Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.

When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.