py-tokenizers

v 0.12.1 Updated: 3 months, 1 week ago

Fast and Customizable Tokenizers

🤗 Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

https://github.com/huggingface/tokenizers

Installable ports:


Add to my watchlist

Installations 0
Requested Installations 0