v 0.22.2 Updated: 2 weeks, 6 days ago
Fast and Customizable Tokenizers
Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Includes BPE, WordPiece, and Unigram tokenizer implementations.
https://github.com/huggingface/tokenizers
| Installations | 0 |
| Requested Installations | 0 |