py310-tokenizers (python/py-tokenizers) Updated: 1 day, 6 hours ago Add to my watchlist
Fast and Customizable TokenizersTokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Includes BPE, WordPiece, and Unigram tokenizer implementations.
Version: 0.22.2 License: Apache-2
GitHub
| Maintainers | No Maintainer |
| Categories | python |
| Homepage | https://github.com/huggingface/tokenizers |
| Platforms | darwin |
| Variants |
|
Subport(s) (8)
"py310-tokenizers" depends on
lib (3)
build (8)
Ports that depend on "py310-tokenizers"
lib (1)run (1)
Port Health:
Loading Port Health