py-tokenizers (python/py-tokenizers) Updated: 1 year, 10 months ago Add to my watchlist
Fast and Customizable TokenizersTokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
Version: 0.13.3 License: Apache-2
GitHub
| Maintainers | No Maintainer |
| Categories | python |
| Homepage | https://github.com/huggingface/tokenizers |
| Platforms | any |
| Variants | - |
Subport(s) (5)
"py-tokenizers" depends on
lib (2)
build (4)
Ports that depend on "py-tokenizers"
No ports
Port Health:
- Sequoia (arm64)
- Sequoia (x86_64)
- Sonoma (arm64)
- Sonoma (x86_64)
- Ventura (arm64)
- Ventura (x86_64)
- Monterey (arm64)
- Monterey (x86_64)
- Big Sur (arm64)
- Big Sur (x86_64)
- Catalina
- Mojave
- High Sierra
- Sierra
- El Capitan
- Yosemite
- Mavericks
- Mountain Lion
- Lion
- Snow Leopard (x86_64)
- Snow Leopard (i386)
- Leopard (ppc legacy)
- No history in app's database
Installations (30 days)
0
Requested Installations (30 days)
0
Livecheck results
py-tokenizers seems to have been updated (port version: 0.13.3, new version: 0.21.1)
livecheck ran: 2 days ago