v 0.10.2 Updated: 1 year, 3 months ago

Fast and Customizable Tokenizers

🤗 Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

To install py39-tokenizers, paste this in macOS terminal after installing MacPorts

sudo port install py39-tokenizers

Add to my watchlist

Installations 3
Requested Installations 1