Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Includes BPE, WordPiece, and Unigram tokenizer implementations.
Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Includes BPE, WordPiece, and Unigram tokenizer implementations.
To install py310-tokenizers, run the following command in macOS terminal (Applications->Utilities->Terminal)
sudo port install py310-tokenizers
To see what files were installed by py310-tokenizers, run:
port contents py310-tokenizers
To later upgrade py310-tokenizers, run:
sudo port selfupdate && sudo port upgrade py310-tokenizers
Reporting an issue on MacPorts Trac
The MacPorts Project uses a system called Trac to file tickets to report bugs and enhancement requests.
Though anyone may search Trac for tickets, you must have a GitHub account in order to login to Trac to create tickets.