v 0.5.2 Updated: 3 months, 3 weeks ago
Fast BPE tokeniser for use with OpenAI's models.
https://github.com/openai/tiktoken