v 0.13.0 Updated: 3 months, 2 weeks ago
tiktoken is a fast BPE tokeniser for use with OpenAI's models
https://github.com/openai/tiktoken