v 0.13.0 Updated: 3 weeks, 4 days ago
tiktoken is a fast BPE tokeniser for use with OpenAI's models
https://github.com/openai/tiktoken