tiktoken

Tech stack & purpose

tiktoken is a fast byte pair encoding tokenizer built for use with OpenAI's language models. Written in Python and available via PyPI, it provides APIs for encoding text into tokens and decoding tokens back to text, with performance reported to be 3-6x faster than comparable open source tokenizers. The project includes an educational submodule for learning about BPE mechanics and supports extension through both direct instantiation of Encoding objects and a plugin mechanism via namespace packages.

Community & reference links

• BPE

Languages

Contributors

• 
  Shantanu29 commits · 1,026 followers
• Shantanu Jain16 commits
• Lőrinc3 commits
• Nathan Goldbaum2 commits
• Nick Stathas1 commits
• Mukunda Rao Katta1 commits
• Alvaro Bartolome1 commits
• Maxime Rey1 commits
• Praneet Nigam1 commits
• l0rinc1 commits
• Lőrinc Pap1 commits
• Ted Sanders1 commits
• Nguyen-Khanh Vu1 commits
• Matt Welsh1 commits
• Rene Leonhardt1 commits
• Fritz Obermeyer1 commits
• Mariatta Wijaya1 commits
• Drew Hintz1 commits
• Logan Kilpatrick1 commits
• Xing Han Lu1 commits
• Salman Chishti1 commits
• Henrik Torget1 commits
• Raj Arun1 commits
• jonathanagustin1 commits
• Arvid Lunnemark1 commits
• youkaichao1 commits