lucascs@proton.me Lucas C.S. tiktoken is a byte pair encoding (BPE) tokeniser from OpenAI. It splits text into the integer tokens that GPT-family models actually consume, and decodes them back, which is what token counting, context-window budgeting and prompt truncation are built on. The hot loop is a Rust extension module bound through pyo3, so it is roughly an order of magnitude faster than pure-Python tokenisers. Encoding tables (r50k_base, p50k_base, cl100k_base, o200k_base, o200k_harmony) are downloaded on first use and cached locally. openai/tiktoken tiktoken https://github.com/openai/tiktoken/issues https://github.com/openai/tiktoken/blob/main/CHANGELOG.md