How it worksttok 1.0 Updates Token Counting Tool to Match New OpenAI ModelsHerald of AIPaul Downey from Berkhamsted, UK, CC BY 2.0, via Wikimedia Commons
How it works
ttok 1.0 Updates Token Counting Tool to Match New OpenAI Models
Simon Willison releases an updated tool that now defaults to the tokenizer used by GPT-5 and GPT-6, after a recent experiment confirmed they share the same system.