ttok 1.0 Updates Token Counting Tool to Match New OpenAI Models

A smarter default for token counting

Simon Willison released ttok 1.0 on October 9, 2026. The tool helps developers count and truncate text based on tokens — the chunks of text that OpenAI and other AI companies use to process language.

The update is more than a routine version bump. It changes the default tokenizer from the one used by GPT-4 to the one used by GPT-5 and GPT-6. Willison said the switch was a good excuse to finally ship a 1.0 release after users reported that newer models were not being counted correctly.

Why the switch matters

Tokenizers split text into pieces that AI models can understand. If a tool uses the wrong tokenizer, it will count tokens incorrectly, which can cause code to cut off text too early or use too much memory. Getting this right matters for anyone building applications on top of large language models.

GPT-5 and GPT-6 are both believed to use the same tokenizer. Willison pointed to a commit by researcher William Liu that tested several GPT models, including GPT-5.5, GPT-5.6, and the different GPT-6 variants known as Astra, Sol, Luna, and Cyber.

All seven models reported 44,794 tokens and matched on every test case. GPT-6 did not introduce any change to the input count, according to the experiment ¹.

OpenAI has not officially confirmed it yet

While the evidence is promising, OpenAI has not publicly confirmed that GPT-6 uses the same tokenizer as the GPT-5 family. There is an open issue discussing the question, and some people in the community have called it frustrating ¹.

That said, the data from Liu's experiment gives developers a strong reason to trust the new default. Willison's tool now reflects that understanding, and anyone upgrading from ttok 0.4 should see more accurate token counts when working with the latest OpenAI models.

What ttok does

ttok is a command-line tool that takes text input and either counts how many tokens it contains or trims it to a specified length. It is useful for developers who need to stay within token limits when sending prompts to AI models.

With the 1.0 release, the tool defaults to the tokenizer shared by GPT-5 and GPT-6, making it easier to work with current OpenAI models without having to manually specify which system to use.

Key facts

• ttok 1.0 now defaults to the tokenizer used by GPT-5 and GPT-6
• William Liu's experiment showed seven GPT models all reporting 44,794 tokens
• OpenAI has not officially confirmed that GPT-6 shares the GPT-5 tokenizer
• The tool counts and truncates text based on tokens for AI applications
• The change fixes incorrect token counts that affected newer model support

Sources

1. Release: ttok 1.0