Course 6, lesson 52 of 100, Ages 12+

Tokens

Words in pieces

Like I’m 5

Tokens are like building blocks for words. Chatbots break words into small blocks, then build their answers block by block.

The big idea

LLMs don't read whole words like we do. They split text into tokens: common words are one token, while rare words are split into pieces, like 'un', 'believ' and 'able'.

Everything is counted in tokens: how much text a model can read at once, how fast it replies and how much it costs to use. Roughly, one English token is about three-quarters of a word.

Examples

  • Splitting: 'Playground' might become 'play' and 'ground'.
  • Counting: A 1,000-word essay is roughly 1,300 tokens.
  • Other languages: Some languages need more tokens for the same sentence.

How it works

  1. Chatbots don’t read whole words. They read small pieces called tokens.
  2. Short words are often one token. Longer words get split, like ‘dream’ and ‘ing’.
  3. Every reply is built one token at a time, and each token is a prediction.

Check your understanding

What is a token?
Options: A small piece of text; A coin for games; A kind of robot.
Answer: A small piece of text. Tokens are the bite-sized pieces of text an AI reads and writes.
Roughly how many tokens is 100 English words?
Options: About 130; About 10; About 10,000.
Answer: About 130. One token is about three-quarters of a word, so 100 words is roughly 130 tokens.

Remember

Tokens are the small pieces of text AI reads and writes.

Talk about it

How would you split “unbelievable” into smaller pieces?

Go deeper

Tokenisers split text into sub-word units so a fixed vocabulary can represent any word in any language. Models have a context window measured in tokens, which limits how much they can consider at once, and AI services are often priced per token.