What are tokens and the context window in an LLM?
Tokens are the small pieces of text (whole words, parts of words or symbols) that an LLM reads and writes. The context window is the maximum number of tokens the model can consider at once, including both your input and its output.
Explain it simply
A model does not read text letter by letter or always word by word. It reads tokens. A token is a chunk of text from a fixed vocabulary. Common words like "the" are usually one token. A rarer word like "placement" might be split into "place" and "ment". Spaces and punctuation are often part of tokens too.
The context window is like the desk space you have in a college library. You can spread out only so many pages at once. Whatever is on the desk, you can refer to. Anything that does not fit is simply not there for you. For an LLM, the desk holds the system instructions, the conversation so far, any documents you pasted, and the answer it is writing.
Example
Text: "Unbelievable placements!"
Tokens: ["Un", "believ", "able", " place", "ments", "!"] (illustrative)
Count: 6 tokens for 2 wordsThe exact split depends on the model's tokenizer, so the same sentence can have a different token count on different models. As a rough rule of thumb, English text averages a little less than one word per token, while many Indian languages written in their own scripts take more tokens for the same meaning, because the vocabulary has fewer ready-made pieces for them.
Now a context-window example. Say a model has a window of 8,000 tokens. You paste a 7,500-token document and ask for a summary. Only about 500 tokens are left for the answer, so the summary may be cut short.
How it works
A tokenizer is built before training, usually with an algorithm such as byte-pair encoding. It starts from bytes or characters and repeatedly merges the most frequent pairs into new tokens until it reaches a target vocabulary size. Every token gets an id number, and the model only ever sees these ids.
The context window is a limit set by how the model was built and trained. Attention compares tokens with each other, so longer inputs cost more memory and time. Chat apps keep the whole conversation in the window on each turn, which is why very long chats eventually drop or summarise older messages.
Real-world use
- API usage is usually billed and rate-limited per token, so teams measure token counts to estimate cost.
- A chatbot for a bank keeps only the recent turns plus a short summary so it stays within the window.
- RAG systems split documents into chunks small enough that the relevant ones fit in the window next to the question.
Common mistakes
- Assuming one word equals one token. Counts vary by language and tokenizer.
- Forgetting that the output also uses the window. A long prompt leaves less room for the answer.
- Assuming a bigger window means the model uses every part equally well. Models can miss details buried in the middle of a very long input, so put key instructions clearly and keep inputs focused.
What to say in the interview
"Tokens are the units an LLM works with, produced by a tokenizer that splits text into subword pieces and maps each to an id. A word can be one token or several, depending on the word and the language. The context window is the maximum number of tokens the model can attend to in one call, counting the prompt, any history or documents, and the generated output. If the conversation is longer than the window, older content has to be dropped, summarised or retrieved selectively."
Follow-up questions
- Why do models use subword tokens instead of whole words or single characters?
- How would you handle a document that is longer than the context window?
- Why does a long chat get slower or more expensive over time?