Why do LLMs often fail at counting letters in a word?
The model sees tokens, not individual letters, so a word like "strawberry" may arrive as two or three chunks and the model never directly sees its spelling. It must guess letter counts from patterns, which is unreliable. Using code for counting fixes this.
Explain it simply
Imagine being shown a word only as a few sealed packets and asked how many times the letter "r" appears inside. You never opened the packets, so you can only guess from memory of the word.
An LLM reads tokens such as "str", "aw", "berry". The letters are hidden inside each token's vector. The model has learned some spelling facts from text that discusses spelling, but it has no built-in step that loops over characters. Reversing strings, counting letters and exact character-level edits are weak spots for the same reason.
The reliable approach is to hand exact work to a tool. Let the model write a line of code, or call a function, and run it.
Example
word = "strawberry"
print(word.count("r")) # 3An LLM asked directly may answer 2. A one-line tool call gets it right every time.
What to say in the interview
"LLMs operate on tokens, not characters, so character-level tasks like counting letters or reversing strings are error-prone. For exact tasks I route to code or a tool and let the model orchestrate."
Common mistakes
- Calling this a reasoning failure only. Tokenization is a major cause.