Tokens, context windows and why your bill exploded
What a token really is, why a 1M context window isn't free, and the two settings that halve your spend.
A token is roughly four characters of English — not a word, not a character. "Understanding" is two tokens; a UUID can be twenty. Every model prices input and output tokens separately, and output is usually three to five times more expensive.
The context window is the maximum number of tokens the model can see in one call: your system prompt, the retrieved documents, the conversation history and the answer it is about to write. It is a ceiling, not an allowance — but you pay for everything you actually put inside it, on every single call.
That is why bills explode without traffic growing. A chat feature that replays the whole conversation each turn has costs that grow quadratically with conversation length. Add a naive retrieval step that stuffs twenty documents into every prompt and you are paying for a small book per question.
The shape of it
- 01
System prompt
Fixed cost, paid on every call. Keep it short and version it.
- 02
History
Grows every turn. Summarise or window it after N turns.
- 03
Retrieved context
Usually the largest slice. Cap the number of chunks.
- 04
Output
The expensive tokens. Set max_tokens deliberately.
Want this applied to your situation?
Sessions are direct and specific — you leave with a decision, not a reading list.
Book a session