Estimated reading time at 200 wpm: 9 minutes
Most people will work with AI models via a standard interface in their browsers when they log in. Those are free or paid subscription packages. Users often don’t think about what’s going on under the ‘hood’, until they are limited by some feature. Then they have take a break for a few hours for the limit to reset or pay money to get on with work.
Whether or not you agree our Fat Disclaimer applies
Understanding what’s under the hood comes from the world of API access to AI services, like OpenRouter. In that domain one gets rich granular detail on every fraction of a penny that is costed and how your money is working.
But there is a wall of jargon that is not easy to grasp. Terms like context windows, cache hits, and quantisation appear on every billing dashboard and settings panel, often sounding far more complicated than they actually are.
Most of these concepts have straightforward equivalents in how the human brain reads, remembers, and speaks. This article will aim to map the technical language onto everyday human habits making it much easier to understand what a model is actually doing—and why it behaves the way it does.
Context Window
Think of this as the size of your physical desk. Everything you are actively working with—reference books, scribbled notes from earlier in the afternoon, and the question someone just asked you—has to fit on that desk at the exact same time. If the desk is large, you can spread out a 600-page manuscript and read it all without putting anything away. Once the desk is completely full, you have to push the oldest papers off the back edge onto the floor just to make room for a new sheet. Those papers on the floor are forgotten; the model genuinely cannot see them anymore.
Tokens
If the context window is the desk, tokens are the actual pieces of paper or index cards you place on it. Humans think in words, but models read and calculate in fragments of roughly three-quarters of a word per token. Each token is a single unit of computation, and pricing them works much like buying biscuits by the wholesale crate.
The price for a million tokens shifts depending on the recipe and the shop. A plain Digestive is cheap to produce, whereas an elaborate chocolate shortbread takes far more effort in the kitchen; similarly, a small, fast model charges pennies per million tokens while a heavy frontier model commands a steep premium for the deeper calculation behind every fragment.
Where you buy matters just as much. You can purchase the exact same packet of Hobnobs from a budget wholesaler, a standard supermarket, or a motorway service station. In AI world when purchasing through a router, the underlying model may be identical, yet one provider sells those tokens at a discount using spare capacity while a priority route charges extra for guaranteed stock and faster service. Even the slicing varies by brand: because each model family uses its own dictionary of word fragments, the exact same English paragraph might break into 100 tokens on one model and 115 on another.
System Prompt
This is your job description or the specific role you adopt before a meeting starts. Before anyone says a word, you already know whether you are supposed to act like a sharp legal editor, a cynical observer, or a patient teacher. It sits permanently at the top-left corner of your desk, colouring how you interpret everything else placed in front of you.
Cache Hit vs. Cache Miss
A cache hit is like walking into a room you left two minutes ago. You do not need to scan the furniture, check the exits, or orient yourself because the mental map is still active in your working memory. Wait ten minutes, get distracted by something else, and that map fades. When you walk back in, you have to re-read the room from scratch, which is a cache miss. So in AI world it’s the same: if the server recognises the data as recent and processed it does not need to do that work again. You get a discount. For some models that could be 80 to 95%. Servers do not keep the data forever, and nor does your brain when you read an article. By the time you got near to the end of an article most of the previous is not exactly in your head. Some meaning may remain but not the exact words or their precise meaning. This is similar to what happens with the models. That’s the next concept.
TTL (Time to Live) vs. LRU (Least Recently Used)
These are the two rules governing how long that mental map of the room or your reading of an article stays alive before fading. TTL is a strict stopwatch: if nobody mentions the topic for five minutes, your brain automatically wipes the slate clean to free up energy. LRU is a crowded waiting room with a fixed number of chairs: your memory stays intact for as long as you keep referencing it, even if hours pass, unless a rush of new visitors arrives and forces you to bump whoever has been sitting quietly the longest out the door.
Context Rot (Lost in the Middle)
Even when you have a massive desk and manage to fit a 500-page file on it, human attention naturally flags in the middle. You remember the opening brief clearly because it set the stage, and you remember the final page because you just finished reading it. The details buried on page 240 tend to blur together unless something draws your eye back to them. Models suffer from this exact same fatigue. Their recall is sharpest at the very start and very end of a long prompt, while facts stuffed into the middle can occasionally get glossed over.
Temperature
This is your level of mental restraint versus improvisation. At a temperature of zero, you are reading strictly from a script or giving the single most predictable, textbook answer every time. Turn the temperature up, and it is a bit like having a glass of wine: you start making looser associations, reaching for less obvious vocabulary, and taking creative risks. Turn it up too high, and you start rambling incoherently.
Top-P (Nucleus Sampling)
If temperature is how loose or tipsy you feel, top-p is your internal editor deciding which words are allowed through the door. Imagine you are about to finish the sentence, “The dog barked at the…” Your brain brings up a ranked list of possibilities: postman (60% likely), cat (25%), shadow (10%), and refrigerator (0.01%). Setting top-p to 0.90 means your mental filter draws a line where the cumulative probability hits 90%, allowing you to pick only from postman, cat, and shadow. Everything below that line gets discarded before you even consider it, keeping you from blurting out refrigerator even when feeling adventurous.
Reasoning / Thinking Tokens
Some models generate thinking blocks before they answer. In human terms, this is the internal monologue or rough scribbling on a scrap pad before speaking out loud. You draft a thought, realise it does not quite hold up, cross it out, try another angle, and only then clear your throat to deliver the final sentence. You still expend mental energy doing the rough work, even though the listener only hears the polished conclusion.
Prefill vs. Generation (Input vs. Output)
Reading a dense document takes far less effort and time than writing a thoughtful response to it. Prefill is the act of reading—scanning all the pages currently on your desk to absorb the situation. Generation is the act of writing—producing new words one at a time, where every new word depends on the exact words put down a second ago. That is why reading a massive prompt happens in a flash, while watching a long answer type out takes noticeably longer.
Latency and Throughput
In human conversation, latency is simply reaction time: the total delay between someone finishing their question and you completing your answer. It combines two different things: how long it takes you to start talking (Time to First Token) and how fast you actually speak once you begin. That speaking speed is throughput, measured in tokens per second. A person who pauses for four seconds to think, then speaks at a brisk, steady clip, has high initial latency but strong throughput.
Quantisation
Think of quantisation as working from a compressed mental summary or a slightly blurry photocopy of a textbook rather than the pristine, original edition. A full-precision model stores every tiny nuance of what it learned during training with microscopic exactness. Quantising it rounds off those numbers to save memory and run faster, much like an experienced specialist relying on sharp rules of thumb rather than recalling every decimal point from a manual. Most of the time, the judgment is indistinguishable from the original, though on exceptionally subtle problems, a heavily quantised model might miss a fine distinction.
Hallucination
This is pure confabulation. When there is a gap in memory or knowledge, the brain dislikes leaving a blank space, so it seamlessly stitches together a plausible-sounding bridge using adjacent facts and pattern recognition. There is no intent to deceive; the model delivers the fabricated detail with the exact same tone, confidence, and fluency as an established fact because, to its internal wiring, it feels like the natural continuation of the sentence.
Putting This Into Practice
Understanding these mechanics shifts how you work with models from guesswork to deliberate control. When you know how the desk, the biscuit crate, and the waiting room behave, you stop paying frontier prices for routine chores.
You can send bulk extraction or initial sorting to a cheap, fast model, reserve expensive reasoning tokens for the final synthesis, and keep your prompts structured so the system reads the static background once and hits the cache on every follow-up question.
Placing critical instructions at the very start or end of a prompt prevents important details from getting lost in the middle, while choosing the right provider route avoids paying priority rates when a standard queue will do.
Instead of treating the API like a mysterious black box with an unpredictable meter, you can match the tool, the memory window, and the price tier to the actual weight of the job in front of you.
