Build Wiz AI Show

LLMs 101: Tokens, KV Cache and Running Models Locally (Ahmad Osman)

15 min · · English

Ahmad Osman's LLMs 101 guide explains how large language models write one token at a time, and why the KV cache, quantization and VRAM decide what runs on your own machine. Bob and Alice unpack it with everyday metaphors and a test to try at home. In this episode: - Tokens, attention and the KV cache, the model's working memory - Prefill vs decode: the pause before the first word, then the stream - Quantization and VRAM math for local LLMs Sources: Ahmad Osman (@TheAhmadOsman); Jay Alammar; Maarten Grootendorst; Simon Willison Big AI ideas, made clear for builders. luongnv.com/podcast

Listen on Build Wiz AI Show Download MP3 Transcript

Based on LLMs 101: A Practical Guide (2026 Edition) by Ahmad (@TheAhmadOsman)

Transcript

Alice: Why does a local AI model pause before its first word, then type its answer like it's in no hurry?

Bob: Two different jobs! First it processes your whole prompt, then it writes the answer one token at a time.

Alice: So what's really going on inside that loop? Welcome to Build Wiz AI Show. Big AI ideas, made clear for builders. I'm Alice.

Bob: And I'm Bob! Today we unpack Ahmad Osman's X article, his practical guide to LLMs, the 2026 edition.

Alice: A practical guide. So beginners are welcome?

Bob: Very welcome! It starts with the mechanics, like tokens, attention and the KV cache, then moves to the local side: quantization, VRAM math and hardware. We'll follow the core story, the loop and its memory, and what it means for running models yourself; the full guide goes much further, into RAG, agents, fine-tuning and licenses.

Alice: Okay, start at the very beginning. What does an LLM actually do?

Bob: It runs one loop, over and over. Turn your text into tokens, score every possible next token, choose one, append it, and repeat until it stops.

Alice: Wait, tokens? Not words?

Bob: Not words! Tokens are small chunks of text, stored as integer IDs: a whole word like hello, a fragment like inter or national, even a punctuation mark.

Alice: Give me the picture.

Bob: Think of a one-word story game around a campfire: each person adds a single word, and everything said so far shapes the next one. That's the model: it never writes the whole answer in one shot, and every new token joins the sequence that steers the next.

Alice: A campfire that never runs out of words!

Bob: Until a stop token or a limit! And tokens are the unit of work: they decide how much text fits in the context window and how big the cache gets.

Alice: Next stop: attention. What is it?

Bob: Attention is how a token decides which earlier tokens matter for the next prediction. Picture a noisy party: through all the chatter, your ear snaps to the one voice saying your name. In the model, attention picks out the earlier tokens that matter right now.

Alice: Maya, you dug into a classic explainer on this?

Maya: Jay Alammar's The Illustrated Transformer uses one sentence: The animal didn't cross the street because it was too tired. Does it mean the street or the animal? That's simple for a human, not as simple for an algorithm.

Alice: And the model gets it right?

Maya: It does. When the model processes the word it, self-attention lets it associate that word with animal. Alammar calls self-attention the way the Transformer bakes the understanding of other relevant words into the current one.

Alice: Baking. Of course there's baking.

Bob: And attention is one reason local models are so hungry for memory, Ahmad says. Classic multi-head attention keeps separate key and value state for many heads, which makes the KV cache large, while newer designs like GQA share key and value heads across groups.

Alice: So two models of the same size can behave differently?

Bob: Very differently! A 7B model with classic attention at 128K context can exhaust a GPU with 24 gigabytes, while a 7B GQA model may fit with room to spare. Parameter count isn't the whole story.

Alice: You keep saying KV cache. What is it, really?

Bob: It's the model's working memory during generation. It stores the key and value states of earlier tokens, so the model doesn't recompute the whole history for every new token.

Alice: Metaphor, please. I'm the beginner today.

Bob: Imagine a waiter at a big table: without a notepad, he'd make everyone repeat the whole order each time he added one dish. The KV cache is that notepad: earlier tokens are already written down, so without it, generation would be brutally inefficient.

Alice: But notepads fill up.

Bob: Exactly, and that's the bill. For older Llama-like 7B models, his rule of thumb is about half a megabyte per token, so 4K tokens can cost around 2 gigabytes just for the cache. At 32K tokens, you may be looking at 16 gigabytes of cache alone.

Alice: Sixteen just to remember the chat? Ouch!

Bob: Ouch indeed. That's why a model can fit with an empty prompt but crash when you load a long document. The weights fit, the working memory did not.

Alice: Back to our teaser. Why the pause, then the slow typing?

Bob: Two phases! Prefill processes your prompt: paste a 20,000-token document, and the model must process all of it before the first answer token. Then decode generates new tokens one at a time, and that's the streaming typing effect.

Alice: Okay, make it kitchen-simple.

Bob: Think of an exam: you take in the whole question first, then you write your answer in pen, word by word, each word depending on the one before. Reading is prefill, which runs in parallel; writing is decode, which is sequential.

Alice: So which one hurts more?

Bob: Long prompts punish prefill, long answers punish decode, and long conversations punish both, because the KV cache grows.

Alice: That's why endless chats get sluggish?

Bob: Right. Let a chat run to 16K tokens, and you pay the memory cost for all of them on every new token. That's why chat apps that keep infinite history eventually slow down or crash.

Alice: And how does it pick each token? Highest score every time?

Bob: That's one option. The model only scores every possible next token; decoding turns those scores into one actual token. It can take the top one, sample from a narrowed set of likely ones, penalize repetition, or stop at a delimiter.

Alice: And temperature is the famous knob?

Bob: It is. Imagine ordering dinner: low temperature means always ordering your usual, high temperature means letting the chef surprise you. For precise work, start low with short limits and stop sequences; for creative work, give the sampler more room.

Alice: So always ordering the usual is the safe bet?

Bob: Not always! Greedy decoding is often brittle: it can get stuck in loops or give generic answers, because it never explores alternatives. His advice: deterministic settings for evals, and for ideation, let the model breathe.

Alice: He also warns about chat templates. What goes wrong there?

Bob: A chat model was trained on one exact conversation format, with markers for system, user and assistant. Use the wrong one and you get gibberish, ignored system prompts or broken tool calls. Then people decide the model is dumb, when the template is the actual bug.

Alice: Picture it for me.

Bob: It's like a tax form where you write your name in the amount box: the clerk isn't stupid, the form is filled in wrong. Ahmad says to treat the template like an API contract: get it wrong, and you're not testing the model you think you are.

Alice: My name in the amount box. Rich at last! Now the context window. Bigger is better, right?

Bob: Bigger costs more! The context window is the most tokens a model can attend to at once, and in 2026 local models range from 8K and 32K up to 128K and beyond. But supported isn't cheap: a model that technically handles 128K may slow to a crawl at 64K.

Alice: So what's the image here?

Bob: Think of a desk: a bigger desk holds more papers, but you can still lose the one buried at the bottom of the pile. Likewise, a model may handle the end of a long document well while missing details near the beginning.

Alice: So just paste everything and hope?

Bob: No! That's where RAG comes in, retrieval-augmented generation: you retrieve the relevant chunks from a knowledge base and give only those to the model. It's like a librarian who brings you the right few pages instead of dumping the whole library on your desk.

Alice: And if the librarian grabs the wrong book?

Bob: Then the answer suffers. Most bad RAG systems aren't bad because of the model, but because of chunking, retrieval and reranking. A good model can't reliably answer from evidence it never received.

Alice: Switching gears: running models on my own machine. What does local really mean?

Bob: A local LLM is one whose weights and runtime are under your control. That freedom comes with work: you're now the ops team, and when something breaks, there's no support ticket to file.

Alice: So what's the everyday version?

Bob: It's like owning a car instead of taking taxis: you go wherever you want, but the oil changes are yours. And local doesn't automatically mean offline, private, safe or cheap: a local app can still phone home. And it's not always worth it. If you need the absolute best quality and don't have the hardware to match, he says a hosted API is the right tool.

Alice: Okay, quantization. It sounds scary.

Bob: It's simple at heart: store the weights in lower precision to save memory. His 2026 rule of thumb, as of the article's date: Q4 is the default consumer sweet spot, and Q8 is near-lossless for many tasks. Q3 or Q2 only when you must fit a bigger model.

Alice: Lena, you brought a visual guide on this?

Lena: I did. Maarten Grootendorst's A Visual Guide to Quantization does the math: a model with 70 billion parameters in full 32-bit precision would need 280 gigabytes just to load. That's why squeezing the bits is so tempting.

Alice: My laptop just fainted.

Lena: The catch is precision. His picture: it's like redrawing a photo with only 8 colors, and the zoomed-in part turns grainy. Quantization cuts the bits, the colors, while keeping as much precision as it can.

Alice: How low can you go?

Lena: Below 8 bits gets hard, because the error grows with each bit you lose. Smart methods reach 4 bits and even 2, but going lower than 4 bits is typically not advised.

Bob: Ahmad agrees: math, code, structured output and tool use degrade first. And a 7B model at Q6 can beat a 13B model at Q2 on reasoning tasks, using less memory. Don't worship parameter count!

Alice: So how do I know if a model fits on my GPU?

Bob: Three memory eaters: weights, KV cache and runtime overhead. Weights are roughly parameters times bytes per parameter: about 2 bytes in FP16, 1 in Q8, and half a byte in Q4, plus format overhead.

Alice: Give me the everyday version.

Bob: Picture packing a suitcase: the clothes are the weights, and the souvenirs you pick up on the trip are the KV cache. Pack it full on day one and the souvenirs won't fit: a 13B model in Q4 may fit at 8K context, then fail at 32K, because the cache quadrupled.

Alice: So leave room in the suitcase.

Bob: Ten to twenty percent headroom, he says. And the most painful setup is the one that almost fits and spills layers to the CPU: it may technically run, but token speed can collapse.

Alice: Why are two cards with the same memory different speeds?

Bob: Bandwidth! Decode is often memory-bandwidth-bound: the GPU streams the weights over and over. It's like drinking a milkshake through a straw: the width of the straw, not the size of the cup, sets the pace.

Alice: Milkshake engineering. Finally, my expertise! Time for counterpoints. Sam, you brought the skeptic's view.

Sam: I did. Simon Willison, on Simon Willison's Weblog, ran Meta's Llama 3.3 70B on his MacBook with 64 gigabytes of memory, in December 2024. His first try consumed every remaining bit of available memory and hard-crashed his Mac.

Alice: Ouch. The suitcase warning, live.

Sam: It worked only after he closed Firefox and his code editor. Then the benchmark catch: the published scores ran on the full-sized release, while his laptop version was quantized, so not exactly the same model, likely with lower scores.

Alice: And how fast was it?

Sam: His own log shows the prompt being processed several times faster than the answer was generated. It runs, but you wait for every generated token.

Bob: That matches Ahmad's own caution: a BF16 leaderboard score isn't your Q4 local reality. Benchmark the stack you'll actually run.

Alice: My take: local is a real tool, not just a hobby, but test it before you trust it. Builder's Corner! Bob, one thing listeners can try this week?

Bob: Run a small model on your own machine. Ahmad's beginner setup: a recent 4B to 9B instruct model, Q4 quantization, 8K to 32K context, and a built-in chat window in a desktop app.

Alice: And then what do I watch?

Bob: Paste a long document and time the pause before the first word: that's prefill. Then watch the answer stream in: that's decode. Measure both, time to first token and decode tokens per second, just like his benchmark list.

Alice: Any bonus round?

Bob: Download two or three models in the same size class and compare them on the same prompts. Your workload is the benchmark that matters.

Alice: Long prompt, short prompt, stopwatch. Got it. Bob, the takeaways?

Bob: One: the model predicts one token at a time, and tokens aren't words. Two: the KV cache is the hidden memory bill. Three: quantization is a tradeoff, and long context isn't free.

Alice: And his one-line summary?

Bob: Local LLMs are mostly memory math plus formatting plus evaluation. Get those right, and the rest of the stack becomes much easier to reason about.

Alice: What's still open?

Bob: Squeezing the cache below 8 bits, for one. He calls it research-heavy and workload-sensitive, to use only after measuring quality on your own prompts.

Alice: And the model scene changes fast, he says. Which picks will still hold next year?

Bob: Nobody knows yet! That's why he says not to let a leaderboard choose your local stack for you. That's our tour of the token loop. Thanks to Ahmad Osman for his practical guide to LLMs, and to Jay Alammar for The Illustrated Transformer. Thanks also to Maarten Grootendorst for A Visual Guide to Quantization, and to Simon Willison for his post on running a big model on his laptop.

Alice: Thanks, everyone! May your KV cache always fit.

Bob: That's all for today on Build Wiz AI Show. Thanks for listening, and see you next time.