How LLM Inference Really Works: Prefill vs Decode

By Hamza Boughanim · 2026-07-28 · 12 min read

Why prefill is compute-bound and decode memory-bound, what TTFT and TPOT measure, and how continuous batching and PagedAttention keep GPUs busy.

Introduction: One Model, Two Completely Different Jobs

Ask an LLM a question and two things happen that feel identical to the user but are almost nothing alike under the hood. The model reads your entire prompt almost instantly, then writes its answer one word at a time, visibly streaming in. That asymmetry isn't an accident of UI design — it's the direct result of two distinct computational phases inside every autoregressive transformer: prefill and decode.

Understanding this split is not a trivia fact. It is the single most important mental model for reasoning about LLM serving cost, latency, and throughput. Every major inference engine — vLLM, TensorRT-LLM, TGI, SGLang — is architected around exploiting the difference between these two phases. This article walks through both phases in depth, explains why the GPU behaves so differently in each, and shows how techniques like continuous batching, chunked prefill, and PagedAttention emerged directly from that asymmetry.

Prefill: Reading the Prompt in One Parallel Pass

When a prompt like "The cat sat down." arrives, the model tokenizes it and pushes every token through the transformer's layers simultaneously, in a single forward pass. Because every token's position and content are already known, there is no sequential dependency preventing the model from processing them all at once.

This matters enormously for how the GPU spends its time. Processing many tokens together lets the attention and feedforward computations be expressed as large, dense matrix multiplications (GEMMs) — exactly the workload GPU tensor cores are built to saturate. During prefill, the model also builds the KV cache: the key and value vectors for every token, at every attention layer, stored so they don't have to be recomputed later.

import torch

def prefill(model, input_ids):
    """
    input_ids: (batch, seq_len) — the full prompt, all tokens known upfront
    Returns logits for the next token AND a populated KV cache.
    """
    with torch.no_grad():
        # One forward pass processes every prompt token in parallel
        outputs = model(input_ids, use_cache=True)
    logits = outputs.logits[:, -1, :]          # next-token distribution
    kv_cache = outputs.past_key_values         # keys/values for every prompt token
    return logits, kv_cache

Because prefill turns into large, batched matrix math, it is typically compute-bound: the GPU's tensor cores stay busy, and FLOPs — not memory traffic — are the limiting factor. The wall-clock time from when a request arrives to when the very first output token appears is called Time to First Token (TTFT), and it is almost entirely a function of prefill cost.

Decode: Generating One Token at a Time

Once the first token is produced, generation becomes strictly sequential. The model predicts a single new token, appends its key/value vectors to the KV cache, feeds that token back in, and predicts the next one. This repeats until an end-of-sequence token or a length limit is hit.

def decode_step(model, last_token_id, kv_cache):
    """
    last_token_id: (batch, 1) — only the most recently generated token
    kv_cache: everything computed so far, reused instead of recomputed
    """
    with torch.no_grad():
        outputs = model(last_token_id, past_key_values=kv_cache, use_cache=True)
    logits = outputs.logits[:, -1, :]
    new_kv_cache = outputs.past_key_values     # grown by one more token
    next_token = torch.argmax(logits, dim=-1)
    return next_token, new_kv_cache

def generate(model, prompt_ids, max_new_tokens=50):
    logits, kv_cache = prefill(model, prompt_ids)
    next_token = torch.argmax(logits, dim=-1, keepdim=True)
    generated = [next_token]

    for _ in range(max_new_tokens - 1):
        next_token, kv_cache = decode_step(model, next_token, kv_cache)
        generated.append(next_token)

    return torch.cat(generated, dim=1)

The compute done per decode step is tiny — one token, not thousands. But every step still has to read the entire KV cache from GPU memory to compute attention against it, and that cache grows with every token generated. Little compute, lots of memory traffic: decode is memory-bandwidth-bound. This is the counterintuitive part — a GPU can look almost idle during decode, with low tensor-core utilization, precisely because it's spending most of its cycles waiting on memory reads rather than computing.

The delay between each streamed token is called Time per Output Token (TPOT), and it's what a user actually perceives as "how fast the response feels" once streaming has started.

Seeing It on a Roofline Plot

The roofline model is the standard tool for reasoning about whether a workload is compute-bound or memory-bound. It plots achievable performance against arithmetic intensity — FLOPs performed per byte moved from memory.

Same weights, same hardware, same model — a completely different bottleneck depending on which phase is running. This single fact is why serious inference engines tune prefill and decode separately, and why some large-scale deployments even run them on physically different hardware pools in what's called disaggregated serving — dedicating compute-heavy GPUs to prefill and memory-bandwidth-heavy GPUs to decode, then shipping the KV cache between them.

Why the KV Cache Is the Real Bottleneck Resource

The KV cache is what makes decode fast at all — without it, generating token n+1 would require reprocessing every prior token from scratch. But that speed comes at a memory cost that scales with sequence length, batch size, number of layers, and number of attention heads. Long contexts from a RAG retrieval pipeline inflate this cost directly.

KV cache size (bytes) ≈
    2 (K and V) × num_layers × num_heads × head_dim × seq_len × batch_size × bytes_per_element

Example — a 7B model, 32 layers, 32 heads, head_dim 128, FP16:
  Per token per sequence: 2 × 32 × 32 × 128 × 2 bytes ≈ 524 KB
  A single 4,000-token conversation: ~2.1 GB of KV cache — before you serve a second user.

This is why techniques like Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) — which share key/value projections across multiple attention heads — exist: they shrink the cache directly, cutting the memory traffic decode has to pay for on every single step. It's also why quantized KV caches (storing keys and values in INT8 or FP8 instead of FP16) have become common in production serving stacks: halving cache precision roughly halves the memory bandwidth decode is bottlenecked on.

From One Request to Many: Why Naive Batching Fails

Serving one user at a time wastes almost all of a modern GPU's capacity — decode's low arithmetic intensity means there's plenty of spare compute sitting idle. The obvious fix is batching multiple requests together. The naive way to do this is static batching: collect a fixed group of requests, run them together, and wait for every request in the group to finish before starting the next batch.

Static batching has an obvious flaw: requests don't finish at the same time. A short "yes/no" answer completes in a handful of decode steps; a long explanation might take hundreds. In a static batch, the GPU slot for the short request sits idle — reserved but unused — until the longest request in the batch finally finishes.

def static_batch_step(model, batch_state):
    """
    Every sequence in the batch advances one decode step together,
    even if some sequences finished generating long ago.
    """
    active = [s for s in batch_state if not s.finished]
    if not active:
        return batch_state  # whole batch idles until ALL are done

    for seq in active:
        seq.next_token, seq.kv_cache = decode_step(model, seq.last_token, seq.kv_cache)
        seq.finished = seq.next_token == EOS_TOKEN

    return batch_state
    # Problem: a finished sequence's GPU slot can't be reused
    # until every sequence in this batch is finished.

Continuous Batching: Filling the Gaps as They Open

Continuous batching (also called in-flight batching, the core idea behind vLLM and TensorRT-LLM) treats the batch as a rolling window instead of a fixed group. At every scheduling step, any sequence that has finished is evicted immediately, and a new request from the queue takes its slot — without waiting for the rest of the batch to catch up.

class ContinuousBatchScheduler:
    def __init__(self, model, max_batch_size=32):
        self.model = model
        self.max_batch_size = max_batch_size
        self.active = []      # sequences currently being decoded
        self.waiting = []     # queued requests not yet admitted

    def step(self):
        # 1. Evict finished sequences, freeing their slots immediately
        self.active = [s for s in self.active if not s.finished]

        # 2. Admit new requests into any freed slots — this is where a
        #    new request's PREFILL can be scheduled alongside other
        #    requests' DECODE steps in the very same batch
        while len(self.active) < self.max_batch_size and self.waiting:
            new_seq = self.waiting.pop(0)
            new_seq.logits, new_seq.kv_cache = prefill(self.model, new_seq.prompt_ids)
            new_seq.last_token = torch.argmax(new_seq.logits, dim=-1, keepdim=True)
            self.active.append(new_seq)

        # 3. Advance every active sequence by exactly one decode step
        for seq in self.active:
            seq.last_token, seq.kv_cache = decode_step(self.model, seq.last_token, seq.kv_cache)
            seq.finished = seq.last_token.item() == EOS_TOKEN

    def add_request(self, prompt_ids):
        self.waiting.append(Sequence(prompt_ids))

The key insight is in step 2: because prefill and decode are just different shapes of the same underlying attention/GEMM computation, an inference engine can mix a newly arrived request's prefill work with ongoing decode work from other requests in the same GPU batch step. A finished request's compute-idle slot gets reused within milliseconds instead of sitting empty for the duration of the slowest sequence in a static batch.

Static vs. Continuous Batching, Side by Side

Chunked Prefill: Stopping Long Prompts From Blocking Everyone

Continuous batching solves the "short request waits on long request" problem for decode, but it introduces a new one: a very long incoming prompt's prefill can itself be a large chunk of compute that, if run in one shot, delays the decode steps of every other in-flight request sharing that batch step — hurting their TPOT.

Chunked prefill fixes this by splitting a long prompt's prefill into smaller pieces and interleaving those chunks with ongoing decode steps, rather than running the entire prefill as one uninterrupted burst. The long request's TTFT goes up slightly, but every other request's decode latency stays smooth and predictable — a deliberate tradeoff that most production schedulers make by default.

PagedAttention: Solving KV Cache Fragmentation

Continuous batching creates a new memory-management problem: KV caches for dozens of simultaneously active sequences, each growing at a different, unpredictable rate, need to live in GPU memory without wasting space or fragmenting it. Naively pre-allocating a large contiguous block per sequence (sized for the maximum possible length) wastes enormous amounts of memory on sequences that finish early.

PagedAttention (introduced by vLLM) borrows the idea of virtual memory paging from operating systems. Instead of one contiguous KV cache per sequence, the cache is split into fixed-size blocks that can be allocated non-contiguously and mapped through a lightweight block table — the same way an OS maps virtual pages to physical memory frames.

Without paging:  [------- reserved for max_len, mostly unused -------]
With paging:     [block][block][block] ← only allocated as tokens are actually generated
                     ↓       ↓       ↓
                 physical GPU memory, reused across sequences as they finish

This eliminates internal fragmentation, lets memory be shared between sequences that share a common prefix (like a system prompt reused across many requests), and — critically — lets the scheduler admit far more concurrent sequences into a continuous batch, because memory is no longer reserved pessimistically for the worst case.

Measuring Serving Performance: The Metrics That Matter

These metrics trade off against each other. Packing more sequences into a batch to raise throughput increases contention for memory bandwidth during decode, which raises TPOT for every request in that batch. Prioritizing a new request's prefill to keep TTFT low can starve ongoing decode steps of scheduling slots, raising their TPOT. Every production inference engine exposes scheduler knobs — max batch size, chunked-prefill chunk size, priority policies — precisely to let operators tune this tradeoff for their traffic pattern. Exposing those knobs safely is a deployment concern as much as a modelling one.

Putting It Together: A Simplified End-to-End Serving Loop

import time

def serve_loop(scheduler, poll_interval=0.001):
    """
    The core loop any production inference server runs: keep stepping
    the batch, admitting new work into freed slots, until shut down.
    """
    while True:
        scheduler.step()  # one prefill/decode-mixed batch iteration on the GPU

        for seq in scheduler.active:
            if seq.just_produced_token:
                stream_token_to_client(seq.request_id, seq.last_token)

        if not scheduler.active and not scheduler.waiting:
            time.sleep(poll_interval)  # nothing to do — wait for new requests

Every piece discussed above — prefill's compute-bound burst, decode's memory-bound trickle, continuous batching's slot reuse, chunked prefill's fairness, and PagedAttention's memory efficiency — exists to keep this loop's GPU utilization as close to 100% as possible without degrading per-request latency below what users will tolerate.

Practical Takeaways for Engineers

• Prefill and decode are different workloads on the same hardware. Optimize them separately — what speeds up one can be irrelevant, or even harmful, to the other.
• TTFT tracks prefill; TPOT tracks decode. If a user complains about a slow first response, look at prompt length and queue depth. If they complain about slow streaming, look at context length and batch contention.
• Continuous batching is not optional at scale. Static batching leaves the majority of a GPU's capacity idle on any workload with variable response lengths.
• KV cache size is the real capacity constraint, not raw VRAM. GQA/MQA, quantized caches, and PagedAttention all exist to fight the same problem: the cache growing faster than useful compute.
• Long prompts are a scheduling hazard, not just a latency cost. Chunked prefill exists specifically to stop one large request from degrading everyone else's experience.

Final Thought

It's tempting to think of "running an LLM" as a single, undifferentiated act of inference. In practice, serving a model well means constantly juggling two workloads with opposite hardware appetites — one that wants dense compute, one that wants fast memory — across dozens of concurrent, unpredictable requests. Continuous batching, chunked prefill, and PagedAttention aren't clever tricks bolted onto inference; they're the direct engineering answer to the prefill/decode split itself. Once you see LLM serving as a compute, memory, and scheduling problem rather than one big model call, the design of every modern inference engine stops looking arbitrary and starts looking inevitable.

Further Reading

Related deep dives on this site, if you're building or serving LLM-backed systems:

• Why GPUs Power Modern AI — the memory-bandwidth and tensor-core fundamentals behind the prefill/decode split
• Building a Production RAG Pipeline — where long retrieved contexts come from, and why they inflate prefill cost
• The Agentic Loop Explained — why agent workloads hammer TTFT with many short, bursty requests
• MLOps in Practice: Docker and FastAPI — packaging an inference service for production
• More AI engineering projects on this site

Primary sources and reference implementations for the techniques covered above:

• Efficient Memory Management for LLM Serving with PagedAttention — Kwon et al., the original vLLM paper
• vLLM documentation — continuous batching and chunked prefill scheduler configuration
• NVIDIA TensorRT-LLM — in-flight batching and quantized KV cache implementations
• Hugging Face Text Generation Inference — production serving reference
• GQA: Training Generalized Multi-Query Transformer Models — Ainslie et al., the grouped-query attention paper