LLM Inference Metrics You Should Always Track

If you’ve ever deployed an LLM in production and had someone ask “why does it feel slow sometimes?”, you’ve probably learned the hard way that “latency” isn’t one number — it’s several different numbers hiding inside a single request, each with a different cause and a different fix. Understanding these metrics individually is the difference between guessing at performance problems and actually diagnosing them.

Every LLM inference request goes through three phases — Prefill, Decode, and Serving — and underneath all three sits a set of GPU and memory metrics that quietly determine how well the other three phases perform. This post walks through all of them.

The Big Picture: What Actually Happens During Inference

Before diving into individual metrics, it helps to see the shape of a single request:

Request arrives
      |
      v
  PREFILL PHASE
  (process the whole prompt at once,
   build the KV cache)
      |
      v
  First token generated  <-- Time to First Token (TTFT) ends here
      |
      v
  DECODE PHASE
  (generate one token at a time,
   reusing the KV cache)
      |
      v
  Final token generated
      |
      v
  Response returned  <-- End-to-End Latency ends here

Prefill and Decode are fundamentally different workloads — Prefill processes your entire prompt in parallel (compute-heavy), while Decode generates tokens one at a time, sequentially (memory-bandwidth-heavy). This is exactly why they need separate metrics: a slow prefill and a slow decode point to completely different bottlenecks and completely different fixes.

1. Prefill Metrics

Prefill is what happens the moment your prompt hits the model — it’s processed all at once to build the initial context (the KV cache) the model will reference while generating each new token.

Time to First Token (TTFT) The time until the first generated token appears. This is the number your users feel most directly — it’s the “is this thing frozen?” moment between hitting send and seeing anything happen. For chat interfaces, TTFT is often the single most important perceived-latency metric.

Prefill Latency The time spent processing the prompt and building the KV cache. This is the underlying cause behind TTFT — if TTFT is high, prefill latency is almost always where the problem lives.

Prompt Throughput Prompt tokens processed per second. This tells you how efficiently your system chews through input text — useful for understanding how well you’re utilizing compute during the prefill phase specifically.

Prompt Length The average number of input tokens, which directly affects both latency and memory usage. Longer prompts mean more work during prefill and a bigger KV cache to build — this is why RAG systems or agents with long conversation histories often feel slower to start responding, even if generation itself is fast.

2. Decode Metrics

Once the first token is out, the model shifts into Decode — generating the rest of the response one token at a time, each new token depending on everything generated before it.

Inter-Token Latency (ITL) The time between consecutive output tokens. This is what determines whether a response feels like it’s “streaming smoothly” or “stuttering” — even if TTFT is fast, high ITL makes the rest of the response feel sluggish.

Decode Latency The time required to generate each token. Closely related to ITL, but typically used to describe the phase’s overall performance rather than the gap between two specific tokens.

Output Tokens/sec Generation throughput — how many tokens the system produces per second during decode. This is the number that most directly determines how fast a full response streams to a user.

Time Per Output Token (TPOT) The average time spent generating each output token after the first. TPOT is often the metric used for capacity planning and SLAs, since it’s a clean, averaged view of decode-phase performance across an entire response.

3. Serving Metrics

While Prefill and Decode describe what happens within a single request, Serving metrics describe how the whole system performs across many requests at once — the metrics that matter most for capacity planning and infrastructure decisions.

End-to-End Latency Total request completion time — prefill plus decode plus any queueing or network overhead. This is the metric that most closely matches what a user actually experiences, start to finish.

Requests/sec The number of requests served each second — your system’s overall request-handling throughput, a key number for understanding how much traffic your deployment can absorb.

Tokens/sec Total generation throughput across all active requests combined — distinct from the per-request Output Tokens/sec above, this is an aggregate, system-wide view.

Concurrent Requests The number of active users being served simultaneously. This number interacts heavily with everything else on this list — more concurrency generally means better GPU utilization, but also more competition for the same KV cache memory, which is where the next section comes in.

4. GPU & Memory Metrics

These are the infrastructure-level metrics sitting underneath everything above. They directly impact throughput and latency across all three phases, and they’re usually where the real root cause of a performance problem is hiding.

GPU Utilization How much of the GPU’s compute capacity is actually being used. Low utilization during prefill-heavy workloads often signals the system is waiting on something else (like memory) rather than being compute-bound.

GPU Memory Usage How much of the GPU’s memory is consumed — by model weights, activations, and critically, the KV cache. This is frequently the actual ceiling on how many concurrent requests a deployment can serve, well before compute becomes the bottleneck.

KV Cache Usage How much of the available KV cache memory is currently occupied. The KV cache stores the “memory” of every token in every active conversation, and it grows with both prompt length and the number of concurrent requests — making it one of the fastest-growing resource pressures in production LLM serving.

KV Cache Fragmentation Similar to memory fragmentation in any system, this measures how much of the KV cache’s free space is scattered in unusable small chunks rather than available as contiguous blocks. High fragmentation can mean a system technically has “enough” free memory but still can’t fit a new request efficiently.

KV Cache Evictions How often the system is forced to evict (discard) cached data to make room for new requests. Frequent evictions are a strong signal you’re memory-constrained — and evictions often force expensive recomputation, directly hurting both latency and throughput.

Why This Framework Matters

The real value of separating metrics this way is diagnostic. When something feels slow, you can now ask a much more precise question than “why is it slow”:

High TTFT, normal decode speed? → Look at prefill latency and prompt length — you’re likely compute-bound on long prompts.
Fast start, but the response stutters as it streams? → Look at ITL and TPOT — likely a decode-phase or GPU-utilization issue.
Everything’s fine with light traffic but degrades under load? → Look at concurrent requests alongside GPU memory usage and KV cache metrics — you’re probably hitting a memory ceiling, not a compute ceiling.
Good GPU utilization but still hitting memory walls? → Look at KV cache fragmentation and eviction rate — you may have plenty of raw memory but poor memory management.

Treating “latency” as one number hides all of this. Treating it as four connected layers — prefill, decode, serving, and the GPU/memory resources underneath all three — turns a vague performance complaint into an actual, traceable root cause.

Cheers,

Sim