LLM Inference Optimization | System Designer

What is LLM inference optimization?

LLM inference optimization is the engineering work that makes an already-trained language model respond within a product's latency, capacity, cost, and quality limits. It changes how requests are admitted, batched, cached, scheduled, represented in memory, and decoded; it does not retrain the model's core knowledge.

In plain language: training creates the model, while inference optimization makes each request practical to serve.

The core invariant is optimize the measured request path without silently breaking the answer contract. A faster kernel is not a production win if queueing gets worse, a quantized checkpoint fails the important task slice, or a cache crosses tenant boundaries.

TTFT

Start quickly

Queueing, routing, cache lookup, and prefill determine time to first token

TPOT

Stream steadily

Decode scheduling and memory movement determine time per output token

TPS

Carry the load

Useful token throughput must be measured at the required latency target

Quality

Preserve the contract

Evaluate task outcomes after every precision, kernel, or decoding change

Trace the complete request before tuning a kernel

End-to-end latency includes work outside the model. Measure each stage independently so an improvement in GPU execution cannot hide a longer admission queue or a slower network path.

A production inference request

TTFT ends when the first token reaches the client. TPOT measures the cadence of later tokens; throughput measures shared system capacity.

Bound demand

Admission and routing

Apply tenant quotas, deadlines, priorities, model routing, and overload policy before expensive work begins.

Reuse exact work

Prefix lookup

Reuse authorized KV blocks only when the model, adapter, tokenized prefix, and cache namespace match.

Read the prompt

Prefill

Process input tokens in parallel and create keys and values for every layer that the decode phase will need.

Generate tokens

Decode scheduler

Interleave active sequences one iteration at a time while protecting latency-sensitive streams from long-running work.

Close the loop

Stream and observe

Deliver tokens, propagate cancellation, and record latency, cache, capacity, quality, and stop-reason evidence.

Keep four measurements separate

Queue delay: time waiting for an eligible worker and enough memory; report p50, p95, and p99 by priority and tenant class.
Prefill latency: time to process uncached input tokens; prompt length and prefix reuse strongly affect it.
Time per output token (TPOT): decode cadence after the first token; active sequences and memory pressure matter.
Goodput: requests or useful tokens completed inside their SLO, not raw tokens generated after clients have timed out.

Understand why prefill and decode need different controls

Prompt phase

Prefill

The model processes many input tokens in parallel. Long prompts create substantial arithmetic work, while exact prefix reuse can skip already-computed blocks.

Generation phase

Decode

Autoregressive generation produces one next-token step at a time. The scheduler repeatedly reads model weights and each sequence's growing KV state.

Contention boundary

Shared serving pool

Prefill bursts can delay interactive decode, while too many active sequences can exhaust KV memory. Scheduling policy decides who waits.

Choose the metric that matches the product

A conversational assistant usually needs a tight TTFT and stable TPOT so the stream feels responsive.
Long-document extraction may accept a slower start if aggregate completion throughput remains high.
Offline batch generation can favor throughput, but still needs deadlines, cancellation, and bounded queue growth.
Real-time voice needs strict tail latency; average tokens per second cannot describe interruptions or jitter.

Disaggregating prefill and decode can isolate their resource policies and tune TTFT separately from TPOT, but KV transfer adds another network and failure boundary. Treat it as an architecture choice, not a universal default.

Allocate latency and KV memory together

Batching, concurrency, and prefix reuse interact. A larger batching window can improve GPU utilization while directly consuming the TTFT budget. More active sequences can raise throughput until their KV state creates memory pressure and preemption. Prefix caching helps prefill only when requests share an exact reusable prefix; it does not accelerate the generation of new tokens.

Use the lab to change those controls and watch the request timeline and KV reservoir move together. Its numbers are a transparent planning model, not a hardware benchmark.

Loading serving budget lab...

Treat the KV cache as live request state

During autoregressive decode, each layer stores keys and values for prior tokens so the model does not recompute the entire prefix for every new token. The cache grows with active sequence length and concurrency.

A useful first-order estimate is:

KV bytes per token = 2 x layers x KV heads x head dimension x bytes per element

The factor of two represents keys and values. Grouped-query or multi-query attention reduces the number of KV heads, and runtime layouts add model- and engine-specific details. Measure the actual allocated pages rather than treating the estimate as an invoice.

Protect the cache contract

Key reusable blocks by the model, adapter, tokenizer output, cache format, and full prefix ancestry.
Include a tenant or trust-domain boundary when prompts or adapters contain private state.
Account for partially filled blocks, copy-on-write branches, preemption, and swap or recompute behavior.
Track allocated KV bytes, fragmentation, eviction, preemption, and prefix-hit tokens together.
Propagate cancellation quickly so abandoned sequences release scheduler and cache capacity.

PagedAttention showed that paging and sharing KV blocks can reduce fragmentation and enable larger effective batches. Prefix caching builds on block reuse, but its benefit depends on repeated prefixes and is concentrated in prefill.

Make admission control part of the optimization

Continuous batching schedules at token-step granularity so completed requests can leave and new work can join without waiting for a whole static batch. This improves utilization, but it does not make capacity infinite.

1
At ingress
Classify the request
Record tenant, priority, prompt and output bounds, streaming mode, deadline, cache eligibility, and cancellation owner.
2
Before execution
Reserve memory
Check model weights, runtime workspace, and projected KV pages before admitting a sequence to the worker.
3
During decode
Schedule by iteration
Batch compatible active sequences while preserving fairness and a bounded TPOT for latency-sensitive work.
4
At the boundary
Shed overload explicitly
Delay, downgrade, or reject before queues exceed the SLO; do not let timeouts become the hidden capacity policy.

Overload controls should specify

maximum queued requests, tokens, and predicted KV bytes per pool;
separate budgets for interactive, batch, and internal retry traffic;
a deadline-aware ordering rule and a starvation bound;
which lower-cost model or shorter-context route is an acceptable fallback;
the response returned when no route can meet the contract.

Calculate a serving envelope before load testing

This runnable example turns a workload profile into effective prefill tokens, modeled TTFT, KV allocation, TPOT pressure, and goodput. The constants are deliberately explicit so teams replace them with measurements from their own model, engine, and hardware.

Serving budget model

Change precision and decoding for different reasons

Quantization primarily changes representation and kernel behavior. Speculative decoding changes how many sequential target-model steps are needed. They can be combined, but they introduce different evidence requirements.

Reference behavior

BF16 or FP16 baseline

Keep a measured reference checkpoint and engine path. It anchors task quality, numerical behavior, memory, latency, and rollback comparisons.

Lower-precision arithmetic

INT8 or FP8

Weights and possibly activations use fewer bits. Kernel and hardware support determine whether fewer bytes become lower latency or higher throughput.

Aggressive weight compression

INT4 weight-only

Methods such as AWQ or GPTQ can reduce weight storage substantially, but calibration, task quality, kernel support, and dequantization overhead remain workload-specific.

Draft then verify

Speculative decoding

A cheaper drafter proposes several tokens and the target verifies them in parallel. Speed depends on acceptance, draft cost, target batch size, and implementation.

Preserve two different quality claims

An exact speculative sampling algorithm can preserve the target model's output distribution; draft quality affects acceptance and speed, not the accepted distribution.
Quantization changes numerical representation and can change model outputs, so every precision recipe needs task-level and safety evaluation.
A provider implementation may support different speculative algorithms, sampling modes, tokenizers, or batch regimes. Validate the actual runtime contract.
Weight memory is approximately parameters x bits / 8 before scales, metadata, padding, runtime workspace, draft weights, and KV cache.

Plan capacity with measured acceptance and quality

A smaller checkpoint may fit more replicas while missing the quality floor. A speculative drafter may reduce target steps at low batch size while consuming memory and compute that could have served another request. Use the lab to pack a fixed HBM pool, adjust measured draft acceptance, and compare capacity with demand.

Loading capacity trade-off lab...

Benchmark combinations, not isolated features

An optimization is deployable only when the exact model, precision, kernel, parallelism plan, scheduler, cache policy, prompt distribution, output distribution, and concurrency level work together.

Build the benchmark matrix

Slice prompts by uncached input length, reusable-prefix length, expected output length, and sampling policy.
Include arrival bursts, cancellations, retries, mixed tenants, and priority inversions rather than sending a perfectly smooth load.
Measure p50, p95, and p99 queue delay, TTFT, TPOT, end-to-end latency, request goodput, and useful-token goodput.
Record GPU compute, memory bandwidth, HBM allocation, KV occupancy, cache hits, evictions, preemptions, and energy or cost when available.
Evaluate task success, refusal and safety behavior, structured-output validity, and long-context retrieval after every numerical change.

Compare at a fair operating point

Compare throughput at the same TTFT and TPOT SLO, not at unconstrained saturation.
Compare cost at the same task-quality floor, not only tokens per dollar.
Warm and cold cache runs answer different questions; report both.
Keep request traces and engine configuration versioned so a result can be reproduced.

Turn the selected plan into an executable check

This example estimates raw weight memory, per-replica footprint, speculative target passes, and pool capacity. It also keeps quantization risk separate from exact speculative verification.

Compression and decoding capacity plan

Recognize optimization failures early

Batching past the interaction budget: throughput rises while TTFT or TPOT misses the product SLO. Cap waiting time and reserve capacity for latency-sensitive traffic.
Counting cache entries instead of tokens saved: many tiny hits look healthy but avoid little prefill work. Measure reused tokens and prefill milliseconds avoided.
Unsafe prefix reuse: a cache key omits tenant, adapter, or model identity. Treat cache namespace design as a data-isolation boundary.
KV oversubscription: long contexts cause preemption or recomputation, producing a tail-latency cliff. Admit from predicted bytes and monitor actual pages.
Quantizing without slice evaluation: average benchmark quality stays flat while a critical domain regresses. Gate release on task, safety, and structured-output slices.
Speculating at the wrong operating point: drafting overhead consumes capacity when acceptance is low or the target already runs in large efficient batches. Enable it only on measured routes.
Optimizing raw throughput: the system generates tokens after deadlines or client cancellation. Track goodput and propagate cancellation to every worker.
Stacking changes without attribution: a new kernel, precision, scheduler, and cache policy ship together. Canary one controlled bundle at a time and preserve rollback evidence.

Operate inference as a versioned policy

Before release

Freeze the model, tokenizer, adapter, quantization recipe, engine, kernel, and scheduler versions.
Define SLOs by workload class and specify the quality and safety floors that performance work may not cross.
Load test healthy, burst, dependency-failure, and cancellation scenarios at realistic sequence lengths.
Canary with bounded traffic and an immediate route back to the reference path.

During production

Alert on queue growth, TTFT, TPOT, goodput, KV pressure, preemption, cache-hit tokens, and quality proxies by route.
Record why each request selected a model, precision, cache path, and decoding policy.
Separate model errors, overload rejection, dependency failure, and client cancellation in telemetry.
Re-run the benchmark matrix after model, driver, kernel, hardware, or workload-distribution changes.

Primary sources behind the design

Orca: A Distributed Serving System for Transformer-Based Generative Models introduces iteration-level scheduling and selective batching for generative-model serving.
Efficient Memory Management for Large Language Model Serving with PagedAttention describes paged KV-cache management, reduced fragmentation, and cache sharing in vLLM.
vLLM automatic prefix caching documents hash-based reuse of matching KV blocks; its usage guide also makes clear that prefix caching accelerates prefill rather than generation.
FlashAttention demonstrates that exact attention can be reorganized around GPU memory IO instead of changing the mathematical result.
Fast Inference from Transformers via Speculative Decoding presents exact speculative sampling whose accepted output preserves the target distribution.
SmoothQuant studies W8A8 post-training quantization; AWQ studies activation-aware low-bit weight-only quantization.
TensorRT-LLM quantization and speculative decoding show why supported precision, hardware, sampling, and batch conditions must be checked in the deployed runtime.

These sources establish mechanisms and measured results in specific systems. They do not guarantee the same speedup for a different model, workload, engine, or GPU.

No quiz questions available

Could not load questions file