Understanding the LLM Inference Stack
Open weight models can be a product moat for AI native products. This series covers key concepts for serving them at scale.
For many AI products, the first model integration is a frontier API. That is usually the right default: closed model APIs offer strong general purpose quality, managed infrastructure, simple billing, and rapid access to new capabilities.
As the product matures, the model call often becomes less like an external utility and more like part of the product itself. A search or recommendation system may need task specific quality that a general assistant model does not optimize for. A workflow product may need strict structured output. A high volume application may need lower unit cost than closed API pricing can provide. A regulated product may need more control over data, deployment geography, fallback behavior, and availability.
That is where open-weight models become interesting as they make those constraints tunable. A product can adapt quality, tune latency, control cost, and design the deployment around the product rather than around a generic API. That control can become a product moat.
Why Open-Weight Models Matter
Self-hosting an open-weight model lets you control the operational levers that shape your product's user experience.
Quality Alignment
Task specific quality is often different from broad benchmark quality. A model used inside a search system, recommendation product, support workflow, or structured extraction pipeline is not being asked to behave like a general assistant. It has to perform well on a domain specific distribution with domain specific constraints.
With open weights, that distribution can become part of the training and evaluation process. The model can be fine-tuned, distilled, constrained, or otherwise adapted to the task. Hence, as workflows become better understood, parts of the system can often be compiled into smaller task-specific models that can often achieve much better performance on specific tasks for product features.
Latency Tuning
Closed APIs are optimized for serving many customers across many workloads. Product teams, however, care about the latency experienced by a specific user in a specific interaction.
Dedicated open-weight deployments make different tradeoffs possible. Models can be selected, quantized, routed, cached, and scheduled around the needs of the application. A real-time autocomplete system may optimize aggressively for time-to-first-token, while a batch workflow may optimize for throughput.
Operational Control
When an application depends on a closed API, the provider’s reliability becomes part of the product’s reliability. For many companies that is a reasonable tradeoff. Managed infrastructure removes significant operational burden.
Open-weight deployments offer a different option. You can design redundancy, load routing, and fallback policies directly into your infrastructure to guarantee specific SLAs.
Cost
At scale, inference is often the largest line items in an AI product. Open-weight deployments can materially reduce unit cost at scale, but the comparison is workload-dependent.
Closed APIs usually charges by input and output tokens. With open-weight deployments, cost becomes something that can be optimized. Model size, quantization, routing policies, caching strategies, scheduler design, and output constraints can reduce inference costs by large factors while preserving product quality. However the economics and cost trade-offs depend on workload characteristics such as utilization, token volume, generation length, batching efficiency, and hardware selection.
LLM Inference
LLM text generation is fundamentally different from traditional model serving. In a classic ML model—such as an image classifier or tabular ranker—a single request triggers a single forward pass, producing a single prediction. In contrast, an LLM processes an input prompt and then runs an autoregressive loop, generating one token at a time.
For each generated token, the model runs a forward pass over the network to predict a probability distribution over its vocabulary. The engine samples a token, appends it to the sequence, and repeats the loop. This continues until it hits an end-of-sequence (<EOS>) token or a configured length limit.
Because each step depends on all prior tokens, the computational and memory access profile changes dramatically between the prompt ingestion and the subsequent token generation steps.
Prefill and Decode
This asymmetry splits the inference process into two distinct phases:
Prefill: The engine processes the entire input prompt—system instructions, retrieved context, conversation history—in a single forward pass. Because all input tokens are processed together, this step uses highly parallel matrix-matrix multiplications and is compute-bound, limited by how fast the GPU can execute arithmetic.
Decode: The engine runs the autoregressive loop, generating one token at a time. Because it processes only a single token per sequence in each step, this phase is memory-bandwidth-bound, limited by how fast the GPU can load the model weights (typically hundreds of gigabytes) from High Bandwidth Memory (HBM) into its compute cores.
This distinction means prompt length and output length demand different systems resources. A long prompt drives up prefill time, directly affecting Time to First Token (TTFT). A long response keeps the sequence active for hundreds of decode iterations, dominating the Inter-Token Latency (ITL) and occupying GPU memory for longer.
Serving Frameworks
In a production environment, you serve hundreds of concurrent streams with varying prompt lengths, output lengths, and latency requirements.
To handle this, modern engines like vLLM and SGLang manage a complex queue. They must batch requests continuously, cache common prefix state, parse structured formats, and schedule prefill and decode tasks dynamically.
These tasks compete for the same hardware resources:
Increasing context limits to support longer prompts reduces the memory pool available for concurrent users.
Enforcing strict JSON schemas with grammar masks prevents output failures but changes token generation distributions, potentially extending the decode tail and delaying other queued requests.
Quantizing model weights or the KV cache to lower precision halves the memory footprint but introduces potential quality trade-offs and kernel compilation overhead.
Every product requirement—whether it is strict schema adherence, low latency, or cost reduction—eventually binds to a specific systems constraint in the serving stack.
The Inference Stack
A useful way to understand the stack is to follow one request from the product boundary to the streamed response:
Prompt Preprocessing
Prompt preprocessing is the host-side CPU work that shapes a raw request into a structured model input. It handles chat template formatting (structuring system prompts, user content, and tool history), appends schemas, and runs the tokenizer to translate text into token IDs. Mismatches here are a common source of silent quality degradation.
Engine Orchestration
The control plane of the serving engine, responsible for coordinating requests before they execute on the GPU. It contains:
Request Router: Directs traffic across replicas, optimizing for queue depth and prefix cache locality.
Scheduler & Batcher: Manages continuous batching, deciding when to admit new requests and how to group token execution steps.
KV Cache Manager: Allocates virtual memory blocks (such as PagedAttention) to maximize memory density and prefix reuse.
Model Execution
The physical GPU execution layer where the forward passes run. It contains:
Model Weights: The static parameters. Quantization reduces their footprint, while Tensor Parallelism or Pipeline Parallelism distributes them across multiple cards.
Prefill Engine: Processes the entire input prompt in a single parallel step to compute the initial key-value attention states.
Decode Engine: Runs the autoregressive token-by-token loop, reading weights and key-values repeatedly.
KV Cache: The physical memory pool allocated to store active key-value states, representing the primary constraint on concurrent capacity.
Sampling & Post-Processing
The Logits & Sampling Engine computes the probability distribution over the vocabulary, applies configuration controls (like temperature and top-p), and enforces constrained decoding grammar masks (such as JSON schemas) before selecting the next token ID.
Response Streaming
The Detokenizer & Streamer converts token IDs back into text chunks, formats the streaming payload (e.g., Server-Sent Events), and delivers it to the network. Buffer delays in proxies or slow clients can introduce latency here that mimics model slowness.
Next in the Series
The the rest of this eries we will cover:
Key Metrics of LLM Inference: Measure performance across the request timeline and isolate latency bottlenecks.
Single Request Optimizations: Optimize prefill and decode execution stages under compute and memory bandwidth constraints.
Serving Concurrent Requests: Navigate tradeoffs between latency, throughput, and memory capacity under multi-request concurrency.
Speeding up Generation: Accelerate token generation using speculative execution and split prefill/decode pipelines.
LLM Model Architecture Choices: Evaluate how quantization, distillation, and structural pruning impact serving efficiency.