Turbocharging LLMs

TenthMatrix on how to turbocharge large language models in production: PagedAttention, vLLM, Self-Debugging, PowerInfer, and EG-MLA — with the trade-offs that actually show up on GPUs. Deep dive: long article. Primer: LLMs explained + Kubernetes.

Watch: what an LLM actually is

Context, not a product demo: Intro to Large Language Models (Andrej Karpathy). Pair with our plain-English LLM + Kubernetes guide.

The scaling problem

LLMs unlocked conversational AI and generation, then immediately became a serving problem. Each decode step appends keys and values to a KV cache that grows with sequence length and batch size. Naive engines pre-reserve huge contiguous slabs. Most of that memory sits idle while other requests wait. Throughput collapses even though the GPU looks “full.”

PagedAttention: virtual memory for attention

PagedAttention (Kwon et al., SOSP 2023) treats KV cache like OS paging: fixed-size blocks, a block table per request, non-contiguous physical pages [arXiv:2309.06180]. The kernel gathers blocks at attention time. You still must tune block size, max model length, and the cache hierarchy (GPU HBM vs CPU offload vs prefix cache). Too-small blocks add mapping overhead; too-large blocks reintroduce waste.

vLLM: near-zero waste serving

vLLM is the serving system built around PagedAttention. It targets near-zero KV waste, continuous batching, and an OpenAI-compatible HTTP surface. In production, watch three things: (1) gpu_memory_utilization vs OOM, (2) time-to-first-token vs decode tokens/s, (3) whether prefix / prompt caching changes answers for your eval set. Caching is a throughput win; it is not free of correctness and tail-latency effects.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", gpu_memory_utilization=0.90)
params = SamplingParams(temperature=0.2, max_tokens=256)
outs = llm.generate(["Explain PagedAttention in one paragraph."], params)
print(outs[0].outputs[0].text)

That is real vLLM usage. A Hugging Face BERT forward() call is not PagedAttention — do not confuse classifier logits with paged KV serving.

Self-Debugging: quality loops with a latency budget

Self-Debugging (Chen et al.) teaches a model to fix its own predicted programs from few-shot traces, matching or beating baselines that emit 10x more candidates [arXiv:2304.05128]. For agents, that is gold — until feedback rounds stack. Cap the number of debug messages, log each round, and fail closed to a human or a smaller specialist model when the budget is gone.

Watch: serving and inference context

Contextual talk on fast LLM serving — not an official TenthMatrix demo. Pair with the vLLM paper and docs.vllm.ai.

PowerInfer: tokens on a single fat GPU

PowerInfer splits hot/cold neurons so a consumer GPU plus CPU can generate tokens quickly. Reported figures: 13.20 tokens/s average, peak 29.08 tokens/s on one NVIDIA RTX 4090, up to 11.69x vs llama.cpp while keeping accuracy [PowerInfer]. At fleet scale you still need placement: which layers stay on GPU, how you shard across nodes, and whether the locality profile of your prompts matches the paper’s models.

EG-MLA: shrink the KV without wrecking the bench

EG-MLA (embedding-gated multi-head latent attention) reports over 91.6% KV cache reduction vs multi-head attention with negligible degradation, extra savings vs MLA (up to 59.9%), and better scores on reasoning suites, scaled past 1B parameters [EG-MLA paper]. Treat it as an architecture choice, not a drop-in flag on a frozen vLLM checkpoint — validate your eval harness before you celebrate the memory graph.

Putting it together

Combine PagedAttention + vLLM for cluster serving, PowerInfer when the box is a workstation GPU, Self-Debugging inside coding agents with a hard round limit, and EG-MLA when you control the model family. Distributed serving adds complexity: KV parallelism, prefill/decode split, and autoscaling that does not thrash the page tables. Measure. Then write the ADR.

Read next

1. Long article — knobs, failure modes, Kubernetes notes.
2. LLMs explained + run your own on Kubernetes
3. Muse Glimmer / local agents · Enterprise AI Lab

Published by TenthMatrix.