LLM inference optimization is the key technology for achieving maximum throughput and minimum latency with limited GPU resources. This post systematically covers quantization, KV Cache, speculative decoding, and other major optimization techniques.
1. Understanding LLM Inference
Inference Stages
LLM inference consists of two stages: Prefill (input processing) and Decode (token generation).
[LLM Inference Stages]
Input: "How to optimize Spark performance?"
↓
[Prefill] — Process all input tokens in parallel
- Generate KV Cache
- Compute-bound (GPU compute intensive)
- Determines TTFT (Time to First Token)
↓
[Decode] — Generate tokens one at a time
- Read + update KV Cache
- Memory-bound (memory bandwidth intensive)
- Determines TPS (Tokens Per Second)
↓
Output: "To optimize Spark performance, you should..."
Key Performance Metrics
Metric
Description
Affected By
TTFT
Time to first token
Prefill speed, model size
TPS
Tokens per second
Decode speed, memory bandwidth
Throughput
Total throughput (tok/s)
Batching, GPU utilization
Latency (P95)
95th percentile response time
Queuing, scheduling
GPU Memory Breakdown
[7B Model GPU Memory (FP16)]
Total GPU Memory: 24 GB (RTX 4090)
Model weights: ██████████████ 14 GB (58%)
KV Cache: ████████ 8 GB (33%)
Activations/temp: ██ 2 GB (9%)
→ KV Cache is the bottleneck determining concurrent request count
2. Quantization
What is Quantization
Reducing the precision of model weights to save memory and computation.
[Static Batching]
Batch: [A(100tok), B(50tok), C(200tok)]
→ Wait for longest (C) to complete, A/B idle after finishing
[Continuous Batching]
Time 1: [A, B, C] processing
Time 2: B done → D immediately added [A, D, C]
Time 3: A done → E immediately added [E, D, C]
→ Minimize GPU idle time
Reuse prefix KV Cache for requests sharing the same system prompt.
# Enable in vLLMpython -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --enable-prefix-caching# → 50-80% TTFT reduction for long system prompts
Tensor Parallelism
Strategy
Splits
Best For
Tensor Parallel
Within layers (weights)
Single node multi-GPU
Pipeline Parallel
Across layers (depth)
Multi-node
Data Parallel
Requests (replicas)
N identical model instances
7. Optimization Strategy Summary
Scenario Guide
Goal
Key Technique
Expected Effect
Reduce model memory
Quantization (AWQ/GGUF Q4)
75% model size reduction
Increase throughput
PagedAttention + Continuous Batching
2-4x throughput
Faster first token
Prefix Caching + FlashAttention
50-80% TTFT reduction
Faster token generation
Speculative Decoding
1.5-2.5x TPS
Run large models
Tensor Parallelism
70B+ model serving
Recommended Priority
1. Apply quantization (easiest, highest impact)
2. vLLM + Continuous Batching (switch serving engine)
3. Enable FlashAttention (usually enabled by default)
4. Prefix Caching (reuse system prompts)
5. Speculative Decoding (large model acceleration)
6. Tensor Parallelism (very large models)
Note: Most optimization techniques can be combined. Applying quantization + PagedAttention + Continuous Batching + Prefix Caching together yields maximum effect.
References
Kwon, W. et al. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP
Dao, T. (2024). "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." ICLR
Leviathan, Y. et al. (2023). "Fast Inference from Transformers via Speculative Decoding." ICML
Frantar, E. et al. (2023). "GPTQ: Accurate Post-Training Quantization." ICLR
Lin, J. et al. (2024). "AWQ: Activation-aware Weight Quantization." MLSys
