Token By Token

Token by Token · episode index

Inference benchmark episodes

Recorded studies and a public experiment roadmap for learning how model-serving systems behave under controlled workloads.

  1. Episode 00 · Warm-up — Follow a request through prefill, decode and the KV cache. Explore the recorded vLLM and SGLang runs, their costs and the limits of the comparison.
  2. Episode 01 · Measure what matters — Compare SGLang and vLLM deployments on one H200 across matched recorded load levels, then inspect latency, throughput and telemetry together.
  3. Episode 02 · Equal-work runtime baseline — Compare HF/PyTorch, vLLM and SGLang under the same model, corpus, template and output contract.
  4. Episode 03 · Prefix reuse — Control exact shared-prefix ratios and compare cold and warm cache behavior.
  5. Episode 04 · Batching, scheduling and mixed traffic — Study chunked prefill and scheduler behavior under controlled long/short traffic mixes.
  6. Episode 05 · Attention kernels and precision — Compare verified attention backends, then separately compare weight and KV precision.
  7. Episode 06 · Speculative decoding and structured outputs — Evaluate compatible proposal paths and constrained generation without assuming speedup.
  8. Episode 07 · Parallelism within one node — Compare DP, TP and justified PP or EP layouts under one declared GPU budget.
  9. Episode 08 · Parallelism across nodes — Test placement and communication costs on a documented multi-node network topology.
  10. Episode 09 · Prefill/decode disaggregation and cache-aware routing — Compare colocated and supported disaggregated serving with queue- and cache-aware routing.
  11. Episode 10 · Slurm and Kubernetes orchestration — Compare representative scheduled jobs and long-running services as platform choices, separate from engines.
  12. Episode 11 · Model internals: weights to optimization — Trace weights, attention, gradients and optimizer state with a small inspectable model.
  13. Episode 12 · LoRA and QLoRA with held-out evaluation — Compare adapter training with fixed data, budgets and held-out quality evaluation.
  14. Episode 13 · SYSTEM 1 and LLMs on labeled decision tasks — Compare typed and LLM decision approaches on one versioned labeled task contract.
  15. Episode 14 · Packaging, accelerator preflight and memory containment — Verify reproducible containers, accelerator visibility, model loading and bounded memory behavior.
  16. Episode 15 · Saturation, SLO and cost — Sweep concurrency and offered rate to find SLO-qualified goodput and attributable cost.
  17. Episode 16 · Release gates, recovery and capstone synthesis — Combine promising settings, reject regressions and exercise rollback and recovery against original gates.