Episode 09 · Planned
Prefill/decode disaggregation and cache-aware routing
Compare colocated and supported disaggregated serving with queue- and cache-aware routing.
Compare a colocated baseline with supported disaggregated serving under the same arrival process, then vary the prefill/decode capacity ratio. Compare simple routing with queue-aware, cache-aware and prompt-length-aware policies. Cache reuse must be balanced against queue delay, load skew and fairness.
Evidence: KV-transfer bytes and duration, queue and utilization in both pools, cache ownership, cancellation/failure behavior, routing decisions, hit benefit versus queue delay, class-specific SLO attainment, starvation and maximum wait.