Episode 07 · Planned
Parallelism within one node
Compare DP, TP and justified PP or EP layouts under one declared GPU budget.
Compare data parallelism (DP) replicas with tensor parallelism (TP) partitioning under the same total GPU budget and workload. Add pipeline parallelism (PP) when model fit or topology justifies it. For mixture-of-experts models, study expert parallelism (EP) and expert load balance as a separate dimension.
Evidence: GPU topology and NVLink/PCIe paths, placement, memory per rank, collective time/bytes, PP stage balance, expert load/skew, throughput and request tails.