WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
A workload-balanced 4D parallelism method for long-context LLM training that combines variable-length packing and outlier delay at the PP level with per-document sharding and …
