A distributed MLLM training system that makes pipeline parallelism frozen-status aware to eliminate bubbles, introduces token workload-balanced context parallelism for the mixed …
A layerwise-adaptive gradient compression framework that formulates per-layer compression parameter selection as a constrained optimization and solves it via dynamic programming, …
A workload-balanced 4D parallelism method for long-context LLM training that combines variable-length packing and outlier delay at the PP level with per-document sharding and …
A dynamic interleaved pipeline system for LMM training that separates modality segments to remove intra-segment imbalance, splits data into modality-aware sub-microbatches to …
An automatic parallelization system for distributed deep learning that unifies intra-operator and inter-operator parallelism in a hierarchical space, using dynamic programming for …
A sparse pseudo-gradient compression method for communication-efficient distributed LLM pre-training, combining Top-k selection, aggressive 2-bit quantization and error feedback on …
Communication optimization, heterogeneous GPU scheduling, and large-scale training acceleration for distributed deep learning.