https://www.acal.org.cn/en/tags/alltoall/AlltoAllHugoBlox Kit (https://hugoblox.com)en-usWed, 01 Jan 2025 00:00:00 +0000https://www.acal.org.cn/media/logo_hu_9c369f2c5735daea.pnghttps://www.acal.org.cn/en/tags/alltoall/https://www.acal.org.cn/en/research/distributed-ml/Wed, 01 Jan 2025 00:00:00 +0000https://www.acal.org.cn/en/research/distributed-ml/<h2 id="overview">Overview</h2> <p>We research communication efficiency optimization, resource scheduling, and heterogeneous computing coordination in large-scale deep learning systems, aiming to maximize training throughput and convergence efficiency under limited hardware resources.</p> <h2 id="research-directions">Research Directions</h2> <ul> <li><strong>Gradient Communication Optimization</strong>: AlltoAll-based sparse gradient communication methods to significantly reduce communication overhead in distributed training</li> <li><strong>Heterogeneous GPU Cluster Scheduling</strong>: Online multi-objective GPU scheduling for heterogeneous clusters</li> <li><strong>Distributed Training Frameworks</strong>: Selective co-location strategies for multimodal LLM distributed training</li> <li><strong>Federated Learning Optimization</strong>: Social welfare control and incentive mechanisms in cross-silo federated learning</li> <li><strong>Convergence Theory</strong>: Rigorous convergence proofs for sparsification and quantization communication algorithms</li> </ul> <h2 id="key-achievements">Key Achievements</h2> <ul> <li>Proposed AlltoAll sparse gradient communication method (ICPP 2024) accelerating distributed deep learning</li> <li>Proposed low-bit sparse gradient communication algorithm (ICPP 2026) with both compression efficiency and convergence guarantees</li> <li>Proposed multi-objective online heterogeneous GPU cluster scheduler (ICIMCT 2025)</li> <li>Proposed SelCo selective co-location multimodal LLM training framework</li> <li>Research on social welfare control and free-riding incentives in federated learning caching (ICASSP 2026)</li> </ul>