Distributed Machine Learning & System Optimization

Overview

We research communication efficiency optimization, resource scheduling, and heterogeneous computing coordination in large-scale deep learning systems, aiming to maximize training throughput and convergence efficiency under limited hardware resources.

Research Directions

• Gradient Communication Optimization: AlltoAll-based sparse gradient communication methods to significantly reduce communication overhead in distributed training
• Heterogeneous GPU Cluster Scheduling: Online multi-objective GPU scheduling for heterogeneous clusters
• Distributed Training Frameworks: Selective co-location strategies for multimodal LLM distributed training
• Federated Learning Optimization: Social welfare control and incentive mechanisms in cross-silo federated learning
• Convergence Theory: Rigorous convergence proofs for sparsification and quantization communication algorithms

Key Achievements

• Proposed AlltoAll sparse gradient communication method (ICPP 2024) accelerating distributed deep learning
• Proposed low-bit sparse gradient communication algorithm (ICPP 2026) with both compression efficiency and convergence guarantees
• Proposed multi-objective online heterogeneous GPU cluster scheduler (ICIMCT 2025)
• Proposed SelCo selective co-location multimodal LLM training framework
• Research on social welfare control and free-riding incentives in federated learning caching (ICASSP 2026)