SparseLoCo: Communication Efficient LLM Pre-training with Sparsity-driven Data Synchronization
A sparse pseudo-gradient compression method for communication-efficient distributed LLM pre-training, combining Top-k selection, aggressive 2-bit quantization and error feedback on …
