# Efficient-Adam: Communication-Efficient Distributed Adam

Article · Congliang Chen, Li Shen, Wei Liu, Zhi-Quan Tom Luo

- Archive id: arxiv:2205.14473
- Published: 2023-01-01
- Venue: IEEE Transactions on Signal Processing
- DOI: https://doi.org/10.1109/tsp.2023.3309461
- arXiv: https://arxiv.org/abs/2205.14473
- Citations: 29
- Source URL: https://doi.org/10.1109/tsp.2023.3309461
- PDF: https://arxiv.org/pdf/2205.14473
- Provenance: openalex+arxiv

## Abstract

Distributed adaptive stochastic gradient methods have been widely used for large-scale nonconvex optimization, such as training deep learning models. However, their communication complexity on finding $\\varepsilon$-stationary points has rarely been analyzed in the nonconvex setting. In this work, we present a novel communication-efficient distributed Adam in the parameter-server model for stochastic nonconvex optimization, dubbed {\\em Efficient-Adam}. Specifically, we incorporate a two-way quantization scheme into Efficient-Adam to reduce the communication cost between the workers and server. Simultaneously, we adopt a two-way error feedback strategy to reduce the biases caused by the two-way quantization on both the server and workers, respectively. In addition, we establish the iteration complexity for the proposed Efficient-Adam with a class of quantization operators, and further characterize its communication complexity between the server and workers when an $\\varepsilon$-stationary point is achieved. Finally, we apply Efficient-Adam to solve a toy stochastic convex optimization problem and train deep learning models on real-world vision and language tasks. Extensive experiments together with a theoretical guarantee justify the merits of Efficient Adam.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Efficient Training & Inference](https://space-kitty.com/topics/efficient-ai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.