DeepReduce: A Sparse-tensor Communication Framework for Federated Deep Learning

Hang Xu, Kelly Kostopoulou, Aritra Dutta, Xin Li, Alxendros Ntoulas, Panos Kalnis

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Abstract

Sparse tensors appear frequently in federated deep learning, either as a direct artifact of the deep neural network’s gradients, or as a result of an explicit sparsification process. Existing communication primitives are agnostic to the peculiarities of deep learning; consequently, they impose unnecessary communication overhead. This paper introduces DeepReduce, a versatile framework for the compressed communication of sparse tensors, tailored to federated deep learning. DeepReduce decomposes sparse tensors into two sets, values and indices, and allows both independent and combined compression of these sets. We support a variety of common compressors, such as Deflate for values, or run-length encoding for indices. We also propose two novel compression schemes that achieve superior results: curve fitting-based for values, and bloom filter-based for indices. DeepReduce is orthogonal to existing gradient sparsifiers and can be applied in conjunction with them, transparently to the end-user, to significantly lower the communication overhead. As proof of concept, we implement our approach on TensorFlow and PyTorch. Our experiments with large real models demonstrate that DeepReduce transmits 320% less data than existing sparsifiers, without affecting accuracy. Code is available at https://github.com/hangxu0304/DeepReduce.
Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 34 (NeurIPS 2021)
EditorsM. Ranzato, A. Beygelzimer, P.S. Liang, J.W. Vaughan, Y. Dauphin
Number of pages15
PublisherNeurIPS Proceedings
Publication dateDec 2021
Publication statusPublished - Dec 2021
Externally publishedYes
SeriesAdvances in Neural Information Processing Systems
ISSN1049-5258

Keywords

  • Distributed Computing
  • federated learning
  • compressed communication
  • sparsification
  • Bloom Filter
  • Polynomial Approximation

Fingerprint

Dive into the research topics of 'DeepReduce: A Sparse-tensor Communication Framework for Federated Deep Learning'. Together they form a unique fingerprint.

Cite this