Abstract: Machine learning (ML) has demonstrated great promise in scientific discovery, healthcare, and education, especially with the rise of large neural networks. However, large models trained on complex and rapidly growing data consume enormous computational resources. In this talk, I will describe my work on exploiting model sparsity and hardware capabilities with randomized algorithms to accelerate large ML systems without accuracy degradation.
I will start by describing SLIDE, an open-source system for efficient sparse neural network training on CPUs that has been deployed by major technology companies and academic labs. SLIDE blends Locality Sensitive Hashing with multi-core parallelism and workload optimization to drastically reduce computations. SLIDE trains industry-scale recommendation models on a 44 core CPU 3.5x faster than TensorFlow on V100 GPU.
Next, I will present Pixelated Butterfly, a simple yet efficient sparse training framework on GPUs. It uses a simple static block-sparse pattern based on butterfly and low-rank matrices, taking into account GPU block-oriented efficiency. Pixelated Butterfly trains up to 2.5x faster (wall-clock) than the dense Vision Transformer and GPT-2 counterparts with no drop in accuracy.
I will conclude by outlining future research directions for further accelerating ML pipelines and making ML more accessible to the general community, such as software-hardware co-design, data-centric AI, and ML for scientific computing and medical imaging.
Bio: Beidi Chen is a postdoctoral scholar in the CS department at Stanford University, working with Prof. Christopher Ré. Her research focuses on large-scale machine learning and deep learning. Specifically, she designs and optimizes randomized algorithms (algorithm-hardware co-design) to accelerate large machine learning systems for real-world problems. Prior to joining Stanford, she received her Ph.D. from the CS department at Rice University, advised by Prof. Anshumali Shrivastava. She received a BS in EECS from UC Berkeley. She has held internships in Microsoft Research, NVIDIA Research, and Amazon AI. Her work has won Best Paper awards at LISA and IISA. She was selected as a Rising Star in EECS by MIT and UIUC.
