Qing Jin’s PhD Proposal Review

Prof. Yanzhi Wang (Advisor)
Prof. David R. Kaeli
Prof. Sunil Mittal
Prof. Jennifer Dy

Deep learning has achieved remarkable success in a variety of modern applications, but this success is often accompanied by inefficiency in terms of storage and inference speed, which can hinder their practical use on resource-constrained hardware. Developing highly efficient neural networks that maintain high prediction accuracy is crucial and challenging. This dissertation explores the potential for simultaneously achieving high efficiency and high prediction accuracy in neural networks, and can be broadly divided into three sections. (1) In Section One, we explore the implementation of highly efficient generative adversarial networks (GANs) capable of generating high-quality images within a predefined computational budget. The key challenge lies in identifying the optimal architecture for the generative model while simultaneously preserving the quality of the generated images from the compressed model, despite its reduced computational cost. To achieve this, we propose a novel neural architecture search (NAS) algorithm and a new knowledge distillation technique. (2) In Section Two, we explore the challenge of quantizing discriminative models without relying on high-precision multiplications. To address this issue, we present an innovative approach to determine the optimal fixed-point formats for both weights and activations based on their statistical properties. Our results demonstrate that high accuracy in quantized neural networks can be achieved without the need for high-precision multiplications. (3) In Section Three, we delve into the challenge of training neural networks for innovative computing platforms, specifically processing-in-memory (PIM) systems. Through a detailed mathematical derivation of the backward propagation algorithm, we facilitate the training of quantized models on these platforms. Additionally, through a thorough theoretical analysis of training dynamics, we ensure convergence and propose a systematic solution for quantizing neural networks on PIM systems.