Tong Jian’s PhD Dissertation Defense

“Robust Sparsified Deep Learning”

Abstract:

This dissertation studies robustness issues around DNN deployments on resource constrained systems, under both environmental and adversarial input adaptation. We propose a means of compressing a Radio Frequency deep neural network architecture through weight pruning, and provide a systems-level analysis of the implementation of such a pruned architecture at resource-constrained edge devices. In particular, we jointly train and sparsify neural networks tailored to edge hardware implementations.

Next, we propose a new learn-prune-share (LPS) algorithm for achieving robustness to environment adaptation in the field of lifelong learning. Our method maintains a parsimonious neural network model and achieves exact no forgetting by splitting the network into task-specific partitions via a weight pruning strategy optimized by the Alternating Direction Methods of Multipliers (ADMM). Moreover, a novel selective knowledge sharing scheme is integrated seamlessly into the ADMM optimization framework to address knowledge reuse.
Furthermore, we investigate the Hilbert-Schmidt Information Bottleneck as regularizer (HBaR) as a means to enhance adversarial robustness. We show that the Hilbert-Schmidt Information bottleneck enhances robustness to adversarial attacks both theoretically and experimentally. In particular, we prove that the HSIC bottleneck regularizer reduces the sensitivity of the classifier to adversarial examples.

Finally, we propose a novel framework Pruning-without-Adversarial-training (PwoA) for the purpose of achieving adversarial robustness on resource-constrained systems. PwoA can efficiently prune a previously trained robust neural network while maintaining adversarial robustness, without further generating adversarial examples. We leverage concurrent self-distillation and pruning to preserve knowledge in the original model as well as regularizing the pruned model via the HBaR.

Committee:

Prof. Stratis Ioannidis (Advisor)

Prof. Jennifer Dy
Prof. Kaushik Chowdhury

Prof. Yanzhi Wang