Zifeng Wang’s PhD Dissertation Defense

Title: Effective and Efficient Continual Learning

Committee Members:
Prof. Jennifer Dy (Advisor)
Prof. Stratis Ioannidis
Prof. Yanzhi Wang

Abstract:
Continual Learning (CL) aims to develop models that mimic the human ability to learn continually without forgetting knowledge acquired earlier. While traditional machine learning methods focus on learning with a certain dataset (task), CL methods adapt a single model to learn a sequence of tasks continually.

In this thesis, we target developing effective and efficient CL methods under different challenging and resource-limited settings. Specifically, we (1) leverage the idea of sparsity to achieve cost-effective CL, (2) propose a novel prompting-based paradigm for parameter-efficient CL, and (3) utilize task-invariant and task-specific knowledge to enhance existing CL methods in a general way.

We first introduce our sparsity-based CL methods. The first method, Learn-Prune-Share (LPS), splits the network into task-specific partitions, leading to no forgetting, while maintaining memory efficiency. Moreover, LPS integrates a novel selective knowledge sharing scheme, enabling adaptive knowledge sharing in an end-to-end fashion. Taking a step further, we present Sparse Continual Learning (SparCL), a novel framework that leverages sparsity to enable cost-effective continual learning on edge devices. SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity, data efficiency, and gradient sparsity.

Secondly, we present a new paradigm, prompting-based CL, that aims to train a more succinct memory system that is both data and memory efficient. We first propose a method that learns to dynamically prompt (L2P) a pre-trained model to learn tasks sequentially under different task transitions, where prompts are small learnable parameters maintained in a memory space. We then improve L2P by proposing DualPrompt, which decouples prompts into complementary “General” and “Expert” prompts to learn task-invariant and task-specific instructions, respectively.

Finally, we propose DualHSIC, a simple and effective CL method that generalizes the idea of leveraging task-invariant and task-specific knowledge. DualHSIC consists of two complementary components that stem from the so-called Hilbert Schmidt independence criterion (HSIC): HSIC-Bottleneck for Rehearsal (HBR) lessens the inter-task interference and HSIC Alignment (HA) promotes task-invariant knowledge sharing.

Comprehensive experimental results demonstrate the effectiveness and efficiency of our methods over the state-of-the-art methods on multiple CL benchmarks.