ECE PhD Proposal Review: Yulun Zhang

PhD Proposal Review: Deep Convolutional Neural Network for Image Restoration and Synthesis

Yulun Zhang

Location: Zoom Link

Abstract: In this presentation, I will introduce how to design powerful deep convolutional neural networks (CNNs) for efficient image restoration and synthesis tasks. Recently, deep convolutional neural network (CNN) has achieved great success for image restoration (IR) and provided hierarchical features at the same time. However, most deep CNN based IR models neglect to make full use of the hierarchical features from the original low-quality images, thereby resulting in relatively-low performance. We propose a novel and efficient residual dense network (RDN) to address this problem in IR. Then, we can make a better tradeoff between efficiency and effectiveness in exploiting the hierarchical features from all the convolutional layers.

We also observe that deeper networks for image SR are more difficult to train. The low-resolution inputs and features contain abundant low-frequency information, which is often treated equally across channels. Such an equal treatment for channels hence hinders the representational ability of CNNs. Residual in residual structure was proposed to firstly train very deep networks (over 400 layers) for image super-resolution. Attention mechanism (e.g., channel attention) is further explored in image restoration.

Plus, we investigate the feature representation in deep CNN for image synthesis, like image style transfer. Most existing methods treat the semantic patterns of style image uniformly. This treatment is not suitable for the real-world case and results unpleasing results on complex styles. In this presentation, we introduce a more flexible and general universal style transfer technique: multimodal style transfer (MST). We find the multimodal style representation and formulate style matching problem as an energy minimization one. Consequently, MST explicitly considers the matching of semantic patterns in content and style images. We also generalize MST to improve some existing methods.