Learning Deep Low-Dimensional Models from High-Dimensional Data: From Theory to Practice

The first part will introduce fundamental properties and results for sensing, processing, analyzing, and learning low-dimensional structures from high-dimensional data. We will first discuss classical low-dimensional models, such as sparse coding and low-rank matrix sensing, and motivate these models by applications in computer vision. Based on convex relaxation, we will characterize the conditions, in terms of sample/data complexity, under which the learning problems of recovering such low-dimensional structures become tractable and can be solved efficiently, with guaranteed correctness or accuracy.

Next, we delve into the core principles of low-dimensional models within various neural network architectures. Specifically, we will introduce unrolled optimization as a design principle for interpretable deep networks. As a simple special case, we will examine several unrolled optimization algorithms for sparse coding, and show that they exhibit striking similarities to current deep network architectures. These unrolled networks are white-box and interpretable ab initio.

The session focuses on the strong conceptual connections between low-dimensional structures and deep models in terms of learned representation. We start with the introduction of an intriguing Neural Collapse phenomenon in the last-layer representation and its universality in deep network, and lay out the mathematical foundations of understanding its cause by studying its optimization landscapes. We then generalize and explain this phenomenon and its implications under data imbalanceness. Furthermore, we demonstrate the practical algorithmic implications of Neural Collapse on training deep neural networks.

Second, we show how low-dimensional structures also emerge in intermediate layers of learned deep neural networks. Specifically, we will characterize the progressive feature compression and discrimination from shallow to deep layers. Such a phenomenon is strongly correlated with the low-dimensional structure in learning dynamics, where the evolution of gradient descent only affects a minimal portion of singular vector spaces across all weight matrices.

We will establish a thorough understanding of the generalization of diffusion models by investigating when and why they are capable of learning the low-dimensional structure of target distributions. We will study the sample complexity to learn these low-dimensional distributions and how they scale with the intrinsic dimensionality of data. We will also show how diffusion models transit from memorization to generalization in terms of model capacity and data size.

We will enhance diffusion models for scientific imaging by improving their efficiency, robustness, and controllability in solving general inverse problems. Our approach focuses on latent-space and patch-based models for efficiency, novel techniques for data consistency in reverse sampling—especially for 3D imaging—and controllable sampling to enforce desired constraints while maintaining sample quality.

We focus on learning data distribution and transforming it into a linear discriminative representation (LDR). Using information-theoretic and statistical principles, we design a loss function called coding rate reduction, which is optimized at such a representation. By unrolling gradient ascent on this loss, we construct ReduNet, a deep network where each operator has a mathematically precise, interpretable role in transforming data toward an LDR. Notably, ReduNet can be built layer-wise through forward propagation, eliminating the need for back-propagation.

Finally, we introduce sparse linear discriminative representations by combining sparse coding with rate reduction, optimizing an objective called sparse rate reduction. This leads to CRATE, a deep network derived by unrolling the optimization and parameterizing feature distributions layer-wise. CRATE's operators are mathematically interpretable, with each layer corresponding to an optimization step, making it a white-box model. Despite its distinct design from ReduNet, both share a common goal, highlighting the flexibility of unrolled optimization. Interestingly, CRATE closely resembles transformer architectures, suggesting that such interpretability advances could enhance our understanding of modern deep networks.