Explaining Machine Learning and Memorization with Statistical Mechanics

Artificial neural networks (NNs) and machine learning (ML) algorithms are poorly understood from a theoretical perspective, which makes it difficult to fully realize their potential and overcome their weaknesses. For instance, ML algorithms train NN weights by moving them along a low-dimensional subspace of their allowed values, but this implicitly low-dimensional learning structure is not properly exploited to improve training because its nature is not well understood. Moreover, trained NNs are easily confused by pervasive adversarial attacks whose theoretical underpinnings are still unclear. This thesis aims to improve our theoretical understanding of NNs and ML, with a particular focus on adversarial attacks and implicitly low-dimensional learning. For this purpose, we use mathematical tools from statistical mechanics to study different types of NNs and ways in which they can fit the data. In particular, we study two classes of models that fit the data with various degrees of learning and memorization: dense associative memory (DAM) and restricted Boltzmann machines (RBM). In the process, we investigate connections between different versions of these models that are useful to make analytical investigations more efficient. First, we study a type of DAM called dense Hopfield network (dense HN) in the teacher-student setting where it is trained using data generated by another dense HN. On the Nishimori line, we show that the phase where dense HNs in the teacher-student setting are able to learn data coincides with the spin-glass phase of dense HNs with random memorized patterns. Outside the Nishimori line, we investigate the noise tolerance and adversarial robustness of dense HNs. In particular, we derive an exact formula for the adversarial robustness of the student at zero temperature, and we clarify why the adversarial robustness of dense HNs changes as a function of the learning regime. Second, we study RBMs in the teacher-student setting. When the teacher's weights are uncorrelated, we validate the conjecture that the performance of the student in learning them is independent of the number of hidden units. Moreover, we show that a student that is larger than necessary to learn the teacher's weights adopts a low-dimensional learning strategy in which only a subset of its hidden units end up correlated with those of the teacher, which we argue can be used as a toy model for studying the lottery ticket hypothesis. When the teacher's weights are correlated together rather than purely random, we show that the student crosses multiple regimes of data representation where it learns them in increasingly detailed ways as the number of samples in its training dataset increases. Finally, we study a type of RBM that belongs to the class of DAMs and is capable of both supervised and unsupervised classification. As before, our methods are based on statistical mechanics calculations in the teacher-student setting. We propose a novel regularization scheme inspired by these calculations, which we find to make training on real data significantly more stable. Moreover, we show that the weights learned by relatively small DAMs trained on both real and synthetic data are saddle points of larger DAMs, and we implement an algorithm that uses this hierarchy to significantly accelerate training on real data.