Department of Statistics, Departmental Seminar
Title: Additivity of Information in Multilayer Networks viaAdditive Gaussian Noise Transforms
Speaker: Galen Reeves, Duke University
Date: November 2
Location: Jordan Hall Building 420, Room 040
Multilayer (or deep) networks are powerful probabilistic models based on multiple stagesof a linear transform followed by a non-linear (possibly random) function. In general,the linear transforms are defined by matrices and the non-linear functions are definedby information channels. These models have gained great popularity due to their abilityto characterize complex probabilistic relationships arising in a wide variety of inferenceproblems. In this talk, we will describe a new method for analyzing the fundamental limitsof statistical inference in settings where the model is known. The validity of our methodcan be established in a number of settings and is conjectured to hold more generally. A keyassumption made throughout is that the matrices are drawn randomly from orthogonallyinvariant distributions.
Our method yields explicit formulas for 1) the mutual information; 2) the minimum meansquarederror (MMSE); 3) the existence and locations of certain phase-transitions withrespect to the problem parameters; and 4) the stationary points for the state evolution ofapproximate message passing algorithms. When applied to the special case of models withmultivariate Gaussian channels our method is rigorous and has close connections to freeprobability theory for random matrices. When applied to the general case of non-Gaussianchannels, our method provides a simple alternative to the replica method from statisticalphysics. A key observation is that the combined effects of the individual components inthe model (namely the matrices and the channels) are additive when viewed in a certaintransform domain.
The paper is available at https://arxiv.org/abs/1710.04580.