Topics in scalable Bayesian posterior estimation

As the complexity and dimensionality of data continue to increase, it is becoming fundamental to develop advanced strategies for statistical inference and to explore their computational properties (Bishop, 2006). This thesis considers Bayesian models, known for their ability to frame prediction and uncertainty within a coherent probabilistic framework. However, achieving accurate estimates of posterior quantities within these models generally requires innovative techniques to accommodate the challenges of modern data analysis. We aim at developing algorithms for exact and approximate posterior estimation exhibiting linear computational cost in the number of parameters, for asymptotic settings where both the numbers of parameters and observations grow to infinity. Such performances are substantially unattainable for state of the art gradient based sampling methods, and are achieved only leveraging the hidden probabilistic structure of the models under consideration. The first and second chapters of this document focus on couplings, a relatively simple probabilistic construction whose potential for unbiased estimation has been recently spotlighted thanks to the work of (Glynn and Rhee, 2014; Jacob et al., 2020). After a brief review on couplings and their applications for unbiased sampling and estimation in Chapter 1, we present in Chapter 2 theoretical results bounding the computational effort required by the coupling construction of Jacob et al. (2020) for certain Gibbs samplers, proving its scalability in a wide range of applications, spanning from crossed random effect to sparse graphical models. Unbiased estimation via couplings therefore presents a promising way to enhance the precision and accuracy of statistical inference, offering insights beyond traditional estimation approaches. Turning to Chapter 3, we cover topics related to variational inference. Variational inference has captured significant attention in the past decades: essentially, it translates the probabilistic problem of finding the posterior distribution as an optimization task (Blei et al., 2017). This chapter not only presents its theoretical foundations but also explores practical implementation and provides results on scalability of the mean field variational approximation for certain large scale hierarchical models. More in detail, assuming the data is randomly generated from a specific distribution, we characterize the rate at which the iterates produced by the coordinate ascent variational inference (CAVI) algorithm converge to a variational minimizer for large scale hierarchical models, proving dimension-free convergence under warm start assumptions. Our work builds upon (Ascolani and Zanella, 2024), highlighting the effectiveness of CAVI in efficiently approximating posterior quantities for models where Gibbs sampling has proved to be effective, given the inherent similarities between these coordinate-wise schemes (Tan and Nott, 2014). Chapter 4 contains some recent advances developed during my visiting period at Warwick University with professor Gareth Roberts. Specifically, we study some properties of Catalytic couplings (Breyer and Roberts, 2001), a coupling procedure well suited for settings where only unnormalized distributions are available and able to couple multiple chains at once. In summary, this dissertation aims at presenting efficient methods in the realm of Bayesian posterior estimation for models with sparse dependencies, such as hierarchical and crossed models, leveraging their probabilistic structure to obtain linear cost estimates. By investigating coupling methods and variational inference, we aim at helping bridge the gap between state-of-the-art statistical procedures and the understanding of their computational properties.