Bayesian Machine Learning

This page contains resources about Bayesian Inference and Bayesian Machine Learning.

Bayesian Networks do not necessarily follow Bayesian approach, but they are named after Bayes' Rule.

Bayes' Rule can be used at both the parameter level and the model level.

Subfields and Concepts

 * Bayesian Model Selection / Bayesian Model Comparison
 * Bayes Factor
 * Bayesian Model Averaging (in Ensemble Learning)
 * Bayesian Information Criterion (BIC)


 * Bayesian Parameter Estimation
 * Bayesian Decision Theory
 * Bayesian Signal Processing
 * Bayesian Nonparametric Methods
 * Bayesian Feature Selection / Bayesian Variable Selection
 * Spike and Slab Method
 * Kuo & Mallick
 * Gibbs Variable Selection (GVS)
 * Stochastic Search Variable Selection (SSVS)
 * Adaptive shrinkage with Jeffreys' prior or Laplace prior
 * Reversible jump MCMC
 * Variational Bayesian Inference
 * Approximate Bayesian Computation (ABC)
 * Automatic Variational ABC
 * Variational Bayes with Intractable Likelihood (VBIL)
 * Bayesian Information Theory
 * The Principle of Maximum Entropy
 * Bayesian Occam's Razor
 * Minimum Message Length (MML)


 * Bayesian Linear (Regression) Model
 * Bayesian Naive Bayes
 * Bayesian Mixture Models
 * Sparse Bayesian Models
 * Bayesian LASSO
 * Bayesian Ridge Regression
 * Relevance Vector Machine (RVM) / Automatic Relevance Determination  / Sparse Bayesian Learning
 * The Spike and Slab Model
 * Bayesian State Space Models
 * Bayesian Linear Dynamical System
 * Bayesian Time Series
 * Bayesian Structural Time Series (BSTS)
 * Kalman filter
 * Spike and Slab Method
 * Bayesian Model Averaging
 * Probabilistic Matrix Factorization
 * Bayesian Multitask Learning
 * Bayesian Optimization
 * Bayesian Reinforcement Learning
 * Bayesian Neural Network
 * Bayesian Deep Learning

Video Lectures

 * Bayesian Learning by Zoubin Ghahramani - VideoLectures.Net
 * Graphical modelling and Bayesian structural learning by Peter Green - VideoLectures.Net

Lecture Notes

 * CSC 2541: Topics in Machine Learning: Bayesian Methods for Machine Learning by Radford Neal
 * CSE 515T: Bayesian Methods in Machine Learning by Roman Garnett
 * Advanced Statistical Machine Learning by Stefanos Zafeiriou

Books and Book Chapters

 * Davidson-Pilon, C. (2015). Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference. Addison-Wesley Professional.
 * Koduvely, H. M. (2015). Learning Bayesian Models with R. Packt Publishing.
 * Theodoridis, S. (2015). "Section 12: Bayesian Learning". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
 * Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2014). Bayesian data analysis. 3rd Ed. Chapman & Hall/CRC.
 * Lee, P. M. (2012). Bayesian statistics: an introduction. 4th Ed. John Wiley & Sons.
 * Barber, D. (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Duda, R. O., Hart, P. E., & Stork, D. G. (2012). Pattern Classification. John Wiley & Sons.
 * Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.
 * Berger, J. O. (2010). Statistical decision theory and Bayesian analysis. 2nd Ed. Springer Science & Business Media.
 * Koller, D., & Friedman, N. (2009). Probabilistic Graphical Models. MIT Press.
 * Barber, D., Cemgil, A. T., & Chiappa, S. (2011). Bayesian time series models. Cambridge University Press.


 * Bishop, C. M. (2006). "Section 2.2: The beta distribution". Pattern Recognition and Machine Learning. Springer.


 * MacKay, D. J. (2003). "Chapter 37: Bayesian Inference and Sampling Theory". Information Theory, Inference and Learning Algorithms. Cambridge University Press.


 * Mitchell, T. M. (1997). "Chapter 6: Bayesian Learning". Machine Learning. McGraw Hill.


 * Pearl, J. (1988). "Chapter 2: Bayesian Inference". Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann.

Scholarly Articles
See also Further Reading.
 * Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.
 * Ghahramani, Z. (2015). Probabilistic machine learning and artificial intelligence. Nature, 521(7553), 452-459.
 * Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight Uncertainty in Neural Network. In Proceedings of The 32nd International Conference on Machine Learning (pp. 1613-1622).
 * Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., & Adams, R. P. (2015, February). Scalable bayesian optimization using deep neural networks. In International Conference on Machine Learning.

Tutorials

 * Heckerman's Bayes Net Learning Tutorial
 * A Brief Introduction to Graphical Models and Bayesian Networks
 * A brief introduction to Bayes' Rule
 * An Introduction to Graphical Models by M. Jordan
 * Bayesian Modelling in Machine Learning: A Tutorial Review
 * Bayesian Methods for Machine Learning - NIPS 2004
 * Bayesian Machine Learning by Ian Murray
 * Bayesian Machine Learning by Zoubin Ghahramani
 * Dynamical Systems, Stochastic Processes and Bayesian Inference - NIPS 2016 workshop

Software

 * Bayesian Probabilistic Matrix Factorization - MATLAB
 * Bayesian Modeling and Monte Carlo Methods - MATLAB
 * Bayesian Optimization (Statistics and Machine Learning Toolbox) - MATLAB
 * Bayesian Methods for Hackers - Python
 * Infer.NET - Developed by Microsoft Research
 * OpenBUGS - Bayesian Inference Using Gibbs Sampling

Other Resources

 * Bayesian Inference with PyMC3 (Part 1, Part 2, Part 3) - Python
 * A Bayesian Approach to Monitoring Process Change (Part 1, Part 2, Part 3) - Python
 * Bayesian Inference in R
 * Bayesian machine learning - Introduction
 * Bayesian machine learning - FastML
 * Bayesian machine learning - Metacademy
 * Bayesian Statistics - Scholarpedia
 * Are "Bayesian networks" Bayesian? - No, Bayesian and Frequentist approaches can both be used.
 * Bayesian Deep Learning - NIPS 2016 workshop
 * Bayesian Methods Research Group
 * Bayesian Learning for Statistical Classification - blog post