Bayesian Parameter Estimation

This page contains resources about Bayesian Parameter Estimation, Bayesian Parameter Learning and Bayes Estimator.

Subfields and Concepts

• For complete (fully observed data):
  • Dirichlet distribution (or other priors)
• For incomplete (hidden/missing data):
  • Markov Chain Monte Carlo (MCMC)
  • Viterbi Algorithm
  • Variational Bayes
    • Stochastic Gradient Variational Bayes (SGVB) Estimator
• Bayesian Hierarchical Modelling / Hierarchical Bayes Model
  • Hyperparameter
  • Hyperprior
• Bayesian Decision Theory
• Bayesian Point Estimation
• Bayesian Signal Processing
• Bayes Risk
• Bayesian Score
  • Posterior variance (when MSE is used)
• Bayes Risk Function / Posterior Expected Loss (i.e. Posterior Expectation Value of Loss Function)
  • Posterior mean / Minimum MSE (MMSE) estimator / Bayes least squared error (BLSE) estimator / Squared error loss
  • Posterior median / Median-unbiased estimator / Absolute error loss
  • Posterior mode
• Bayes estimator
  • MMSE / BLSE estimator
  • Median-unbiased estimator
  • Bayes estimator for conjugate priors (eg. exponential family)
• Maximum Likelihood Estimation (MLE)
• Asymptotics of Maximum Likelihood
• Cramer-Rao bound / Cramer-Rao lower bound
• Fisher information
• Uninformative priors / Noninformative priors / Maximum entropy priors
  • Jeffreys prior
• Maximum Entropy (Maxent) Models / Entropic priors
• Exponential family
• Beta distribution
• Bayesian Online Parameter Estimation
• Recursive Bayesian Estimation / Bayes filter
  • Kalman filter (special case of Bayes filter)
  • Wiener filter (special case of Kalman filter)
• Bayesian Density Estimation
• Nonparametric Empirical Bayes (NPEB)
• Parametric Empirical Bayes Point Estimation
• Iterative proportional fitting (IPF)
• Nonparametric Methods
  • Kernel Density Estimation (KDE)
  • k-Nearest Neighbours
  • Bayesian Nonparametrics

Books and Book Chapters

• Theodoridis, S. (2015). "Chapter 12: Bayesian Learning" Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
• Aster, R. C., Borchers, B., & Thurber, C. (2012). "Chapter 11: Bayesian Methods". Parameter estimation and inverse problems. Academic Press
• Barber, D. (2012). "Section 9.1: Learning as Inference". Bayesian Reasoning and Machine Learning. Cambridge University Press.
• Barber, D. (2012). "Chapter 18: Bayesian Linear Models". Bayesian Reasoning and Machine Learning. Cambridge University Press.
• Duda, R. O., Hart, P. E., & Stork, D. G. (2012). "Chapter 2: Bayesian Decision Theory". Pattern Classification. John Wiley & Sons.
• Murphy, K. P. (2012). "Section 5.7: Bayesian Decision Theory". Machine Learning: A Probabilistic Perspective. MIT Press.
• Koller, D., & Friedman, N. (2009). "Chapter 17: Parameter Estimation". Probabilistic Graphical Models. MIT Press.
• Theodoridis, S., Pikrakis, A., Koutroumbas, K., & Cavouras, D. (2008). "Chapter 2: Bayesian Decision Theory". Pattern Recognition. 4th Ed. Academic Press.
• Bishop, C. M. (2006). "Chapter 2: Probability Distributions". Pattern Recognition and Machine Learning. Springer.
• MacKay, D. J. (2003). "Chapter 24: Exact Marginalization". Information Theory, Inference and Learning Algorithms. Cambridge University Press.
• MacKay, D. J. (2003). "Chapter 36: Decision Theory". Information Theory, Inference and Learning Algorithms. Cambridge University Press.
• Bretthorst, G. L. (1998). Bayesian spectrum analysis and parameter estimation. Springer Science & Business Media.
• Berger, J. O. (1993). Statistical decision theory and Bayesian analysis. 2nd Ed. Springer Science & Business Media.

Scholarly Articles

• Caticha, A. (2010). Entropic inference. arXiv preprint arXiv:1011.0723.
• Caticha, A., & Preuss, R. (2004). Maximum entropy and Bayesian data analysis: Entropic prior distributions. Physical Review E, 70(4), 046127.
• Ghahramani, Z. (2003). "Graphical models: Parameter learning". In Handbook of Brain Theory and Neural Networks.
• Malouf, R. (2002). A comparison of algorithms for maximum entropy parameter estimation. In proceedings of the 6th conference on Natural language learning-Volume 20 (pp. 1-7). Association for Computational Linguistics.

See also

• Bayesian Model Selection
• Estimation Theory

Other resources

• Bayesian parameter estimation - Metacademy
• Maximum Entropy Modelling