Probability and Statistics



This page contains resources about Probability Theory and Statistics in general.

More specific information is included in each subfield.

Subfields and Concepts
See Category:Probability and Statistics for all its subfields.
 * Statistical Inference / Inferential Statistics
 * Frequentist Inference
 * Statistical Hypothesis Testing / Statistical Tests
 * Fisher's Null Hypothesis Testing
 * Neyman-Pearson Theory
 * Analysis of Variance (ANOVA)
 * Analysis of Covariance (ANCOVA)
 * Multivariate Analysis of Variance (MANOVA)
 * T-test
 * F-test
 * Tests of Goodness-of-Fit
 * Confidence Intervals
 * Bootstrapping
 * Bayesian Inference
 * Bayes Factor
 * Credible Intervals
 * Variational Bayesian Inference
 * Bayesian Nonparametrics
 * Empirical Bayes / Maximum Marginal Likelihood
 * Hierarchical Bayes
 * Inductive inference
 * Causal Inference
 * Interval Estimation
 * Estimation Theory / Point Estimation
 * Least Squares filters
 * Kalman filter
 * Wiener filter
 * Monte Carlo Methods
 * Expectation-Maximization Algorithm
 * Maximum Likelihood Estimator (MLE)
 * Maximum a posteriori (MAP) estimator
 * Bayes Estimator
 * Bayesian Decision Theory
 * Decision Theory
 * Neyman-Pearson Theory
 * The Expected Loss Principle
 * Optimal decision rules
 * Bayesian Decision Theory / Bayesian Estimator
 * Cost function / Loss function
 * Risk function
 * Admissibility
 * Unbiasedness
 * Minimaxity
 * Algorithmic Information Theory
 * Minimum Description Length (MDL)
 * Minimum Message Length (MML)
 * Occam's Razor
 * Kolmogorov Complexity
 * Model Selection and Evaluation
 * Akaike Information Criterion (AIC)
 * Bayesian Information Criterion (BIC)
 * Deviance Information Criterion (DIC)
 * Bayesian Predictive Information Criterion (BPIC)
 * Focused Information Criterion (FIC)
 * Bayesian Model Selection / Bayesian Model Comparison
 * Bayesian Model Averaging
 * Bayesian Parameter Estimation
 * Bayesian Nonparametrics
 * Minimum Description Length (MDL)
 * Minimum Message Length (MML)
 * Akaike Final Prediction Error (FPE)
 * Parzen's Criterion Autoregressive Transfer Function (CAT)
 * Cross-Validation
 * Statistical Hypothesis Testing (for Multilevel Models / Nested Models only)
 * Lagrange multiplier test / Score test / Score Method
 * Likelihood-ratio test
 * Wald test
 * Statistical Models
 * Regression Analysis
 * Linear Regression Model
 * Simple Linear Regression
 * Multiple Linear Regression (not to be confused with Multivariate Linear Regression)
 * General Linear Model / Multivariate Linear Regression
 * Generalized Linear Model (GLM or GLIM)
 * Poisson Regression
 * Least Squares Methods
 * Ordinary Least Squares / Linear Least Squares
 * Weighted Least Squares
 * Nonlinear Least Squares
 * Logistic Regression Model / Logit Model
 * Probit Model
 * Fixed Effects Model
 * Hierarchical Linear Models / Multilevel Models / Nested Data Models
 * Random Effects Model / Variance Components Model
 * Mixed Effects Models (not to be confused with Mixture Models)
 * Nonparametric Regression Models
 * Nonlinear Regression Models
 * Robust Regression Models
 * Random sample consensus (RANSAC)
 * Regularization
 * Ridge regression / Tikhonov regularization
 * Least absolute shrinkage and selection operator (LASSO)
 * Elastic Nets
 * Probabilistic Models
 * Stochastic Models (Stochastic Processes, Random Fields, ...)
 * Probabilistic Graphical Models
 * Latent Variable Models (i.e. Partially Observed Probabilistic Models)
 * Continuous Latent Variable Models
 * Discrete Latent Variable Models
 * State Space Models
 * Time Series Models
 * Probability Theory
 * Random Variables
 * Continuous Random Variables
 * Probability Density Function
 * Discrete Random Variables
 * Probability Mass Function
 * Jointly Distributed Random Variables
 * Joint Density Function
 * Independent Random Variables
 * Uncorrelated Random Variables
 * Moments of a distribution
 * First Moment / Mean
 * Second Moment / Variance
 * Third Moment / Skewness
 * Fourth Moment / Kurtosis
 * Probabilistic Models
 * Stochastic Convergence
 * Probability Space
 * Measure Space
 * State Space
 * Theorem of Total Probability
 * Central Limit Theorem
 * Queueing Theory
 * Martingale Theory
 * Ergodic Theory
 * Decision Theory
 * Measure Theory
 * Utility Theory

Video Lectures

 * Probabilistic Systems Analysis and Applied Probability by John Tsitsiklis
 * Introduction to Probability - The Science of Uncertainty by edX  - very similar to the above
 * Probability by Salman Khan
 * Statistics by Salman Khan
 * Combinations - Counting Using Combinations

Lecture Notes

 * Introduction to Probability and Statistics by Dmitry Panchenko
 * Economic Theory I by Eric Zivot

Statistical Inference and Theory of Statistics

 * Bruce, P., & Bruce, A. (2017). Practical Statistics for Data Scientists: 50 Essential Concepts. O'Reilly Media.
 * Imbens, G. W., & Rubin D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction.
 * Ross, S. M. (2014). Introduction to probability models. 11th Ed. Academic Press.
 * Smith, R. C. (2013). Uncertainty quantification: theory, implementation, and applications. SIAM.
 * Gentle, J. E. (2013). Theory of statistics. (link)
 * DeGroot, M. H., & Schervish, M. J. (2012). Probability and statistics. 4th Ed. Pearson.
 * Abu-Mostafa, Y. S., Magdon-Ismail, M., & Lin, H. T. (2012). Learning From Data. AMLBook.
 * Diez, D. M., Barr, C. D., & Cetinkaya-Rundel, M. (2012). OpenIntro Statistics. CreateSpace.
 * Ramachandran, K. M., & Tsokos, C. P. (2012). Mathematical Statistics with Applications in R. Elsevier.
 * Gentle, J. E. (2007). Matrix algebra: theory, computations, and applications in statistics. Springer Science & Business Media.
 * Rice, J. (2006). Mathematical statistics and data analysis. 3rd Ed. Duxbury Press.
 * Cox, D. R. (2006). Principles of statistical inference. Cambridge University Press.
 * Lavine, M. (2005). Introduction to Statistical Thought. Michael Lavine.
 * Young, G. A., & Smith, R. L. (2005). Essentials of statistical inference. Cambridge University Press.
 * Lehmann, E. L., & Casella, G. (2003). Theory of point estimation. Springer.
 * Bertsekas, D. P., & Tsitsiklis, J. N. (2002). Introduction to Probability. Athena scientific.
 * Casella, G., & Berger, R. L. (2002). Statistical inference. Cengage Learning.
 * Garthwaite, P. H., Jolliffe, I. T., & Jones, B. (2002). Statistical inference. Oxford University Press.
 * Shao, J. (2000). Mathematical Statistics. Springer.
 * Mukhopadhyay, N. (2000). Probability and statistical inference. CRC Press.
 * Schervish, M. J. (1995). Theory of statistics. Springer Science & Business Media.

Regression Analysis and Generalized Linear Models

 * Harrell, F. (2015). Regression modeling strategies. 2nd Ed. Springer.
 * Chatterjee, S., & Hadi, A. S. (2012). Regression analysis by example. 5th Ed. John Wiley & Sons.
 * Goldstein, H. (2010). Multilevel statistical models. 4th Ed. John Wiley & Sons.
 * Dobson, A. J., & Barnett, A. (2008). An introduction to generalized linear models. 3rd Ed. CRC press.
 * Fox, J. (2008). Applied regression analysis and generalized linear models. 2nd Ed. Sage Publications.
 * Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
 * Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). Semiparametric regression. Cambridge University Press.
 * Faraway, J. J. (2002). Practical regression and ANOVA using R. (link)


 * Draper, N. R., & Smith, H. (1998). Applied regression analysis. 3rd Ed. John Wiley & Sons.
 * Long, J. S., & Freese, J. (1997). Regression models for categorical dependent variables. Sage Publications.
 * McCullagh, P., & Nelder, J. A. (1989). Generalized linear models. CRC press.

Counting and Probability

 * Shu, Z. (2016). Probability and Expectation (Volume 14). World Scientific.


 * Zhou, X. (2015). Counting: Math for Gifted Students. CreateSpace.
 * Hollos, S. & Hollos, J. R. (2013). Probability Problems and Solutions. Abrazol Publishing.
 * Patrick, D. (2007). Introduction to Counting and Probability. 2nd Ed. AoPS Incorporated.
 * Patrick, D. (2007). Intermediate Counting and Probability. AoPS Incorporated.

Software
See List of Statistical packages for a complete list.
 * The Lightspeed Matlab Toolbox
 * Statistics and Machine Learning Toolbox - MATLAB
 * Statistical functions (scipy.stats) - Python
 * Statistics (numpy) - Python
 * Statsmodels - Statistical Modeling and Econometrics in Python
 * revrand - Python

Other Resources

 * Video Tutorials - Youtube channel of 'Mathematical Monk'
 * Probability and Statistics by Khan Academy
 * Statistics by Wikibooks
 * Statistics by Wikiversity
 * Statistics - Notebook
 * Probability Theory - Notebook
 * Algorithmic Information Theory - Notebook
 * Bayesian statistics: a comprehensive course by Ox Educ - Youtube
 * Random - Lessons in Probability, Mathematical Statistics and Stochastic Processes
 * Learning Machine Learning — Probability Theory Fundamentals - Medium