Optimization



This page contains resources about Mathematical Optimization, Computational Optimization and Operations Research.

More specific information is included in each subfield.

Subfields and Concepts
See Category:Optimization for some of  its subfields.
 * Cost Function / Loss Function / Objective Function
 * Convex Optimization
 * Linear Programming
 * Big M method
 * Simplex Algorithm
 * Quadratic Programming
 * Nonlinear Programming
 * Karush–Kuhn–Tucker (KKT) conditions
 * Combinatorial / Discrete Optimization
 * Integer Programming
 * Dynamic Programming / Optimal Control
 * Deterministic Optimal Control
 * Stochastic Optimal Control
 * Lyapunov Optimization
 * Greedy Algorithm
 * Travelling Salesman Problem (TSP)
 * Approximation Algorithms
 * Online Convex Optimization
 * Mini-Batch Learning
 * Variational Analysis‎
 * Calculus of variations
 * Robust Optimization
 * Lagrange Multipliers
 * Online Optimization
 * Iterative Methods
 * Powell's Method / Powell's Conjugate Direction Method
 * Nelder–Mead Method / Downhill Simplex Method / Amoeba Method
 * Constrained Optimization by Linear Approximation (COBYLA)
 * Expectation-Maximization (EM) Algorithm
 * Levenberg–Marquardt Algorithm
 * Iteratively Reweighted Least Squares
 * Nonlinear Least Squares
 * Ordinary Least Squares / Linear Least Squares
 * Weighted Least Squares
 * Sequential Quadratic Programming (SQP) / Sequential Least Squares Programming (SLSQP)
 * Gauss–Newton Algorithm
 * Subgradient Methods (used for Convex Minimization)
 * Gradient Descent / Steepest Descent
 * Krylov Subspace Methods
 * Conjugate Gradient Method
 * Biconjugate Gradient stabilized (BiCGSTAB) Method
 * Arnoldi Method
 * Lanczos Method
 * Generalized Minimal Residual (GMRES) Method
 * Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm
 * Limited-memory BFGS (L-BFGS)
 * Truncated Newton Methods / Hessian-free Optimization
 * Resilient Backpropagation (Rprop)
 * Bayesian Optimization
 * Stochastic Optimization
 * Gradient Descent Methods (either full-batch or mini-batch or both)
 * Stochastic Gradient Descent (SGD)
 * Stochastic Gradient Descent with Cyclical Learning Rates (using Triangular Policy)
 * Stochastic Gradient Descent with Restarts (SGDR) / Cyclic Cosine Annealing
 * Stochastic Weight Averaging (SWA)
 * SGD with Momentum
 * Averaged SGD
 * AdaDelta
 * AdaGrad
 * Adam
 * RMSprop
 * Nesterov’s Accelerated Gradient (NAG) Descent
 * NAdam (NAG/Nesterov Adam)
 * Projected Gradient Descent
 * Particle Mirror Descent (PMD)
 * Regularized Dual Averaging (RDA)
 * Follow the regularised leader (FTRL)
 * Online Gradient Descent
 * Adaptive Online Gradient Descent
 * Natural Gradient Descent
 * Stochastic Gradient Fisher Scoring
 * Stochastic Gradient Langevin Dynamics (SGLD)
 * Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
 * Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC)
 * Stochastic Gradient Markov Chain Monte Carlo (SGMCMC)
 * Stochastic Gradient Nose-Hoover Thermostat (SGNHT)
 * Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo
 * Stochastic Approximation
 * Robbins-Monro Algorithm (using noisy estimates of the gradient)
 * Metaheuristics
 * Population-based search
 * Evolutionary Algorithms
 * Genetic Algorithms
 * Swarm Intelligence
 * Single point search / Trajectory methods / Single-state methods
 * Hill-Climbing
 * Simulated Annealing
 * Tabu search
 * Explorative search methods
 * Greedy Randomized Adaptive Search Procedure (GRASP)
 * Variable Neighborhood Search (VNS)
 * Guided Local Search (GLS)
 * Iterated Local Search (ILS)
 * Inverse Problems
 * Regularization
 * L1-regularization / Least absolute shrinkage and selection operator (LASSO)
 * L2-regularization / Ridge Regression / Tikhonov Regularization
 * Early Stopping
 * Total Variation (TV) Regularization
 * Dropout
 * Stochastic Simulation / Monte Carlo Methods
 * Multi-Objective Optimization / Multicriteria Optimization / Pareto Optimization
 * Multi-Objective Linear Programming

Video Lectures

 * Optimization Course by Michael Zibulevsky
 * Convex Optimization I by Stephen P. Boyd
 * Convex Optimization II by Stephen P. Boyd
 * Discrete Optimization by Professor Pascal Van Hentenryck - Coursera
 * Linear and Discrete Optimization by Friedrich Eisenbrand - Coursera
 * Advanced Optimization and Randomized Methods by A. Smola and S. Sra
 * Optimization, Learning and Systems by Martin Jaggi
 * Approximate Dynamic Programming Lectures by Dimitri P. Bertsekas
 * Optimization

Introductory

 * Practical Optimization: A Gentle Introduction by John W. Chinneck - a very introductory course
 * Lecture Notes on Optimization by Pravin Varaiya
 * Lecture Notes on Engineering Optimization by Fraser J. Forbes and Ilyasse Aksikas
 * Optimization by Geoff Gordon and Ryan Tibshirani
 * Optimization: An introduction by A. Astolﬁ
 * Optimization for Differential Equations by Martin Berggren and Krister Svanberg
 * Quantitative Methods by Michael Trick
 * Operations Research. Linear Programming by Ana Zelaia Jauregi

Specialized

 * Convex Analysis and Optimization by Dimitri Bertsekas
 * Combinatorial Optimization by Santosh Vempala
 * Discrete Optimization for Vision and Learning by Nikos Komodakis and Pawan Kumar
 * Lectures on Modern Convex Optimization by Aharon Ben-Tal and Arkadi Nemirovski
 * Numerical Optimization by Christopher Griffin
 * Lecture Notes on Continuous Optimization by Kok Lay Teo and Song Wang
 * Introduction to Online Optimization by Sébastien Bubeck
 * Lecture Notes on Online Learning by Alexander Rakhlin
 * EECS260 Optimization by Miguel Á. Carreira-Perpiñán
 * Optimization Techniques in Engineering by Alan R. Parkinson and John D. Hedengren
 * Multidisciplinary System Design Optimization by Olivier de Weck and Karen Willcox
 * ORF523: Advanced Optimization by Sébastien Bubeck
 * Dynamic Programming and Optimal Control by Raffaello D'Andrea
 * An Introduction to Inverse Problems by Colin Fox, Geoff K. Nicholls and Sze M. Tan
 * Inverse Problem in Imaging by Simon Arridge

Introductory

 * Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons.
 * Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer.

Specialized

 * Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control. 4th Ed. Athena Scientific.
 * Bertsekas, D. P. (2016). Nonlinear Programming. 3rd Ed. Athena scientific.
 * Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2(3-4), 157-325.
 * Bertsekas, D. P. (2015). Convex Optimization Algorithms. Athena Scientific.
 * Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
 * Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons.
 * Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and Trends® in Machine Learning, 4(2), 107-194.
 * Sra, S., Nowozin, S., & Wright, S. J. (2012). Optimization for machine learning. MIT Press.
 * Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu.
 * Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press.
 * Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer.
 * Ruszczyński, A. (2006). Nonlinear Optimization. Princeton University Press.
 * Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press.
 * Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Springer.
 * Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam.
 * Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam.
 * Kelley, C. T. (1999). Iterative Methods for Optimization. Siam.
 * Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam.
 * Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier.
 * Scales, L. E. (1985). Introduction to non-linear optimization. Springer-Verlag.

Software
See List of Optimization Software  for the complete list.
 * Optimization Toolbox - MATLAB
 * Optimization (scipy.optimize) - Python
 * climin - Python
 * PyOpt - Python
 * dlib - C++ (with Python API)
 * CVXPY - Python
 * CVX - MATLAB
 * libLBFGS - C
 * PyLBFGS - Python
 * NLopt - C, C++, Fortran, Matlab or GNU Octave, Python, GNU Guile, Julia, GNU R, Lua, OCaml
 * hessianfree - Hessian-free Optimization for Deep Networks
 * MOSEK Fusion API - Python framework for conic optimization
 * MOSEK - C, Java, MATLAB, .NET, Python, R
 * Gurobi - various languages

Other Resources

 * Mathematical Optimization - Google Scholar Metrics (Top Publications)
 * NEOS Optimization Guide
 * COIN-OR - Computational Infrastructure for Operations Research
 * Decision Tree for Optimization Software Links to optimization source codes
 * Mathematical Programming Glossary
 * Optimization Related Links
 * Optimization - Notebook
 * Stochastic Approximation - Notebook
 * Convex Optimization - Metacademy
 * OPTML - NIPS Workshop on Optimization for Machine Learning
 * Metaheuristics and Local Search
 * A good book for Inverse Problems - Math StackExchange
 * A Global Optimization Algorithm Worth Using (for hyperparameters) - blog post in dlib
 * opt (GitHub) - code
 * Linear Regression – Theory - blog post
 * Gradient Descent - blog post
 * Implement Gradient Descent in Python (Medium) - blog post
 * Visualizing the gradient descent method - blog post