This page contains resources about Artificial Neural Networks.
For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems.
Subfields and Concepts[]
- Feedforward Neural Network
- Single-Layer Perceptron (i.e. with no hidden layers)
- Multi-Layer Perceptron (MLP) / Standard Neural Network
- Radial Basis Function (RBF) Network
- Extreme Learning Machine (ELM)
- Convolutional Neural Network (CNN or ConvNet)
- Capsule Network (CapsNet)
- Recurrent Neural Network (RNN)
- Hopfield Network
- Boltzmann Machine
- Bidirectional RNN
- Bidirectional associative memory (BAM)
- Long short-term memory (LSTM)
- Gated Rectified Unit RNN (GRU-RNN)
- Simple Recurrent Network (SRN)
- Continuous Time RNN (CTRNN)
- RNN-RBM
- Echo State Network (ESN)
- Unitary RNN (uRNN)
- Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
- Helmholtz Machine
- Boltzmann Machine
- Restricted Boltzmann Machine (RBM)
- Conditional RBM (CRBM)
- Autoassociative memory
- Generative Stochastic Network
- Generative Adversarial Network
- Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
- Stochastic Computation Graph
- Variational Autoencoder (VAE)
- Natural-Parameter Network
- Variance Network
- Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
- Probabilistic Nerual Network
- Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
- Probabilistic Backpropagation
- Bayes by Backprop
- Bayesian Dark Knowledge (BDK)
- Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
- Gamma NPN
- Gaussian NPN
- Poisson NPN
- Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
- Random Neural Network
- Autoencoder (used for Dimensionality Reduction)
- Linear Autoencoder (equivalent to PCA)
- Stacked Denoising Autoencoder
- Generalized Denoising Autoencoder
- Sparse Autoencoder
- Contractive Autoencoder (CAE)
- Variational Autoencoder (VAE)
- Deep Neural Network (i.e. more than two hidden layers)
- Deep Multi-Layer Perceptron
- Deep Belief Network (DBN)
- Convolutional Deep Neural Network
- Long short-term memory (LSTM)
- Deep Autoencoder (i.e. two symmetrical DBN)
- Neural Module Network (NMN)
- HyperNetwork
- HyperLSTM
- Training
- Automatic Differentiation
- Backpropagation Algorithm
- Backpropagation Through Time (for training RNNs)
- Stochastic Backpropagation
- Optimization
- Stochastic Gradient Methods
- Stochastic Gradient Descent (SGD)
- SGD with Momentum
- Simulated Annealing
- Genetic Algorithms (for training RNNs)
- Stochastic Gradient Methods
- Contrastive Divergent (CD) Algorithm (for training RBMs)
- Persistent CD (PCD)
- Wake-Sleep Algorithm (for Stochastic ANNs)
- Generative Stochastic Networks (GSN) for probabilistic models
- Auto-Encoding Variational Bayes (AEVB) Algorithm
- Automatic Differentiation
- Activation Functions / Transfer Functions for deterministic units (must be differentiable)
- Logistic
- Rectifier (ReLU)
- Softmax
- Hyperbolic tangent
- Swish
- Cost Functions / Loss Functions / Objective Functions
- Least-Squares
- Cross-entropy
- Relative Entropy / KL Divergence
- Connectionist Temporal Classification (CTC)
- Energy-Based Model (EBM)
- Free energy (i.e. the contrastive term)
- Regularization term
- Loss Functionals / Loss Functions
- Energy Loss
- Generalized Perceptron Loss
- Generalized Margin Losses
- Negative Log-Likelihood Loss
- Improve Generalization (to prevent overfitting)
- Early stopping
- Regularization / Weight decay
- L1-regularization / Laplace prior
- L2-regularization / Gaussian prior
- Max norm constraints
- Dropout
- Add noise
- Theory of ANNs
- Representation Theorem
- Universal Approximation Theorem
- Universal Turing Machine
Online Courses[]
Video Lectures[]
- Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
- Neural networks class by Hugo Larochelle (Youtube )
- Deep Learning and Neural Networks by Kevin Duh
Lecture Notes[]
- CSC321: Introduction to Neural Networks and Machine Learning by Tijmen Tieleman - this might be a bit advanced for beginners
- CSC2515: Introduction to Machine Learning by Geoffrey Hinton - this is very similar to the above
- Neural Networks and Pattern Recognition by Ömer Cengiz ÇELEBİ
Books and Book Chapters[]
See Deep Learning Books.
Scholarly Articles[]
- Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11).
- Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
- Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation, 17(6), 1223-1263.
Tutorials[]
Software[]
See also[]
Other Resources[]
General[]
- artificial_neural_networks (Github)
- What is the difference between CNNs, RBMs and Autoencoders - Stackexchange
- Software Tools for RL, ANNs and Robotics - Python and MATLAB
- Neural Networks and Deep Learning - free online book
- Feedforward Neural Network - Metacademy
- Backpropagation - Metacademy
- Weight decay in Neural Networks - Metacademy
- The Neural Network Zoo - Blog post with some of the ANN architectures
- Neural Nets, Connectionism, Perceptrons, etc. - Notebook
- Neural Networks, Manifolds and Topology - Blog post
- Bayesian Neural Network in PyMC3 (and Theano) - Blog post
- Bayesian Neural Network in Edward (and Tensorflow) - Blog post
- Hierarchical Bayesian ANN in PyMC3 (and Theano Lasagne) - Blog post
- 10 Misconceptions about Neural Networks - Blog post
- Artificial Neural Networks Explained - blog post
- Pruning deep neural networks to make them fast and small - blog post
- Neural Networks gone wild! They can sample from discrete distributions now! - blog post
- keras-surgeon - Github
- tensorpack - Github
- Sequence Modeling With CTC - blog post
- gumbel-softmax (Github) - code
- bayesgan (Github) - code
- Detectron - code
- uTensor(Github) - AI inference library based on Mbed and TensorFlow
- When is L2 weight decay better than L1 and when is max norm better? - Quora
TensorFlow[]
- tensorflow-tutorial - Github
- TensorFlow-Tutorials (Hvass-Labs) - Github
- TensorFlow-Tutorials (nlintz) - Github
- TensorFlow-Examples - Github
- stanford-tensorflow-tutorials - Github
- tensorlayer - Github
- TensorFlow-Book - Github
- Tensorflow-101 - Github
- tensorflow-generative-model-collection - Github
RNN[]
- Recurrent Neural Network - Metacademy
- Advanced Recurrent Neural Networks - blog post
- Recurrent Neural Networks for Language Modeling - blog post
- Awesome-RNN (Github) - A curated list of resources dedicated to RNN
- Predict Stock Prices Using RNN (Part 1, Part 2) - blog post
- Introduction to LSTMs with TensorFlow - Blog post
- Understanding LSTM Networks - Blog post
- Understanding LSTM Networks by Example using Torch - Blog post
- Time Series Forecasting with the Long Short-Term Memory Network in Python - blog post
- Multi-step Time Series Forecasting with Long Short-Term Memory Networks in Python - blog post
- How to Seed State for LSTMs for Time Series Forecasting in Python - blog post
- Multivariate Time Series Forecasting with LSTMs in Keras - blog post
- Unfolding RNNs (Part 1, Part 2) - blog post
- LSTM implementation explained - blog post
- Time Series Prediction Using LSTM Deep Neural Networks - blog post
- Stock Market Predictions with LSTM in Python - blog post
- Stock prediction LSTM using Keras (Kaggle)
- Predict stock prices with LSTM (Kaggle)
- New-York-Stock-Exchange-Predictions-RNN-LSTM (GitHub) - code
- Vanilla Recurrent Neural Networks - blog post
- How (not) to use Machine Learning for time series forecasting: Avoiding the pitfalls - blog post
- Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras