Artificial Neural Network

This page contains resources about Artificial Neural Networks.

For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems.

Subfields and Concepts

• Feedforward Neural Network
  • Single-Layer Perceptron (i.e. with no hidden layers)
  • Multi-Layer Perceptron (MLP) / Standard Neural Network
  • Radial Basis Function (RBF) Network
  • Extreme Learning Machine (ELM)
  • Convolutional Neural Network (CNN or ConvNet)
  • Capsule Network (CapsNet)
• Recurrent Neural Network (RNN)
  • Hopfield Network
  • Boltzmann Machine
  • Bidirectional RNN
  • Bidirectional associative memory (BAM)
  • Long short-term memory (LSTM)
  • Gated Rectified Unit RNN (GRU-RNN)
  • Simple Recurrent Network (SRN)
  • Continuous Time RNN (CTRNN)
  • RNN-RBM
  • Echo State Network (ESN)
  • Unitary RNN (uRNN)
• Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
  • Helmholtz Machine
  • Boltzmann Machine
  • Restricted Boltzmann Machine (RBM)
  • Conditional RBM (CRBM)
  • Autoassociative memory
  • Generative Stochastic Network
  • Generative Adversarial Network
  • Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
  • Stochastic Computation Graph
  • Variational Autoencoder (VAE)
  • Natural-Parameter Network
  • Variance Network
• Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
• Probabilistic Nerual Network
  • Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
    • Probabilistic Backpropagation
    • Bayes by Backprop
  • Bayesian Dark Knowledge (BDK)
  • Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
    • Gamma NPN
    • Gaussian NPN
    • Poisson NPN
• Random Neural Network
• Autoencoder (used for Dimensionality Reduction)
  • Linear Autoencoder (equivalent to PCA)
  • Stacked Denoising Autoencoder
  • Generalized Denoising Autoencoder
  • Sparse Autoencoder
  • Contractive Autoencoder (CAE)
  • Variational Autoencoder (VAE)
• Deep Neural Network (i.e. more than two hidden layers)
  • Deep Multi-Layer Perceptron
  • Deep Belief Network (DBN)
  • Convolutional Deep Neural Network
  • Long short-term memory (LSTM)
  • Deep Autoencoder (i.e. two symmetrical DBN)
  • Neural Module Network (NMN)
• HyperNetwork
  • HyperLSTM
• Training
  • Automatic Differentiation
    • Backpropagation Algorithm
    • Backpropagation Through Time (for training RNNs)
    • Stochastic Backpropagation
  • Optimization
    • Stochastic Gradient Methods
      • Stochastic Gradient Descent (SGD)
      • SGD with Momentum
    • Simulated Annealing
    • Genetic Algorithms (for training RNNs)
  • Contrastive Divergent (CD) Algorithm (for training RBMs)
    • Persistent CD (PCD)
  • Wake-Sleep Algorithm (for Stochastic ANNs)
  • Generative Stochastic Networks (GSN) for probabilistic models
  • Auto-Encoding Variational Bayes (AEVB) Algorithm
• Activation Functions / Transfer Functions for deterministic units (must be differentiable)
  • Logistic
  • Rectifier (ReLU)
  • Softmax
  • Hyperbolic tangent
  • Swish
• Cost Functions / Loss Functions / Objective Functions
  • Least-Squares
  • Cross-entropy
  • Relative Entropy / KL Divergence
  • Connectionist Temporal Classification (CTC)
• Energy-Based Model (EBM)
  • Free energy (i.e. the contrastive term)
  • Regularization term
  • Loss Functionals / Loss Functions
    • Energy Loss
    • Generalized Perceptron Loss
    • Generalized Margin Losses
    • Negative Log-Likelihood Loss
• Improve Generalization (to prevent overfitting)
  • Early stopping
  • Regularization / Weight decay
    • L1-regularization / Laplace prior
    • L2-regularization / Gaussian prior
    • Max norm constraints
  • Dropout
  • Add noise
• Theory of ANNs
  • Representation Theorem
  • Universal Approximation Theorem
  • Universal Turing Machine

Online Courses

Video Lectures

• Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
• Neural networks class by Hugo Larochelle (Youtube )
• Deep Learning and Neural Networks by Kevin Duh

Lecture Notes

• Convolutional Neural Networks for Visual Recognition by Fei-Fei Li & Andrej Karpathy

• CSC321: Introduction to Neural Networks and Machine Learning by Tijmen Tieleman - this might be a bit advanced for beginners
• CSC2515: Introduction to Machine Learning by Geoffrey Hinton - this is very similar to the above
• Neural Networks and Pattern Recognition by Ömer Cengiz ÇELEBİ

Books and Book Chapters

See Deep Learning Books.

Scholarly Articles

• Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11).
• Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
• Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation, 17(6), 1223-1263.

Software

See Deep Learning Software.

See also

• Probabilistic Graphical Models
• Online Learning

Other Resources

General

• artificial_neural_networks (Github)
• What is the difference between CNNs, RBMs and Autoencoders - Stackexchange
• Software Tools for RL, ANNs and Robotics - Python and MATLAB
• Neural Networks and Deep Learning - free online book
• Feedforward Neural Network - Metacademy
• Backpropagation - Metacademy
• Weight decay in Neural Networks - Metacademy
• The Neural Network Zoo - Blog post with some of the ANN architectures
• Neural Nets, Connectionism, Perceptrons, etc.  - Notebook
• Neural Networks, Manifolds and Topology - Blog post
• Bayesian Neural Network in PyMC3 (and Theano) - Blog post
• Bayesian Neural Network in Edward (and Tensorflow) - Blog post
• Hierarchical Bayesian ANN in PyMC3 (and Theano Lasagne) - Blog post
• 10 Misconceptions about Neural Networks - Blog post
• Artificial Neural Networks Explained - blog post
• Pruning deep neural networks to make them fast and small - blog post
• Neural Networks gone wild! They can sample from discrete distributions now! - blog post
• keras-surgeon - Github
• tensorpack - Github
• Sequence Modeling With CTC - blog post
• gumbel-softmax (Github) - code
• bayesgan (Github) - code
• Detectron - code
• uTensor(Github) - AI inference library based on Mbed and TensorFlow
• When is L2 weight decay better than L1 and when is max norm better? - Quora

TensorFlow

• tensorflow-tutorial - Github
• TensorFlow-Tutorials (Hvass-Labs) - Github
• TensorFlow-Tutorials (nlintz) - Github
• TensorFlow-Examples - Github
• stanford-tensorflow-tutorials - Github
• tensorlayer - Github
• TensorFlow-Book - Github
• Tensorflow-101 - Github
• tensorflow-generative-model-collection - Github

RNN

• Recurrent Neural Network - Metacademy
• Advanced Recurrent Neural Networks - blog post
• Recurrent Neural Networks for Language Modeling - blog post
• Awesome-RNN (Github) - A curated list of resources dedicated to RNN
• Predict Stock Prices Using RNN (Part 1, Part 2) - blog post
• Introduction to LSTMs with TensorFlow - Blog post
• Understanding LSTM Networks - Blog post
• Understanding LSTM Networks by Example using Torch - Blog post
• Time Series Forecasting with the Long Short-Term Memory Network in Python - blog post
• Multi-step Time Series Forecasting with Long Short-Term Memory Networks in Python - blog post
• How to Seed State for LSTMs for Time Series Forecasting in Python - blog post
• Multivariate Time Series Forecasting with LSTMs in Keras - blog post
• Unfolding RNNs (Part 1, Part 2) - blog post
• LSTM implementation explained - blog post
• Time Series Prediction Using LSTM Deep Neural Networks - blog post
• Stock Market Predictions with LSTM in Python - blog post
• Stock prediction LSTM using Keras (Kaggle)
• Predict stock prices with LSTM (Kaggle)
• New-York-Stock-Exchange-Predictions-RNN-LSTM (GitHub) - code
• Vanilla Recurrent Neural Networks - blog post
• How (not) to use Machine Learning for time series forecasting: Avoiding the pitfalls - blog post
• Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras