This page contains resources about Artificial Neural Networks.

For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems.

Subfields and Concepts Edit

  • Feedforward Neural Network
    • Single-Layer Perceptron (i.e. with no hidden layers)
    • Multi-Layer Perceptron (MLP) / Standard Neural Network
    • Radial Basis Function (RBF) Network
    • Extreme Learning Machine (ELM)
    • Convolutional Neural Network (CNN or ConvNet)
    • Capsule Network (CapsNet)
  • Recurrent Neural Network (RNN)
    • Hopfield Network
    • Boltzmann Machine
    • Bidirectional RNN
    • Bidirectional associative memory (BAM)
    • Long short-term memory (LSTM)
    • Gated Rectified Unit RNN (GRU-RNN)
    • Simple Recurrent Network (SRN)
    • Continuous Time RNN (CTRNN)
    • RNN-RBM
    • Echo State Network (ESN)
    • Unitary RNN (uRNN)
  • Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
    • Helmholtz Machine
    • Boltzmann Machine
    • Restricted Boltzmann Machine (RBM)
    • Conditional RBM (CRBM)
    • Autoassociative memory
    • Generative Stochastic Network
    • Generative Adversarial Network
    • Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
    • Stochastic Computation Graph
    • Variational Autoencoder (VAE)
    • Natural-Parameter Network
    • Variance Network
  • Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
  • Probabilistic Nerual Network
    • Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
      • Probabilistic Backpropagation
      • Bayes by Backprop
    • Bayesian Dark Knowledge (BDK)
    • Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
      • Gamma NPN
      • Gaussian NPN
      • Poisson NPN
  • Random Neural Network
  • Autoencoder (used for Dimensionality Reduction)
    • Linear Autoencoder (equivalent to PCA)
    • Stacked Denoising Autoencoder
    • Generalized Denoising Autoencoder
    • Sparse Autoencoder
    • Contractive Autoencoder (CAE)
    • Variational Autoencoder (VAE)
  • Deep Neural Network (i.e. more than two hidden layers)
    • Deep Multi-Layer Perceptron
    • Deep Belief Network (DBN)
    • Convolutional Deep Neural Network
    • Long short-term memory (LSTM)
    • Deep Autoencoder (i.e. two symmetrical DBN)
    • Neural Module Network (NMN)
  • HyperNetwork
    • HyperLSTM
  • Training
    • Automatic Differentiation
      • Backpropagation Algorithm
      • Backpropagation Through Time (for training RNNs)
      • Stochastic Backpropagation
    • Optimization
    • Contrastive Divergent (CD) Algorithm (for training RBMs)
      • Persistent CD (PCD)
    • Wake-Sleep Algorithm (for Stochastic ANNs)
    • Generative Stochastic Networks (GSN) for probabilistic models
    • Auto-Encoding Variational Bayes (AEVB) Algorithm
  • Activation Functions / Transfer Functions for deterministic units (must be differentiable)
    • Logistic
    • Rectifier (ReLU)
    • Softmax
    • Hyperbolic tangent
    • Swish
  • Cost Functions / Loss Functions / Objective Functions
    • Least-Squares
    • Cross-entropy
    • Relative Entropy / KL Divergence
    • Connectionist Temporal Classification (CTC)
  • Energy-Based Model (EBM)
    • Free energy (i.e. the contrastive term)
    • Regularization term
    • Loss Functionals / Loss Functions
      • Energy Loss
      • Generalized Perceptron Loss
      • Generalized Margin Losses
      • Negative Log-Likelihood Loss
  • Improve Generalization (to prevent overfitting)
    • Early stopping
    • Regularization / Weight decay
      • L1-regularization / Laplace prior
      • L2-regularization / Gaussian prior
      • Max norm constraints
    • Dropout
    • Add noise
  • Theory of ANNs
    • Representation Theorem
    • Universal Approximation Theorem
    • Universal Turing Machine

Online Courses Edit

Video LecturesEdit

Lecture NotesEdit

Books and Book Chapters Edit

See Deep Learning Books.

Scholarly Articles Edit

  • Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11).
  • Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
  • Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation17(6), 1223-1263.

Tutorials Edit

Software Edit

See Deep Learning Software.

See also Edit

Other Resources Edit

General Edit

TensorFlow Edit

RNN Edit

Community content is available under CC-BY-SA unless otherwise noted.