Deep learning

From Wikipedia, the free encyclopedia

For deep versus shallow learning in educational psychology, see Student approaches to
learning.

Machine learning and

data mining

Problems[show]
Supervised learning
(classification regression)

[show]

Clustering[show]

Dimensionality reduction[show]

Structured prediction[show]

Anomaly detection[show]

Neural nets[show]

Reinforcement Learning[show]

Theory[show]

Machine learning venues[show]

Machine learning portal

Deep learning (also known as deep structured learning, hierarchical learning or deep
machine learning) is a branch of machine learning based on a set of algorithms that
attempt to model high level abstractions in data. In a simple case, you could have two sets
of neurons: ones that receive an input signal and ones that send an output signal. When
the input layer receives an input it passes on a modified version of the input to the next
layer. In a deep network, there are many layers between the input and output (and the
layers are not made of neurons but it can help to think of it that way), allowing the algorithm
to use multiple processing layers, composed of multiple linear and non-linear
transformations.[1][2][3][4][5][6][7][8][9]
Deep learning is part of a broader family of machine learning methods based on learning
representations of data. An observation (e.g., an image) can be represented in many ways
such as a vector of intensity values per pixel, or in a more abstract way as a set of edges,
regions of particular shape, etc. Some representations are better than others at simplifying
the learning task (e.g., face recognition or facial expression recognition[10]). One of the
promises of deep learning is replacing handcrafted features with efficient algorithms
for unsupervised or semi-supervised feature learning and hierarchical feature extraction.[11]
Research in this area attempts to make better representations and create models to learn
these representations from large-scale unlabeled data. Some of the representations are
inspired by advances in neuroscience and are loosely based on interpretation of
information processing and communication patterns in a nervous system, such as neural
coding which attempts to define a relationship between various stimuli and associated
neuronal responses in the brain.[12]
Various deep learning architectures such as deep neural networks, convolutional deep
neural networks, deep belief networks and recurrent neural networks have been applied to
fields like computer vision, automatic speech recognition, natural language processing,
audio recognition and bioinformatics where they have been shown to produce state-of-theart results on various tasks.
Deep learning has been characterized as a buzzword, or a rebranding of neural networks.[13]
[14]

Companies such as Google are giving their employees classes in how to use it. [15]
Contents
[hide]

1Introduction
o

1.1Definitions

1.2Fundamental concepts
2Interpretations

2.1Universal approximation theorem interpretation

2.2Probabilistic interpretation

3History

4Artificial neural networks

5Deep neural network architectures

5.1Brief discussion of deep neural networks

5.1.1Backpropagation

5.1.2Problems with deep neural networks

5.2First deep learning networks of 1965: GMDH

5.3Convolutional neural networks

5.4Neural history compressor

5.5Recursive neural networks

5.6Long short-term memory

5.7Deep belief networks

5.8Convolutional deep belief networks

5.9Large memory storage and retrieval neural networks

5.10Deep Boltzmann machines

5.11Stacked (de-noising) auto-encoders

5.12Deep stacking networks

5.13Tensor deep stacking networks

5.14Spike-and-slab RBMs

5.15Compound hierarchical-deep models

5.16Deep coding networks

5.17Deep Q-networks

5.18Networks with separate memory structures

5.18.1LSTM-related differentiable memory structures

5.18.2Semantic hashing

5.18.3Neural Turing machines

5.18.4Memory networks

5.18.5Pointer networks

5.18.6Encoderdecoder networks
6Other architectures

6.1Multilayer kernel machine

7Applications

7.1Automatic speech recognition

7.2Image recognition

7.3Natural language processing

7.4Drug discovery and toxicology

7.5Customer relationship management

7.6Recommendation systems

7.7Biomedical Informatics

8Theories of the human brain

9Commercial activities

10Criticism and comment

11Software libraries

12See also

13References

14External links

Introduction[edit]
Definitions[edit]
Deep learning is characterized as a class of machine learning algorithms that[2](pp199200)

use a cascade of many layers of nonlinear processing units for feature

extraction and transformation. Each successive layer uses the output from the previous

layer as input. The algorithms may be supervised or unsupervised and applications

include pattern analysis (unsupervised) and classification (supervised).

are based on the (unsupervised) learning of multiple levels of features or

representations of the data. Higher level features are derived from lower level features
to form a hierarchical representation.
are part of the broader machine learning field of learning representations of data.
learn multiple levels of representations that correspond to different levels of
abstraction; the levels form a hierarchy of concepts.

These definitions have in common (1) multiple layers of nonlinear processing units and (2)
the supervised or unsupervised learning of feature representations in each layer, with the
layers forming a hierarchy from low-level to high-level features.[2](p200) The composition of a
layer of nonlinear processing units used in a deep learning algorithm depends on the
problem to be solved. Layers that have been used in deep learning include hidden layers of
an artificial neural network and sets of complicated propositional formulas.[3] They may also
include latent variables organized layer-wise in deep generative models such as the nodes
in Deep Belief Networks and Deep Boltzmann Machines.
Deep learning algorithms transform their inputs through more layers than shallow learning
algorithms. At each layer, the signal is transformed by a processing unit, like an artificial
neuron, whose parameters are 'learned' through training.[5](p6) A chain of transformations
from input to output is a credit assignment path (CAP). CAPs describe potentially causal
connections between input and output and may vary in length for a feedforward neural
network, the depth of the CAPs (thus of the network) is the number of hidden layers plus
one (as the output layer is also parameterized), but for recurrent neural networks, in which
a signal may propagate through a layer more than once, the CAP is potentially unlimited in
length. There is no universally agreed upon threshold of depth dividing shallow learning
from deep learning, but most researchers in the field agree that deep learning has multiple
nonlinear layers (CAP > 2) and Juergen Schmidhuber considers CAP > 10 to be very deep
learning.[5](p7)

Fundamental concepts[edit]
Deep learning algorithms are based on distributed representations. The underlying
assumption behind distributed representations is that observed data are generated by the
interactions of factors organized in layers. Deep learning adds the assumption that these
layers of factors correspond to levels of abstraction or composition. Varying numbers of
layers and layer sizes can be used to provide different amounts of abstraction. [4]
Deep learning exploits this idea of hierarchical explanatory factors where higher level, more
abstract concepts are learned from the lower level ones. These architectures are often
constructed with a greedy layer-by-layer method. Deep learning helps to disentangle these
abstractions and pick out which features are useful for learning.[4]
For supervised learning tasks, deep learning methods obviate feature engineering, by
translating the data into compact intermediate representations akin to principal
components, and derive layered structures which remove redundancy in representation. [2]
Many deep learning algorithms are applied to unsupervised learning tasks. This is an
important benefit because unlabeled data are usually more abundant than labeled data.

Examples of deep structures that can be trained in an unsupervised manner are neural
history compressors[16] and deep belief networks.[4][17]

Interpretations[edit]
Deep neural networks are generally interpreted in terms of: Universal approximation
theorem[18][19][20][21][22] or Probabilistic inference.[2][3][4][5][17][23]

Universal approximation theorem interpretation[edit]

The universal approximation theorem concerns the capacity of feedforward neural
networks with a single hidden layer of finite size to approximate continuous functions.[18][19][20]
[21][22]

In 1989, the first proof was published by George Cybenko for sigmoid activation
functions[19] and was generalised to feed-forward multi-layer architectures in 1991 by Kurt
Hornik.[20]

Probabilistic interpretation[edit]
The probabilistic interpretation[23] derives from the field of machine learning. It features
inference,[2][3][4][5][17][23] as well as the optimization concepts of training and testing, related to
fitting and generalization respectively. More specifically, the probabilistic interpretation
considers the activation nonlinearity as a cumulative distribution function.[23] See Deep belief
network. The probabilistic interpretation led to the introduction of dropout as regularizer in
neural networks.[24]
The probabilistic interpretation was introduced and popularized by Geoff Hinton, Yoshua
Bengio, Yann LeCun and Juergen Schmidhuber.

History[edit]
The first general, working learning algorithm for supervised deep feedforward multilayer
perceptrons was published by Ivakhnenko and Lapa in 1965.[25] A 1971 paper[26] described a
deep network with 8 layers trained by the Group method of data handling algorithm which is
still popular in the current millennium[citation needed]. These ideas were implemented in a computer
identification system "Alpha", which demonstrated the learning process. Other Deep
Learning working architectures, specifically those built from artificial neural networks (ANN),
date back to the Neocognitron introduced by Kunihiko Fukushima in 1980.[27] The ANNs
themselves date back even further. The challenge was how to train networks with multiple
layers. In 1989, Yann LeCun et al. were able to apply the
standard backpropagation algorithm, which had been around as the reverse mode
of automatic differentiation since 1970,[28][29][30][31] to a deep neural network with the purpose of
recognizing handwritten ZIP codes on mail. Despite the success of applying the algorithm,
the time to train the network on this dataset was approximately 3 days, making it
impractical for general use.[32] In 1993, Jrgen Schmidhuber's neural history
compressor[16] implemented as an unsupervised stack of recurrent neural networks (RNNs)
solved a "Very Deep Learning" task[5] that requires more than 1,000 subsequent layers in an
RNN unfolded in time.[33] In 1995, Brendan Frey demonstrated that it was possible to train a
network containing six fully connected layers and several hundred hidden units using

the wake-sleep algorithm, which was co-developed with Peter Dayan and Geoffrey Hinton.
[34]
However, training took two days.
Many factors contribute to the slow speed, one being the vanishing gradient
problem analyzed in 1991 by Sepp Hochreiter.[35][36]
While by 1991 such neural networks were used for recognizing isolated 2-D hand-written
digits, recognizing 3-D objects was done by matching 2-D images with a handcrafted 3-D
object model. Juyang Weng et al. suggested that a human brain does not use a monolithic
3-D object model, and in 1992 they published Cresceptron,[37][38][39] a method for performing
3-D object recognition directly from cluttered scenes. Cresceptron is a cascade of layers
similar to Neocognitron. But while Neocognitron required a human programmer to handmerge features, Cresceptron automatically learned an open number of unsupervised
features in each layer, where each feature is represented by a convolution kernel.
Cresceptron also segmented each learned object from a cluttered scene through backanalysis through the network. Max pooling, now often adopted by deep neural networks
(e.g. ImageNet tests), was first used in Cresceptron to reduce the position resolution by a
factor of (2x2) to 1 through the cascade for better generalization. Despite these
advantages, simpler models that use task-specific handcrafted features such as Gabor
filters and support vector machines (SVMs) were a popular choice in the 1990s and 2000s,
because of the computational cost of ANNs at the time, and a great lack of understanding
of how the brain autonomously wires its biological networks.
In the long history of speech recognition, both shallow and deep learning (e.g., recurrent
nets) of artificial neural networks have been explored for many years. [40][41][42] But these
methods never won over the non-uniform internal-handcrafting Gaussian mixture
model/Hidden Markov model (GMM-HMM) technology based on generative models of
speech trained discriminatively.[43] A number of key difficulties have been methodologically
analyzed, including gradient diminishing[35] and weak temporal correlation structure in the
neural predictive models.[44][45] Additional difficulties were the lack of big training data and
weaker computing power in these early days. Thus, most speech recognition researchers
who understood such barriers moved away from neural nets to pursue generative
modeling. An exception was at SRI International in the late 1990s. Funded by the US
government's NSA and DARPA, SRI conducted research on deep neural networks in
speech and speaker recognition. The speaker recognition team, led by Larry Heck,
achieved the first significant success with deep neural networks in speech processing as
demonstrated in the 1998 NIST (National Institute of Standards and Technology) Speaker
Recognition evaluation and later published in the journal of Speech Communication.
[46]
While SRI established success with deep neural networks in speaker recognition, they
were unsuccessful in demonstrating similar success in speech recognition. Hinton et al.
and Deng et al. reviewed part of this recent history about how their collaboration with each
other and then with colleagues across four groups (University of Toronto, Microsoft,
Google, and IBM) ignited a renaissance of deep feedforward neural networks in speech
recognition.[47][48][49][50]
Today, however, many aspects of speech recognition have been taken over by a deep
learning method called Long short-term memory (LSTM), a recurrent neural
network published by Sepp Hochreiter & Jrgen Schmidhuber in 1997.[51] LSTM RNNs
avoid the vanishing gradient problem and can learn "Very Deep Learning" tasks[5] that
require memories of events that happened thousands of discrete time steps ago, which is
important for speech. In 2003, LSTM started to become competitive with traditional speech
recognizers on certain tasks.[52] Later it was combined with CTC[53] in stacks of LSTM RNNs.
[54]
In 2015, Google's speech recognition reportedly experienced a dramatic performance

jump of 49% through CTC-trained LSTM, which is now available through Google Voice to
all smartphone users,[55] and has become a show case of deep learning.
The use of the expression "Deep Learning" in the context of Artificial Neural Networks was
introduced by Igor Aizenberg and colleagues in 2000. [56] A Google Ngram chart shows that
the usage of the term has gained traction (actually has taken off) since 2000. [57] In 2006, a
publication by Geoffrey Hinton and Ruslan Salakhutdinov drew additional attention by
showing how many-layered feedforward neural network could be effectively pre-trained one
layer at a time, treating each layer in turn as an unsupervised restricted Boltzmann
machine, then fine-tuning it using supervised backpropagation.[58] In 1992, Schmidhuber
had already implemented a very similar idea for the more general case of unsupervised
deep hierarchies of recurrent neural networks, and also experimentally shown its benefits
for speeding up supervised learning.[16][59]
Since its resurgence, deep learning has become part of many state-of-the-art systems in
various disciplines, particularly computer vision and automatic speech recognition (ASR).
Results on commonly used evaluation sets such as TIMIT (ASR) and MNIST (image
classification), as well as a range of large-vocabulary speech recognition tasks are
constantly being improved with new applications of deep learning.[47][60][61] Recently, it was
shown that deep learning architectures in the form of convolutional neural networks have
been nearly best performing;[62][63] however, these are more widely used in computer vision
than in ASR, and modern large scale speech recognition is typically based on CTC [53] for
LSTM.[51][55][64][65][66]
The real impact of deep learning in industry apparently began in the early 2000s, when
CNNs already processed an estimated 10% to 20% of all the checks written in the US in
the early 2000s, according to Yann LeCun.[67] Industrial applications of deep learning to
large-scale speech recognition started around 2010. In late 2009, Li Deng invited Geoffrey
Hinton to work with him and colleagues at Microsoft Research in Redmond, Washington to
apply deep learning to speech recognition. They co-organized the 2009 NIPS Workshop on
Deep Learning for Speech Recognition. The workshop was motivated by the limitations of
deep generative models of speech, and the possibility that the big-compute, big-data era
warranted a serious try of deep neural nets (DNN). It was believed that pre-training DNNs
using generative models of deep belief nets (DBN) would overcome the main difficulties of
neural nets encountered in the 1990s.[49] However, early into this research at Microsoft, it
was discovered that without pre-training, but using large amounts of training data, and
especially DNNs designed with corresponding large, context-dependent output layers,
produced error rates dramatically lower than then-state-of-the-art GMM-HMM and also than
more advanced generative model-based speech recognition systems. This finding was
verified by several other major speech recognition research groups. [47][68] Further, the nature
of recognition errors produced by the two types of systems was found to be
characteristically different,[48][69] offering technical insights into how to integrate deep learning
into the existing highly efficient, run-time speech decoding system deployed by all major
players in speech recognition industry. The history of this significant development in deep
learning has been described and analyzed in recent books and articles. [2][70][71]
Advances in hardware have also been important in enabling the renewed interest in deep
learning. In particular, powerful graphics processing units (GPUs) are well-suited for the
kind of number crunching, matrix/vector math involved in machine learning. [72][73] GPUs have
been shown to speed up training algorithms by orders of magnitude, bringing running times
of weeks back to days.[74][75]

Artificial neural networks[edit]

Some of the most successful deep learning methods involve artificial neural networks.
Artificial neural networks are inspired by the 1959 biological model proposed by Nobel
laureates David H. Hubel & Torsten Wiesel, who found two types of cells in the primary
visual cortex: simple cells and complex cells. Many artificial neural networks can be viewed
as cascading models[37][38][39][76] of cell types inspired by these biological observations.
Fukushima's Neocognitron introduced convolutional neural networks partially trained
by unsupervised learning with human-directed features in the neural plane. Yann LeCun et
al. (1989) applied supervised backpropagation to such architectures.[77] Weng et al. (1992)
published convolutional neural networks Cresceptron[37][38][39] for 3-D object recognition from
images of cluttered scenes and segmentation of such objects from images.
An obvious need for recognizing general 3-D objects is least shift invariance and tolerance
to deformation. Max-pooling appeared to be first proposed by Cresceptron[37][38] to enable the
network to tolerate small-to-large deformation in a hierarchical way, while using
convolution. Max-pooling helps, but does not guarantee, shift-invariance at the pixel level. [39]
With the advent of the back-propagation algorithm based on automatic differentiation,[28][30][31]
[78][79][80][81][82][83][84]
many researchers tried to train supervised deep artificial neural networks from
scratch, initially with little success. Sepp Hochreiter's diploma thesis of 1991[35][36] formally
identified the reason for this failure as the vanishing gradient problem, which affects manylayered feedforward networks and recurrent neural networks. Recurrent networks are
trained by unfolding them into very deep feedforward networks, where a new layer is
created for each time step of an input sequence processed by the network. As errors
propagate from layer to layer, they shrink exponentially with the number of layers, impeding
the tuning of neuron weights which is based on those errors.
To overcome this problem, several methods were proposed. One is Jrgen Schmidhuber's
multi-level hierarchy of networks (1992) pre-trained one level at a time by unsupervised
learning, fine-tuned by backpropagation.[16] Here each level learns a compressed
representation of the observations that is fed to the next level.
Another method is the long short-term memory (LSTM) network
of Hochreiter & Schmidhuber (1997).[51] In 2009, deep multidimensional LSTM networks
won three ICDAR 2009 competitions in connected handwriting recognition, without any
prior knowledge about the three languages to be learned.[85][86]
Sven Behnke in 2003 relied only on the sign of the gradient (Rprop) when training his
Neural Abstraction Pyramid[87] to solve problems like image reconstruction and face
localization.
Other methods also use unsupervised pre-training to structure a neural network, making it
first learn generally useful feature detectors. Then the network is trained further by
supervised back-propagation to classify labeled data. The deep model of Hinton et al.
(2006) involves learning the distribution of a high-level representation using successive
layers of binary or real-valued latent variables. It uses a restricted Boltzmann
machine (Smolensky, 1986[88]) to model each new layer of higher level features. Each new
layer guarantees an increase on the lower-bound of the log likelihood of the data, thus
improving the model, if trained properly. Once sufficiently many layers have been learned,
the deep architecture may be used as a generative model by reproducing the data when

sampling down the model (an "ancestral pass") from the top level feature activations.
[89]
Hinton reports that his models are effective feature extractors over high-dimensional,
structured data.[90]
In 2012, the Google Brain team led by Andrew Ng and Jeff Dean created a neural network
that learned to recognize higher-level concepts, such as cats, only from watching unlabeled
images taken from YouTube videos.[91][92]
Other methods rely on the sheer processing power of modern computers, in
particular, GPUs. In 2010, Dan Ciresan and colleagues[74] in Jrgen Schmidhuber's group at
the Swiss AI Lab IDSIA showed that despite the above-mentioned "vanishing gradient
problem," the superior processing power of GPUs makes plain back-propagation feasible
for deep feedforward neural networks with many layers. The method outperformed all other
machine learning techniques on the old, famous MNIST handwritten digits problem of Yann
LeCun and colleagues at NYU.
At about the same time, in late 2009, deep learning feedforward networks made inroads
into speech recognition, as marked by the NIPS Workshop on Deep Learning for Speech
Recognition. Intensive collaborative work between Microsoft Research and University of
Toronto researchers demonstrated by mid-2010 in Redmond that deep neural networks
interfaced with a hidden Markov model with context-dependent states that define the neural
network output layer can drastically reduce errors in large-vocabulary speech recognition
tasks such as voice search. The same deep neural net model was shown to scale up to
Switchboard tasks about one year later at Microsoft Research Asia. Even earlier, in 2007,
LSTM[51] trained by CTC[53] started to get excellent results in certain applications. [54] This
method is now widely used, for example, in Google's greatly improved speech recognition
for all smartphone users.[55]
As of 2011, the state of the art in deep learning feedforward networks alternates
convolutional layers and max-pooling layers,[93][94] topped by several fully connected or
sparsely connected layer followed by a final classification layer. Training is usually done
without any unsupervised pre-training. Since 2011, GPU-based implementations [93] of this
approach won many pattern recognition contests, including the IJCNN 2011 Traffic Sign
Recognition Competition,[95] the ISBI 2012 Segmentation of neuronal structures in EM
stacks challenge,[96] the ImageNet Competition,[97] and others.
Such supervised deep learning methods also were the first artificial pattern recognizers to
achieve human-competitive performance on certain tasks.[98]
To overcome the barriers of weak AI represented by deep learning, it is necessary to go
beyond deep learning architectures, because biological brains use both shallow and deep
circuits as reported by brain anatomy[99] displaying a wide variety of invariance.
Weng[100] argued that the brain self-wires largely according to signal statistics and, therefore,
a serial cascade cannot catch all major statistical dependencies. ANNs were able to
guarantee shift invariance to deal with small and large natural objects in large cluttered
scenes, only when invariance extended beyond shift, to all ANN-learned concepts, such as
location, type (object class label), scale, lighting. This was realized in Developmental
Networks (DNs)[101] whose embodiments are Where-What Networks, WWN-1 (2008)
[102]
through WWN-7 (2013).[103]

Deep neural network architectures[edit]

This section may be too technical for most readers to

understand. Please help improve this section to make it
understandable to non-experts, without removing the technica
details. The talk page may contain suggestions. (July 2016) (Lear
and when to remove this template message)

There are huge number of variants of deep architectures. Most of them are branched from
some original parent architectures. It is not always possible to compare the performance of
multiple architectures all together, because they are not all evaluated on the same data
sets. Deep learning is a fast-growing field, and new architectures, variants, or algorithms
appear every few weeks.

Brief discussion of deep neural networks[edit]

A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden
layers of units between the input and output layers.[3][5] Similar to shallow ANNs, DNNs can
model complex non-linear relationships. DNN architectures, e.g., for object
detection and parsing, generate compositional models where the object is expressed as a
layered composition of image primitives.[104] The extra layers enable composition of features
from lower layers, giving the potential of modeling complex data with fewer units than a
similarly performing shallow network.[3]
DNNs are typically designed as feedforward networks, but research has very successfully
applied recurrent neural networks, especially LSTM,[51][105] for applications such as language
modeling.[106][107][108][109][110] Convolutional deep neural networks (CNNs) are used in computer
vision where their success is well-documented.[111] CNNs also have been applied to acoustic
modeling for automatic speech recognition (ASR), where they have shown success over
previous models.[63] For simplicity, a look at training DNNs is given here.
Backpropagation[edit]
A DNN can be discriminatively trained with the standard backpropagation algorithm.
According to various sources,[5][8][84][112] basics of continuous backpropagation were derived in
the context of control theory by Henry J. Kelley[79] in 1960 and by Arthur E. Bryson in 1961,
[80]
using principles of dynamic programming. In 1962, Stuart Dreyfus published a simpler
derivation based only on the chain rule.[81] Vapnik cites reference[113] in his book on Support
Vector Machines. Arthur E. Bryson and Yu-Chi Ho described it as a multi-stage dynamic
system optimization method in 1969.[114][115] In 1970, Seppo Linnainmaa finally published the
general method for automatic differentiation (AD) of discrete connected networks of
nested differentiable functions.[28][116] This corresponds to the modern version of
backpropagation which is efficient even when the networks are sparse. [5][8][29][78] In
1973, Stuart Dreyfus used backpropagation to adapt parameters of controllers in proportion
to error gradients.[82] In 1974, Paul Werbos mentioned the possibility of applying this
principle to artificial neural networks,[117] and in 1982, he applied Linnainmaa's AD method to
neural networks in the way that is widely used today.[8][31] In 1986, David E.
Rumelhart, Geoffrey E. Hinton and Ronald J. Williams showed through computer
experiments that this method can generate useful internal representations of incoming data
in hidden layers of neural networks.[83] In 1993, Eric A. Wan was the first[5] to win an
international pattern recognition contest through backpropagation.[118]
The weight updates of backpropagation can be done via stochastic gradient descent using
the following equation:

Here, is the learning rate, is the cost function and a stochastic term. The choice of the
cost function depends on factors such as the learning type (supervised,
unsupervised, reinforcement, etc.) and the activation function. For example, when
performing supervised learning on a multiclass classification problem, common choices
for the activation function and cost function are the softmax function and cross
entropy function, respectively. The softmax function is defined as where represents
the class probability (output of the unit ) and and represent the total input to
units and of the same level respectively. Cross entropy is defined
as where represents the target probability for output unit and is the probability output
for after applying the activation function.[119]
These can be used to output object bounding boxes in the form of a binary mask. They
are also used for multi-scale regression to increase localization precision. DNN-based
regression can learn features that capture geometric information in addition to being a
good classifier. They remove the limitation of designing a model which will capture
parts and their relations explicitly. This helps to learn a wide variety of objects. The
model consists of multiple layers, each of which has a rectified linear unit for non-linear
transformation. Some layers are convolutional, while others are fully connected. Every
convolutional layer has an additional max pooling. The network is trained to minimize
L2 error for predicting the mask ranging over the entire training set containing bounding
boxes represented as masks.
Problems with deep neural networks[edit]
As with ANNs, many issues can arise with DNNs if they are naively trained. Two
common issues are overfitting and computation time.
DNNs are prone to overfitting because of the added layers of abstraction, which allow
them to model rare dependencies in the training data. Regularization methods such as
Ivakhnenko's unit pruning[26] or weight decay (-regularization) or sparsity (regularization) can be applied during training to help combat overfitting. [120] A more
recent regularization method applied to DNNs is dropout regularization. In dropout,
some number of units are randomly omitted from the hidden layers during training. This
helps to break the rare dependencies that can occur in the training data. [121]
The dominant method for training these structures has been error-correction training
(such as backpropagation with gradient descent) due to its ease of implementation and
its tendency to converge to better local optima than other training methods. However,
these methods can be computationally expensive, especially for DNNs. There are
many training parameters to be considered with a DNN, such as the size (number of
layers and number of units per layer), the learning rate and initial weights. Sweeping
through the parameter space for optimal parameters may not be feasible due to the
cost in time and computational resources. Various 'tricks' such as using mini-batching
(computing the gradient on several training examples at once rather than individual
examples)[122] have been shown to speed up computation. The large processing
throughput of GPUs has produced significant speedups in training, due to the matrix
and vector computations required being well suited for GPUs. [5] Radical alternatives to
backprop such as Extreme Learning Machines,[123] "No-prop" networks,[124] training
without backtracking,[125] "weightless" networks,[126] and non-connectionist neural
networks are gaining attention.

First deep learning networks of 1965: GMDH [edit]

According to a historic survey,[5] the first functional Deep Learning networks with many
layers were published by Alexey Grigorevich Ivakhnenko and V. G. Lapa in 1965.[25]
[127]
The learning algorithm was called the Group Method of Data Handling or GMDH.
[128]
GMDH features fully automatic structural and parametric optimization of models. The
activation functions of the network nodes are Kolmogorov-Gabor polynomials that
permit additions and multiplications. Ivakhnenko's 1971 paper[26] describes the learning
of a deep feedforward multilayer perceptron with eight layers, already much deeper
than many later networks. The supervised learning network is grown layer by layer,
where each layer is trained by regression analysis. From time to time useless neurons
are detected using a validation set, and pruned through regularization. The size and
depth of the resulting network depends on the problem. Variants of this method are still
being used today.[129]

Convolutional neural networks[edit]

Main article: Convolutional neural network
CNNs have become the method of choice for processing visual and other twodimensional data.[32][67] A CNN is composed of one or more convolutional layers with fully
connected layers (matching those in typical artificial neural networks) on top. It also
uses tied weights and pooling layers. In particular, max-pooling [38] is often used in
Fukushima's convolutional architecture.[27] This architecture allows CNNs to take
advantage of the 2D structure of input data. In comparison with other deep
architectures, convolutional neural networks have shown superior results in both image
and speech applications. They can also be trained with standard backpropagation.
CNNs are easier to train than other regular, deep, feed-forward neural networks and
have many fewer parameters to estimate, making them a highly attractive architecture
to use.[130] Examples of applications in Computer Vision include DeepDream.[131] See the
main article on Convolutional neural networks for numerous additional references.

Neural history compressor[edit]

The vanishing gradient problem[35] of automatic differentiation or backpropagation in
neural networks was partially overcome in 1992 by an early generative model called
the neural history compressor, implemented as an unsupervised stack of recurrent
neural networks (RNNs).[16] The RNN at the input level learns to predict its next input
from the previous input history. Only unpredictable inputs of some RNN in the hierarchy
become inputs to the next higher level RNN which therefore recomputes its internal
state only rarely. Each higher level RNN thus learns a compressed representation of
the information in the RNN below. This is done such that the input sequence can be
precisely reconstructed from the sequence representation at the highest level. The
system effectively minimises the description length or the negative logarithm of the
probability of the data.[8] If there is a lot of learnable predictability in the incoming data
sequence, then the highest level RNN can use supervised learning to easily classify
even deep sequences with very long time intervals between important events. In 1993,
such a system already solved a "Very Deep Learning" task that requires more than
1000 subsequent layers in an RNN unfolded in time. [33]
It is also possible to distill the entire RNN hierarchy into only two RNNs called the
"conscious" chunker (higher level) and the "subconscious" automatizer (lower level).
[16]
Once the chunker has learned to predict and compress inputs that are still

unpredictable by the automatizer, the automatizer is forced in the next learning phase
to predict or imitate through special additional units the hidden units of the more slowly
changing chunker. This makes it easy for the automatizer to learn appropriate, rarely
changing memories across very long time intervals. This in turn helps the automatizer
to make many of its once unpredictable inputs predictable, such that the chunker can
focus on the remaining still unpredictable events, to compress the data even further.[16]

Recursive neural networks[edit]

Main article: Recursive neural network
A recursive neural network[132] is created by applying the same set of
weights recursively over a differentiable graph-like structure, by traversing the structure
in topological order. Such networks are typically also trained by the reverse mode
of automatic differentiation.[28][78] They were introduced to learn distributed
representations of structure, such as logical terms. A special case of recursive neural
networks is the RNN itself whose structure corresponds to a linear chain. Recursive
neural networks have been applied to natural language processing.[133] The Recursive
Neural Tensor Network uses a tensor-based composition function for all nodes in the
tree.[134]

Long short-term memory[edit]

Main article: Long short-term memory
Numerous researchers now use variants of a deep learning RNN called the Long shortterm memory (LSTM) network published by Hochreiter & Schmidhuber in 1997. [51] It is a
system that unlike traditional RNNs doesn't have the vanishing gradient problem. LSTM
is normally augmented by recurrent gates called forget gates.[105] LSTM RNNs prevent
backpropagated errors from vanishing or exploding.[35] Instead errors can flow
backwards through unlimited numbers of virtual layers in LSTM RNNs unfolded in
space. That is, LSTM can learn "Very Deep Learning" tasks[5] that require memories of
events that happened thousands or even millions of discrete time steps ago. Problemspecific LSTM-like topologies can be evolved.[135] LSTM works even when there are
long delays, and it can handle signals that have a mix of low and high frequency
components.
Today, many applications use stacks of LSTM RNNs[136] and train them by Con