Autoencoders

Autoencoders
Motivation – Representation Learning

• The success of machine learning heavily depends on data
representation (feature).
• Representation learning helps to improve performance of machine
learning models by extracting features from high dimensional data,
such as image or audio.
• Sometimes we would like to have a representation with a lower
dimension than the input dimension, to avoid complicated
calculations.
• But as we reduce dimensionality, there may be some loss of
information during compression.
Motivation – Representation Learning
• There are two broad parallel lines of representation learning approaches:
• Probabilistic graphical model – The hidden units are considered latent
random variables, for example: RBM.
• Neural networks – The hidden units in the model are considered as
computational graph, for example: Autoencoders.
• Manifold learning - Another approach for representation learning,

based on the associated geometry and the manifold assumption.
Autoencoders – An Introduction
• An Autoencoder is a type of Neural Network used to learn efficient data
encodings in an unsupervised manner.
• The goal of unsupervised feature learning(representation learning) is often
to discover low-dimensional features that captures some structure
underlying the high-dimensional input data.
• The aim of an Autoencoder is to learn a representation (encoding) for a set
of data, typically for dimensionality reduction. Along with the reduction
side, a reconstructing side is learnt, where the autoencoder tries to
generate from the reduced encoding a representation as close as possible
to its original input.
• Autoencoders are effectively used for solving many applied problems,
from face recognition to acquiring the semantic meaning of words.
• An autoencoder is a neural network that learns to
copy its input to its output. It has an internal
(hidden) layer that describes a code used to
represent the input, and it is constituted by two
main parts: an encoder that maps the input into
the code, and a decoder that maps the code to a
reconstruction of the original input.
• The Input and the output are approximately the
same.
• The middle hidden layer is the “Bottleneck Layer”
where the encoded representation is obtained.
• From the input layer to the bottleneck layer is the
“Encoder”
• From the bottleneck layer to the output layer is the
“Decoder”
Auto-encoder Basic Architecture
Target Target
= input
Y =X = input
Code Z Code
Input X Input
“Bottleneck” code “Overcomplete” code

i.e., low-dimensional, i.e., high-dimensional,
typically dense, always sparse,
distributed distributed
representation representation
Autoencoders
• Formally, we learn two functions:
• f(x) is the encoder
• g(h) is the decoder
• With respect to the reconstruction loss:
1
෍ 𝑥𝑘 − 𝑥ො𝑘 2
2
𝑘
• 𝑥 is the input and 𝑥ො is the output
(reconstructed input)
• h is the low dimensional
representation obtained at the
bottleneck layer
Undercomplete Autoencoder
• An Autoencoder is undercomplete if the dimension of the hidden layer is smaller than the
dimension of the input layer.
• The hidden layer “compresses” the input.
Hinton and Salakhutdinov,

“Reducing the Dimensionality of Data with Neural Networks”, 2006
Overcomplete Autoencoder
• An Autoencoder is overcomplete if the dimension of the hidden layer is larger than (or
equal to) the dimension of the input layer.
• In this case, we might have a duplication of the input in the features with perfect
reconstruction (identity function), without learning meaningful features.
• We use different forms of regularization to avoid this.
Applications of Autoencoders
• Dimensionality Reduction, Information Retrieval, Data Compression
• Noise Removal, Anomaly Detection, Face Recognition, Drug Discovery
• Training can use typical backpropagation methods
Trained
autoencoder
The Details
• In summary, basic autoencoder training consists in finding a value of
parameter vector minimizing reconstruction error:
• This minimization is usually carried out by stochastic gradient descent
12
Noise Removal from MNIST Dataset using Convolutional
Autoencoder
(CODE IN THE FOLLOWING SLIDE)
• By training an autoencoder, we are really training both the encoder

and the decoder at the same time.
• We can build an encoder and use it to compress MNIST digit images.
• Let’s also build a decoder so that we can decompress the compressed
image to the original image size.
• The UpSampling2D will repeats the rows and columns twice.
Effectively reversing the effect of the MaxPooling2D.
• In short, the UpSampling2D is doubling the height and width.
• We reshape the image data to the format the convolutional
autoencoder expects for training.
• We add random noises to the MINST image data and use them as
input for training.
• We train a new autoencoder with the noisy data as input and the
original data as expected output.
• During the training, the autoencoder learns to extract important
features from input images and ignores the image noises because the
labels have no noises.
Thank you
Prepared by: Phenil Buch

[email protected]

Autoencoders - Presentation

Uploaded by

Copyright:

Available Formats

Autoencoders - Presentation

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Autoencoders - Presentation

Uploaded by

Copyright:

Available Formats

What are the two broad parallel lines of representation learning approaches mentioned?

What is the goal of unsupervised feature learning?

Motivation – Representation Learning

• Manifold learning - Another approach for representation learning,

“Bottleneck” code “Overcomplete” code

Hinton and Salakhutdinov,

• By training an autoencoder, we are really training both the encoder

Prepared by: Phenil Buch

You might also like