See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/298263303

Introduction to Deep Convolutional Neural Networks

Presentation · March 2016

CITATIONS READS

3 1,859

1 author:

Asifullah Khan
Pakistan Institute of Engineering and Applied Sciences
223 PUBLICATIONS   3,496 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Transfer Learning in Deep Neural networks View project

Machine Learning based Analysis of Biomedical Data View project

All content following this page was uploaded by Asifullah Khan on 08 November 2016.

The user has requested enhancement of the downloaded file.

Introduction to Deep
Convolutional Neural Networks

Dr. Asifullah Khan,

Professor,
DCIS, PIEAS, Islamabad,
Pakistan
Outline
 Artificial Neurons and Perceptron
 Building blocks of CNN
 Convolution
 Pooling
 Motivations for CNN
 Working of a CNN
 Why use Hierarchical Learning
 Visualizing CNN Activations (features)
 Deep CNN architectures: case studies
 Transfer Learning

2
Biological neuron

Image from: http://hplusmagazine.com/2012/10/17/four-statements-about-

the-future
3
Single layer perceptron

Image from: https://battleprogrammer.wordpress.com/2011/03/23/jaringan-syaraf-tiruan-

apa-apa-apa/
4
Multi-layered Neural Net

Image from: Pasini, Antonello. "Artificial neural networks for

small dataset analysis."Journal of thoracic disease 7.5 (2015):
953.
5
Convolution

1 0 1
0 1 0
1 0 1

Filter/kernel

6
Convolution

7
Convolution

8
Convolution

9
Convolution

10
 Convolution
Wconv = Wimg – Wfilt + 1
Hconv = Himg – Hfilt + 1

Convolution and Correlation becomes the same if

the Kernel is symmetric; no effect from flipping.
In Machine Learning community, mostly 11
correlation Kernels are used.
Convolution

The step of
the mask is
known as a
stride

Image from: http://www.slideshare.net/uspace/ujavaorg-deep-learning-with-convolutional-

neural-network 12
Pooling

13
Pooling

14
 One of our Proposed CNN
Architecture;
Inupt Preprocessing Automatic feature extraction Classification

HPF Feature maps Feature maps Pooled maps Feature maps F5:512 Output layer
2084x2084x3 80x80x3 C1:128@37x37 C2:256@16x16 S3:256@8x8 C4:512@3x3

ReLU ReLU ReLU

:

Convolution Convolution Max pooling Convolution

Rotation, flipping and 8x8 7x7 2x2 4x4 Fully connected
histogram equalization Stride:2 Stride:2 Stride:2 Stride:2

featureMapSize = [(inputsize – filtersize + 2xzeropadding)/stride]+1

pooledMapSize = [(inputsize – filtersize)/stride]+1

15
Filters/Kernels/Mask/Feature-extractor

Static/hand-made Filters
16
Motivation for CNN
 Invariant Features: Detection and classification
independent of pose, scale, illumination, occlusion
and clutter
 Dynamic Feature Extraction: how could an artificial
vision system learn appropriate internal
representations automatically, the way humans seem
to by simply looking at the world?
 Reduced Learnable Parameters compared to a
BackPropagator.
 Hierarchical Learning

17
Motivation for CNN; Invariance
Goal
 Detection and classification independent of pose,
scale, illumination, occlusion and clutter

Image from: Dr. Richard E. Turner presentation (2014)

18
 Static Feature Extraction: Standard image
classification approach
Use simple classifier
Input Extract features e.g., logistic
regression, SVMs
Computer$vision$features$

SIFT$ Spin$image$
Face
HoG$ RIFT$

Textons$ GLOH$
Slide$Credit:$Honglak$Lee$

Deep Neural Network and Transfer Learning 19

Static Feature Extraction: many hand crafted
Computer$vision$
Features features$
exist…

SIFT$ Spin$image$

HoG$ RIFT$

Textons$ GLOH$
Slide$Credit:$Honglak$Lee$

… but very painful to design

20
 Dynamic Feature Extraction: Change Image
Classification Approach?
Use simple classifier
Input Extract features dynamically
e.g., SVM, etc.

Computer$vision$features$

SIFT$ Spin$image$
Face
Can we automatically learn
features
HoG$ RIFT$

from data?
Textons$ GLOH$
Slide$Credit:$Honglak$Lee$

Deep Neural Network and Transfer Learning 21

Motivation for CNN; Less Parameters
Cons of multilayer perceptron
 The number of trainable parameters becomes
extremely large. For example, a 24 × 24 input layer
would already have 576 connections per single
neuron in the hidden layer
 Secondly, it offers little or no invariance to shifting,
scaling, and other forms of distortion
 Hand-crafted feature extraction requires a great deal
of time

22
Motivation for CNN; Less Parameters
Motivation for CNN; Hierarchical Learning
 Hubel/Wiesel Architecture
 D. Hubel and T. Wiesel (Nobel Prize 1981)
 Visual cortex consists of a hierarchy of simple,
complex, and hyper-complex cells

Image from: http://cns-alumni.bu.edu/~slehar/webstuff/pcave/hubel.html 24

Motivation for CNN: Hierarchical Learning

Image from: http://www.csrp.riken.jp/application_b_e.html

25
Motivation for CNN; Hierarchical Learning

 In 1995, Yann LeCun and Yoshua Bengio introduced

the concept of CNN

 Neurobiologically motivated by the findings of locally

sensitive and orientation-selective nerve cells in the
visual cortex of the cat

26
 Motivation for CNN; Why use hierarchical
multi-layered models?
 visual scenes are hierarchically organized

Slide credit: Dr. Richard E. Turner presentation (2014)

27
 Motivation for CNN; Why use hierarchical
multi-layered models?
• biological vision is hierarchically organized

Slide credit: Dr. Richard E. Turner presentation (2014)

28
 Motivation for CNN; Why use
hierarchical multi-layered models?
 Shallow architectures are inefficient at
representing deep functions
 Deep net, deep (enriched) features

These units fine-tune the

features learned by those
in the previous layer
Params: 5x5+5=30 Params: 5x3+6+2=23

Deep residual learning for image recognition, Noorul Wahab, (26 Aug. 2016) 29
Motivation for CNN; Hierarchical Representation
Example

Yoshua Bengio (2009)

30
Working of Convolutional Neural
Networks

 Neural network with specialized connectivity

structure
 Stack multiple stages of feature extractors
 Higher stages compute more global, more invariant
features
 Classification layer at the end

31
Working of Convolutional Neural
Network
 Feed-forward feature extraction:
 1. Convolve input with filters
 2. Non-linearity (ReLU)
 3. Spatial pooling
 4. Normalization
Supervised training of convolutional filters by back-
propagating classification error.
A CNN can be both Generative and Discriminative
Learning Mechanism

32
Working of Convolutional Neural
Network
• Feed-forward feature extraction:

33
Working of Convolutional Neural
Network
 CNN is a type of feed-forward artificial neural
network.
 Individual neurons are tiled in such a way that they
respond to overlapping regions in the visual field
 The lower layers obtain low-level features (like pixels,
edges, lines and corners) while the higher layers
obtain high-level features (like shapes)
 The more layers the network has, the higher-level
features it will get

34
Working of Convolutional Neural
Network; Convolutional layer
 Unlike a hand-coded convolution kernel (Sobel,
Prewitt, Roberts), in a CNN, the parameters of each
convolution kernel are trained by the
backpropagation algorithm
 Convolution operators extract different features of
the input
 There are many convolution kernels in each layer,
and each kernel is replicated over the entire image
with the same parameters (weights and bias)

35
Working of Convolutional Neural
Network: Convolutional layer

Slide credit: Rob Fergus (NIPS 2013 tutorial)

36
http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf
Working of Convolutional Neural
Network: Non-linearity

Slide credit: Rob Fergus (NIPS 2013 tutorial)

37
http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf
Working of Convolutional Neural
Network: Pooling Layer
 In order to reduce variance, pooling layers
compute the max or average value of a
particular feature over a region of the image
 This will ensure that the same result will be
obtained, even when image features have
small translations
 This is an important operation for object
classification and detection

38
Working of Convolutional Neural
Network: Pooling Layer

Slide credit: Rob Fergus (NIPS 2013 tutorial)

39
http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf
Working of Convolutional Neural
Network: Normalization

Slide credit: Rob Fergus (NIPS 2013 tutorial)

http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf 40
 Visualizing activations in CNN
Visualizing activations

Zeiler, Matthew (2014)

41
 Visualizing activations in CN cont..
Visualizing activations

Zeiler, Matthew (2014)

42
Deep CNN recent examples:
GoogLeNet (ILSVRC 2014 winner)

Inception module

Szegedy, et al (2015)
43
Deep CNN recent examples:
ResNet (ILSVRC 2015 winner)

Residual learning: a building block

152 layers; 3.57 top 5% error
ImageNet 2012 dataset;
1000 classes, 1.28 million
training Images,
50K validation images, 100 K
test images;
millions of parameters;
44
Kaiming He, et al. (2015)
CNN; Competition in Depth

Slide from Kaiming He et al., 45

Transfer Learning; Image Classification
example

Features Task One Model One

Deep Neural Network and Transfer Learning 46

 Transfer Learning; Image Classification
example
Reuse
Features Task One

Cars

Features Task Two

Model Two

Motorcycles
Task Two
Deep Neural Network and Transfer Learning 47
Convolutional Neural Networks
To ponder on:
 Computational cost vs. performance/results
 What should be the filter sizes at different layers?
 How much pooling?
 How many neurons to keep in different layers?
 How many layers to employ?
 How to increase generalization of a CNN?
 Use a good cross-validation startegy
 Use pruning (Dropout, Swapout, etc.)
 Use unsupervised pre-training

48
References
• http://ufldl.stanford.edu/tutorial/supervised/FeatureExtractio
nUsingConvolution/
• Tutorial on Deep Learning and Applications Honglak Lee
(University of Michigan)
• http://docs.gimp.org/en/plug-in-convmatrix.html
• https://developer.apple.com/library/ios/documentation/Perfo
rmance/Conceptual/vImage/ConvolutionOperations/Convolut
ionOperations.html
• http://www.slideshare.net/AhmedMahany/convolution-
neural-networks
• http://www.robots.ox.ac.uk/~vgg/research/text/index.html

49
References cont…
• http://learning.eng.cam.ac.uk/pub/Public/Turner/Teaching/ml-
lecture-3-slides.pdf
• LeCun, Yann, et al. "Gradient-based learning applied to document
recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.
• Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding
convolutional networks." Computer vision–ECCV 2014. Springer
International Publishing, 2014. 818-833.
• Bengio, Yoshua. "Learning deep architectures for AI." Foundations
and trends® in Machine Learning 2.1 (2009): 1-127.
• Szegedy, Christian, et al. "Going deeper with
convolutions." Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. 2015.
• He, Kaiming, et al. "Deep Residual Learning for Image
Recognition." arXiv preprint arXiv:1512.03385 (2015).

50
View publication stats