Machine Learning

and
Deep Learning
By:
Dr. Amit Ganatra
What is Learning?
• “Learning denotes changes in a system that ... enable a system to do
the same task … more efficiently the next time.” - Herbert Simon
• “Learning is constructing or modifying representations of what is
being experienced.” - Ryszard Michalski
• “Learning is making useful changes in our minds.” - Marvin Minsky

“Machine learning refers to a system capable of the autonomous

acquisition and integration of knowledge.”
Learning…
• Most perception (Input Processing) in the brain may be due to one
learning algorithm.
• Build learning algorithms that mimic the brain
• Most human intelligence is due to one learning algorithm
• Learning from tagged data (Supervised Learning)
• Bigger Data – More training is better
• Untagged data (Unsupervised Learning)
• Deep Learning
Dimensions of Learning Systems
• Type of feedback
• Supervised (labeled examples)
• Unsupervised (unlabeled examples)
• Semi supervised (labeled examples and unlabeled examples)
• Reinforcement (reward)
• Representation
• Attribute-based (feature vector)
• Relational (first-order logic)
• Use of knowledge
• Empirical (knowledge-free)
• Analytical (knowledge-guided)
Artificial Intelligence
What is Artificial Intelligence ?

THOUGHT Systems that think Systems that think

like humans rationally

Systems that act Systems that act

BEHAVIOUR like humans rationally

HUMAN RATIONAL
Artificial
• Produced by human art or effort, rather than originating naturally.
• “Intelligence is the ability to acquire knowledge and use it" -
Pigford and Baur

• So AI was defined as:

• AI is the study of ideas that enable computers to be intelligent.
• AI is the part of computer science concerned with design of computer
systems that exhibit human intelligence (From the Concise Oxford
Dictionary)
The main topics in AI
Artificial intelligence can be considered under a number of headings:
• Search (includes Game Playing).
• Representing Knowledge and Reasoning with it.
• Planning.
• Machine Learning.
• Natural language processing.
• Expert Systems.
• Interacting with the Environment
(e.g. Vision, Speech recognition, Robotics)
Some Advantages of Artificial Intelligence

• More powerful and more useful computers

• New and improved interfaces
• Solving new problems
• Better handling of information
• Relieves information overload
• Conversion of information into knowledge
The Disadvantages

• Increased costs
• Difficulty with software development - slow and expensive
• Few experienced programmers
• Few practical products have reached the market as yet.
Areas of AI

Knowledge
Search Logic Representation

Machine
Planning
Learning

Expert
NLP Vision Robotics Systems
Machine Learning
Machine Learning – A definition

“A computer program is said to learn from experience E with respect

to some class of tasks T and performance measure P if its
performance at tasks in T, as measured by P, improves with
experience E ” - Tom Mitchell

ML is the study of computer algorithms that allow computer programs

to automatically improve through experience..
Steps for Machine Learning:
• Collecting data

• Exploring and preparing the data

• Selecting the Machine Leaning approach

• Training the model on the data

• Evaluating model performance

• Improving model performance

After these steps have been completed, if the model appears to
be performing satisfactorily, it can be deployed for its intended
task
Why Machine Learning?
• No human experts
• Industrial/Manufacturing control
• Mass spectrometer analysis, drug design, astronomic discovery
• Black-box human expertise
• Face/Handwriting/Speech Recognition
• Driving a car, Flying a plane
• Rapidly changing phenomena
• Credit scoring, Financial Modeling
• Diagnosis, Fraud Detection
• Need for customization/personalization
• Personalized news reader
• Movie/Book Recommendation
Learning Methods
• Supervised learning ( )
• Prediction
• Classification (discrete labels), Regression (real values)
• Unsupervised learning ( )
• Clustering
• Probability distribution estimation
• Finding association (in features)
• Dimension Reduction
• Semi-supervised learning
• Reinforcement learning
• Decision making (robot, chess machine)
Supervised learning Unsupervised learning

Semi-supervised learning

Image Source: Yi-Fan Chang, An Overview of Machine Learning

Supervised learning

Image Source: Yi-Fan Chang, An Overview of Machine Learning

Classification and Regression
• ; Binary Classification
• ; Regression

Image Source:
https://utkuufuk.github.io/2018/06/03/one-vs-all-
classification/
https://www.edvancer.in/foundations-of-data-science-
made-simple-part-2/
Linear and Non Linear Classification
• Linear Classification • Non-Linear Classification
Find a straight line to separate (classify) There is no way that a straight line can be drawn to
green from blue discriminate the two classes

Examples of linear classifier are: Examples of non-linear classifiers are:

Linear Discriminant Classifier, Naive Bayes, Quadratic Discriminant Classifier, Multi-
Logistic Regression, Perceptron, layer Perceptron (most deep networks),
SVM (with linear kernel) Decision Trees, Random Forest, K-Nearest
Neighbour
The Classification framework
• Apply a prediction function to a feature representation of
the image to get the desired output:

Slide credit: L. Lazebnik

The Classification framework
x
output prediction Image
function feature

• Training: given a training set of labeled examples {(x1,y1), …,

(xN,yN)}, estimate the prediction function f by minimizing the
prediction error on the training set
• Testing: apply f to a never before seen test example x and
output the predicted value y = f(x)

Slide credit: L. Lazebnik

Supervised learning categories and techniques
• Linear classifier (Numerical Functions)
• Parametric (Probabilistic Functions)
• Naïve Bayes, Gaussian discriminant analysis (GDA), Hidden Markov models (HMM),
Probabilistic graphical models
• Non-parametric (Instance-Based Functions)
• K-nearest neighbors, Kernel regression, Kernel density estimation, Local regression
• Non-metric (Symbolic Functions)
• Classification and regression tree (CART), decision tree
• Aggregation
• Bagging (Bootstrap + Aggregation), Adaboost, Random forest
Applications of Supervised learning :
• Pattern recognition
• Face recognition: Pose, lighting, occlusion (glasses, beard), make-
up, hair style
• Character recognition: Different handwriting styles.
• Speech recognition: Temporal dependency.
• Use of a dictionary or the syntax of the language.
• Sensor fusion: Combine multiple modalities; eg, visual (lip image) and
acoustic for speech
• Medical diagnosis: From symptoms to illnesses
• Web Advertising: Predict if a user clicks on an ad on the Internet.
Unsupervised learning

Image Source: Yi-Fan Chang, An Overview of Machine Learning

Unsupervised learning categories and techniques

• Clustering
• K-means clustering
• Spectral clustering
• Density Estimation
• Gaussian mixture model (GMM)
• Graphical models
• Dimensionality reduction
• Principal component analysis (PCA)
• Factor analysis
Applications of unsupervised learning:
• Clustering: Grouping similar instances
• Other applications: Summarization, Association Analysis
• Example applications:
• Customer segmentation in CRM
• Image compression: Color quantization
• Bioinformatics: Learning motifs
Reinforcement Learning
• A computer program interacts with a dynamic environment in which it must
perform a certain goal. The program is provided feedback in terms of rewards
and punishments as it navigates its problem space.
• Policies: what actions should an agent take in a particular situation
• Utility estimation: how good is a state (used by policy)
• No supervised output but delayed reward
• Credit assignment problem (what was responsible for the
outcome)
• Applications:
• Game playing
• Robot in a maze
• Multiple agents, partial observability, ...
Image Source: https://towardsdatascience.com/types-of-machine-learning-algorithms-you-should-know-953a08248861
Examples of real-life problems in the context of supervised and unsupervised learning tasks: Spam filtering
as a classification task and House price estimation as a regression task are part of supervised learning;
Clustering is part of unsupervised learning in which customers are grouped into three different categories
based on their purchasing behavior.

Image Source: DEEP GENERATIVE MODELS FOR SYNTHETIC RETINAL IMAGE GENERATION, Lappeenranta University of Technology, July 2017
Image Source: https://www.datarobot.com/wiki/machine-learning-life-cycle/
Machine Learning Applications
• Recognizing Patterns:
• Facial identities or facial expressions
• Handwritten or spoken words
• Medical images
• Generating Patterns:
• Generating images or motion sequences
• Recognizing Anomalies:
• Unusual sequences of credit card transactions
• Unusual patterns of sensor readings in a nuclear power plant or unusual sound
in your car engine.
• Prediction:
• Future stock prices or currency exchange rates
Some web-based examples of machine learning
• The web contains a lot of data. Tasks with very big datasets often use machine
learning
• especially if the data is noisy or non-stationary.
• Spam filtering, fraud detection:
• The enemy adapts so we must adapt too.
• Recommendation systems:
• Lots of noisy data. Million dollar prize!
• Information retrieval:
• Find documents or images with similar content.
• Data Visualization:
• Display a huge database in a revealing way.
Python Libraries for Machine Learning
Mathematics for Data Manipulation
Machine Learning Visualization
Data Science & Analysis
Scikit-Learn - (formerly scikits.learn)
It features various classification, regression and clustering algorithms
including support vector machines, random forests, gradient
boosting, k-means and DBSCAN, and is designed to interoperate with
the Python numerical and scientific libraries NumPy and SciPy.
Scikit-Learn
TensorFlow
TensorFlow™ is an open source software library for high
performance numerical computation. Its flexible architecture allows
easy deployment of computation across a variety of platforms
(CPUs, GPUs, TPUs), and from desktops to clusters of servers to
mobile and edge devices. Originally developed by researchers and
engineers from the Google Brain team within Google’s AI
organization, it comes with strong support for machine learning and
deep learning and the flexible numerical computation core is used
across many other scientific domains.
Keras
• Keras is a high-level neural networks API, written in Python and
capable of running on top of TensorFlow, CNTK, or Theano. It was
developed with a focus on enabling fast experimentation.
• Allows for easy and fast prototyping (through user friendliness,
modularity, and extensibility).
• Supports both convolutional networks and recurrent networks, as
well as combinations of the two.
• Runs seamlessly on CPU and GPU.
OpenCV
• OpenCV is the leading open source library for computer vision,
image processing and machine learning, and now features GPU
acceleration for real-time operation.
• OpenCV is released under a BSD license and hence it’s free for both
academic and commercial use. It has C++, C, Python and Java
interfaces and supports Windows, Linux, Mac OS, iOS and Android.
OpenCV was designed for computational efficiency and with a
strong focus on real-time applications. Written in optimized
C/C++, the library can take advantage of multi-core processing.
Research issues in Machine Learning
• Handling large amount of data
• High dimensionality
• Training Time is more (Hours or Days)
• Labeling of data or labeling of Images & Videos
• Processing Performance for Training and Testing (Validating)
• Accessing Reference Model – Transfer Learning
• Optimizing Hyper parameters
• Rapid and Optimized deployment
• Curse of Modularity
• Class Imbalance
• Feature Engineering
• Variance and Bias
• Feature Extraction
• Applications specific training – Less Generalization
Growth of Machine Learning – Current State

• Machine learning is preferred approach to

• Speech recognition, Natural language processing
• Computer vision
• Medical outcomes analysis
• Robot control
• Computational biology
• This trend is accelerating – Deep Learning…
• Improved machine learning algorithms
• Improved data capture, networking, faster computers
• Software too complex to write by hand
• New sensors / IO devices
• Demand for self-customization to user, environment
Deep Learning
Biological Inspiration

Image Source: http://www.andreykurenkov.com/writing/ai/a-brief-history-of-neural-nets-and-deep-learning/

Practical issues with Neural Networks
• Data augmentation
• Data normalization
• Architecture optimization
• Loss function
• Weight initialization
• Learning rate and its decay schedule
• Overfitting: regularization, early stopping
• Momentum
Deep Learning – A Definition
Deep Learning is a subfield of Machine Learning which uses
computational models, with hierarchical architectures composed
by multiple processing layers, to learn representations of complex
data such as images, sound and text.

Image Source: https://www.linkedin.com/pulse/intuition-deep-neural-networks-abhishek-parbhakar

Deep learning
“Deep learning is a particular kind of machine learning that achieves
great power and flexibility by learning to represent the world as
nested hierarchy of concepts, with each concept defined in relation
to simpler concepts, and more abstract representations computed
in terms of less abstract ones.”
 A bit of History

1965 1986
Alexey Ivakhnenko Geoffrey Hinton

First works on
backpropagation
Deep neural networks
algorithm
 A bit of History

Computer vision and

speech communities do
not use neural nets
anymore

Hard to train 2006 Efficient training

No computational resources GPU /multi-core processors
small datasets large datasets
 A bit of History

2011 2012
Microsoft Geoffrey Hinton
(University of Toronto)
breakthrough
in speech recognition breakthrough
in computer vision
Researchers of Deep Learning

Andrew Ng Fei Fei Li Yann LeCun Yoshua Bengio Ian Goodfellow

Andrej Karpathy Jeff Dean Nitish Srivastava Dr Ketan Kotecha

Difference between ML and DL

Image Source: https://www.xenonstack.com/blog/log-analytics-with-deep-learning-and-machine-learning

 Continue…
Data dependencies
ML DL
• Traditional machine learning • When the data is small, deep
algorithms with their learning algorithms don’t
handcrafted rules succeed in perform that well. This is
this scenario. because deep learning
algorithms need a large
amount of data to understand
it perfectly.
 Continue…
Hardware dependencies
ML DL
• Traditional machine learning • Deep learning algorithms heavily
algorithms can work on low-end depend on high-end machines
machines. because deep learning
algorithms inherently do a large
amount of matrix multiplication
operations. These operations
can be efficiently optimized
using a GPU because GPU is built
for this purpose.
 Continue…
Feature engineering
ML DL
• Most of the applied features need to • Deep learning algorithms try to
be identified by an expert and then learn high-level features from data.
hand-coded as per the domain and • Deep learning reduces the task of
data type. developing new feature extractor for
• For example, features can be pixel every problem. Like, Convolutional
values, shape, textures, position and NN will try to learn low-level
orientation. The performance of features such as edges and lines in
most of the Machine Learning early layers then parts of faces of
algorithms depends on how people and then high-level
accurately the features are identified representation of a face.
and extracted.
 Continue…
Problem Solving approach
ML DL
• Traditional machine learning is • Deep learning in contrast
generally recommended to advocates to solve the problem
break the problem down into end-to-end.
different parts, solve them
individually and combine them • Image would be passed to
to get the result. network as an input, and it
• For object detection problem, would give out the location
you would divide the problem along with the name of object.
into two steps, object detection (i.e. YOLO net)
and object recognition.
 Continue…
Execution time
ML DL
• Machine learning • A deep learning algorithm
comparatively takes much less takes a long time to train. This
time to train, ranging from a is because there are so many
few seconds to a few hours. parameters in a deep learning
algorithm that training them
takes longer than usual.
• State of the art deep learning
algorithm ResNet takes about
two weeks to train completely
from scratch.
 Continue…
Interpretability
ML DL
• Easy to interpret the reasoning • No one interprets what
behind it. happens in a node of DL model.

Let’s take an example. Suppose we use deep learning to give automated scoring to essays. The
performance it gives in scoring is quite excellent and is near human performance. But there’s is an issue.
It does not reveal why it has given that score. Indeed mathematically you can find out which nodes of a
deep neural network were activated, but we don’t know what these neurons were supposed to model
and what these layers of neurons were doing collectively. So we fail to interpret the results.

On the other hand, machine learning algorithms like decision trees give us crisp rules as to why it chose
what it chose, so it is particularly easy to interpret the reasoning behind it. Therefore, algorithms like
decision trees and linear/logistic regression are primarily used in industry for interpretability.
Where is Machine Learning and Deep Learning being
applied right now?
• Computer Vision: for applications like vehicle number plate identification
and facial recognition.
• Information Retrieval: for applications like search engines, both text search,
and image search.
• Marketing: for applications like automated email marketing, target
identification
• Medical Diagnosis: for applications like cancer identification, anomaly
detection
• Natural Language Processing: for applications like sentiment analysis,
photo tagging
• Online Advertising
 Why Deep Learning taking off?

1. Lots of Data – Structured and

Unstructured

2. Computation Power – GPU

3. Efficient Algorithms
Types of Deep Networks
• There are different deep networks like
• Convolutional Neural Network (CNN),
• Recurrent Neural Network (RNN),
• Deep Belief Network (DBN),
• Recursive Neural Tensor Network (RNTN),
• Long and Short Term Memory (LSTM)
• Restricted Boltzman Machine (RBM).
• Auto Encoder
• GAN
Types of Deep Network
Applications Deep Network

Text Processing RNN, RNTN

Image Recognition CNN, DBN

Object Recognition CNN, RNTN

Speech Recognition RNN

Time Series Analysis RNN

Unlabeled data-pattern recognition RBM

Comparison of CNN and RNN
CNN RNN
CNN takes a fixed size inputs and generates fixed-size outputs. RNN can handle arbitrary input/output lengths.

CNN is a type of artificial neural network - are variations of RNN unlike neural networks - can use their
multilayer perceptron which are designed to use minimal internal memory to process arbitrary sequences
amounts of preprocessing. of inputs.

CNNs use connected pattern between its neurons and is

inspired by the organization of the animal visual cortex, whose Recurrent neural networks use time-series or
individual neurons are arranged in such a way that they respond sequenced information.
to overlapping regions tiling the visual field.

It is used when there is analysis of image data set , for example, RNNs are ideal for text and speech analysis. It is
classify or detect objects. CNNs are ideal for images and video used in sentiment analysis for text containing
processing. product reviews.
 Comparison of Deep Learning Models
Auto-
Model CNN RNN LSTM RBM DBN
encoder
Type Discriminative Discriminative Discriminative Generative Generative Generative
Unsupervised Unsupervised
Dimensionality
Purpose Classification Prediction Prediction Feature Feature
Reduction
Learning Learning
Creating a Creating a
Learning
Processing two- Processing Processing long probabilistic compact
Suitable for distribution of
dimensional data sequence data sequence data recommendati representation
data
on of data of data

Generate Trained layer

Language
Language Models samples from used as
Example Images/ Videos Models and PCA-like tasks
and speech learned hidden feature
speech
representation detector
 Convolution
CNN’s make use of filters (also known
as kernels), to detect what features, such
as edges, are present throughout an
image. A filter is just a matrix of values,
called weights, that are trained to detect
specific features. The filter moves over
each part of the image to check if the
feature it is meant to detect is present.
To provide a value representing how
confident it is that a specific feature is
present, the filter carries out
a convolution operation, which is an
element-wise product and sum between
two matrices.
CNN
• In Computer vision problem, we don’t want to be stuck using only
tiny little images, but we want to work with large images so
convolution operator is very important and it is basic block of
convolution Neural Network. Different layers of CNN use different
activation functions. Following is the summary of activation functions.
 Activation Function
Name of
Mathematical
the Advantage Disadvantage
Representation
function
Rectified Bounded below by 0 Tends to blowup the activations
Does not saturate (in +region)
Linear
Very computationally efficient
Unit Converges much faster than
(ReLU) sigmoid/tanh in practice
Squashes the neuron’s pre-activations Tends to vanish the gradient
between [0,1], descent as activation increases
Historically popular
Sigmoid Sigmoid outputs are not zero-
centered

𝑒 is a bit compute extensive

Squashes the neuron’s pre-activations Tends to vanish the gradient
between descent as activation increases
[-1,1],
Tanh
𝑒 is a bit compute extensive
Zero centered
Pooling layer
• CNNs also incorporate pooling layers, where an operation is
applied to all elements within the filtering extent. This
corresponds, effectively, to down sampling.
• The pooling filter has width and height (wp, hp) and is applied
with a given stride. It is most common to use the max() operation
as the pooling operation.
Convolutional layers have sparse interactions

• Sparse interactions reduce computational memory.

• Sparse interactions reduce computation time.

• Convolutional layers share parameters

A typical workflow of images/video processing
Deep Learning Library

Deep Learning Cloud Services

In case of CNN, following hyper parameters to be optimized
• Convolution
• Filter Size
• Number of filters
• Padding
• stride
• Pooling
• Window size
• Stride
• Fully Connected
• Number of neurons
 Training Time: CPU vs GPU (Case Study)
Dataset Deep Learning Framework Deep Learning Framework
CPU GPU
Training Time (Second) Training Time
MNIST DL Caffe Tensor Torch Keras DeepLearni DLL Caffe TensorFl Torc Keras DeepLearni
L Flow ng4J ow h ng4J
35 1,781 1,303 1,573 1,282 2,953 355 564 160 988 185 1,193
5

CIFAR 1,0 2,677 1,790 2,097 1,741 1,0 572 272 1,02 313
61 61 7

ImageNet 3,0 6,875 5,912 9,682 5,034 30,954 3,0 1,475 945 2,43 903 1.35* 105
25 25 2
Findings
• As long as the data is small enough to fit on one machine, one
machine with GPU capability outperforms other architectures .
• Network communication varies directly with model size.
• The local implementations have no data-parallelism and no
network-overhead.
• GPUs speeding up mathematical operations.
• GPU has its own memory, so GPU memory can become a
bottleneck if matrices are too large to fit the somewhat limited
typical GPU memory.
 Comparison of Framework
Framework Core Bindings Single Multi Open Distributed Comments
Language GPU GPU Source
Caffe C++ Python,    See Mostly for Image Classification, Models/
Matlab com.yahoo.ml.C layers expressed in proto format
affeOnSpark

Theano/Pylearn 2 Python  In  No Transparent use of GPU, General purpose,

progress Computation as DAG
Torch7 Lua   See Twitter’s Baidu’s Deep Speech open sourced. Very
torch-distlearn efficient
TensorFlow C++ Python  Upto 4 Not Open Slower than theano/ Torch, Computation
GPUs Sourced as DAG
DL4J Java   Distributed Supports GPUs via CUDA, Support for
Hadoop/ Spark
SystemML Java Python, In Not yet Distributed
Scala Process
Minerva/ CXXNet C++ Python    Distributed https://github.com/dmlc.Minerva~Thean
(Smola) o and CXXNet~ Caffe
Transfer Learning
• Instead of training the network from scratch, transfer learning
utilizes a trained model on a different dataset, and adapts it to the
problem that we're trying to solve.

2 strategies for transfer learning

• Utilize the trained model as a fixed feature extractor

• Fine-tune the trained model

Why deep representations?
• Deep NN makes relations with data from simpler to complex.
• In each layer it tries to make a relations between the previous layer.
• Face recognition application:
• Image ==> Edges ==> Face parts ==> Faces ==> desired face
• Audio recognition application:
• Audio ==> Low level sound features ==> Phonemes ==> Words ==>
Sentences
• Neural Researchers thinks that deep neural networks thinks like
brains (Simple ==> Complex)
Image Source: https://www.kiv.zcu.cz/studies/predmety/uir/NS/Neocognitron/en/hierarch-det.html
Deep Learning Architectures
1. CNN (Convolutional Neural Network)

Image Source: https://www.clarifai.com/technology

 Binary Classification

1: CAT
0: NOT CAT

model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False)
Algorithm:
• Define the model structure (such as number of inputs and
outputs)
• Initialize the model's parameters.
• Loop.
• Calculate current loss (forward propagation)
• Calculate current gradient (backward propagation)
• Update parameters (gradient descent)
• Repeat Until it Converges
y = mx + c

y - y1 = m(x - x1)

y - 4 = (-1.6)(x - (-2))
y - 4 = -1.6x - 3.2
y = -1.6x - 3.2 + 4

y = -1.6x + 0.8

y = -1.6(3) + 0.8
y= -4
Activation Function
1. Forward Propagation Equation

Loss

Cost
2. Backward Propagation
Gradient Descent
3. Update parameters (gradient descent)

The picture can't be display ed.

 𝝏𝑱
𝒘=𝒘 −𝜶
𝝏𝒘

𝝏𝑱
𝒃=𝒃 −𝜶
𝝏𝒃

model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False)
Input
2 layer NN (1 HL + 1 OL)
Parameters and Hyperparameters
Parameter - Weight and Bias
Hyperparameter (parameters that control the algorithm):
1) No of hidden layers - Trial and error
2) No of neurons in each hidden layer - Trial and error
3) Learning rate. (The most important parameter) – 0.1, 0.01, 0.001, 0.0001, .00001
4) Activation functions – Sigmoid, Relu, Leaky Relu, Tanh
5) Number of iteration – Trial and error
6) Epoch – Trial and error
7) Batch size – 4, 8, 16, 32, 64, 128… (Power of 2)
8) Regularization – Dropout, L1, L2
9) Regularization Rate and Dropout lambda
10)Normalizing inputs – Min-max, mean, Z-Score
11)Weights and Bias Initialization – Zero, Random
12)optimization algorithm
13)Learning rate decay
14)Momentum – 0.9
Iterations, Epoch and Batch Size
• batch size = the number of training examples in one
forward/backward pass. The higher the batch size, the more
memory space you'll need.
• number of iterations = number of passes, each pass using [batch
size] number of examples. To clear, one pass = one forward pass +
one backward pass (we do not count the forward pass and
backward pass as two different passes).
• one epoch = one forward pass and one backward pass of all the
training examples.
• Example: if you have 5000 training examples, and your batch size
is 32, then it will take 157 iterations to complete 1 epoch.
• 5000/32 = 156.25 =157 iterations
Empirical Process
Application of CNN
2. RNN (recurrent neural network)
 Application of RNN
1. Automatically Adding Sounds To Silent Movies
2. Automatic Text Generation
3. RBM (Restricted Boltzmann Machine)

Applications of RBM:
 Feature extraction
 Labelling of dataset
 Construct Deep Belief Networks
4. Deep Belief Network
1. Unsupervised pre-learning provides a
good initialization of the network

2. Supervised fine-tuning

Application:
• Image/ Face Recognition
• Video Sequence recognition
• Motion-capture data

NASA is using DBN to classify high-

resolution, highly diverse satellite
imagery.
General Applications of Deep Learning:
Challenges in Computer Vision

Image Source: http://cs231n.github.io/classification/

A typical workflow of images/video processing
Deep Learning Library

Deep Learning Cloud Services

Comparison of Deep Learning Library
Software Open CUDA
Software Creator Platform Written in Interface
license source support
Google Linux, Mac Python, (C/C++public API
TensorFlow Apache 2.0 Yes C++,Python Yes
Brainteam OS, Windows only for executing graphs)
François Linux, Mac
Keras MIT license Yes Python Python Yes
Chollet OS, Windows
Linux, Mac
Adam OS, Windows,
Deeplearning4j Apache 2.0 Yes C, C++ Java, Scala Yes
Gibson Android (Cross-
platform)
BSD Linux, Mac
Caffe Yangqing Jia Yes C++ Python, Matlab Yes
License OS, Windows
Université BSD
Theano Yes Cross-platform Python Python Yes
de Montréal License

Ronan
Linux, Mac OS,
Collobert,Ka BSD Lua, C, utility library for
Torch Yes Windows, C, Lua Yes
vukcuoglu, License C++/OpenCL
Android, iOS
Clement
Python, C++,
Microsoft
Microsoft Command line,
Cognitive MIT license Yes Windows, Linux C++ Yes
Research BrainScript(.NET on
Toolkit - CNTK
roadmap)
Schematic Diagram

Train and Training/ Testing

Test
Dataset
using DL Accuracy and Weights
model file

Input image/
video Applications output Class/object
and (Python scripts) score
weight file
(Trained model)
Evaluation Parameters
• Accuracy – Confusion matrix
• Precision
• Recall
• F1 Score (try to maximize - to create a balanced classification
model)
• mAP – IOU (For Object Detection)
• Loss/Cost – Minimize the difference between actual and predicated
• Entropy
 Dear, Do you remember all wedding
anniversary surprises from me?

They are celebrating 6th wedding anniversary.

If you can recall all 5 gifts correctly, then, your recall

ratio is 1.0 (100%). If you can recall 4 events
correctly, your recall ratio is 0.8 (80%).

After some days, again she asked same question……

Most
Dangerous
Question!!! For example, you answers 15 times, 10 times are correct and 5
times are wrong. This means you can recall all events but it’s
not so precise.

That means: (15 answers: 10 correct answers, 5 wrong

answers), you get 100% recall but your precision is
only 66.67% (10 / 15).
Facts of Deep Learning
• Majority research work of Deep Learning published in only
reputed conference like CVPR, ECCV, ICCV not in journals.
• Cold war between image processing and deep learning researcher,
because of the nature of deep leaning – It is BlackBox. No one is
knowing “How it works from inside?”
• Very hard to publish paper in reputed journal. It gets published
only like strong survey paper or hybrid branch paper like
bioinformatics.
• It follows very old methods. It becomes popular due to huge
amount of data and processing power.
• Deep Learning is empirical process. It takes time to get expertise.
Online DEMO

• https://playground.tensorflow.org

• http://scs.ryerson.ca/~aharley/vis/conv/
NVIDIA Titan Xp
(Grant by Nvidia Graphics Pvt. Ltd.)
 Configuration
• GPU Architecture – Pascal

• NVIDIA Cuda Cores – 3840

• Memory Size – 12 GB GDDR5X

• Memory Speed - 11.4 Gbps

• Nvidia Cuda Compute Capblity – 6.1

• VR Ready
Work Station

 Configuration
• 32 GB RAM

• Core i7 8th Generation CPU

• 2 TB HDD

• ASRock Mother Board Z series

• Cooler Master Cabinet

MNIST Dataset
• The MNIST database of handwritten digits, available from this
page, has a training set of 60,000 examples, and a test set of
10,000 examples. It is a subset of a larger set available from NIST.
The digits have been size-normalized and centered in a fixed-size
image.
• Each Image Size – 28 x 28
• Color Channel – 1 (Black and White)
Results in CPU of CNN Model

Model Loss vs. Epoch in CPU Model Accuracy vs. Epoch in CPU
Results in GPU of CNN Model

Model Loss vs. Epoch in GPU Model Accuracy vs. Epoch in GPU
Comparison of training time CNN model in CPU
and GPU
Training Time vs Processing Unit
900

800

700

600
Time (Seconds)

500

400

300

200

100

0
CPU GPU
Training Time 845 80
ML/DL Conferences/Journals
• CVPR - Conference on Computer Vision and Pattern Recognition
• NIPS - Neural Information processing systems
• ICML - International conference on machine learning
• ECCV - European Conference on Computer Vision
• ICCV - International Conference on Computer Vision
• ECML - European Conference on Machine Learning
• IJCAI - International Joint Conference on Artificial Intelligence
• JML - Journal of Machine Learning
• IEEE Transactions on Neural Networks
• IEEE Transactions on Pattern Analysis and Machine Intelligence
• Neural Networks
• Artificial Intelligence
• Pattern Recognition
• Expert Systems with Applications
• IEEE Transactions on Knowledge and Data Engineering
• Neural Computing and Applications
• IEEE Transactions on Audio, Speech and Language Processing
• International Journal of Neural Systems
Challenges with Deep Learning

• Handling a large amount of data

• Avoid over fitting problem
• Hyper parameters Optimization
• Requires high-performance hardware
• Rapid and Optimized deployment
Research issues in Deep Learning
• Huge amount of data
• Overfitting in Neural Networks
• Hyperparameter Optimization
• High-performance Hardware Requirement
• Essentially – It is a Blackbox
• Lack of Flexibility, Scalability and Multitasking
• Generative Models – learn and then generate new images
• Video Processing and Analytics
• Deep Learning in Embedded System
• Generalization of algorithms – (Partial Solution: Transfer Learning)
• CPU to GPU Data Transfer Time
Latest Research Trends
Future Trends
• Deep learning is surprising us each and every day, and will continue to
do so in the near future. This is because Deep Learning is proving to be
one of the best technique to be discovered with state-of-the-art
performances.

• Research is continuous in Machine Learning and Deep Learning. But

unlike in previous years, where research was limited to academia,
research in Machine Learning and Deep Learning is exploding in both
industry and academia.
Future Trends
• First of all, seeing the increasing trend of using data science
and machine learning in the industry, it will become increasing
important for each company who wants to survive to
inculcate Machine Learning in their business. Also, each and every
individual would be expected to know the basics terminologies.
• Deep learning is surprising us each and every day, and will
continue to do so in the near future. This is
because Deep Learning is proving to be one of the best technique
to be discovered with state-of-the-art performances.
• Research is continuous in Machine Learning and Deep Learning.
But unlike in previous years, where research was limited to
academia, research in Machine Learning and Deep Learning is
exploding in both industry and academia
Reading Materials
• Text Book: https://www.deeplearningbook.org/
• https://www.coursera.org/learn/machine-learning
• deeplearning.ai
• http://cs231n.stanford.edu/
• https://github.com/floodsung/Deep-Learning-Papers-Reading-
Roadmap
• https://github.com/terryum/awesome-deep-learning-papers
 Thank you!

Any Question..?