Deep Learning:

What is Deep Learning?

Deep learning refers to an artificial intelligence (AI) function that imitates the functionality of the human
brain in processing data and developing patterns for use in key decision making.

Deep learning falls under the larger the subset of machine learning in AI that has networks of learning
unsupervised from unstructured/unlabeled data. Other names for deep learning include deep neural
learning or deep neural networks.

Deep learning has evolved with the rise of the 4 th Industrial Revolution (4IR) which brought about a rise
of data in all forms from a variety of industries and sectors of society. This data, coined as ‘big data’,
arrives from many platforms and is readily accessible but the vast amount of this data is unstructured
which ultimately means that extraction and comprehension by humans to get relevant information
would be time intensive and tedious. Deep learning was used to adapt existing AI systems to learn from
the vast amounts of unstructured data and process and understand it much faster than humans.

Machine learning is a self-adaptive algorithm that improves analysis of patterns and data with
experience or with new data being constantly added to what it has seen.

Deep learning utilizes a hierachical level of artificial neural networks (ANNs) to carry out the machine
learning process. ANNs are built akin to the human brain, with neuron nodes being connected together
in a complex pattern. Traditional data processing programs build analysis of data in a very linear manner
whereas the hierachical functionality of deep learning systems allows machines to process data in a very
non-linear approach. An application of deep learning was developed by Panasonic in conjunction with
universities and research institutions to use deep learning in computer vision technologies.

An example of an application is detecting fraud in a digital payment system where a machine learning
computational algorithm built into a computer model can process all transactions happening on the
digital platform hence finding patterns automatically in the data set and pointing out any anomaly
detected by the pattern. Traditionally fraud detection relies on the amount of the transaction that
ensues while a non-linear deep learning technique includes time, geographic location, IP address, type
of retailer and other features that is likely to point to fraud.

A neural network’s first layer processes raw data as an input such as the amount of the transaction and
then passes it on to the next layer as an output. The second layer processes the previous layer’s
information by including additional information like the user’s IP address and passes on the result to the
next layer. The next layer takes the second layer’s information and includes raw data like geographic
location and improves the machine’s pattern. This process continues across all levels of the neuron
network.

Each layer of a neural network builds on its previous layer with added data. Deep learning algorithms
are trained to not just create patterns from all transactions but also know when a pattern is signalling
the need to investigate fraud. The final layer of the neural network relays a signal to an analyst who may
freeze the user’s account until all pending investigations are finalised.
Deep learning can be used in a number of different applications such as image recognition tools, open-
source platforms with consumer recommendation applications and medical research tools that explore
different uses of drugs for new ailments.

Above is taken from:

https://www.investopedia.com/terms/d/deep-learning.asp

From Mathworks:

Deep learning is a machine learning technique that ‘teaches’ computers to execute functions done by
the human brain, through the process of learning by example. Deep learning is the key proponent
behind driverless cars which allows to recognize stop signs or to distinguish a pedestrian walking across
the road from a traffic light.

Deep learning is used in voice control in consumer devices such as phones, tablets, TVs and bluetooth
speakers. Specifically, in deep learning, a computer model learns to perform classification tasks directly
from images, text or sound. The capability of deep learning models is quite astounding as they can
achieve higher levels of accuracy than human performance. These models are trained by using a large
set of labeled data and multi-layer neural networks architectures.

Why deep learning matters?

Why is deep learning so useful?

1) Deep learning takes in large amounts of labeled data.

2) Deep learning requires substantial computing power. High-performance GPUs have a parallel
architecture which is efficient for deep learning. If combined with cloud computing, this enables
development teams to reduce training time for a deep learning network.

Applications of deep learning:

 Automated Driving: Deep learning becomes useful in automatically detecting objects such as
stop signs and traffic lights. It can help decrease accidents as pedestrians can be detected.
 Aerosapce and Defence: Deep learning is used to identify objects from satellites that locate
areas of interest, and identifying safe/unsafe zones for combat troops.
 Medical Research: Cancer researchers are using deep learning to automatically detect cancer
cells. UCLA have built an advanced microscope that yields a high-dimensional data set used to
train a deep learning application to accurately identify cancer cells.
 Industrial Automation: Improves worker safety around heavy machinery by automatically
detecting when people or objects are within unsafe distance of machinery.
 Electronics: Deep learning is being used in automated hearing and speech translation. E.g. home
assistance devices that respond to your voice and know your preferences are powered by deep
learning applications.

Typical Neural Network Architecture:

Most deep learning methods use ANNs or coined as deep neural networks. The term “deep” usually
refers to the number of hidden layers in the neural network. Traditional neural networks typically
contain 2-3 hidden layers, while deep neural networks can have as many as 150 layers.

Deep learning models are trained by using large sets of labeled data and NNAs that learn features
directly from the data without the need for manual feature extraction.

Neural networks are organized in layers consisting of interconnected nodes. One of the most popular
types of DNNs is known as Convolutional Neural Networks (CNN or ConvNet). A CNN convolves learned
features with input data, and uses 2D convolutional layers, making this architecture suitable to
processing 2D data such as images.

CNNs eliminate the need for using manual feature extraction

4 different categories:

1) General Deep Learning (Fully Connected Nets)

2) Image Models (ConvNets)
3) 1D Sequence Models (RNN’s, LSTM’s, Attention Models etc.)
4) Other Miscellaneous (Unsupervised Learning, Reinforcement Learning, Sparse Coding, Slow
Feature Analysis etc.)

How to create and train a deep learning model?

3 common ways to train a deep learning model:

1) Training from Scratch

To train a model from scratch, you need to gather a very large labeled data set and design a network
architecture that will learn the features and model. This is advantageous for new applications, or
applications that will have a large number of output categories. However, this is a less coomon approach
because of the large amount of data and the expected rate of learning with these type of networks
typically taking days or weeks to properly train.

2) Transfer Learning

Most deep learning applications use an approach of transfer learning which involves fine tuning a pre-
trained deep learning model. It begins with an existing network e.g. AlexNet or GoogLeNet and new data
containing previously unknown classes are fed in. With some minor adjustments to the network, a new
task can be performed such as categorizing a single class type from a group of 1000 different classed
objects. The advantage of this method is needing much less data (processing thousands of images,
rather than millions), so the computation time drops to minutes or hours.

However, transfer learning requires an interface to the internal structure of the pre-existing network, so
it can be ‘surgically’ modified and improved for accomplishing the new task it was meant to do. Such an
interface can be any GUI or Code compiler that can be the ‘middle man’.

3) Feature Extraction

This is a slightly less common method and is a more specialized approach to deep learning where the
network is used as a feature extractor. Since all of the layers are tasked with learning specific features
from data sets, certain features can be pulled out from the network at any time during the training
process. These features can then be used as input to other machine learning models such as Support
Vector Machines (SVM).

Different Network Structures:

 AlexNet
 ZFNet
 NIN
 GoogLeNet/Inception
 VGGNet
 ResNet
 DenseNet
 Xception
 CapsNet
 ImageNet and MS COCO Data Sets
 Object Detection Networks (ODNs)
 Image Segmentation Networks (ISNs)
 Recurrent Neural Networks (RNNs)
 Generative Adversarial Nets (GANs)

Additional Training Methods:

 Gradient Descent Optimization Algorithms: SGD, Momentum, Adagrad, RMSProp, Adam

 Training Strategies: Dropout, batch normalization, group normalization etc.
Comparison between machine learning and deep learning:

Machine learning extracts the features of images such as corners and edges in order to create models of
the various objects. It later uses these models to identify the objects. Machine learning does not require
high performance processors and more data.

Deep Learning is subtype of machine learning. Deep Learning is a machine learning technique which
learns features and tasks directly from data. The data can be images, text files or sound.

The figure above depicts processes followed to identify the object in both machine learning and deep
learning. Deep Learning does not require feature extraction manually and takes images directly as an
input. Deep learning requires high performance GPUs and lots of data. Feature extraction and
classification are carried out by deep learning algorithms known as CNNs. CNN takes care of feature
extraction as well as classification based on multiple images.