UNIT 1

Introduction to ANN
INTRODUCTION
• The term "Artificial neural network" refers to a biologically inspired
sub-field of artificial intelligence modeled after the brain.
• An Artificial neural network is usually a computational network based
on biological neural networks that construct the structure of the
human brain.
• Similar to a human brain has neurons interconnected to each other,
artificial neural networks also have neurons that are linked to each
other in various layers of the networks.
.
● Artificial Neural Networks
● Artificial Neural Networks contain artificial neurons which are
called units. These units are arranged in a series of layers that
together constitute the whole Artificial Neural Network in a system.
A layer can have only a dozen units or millions of units as this
depends on how the complex neural networks will be required to
learn the hidden patterns in the dataset.
.
● An Artificial Neural Network in the field of Artificial
intelligence where it attempts to mimic the network of neurons
makes up a human brain so that computers will have an option to
understand things and make decisions in a human-like manner.
● The artificial neural network is designed by programming
computers to behave simply like interconnected brain cells.
.
● there are around 1000 billion neurons in the human brain.
● Each neuron has an association point somewhere in the range of
1,000 and 100,000.
● In the human brain, data is stored in such a manner as to be
distributed, and we can extract more than one piece of this data
when necessary from our memory parallelly.
● We can say that the human brain is made up of incredibly amazing
parallel processors.
.
● What is Artificial Neural ● .
Network?
● The term "Artificial Neural
Network" is derived from
Biological neural networks that
develop the structure of a
human brain. Similar to the
human brain that has neurons
interconnected to one another,
artificial neural networks also
have neurons that are
interconnected to one another
ADVANTAGES OF NEURAL NETWORKS
● Self-organization: An ANN can generate its
own representation of the information that it
receives at the time of learning.
● Real-Time Operation: Real-Time Operation:
Neural networks can perform calculations
simultaneously, allowing for real-time
operation in certain applications.
● Adaptive learning: Capability to learn how to
solve tasks is based on the data given for the
training set.
● Redundant Information Coding Through Fault
Tolerance: Redundant Information Coding
Through Fault Tolerance: Neural networks can
exhibit fault tolerance, meaning they can still
function even if some parts of the network are
damaged or destroyed.
 APPLICATION SCOPE OF NEURAL
NETWORKS
● Neural networks, specifically artificial neural networks (ANNs), have
a broad application scope across various domains. Here are some
notable areas where neural networks are commonly applied:
● Computer Vision: Neural networks are extensively used in
computer vision tasks, such as image classification, object
detection, image segmentation, facial recognition, and video
analysis. Convolutional Neural Networks (CNNs) are particularly
popular in this domain due to their ability to learn hierarchical
representations of visual data.
 APPLICATION SCOPE OF NEURAL
NETWORKS
● Natural Language Processing (NLP): Neural networks play a crucial role in NLP tasks, including
sentiment analysis, text classification, machine translation, named entity recognition, question
answering, and language generation. Recurrent Neural Networks (RNNs) and Transformer models
have significantly advanced the performance in these areas.
● Speech and Audio Processing: Neural networks are utilized in speech recognition, speech synthesis,
speaker identification, music generation, and audio classification tasks. Recurrent Neural Networks
(RNNs) and Convolutional Neural Networks (CNNs) are commonly employed for analyzing and
generating speech and audio signals.
● Recommendation Systems: Neural networks are employed in recommendation systems to provide
personalized recommendations for products, movies, music, and more. Collaborative filtering
techniques and deep learning architectures are used to model user preferences and make accurate
recommendations.
APPLICATION SCOPE OF NEURAL
NETWORKS
● Time Series Analysis and Forecasting: Neural networks, particularly Recurrent
Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks, are used
for time series analysis, forecasting, and modeling. They can capture temporal
dependencies and make predictions based on historical data, finding applications in
finance, weather prediction, stock market analysis, and more.
● Robotics and Control Systems: Neural networks are employed in robotics and
control systems to enable intelligent decision-making, motion planning, and control.
They can learn to navigate complex environments, manipulate objects, and perform
tasks requiring perception and action coordination.
● Anomaly Detection: Neural networks are used for anomaly detection in various
domains, including network security, fraud detection, manufacturing, and
healthcare. They can learn normal patterns from data and identify deviations or
outliers, enabling the detection of unusual or suspicious behavior.
APPLICATION SCOPE OF NEURAL
NETWORKS
● Healthcare: Neural networks have applications in various healthcare areas, including
disease diagnosis, medical imaging analysis, drug discovery, genomics, and
personalized medicine. They can aid in diagnosing diseases, identifying medical
conditions from medical images, predicting treatment outcomes, and discovering
potential drug candidates.
● Financial Analysis: Neural networks are used in financial institutions for tasks such
as credit scoring, fraud detection, algorithmic trading, and risk assessment. They can
analyze large volumes of financial data, identify patterns, and make predictions or
decisions based on market trends and risk factors.
● Gaming and Simulation: Neural networks have been applied in gaming and
simulation for tasks like game playing, character animation, and game level
generation. They can learn strategies, simulate virtual environments, and generate
content to enhance the gaming experience
Artificial Neural Networks
● Artificial Neural Network has an input
layer, an output layer as well as hidden
layers.
● The input layer receives data from the
outside world which the neural
network needs to analyze or learn
about.
● Then this data passes through one or
multiple hidden layers that transform
the input into data that is valuable for
the output layer.
● Finally, the output layer provides an
output in the form of a response of the
Artificial Neural Networks to input data
provided.
Neuron
● Each neuron has an internal state, which
is called an activation signal.
● Output signals, which are produced
after combining the input signals and
activation rule, may be sent to other
neurons
Working of Neuron
● Let there are two neurons X and Y which is
transmitting signal to another neuron Z .
● Then , X and Y are input neurons for
transmitting signals and Z is output neuron for
receiving signal .
● The input neurons are connected to the
output neuron , over a interconnection links
( A and B ) as shown in figure .
 In neuron architecture, the net input has to be
.
●
calculated in the way.
I = xA + yB
where x and y are the activations of the input
neurons X and Y.
A and B are Weights
The output neuron Z can be obtained by applying
activations over the net input.
O = f(I)
Output = Function ( net input calculated )
Each neuron has internal state of its own, it is
called activation
The function to be applied over the net input is
called activation function .
The main purpose of the activation function is to
convert the weighted sum of input signals of a
neuron into the output signal.
Neural net of pure linear equation
● The above calculation of
netinput is similar to the
calculation of output of a pure
linear equation(y=mx).
● To obtain the output y, the
slope m directly applied with
the input signal.
● Y = mx
Biological Neural Network
● Neurons are the fundamental unit of the
nervous system. Neurons are also called as
nerve cells.
● The main function of the neurons is to pass the
receiving information and send appropriate
signals to the rest parts of the body.
● The signals received are in the form of
electrical signals. Neuron comprises of
dendrite, axon and cell body.
● Dendrites receives messages from the
surrounding and sends it to the cell body.
● Cell body consists of nucleus, mitochondria
and other organelles.
● Axon transmits the message away from the cell
body and pass it to the the next receiving
neuron.
Biological neural network and artificial neural
network:
Biological neural network and artificial neural
network:
● A biological neuron receives its input signals
from other neurons through dendrites (small
fibers). Likewise, a neuron receives its data
from other neurons through input
neurons that take numbers.
● The connection points between dendrites
and biological neurons are called synapses.
Likewise, the connections between inputs and
neurons are called weights. They measure the
importance level of each input.
● In a biological neuron, the nucleus produces an
output signal based on the signals provided by
dendrites. Likewise, the nucleus (colored in
blue) in a neuon performs some calculations
based on the input values and produces an
output.
● Axon
● Axon is a tube-like structure that carries
Biological neural network and artificial neural
network:
● In a biological neuron, the ● .
output signal is carried away by
the axon. Likewise, the axon in
a neuron is the output
value which will be the input
for the next neurons.
Relationship between Biological neural network and artificial
neural network:

Biological Neural Network Artificial Neural Network

Dendrites Inputs
Cell nucleus Nodes
Synapse Weights
Axon Output
Characteristics of Artificial Neural Network
● It is neuraly implemented mathematical model
● It contains huge number of interconnected processing elements
called neurons to do all operations
● Information stored in the neurons are basically the weighted
linkage of neurons
● The input signals arrive at the processing elements through
connections and connecting weights.
● It has the ability to learn , recall and generalize from the given data
by suitable assignment and adjustment of weights.
● The collective behavior of the neurons describes its computational
power, and no single neuron carries specific information .
●
.
In 1949, Donald Hebb reinforced the concept of neurons in his book, The Organization of Behavior. It pointed out that neural pathways are
strengthened each time they are used.
● In the 1950s, Nathanial Rochester from the IBM research laboratories led the first effort to simulate a neural network.
● In 1956 the Dartmouth Summer Research Project on Artificial Intelligence provided a boost to both artificial intelligence and neural networks.
This stimulated research in AI and in the much lower level neural processing part of the brain.
● In 1957, John von Neumann suggested imitating simple neuron functions by using telegraph relays or vacuum tubes.
● In 1958, Frank Rosenblatt, a neuro-biologist of Cornell, began work on the Perceptron. He was intrigued with the operation of the eye of a fly.
Much of the processing which tells a fly to flee is done in its eye. The Perceptron, which resulted from this research, was built in hardware and is
the oldest neural network still in use today. A single-layer perceptron was found to be useful in classifying a continuous-valued set of inputs into
one of two classes. The perceptron computes a weighted sum of the inputs, subtracts a threshold, and passes one of two possible values out as
the result.
● In 1959, Bernard Widrow and Marcian Hoff of Stanford developed models they called ADALINE and MADALINE. These models were named for
their use of Multiple ADAptive LINear Elements. MADALINE was the first neural network to be applied to a real-world problem. It is an adaptive
filter which eliminates echoes on phone lines. This neural network is still in commercial use.
● Marvin Minsky & Seymour Paper proved the Perceptron to be limited in their book, Perceptrons
● Progress on neural network research halted due fear, unfulfilled claims, etc. until 1981. This caused respected voices to critique the neural
network research. The result was to halt much of the funding. This period of stunted growth lasted through 1981.
● 1982 — John Hopfield presented a paper to the national Academy of Sciences. His approach to create useful devices; he was likeable, articulate,
and charismatic.
● 1982- US-Japan Joint Conference on Cooperative/ Competitive Neural Networks at which Japan announced their Fifth-Generation effort resulted
US worrying about being left behind. Soon funding was flowing once again.
● 1985 — American Institute of Physics began what has become an annual meeting — Neural Networks for Computing. By 1987, the Institute of
Electrical and Electronic Engineer’s (IEEE) first International Conference on Neural Networks drew more than 1,800 attendees.
● In 1997 — A recurrent neural network framework, Long Short-Term Memory (LSTM) was proposed by Schmidhuber & Hochreiter.
● In 1998, Yann LeCun published Gradient-Based Learning Applied to Document Recognition.
.
● ANNs are composed of multiple nodes,
which imitate biological neurons of the
human brain. The neurons are
connected by links and they interact
with each other. The nodes can take
input data and perform simple
operations on the data. The result of
these operations is passed to other
neurons. The output at each node is
called its activation or node value.
● Each link is associated
with weight. ANNs are capable of
learning, which takes place by altering
weight values. The following
illustration shows a simple ANN –
Basic models of ANN
• Interconnections:
• Interconnection can be defined as the way processing elements
(Neuron) in ANN are connected to each other. Hence, the
arrangements of these processing elements and geometry of
interconnections are very essential in ANN.
• Five basic types of neuron connection architecture :
• Single-layer feed-forward network
• Multilayer feed-forward network
• Single node with its own feedback
• Single-layer recurrent network
• Multilayer recurrent network
Single-layer feed-forward network
•. • #1) Single-Layer Feed-Forward Network
• A layer is a network formed of neurons.
These neurons are connected to the other
neurons of the next layer.
• For a single layer, there are only the input
and output layers. The input layer is
connected to the output layer nodes with
weights.
• All the input nodes are connected to each
of the output nodes.
• The term feed-forward depicts that there
is no feedback sent from the output layer
to the input layer. This forms a single layer
feed-forward network
•.
Multilayer feed-forward network
• Multi-Layer Feed-Forward Network
• The Multi-layer network consists of one or
more layers between the input and output.
The input layer just receives a signal and
buffers it while the output layer shows the
output. The layers between the input and
output are called the hidden layers.
• The hidden layers are not in contact with the
external environment. With more number of
hidden layers, the output response is more
efficient. The nodes in the previous layer are
connected to each node in the next layer.
• As there is no output layer connected to the
input or hidden layers, it forms a multi-layer
feed-forward network.
Single node with its own feedback
• #3) Single Node With Its Own
Feedback
• The networks where the output
layer output is sent back as an
input to the input layer or the other
hidden layers is called Feedback
Networks
• . In single-node feedback systems,
there is a single input layer where
the output is redirected back as
feedback.
•
 Single Layer Recurrent Network
In a single layer recurrent
network, the feedback
network forms a closed
loop.
In this model, a single
neuron receives feedback to
#4)
itself or the other neurons
in the network or both.
MLRN
• In Multi-Layer recurrent
network, multiple hidden layers
exist and the output is
redirected back to the neurons
of the previous layers and other
neurons in the same layers or
same neuron itself.
• Adjustments of Weights or Learning • .
• Learning, in artificial neural network, is the method of modifying the
weights of connections between the neurons of a specified network.
Learning in ANN can be classified into three categories namely supervised
learning, unsupervised learning, and reinforcement learning.
• Supervised Learning deals with two main tasks Regression and
Classification.
• Unsupervised Learning deals with clustering and associative rule mining
problems.
• Whereas Reinforcement Learning deals with exploitation or exploration,
Markov's decision processes, Policy Learning, Deep Learning and value
learning.
• Supervised Learning
• As the name suggests, this type of
. learning is done under the supervision
of a teacher. This learning process is
dependent.
• During the training of ANN under
supervised learning, the input vector is
presented to the network, which will
give an output vector.
• This output vector is compared with
the desired output vector.

• An error signal is generated, if there is

a difference between the actual
output and the desired output vector.
On the basis of this error signal, the
weights are adjusted until the actual
output is matched with the desired
output.
• Unsupervised Learning
• As the name suggests, this type of learning
. is done without the supervision of a
teacher. This learning process is
independent.
• During the training of ANN under
unsupervised learning, the input vectors
of similar type are combined to form
clusters.
• When a new input pattern is applied, then
the neural network gives an output
response indicating the class to which the
input pattern belongs.
• There is no feedback from the
environment as to what should be the
desired output and if it is correct or
incorrect.
• Hence, in this type of learning, the
network itself must discover the patterns
and features from the input data, and the
relation for the input data over the
output.
• Reinforcement Learning
.
• As the name suggests, this type of learning
is used to reinforce or strengthen the
network over some critic information. This
learning process is similar to supervised
learning, however we might have very less
information.
• During the training of network under
reinforcement learning, the network
receives some feedback from the
environment.
• This makes it somewhat similar to
supervised learning. However, the
feedback obtained here is evaluative not
instructive, which means there is no
teacher as in supervised learning.
• After receiving the feedback, the network
performs adjustments of the weights to
get better critic information in future.
Criteria Supervised ML Unsupervised ML Reinforcement ML
Trained using unlabelled
Learns by using labelled Works on interacting with
Definition data without any
data the environment
guidance.
Type of data Labelled data Unlabelled data No – predefined data
Regression and
Type of problems Association and Clustering Exploitation or Exploration
classification
Supervision Extra supervision No supervision No supervision
Linear Regression, Logistic K – Means, Q – Learning,
Algorithms
Regression, SVM, KNN etc. C – Means, Apriori SARSA
Discover underlying
Aim Calculate outcomes Learn a series of action
patterns
Risk Evaluation, Forecast Recommendation System, Self Driving Cars, Gaming,
Application
Sales Anomaly Detection Healthcare
Components of ANN
• Inputs: They’re usually represented as features of a dataset which are
passed on to a neural network to make predictions.
• Weights: These are the real values associated with the features. They are
significant as they tell the importance of each feature which is passed as an
input to the artificial neural network.
• Bias: Bias in a neural network is required to shift the activation function
across the plane either towards the left or the right. We will cover it in
more detail later.
• Summation function: It is defined as the function which sums up the
product of the weight and the features with bias.
• Activation function: It is required to add non-linearity to the neural network
model
.
● Activation functions are really important for an Artificial Neural
Network to learn and make sense of something reallocated and
Non-linear complex functional mappings between the inputs and
response variable. They introduce non-linear properties to our
Network.
● The activation function is applied over the net input to calculate
the output of an ANN.
● The information processing of a processing element can be viewed
as consisting of two major parts: input and output.
● An integration function is associated with the input of a

processing element. This function serves to combine activation,

information or evidence from an external source or other processing
elements into a net input to the processing element.
Terminologies of ANN
● Weights: In an ANN, weights are numerical values assigned to the
connections between neurons. They represent the strength of the
connections and determine the impact of one neuron on another
in the network. The weights are adjusted during the learning
process to optimize the network's performance.
● Bias: Bias is an additional parameter in each neuron of an ANN. It
represents a constant value that is added to the weighted sum of
inputs to introduce a shift or offset. Bias helps the model to learn
and represent complex relationships that may not be captured by
the input data alone.
● Bias: Suppose the output neuron in the previous example has a bias
term of b = 0.2. The bias is added to the weighted sum of inputs
before passing it through the activation function. It allows the
model to introduce a shift or offset in the output.
● Learning Rate: The learning rate is a hyperparameter that controls
the step size at which the weights of the neural network are
updated during the training process. It determines how much the
weights are adjusted based on the calculated error. A higher
learning rate can lead to faster convergence, but it may also risk
overshooting the optimal solution. A lower learning rate may result
in slower convergence but can provide more accurate results.
● Learning Rate: Assume during the training process, the learning
rate is set to α = 0.01. This value determines the step size at which
the weights are adjusted based on the calculated error. It controls
how quickly or slowly the neural network learns from the training
data.
● Threshold: In the context of ANNs, the threshold usually refers to
the activation threshold or activation function threshold. It is a
value that determines whether the neuron will fire or activate
based on the weighted sum of inputs and bias. If the sum exceeds
the threshold, the neuron will be activated and produce an output;
otherwise, it remains inactive.
● Vigilance Parameters: In the context of Adaptive Resonance Theory
(ART) networks, a vigilance parameter determines the level of
similarity or tolerance required for pattern recognition. For
instance, if the vigilance parameter is set to 0.8, the network will
accept input patterns as similar or close matches only if their
similarity exceeds this threshold.
● Momentum Factor: Momentum is a technique used during the
training of neural networks to speed up convergence and prevent
the model from getting stuck in local minima. The momentum
factor is a hyper parameter that determines the contribution of the
previous weight update to the current update. It allows the model
to accumulate momentum in a certain direction, helping it to move
past flat or shallow areas in the error surface.