INTRODUCTION TO

ARTIFICIAL NEURAL NETWORKS

(ANN)
Dr Visalakshi P
Professor
Dept of ECE
The idea of ANNs..?
NNs learn relationship between cause and effect or
organize large volumes of data into orderly and
informative patterns.
It’s a frog
frog

lion

bird

What is that?
 Neural networks to the rescue…

• Neural network: information processing

paradigm inspired by biological nervous
systems, such as our brain
• Structure: large number of highly interconnected
processing elements (neurons) working together
• Like people, they learn from experience (by
example)

3
Definition of ANN
“Data processing system consisting of a
large number of simple, highly
interconnected processing elements
(artificial neurons) in an architecture inspired
by the structure of the cerebral cortex of the
brain”

(Tsoukalas & Uhrig, 1997).

4
 Neural Networks
What is a Neural Network?
•Biologically motivated approach to
machine learning

Similarity with biological network

Fundamental processing elements of a neural network
is a neuron
1.Receives inputs from other source
2.Combines them in someway
3.Performs a generally nonlinear operation on the
result
4.Outputs the final result
 Inspiration from Neurobiology
Human Biological Neuron

6
 Similarity with Biological Network

• Fundamental processing element of a

neural network is a neuron
• A human brain has 100 billion neurons
• An ant brain has 250,000 neurons
Biological Neural Networks

Biological neuron
 Biological Neural Networks

A biological neuron has

three types of main
components; dendrites,
soma (or cell body) and
axon.
Dendrites receives
signals from other
neurons.
The soma, sums the incoming signals. When
sufficient input is received, the cell fires; that is it
transmit a signal over its axon to other cells.
Biological Neural Networks
• The majority of neurons encode their
outputs or activations as a series of brief
electical pulses (i.e. spikes or action
potentials).

• Dendrites are the receptive zones that

receive activation from other neurons.

• The cell body (soma) of the neuron’s

processes the incoming activations and
converts them into output activations.

• 4. Axons are transmission lines that send

activation to other neurons.

• 5. Synapses allow weighted transmission of

signals (using neurotransmitters) between
axons and dendrites to build up large neural
networks.
 Artificial Neurons

ANN is an information processing system that has

certain performance characteristics in common
with biological nets.
Several key features of the processing elements of
ANN are suggested by the properties of biological
neurons:

1. The processing element receives many signals.

2. Signals may be modified by a weight at the receiving
synapse.
3. The processing element sums the weighted inputs.
4. Under appropriate circumstances (sufficient input), the
neuron transmits a single output.
5. The output from a particular neuron may go to many other
neurons.
 Artificial Neurons
• From experience:
A physical neuron
examples / training
data
• Strength of connection
between the neurons
is stored as a weight-
value for the specific
connection.
• Learning the solution
to a problem =
changing the
connection weights

An artificial neuron
12
 Artificial Neurons

ANNs have been developed as generalizations of

mathematical models of neural biology, based on
the assumptions that:

1. Information processing occurs at many simple elements

called neurons.
2. Signals are passed between neurons over connection links.
3. Each connection link has an associated weight, which, in
typical neural net, multiplies the signal transmitted.
4. Each neuron applies an activation function to its net input
to determine its output signal.
 Artificial Neuron

Four basic components of a human biological The components of a basic artificial neuron
neuron

14
 Model Of A Neuron
Wa
X1

Wb Y
X2  f()

Wc
X3

Input units Connection Summing

computation
weights function

(dendrite) (synapse) (axon)

(soma)
15
• A neural net consists of a large number of
simple processing elements called
neurons, units, cells or nodes.

• Each neuron is connected to other neurons by

means of directed communication links, each
with associated weight.

• The weight represent information being used by

the net to solve a problem.

16
• Each neuron has an internal state, called
its activation or activity level, which is a
function of the inputs it has received.
Typically, a neuron sends its activation as
a signal to several other neurons.

• It is important to note that a neuron can

send only one signal at a time, although
that signal is broadcast to several other
neurons.
17
• Neural networks are configured for a specific
application, such as pattern recognition or
data classification, through a learning
process
• In a biological system, learning involves
adjustments to the synaptic connections
between neurons
 same for artificial neural networks (ANNs)

18
 Artificial Neural Network
Synapse Nukleus

x1 w1
 
y
Axon
x2 w2 Activation Function:
yin = x1w1 + x2w2 (y-in) = 1 if y-in >= 
and (y-in) = 0

Dendrite
-A neuron receives input, determines the strength or the weight of the input, calculates the total
weighted input, and compares the total weighted with a value (threshold)

-The value is in the range of 0 and 1

- If the total weighted input greater than or equal the threshold value, the neuron will produce the
output, and if the total weighted input less than the threshold value, no output will be produced

19
 One Neuron as
a Network
• Here x1 and x2 are normalized attribute value of data.

• y is the output of the neuron , i.e the class label.

• x1 and x2 values multiplied by weight values w1 and w2 are input to the neuron x.

• Value of x1 is multiplied by a weight w1 and values of x2 is multiplied by a weight w2.

• Given that

– w1 = 0.5 and w2 = 0.5

– Say value of x1 is 0.3 and value of x2 is 0.8,

– So, weighted sum is :

– sum= w1 x x1 + w2 x x2 = 0.5 x 0.3 + 0.5 x 0.8 = 0.55

•
 One Neuron as a Network

– The neuron receives the weighted sum as input and calculates

the output as a function of input as follows :

– y = f(x) , where f(x) is defined as

– f(x) = 0 { when x< 0.5 }

– f(x) = 1 { when x >= 0.5 }

– For our example, x ( weighted sum ) is 0.55, so y = 1 ,

– That means corresponding input attribute values are classified in

class 1.

– If for another input values , x = 0.45 , then f(x) = 0,

– so we could conclude that input values are classified to
class 0.

•
 Bias of a Neuron

• We need the bias value to be added to the weighted

sum ∑wixi so that we can transform it from the origin.
v = ∑wixi + b, here b is the bias

x1-x2= -1
x2 x1-x2=0

x1-x2= 1

x1
 Neuron with Activation
• The neuron is the basic information processing unit of a
NN. It consists of:

1 A set of links, describing the neuron inputs, with

weights W1, W2, …, Wm

2. An adder function (linear combiner) for computing the

weighted sum of the inputs (real numbers):
m
u   wjxj
j 1

3 Activation function : for limiting the amplitude of the

neuron output.
y   (u  b)
 History
• 1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model

• 1949 Hebb published his book The Organization of Behavior, in which the Hebbian
learning rule was proposed.

• 1958 Rosenblatt introduced the simple single layer networks now called Perceptrons.

• 1969 Minsky and Papert’s book Perceptrons demonstrated the limitation of single layer
perceptrons, and almost the whole field went into hibernation.

• 1982 Hopfield published a series of papers on Hopfield networks.

• 1982 Kohonen developed the Self-Organizing Maps that now bear his name.

• 1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was re-
discovered and the whole field took off again.

• 1990s The sub-field of Radial Basis Function Networks was developed.

• 2000s The power of Ensembles of Neural Networks and Support Vector Machines
becomes apparent.
• ……………………………………..
24
 Characterization
• Architecture
– a pattern of connections between neurons
• Single Layer Feedforward
• Multilayer Feedforward
• Recurrent
• Strategy / Learning Algorithm
– a method of determining the connection weights
• Supervised
• Unsupervised
• Reinforcement
• Activation Function
– Function to compute output signal from input signal

25
 Single Layer Feedforward NN

x1 w11

w12 ym

w21

x2 yn

w22
output layer
Input layer

Contoh: ADALINE, Hopfield, LVQ, Perceptron, SOFM

26
 Multilayer Neural Network
z1
V11
x1
w12
  w11
V1n
w12 y1

x2 z2

   y2


 zn

xm Vmn  
Input layer Output layer
Hidden layer
Contoh: CCN, GRNN, MADALINE, MLFF with BP, Neocognitron, RBF, RCE
27
 Recurrent NN
Input Outputs

Hidden nodes

Contoh: ART, BAM, BSB, Boltzman Machine, Cauchy Machine,

Hopfield, RNN
28
 Strategy / Learning Algorithm
Supervised Learning

• Learning is performed by presenting pattern with target

• During learning, produced output is compared with the desired output
– The difference between both output is used to modify learning
weights according to the learning algorithm
• Recognizing hand-written digits, pattern recognition and etc.
• Neural Network models: perceptron, feed-forward, radial basis function,
support vector machine.

29
 Unsupervised Learning

• Targets are not provided

• Appropriate for clustering task
– Find similar groups of documents in the web, content
addressable memory, clustering.
• Neural Network models: Kohonen, self organizing maps,
Hopfield networks.

30
 Reinforcement Learning

• Target is provided, but the desired output is absent.

• The net is only provided with guidance to determine the
produced output is correct or vise versa.
• Weights are modified in the units that have errors

31
 Activation Functions
• Identity
f(x) = x
• Binary step
f(x) = 1 if x >= 
f(x) = 0 otherwise
• Binary sigmoid
f(x) = 1 / (1 + e-x)
• Bipolar sigmoid
f(x) = -1 + 2 / (1 + ex)
• Hyperbolic tangent
f(x) = (ex – e-x) / (ex + e-x)
32
x1 w1= 0.5

  y
x2 w2 = 0.3

Activation Function:
yin = x1w1 + x2w2 Binary Step Function
 = 0.5,

(y-in) = 1 if y-in >= 

dan (y-in) = 0

33
Where can neural network systems help…
• when we can't formulate an algorithmic
solution.
• when we can get lots of examples of the
behavior we require.
‘learning from experience’
• when we need to pick out the structure
from existing data.

34
 Who is interested?...
• Electrical Engineers – signal processing,
control theory
• Computer Engineers – robotics
• Computer Scientists – artificial
intelligence, pattern recognition
• Mathematicians – modelling tool when
explicit relationships are unknown

35
 Problem Domains
• Storing and recalling patterns
• Classifying patterns
• Mapping inputs onto outputs
• Grouping similar patterns
• Finding solutions to constrained
optimization problems

36
 Applications of ANNs

• Signal processing
• Pattern recognition, e.g. handwritten
characters or face identification.
• Diagnosis or mapping symptoms to a
medical case.
• Speech recognition
• Human Emotion Detection
• Educational Loan Forecasting
37
 Advantages Of NN
NON-LINEARITY
It can model non-linear systems

INPUT-OUTPUT MAPPING
It can derive a relationship between a set of input & output
responses

ADAPTIVITY
The ability to learn allows the network to adapt to changes in
the surrounding environment

EVIDENTIAL RESPONSE
It can provide a confidence level to a given solution

38
 Advantages Of NN
CONTEXTUAL INFORMATION
Knowledge is presented by the structure of the network.
Every neuron in the network is potentially affected by the
global activity of all other neurons in the network.
Consequently, contextual information is dealt with naturally
in the network.

FAULT TOLERANCE
Distributed nature of the NN gives it fault tolerant capabilities

NEUROBIOLOGY ANALOGY
Models the architecture of the brain

39
Comparison of ANN with conventional AI methods

40