AUTOMATIC PATTERN CLASSIFICATION BY UNSUPERVISED LEARNING

USING DIMENSIONALITY REDUCTION OF DATA WITH MIRRORING

NEURAL NETWORKS

Name(s) Dasika Ratna Deepthi (1), G.R.Aditya Krishna (2) and K. Eswaran (3)
Address: (1) Ph. D. Student, College of Engineering, Osmania University, Hyderabad-500007, A.P.,
(2) Altech Imaging & Computing, Sri Manu Plaza, A.S. Rao Nagar, Hyderabad, 500062, A.P.,
(3) Sreenidhi Institute of Science and Technology, Yamnampet, Ghatkesar, Hyderabad - 501 301. A.P.,
Country India
Email addresses [email protected], [email protected], [email protected]

ABSTRACT as possible for the characterization of data, are important

This paper proposes an unsupervised learning technique steps in computer aided pattern recognition. This may be
by using Multi-layer Mirroring Neural Network and supervised or unsupervised depending on whether or not
Forgy’s clustering algorithm. Multi-layer Mirroring the pattern label information is provided at the time of
Neural Network is a neural network that can be trained training and classification. This paper deals with
with generalized data inputs (different categories of image extracting feature set from the input using MMNN and
patterns) to perform non-linear dimensionality reduction unsupervised classification using Forgy’s clustering
and the resultant low-dimensional code is used for algorithm. There are many techniques for features
unsupervised pattern classification using Forgy’s extraction of the object and machine learning [1], [2]
algorithm. By adapting the non-linear activation function which can be applied to recognition. An elaborate
(modified sigmoidal function) and initializing the weights discussion on several techniques used to perform
and bias terms to small random values, mirroring of the nonlinear dimensionality reduction can be found in [3].
input pattern is initiated. In training, the weights and bias [4] and [5] deal with the neural network approach to
terms are changed in such a way that the input presented dimensionality reduction. For dimensionality reduction
is reproduced at the output by back propagating the error. there are other techniques like PCA, ICA. In [6], [7] and
The mirroring neural network is capable of reducing the [8] dimensionality reduction is used for visualization of
input vector to a great degree (~ 1/30th the original size) patterns in data using PCA and multi-dimensional scaling.
and also able to reconstruct the input pattern at the output According to the discussion given in [9] neural network
layer from this reduced code units. The feature set (output approach performs better than PCA.
of central hidden layer) extracted from this network is fed
to Forgy’s algorithm, which classify input data patterns This paper deals with
into distinguishable classes. In the implementation of  Nonlinear dimensionality reduction (Feature
Forgy’s algorithm, initial seed points are selected in such extraction)
a way that they are distant enough to be perfectly grouped  Reconstruction of the input patterns
into different categories. Thus a new method of  Classification of data in unsupervised mode
unsupervised learning is formulated and demonstrated in
this paper. This method gave impressive results when For dimension reduction and reconstruction MMNN [10]
applied to classification of different image patterns. is used and to classify data in an unsupervised mode
Forgy’s algorithm is used.
KEY WORDS
Unsupervised learning, Multi-layer Mirroring Neural MMNN is a generalized network which can take different
Network (MMNN), pattern classification, Forgy’s input patterns and produce their mirrors (same as input) as
algorithm, modified sigmoidal function, feature set. output. At the same time, it also reduces the dimension of
input data. This is a very simple generalized network
From now on Multi-layer Mirroring Neural Network will where there is no need to change the network architecture
be abbreviated as MMNN. as training proceeds (as done in other neural network
approach [11]). This approach has good ability to learn
1. Introduction and reconstruct image patterns having extremely different
features fed to the network. With this approach we report
Feature extraction of patterns in data and the task of a 95% success rate in the classification of different
dimension reduction, that is, choosing as few parameters patterns.
2. Pattern Classification by Unsupervised
Input Pattern
Learning

The algorithm presented here is an attempt of classifying

different data patterns using mirroring neural network and
Forgy’s unsupervised classification algorithm.
Generalized MMNN (Trained with 3 categories of image
We first use the mirroring neural network to reduce the patterns)
dimensions of input data pattern. The reduced dimension
feature vectors thus obtained are then classified by using
Forgy’s unsupervised classification algorithm. Thus we
obtain an algorithm which not only performs dimension
reduction of the features but also clusters similar data
patterns into one group. Forgy’s clustering technique

In this work, we present an MMNN having compatible

number of nodes at the first hidden layer to participate in
the learning process by accepting input patterns from the Group 1 Group 2 Group 3
input layer. This network is designed to have least
possible number of nodes at central hidden layer to reduce
the dimension of input pattern the output layer has same
Fig. 1 MMNN that accepts generalized input patterns and classifies
number of nodes as input layer and is used to reconstruct them in to one of the three categories with the help of Forgy’s
(mirror) the input data pattern. Using least dimensional algorithm
central hidden layer outputs as input feature vector for
Forgy’s algorithm, data patterns are grouped into different 2.1 Inputs to the Multi-layer Mirroring Neural
clusters. In contrast to the typical training approach where Network
the neural network accepts categorized input pattern, this
algorithm learns different patterns and categorize them Inputs to MMNN are three different categories of 8-Bit
into groups by itself. images of size 26X26. These include faces, flowers and
furniture. MMNN uses hyperbolic tangent function
A pictorial representation of MMNN architecture is given (sigmoidal function) that accepts input from -1 to +1. To
in Fig. 1. MMNN architecture resembles the architecture fit in to this range input intensities are linearly rescaled
given in [12]. The difference between these two is that in [13] from [0,255] to [-1, +1].
[12], the architecture is designed with specialized
mirroring networks for each category of the input pattern, I input = (I input – 128) / 128
where as in MMNN, the network is a generalized network
that can accept any type of input pattern. Using Forgy’s Where I input is input pixel intensity.
unsupervised clustering algorithm input pattern is
categorized into one of the different groups. To accelerate the learning process a truncation function
(as discussed in [14]) is implemented at the input layer.

Truncation function is defined as:

f trc (I input) = -0.9 if I input ≤ (-0.9)

= +0.9 if I input ≥ (+0.9)
= I input otherwise

2.2 MMNN Architecture

The mirroring neural network is trained with different

categories of image patterns rescaled according to the
discussion given in 2.1. Fixing the number of layers and
the number of nodes in each layer is done after
considerable experimentation. The first hidden layer of
size 30 followed by the second hidden layer of size 20 and
an output layer of size equal to the input was found to be Automatic pattern classification of the training set as well
the most suitable architecture for our network. So, we as test set is done by simple partitional clustering
have chosen 676-30-20-676 multilayer mirroring network technique which is widely used in engineering
architecture which is in contrast to the symmetrical applications as described in [17]. We implemented this
dimensions of the encoder and decoder layers of the using Forgy’s algorithm [18].
“autoencoder” described in [9] and “auto-associative”
neural networks discussed in [3]. With this, it is apparent Forgy’s algorithm:
that there exists a nonlinear relationship between the 1. Select initial seed points from input data set.
dimensions of adjacent layers (excluding the input layer) 2. For all the input datasets repeat step 2.
to accurately reconstruct the input at the output of the a. Calculate distance between each input data set
network. We modified the sigmoidal activation function and the each of the seed points representing a
and used hyperbolic tangent function, instead of logistic cluster.
function implemented in [9], as it was found to be
appropriate for our input data as described in [15]. The b. Place the input data into the group associated
functional output at each node of the hidden layers and with the seed point which is closest to input data
output layer with activation function of the form Sgm(s) = set (least of the distances in step 2 a)
tanh (s/2) passed through modified hyperbolic tangent 3. Centroids of all the clusters are considered as new seed
function as discussed below. points.
4. Repeat step 2, 3 as long as the data sets leave one
The output y, which has to be passed through the filter cluster to join another in step 2 b.
that produces modified sigmoidal function having the
upper and lower bounds at +0.9 and -0.9 respectively, is Input to the Forgy’s algorithm is the dimensionally
given by: reduced central hidden layer’s output. This is the feature
f mod(y) = -0.9 if y ≤ (-0.9) set extracted from MMNN.
= +0.9 if y ≥ (+0.9)
= y otherwise The number of initial selected seed points is equal to the
number of distinguishable patterns in the training set.
Where y is the sigmoidal hyperbolic tangent function Initially, the seed points selected must be distinct in such a
defined as: way that they perfectly cluster the input patterns and test
patterns. The Euclidean distance between any two initial
y = Sgm(s) = tanh (s/2) = (1 - e-s ) / (1 + e-s) seed points belonging to two different categories is a
representation of the inter-set distance between the two
Modified Hyperbolic Tangent Function forces the output categories of images. The inter-set Euclidean distance is
of the multilayer mirroring neural network in the range of fixed by experimentation depending on the categories of
-0.9 and +0.9 instead of -1.0 and +1.0. This prevents the images with which we train the mirroring neural network.
multilayer neural network from out-of-range values and We have used a threshold value 1.3 i.e., euclidean
helps in faster convergence. MMNN is a back propagating distances between any two initial seed points should be
network that minimizes the error between the input and its greater than 1.3.
reconstruction using gradient descent [16]. Learning
parameter, weights and bias terms are initialized to very For the initial seed points the following procedure is
small random values to effectively learn different patterns. followed: First seed point is randomly selected from the
Training the input patterns to the specified accuracy input data set. Then, to select second seed point (from the
(above 95%) is achieved by fixing a considerable amount input data set), the Euclidean distance between the first
of threshold distance in between the input and its mirror. and the second must be greater than 1.3. Similarly to
select nth seed point, the Euclidean distance between the
2. 3 Output Rescaling present point and already selected n-1 seed points should
be greater than 1.3. In our approach there are 3 categories
As the modified hyperbolic tangent function itself results of input data. So, we go for 3 initial seed points to
an output value in the range of -0.9 to +0.9 which is separate the input into 3 distinguishable groups. After
compatible with the truncated input, there is no need to selecting the 3 initial seed points using the aforementioned
rescale the desired output again at the time of training the procedure and initializing them as cluster centroids, find
patterns. the cluster centroid nearest to each sample in the training
set by comparing the Euclidean distances (distance
2.4 Pattern Classification using unsupervised learning between the input dataset and the cluster centroid). Mark
the sample as of its nearest cluster. This is done for each
of the samples in the training set (and/or test set). After
completing the process of grouping the samples, compute 84 flower images and 1 face image. Group 3
the new centroid of the resulting clusters. Based on the contained 87 face images and 4 flower images.
new centroids of the resultant clusters, group the samples From 264 input images, 259 images were
again by marking it as of its nearest cluster centroid. classified correctly (98.1% success rate).
Repeat this procedure till there is no change in the cluster  The algorithm was tested with 90 new test
groups. This is done in unsupervised mode because while images (30 face images, 30 furniture images and
training the network we are not giving any information 30 flower images) using the aforementioned
regarding the category of the input pattern, and weights and bias terms. These 90 images were
classification is simply based on the selection of distant entirely new images not occurring in the training
initial seed points. samples. The algorithm automatically classified
the input images into three groups. The first
2.5 Results group consisted of 30 furniture images (100%
classification was achieved), second group
consisted of 28 flower images. The third group
The inputs for training the multilayer mirroring neural consisted of 30 face and 2 flower images. From a
network are 8-Bit grayscale images of size 26X26. test sample space of 90 images, 88 images were
classified correctly (97.77% success rate).
Experiment 1:
(Two different input patterns; Face and Furniture) We have also tested the mirroring neural network for
generalized acceptance, learning competence and output
 While training MMNN, 176 images (88 face reconstruction (please refer to the fig 2 for reconstruction).
images and 88 furniture images) were given as
input. MMNN was trained till it successfully The input and output images of the multilayer neural
mirrored 95% of the input images. With these network are given in fig. 2.
weights and bias terms, the algorithm
automatically classified the 176 images into two Inputs to the network
groups. The first group contained 89 images (88
furniture images and 1 face image).The second
group contained 87 face images. From 176 input
images, 175 images were classified correctly
(99.43% success rate). Corresponding Outputs from the network
 The algorithm was tested with 80 new test
images (40 face images, 40 furniture images)
using the aforementioned weights and bias terms.
These 80 images were entirely new and did not
occur in the training set. The algorithm Fig. 2 Input and their corresponding output images
automatically classified the 80 test images into
two groups. The first group contained 42 images
(40 furniture images and 2 face images). The 3. Conclusion & Future Work
second group contained 38 face images. From a
test sample space of 80 images 78 images were The algorithm proposed in this paper is a method of
classified correctly (97.5% success rate). unsupervised learning and classification using mirroring
neural networks and forgy’s clustering technique.
Experiment 2:
(Three different input patterns; Face, Flower and To improve the performance of Forgy’s clustering
Furniture) technique as applied to our application, we set a threshold
distance between randomly selected initial seed points.
 While training MMNN, 264 images (88 face This threshold made the randomly selected seed points
images, 88 furniture images, 88 flower images) sufficiently far apart as to make Forgy’s technique cluster
were given as input. MMNN was trained till it the input patterns perfectly.
successfully mirrored 95% of the input images.
With these weights and bias terms, the algorithm The results of the algorithm over three different input
automatically classified the 264 input images into patterns were encouraging. This can be extended to an
three groups. Group 1 contained 88 furniture architecture wherein the network will not only classify an
images (i.e. for furniture samples 100% input image pattern but it can also learn and classify any
classification was achieved). Group 2 contained new pattern which is not one of the trained patterns.
 [12] K. Eswaran, “System & Method of Identifying
References Patterns”, patents filed in IPO on 19/7/06 and 19/03/07.
Also see “New Automated Techniques for Pattern
[1] C. J. C. Burges, Geometric methods for feature Recognition & their Applications”, Altech Report, 6th July
extraction and dimensional reduction, In L. Rokach and 2006.
O.Maimon, editors, Data Mining and Knowledge
Discovery Handbook: A Complete Guide for Practitioners [13] R. C Gonzalez and R. E. Woods, Digital image
and Researchers (Kluwer Academic Publishers, 2005) processing (Englewood Cliffs, NJ: Prentice-Hall, 2002).

[2] J. Zhang, S. Z. Li and J. Wang, Manifold learning and [14] J. –S. R. Jang, C. –T. Sun & E. Mizutani, Neuro-fuzzy
applications in recognition, In Intelligent Multimedia and soft computing (Delhi, India: Pearson Education
Processing with Soft Computing, Springer-Verlag, (Singapore) Pte. Ltd., 1997 (Second Indian Reprint,
Heidelberg, 2004 2005)).

[3] Hiu Chung Law, Clustering, Dimensionality [15] Christopher M. Bishop, Neural networks for pattern
Reduction and Side Information (Michigan State recognition (United States: Oxford University Press Inc.,
University, 2006) New York, 1999).

[4] P. Baldi and K. Harnik, Neural networks and principal [16] James A. Freeman & David M. Skapura, Neural
component analysis: learning from examples without local networks (Eastern Press (Bangalore) Pvt. Ltd.: Addison-
minima, Neural Networks, 2, 1989, 53-58. Wesley Publishing Company, Inc., 1991 (First ISE reprint,
1999)).
[5] D. DeMers and G. Cottrell, Non-linear dimensionality
reduction, In Advances in Neural Information Processing [17] Anil K. Jain, Richard C. Dubes, Algorithms for
Systems 5, Morgan Kaufmann, 1993, 580-587. clustering data (Englewood Cliffs, NJ: Prentice-Hall, Inc.,
1988).
[6] Jens Nilsson, Nonlinear dimensionality reduction of
gene expression data (Sweden: KFS, 2006) [18] Earl Gose, Richard Johnsonbaugh & Steve Jost,
Pattern recognition and image analysis (New Delhi:
Prentice Hall of India, 2000)
[7] J. Nilsson, T. Fioretos, M. Hoglund and M. Fontes,
Approximate geodesic distances reveal biologically
relevant structures in microarray data, Bioinformatics,
20(6), 2004, 874-880.

[8] F. Andersson and J. Nilsson, Nonlinear dimensionality

reduction using circuit models, Proc. 14th Scandinavian
Conf. on Image Analysis, Springer Verlag, 2005.

[9] G. E. Hinton & R. R. Salakhutdinov, Reducing the

Dimensionality of Data with Neural Networks, Science,
313, July 2006, 504-507.

[10] Dasika Ratna Deepthi, Sujeet Kuchibholta, K.

Eswaran, Dimentionality reduction and reconstruction
using mirroring neural networks and object recognition
based on reduced dimension characteristic vector,
submitted to Advances in Computer Vision and
Information Technology (ACVIT-07) on Artificial
Intelligence, November 2007.

[11] D. Michie, D. J. Spiegelhalter, C.C. Taylor, Machine

learning, neural and statistical classification (February,
1994).