A design strategy for logical neural networks

A zyxwvutsrqpo zyxwvu zyxwvuts DESIGN STRATEGY FOR LOGICAL NEURAL NETWORKS T.J. STONHAM AND R. AL-ALAWI DEPARTMENT OF ELECTRICAL ENGINEERING BRUNEL, UNIVERSITY OF WEST LONDON. UXBRIDGE. U.K. UB83PH ABSTRACT I1 THE DIGITAL NEURAL NETWORK The functionality of a neural network determines its generalisation properties. Multilayer networks have a restricted functionality. This is determined by the topology of the network and in particular the number of nodes and the method of interconnection. This paper presents a strategy whereby a given logical neural network structure can be evaluated in order t o determine whether it will support the functionality implied by a given training set. Neural networks can be classified as either analogue or digital depending on the type of nodal operation. The analogue node first proposed in 1943 [ l l has dominated the research although digital nodes performing logical operations offer a significant advantage in terms of training characteristics and implementation costs [21 [31. The analogue node performs the function. x e clasrl eh? g I zyxwv zyxw zyxwv E clasn ifY.X>t (1) ifY&t where is the input stimulus, 0 an internal weight vector and t a threshold function. The logical node performs the function INTRODUCTION A neural network can be regarded as providing a mapping from a fuzzy high dimensional input space to a low dimensional deterministic output. The specific characteristics of the mapping must be self-evolvingfrom a training strategy and furthermore the trained system must have generalisation properties such that the network can partition input data which was not seen during the training procedure, but is deemed to be similar, often by a human assessor, to the data on which the net was trained. Neural networks therefore offer the prospects of providing an enabling technology for many pattern recognition problems where formal analysis of the data base cannot be contemplated. (A television resolution image can be regarded as a vector of a t least 250,000 elements or variables) where f lXJ is a Boolean operator. The functionality of the analogue node has been analysed by Muroga [4]. The functionality F(c), of the logical node is given by F(c)=2" where C is the number of inputs to the node. Comparitive figures are given in Table 1. zyxwvutsrq zyxwvutsrq zyxw The logical node is universal - it can support any function of its input variables and therefore cannot generalise. In a neural network context, it will only ever recognise the training set. The analogue node has limited functionality. It will generalise but cannot support linearly inseparable functions. Multilayer systems have increased generalisation. Functionality analysis on multilayer logical networks 151 shows that the functional capacity of a network decreases dramatically with the number of layers. For example, by analysing six topologies given in Fig 1, all of which have 16 input variables and 1 output, the corresponding functionalities summarised in Table 2 can be obtained. Given a finite training set for a neural network its behaviour is dependent on 3 parameters i) the resulting nodal functions ii) the number of nodes and layers iii) the connectivity and topology of the network The nodal functionality is determined by the training strategy, whilst the size and connectivity of the system are specified by the user when initiating the network. This paper presents a design strategy which can establish whether a given network can support the functionality implied by a training set. 355 CH3051-0/91/0000-0355 $1.00 0 1991 ~ IEEE ~~ 1 zyxw zyxwv zyxwvutsrqponmlkji zyxwvutsrqponm 4 functions, the problem is increased [6] a s analogue nodes themselves have limited functionality, whereas logical nodes are universal. In this paper, an algorithm is presented which can establish whether a given network structure can support the functionality inplied by a training set. 4 I I I11 A STRATEGY TO DETERMINE CONVERGENCE IN A LOGICAL NEURAL NETWORK Consider the general two-layer network in Fig 2. A two-layer network is employed because, given suitable input connectivity and sufficient numbers of nodes on the input layer, it can implement a canonical form of the function and every function has its equivalent canonical form. Table I Comparison of functionality of analogue and logical nodes Figure 2 A 2 layer network having nodes of C, inputs on the 1st layer and C, inputs on the 2nd layer zyxwvutsrqponmlkjihgfedc where h, is the function performed by the ithunit of the 1st layer:- STRUCTURE L STRUCNAE 5 zyxwvutsrq zyxwvutsrqponmlkjih h, = A~IIJ,z..Jkl) (6) Figure 1 6 Network Topologies lunctlonahty &-,. (1. . c.1 1 2 x ]0’9”8 3 35 x 1 0 ~ 5 ‘ 1 745 x I zyxwvutsrqponmlk . Pcrcentdgc Absolute Slruciurc lunctionahty Storage s 6n36 j16 [I%,. Ll. ..,/2’1 80 6 16 76 3 IO” 2 88 x IO” 6037 x 10l6 68 0 3s 0 02 6 I I3 x IO” 60 9 7 9 x 10-s 1 x 100 100 I5 5 Applying Shannon’s expansion theorem to (4) gives Z=h, Ahp$ ,...h,-l,lJl,+l,...hc)+~A~lJs~...~,~l.OJI,+l....~cz) (8) Table 11 Functional Capacity of Networks in Fig 1. In applying a neural network to a real-world problem, a network topology is selected and a training strategy followed. The probability of the network converging to a solution is low. Should, for example, structure 4 be selected, it can support any one of 2.88 x 10,’ functions but this is an infinitesimally small proportion of the total possible number of 1 6 variable functions which can be specified (and implemented by the universal node, structure 1). If the nodes are replaced by analogue Hence the fimction Z can be expressed:- 356 zyxwvuts zyxwvutsrqp zyxwvutsrq zyxwvutsrqp zyxwvutsrq Again given appropriate assignments the necessary condition of no more than two sub-functions 4 2 different row entries) is satisfied. Interpreting equations (5) and (111, the following rule may be obtained:A given function can be implemented in a specific network structure provided that there are never more than 2 subfunctions &, and Z, performed by the net when the inputs of every node i are set in turn to all possible input states. There can be less than 2 functions because Zi, and Z, are mutually independant and the case where Z, = Zi, is allowable. IV APPLICATION OF CONVERGANCE PROBLEM STRATEGY TO Table V Partition map for node 2 A Given the training data in Table 3 which represents limited examples of a 'power of 2' detector, the above techniques can be used to determine whether the function can be supported in a 2,3 logical net shown in Fig. 2. The algorithm is demonstrated by using partition maps. In Table 4, Unit 1 is considered and outputs for all combinations of inputs xl, and xlz are entered (Rows r, to r3). The desired outputs are initially shown in bold type. The necessary condition that there should be no more than two subfunctions (Zll and Z,,) require there to be no more than 2 different function rows. This is achieved by assigning appropriate logic values (light type) to the cells not addressed by the training data. These are initially 'don't care' conditions. Finally re-ordering for unit 3, the partition map is shown in Table 6 For all units the necessary condition of no more than 2 subfunctions is satisfied and the function implied by the. training set can be set up in the selected topology Fig.2: From the final partition map one cell remains in the 'don't care' state Input (000000). This indicates that 2 functions only will satisfy both the functionality of the training data and be implementable in the chosen The partition map is then reordered about inputs q1and xZ2,carrying over the function output data and assigned states. See Table 5. Table VI Partition map for node 3 =WIT1 q network topology. This is out of a total of 264possible functions of 6 variable and 2'' functions that exist that will have the same response to the specified 15 training patterns. The technique is currently being used to design associative memories using input vector resolutions of up to 96 bits. *I, - 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0 1 0 0 -1 1 1 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 1 , 0 0 i i i 1 0 0 0 1 0 l l O l 0 1 l 0 O 1 0 .~ I I 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 0 1 1 0 0 0 1 1 0 1~ 0 ! -1 0 zyxwvutsr W.S. McCulloch and W Pitts, "A Logical Calculus of the Ideas Imminent in Nervous Activity", Bulletin of Mathematical Biophysics, Vol. 5, pp 115-133, 1943. Table 111 Training data for a 'power of two' detector W. Penny and T.J. Stonham, "Learning Algorithms for Logical Neural Networks", Proc. IEEE Conference Systems Engineering, pp 625628, Pittsburg 1990. I. Aleksander, "The Logic of Connectionist Systems" in Neural Computing Architectures, pp 133-155, North Oxford Academic Press, 1989. Table IV Partition map for node 1 357 zyxwvutsrqpon zyxwvutsrq [41 S. Muroga, "Enumerationof Threshold Functions of Eight Variables",IEEE Trans. Computers, Vol. C-19, No.9, pp 818-825, (1970). [51 R. Al-Alawi and T.J. Stonham, "TheFunctionality of Multilayer Boolean Neural Networks", Electronics Letters, Vo1.25, No 10, pp 657-658, 1989. [61 R. Al-Alawi "The Functionality, Training and Topological Constraints of Digital Neural Networks",Ph.D. Thesis, Brunel University, 1990. 358