Artificial Neural Network (ANN)

ISSN 1925-251X [Print] ISSN 1925-2528 [Online] www.cscanada.net www.cscanada.org Progress in Applied Mathematics Vol. 3, No. 2, 2012, pp. 28-34 DOI: 10.3968/j.pam.1925252820120302.1255 On the Power Efficiency of Artificial Neural Network (ANN) and the Classical Regression Model Chukwu, A. U1,∗ ; Adepoju K. A1 1 Department of Statistics, University of Ibadan, Ibadan, Nigeria. author. Address: Department of Statistics, University of Ibadan, Ibadan, Nigeria. ∗ Corresponding Received December 2, 2011; accepted April 20, 2012 Abstract This research work presents new development in the field of natural science, where comparison is made theoretically on the efficiency of both classical regression models and that of artificial neural network models, with various transfer functions without data consideration. The results obtained based on variance estimation indicates that ANN is better which coincides with the results of Authors in the past on the efficiency of ANN over the traditional regression models. The certain conditions required for ANN efficiency over the conventional regression models were noted only that the optimal number of hidden layers and neurons needed to achieve minimum error is still open to further investigation. Key words Artificial neural network models; Transfer functions; Hidden layers; Regression models Chukwu, A. U, Adepoju K. A (2012). On the Power Efficiency of Artificial Neural Network (ANN) and the Classical Regression Model. Progress in Applied Mathematics, 3(2), 28-34. Available from: URL: http://www.cscanada.net/index.php/pam/article/view/j. pam.1925252820120302.1255 DOI: http://dx.doi.org/10.3968/j.pam.1925252820120302.1255 1. INTRODUCTION Neural networks are being widely used in many fields of study. This can be attributed to the fact that these networks are attempts to model the capabilities of human brains. Since the last decade, neural networks have been used as a theoretically sound alternative to traditional regression models. Although neural networks (NNs) originated in mathematical neurology, the rather simplified practical models currently in use have moved steadily towards the field of statistics. A number of researchers have illustrated the connection of neural networks to traditional statistical methods. For example Gallinari, Thiria, Badran and Fogelman-Soullie (1991) have presented analytical results that establish a link between discriminant analysis and multilayer perceptions (MLP) used for classification problems. Cheng and Titterington ((1994) made a detailed analysis and comparison of various neural network models with traditional statistical methods. Neural networks are being used in the areas of prediction and classification, areas where regression models and other related statistical techniques have traditionally been used. Ripley (1994) discusses the statistical aspects of neural networks and classifies neural networks as one of a class of flexible non-linear regression methods. 28 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 2. AIM AND OBJECTIVES OF THE STUDY The main purpose of this study is to compare efficiency of traditional regression model and that of artificial neural network models with an attempt to recognize the one with better discriminating and predictive power. For the realization of the above intention, the following measures are the underlined objectives: (i) To compare theoretically a ANN model with a logistic transfer function and a logistic regression model. (ii) To compare analytically the ANN model with linear transfer function and linear regression model. (iii) To estimate the means of the estimates of parameters of various classes of non linear regression and that of the ANN. (iv) To estimate and compare variances of the parameters of both ANN and that of traditional regression model. (v) To identify a better model based on the result obtained from the comparison. 3. SIGNIFICANCE OF THE STUDY Methodological disputes that arise in practice often turn on questions of the nature interpretation and justifications of methods and models that relieved on to learn from incomplete and often “observational” (or non experimental) data, the methodology of statistical inference and statistical modeling. This research work is of very high significance as it attempts to unravel the truth and settle the scores of methodological disputes in the field of mathematical statistics, Accounting and finance, Health and medicine, Engineering and Manufacturing, Marketing and general body of knowledge with regards to using classical regression and artificial neural modeling. As the performance of a particular technique in comparison to other technique depend on various factors like the size of the sample, among others, in this study, attempt is comparing both techniques (ANN and logistic models) analytically without data consideration. 4. METHODOLOGY The purpose of the study is to have the theoretical explanation of both techniques, which include their variance analysis. The classical regression model includes: (i) Log-linear model lnY = α0 + α1 x + ei (ii) Linear-log model Y = α0 + α1 lnx + ei (iii) YLR = α̂0 + α̂1 (1 + e−x )−1 Logistic regression The ANN class of models i−1 h (i) Y = α0 + α1 1 + e−(γ0 +γ1x ) + ei Logistic transfer function   γ +γ −(γ0 +γ1x ) + ei Hyperbolic transfer function (ii) Y = α0 + α1 eeγ00 +γ1x1x −e +e−(γ0 +γ1x ) YLNN = α̂0 + α̂1 γ̂0 + α̂1 γ̂1 x linear transfer function 29 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 5. COMPARATIVE ANALYSIS OF THE VARIANCES OF TRADITIONAL REGRESSION MODEL AND ANN MODELS 5.1 Linear-log Versus ANN with Logistic as Transfer Function In the section, the variances of the model under review will be compared. Let YLG represents a lines-log model and YLANN denotes variable of a ANN with logistic as transfer function. Var [YLG ] = Var [(YLG − YLANN ) + YLANN ] Recall that Var(Z+M) = Var(Z) + Var(M) + 2Cov(Z, M) Let Z = YLG − YLANN and M = YLANN Var [YLG ] = Var (YLG − YLANN ) + Var (YLANN ) + 2Cov (YLG − YLANN ) (YLANN )  −1 Since YLG = α̂0 + α̂1 ln x and YLANN = α̂0 + α̂1 1 + e−(γ0 +γ1x ) We have    −1   −1  + Var α̂0 + α̂1 1 + e−(γ0 +γ1x ) Var (YLG ) = Var α̂0 + α̂1 ln x − α̂0 − α̂1 1 + e−(γ0 +γ1x )   −1  + 2Cov α̂0 + α̂1 ln x, α̂0 + α̂1 1 + e−(γ0 +γ1x )    −1  Var (YLG ) = Var (YLANN ) + Var α̂1 ln x − α̂0 − α̂1 1 + e−(γ0 +γ1x )   −1  + 2Cov α̂0 + α̂1 , ln x − α̂0 + α̂1 1 + e−(γ0 +γ1x ) Considering the nature of independency of the variables, one expect Cov(Z,M) = 0 ⇒ E (ZM) = E (Z) · E (M)    −1  E (α̂0 + α̂1 ln x) α̂0 + α̂1 1 + e−(γ0 +γ1x )   −1  = (α0 + α1 ln x) α0 + α1 1 + e−(γ0 +γ1x ) Therefore E (α̂0 + α̂1 ln x) = α0 + α1 ln x since E (α̂0 ) = α0     −1  E (α̂1 ) = α1 and = α0 + α1 1 + e−(γ0 +γ1x ) E α̂0 + α̂1 1 + e−(γ0 +γ1x ) Cov(ZM) = 0    −1  Var (YLG ) = Var (YLANN ) + Var α̂1 ln x − 1 + e−(γ0 +γ1x )   −1 2 Var (α̂1 ) Var (YLG ) = Var (YLANN ) + ln x − 1 + e−(γ0 +γ1x ) Var (α̂1 ) is non negative function [ln x − (1 + e−(γ0 +γ1x ) )−1 ]2 is also a non-negative function for every x. Therefore, Var (YLG ) ≥ Var (YHANN ) 30 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 5.2 Linear-log versus ANN with Hyperbolic as transfer function If YLG and YLANN denote linear-log and ANN with hyperbolic as transfer function respectively, the variance of linear-log is as follows Var [YLG ] = Var [(YLG − YHANN ) + YHANN ] Var (YLG ) = Var (YLG − YHANN ) + V (YANN ) + 2Cov [(YLG − YHANN ) , YANN ] With YLG = α̂0 + α̂1 ln x and YHANN = α̂0 + α̂1 " eγ0 +γ1x − e−(γ0 +γ1x ) eγ0 +γ1x + e−(γ0 +γ1x ) # i " γ0 +γ1x " " γ0 +γ1x ## " ## e e − e−(γ0 +γ1x ) − e−(γ0 +γ1x ) Var (YLG ) = Var α̂0 + α̂1 ln x − α̂0 − α̂1 γ +γ + V α̂ + α̂ 0 1 e 0 1x + e−(γ0 +γ1x ) eγ0 +γ1x + e−(γ0 +γ1x ) " # ## " γ0 +γ1x " eγ0 +γ1x − e−(γ0 +γ1x ) e − e−(γ0 +γ1x ) , α̂ + α̂ + 2Cov α̂0 + α̂1 ln xα̂0 − α̂1 γ +γ 0 1 e 0 1x + e−(γ0 +γ1x ) eγ0 +γ1x + e−(γ0 +γ1x ) " !# " eγ0 +γ1x eγ0 +γ1x − e−(γ0 +γ1x ) Var (YLG ) = Var α̂1 ln x − γ +γ + Var α̂ + α̂ 0 1 e 0 1x + e−(γ0 +γ1x ) eγ0 +γ1x ! " √ γ0 +γ1x −(γ0 +γ1x ) e −e , α̂0 + α̂1 + 2Cov α̂1 b2 − 4ac ln x − γ +γ e 0 1x + e−(γ0 +γ1x ) !# − e−(γ0 +γ1x ) + e−(γ0 +γ1x ) !!# eγ0 +γ1x − e−(γ0 +γ1x ) eγ0 +γ1x + e−(γ0 +γ1x ) Similarly Covariance structure can be shown to be equal to zero. Then we have !!# " eγ0 +γ1x − e−(γ0 +γ1x ) Var (YLG ) = Var (YHAAN ) + Var α̂1 ln x − γ +γ e 0 1x + e−(γ0 +γ1x ) " !#2 eγ0 +γ1x − e−(γ0 +γ1x ) Var (YLG ) = Var (YHANN ) + ln x − γ +γ Var (α̂1 ) e 0 1x + e−(γ0 +γ1x ) Since Var (α̂1 ) is a non-negative function, then  Var (YLG ) ≥ Var (YHANN ) if and only if ln x − eγ0 +γ1x −e−(γ0 +γ1x ) eγ0 +γ1x +e−(γ0 +γ1x ) 2 ≥0 5.3 Log-linear Versus ANN with Logistic Transfer Function Similarly, by letting YLL and YLANN denote the Log-linear and ANN respectively, the variance of one can be obtained in terms of the other. The log-linear model is defined as follows YLL = ln Y = α̂0 + α̂1 x Var (YLL ) = Var [(YLL − YLANN ) + YLANN ]  −1 YLL − YLANN = α̂0 + α̂1 − α̂0 − α̂1 1 + e−(γ0 +γ1x )   −1  = α̂1 x − 1 + e−(γ0 +γ1x )     −1   −1  + 2Cov + Var α̂0 + α̂1 1 + e−(γ0 +γ1x ) Var (YLL ) = Var α̂1 x − 1 + e−(γ0 +γ1x )    −1   −1  , α̂0 α̂1 1 + e−(γ0 +γ1x ) α̂1 x − 1 + e−(γ0 +γ1x ) 31 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 Since covariance structure is zero, we have   −1 2 Var (α̂1 ) + Var (YLANN ) Var (YLL ) = x − 1 + e−(γ0 +γ1x ) Since Var (α̂1 ) is a non-negative function,   −1 2 is equal to or greater than zero. Var (YLL ) ≥ Var (YLANN ) iff x − 1 + e−(γ0 +γ1x ) 5.4 ANN Model with a Logistic Transfer Function Versus Logistic Regression Model The logistic regression model of the form YLR = α̂0 + α̂1 (1 + e−x )−1 is considered A perception model with a logistic transfer function is given as  −1 YLT MN = α̂0 + α̂1 1 + e−(γ0 +γix )  −1 
−1  YLOR − YLT NN = α̂1 1 + e−x − 1 + e−(γ0 +γix ) V (YLOR ) = V (YLOR − YLT NN ) + V (YLT NN ) + 2Cov [(YLOR YLT NN ) YLT NN ] Since Cov [YLOR − YLT NN , YLT NN ] = 0, we have V (YLOR ) = V (YLOR − YLT NN ) + V (YLT NN )  −1 2
−1  V (YLOR ) = V (YLT NN ) + V (α̂1 ) 1 + e−x − 1 + e−(γ0 +γix ) V (YLOR ) ≥ V (YLT NN ) iff   −1 2 V (α̂1 ) (1 + e−x )−1 − 1 + e−(γ0 +γix ) ≥ 0 and e−x , e−(γ0 +γ1x ) 5.5 ANN Model with a Linear Transfer Function Versus Univariate Linear Regression Model YL = α̂0 + α̂1 x YLNN = α̂0 + α̂1 γ̂0 + α̂1 γ̂1 x YL − YLNN = α̂1 x − α̂1 γ̂0 − α̂1 γ̂0 x = α̂1 (x − γ̂0 − γ̂0 x)   YL − YLNN = α̂1 x (1 − γ̂0 ) − γ̂0 V (YL ) = V [YL − YLNN ] + V (YLNN ) + 2Cov [YL − YLNN , YLNN ] Since Cov [YL − YLNN , YLNN ] = 0 V (YL ) = V (YLNN ) + V (α1 ) [x (1 − α̂1 ) − α̂0 ]2 V (YL ) ≥ V (YLNN ) iff   V (α1 ) x (1 − γ̂1 ) − γ̂0 2 ≥ 0 32 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 6. CERTAIN CONDITIONS From the critical analysis of the variances of the models under review, the following are noted. (i) ANN model with logistic function out-performed the linear-log if and only if   −1 2 ln x − 1 + e−(γ0 +γ1x ) >0 (ii) ANN model with Hyperbolic as transfer function will be better than that of logistic regression iff " !#2 eγ0 +γ1x − e−(γ0 +γ1x ) ln x − γ +γ >0 e 0 1x + e−(γ0 +γ1x ) (iii) ANN model with logistic as transfer function posses a better efficiency over log-linear model iff   −1 2 x − 1 + e−(γ0 +γ1x ) >0 7. SUMMARY, CONCLUSION AND RECOMMENDATION 7.1 Summary This research work has presented an analytically the efficiency of both traditional regression model of various classes and the ANN of various transfer function. Analysis of variance of each model was also conducted without data consideration. 7.2 Conclusion Based on the result obtained from the variable analysis, the following measures are hereby concluded: (i) Without data consideration, the results obtained from variance analysis confirmed the competence of artificial neural network model of various transfer functions over the conventional regression model, which coincided with the earlier work by different Authors on the competence of ANN over the traditional regression models using data across fields. 7.3 Recommendation This research work is conducted to put end to methodological dispute among the users of both classical regression and Artificial/neural network model. Therefore, from the conclusion arrived at, the following measures are recommended objectively: (i) That World body of statistics should incorporate the teaching of artificial neural network into their Curriculum. (ii) That ANN should be welcomed as very good replacement for alternative approach to the classical regression model. REFERENCES [1] Akaike H. (1973). Information Theory and an Estimation of the Maximum Likelihood Principle. Second International Symposium on Information Theory. Budapest. [2] Amemiya T. (1980). Selection of Regressors. International Economics Review, 21, 331-354 leisten. [3] Anders U. (1996). Was Neuronale Netze Wirklich Leisten. Die Bank, 3, 162-165. 33 Chukwu, A. U; Adepoju K. A/Progress in Applied Mathematics Vol.3 No.2, 2012 [4] Davidson R. Mackinnon J.G. (1993). Estimation and Inference in Econometrics. Oxford University Press. [5] Efron B., Tibshirani R. (1986). Boostrap Methods for Standard Errors Confidence Intervals and Other Measures of Statistical Accuracy. Statistical Science, 1(1), 54-77. [6] Hardle W. (1993). Applied NonParametrics Regression. Cambridge University Press. [7] Hornik K., Stinchcombe M., White H. (1989). Multilayer Feedforward Networks are Universal Approximators. Neural Networks, 2, 356-366. 34