See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/229728883

Analysis of Regression in Game Theory Approach

Article in Applied Stochastic Models in Business and Industry · October 2001

DOI: 10.1002/asmb.446

CITATIONS READS
497 10,992

2 authors:

Stan Lipovetsky Michael Conklin

independent consultant 56Stats LLC
453 PUBLICATIONS 4,526 CITATIONS 57 PUBLICATIONS 1,408 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Stan Lipovetsky on 18 March 2016.

The user has requested enhancement of the downloaded file.

APPLIED STOCHASTIC MODELS IN BUSINESS AND INDUSTRY
Appl. Stochastic Models Bus. Ind., 2001; 17:319}330 (DOI: 10.1002/asmb.446)

Analysis of regression in game theory approach

Stan Lipovetsky* R and Michael Conklin

Custom Research Inc., 8401 Golden Valley Road, Minneapolis, MN 55427, U.S.A.

SUMMARY
Working with multiple regression analysis a researcher usually wants to know a comparative importance of
predictors in the model. However, the analysis can be made di$cult because of multicollinearity among
regressors, which produces biased coe$cients and negative inputs to multiple determination from presum-
ably useful regressors. To solve this problem we apply a tool from the co-operative games theory, the
Shapley Value imputation. We demonstrate the theoretical and practical advantages of the Shapley Value
and show that it provides consistent results in the presence of multicollinearity. Copyright  2001 John
Wiley & Sons, Ltd.

KEY WORDS: co-operative games; Shapley Value; multicollinearity; regressors net e!ects

1. INTRODUCTION

In this work we consider a common problem often faced by researchers and practitioners
applying regression modelling*estimation of relative importance, or shares of in#uence that
independent variables contribute to the model. Widely used statistical measures of predictors'
importance include coe$cients of regression, their t-statistics, partial F-statistics, p-values, inputs
to multiple determination R from net e!ects of each variable. For real data the variables are
correlated, often highly correlated, and could be stochastically dependent in subsets of several
variables, or multicollinear. Multicollinearity is not a problem for prediction by regression, but it
has several detrimental e!ects in the analysis of the regressors' in#uence on the criterion variable.
Such e!ects are as follows: parameter estimates can #uctuate wildly with a negligible change in
the sample, they can have signs opposite to signs of easily understood pair-wise correlations, and
theoretically important variables can have insigni"cant coe$cients. Multicollinearity causes
a reduction in statistical power, or the ability of statistical tests to detect true di!erences in the
population. This leads to wider con"dence intervals around the coe$cients implying that they
could be incorrectly identi"ed as being insigni"cant, and the ability to determine the di!erence
between parameters is degraded [1].

* Correspondence to: Stan Lipovetsky, Custom Research Inc., 8401 Golden Valley Road, Minneapolis, MN 55427, U.S.A.
R E-mail: [email protected]

Published online 1 October 2001 Received 17 July 2000

Copyright  2001 John Wiley & Sons, Ltd. Revised 26 April 2001
320 S. LIPOVETSKY AND M. CONKLIN

Many di!erent techniques are used in applied regression analysis to evaluate the relative
importance of the predictors. One particularly useful technique is a decomposition of the
coe$cient of multiple determination into direct, indirect and net e!ects associated with each
variable [2]. The net e!ect of a predictor is a combination of the direct (as measured by its
regression coe$cient squared) and the indirect e!ects (measured by the combination of its
correlations with other variables). The net e!ects have the nice property of summing to the total
coe$cient of multiple determination R of the model. They explicitly take into account the
correlations that predictor variables have with each other. However, the net e!ect values
themselves are in#uenced by the collinear redundancy in the data so that the estimated net e!ects
can be negative, that is di$cult to interpret. On the other hand, even in presence of multicollinear-
ity, it is often desirable to keep available variables in the model and to estimate comparative
importance of their relation to the dependent variable. It makes sense because all variables do not
represent each other exactly, rather each of them plays its own speci"c role in "tting and
understanding behaviour of the dependent variable.
Another important issue is that in most real-world situations, researchers and practitioners
have neither a comprehensive theory, nor a complete control over all variables that could
describe numerous speci"c features of a complex object or process (see, for example, References
[3,4]). When a model does not incorporate some variables that correlate with included variables,
regression estimates are biased and inconsistent (see References [5, pp. 334}350; 6, pp. 24}25]).
However, in the absence of knowing what variables are necessary for the equation, we can
consider di!erent models, even limiting consideration to linear regressions. Also some kind of
averaging of the estimated characteristics over all the possible models can be applied.
Various techniques were elaborated for choosing and averaging among the regression models
to "nd the best subset [7}10]. However, given the power reduction caused by multicollinearity it
is di$cult to be sure that the best models found by such search are really superior to the many
other models that are rejected. Many techniques have also been proposed for combining
alternative models. Bagging, or bootstrap aggregation [11], boosting [12] and bundling [13] are
just some of the techniques that have been shown to improve predictions over choosing a single
model. The disadvantage in these techniques is that inference about speci"c variables is di$cult.
In bootstrapping experiments we can demonstrate that a model with typical levels of multicol-
linearity could consist of spurious coe$cients corresponding to a random distribution around
zero. In the presence of multicollinearity, the standard error of every coe$cient can be several
times more and at least not less than the coe$cient itself. This means that an actually arbitrary
decision could be made based on the analysis of the regression coe$cients or their shares of
in#uence.
Thus, we need a decision tool that can produce clear results for estimation of regressors even if
they are collinear, and when there actually are many possible models by various subsets of the
predictors. The appropriate tool we "nd in the co-operative game theory. We can think of the
particular model as a way of building coalitions among players (predictor variables) to maximize
the total value (quality of "tting). In the "eld of co-operative games a useful analysis and decision
tool is the Shapley Value imputation [14}17]. Results of the regression calculations can serve as
the initial data to "nding Shapley Value that in its turn reshapes the regression net e!ects and
coe$cients. We proved this technique to be useful during several years of solving various
complicated problems in the marketing research "eld [18}23]. Of course, some other game-
theoretical techniques can be used in statistical decisions under consideration, but we prefer the
Shapley Value imputation because it is not an heuristic procedure, it was derived as an axiomatic

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
 RELIABLE REGRESSION 321

approach, and it produces a unique solution satisfying general requirements of Nash equilibrium
[14,15].
This paper is organized as following. In Section 2 we describe how to estimate the predictors
importance, and in Section 3 we consider Shapley Value analysis. Its application to net e!ects and
a possibility to adjust regression coe$cients is considered in Section 4. Numerical example of
Shapley Value in regression analysis with bootstrapping estimation is given in Section 5. In
Section 6 we summarize results of Shapley Value application to analysing regression models.

2. PREDICTORS' CONTRIBUTION IN REGRESSION

Let us consider brie#y some properties of linear regression that will be used in further analysis.
A multiple regression model is

y "
x #
x #2#
x #e (1)

G  G  G L LG G

where y and all x's are standardized variables (centred and normalized by standard deviations),
e denotes normal random error and i is a number of observation (i"1, 2,2, N). Coe$cients of
the regression can be found in the least-squares (LS) approach that corresponds to minimization
of the objective

S"ee"(y!Xb)(y!Xb)"1!2br#bCb (2)

where y and e are the Nth-order vector-columns of the dependent variable and error term,
respectively, X is the matrix of N;n order of standardized independent variables, b is the
nth-order vector-column of LS estimates for beta-coe$cients of the regression (1), C"XX is the
nth-order matrix of sample correlations r among all pairs of variables x and x , vector-column
HI H I
r"Xy of size n de"nes sample correlations r between each x and the dependent variable y and
H H
prime denotes transposition. Minimizing (2) yields the normal system of equations

Cb"r (3)

and solution of system (3) is

b"C\r (4)

where C\ is the inverted matrix of correlations.

Minimizing deviations (2) is equivalent to maximizing of the regression quality estimated by
the coe$cient of multiple determination R (this problem in the context of canonical analysis is
discussed in References [24}26]), that could be presented as

R"1!S"2br!bCb"
b (2r!Cb) (5)

H H
H
Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
322 S. LIPOVETSKY AND M. CONKLIN

and reduced to the same solution (3) and (4). Substituting (3) into (5), we "nd the maximum of the
multiple determination as the scalar product of vectors b and r:

R"br"
b r (6)
H H
H
Items of the total R (6) de"ne the so-called net e!ects (NEF) of each jth regressor

NEF "b r (7)

H H H
Multicollinearity can change a sign of b in the multiple regression to opposite in comparison with
H
the pairwise regression y by x (pair regression coe$cient coincides with the correlation r ), then
H H
jth net e!ect becomes negative.
The values of multiple determination (6) and net e!ects (7) are widely used in practice of
regression modelling, although there are theoretically more reasonable indices for evaluation of
predictor importance. For example, the share of x could be de"ned by the square of the partial
H
correlation between y and x with "xed other x's (see References [27,28]):
H
R "(R!R )/(1!R ) (8)
WH \H \H
where R denotes multiple determination in model (1) with all n predictors including x , and
H
R denotes multiple determination in the model with n!1 predictors without x .
\H H
Another measure of relative importance of predictors is considered in References [29, 30]. It
consists in evaluation of usefulness of each regressor via the increment of multiple determination
R of the model with this particular x in the set of regressors in comparison with the model
H
without x *i.e. just the numerator in (8):
H
; "R!R (9)
H \H
Coe$cient (8) corresponds to a relative value of measure (9), and it reduces to (9) when R 1.
\H
The characteristic of usefulness (9) can be represented as the scalar product

; "(b!b*)r (10)
H
where b and r are the same vectors as in (6), and b* is a vector of the nth order with zero in the
position of the variable x and all the other elements equal to the coe$cients in the regression
H
model without x . It is possible to represent (10) as follows [30]:
H
; "b (1!R ) (11)
H H H 
where b is the jth coe$cient of regression (1) by all n variables, and R equals to the multiple
H H 
determination in the regression of x by all the others n!1 predictors. Recognizing the second
H
term in (11) as the jth variance in#ation factor [31,32] we can rewrite (11) in a simple form

; "b /C\ (12)

H H HH
Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
 RELIABLE REGRESSION 323

where in denominator we have the jth diagonal element of the inverted correlation matrix C\
(4) of all the x's.
In the presence of highly correlated variables among the x's, the matrix C (3) degenerates to the
ill-conditioned matrix, and solution (4) produces poor results both for coe$cients of regression
and for net e!ects. In the limit of very high multicollinearity among the x's solution (4) simply
does not exist, and the techniques of the ridge regression analysis can be recommended (for
example, References [9,10,32]). Let us consider now new possibilities that the Shapley Value
analysis suggests for estimation of regressors' shares in their mutual in#uence on the dependent
variable, and for evaluation of the coe$cients in the regression model.

3. SHAPLEY VALUE IMPUTATION

The Shapley Value, hereafter referred to as SV, was developed to evaluate an ordering of the
worth of players in a multiplayer co-operative game. The key to understanding its utility is that it
represents the worth of each player over all possible combinations of players. Extending this to
the problem of comparative usefulness of regressors, the SV assigns a value for each predictor
calculated over all possible combinations of predictors in regressions.
The SV approach to the problem provides a solution that is closer to the actual modeling for
any complex process or object, because it compares and averages over all possible subsets of
predictors in the model. This is an advantage of the SV solution because by comparing across all
possible models it includes the possibility of competitive in#uence of any subsets of predictors in
the analysis.
The Shapley Value is de"ned as each jth participant's input to a coalition

S "
 (M) [ (M j)! (M)] (13)

H L
+
with weights of proportions to enter into a coalition M de"ned as

 (M)"m!(n!m!1) !/n! (14)

L
In (13) and (14), n is the total number of all the participants, m is the number of participants in the
Mth coalition, and  ( ) is the characteristic function used for estimation of utility for each
coalition. By M j a set of participants which includes the jth participant is denoted, when
M means a coalition without the jth participant. In our case, the participants of the coalition
game are predictors incorporated into the regression model.
Regression output supplies us with R values, or per cent of explained variability reached for
each set of variables in regression modelling. For ease of exposition, let us use notations A, B and
C, etc. for variables x , x and x , etc., respectively. Then R , for example, de"nes the multiple
    !
determination in model (1) with the regressors A, B and C (or, the same, x , x and x ).
  
We de"ne characteristic function  (13) via these R values estimated by the results of
regression modelling. Let us construct, for the example of n"5, the characteristic function for the
variable A (where there are other variables B, C, D and E):

 (0)"0,  (A)"R ,  (AB)"R ,2,  (ABCDE)"R , (15)

   !"#
Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
324 S. LIPOVETSKY AND M. CONKLIN

where all the right-hand-side values are estimations of the multiple determination coe$cients in
di!erent regressions containing the regressor A. Substituting characteristic function (15) into SV
expression (13), we can see that each item in brackets (13) actually coincides with the usefulness
de"ned in (9). That means that SV for a predictor A is a measure of its usefulness averaged by all
the models that contain this regressor A.
Weights of imputation (14) for n"5 are

(0)"(4)"0.20, (1)"(3)"0.05, (2)"0.033 (16)

Then the SV (13) for the variable A can be written explicitly as

S "0.2(; )#0.05(; #; #; #; )
   ! " #

#0.033(; #; #; #; #; #; )
 !  "  # !" !# "#

#0.05(; #; #; #; )#0.2(; ) (17)

 !"  !# !"#  "#  !"#
where the values of usefulness (9) for our sets of regressors are as follows:

; "R , ; "R !R ,2, ; "R !R ,2

     !  ! !
; "R !R ,2, ; "R !R (18)
 !"  !" !"  !"#  !"# !"#
The items in sum (17) correspond to usefulness' margins from the variable A to all the coalitions,
and the Shapley Value imputation S corresponds to the mean margin of the variable A,

estimated by averaging over its possible participation in all coalitions. Similar formulas are used
for each of the other variables B, C, D and E, and their Shapley Values (13) de"ne margins from
each of these regressors. The total of the margins from all the variables equals the maximum value
of R in the model with all the regressors together, that is (due to (15))

L

S " (all)"R (19)
I  !"#
I
Thus, the SV are shares of the total R and they de"ne importance of each regressor in the model.
Regrouping the items in (17) with help of (18), we represent the Shapley Value imputation
formula in the following form:

S "(R !RM )/(n!1)#(RM  *!RM )/(n!2)#(RM  **!RM )/(n!3)

      
#2#(RM  **!RM  )/(n!(n!1))#R /n (20)
 L\  2
,
In the "rst item of sum (20) we see a di!erence of R for the model with one regressor A and mean

value of RM  (marked by bar over R) for all the models with just one regressor (marked by

sub-index 1). In the second item of sum (20) we see is a di!erence between mean RM  * for all the

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
 RELIABLE REGRESSION 325

models with two regressors one of which is A (marked by sub-index A* with asterisk denoting any
other variable x), and mean RM  for all the models with any two regressors (marked by sub-index

2), etc. The last item represents a share that the regressor A has in the total R of model (1) with all
the x's together.

4. SV NET EFFECTS AND COEFFICIENTS OF ADJUSTED REGRESSION

Suppose, we found SV (20) for each regressor, with their total equals multiple determination (19)
for model (1) with all the variables. These Shapley Values (19) are nothing else but estimations of
the net e!ects (7) obtained via averaging by all possible models in the co-operative game
approach. Returning from lettered indices to the index j, let us denote by SV the net e!ects
H
estimated by Shapley Value imputation for each regressor. Then in place of (6) and (7) for regular
net e!ects for multiple regression (1) we can write decomposition of the multiple determination by
the net e!ects estimated as Shapley Values

R"
S< (21)

H
H

Each item in (21) is a very stable estimate of net e!ect because Shapley Value is an average across
all possible linear models with di!erent subsets of the regressors. Thus, it is not so volatile as
regular net e!ect and is not prone to multicollinearity distortion. In comparison to net e!ects (7),
SV net e!ects (21) are always positive, so they are interpretable and suggest an easy way of
graphical (pie-charts) presentation of regressors' shares in their contribution to explanation of the
behaviour of dependent variable

(SV /R)"1 (22)

H
H

Let us consider a simple possibility of estimating which of the shares (22) are statistically
signi"cant, or which of the regressors are important in their contribution to R. Suppose we take
level of signi"cance  (for example, 5 per cent) for checking the di!erence of the coe$cient R (21)
from zero. In the assumption of independence of the items in multiple comparison and for equal
con"dential probability 1! for each of them, we can obtain the joint probability 1! as the
product of the probabilities of all the items (1!)L. Then the so-called Bonferroni con"dential
interval [33] for each item among n of them can be de"ned as

"1!(1!)L (23)

that for n'1 is always less than  level. Standard deviation  for the coe$cient of multiple
0
regression R (square root of multiple determination R) equals

 "((1!R)/(N!n!1) (24a)
0
Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
326 S. LIPOVETSKY AND M. CONKLIN

where N!n!1 is the number of degrees of freedom. Let us denote sample t-value for R as
t "R/ . On the level of signi"cance  (23) the interval estimates for R are
0 0
R$"R$t  (24b)
A 0
with two-tailed t-statistics t . Then the relative squared deviation is
A
"((R$!R)/R)"t  /R"t /t (25)
A 0 A 0
that equals the ratio of critical and empirical squared t-values. The shares of net e!ects (22) higher
than the threshold (25), SV /R' , can be considered as important (signi"cantly di!erent from
H
zero), and those SV /R(  correspond to the variables that can be neglected.
H
Let us consider adjusting regression coe$cients by SV net e!ects. Using calculated SV , we can
H
rewrite relation (7) for net e!ect as SV "a r where by a we denote unknown parameters of an
H H H H
adjusted regression with such a property: product of each coe$cient a with correlations r yields
H H
Shapley Value net e!ect. Then solving these simple equations we have:

a "SV /r . (26)
H H H
Coe$cients (26) are obtained via de"nition (7) for net e!ects. However, this de"nition (7)
corresponds to items in expression (6) for multiple determination obtained as a result of
substitution of the least-squares normal system of Equations (3) into the general objective for
R (5). To re-estimate coe$cients of the regression with the obtained Shapley Values, we suggest
to use not a simple relation SV "a r but a more complicated expression for net e!ect de"ned as
H H H
the items in sum (5) with the coe$cients a . So we can write equations for "nding coe$cients of
H
regression adjusted by Shapley Values as follows:

a (2r!Ca) "SV (27)

H H H
For already found Shapley Values, relations (27) present a system of n quadratic equations. This
system can be solved for a by a non-linear minimizing of the objective
H

F"
[SV !a (2r!Ca) ]"
(SV !2a r #a
r a ) (28)

H H H H H H H HI I
H H I
Coe$cients (26) can be used as the initial approximation a in the minimizing procedure (28).
H
Parameters a obtained in (28) are coe$cients of the adjusted regression estimated via Shapley
H
Values. These coe$cients are not prone to distortion from multicollinearity, and have interpreta-
ble signs and values.
Let us consider a convenient characteristic of the di!erence between two solutions for
regression coe$cients. It can be constructed as a ratio of the residual sum of squares for
a non-least-squares regression to the residual sum of squares of the least-squares regression (see
References [31,34]). Denote residual sum of squares in model (1) as S(a) for the solution
a obtained by (28), and S(b) for beta-coe$cients b (4). Using norms of vectors, we have

S(a)" y!Xa " (y!Xb)!X(a!b) " y!Xb !2(a!b)X(y!Xb)

#(a!b)XX(a!b)"S(b)#(a!b)C(a!b) (29)

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
 RELIABLE REGRESSION 327

In derivation (29) we use a property X(y!Xb)"0 satis"ed due to solution (3). From (29) we get
a relative index

S(a)/S(b)"1#((a!b)C(a!b))/(1!R) (30)

This expression de"nes e$ciency of an adjusted regression versus least-squares model: if value
(30) equals (1#d) per cent, than adjusted regression has d per cent bigger residual variance than
regular regression. This is the price of the trade-o! for an adjusted regression with interpretable
coe$cients and positive net e!ects.

5. NUMERICAL ESTIMATIONS

Formula (20) presents Shapley Value as a marginal input from each variable averaged by all
possible coalitions. The important feature of this formula is the presentation of subsequent inputs
of coalitions of the "rst, second, etc., levels to the total Shapley Value. If the data is available only
on the several initial stages of coalitions with one, two, and some other subsets of variables, it is
possible to use (20) for estimation of partial inputs to the total SV. Comparison of such
cumulative values for each variable A, B, C, etc., allows to evaluate stability of the SV from partial
data. This suggests an approach for reducing the computation time of the SV by limiting
computation to the number of levels where stability is achieved. We can see by (20) that each term
is constructed by calculating a mean value of combinations with the product and a mean value of
combinations without it, thus, we can estimate those means by sampling combinations. This
could be easily done and incorporated in the code whenever the number of regressors being
evaluated is above 10 (see Reference [19]).
Let us consider an example of a real project on customer satisfaction study for a telephone
customer service center. The variables are measured in scale from 1 (totally dissatis"ed, or
disagreed) to 7 (totally satis"ed, or agreed). They include: y2overall satisfaction of clients with
the company in general (dependent variable); x1*customer satisfaction with service representa-
tives; x2*service representatives are courteous; x3*they provide all the needed information;
x4*they give quick response; x5*they show care with customer problems; x6*they are
accurate in the answers; x7*they take all the necessary actions. The data gathered by 242
respondents is available. All variables are positively correlated (pair correlations are from 0.52 to
0.89). The aim of the modeling was to measure the input of the predictors in their in#uence on the
dependent variable.
In Table I some numerical results are presented. In the "rst row of this table we see that
dependent variable is correlated with all the regressors rather evenly, so each variable can be
more or less equally important in the model. However, by beta-coe$cients and their t-statistics in
the next two rows it is clear that most of the predictors are insigni"cant in the model. Two x's
have negative beta-coe$cients, and their net e!ects are negative too (next two rows in Table I),
although pair correlations (and pairwise regressions of y by each x separately) have positive signs
and similar values. Of course, it is the e!ects of multicollinearity, but knowing it does not help
much in comparison of the variables' importance. And what do we do if we want to construct
a pie-chart by shares of net e!ects with two of those negative? Multiple determination equals
0.356 and its t-statistics is t "11.52 (see Table I), so the model is good for forecasting, but it is
0
hardly useful for the analysis of the regressors.

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
328 S. LIPOVETSKY AND M. CONKLIN

Table I. Regressor coe$cients and net e!ects.

Predictor X1 X2 X3 X4 X5 X6 X7 R

Correlation r 0.543 0.450 0.545 0.433 0.511 0.546 0.505

H
*
Beta (4) 0.255 !0.022 0.177 !0.039 0.052 0.195 0.029 0.356
t-statistics 2.26 !0.25 1.28 !0.45 0.50 1.49 0.28 11.52
Net e!ect (7) 0.139 !0.010 0.096 !0.016 0.027 0.106 0.014 0.356
Share, % 38.91 !2.72 27.08 !4.72 7.51 29.90 4.04 100
SV (21) 0.068 0.034 0.064 0.031 0.048 0.065 0.046 0.356
Share, % (22) 19.10 9.55 17.97 8.71 13.47 18.28 12.92 100
a (26) 0.125 0.075 0.117 0.072 0.094 0.119 0.092 0.356
H
a (28) 0.118 0.075 0.111 0.073 0.092 0.113 0.090 0.345
H

The next rows of Table I show the Shapley Value net e!ects*they are positive, thus,
interpretable. Moreover, the net e!ects become closer to one another*it makes more sense
taking into account similar values of the correlations with the dependent variable (Table I, the
"rst row). These net e!ects can be used without any di$culties, particularly in graphical
presentation of the regressors contribution into the model. Taking con"dential probability "5
per cent, we "nd by (23) con"dential probability for net e!ects "0.73 per cent, so t "2.68.
A
The relative index  (25) equals 5.84 per cent, so all Shapley Value net e!ect shares are above this
level (see Table I), thus, the contribution from each regressor is signi"cant in the model.
Using Shapley Value, we also construct the adjusted coe$cients of regression in the approach
(26) and (28). We see (in the last two rows of Table I) that they are very close (although a (28) was
H
obtained after several dozens of iterations by the &nlminb' function in SPLUS software). Using the
"nal set of coe$cients we estimate that multiple determination for this model equals 0.345 (last
row and last column in Table I)*it is just a little less than the original value of 0.356. E$ciency
index (30) equals 1.017, i.e. residual variance of the adjusted regression is just 1.7 per cent higher
than that of a regular regression, but at the same time all re-estimated parameters of regression
become positive and interpretable.
Additionally, we use bootstrapping for evaluation of the considered characteristics*see
Table II.
Again Shapley Value, based on the averaged sets of regressors, demonstrates stable results and
robust decision rule. In Table II we represent beta-coe$cients, net e!ects, Shapley Value net
e!ects, and adjusted coe$cients of regression (the "rst row in each group). By bootstrapping with
50 replications we estimated mean values and standard deviation of all characteristics*see the
second and the third rows in each group of the results. The t-ratio of means to standard
deviations is shown in the last row of each group. By the results in Table II we see a high volatility
of beta-coe$cients and regressor net e!ects*these characteristics are biased from their means,
and the means are usually less than their standard deviations. At the same time all characteristics
obtained via Shapley Value (SV net e!ects and adjusted coe$cients of regression) are very stable.
They are almost unbiased from their mean values, and these means are several times more than
the corresponding standard errors. Therefore, when using SV net e!ects and adjusted coe$cients
of regression we can be sure of the contribution of individual regressors and in the estimation of
the regressors' in#uence on the dependent variable.

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
 RELIABLE REGRESSION 329

Table II. Bootstrapping for regression coe$cients and net e!ects.

Predictor X1 X2 X3 X4 X5 X6 X7

Beta (4) 0.255 !0.022 0.177 !0.039 0.052 0.195 0.029

Mean 0.183 0.013 0.203 !0.035 0.090 0.172 0.045
Std 0.246 0.170 0.210 0.127 0.178 0.187 0.154
t 0.74 0.08 0.97 !0.27 0.51 0.92 0.29
NEF (7) 0.139 !0.010 0.096 !0.016 0.027 0.106 0.014
Mean 0.145 0.018 0.106 !0.002 0.039 0.097 0.013
Std 0.135 0.071 0.121 0.061 0.139 0.108 0.084
t 1.07 0.25 0.88 !0.03 0.28 0.90 0.15
SV (21) 0.068 0.034 0.064 0.031 0.048 0.065 0.046
Mean 0.071 0.040 0.071 0.035 0.057 0.74 0.053
Std 0.025 0.016 0.022 0.008 0.022 0.022 0.020
t 2.84 2.50 3.23 4.38 2.59 3.36 2.65
a (28) 0.118 0.075 0.111 0.073 0.092 0.113 0.090
H
Mean 0.112 0.072 0.107 0.074 0.099 0.113 0.086
Std 0.035 0.023 0.025 0.021 0.033 0.026 0.026
t 3.20 3.13 4.28 3.52 3.00 4.35 3.31

6. SUMMARY

We considered application of a tool from the cooperative game theory, namely, Shapley Value
analysis, for evaluation of coe$cients of regression and relative usefulness of the predictors in the
model. The results are very encouraging*they show that it is possible to perform a reliable
analysis even with a high degree of multicollinearity. The results can be understood by the speci"c
structure of Shapley Value inputs as averages of the net e!ects over all possible coalitions of
regressors. While regular regression coe$cients and shares of importance are highly prone to
multicollinearity distortions, all Shapley Value characteristics, being averaged values, are very
consistent and demonstrate very stable bootstrapping output.
Due to several years of our experience actively using the described approach, the Shapley Value
technique can be successfully combined with multiple regression, signi"cantly facilitating analysis
of the regression models in numerous practical applications.

ACKNOWLEDGEMENTS

The authors wish to thank a referee for the suggestions that improved the paper.

REFERENCES

1. Mason ChH, Perreault Jr WD. Collinearity, power, and interpretation of multiple regression analysis. Journal of
Marketing Research 1991; 28:268}280.
2. Ferber R. Marketing Research. Ronald Press: New York, 1964.
3. Tishler A, Lipovetsky S. The #exible CES-GBC family of cost functions: derivation and application. ¹he Review of
Economics and Statistics 1997; LXXIX:638}646.

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
330 S. LIPOVETSKY AND M. CONKLIN

4. Tishler A, Lipovetsky S. A globally concave, monotone and #exible cost function: derivation and application. Applied
Stochastic Models in Business and Industry 2000; 16:279}296.
5. Kmenta J. Elements of Econometrics. Macmillan: New York, 1986.
6. Long JS. Regression Models for Categorical and ¸imited Dependent <ariables. SAGE Publications: London, 1997.
7. Madigan D, Raftery AE. Model selection and accounting for model uncertainty in graphical models using Occam's
window. Journal of American Statistical Association 1994; 89:1535}1546.
8. Raftery AE, Madigan D, Hoeting JA. Bayesian model averaging for linear regression model. Journal of American
Statistical Association 1997; 92:179}191.
9. Lipovetsky S, Conklin M. CRI: a collinearity resistant implement for analysis of regression problems. Proceedings
of the 31st Symposium on the Interface: Computing Science and Statistics, Schaumburg, IL, 9}12 June, 1999;
282}287.
10. Lipovetsky S, Conklin M. Multiobjective regression modi"cations for collinearity. Computers and Operations
Research, 2001, forthcoming.
11. Breiman L. Bagging predictors. Machine ¸earning 1994; 26:123}140.
12. Freund Y, Schapire RE. A decision-theoretic generalization of on-line learning and an application to boosting.
Journal of Computer and System Sciences 1995; 55:119}139.
13. Lee SS, Elder JF. Bundling heterogeneous classi"ers with advisor perceptrons. Elder Research Technical Report,
University of Idaho, 1997.
14. Myerson RB. Game ¹heory: Analysis of Con-ict. Harvard University Press: Cambridge, MA and London, England,
1991.
15. Owen G. Game ¹heory. Academic Press: New York, 1982.
16. Roth AE. The Shapley Value as a Von Neumann}Morganstern utility. Econometrica 1977; 45:657}664.
17. Roth AE. (ed.). ¹he Shapley <alue: Essays in Honor of ¸loyd S. Shapley. Cambridge University Press: Cambridge,
1988.
18. Conklin M, Lipovetsky S. Choosing product line variants: a game theory approach. Proceedings of the 30th
Symposium on the Interface: Computing Sciences and Statistics: Dimension Reduction, Computational Complexity and
Information. Minneapolis, MN, Vol. 30. 1998; 164}168.
19. Conklin M, Lipovetsky S. Modern marketing research combinatorial computations: Shapley Value versus
TURF tools. Proceedings of 1998 International S-Plus ;ser Conference, MathSoft Inc., Washington, DC, 8}9 October
1998.
20. Conklin M, Lipovetsky S. A winning tool for CPG . Marketing Research: A Magazine of Management and Applica-
tions 2000; 11:23}27.
21. Conklin M, Lipovetsky S. A new approach to choosing #avors. Proceedings of the 11th Annual Advanced Research
¹echniques Forum of the American Marketing Association, Monterey, CA, 4}7 June 2000.
22. Conklin M, Lipovetsky S. Identi"cation of key dissatis"ers in customer satisfaction research. ¹he 11th Annual
Advanced Research ¹echniques Forum of the American Marketing Association, Monterey, CA, 4}7 June 2000.
23. Conklin M, Lipovetsky S. Evaluating the importance of predictors in the presence of multicollinearity. Proceedings of
the 12th Annual Advanced Research ¹echniques Forum of the American Marketing Association, Amelia Island, FL,
24}27 June 2001.
24. Lipovetsky S, Tishler A. Linear methods in multimode data analysis for decision making. Computers and Operations
Research 1994; 21:169}183.
25. Tishler A, Lipovetsky S. Canonical correlation analysis for three data sets: a uni"ed framework with application to
management. Computers and Operations Research 1996; 23:667}679.
26. Tishler A, Lipovetsky S. Modelling and forecasting with robust canonical analysis: method and application.
Computers and Operations Research 2000; 27:218}232.
27. Goldberger A. Econometrics. Wiley: New York, 1964.
28. Timm N. Multivariate Analysis with Applications in Education and Psychology. Brooks-Cole: Monterey,
CA, 1975.
29. Darlington R. Multiple regression in psychological research and practice. Psychological Bulletin 1968;
79:161}182.
30. Harris R. A Primer of Multivariate Statistics. Academic Press: New York and London, 1975.
31. Weisberg S. Applied ¸inear Regression. Wiley: New York, 1985.
32. Marquardt D. Generalized inverses, ridge regression and biased linear estimation. ¹echnometrics 1970; 12:591}612.
33. Hsu JC. Multiple Comparisons: ¹heory and Methods. Chapman & Hall: London, 1996.
34. Ehrenberg ASC. How good is best. Journal of Royal Statistical Society, A, 1982; 145:364}366.
35. Cooley W, Lohnes P. Multivariate Data Analysis. Wiley: New York, 1971.
36. Grapentine T. Managing multicollinearity. Marketing Research 1997; 11}21.
37. Green PE, Carroll JD, DeSarbo WS. A new measure of predictor variable importance in multiple regression. Journal
of Marketing Research 1978; 20:356}360.

Copyright  2001 John Wiley & Sons, Ltd. Appl. Stochastic Models Bus. Ind. 2001; 17:319}330
View publication stats