Expert Systems with Applications 42 (2015) 6717–6727

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier.com/locate/eswa

Development, test and comparison of two Multiple Criteria Decision

Analysis (MCDA) models: A case of healthcare infrastructure location
Benjamin Dehe ⇑, David Bamford 1
University of Huddersfield – The Business School, University of Huddersfield, Queensgate, Huddersfield HD1 3DH, United Kingdom

a r t i c l e i n f o a b s t r a c t

Article history: When planning a new development, location decisions have always been a major issue. This paper exam-
Available online 30 April 2015 ines and compares two modelling methods used to inform a healthcare infrastructure location decision.
Two Multiple Criteria Decision Analysis (MCDA) models were developed to support the optimisation of
Keywords: this decision-making process, within a National Health Service (NHS) organisation, in the UK. The pro-
Multiple Criteria Decision Analysis (MCDA) posed model structure is based on seven criteria (environment and safety, size, total cost, accessibility,
Analytical Hierarchy Process (AHP) design, risks and population profile) and 28 sub-criteria. First, Evidential Reasoning (ER) was used to solve
Evidential Reasoning (ER)
the model, then, the processes and results were compared with the Analytical Hierarchy Process (AHP). It
Location decision
was established that using ER or AHP led to the same solutions. However, the scores between the alter-
natives were significantly different; which impacted the stakeholders’ decision-making. As the processes
differ according to the model selected, ER or AHP, it is relevant to establish the practical and managerial
implications for selecting one model or the other and providing evidence of which models best fit this
specific environment. To achieve an optimum operational decision it is argued, in this study, that the
most transparent and robust framework is achieved by merging ER process with the pair-wise compar-
ison, an element of AHP. This paper makes a defined contribution by developing and examining the use of
MCDA models, to rationalise new healthcare infrastructure location, with the proposed model to be used
for future decision. Moreover, very few studies comparing different MCDA techniques were found, this
study results enable practitioners to consider even further the modelling characteristics to ensure the
development of a reliable framework, even if this means applying a hybrid approach.
Ó 2015 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

1. Introduction Bamford, & Moxham, 2011; Feldmann & Olhager, 2013). Formal
decision-making models and intelligent systems can be used to
The National Health Service (NHS) in the UK has a requirement support the decision-making processes and it is suggested, in this
to develop reliable, robust and transparent operational paper, that MCDA models are appropriate techniques to resolve
decision-making processes; and, when appropriate, to include the the location issue, especially when considering the NHS environ-
local population within the process (Department of Health (DoH), ment and the objectives set by the stakeholders: robustness and
2010). Within the planning of new healthcare infrastructure such transparency. Healthcare organisations are becoming increasingly
as hospitals, clinics or healthcare centres, a key operational deci- accountable to the local population (DoH, 2010) and modelling
sion is the choice of the location. The infrastructure site decision techniques such as location-allocation models and Geographic
influences and shapes the overall healthcare network within an Information Systems (GIS) have been promoted to optimise site
area. It is also a delicate decision for the local population who locations (Rosero-Bixby, 2004). However, it is suggested that these
are traditionally consulted via a mailed questionnaire. However, types of modelling technique do not entirely satisfy the trans-
this method is not the most rational or transparent way for opti- parency and inclusivity objectives of the NHS; it is difficult to sim-
mising the location and other methods are needed to improve ulate and model the more qualitative criteria and inputs gathered
the site location decision-making process (Dehe, Bamford, from the ‘Voice of the Customer’ (Bamford & Forrester, 2010).
This paper reports on the empirical differences perceived
between the operational application of ER and AHP, when applied
⇑ Corresponding author. Tel.: +44 1484 473132.
to the healthcare site selection, as well as identifying some of the
E-mail addresses: [email protected] (B. Dehe), [email protected]
(D. Bamford). managerial and practical implications for the decision-makers.
1
Tel.: +44 01484 472278. The research make a practical contribution rather than a purely

http://dx.doi.org/10.1016/j.eswa.2015.04.059
0957-4174/Ó 2015 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
6718 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727

theoretical one, hence the technical and mathematical background decision-maker will select the optimal alternative, according to
will not be considered, and rather it is the socio-technical aspect of the identified set of criteria, denoted by C = {C1, C2, . . ., Cn}. Also,
the models implementation that are the focus in this paper (Singh an interval weight vector, denoted by X = (x1, x1, . . ., xn), will
& Wood-Harper, 2011). To direct the study two specific research be given, where xj = [xL j, xR j] (j e N = {1, 2, . . ., n}) and 0 6 xL
questions (RQ) were developed: RQ1: Are the operational processes j 6 xR j 6 1. This represents the relative importance of each
and outcomes significantly different according to the MCDA model criterion.
implemented: ER or AHP; RQ2: According to the decision-makers, what Tavana and Sodenkamp (2010) explained that MCDA enables
is the most reliable and appropriate modelling techniques to provide a the stakeholders to create a framework to exchange their informa-
rational, inclusive and transparent operational solution? tion and knowledge while exploring their value systems through
the weighting and scoring mechanisms. Furthermore, Ormerod
(2010) suggested that different frameworks and mechanisms
2. Literature – MCDA in context and practise
inform the stakeholders’ beliefs about the relationship between
the options and the outcomes. While, Belton and Stewart (2002)
2.1. Modelling theory and roles of models
explained the myths of MCDA, emphasising that there are no right
answers due to the subjectivity of the inputs. The subjectivity is
Various model definitions have been discussed over the years,
inherent to the choice of criteria, the weighting and the assess-
for instance Ackoff and Sasieni (1968) defined a model as a repre-
ment. Therefore, according to the framework selected, the subjec-
sentation of the reality. However, Pidd (2003) explained this sim-
tivity might be different, even when the common final aim leans
plistic definition did not address the fact that people have
towards a transparent, informed and sensitive decision.
different worldview and perception of the reality, as well as that
Xu and Yang (2001) wrote that there are many methods avail-
a model can never be entirely complete and accurate. Therefore,
able for solving MCDA problems. Amongst the most theoretical
Pidd (2003) preferred defining a model as an external and explicit
and empirically sound techniques, there are ER and AHP (Guo,
representation part of a reality as seen by the decision-makers and
Yang, Chin, & Wang, 2007; Saaty, 1980; Saaty & Vargas, 2001; Xu
modellers. This means, models are an approximation of the reality
& Yang, 2001). Other methods which can be found are: TOPSIS,
and that according to the specific model used to look at a real
VIKOR, ELECTRE, and UTASTAR (De Moraes, Garcia, Ensslin, Da
world problem the processes and outcomes might be different. In
Conceição, & De Carvalho, 2010; Grigoroudis, Orfanoudaki, &
this paper, it is intended to establish, whether by looking at the
Zopounidis, 2012; Liao & Xu, 2013; Santos et al., 2002; Yang,
same real world problem – the site location for a new healthcare
2001). The literature reports several applications of MCDA. Some
centre – throughout two different MCDA models: ER and AHP,
applications are associated with a sector of activity; manufactur-
the processes and the outcomes are different or not, and whether
ing, healthcare or construction. Other applications are related to
one is more appropriate than the other, in this particular setting.
a specific type of decision. Is one technique more appropriate than
According to Box and Draper (1987, p.424) ‘‘Essentially, all
another, in a specific context?
models are wrong, but some are more useful than others’’. Hence,
models have different characteristics, and one may want to
2.3. MCDA in healthcare
identify the most appropriate model to use for solving a specific
problem in an identified environment. To identify the most
The literature shows a worldwide use of MDCA in the health-
appropriate model, one may want to look at: (i) the robustness
care sector. Its use and applications remain varied, to support both
and the representativeness of the results generated, which are
clinical (Miot, Wagner, Khoury, Rindress, & Goetghebeur, 2012;
measures and perceptions of accuracy; and (ii) the repeatability
Tony et al., 2011; Youngkong, Teerawattananon, Tantivess, &
and the reproducibility, associated with the consistency and
Baltussen, 2012) and managerial (De Moraes et al., 2010; Dey,
transparency, which are measures of precision of the model and
Hariharan, & Clegg, 2006; Grigoroudis et al., 2012; Kornfeld &
its process (Breyfogle, 2003). For instance, will the models allow
Kara, 2011) decision-making during complex problem solving.
the decision-makers and participants to be consistent at a different
Büyüközkan, Çifçi, and Güleryüz (2011) showed how a fuzzy
time? And, how representative of the perceived reality are the
AHP model supported the evaluation and the perception of the ser-
results? Moreover, the consistency, transparency and the facilita-
vice quality in a Turkish hospital; they determined the factors and
tion or practicality must be taken into account when implementing
criteria that hospitals should focus on to optimise their service
a model. Fig. 1 illustrates an assessment framework to determine
quality.
what model would lead towards the optimum solution.
2.4. MCDA in site selection
2.2. Multiple Criteria Decision Analysis MCDA
Site selection is a critically strategic decision as it could poten-
Ram, Montibeller, and Morton (2011) and Golmohammadi and tially make or break a business, independently of the industry
Mellat-Parast (2012) stated that when strategic options are being because location decisions involve long term resource commit-
evaluated for instance in supplier or location selections, MCDA is ment and have significant impacts on the operations strategy and
the suitable approach to handle conflicting and both qualitative the key operations performance indicators such as cost, flexibility,
and quantitative objectives. MCDA provides a framework to aid speed and dependability (Ertuğrul & Karakasßoğlu, 2008; Salles,
with making complex decision by creating a platform where all 2007; Yang & Lee, 1997). The literature is very diverse regarding
stakeholders can share information, in order to develop a consen- site selection or facility location, however, as for complex pro-
sus or find a compromise. The sequence of tasks becomes logical, cesses it requires rationalised decision-making, often subject to
first by structuring the problem; second, by modelling the criteria uncertainty (Hodgett, Martin, Montague, & Talford, 2013).There
preference and their importance; then, by aggregating the alterna- are numerous MCDA applications in the site selection problem;
tives evaluation; and finally allowing the decision to be made this is one of the first problems studied in the MCDA literature
(Saaty, 1980; Santos, Belton, & Howick, 2002; Yang, 2001). for instance with the work of Keeney and Raiffa (1993) where they
Ren, Gao, and Bian (2013, p. 3) pointed out that, from a explore airport location. Furthermore, several papers have been
mathematical perspective, a MCDA model is defined by a set published regarding landfill site selection considering the eco-
of alternatives, denoted by A = {a1, a2, . . ., am}, from which a nomic, ecological and environmental issues associated with the
 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727 6719

Processes (precision) Results (accuracy)

Consistency (repeatability)
Robustness (sensitivity)

Transparency (reproducibility)

Representativeness
Facilitation & Access

Optimum solution (rational)

Fig. 1. A compiled framework for MCDA comparison (adapted from Breyfogle, 2003).

decision, often the MCDA models were associated with the use of 2.5. Evidential Reasoning ER and its application
Geographic Information System (Fatta, Saravanos, & Loizidou,
1998; Gorsevski, Donevska, Mitrovski, & Frizado, 2012; Guiqin, Li, The ER approach is amongst the latest MCDA technique, devel-
Guoxue, & Lijun, 2009; Onut & Soner, 2007). Other papers, less oped to handle uncertainty and randomness. Xu (2011), Liu, Bian,
specific, presented MCDA models for other infrastructure location, Lin, Dong, and Xu (2011) and Wang and Elhag (2008) stated that
for instance, Chen (2006) explained the complexity in the conven- the ER was first developed by Yang and Singh (1994) to solve mul-
tion site selection and suggested AHP as a method to support the tiple criteria decision problems taking into account qualitative and
decision by making sense of the multitude of variables encom- quantitative attributes as well as the inherent uncertainty, by com-
passed; they demonstrated the use of their five criteria and 17 bining the Dempster–Shafer (D–S) theory (Shafer, 1976) with a dis-
sub-criteria model within a site selection in Taiwan. Ertuğrul and tributed modelling framework. The difference with the other more
Karakasßoğlu (2008) chose to demonstrate the MCDA application traditional MCDA models is that ER uses an extended decision
to optimise the facility location of a textile organisation in Turkey. matrix in which each attribute of an alternative is described by a
However, case studies investigating the healthcare site selec- distributed assessment using a belief structure (Liu et al., 2011;
tion problem, using MCDA, are limited. It was identified that, in Xu & Yang, 2001; Xu & Yang, 2003). For instance the distributed
their paper, only Vahidnia, Alesheikh, and Alimohammadi (2009) assessment results of the sub-criteria regeneration impact for
developed an AHP model to find the best site for a new hospital. alternative A can be {(Best, 33%), (Good, 33%), (Average, 33%),
Their model has five criteria: distance from arterial routes, travel (Poor, 0%), (Worst, 0%)}, whereas for B it can be {(Best, 0%),
time, contamination, land cost and population density. (Good, 50%), (Average, 50%), (Poor, 0%), (Worst, 0%)}. ER uses a
Additionally, very few studies comparing results between dif- Simple Additive Weighting as scoring methods to calculate the
ferent models were found. Only Ertuğrul and Karakasßoğlu (2008) overall score of an alternative as the weighted sum of the attribute
compared the AHP method with TOPSIS, and Zhang, Wang, Sun, scores or utilities (Xu, 2011; Xu & Yang, 2001; Xu & Yang, 2003;
and Wang (2011) who compare their methods with two different Yang, 2001). This process can be facilitated by the Intelligent
authors Beynon (2002) and Hua, Gong, and Xu (2008) methods, Decision Systems (IDS) software developed and tested by Yang
which lead them to observe contradictory results. This is noted, and his collaborators between 1998 and 2006 (Wang & Elhag,
despite the common recognition of the compensation involved in 2008; Xu, 2011; Yang 2007). Xu and Yang (2001), Xu and Yang
any aggregation models and the subjectivity incurred in a frame- (2003) also clearly explained that by using a distributed assess-
work. For example, Grigoroudis et al. (2012) explained that results ment technique decision-makers can capture the diverse type of
are affected by both the model reference sets and by the uncertainties and model subjective judgement. Hence, they clari-
decision-makers consistency and interpretation of the model fied that ER approach uses the Dempster–Shafer (D–S) theory as
mechanisms. In their paper, however, Ertuğrul and Karakasßoğlu aggregation mechanisms; Bi, Guan, and Bell (2008) explained that
(2008) contrasted two modelling techniques: AHP and TOPSIS, the D–S theory is an appropriate and suitable approach for dealing
and concluded that, despite that both AHP and TOPSIS having their with uncertainty and imprecision. It provides a coherent frame-
own characteristics, the ranking of the three alternatives was the work to cope with the lack of evidence and discard the insufficient
same. They demonstrated that, when the decision-makers were reasoning principle. ER enables to translate the relationship
consistent, both methods could be appropriate, even if they recog- between the object and the degree of goodness or badness of its
nised that decision-makers should choose the methods fitting the sub-criteria, which is measured by both ‘‘the degree to which that
problems and the situation. However, the study did not address sub-criteria is important to the object and the degree to which the
the process differences and preferences of the decision-makers in sub-criteria belongs to the good (or bad) category’’ (Xu & Yang,
great depth, and this is the reason why it will be attempted here 2001, p. 8). Furthermore, it allows decision-makers preferences
in comparing two methods: AHP and ER, and evaluate the manage- to be aggregated in a structured and rigorously without accepting
rial consequences of choosing one or the other. the linearity assumption (Chin, Wang, Yang, & Poon, 2009). This
There are many methods available for solving MCDA problems, makes to some extent ER different from other MCDA approach
however, some methods were criticised for lacking theoretical such as AHP or TOPSIS (Ertuğrul & Karakasßoğlu, 2008; Seçme,
soundness and empirical evidence (Xu & Yang, 2003). Bayrakdaroglu, & Kahraman, 2009; Zhang et al., 2011).
Nevertheless, both ER and AHP are both theoretical and empirical Furthermore, ER has been applied in different sectors and indus-
grounded (Saaty, 1980; Saaty & Vargas, 2001; Xu & Yang, 2003). tries construction, security, transport, and IT, with diverse applica-
Therefore, it was useful to test whether or not by implementing tions such as supplier selection, performance measurement,
this two different MCDA models the optimisation of the assessment, risk management, new product development, and data
decision-making process of the site selection was going to be aggregation (Chin, Xu, Yang, & Lam, 2008; Liu et al., 2011; Wang &
affected. Elhag, 2008; Wang, Yang, & Xu, 2006; Yang, Wang, Bonsall, & Fang,
6720 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727

2009; Zhang, Deng, Wei, & Deng, 2012). However, not many publi- Table 1
cations were found in the Healthcare sector, only Tang et al. (2012) Fundamental pair-wise comparison scale (ADAPTED from Saaty & Vargas, 2001).

used ER in order to assess and analysed the risks in an NHS Intensity of Definition Explanations
organisation. importance
1 Equally Two activities contribute equally to the
2.6. Analytical Hierarchy Process AHP and its application preferred objective
3 Moderately Experience and judgment slightly or
preferred moderately favour one activity
AHP is a general theory of measurement; it is an effective 5 Strongly Experience and judgment strongly favour one
approach to handling decision-making and certainly the most preferred activity
popular MCDA methodology (Bozbura, Beskese, & Kahraman, 7 Very Experience and judgment very strongly
2007; Chen & Huang, 2007; Jakhar & Barua, 2013; Kang & Lee, strongly favour one activity
preferred
2007; Partovi, 2007). It was developed by Saaty in the 1980’s for
9 Extremely The evidence favouring one activity over
resolving unstructured problems in any disciples or business areas preferred another is of the highest possible order of
(Wu, Lee, Tah, & Aouad, 2007). Saaty and Vargas (2001) explained affirmation
that it was designed to cope with the uncertainty, and to optimise
the evaluation the available alternatives. By undertaking pair-wise
comparison judgments and aggregating the scores, a ranking of weighting; whereas AHP focuses on the aggregate criteria and uses
alternative is developed. The advantage resides in the fact that it pair-wise comparison, as Fig. 2 illustrates. These differences influ-
allows inconsistency to be assessed but simultaneously improving ence the subjectivity within the modelling process, and may lead
the consistency of the decision (Saaty & Vargas, 2001). to practical and managerial implications.
The logic behind AHP is in building a three level hierarchy Both ER and AHP use equivalent hierarchical structures there-
model with the goal, the criteria and the alternatives to be fore one can follow the same process with the identified group of
assessed. Cousins, Lamming, Lawson, and Squire (2008) explained stakeholders to satisfy the accountability objectives by engaging
that to express the relative importance of one criterion over with the stakeholders. However, the differences will take place in
another AHP uses the pair-wise comparison method. The the weighting and scoring phases. The assessment of alternatives
scale can be selected to accommodate the needs of the follows different type of mechanisms. Also, one can wonder
decision-makers as Tiwari and Banerjee (2010) demonstrated. whether, by using one or the other method, it will influence the
We have used the fundamental five levels scale to offer a wide results interpretation.
range of possibilities as Table 1 shows. This fundamental scale
was defined by Saaty and Vargas (2001), and has been theoretically
justified and its effectiveness validated. This scale is used with the 3. Methodology
reciprocals values when the relationship between two activities is
inverted. This research used an embedded single-case study in order to
Belton and Gear (1983), Chin et al. (2008) and Taround and develop, test and compare the two MCDA models (Yin, 2009).
Yang (2013, p. 1222) recognised the excellence of the AHP The research is designed around a series of eight workshops; it
approach. However, they also explained that it has a number of was adopted to gather rich data in order to develop an understand-
limitations. Firstly, as AHP treats criteria weights and scores in ing of the use of Evidential Reasoning and Analytical Hierarchy
the same way, applying pair-wise comparison, which, they Process, as well as understanding the socio-technical processes
believed, leads to ranking reversal problems, moreover, one needs informing the final location decision. The researchers had a direct
to be concerned with the number of judgments required to derive access to the organisation over an extended time period: two years,
relative priorities, which can create inconsistency issues (Mustafa and this experiment lasted about 6 months where eight workshops
& Al-Bahar, 1991). Furthermore, AHP lacks the capacity to cope were organised and attended by the different groups of stakehold-
with uncertainty. Finally, the introduction of new criteria, or alter- ers. There are shortfalls associated with a single case study; these
natives, will require the modification of the whole model (Belton & are often related to the external validity and the generalisation
Gear, 1983; Belton & Stewart, 2002). The limitations of AHP do not (Gay & Bamford, 2007). Nevertheless, it remains a popular research
undermine its usefulness, but have stimulated researchers to methods and many important operational concepts have been
develop alternative techniques, such as ER (Taround & Yang, developed by using a case study approach (Voss, Tsikriktsis, &
2013, p. 1222). Frohlich 2002). The case study is therefore a valid method to con-
To solve the developed model, a software called ‘Make it tribute to the body of knowledge by developing an understanding
Rational’ (MiR) was used (http://makeitrational.com/). This of the causal mechanisms of a particular phenomenon (Yin, 2009).
allowed comparing ‘like-for-life’ modelling techniques; it was felt
that by not implementing both models via a software interface 3.1. The case study
the results could have been compromised or at least biased toward
one or another model. The model structure and hierarchy was developed in collabora-
tion with an NHS organisation in the UK. This healthcare organisa-
2.7. General differences and similarities tion commissions the full range of clinical services throughout 58
community-based health services across 100 sites, within the 30
Evidential Reasoning (ER) and Analytical Hierarchy Process Local Authority wards, each of them with a population of about
(AHP) were the two approaches presented and selected, because 17,000 inhabitants. The total catchment area represents approxi-
it was considered that AHP was the most popular approach, and mately 500,000 people, living in both urban and rural area. The
ER was an excellent complementary approach. However, the organisation has set particular priorities: for instance, the reduc-
researcher recognises and acknowledges the other powerful tech- tion of health inequalities, the improvement of the clinical quality
niques, such as VIKOR and TOPSIS. ER and AHP major practical dif- and safety, as well as increasing the patient experience through
ferences reside in the assessment level and in the assessment enhance efficiency and effectiveness performances. This can be
technique. ER focuses on the sub-criteria level of the model, uses accommodated by a move toward more community-based
a degree of belief for the assessment, and the Likert scale for the care provision, as specified within Lord Darzi’s Report (2007).
 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727 6721

Fig. 2. Differences between ER and AHP.

However, to achieve these objectives the organisation has to weightings (c.f. Fig. 3). Therefore, in total six facilitated workshops,
undertake extensive infrastructure development over the next dec- which involved a total of 45 stakeholders, enable the authors to
ade, but lacks mechanisms, systems or procedures for overseeing collect qualitative and quantitative data to be able to compile
their planning and ensuring that the organisation’s future strategic and solve both final ER and AHP models as the findings section will
needs are achieved. present, an extra two workshop were held to compare the models.

3.2. The research rationale and process 3.4. Reflection on the approach using semi-structured interviews and
group discussions
The objective was to optimise healthcare site selection
decision-making processes within a National Health Service In order to answer the second research question, and identify
(NHS) organisation in the North of England. It was also aimed at the most reliable and appropriate modelling techniques, it was rel-
establishing the most reliable and appropriate modelling tech- evant to gather information directly from the decision-makers,
niques to tend toward a rational, inclusive and transparent solu- who, in the future, will own the process. The authors were keen
tion, which fits with the key objectives and indicators of the to collect qualitative information regarding both processes ER
organisation. For these reasons the two research questions men- and AHP during and after the experiments. The rational was to
tioned, in the introduction, were developed. In the first instance, understand what are the models’ characteristics that the
the MCDA model was developed with a wide range of carefully decision-makers require to optimise the process. Therefore,
selected stakeholders, and was subsequently validated. The assess- semi-structured interviews and group discussions were organised,
ment was conducted by the team of experts to reduce the informa- during the last two workshops, around the following questions:
tion asymmetry and be as informed and sensitive as possible. The ‘what did think of ER?’; ‘How was the APH process?’; ‘How did you find
AHP assessment was undertaken concomitantly using Make it the pair-wise comparison?’; Between ER and AHP, which one did you
Rational (MiR) software in order to answer the stated research prefer?’; Building the ER model, was it cumbersome? ‘Did you feel that
questions and be able to compare ER and AHP as objectively as pos- your opinion was well integrated within the final AHP model?’; Overall
sible. This process was undertaken as an experiment. was ER and AHP a complex process to go through?’. Moreover, during
both processes ER and AHP, the authors made observations
3.3. Data collection to build the models regarding the interactions and the dynamic between the
decision-makers. It was important to perceive how the stakehold-
A substantial data set from public consultations was considered ers and decision-makers responded during the processes. This
with both qualitative and quantitative information that supported information was recorded to support the discussion in this paper.
the design of the final MCDA model as Fig. 3 shows. Furthermore, This paper reports an experiment of applying two different
data were gathered from four specific workshops to compile the MCDA techniques: ER and AHP, to optimise the healthcare site
final model and solve it using ER and two extra workshops were selection. It had for objectives to establish (i) whether there is a dif-
set up for solving the AHP model. These workshops were organised ference between the two models processes and results and (ii)
to capture both the ‘voice of the local community’ and the ‘experts identify what would be the optimum process within this environ-
judgment’ such as: Estates, Primary care, Planners, Clinicians and ment. These objectives were achieved by combining both quantita-
other key decision-makers from the senior management. These tive and qualitative data in order to develop and solved the models
sets of data were used to identify and agree the seven criteria: and qualitative data in order to gather the perception of the
environment and safety, size, total cost, accessibility, design, risks, decision-makers and establish the most reliable and appropriate
and population profile; and the 28 sub-criteria and their associated modelling techniques.
6722 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727

Neutral location – QL - 0.1

Environment & safety Vandal proof – QL - 0.2

(weight mean = 4)
Open location – QL – 0.2

Regeneration potential – QL – 0.5

Parking spaces – Q - 0.3

Size (weight mean = Clinical space – Q - 0.4

6)
Admin space – Q - 0.1

Expansion capacity – Q - 0.2

Construction costs – Q - 0.2

Total cost (weight Land costs – Q - 0.3

mean = 10)
Rates and taxes – Q - 0.1

Value for money – Q - 0.4

Location of
New healthcare
centre Road and traffic – QL - 0.3

Public transport – QL - 0.4

Accessibility (weight
mean = 9) Pedestrian and disabled access – QL - 0.2

Commuting Affordability– QL - 0.1

Flexibility in design – QL - 0.6

Number of storeys – Q - 0.1

Design (weight mean Fits in with the surrounding area – QL - 0.1

= 6)
Potential use of renewable energy – QL - 0.1

Pharmacy required – QL - 0.1

Land risk – QL - 0.3

Construction risk – QL - 0.2

Risks (weight mean =
6) Service disruption risk – QL - 0.2

Delivery time and speed risk – QL - 0.3

Demographic – QL – 0.5

Population profile
Geographic – QL - 0.2
(weight mean = 4)
Epidemiologic – QL - 0.3

Fig. 3. ER model structure-criteria and sub-criteria weightings.

4. Findings – comparing ER and AHP models noted QL in Fig. 3. Once the weightings were identified and vali-
dated they were normalised, which are used in this analysis fur-
4.1. The ER model ther. The normalisation process helped to compare the results
generated by the two models.
From the facilitated workshops, which involved a total of 45
stakeholders, it was possible to compile the final model, with the
associated weightings, which were the rounded average of each 4.2. The AHP model
individual score, as shown in Fig. 3 below. This model is composed
of seven criteria and 28 sub-criteria. As mentioned previously, in The AHP model has traditionally three levels: the goal, the cri-
the ER approach the assessment takes place at the sub-criteria teria and the alternatives as illustrated in Fig. 4. The set of the
level, therefore, it was required to identify whether the seven criteria is the common structure, as it is independent of
sub-criteria are evaluated quantitatively, noted Q, or qualitatively, the selected modelling techniques.
 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727 6723

Env & safety

Size

Total cost
Location A

Location of New
Accessibility
Health Centre

Location B
Design

Risks

Population profile

Fig. 4. AHP model structure.

In the AHP model, the weightings of criteria are pair-wise com- the assessment it was established that when stakeholders use a
pared and the results are shown in Table 2. This was established by Likert scale from 1 to 10 it is likely that little difference can be per-
the group of decision-makers, and is consistent with the weighting ceived between the criteria but that the uniformity is respected
determined in the ER model. and it is highly transparent. However, using pair-wise comparison,
While ER uses the Likert scale from 1 to 10 (10 being the high- the difference is amplified, but there is room for inconsistency
est) to identify the weighting of each criteria, AHP uses pair-wise when criteria are being compared against other criteria, and stake-
comparison, modelling (7  7) 49 relationships. This means for holders might have a less transparent perception of the weighting
instance that: Environment and safety is equally preferred to phase. The second relevant point to mention is that in this case
Population profile; then a value of 1 is captured; Size is strongly using ER or AHP led to the same ranking, which is positive, and
non-preferred to Total cost; then a value of 1/5 is captured. This translate that the decision-makers were consistent in their
is also automatically recorded in the reciprocal cell as Total cost approach, and gave confidence to proceed with the comparison.
is strongly preferred to Size where a value 5 is registered. Once this
is averaged and normalised, the weightings are obtained. 4.4. ER and AHP the assessments
It is important to note that AHP model logic works at the high-
est level (e.g.: at the criteria level). The sub-criteria are also taken The next step was the assessment of alternatives: A and B in this
into account by forming the overall definition of each criterion. It case; which allowed the ranking of the alternatives. With ER the
would have been impractical to facilitate the pair-wise comparison degree of belief for each sub-criteria is established independently,
process to identify the weights and the assessments for the 28 whereas AHP remains at the criteria level and assessed the alterna-
sub-criteria and the decision-makers would not have bought into tive against each other using the pair-wise comparison. Table 4
the process; a criticism of AHP. compiled the results from both assessments at the criteria level;
note that even if the results provided shows that location A is sig-
4.3. ER and AHP the weightings nificantly the preferred option in 3 criteria, and location B in 2 cri-
teria, and that overall A is the preferred option, the quantification
In Table 3, the weighting assigned for both models are pre- differences which is the most paramount indicator for the final
sented. Note that the weighting range is different whether ER or decision is substantially different according the selected modelling
AHP was the selected framework to solve the problem. With ER approach. Therefore, for this reason a statistical test: 2 Proportion
the weighting are included into a range from 8.90% to 22.22%; Test was undertaken. Hypothesis testing: is there any significant
whereas with AHP the range is wider from 3.53% to 38.89%. From difference between the results scoring range of ER and AHP? H1:
6724 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727

Table 2
AHP Pair-wise comparison table for the criteria weightings.

Table 3
criterion on its own, but rather as an aspect linked to each of the
Criteria weightings and rankings comparison.
five criteria transversal to both the process and results. This was
Criteria ER AHP achieved by the reflective work that the authors have undertaken
Weight (%) Rank Weight (%) Rank with the decision-makers during the last two workshops.
Environment & safety 8.90 6 3.53 6
Size 13.33 3 8.44 3
5.1. Processes and their precisions
Total Cost 22.22 1 38.89 1
Accessibility 20.00 2 28.75 2
Design 13.33 3 8.44 3 The processes for weighting and assessing the criteria had to be
Risks 13.33 3 8.44 3 consistent, repeatable and transparent, because they were used by
Population Profile 8.90 6 3.53 6 the group of decision-makers at different times. To test this, parts
of the processes were selected and tested by asking the stakehold-
ers to re-weight and re-assess criteria and alternatives, in order to
proportion [ER(a b) – AHP(a b)]. P value < 0.05 (with a = 0.05), establish to what extent the same weightings and assessments
hence, we can be 95% confident that there is a difference between could be reproduced, and to test the capabilities of the measure-
the results from ER and AHP. ment models (Breyfogle, 2003). This goes some way towards
With ER, it is suggested that both alternatives reach similar addressing the concerns regarding the myths of MCDA, which state
scores (A = 56 and B = 54 or normalised A = 51 and B = 49), it can that it does not always provide a consistent answer, as Belton and
be interpreted as location A and B are performing similarly; how- Stewart (2002) suggested. Moreover, different groups of stakehold-
ever, using the AHP model, there is less doubt that alternative A ers were asked to weight and assess the same criteria and alterna-
significantly outstrips the alternative B (A = 62.35 and B = 37.65 tives, based on the same given information, to establish whether
normalised). Having said that, this does not indicate which model the differences were significant or not. This relates to view
provides the optimum solution in this example and in this context. MCDA as being highly useful for exchanging knowledge (Tavana
Would the final recommendations change based on the ER or & Sodenkamp, 2010).
AHP results? What are the most suitable models to optimise the By using ER, the weighting and assessment processes generated
decision-making process in this environment? good consistency. Over time, participants were able to repeat their
assessments, quite confidently, by using the Likert scale methodol-
5. Discussion ogy. However, by using AHP and the pair-wise comparison, the
process was found to be less consistent, especially as the model
The framework developed from the literature (Fig. 1) structured became bigger anomalies and contradictions were created, as
the assessment of both models and focussed the discussion of their observed the decision-makers. This could partly be explained by
implications, from a practical and managerial perspectives. The fol- the decision-makers not being familiar with pair-wise comparison
lowing section will deliberate the models and compare them methods, considered confusing by the group of participants.
against the framework criteria: the process consistency, the pro- Therefore, it can be suggested that ER is more likely to be a consis-
cess transparency and its facilitation and access for the decision tent method for assessing alternatives, but could lead to some
makers; as well as the robustness of the result and their represen- inconsistency within the weighting process, as participants and
tativeness. The rationality element was not considered as a decision-makers were reluctant to use the whole scale and the
range of most of the weightings were only between 6 and 9 on
the entire, 1–10 scale, which could affect the final results.
Table 4 Therefore, in terms of consistency, it was recommended that
Scoring differences between ER and AHP. the pair-wise comparison is used at the criteria level, and the
Criteria Scoring degree of belief technique is used in the assessment, so as to reach
an optimum process consistency.
ER (IDS) AHP (MiR)
Transparency was the primary criteria for justifying the MCDA
A B A B route, as discussed earlier in this paper. The objectives were to
Env & safety 67 56 75 25 embed inclusive processes and make them easy to understand
Size 50 91 12.5 87.5 for the large range of stakeholders involved. In this case, ER seemed
Total cost 66 34 87.5 12.5
easier for the majority of the participants involved; ‘‘ER was more
Accessibility 45 45 50 50
Design 60 81 16.67 83.33 straightforward than AHP’’ according to the participants. This is
Risks 58 46 75 25 reinforcing the findings from the literature, which states that ER
Population profile 50 50 50 50 is a ‘simple’ process, and that there are many different ways to
Aggregate 56 54 62.4 37.6
compile and aggregate the results, as Xu and Yang (2001), and
Normalised 51 49 62.4 37.6
Xu (2011) explained. Also, the pair-wise comparison had to be
 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727 6725

established by a consensus, and some of the stakeholders and

decision-makers found it slightly confusing and rather redundant, MCDA Model
which reduced the transparency factor.
Hence, it was confirmed that, for the large range of stakehold-
ers, ER was more a transparent process than the pair-wise compar- ER (IDS)
ison. It was easier to track, as the individual inputs could be
highlighted, as part of the process is to average the different scores
given by all the participants, and the process allows the average Weighting:
scores to be reproduced on different occasions. By using AHP, it
was necessary to identify the pair-wise weight, or assessment, Level: criteria
based on the general consensus given at the specific time. Method: Pairwise comparison
However, it was found that it did not keep track of what happened
during the process, which could, arguably, make it less transparent
than ER. Possibly, to overcome this issue, when using AHP, every
Alternative assessment:
stakeholder could provide their own pair-wise comparison and
an aggregated mean of the individual judgment could be gener-
Method: degree of belief
ated. However, this was considered impractical at the time of the
experiment but will be extremely relevant in the future when
the MCDA maturity of the organisation has grown.
Both models can be facilitated using a large number of stake- Results analysis process
holders. It was felt that AHP was easier and faster, as it interacts
with a higher level of the structure. Moreover, AHP uses one mech- Fig. 5. Merging ER process with the pair-wise comparison.
anism for both weighting and assessing (i.e.: pair-wise compar-
ison); whereas, ER uses the Likert scale for the weighting, then
the degree of belief for the assessment of the alternatives. More points of difference, whereas ER model only established a 2 points
time needed to be allowed for facilitating ER as opposed to AHP. difference in (both cases normalised (56–54) and non-normalised
AHP was also easily facilitated by an excel spread sheet, which (51–49) as illustrated in Table 4.
proved convenient for the decision-makers. Having said that, from The rationality aspect was defined by asking the stakeholders
the feedback received, participants were more comfortable using what was the process they perceived the most rational, ER came
the Likert scale and degree of belief system than pair-wise compar- out in front: ‘‘ER seems more scientific’’ argued one the
ison, despite the training provided beforehand. It was felt that AHP decision-makers. However, from the example AHP results seem
was more accessible, as it remained at the aggregate level of the to be more in-line with reality; in the past the questionnaire was
hierarchy model – very useful for unstructured problem solving the tool used to make the final decision for the site locations. In
– whereas, ER goes down to the smallest level of the model; in this this case the reality was translated more through the AHP model
case, the sub-criteria (Saaty, 1980; Wang et al., 2006). than with the ER model. As mentioned previously, this was due
to the tendency that with ER, only a part of the scale was used
5.2. Results and their accuracy (6–9), especially during the weighting stage, this was clear in the
presented case, which is translated by the small range for the cri-
The robustness of the results was hampered by the possibility of teria weighting varying from 8.90 to 22.22 (once normalised)
introducing bias; the stability of the models and the sensibility whereas with AHP, it fluctuates from 3.53 to 38.89 (see Tables 2
aspect of the results were other factors considered. Ideally, the and 3). This has had a substantial impact on the results. Hence, it
model needed to be bias proof and sensible enough to adequately was felt that the AHP model was appropriate to translate better
translate the results. It was suggested that, potentially, AHP was the reality, as seen by the local population, thanks to its criteria
the more sensible option, as the spread of the results shown; how- pair-wise comparison element.
ever, it was more likely to introduce bias into the results, by finding
consensus based on the strongest personality in the room, while 6. Conclusion
weighting and assessing criteria as well as the alternatives.
Moreover, the AHP method could possibly introduce unsteady 6.1. Research questions answered
elements by not following a logical and consistent pair-wise
assessment, and there is a danger that contradictions might be To provide specifically focused conclusions and evidence-based
input into the model. Both techniques provide sensitivity analysis. the originality of the paper, the research questions are answered
This translates the robustness of the results, as one can further each in turn. RQ1: Are the operational processes and outcomes
understand what the ranking means, plus what influence changing significantly different according to the MCDA model implemented:
a weighting, or unit of assessment would have on the results. ER or AHP? There were significant differences between the process
Therefore, , it was analysed that ER was less subject to bias and and the outcomes of the two models. According to the model
was slightly more stable than AHP, perhaps because it works at selected, the results were statistically and significantly different;
the lower level of the model, in line with the observations of Xu thus, this would have impacted the final decision. The process
and Yang (2001). selected also impacts the practical and managerial implications
It was also important to evaluate if the model distorts reality by and behaviours for both the participants and decision-makers. ER
appreciating the level of subjectivity. The mechanism, for estab- uses different methods for weighting and assessing and works at
lishing whether or not this was the case, was to compare the the lower level of the model, which supports the transparency
results of the model against other measurements. In this case, and robustness elements; whereas, the decision-makers found
the measurement available was the extensive survey of N = 3055, AHP to be more flexible, very efficient and extremely relevant in
undertaken by the organisation during the public consultation, a smaller strategic committee, in which the level of transparency
from which 92% of the participants were in favour of Location A. for the local population was not necessarily the prime issue.
The AHP model shows a wider range between A and B with 24.7 Moreover, the pair-wise comparison seemed more appropriate
6726 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727

for grasping the real, or subjective, differences. This section leads to more optimum operational solutions and a more seamless
reinforces the quote, stated at the beginning, that ‘‘essentially, all process from the decision-makers perspectives, than the tradi-
models are wrong, but some are more useful than others’’ (Box & tional ER or AHP. The advantage is that decision-makers can gain
Draper, 1987). From the findings, it was felt that this is greatly enhanced confidence in the results generated by the model and
dependent upon the environment. can justify further the reasons for the model characteristics.
With regards to the second question, RQ2: According to the There are a number of potential areas of further operational
decision-makers, what is the most reliable and appropriate modelling research in order to enrich this study and overcome its limitations.
techniques to provide a rational, inclusive and transparent operational Firstly, it is relevant to facilitate the pair-wise comparison individ-
solution? The most reliable and appropriate modelling technique to ually and develop a geometric mean of the assessments, within
use in the specific context of site selection for future healthcare AHP, instead of seeking a general consensus. However, as
infrastructure, when seeking a rational, inclusive and transparent explained, this would have been impractical at the time of the
solution, would be a hybrid version of both ER and AHP. It was experiment; nevertheless, it will be considered in the future.
agreed that both models were reliable techniques with different Secondly, the perceptions of the decision-makers were gathered
characteristics. Thus, to optimise both the process transparency qualitatively, it might be appropriate to develop and validate a
and consistency, the use of ER, merged with the AHP pair-wise construct, in order to measure quantitatively aspect of rationality
comparison at the criteria weighting process, is suggested. It is and transparency for each model. Thirdly, it would be relevant to
believed that, by implementing this hybrid version, the rationality test this new hybrid model from the beginning of the process,
of the decision can be optimised even further, by developing an and compare the results with the ER and AHP models. These sug-
informed, sensitive and transparent decision for the site selection. gestions would strengthen the validity of the results presented in
Consequently, it is recommended to work at the lower level of the this paper. Finally, to explore further the phenomenon of this
model, as ER suggests, in order to reduce the information asymme- hybrid model and strengthen the impact to Expert and Intelligent
try; but that the weightings of the criteria are established, through Systems, this technique will be tested, as part of future research,
the pair-wise comparison, as AHP implies, and as is illustrated in in different sector and for different type decision-making. These
Fig. 5. This goes towards Zhang et al. (2012) who also used a mixed findings should also encourage Expert and Intelligent Systems
AHP and ER approach to propose a flexible and practical model to researchers to compare other MCDA techniques such as TOPSIS,
cope with qualitative and quantitative data as well as with uncer- VIKOR, ELECTRE, and UTASTAR, in order to establish optimum
tainty for the assessment of e-commerce security. combination characteristics.
Therefore, by solving this hybrid model, using pair-wise com-
parison to assess the criteria weights and the degree of belief to
assess the alternatives, the normalised results are that A is the pre- References
ferred options with 56%, and B has a total score of 44%.
Ackoff, R. L., & Sasieni, M. W. (1968). Fundamentals of operations research. New York:
John Wiley.
6.2. Practical contribution, limitations and further research Bamford, D., & Forrester, P. (2010). Essential guide for operations managers: Concepts
and case notes. London: John Wiley & Sons.
The use of these models directly influenced the board of direc- Belton, V., & Gear, T. (1983). On a short-coming of Saaty’s method of analytic
hierarchies. Omega, 11(3), 228–230.
tors of this National Health Service (NHS) organisation to make an Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: An integrated
informed operational decision for the location of the £15 million approach. Boston/Dordrecht/London: Kluwer Academic Publishers.
health centre. As several attributes were conflicting these tech- Beynon, M. J. (2002). DS/AHP method: A mathematical analysis, including an
understanding of uncertainty. European Journal of Operational Research, 140(1),
niques were useful to aggregate the different stakeholders’ per- 149–165.
spectives and to reach agreement in selecting the key factors in Bi, Y., Guan, J., & Bell, D. (2008). The combination of multiple classifiers using an
identifying the optimum healthcare centre location. By going evidential reasoning approach. Artificial Intelligence, 172, 1731–1751.
Box, G. E., & Draper, N. R. (1987). Empirical model-building and response surfaces.
through this process the healthcare organisation became more United States of America: John Wiley.
informed and sensitive in appreciating the alternatives’ differ- Bozbura, F. T., Beskese, A., & Kahraman, C. (2007). Prioritization of human capital
ences; ultimately this allowed a more rational ranking of alterna- measurement indicators using fuzzy AHP. Expert Systems with Applications,
32(2), 1100–1112.
tive by preferences. It has also been beneficial to the future
Breyfogle, F. W. (2003). Implementing six sigma (2nd ed.). New Jersey: John Wiley &
patients, who were able to follow and take part in the evidence Sons.
based decision-making process. This paper makes a defined techni- Büyüközkan, G., Çifçi, G., & Güleryüz, S. (2011). Strategic analysis of healthcare
cal and practical contribution by examining the use of MCDA mod- service quality using fuzzy AHP methodology. Expert Systems with Applications,
38, 9407–9424.
els in operational location decision-making, and by evidencing the Chen, C.-F. (2006). Applying the analytical hierarchy process (AHP) approach to
most relevant model via a thorough comparison. Furthermore, the convention site selection. Journal of Travel Research, 45, 167–174.
model structure is being used as a starting point to replicate future Chen, Y. M., & Huang, P.-N. (2007). Bi-negotiation integrated AHP in suppliers
selection. International Journal of Operations & Production Management, 27(11),
infrastructure selection decisions, which has been a long standing 1254–1274.
issue. To put this into perspective, over the next six years, ten new Chin, K. S., Wang, Y. M., Yang, J. B., & Poon, K. K. G. (2009). An evidential reasoning
schemes are planned in this specific organisation, representing based approach for quality function deployment under uncertainty. Expert
Systems with Applications, 36, 5684–5694.
more than £150 million of investment. For this reason, the site Chin, K. S., Xu, D. L., Yang, J. B., & Lam, J. P. K. (2008). Group-based ER–AHP system
selection and location decisions will be scrutinised and the emerg- for product project screening. Expert Systems with Applications, 35, 1909–1929.
ing hybrid methodologies will help provided effective and efficient Cousins, P., Lamming, R., Lawson, B., & Squire, B. (2008). Strategic supply
management: Principles, theories, practice. London: Prentice Hall.
guidance. De Moraes, L., Garcia, R., Ensslin, L., Da Conceição, M. J., & De Carvalho, S. M. (2010).
The authors appreciate that ER and AHP have different inherent The multicriteria analysis for construction of benchmarkers to support the
characteristics and assumptions, hence the comparison at a theo- clinical engineering in the healthcare technology management. European
Journal of Operational Research, 200, 607–615.
retical level could be difficult to justify; however, the comparison
Dehe, B., Bamford, D., Bamford, J., & Moxham, C. (2011). An application of a MCDA
is meaningful at the practical and practitioner levels, as model for future healthcare site selection. In: Production and operations
decision-makers use the model to support the complex operational management society (POMs) Conference 2011. Reno, USA.
decisions to be resolved. Therefore, according to the Dey, P. K., Hariharan, S., & Clegg, B. T. (2006). Measuring the operational
performance of intensive care units using the analytic hierarchy process
decision-makers one method can be better than the other. This approach. International Journal of Operations & Production Management, 26(8),
research has evidence-based that the proposed hybrid version 849–865.
 B. Dehe, D. Bamford / Expert Systems with Applications 42 (2015) 6717–6727 6727

DoH (2010). Equity and excellence, Liberating the NHS. Crown Copyright, Department Saaty, T. L. (1980). The analytic hierarchy process – planning, priority setting, resource
of Health: London. allocation. London: McGraw-Hill.
Ertuğrul, I., & Karakasßoğlu, N. (2008). Comparison of fuzzy AHP and fuzzy TOPSIS Saaty, T. L., & Vargas, L. G. (2001). Models, methods, concepts and applications of the
methods for facility location selection. International Journal of Advanced analytic hierarchy process. International Series in Operations Research &
Manufacturing Technology, 39, 783–795. Management Science, 34, 1–25.
Fatta, D., Saravanos, P., & Loizidou, M. (1998). Industrial waste facility site selection Salles, M. (2007). Decision making in SMEs and information requirements for
using geographical information system techniques. International Journal of competitive intelligence. Production Planning and Control, 17(3), 229–237.
Environmental Studies, 56, 1–14. Santos, S. P., Belton, V., & Howick, S. (2002). Adding value to performance
Feldmann, A., & Olhager, J. (2013). Plant roles: Site competence bundles and their measurement by using system dynamics and multicriteria analysis.
relationships with site location factors and performance. International Journal of International Journal of Operations & Production Management, 22(11),
Operations & Production Management, 33(6), 722–744. 1246–1272.
Gay, W., & Bamford, D. (2007). A case study into the management of racial diversity Seçme, N. Y., Bayrakdaroglu, A., & Kahraman, C. (2009). Fuzzy performance
within an NHS teaching hospital. The International Journal of Public Sector evaluation in Turkish banking sector using analytic hierarchy process and
Management, 20(4), 257. TOPSIS. Expert Systems with Applications, 36, 11699–11709.
Golmohammadi, D., & Mellat-Parast, M. (2012). Developing a grey-based decision- Shafer, G. (1976). A mathematical theory of evidence. Princeton: Princeton University
making model for supplier selection. International Journal of Production Press.
Economics. http://dx.doi.org/10.1016/j.ijpe.2012.01.025. Singh, R., & Wood-Harper, T. (2011). The socio-technical balanced scorecard for
Gorsevski, P., Donevska, K., Mitrovski, C., & Frizado, J. (2012). Integrating multi- assessing a public university. In E. Alkhalifa (Ed.), E-strategies for resource
criteria evaluation techniques with geographic information systems for landfill management systems: Planning and implementation (pp. 47–60). Hershey, PAs:
site selection: A case study using ordered weighted average. Waste IGI, Global.
Management, 32, 287–296. Tang, D., Yang, J. B., Bamford, D., Xu, D. L., Waugh, M., Bamford, J., et al. (2012). The
Grigoroudis, E., Orfanoudaki, E., & Zopounidis, C. (2012). Strategic performance evidential reasoning approach for risk management in large enterprises.
measurement in a healthcare organisation: A multiple criteria approach based International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 20,
on balanced scorecard. Omega, 40, 104–119. 17–30.
Guiqin, W., Li, Q., Guoxue, L., & Lijun, C. (2009). Landfill site selection using spatial Taround, A., & Yang, J. B. (2013). A DST-based approach for construction project risk
information technologies and AHP: A case study in Beijing, China. Journal of analysis. Journal of the Operational Research Society, 64, 1221–1230.
Environmental Management, 90, 2414–2421. Tavana, M., & Sodenkamp, M. A. (2010). A fuzzy multi-criteria decision analysis
Guo, M., Yang, J. B., Chin, K. S., & Wang, H. W. (2007). Evidential reasoning based model for advanced technology assessment at Kennedy space centre. Journal of
preference programming for multiple attribute decision analysis under the Operational Research Society, 61, 1459–1470.
uncertainty. European Journal of Operational Research, 182(3), 1294–1312. Tiwari, M. K., & Banerjee, R. (2010). A decision support system for the selection of a
Hodgett, R. E., Martin, E. B., Montague, G., & Talford, M. (2013). Handling uncertain casting process using analytic hierarchy process. Production Planning and
decisions in whole process design. Production Planning and Control. http:// Control, 12(7), 689–694.
dx.doi.org/10.1080/09537287.2013.798706. Tony, M., Wagner, M., Khoury, H., Rindress, D., Papastavros, T., Oh, P., et al. (2011).
Hua, Z. S., Gong, B. G., & Xu, X. Y. (2008). A DS-AHP approach for multi-attribute Bridging health technology assessment (HTA) with multicriteria decision
decision making problem with incomplete information. Expert Systems with analyses (MCDA): Field testing of the EVIDEM framework for coverage
Applications, 34(3), 2221–2227. decisions by a public payer in Canada. BMC Health Services Research, 11, 1–13.
Jakhar, S. K., & Barua, M. K. (2013). An integrated model of supply chain Vahidnia, M., Alesheikh, A., & Alimohammadi, A. (2009). Hospital site selection
performance evaluation and decision-making using structural equation using fuzzy AHP and its derivatives. Journal of Environmental Management, 90,
modelling and fuzzy AHP. Production Planning and Control. http://dx.doi.org/ 3048–3056.
10.1080/09537287.2013.782616. Voss, C., Tsikriktsis, N., & Frohlich, M. (2002). Case research in operations
Kang, H. Y., & Lee, H. I. (2007). Priority mix planning for semiconductor fabrication management. International Journal of Operations & Production Management,
by fuzzy AHP ranking. Expert Systems with Applications, 32(2), 560–570. 22(2), 195–219.
Keeney, R. L., & Raiffa, H. (1993). Decisions with Multiple Objectives, Preferences and Wang, T.-M., & Elhag, T. (2008). Evidential reasoning approach for bridge condition
Value Tradeoffs. Cambridge: Cambridge University Press. assessment. Expert Systems with Applications, 34, 689–699.
Kornfeld, B. J., & Kara, S. (2011). Project portfolio selection in continuous Wang, T.-M., Yang, J. B., & Xu, D. L. (2006). Environmental impact assessment using
improvement. International Journal of Operations & Production Management, the evidential reasoning approach. European Journal of Operational Research, 174,
31(10), 1071–1088. 1885–1913.
Liao, H., & Xu, Z. (2013). A VIKOR-based method for hesitant fuzzy multi-criteria Wu, S., Lee, A., Tah, J. H. M., & Aouad, G. (2007). The use of a multi-attribute tool for
decision making. Fuzzy Optimization and Decision Making. http://dx.doi.org/ evaluating accessibility in buildings: The AHP approach. Facilities, 25, 375–389.
10.1007/s10700-013-9162-0. Xu, D. L. (2011). An introduction and survey of the evidential reasoning approach
Liu, H.-C., Bian, Q.-H., Lin, Q.-L., Dong, N., & Xu, P.-C. (2011). Failure mode and effects for multiple criteria decision analysis. Annals of Operations Research, 195,
analysis using fuzzy evidential reasoning approach and grey theory. Expert 163–187.
Systems with Applications, 38, 4403–4415. Xu, D. L., & Yang, J. B. (2001). Introduction to Multi-Criteria Decision Making and the
Miot, J., Wagner, M., Khoury, H., Rindress, D., & Goetghebeur, M. (2012). Field testing Evidential Reasoning approach, Working Paper No. 0106, pp.1-21.
of a multicriteria decision analysis (MCDA) framework for coverage of a Xu, D. L., & Yang, J. B. (2003). Intelligent decision system for self-assessment. Journal
screening test for cervical cancer in South Africa. Cost Effectiveness and Resource of Multi-Criteria Decision Analysis, 12(1), 43–60.
Allocation, 10, 1–12. Yang, J. B. (2001). Rule and utility based evidential reasoning approach for multiple
Mustafa, M. A., & Al-Bahar, J. F. (1991). Project risk assessment using the analytic attribute decision analysis under uncertainty. European Journal of Operational
hierarchy process. IEEE Transactions on Engineering Management, 38(1), Research, 131, 31–61.
46–52. Yang, J. B. (2007). IDS multicriteria assessor manual. The University of Manchester:
Onut, S., & Soner, S. (2007). Transshipment site selection using the AHP and TOPSIS IDS Limited (pp. 1–58). The University of Manchester: IDS Limited.
approaches under fuzzy environment. Waste Management, 28, 1552–1559. Yang, J., & Lee, H. (1997). An AHP decision model for facility location selection.
Ormerod, R. J. (2010). OR as rational choice: A decision and game theory Facilities, 15, 241–254.
perspective. Journal of the Operational Research Society, 61, 1761–1776. Yang, J. B., & Singh, M. G. (1994). An evidential reasoning approach for multiple
Partovi, F. Y. (2007). An analytical model of process choice in the chemical industry. attribute decision making with uncertainty. IEEE Transactions on Systems, Man,
International Journal of Production Economics, 105, 213–227. and Cybernetics, 24, 1–18.
Pidd, M. (2003). Tools for thinking – Modelling in management science (2nd ed.). Yang, J. B., Wang, J., Bonsall, S., & Fang, Q. C. (2009). Use of fuzzy evidential
Chichester: John Wiley & Sons. reasoning in maritime security assessment. Risk Analysis, 29(1), 95–120.
Ram, C., Montibeller, G., & Morton, A. (2011). Extending the use of scenario planning Yin, K. R. (2009). Case study research: Design and methods. London: Sage.
and MCDA for the evaluation of strategic options. Journal of the Operational Youngkong, S., Teerawattananon, Y., Tantivess, S., & Baltussen, R. (2012). Multi-
Research Society, 62, 817–829. criteria decision analysis for setting priorities on HIV/AIDS interventions in
Ren, J., Gao, Y., & Bian, C. (2013). Multiple criteria decision making based on discrete Thailand. Health Research Policy and Systems, 10, 1–8.
linguistic stochastic variables. Mathematical Problems in Engineering, 1–11. Zhang, Y., Deng, X., Wei, D., & Deng, Y. (2012). Assessment of E-Commerce security
Hindawi Publishing Corporation. using AHP and evidential reasoning. Expert Systems with Applications, 39,
Report, Darzi (2007). High quality care for all. London: Department of Health, Crown 3611–3623.
Copyright. Zhang, H. T., Wang, H., Sun, K., & Wang, D. P. (2011). A method for multi-attribute
Rosero-Bixby, L. (2004). Spatial access to health care in Costa Rica and its equity: A decision making based on ER-AHP. In: The 18th international conference on
GIS based study. Social Science & Medicine, 58, 1271–1284. management science & engineering, Rome, Italy (pp. 123–128).