STATE OF THE ART ON EXPERT JUDGEMENT METHODOLOGIES AND THEIR

APPLICATION TO FAILURE ANTICIPATION

Lela Bouzane
1
, Andr Lannoy
2
, Franois Prs
1

1. Laboratoire PL, Ecole Centrale Paris, Grande voie des vignes 92295 Chatenay
Malabry, France, [email protected]

2. EDF-R&D, 6 Quai Watier, 78401 Chatou Cedex, France
[email protected]
Abstract:
In many industrial Iields and specially in the nuclear industry, risk management is crucial
to ensuring plant and personnel saIety. In this context, anticipation has an important role in
risk prevention, deIinition oI solutions and decision making orientation aimed at optimising
plant maintenance programs and liIetime management.
In this paper, we Iocus on Iailure anticipation. Expert judgement is essential to complete
the data Ieedback and to identiIy potential Iailures. Expert opinion has been systematically
used in many Iields including aerospace programs, military intelligence and nuclear energy.
This paper presents some expert judgement methodologies useIul in Iailure anticipation.
1. Introduction
Failures oI a component are generally well known during the design process. However iI
some Iailures are eIIectively observed, others are never observed, the degradation speeds
being very low, oIten lower than the previously expected ones.
Moreover , generally Ior economic reasons, when degradation mechanisms are considered
well controlled, the question is to extend the liIetime oI the component beyond its design
liIetime. New problems not considered at the design stage by Iunctional analysis or FMEA,
can occur. These problems can occur when modiIying Ior instance the operation procedures
or when improving the perIormance oI installations or when ageing has not been detected or
is not correctly managed. Consequently Iailures not predicted can occur, maintenance
programmes can be inadequate and it is indispensable to anticipate these potential Iailures
which can occur during the end oI liIe phase, or during the extension liIe phase, oI the
component. This anticipation problem and its consequences in terms oI decreasing the
perIormance oI a component (availability, saIety, costs) have to be determined.
This paper is Iocussed on this aspect. The purpose is to provide an expert judgement
elicitation methodology Ior anticipating the Iailures oI a component, up to the end oI its
design liIe cycle period, including eventually an extension period.
We deIine Iailure anticipation as the identiIication oI events which are potentially
objectionable as concerns cost, saIety or availability, beIore they occur to evaluate the risks
which they represent and to prepare and implement the appropriate preventive or exceptional
measures which may be required.
The paper is divided into three parts.
The Iirst part deals with the use oI expert judgement as an essential source oI inIormation
in a decision-making context. As risk analysis typically deals with rare events, this makes
relevant data scarce. For this reason, the use oI expert judgement is strengthened. This is even
truer when dealing with nuclear systems with a high quality design and a very demanding
maintenance programme, where Iailures are very rare.
The second part, is a state oI art on expert judgement methodologies. Some expert
judgement methodologies already used in nuclear studies are presented, analysed and
compared. Theses methodologies are then compared to our case study and classiIied
according to their appropriateness to anticipation.
This leads us to identiIy recommendations aimed at building an expert judgement
methodology well suited to Iailure anticipation.
2. Failure anticipation and expert judgement
This study has been carried out within the Iramework oI equipment liIe cycle
considerations (LiIe Cycle Management). Certain equipments represent major investments Ior
the company. In addition to the cost oI design, manuIacturing and installation, such
equipment can oIten require signiIicant maintenance. However, iI an equipment has been
designed with a high level oI quality and is properly maintained, it is possible to envisage
extension oI its service liIe beyond the service liIe deIined during the design process. This liIe
cycle extension would make it possible to Iurther amortize the initial investment.
Problems other than those identiIied during the deIined process can appear. For this reason,
it is useIul to anticipate these potential Iailures which can occur during the end oI liIe cycle
period. To anticipate, it is necessary to take account oI past Ieedback concerning the
equipment and also oI Ieedback relative to similar equipment installed in other units under the
same environment, operating and maintenance conditions. It is also necessary to take account
oI modiIications with respect to the design and current and Iorecast operating and
maintenance conditions.
Two important aspects must be considered :
- management oI physical aging oI component,
- cost management.
Due to the very special Iramework which the nuclear context represents, equipment used in
this context presents several special characteristics :
- speciIic equipment,
- importance oI saIety,
- high quality design,
- stringent maintenance.
These characteristics result in limited Ieedback (low number oI Iailures) which can make a
statistical study diIIicult.
To compensate Ior this limited inIormation, the classic solution consists in gathering expert
survey inIormation. The expert survey contributes to Iilling in the gaps oI the Ieedback data.
The expert is considered as a relevant source oI inIormation.


























Figure 1 : Actors involved in anticipation process information stream.
Expert
(Knowledge)
Analyst
(Failure
anticipation
approach)
Decision-maker
(Decision-making process,
strategic orientations)
Decisions
Information for anticipation
Decision-aid information
Existing data
(design, maintenance,...)
Feedback concerning similar equipment
Available objectives and data concerning equipment
Equipment data completed by expert survey
Equipment data requested
3. State of the art on expert judgement methodologies
For this study, we have considered 10 expert judgement methodologies already used in
nuclear studies. Six oI them were considered on a benchmark exercise Ior a PSA Study,
experiment L-24 oI the JRC-ISIS, FARO Iacility Ior Iuel coolant interaction studies in a
nuclear reactor accident | 1 | :
NNC methodology,
FEJ-GRS methodology,
STUK-VTT methodology,
NUREG-1150 methodology,
KEEJAM methodology,
CTN-UPM methodology.
Other methods covering diIIerent saIety applications have also been studied:
Procedure guide Ior structured expert judgment,
LCM methodology developed by EPRI (LiIe Cycle Management),
TRIZ-AFD methodology (Iailure anticipation),
RIPBR, Risk-InIormed, PerIormance-Based Regulation, developed by the Department
oI Nuclear Engineering, MIT (risk management and maintenance optimisation).
3.1 Presentation of the methodologies
AAC methodology|1|
This methodology was developed in 1996. It is based on the quality principles and
procedures in the NNC Quality Procedures and Engineering Manual, U.K.. NNC is a Quality
based methodology : based on quality assurance methods oI the sources oI inIormation and oI
the problem solving processes, this approach is based on individual estimates. It involves a
multi-disciplinary team, deIined as a set oI individuals with diIIerent but complementary
skills.
As there is no rigorous Iormal elicitation process, the NNC approach may be called
inIormal expert judgement.
FE1-CRS methodology|1|
This methodology was developed in 1985 by GRS, Germany. The methodology has been
developed to quantiIy the state oI knowledge in elements oI a breakdown oI the question and
to propagate it through this breakdown to arrive at a quantitative uncertainty statement Ior the
answer.
The methodology aggregates the judgements at lower levels and propagates them through
the breakdown to arrive at a quantitative expression oI the resulting state oI knowledge at the
model output level.
1UK-J11 methodology|1|
This methodology was developed in 1997 by VTT Automation, STUK, Finland. The
methodology was originally intended Ior use in various kinds oI quantitative risk and
reliability assessments, and in engineering and economical analyses, where remarkable
uncertainties are present.
The methodology is based on probabilistic representation oI uncertainties. The predictions
obtained Irom experts are expressed as probability distributions. The combination oI these
assessments is based on hierarchical Bayes models (belieI networks). Due to this property, it
is also possible to deal with experts who are not Iamiliar with the concepts oI probability.
Although, there are no restrictions as to the applicability oI the method, it is at its best when
applied to generate predictions to physical parameters
AUREC-115 methodology|1, 2, 3|
This methodology was developed in 1987-1990 by US-NRC, USA.
Highly structured, this approach includes training oI the experts, review oI discussions,
individual elicitations, composition and aggregation oI the opinions and review by experts.
In the NUREG-1150 approach, the domain experts write reports on the issue and their Iinal
estimates are elicited individually aIter expert`s discussions, then averaged on an equal weight
basis.
KEE1AM methodology|1, 3|
This methodology was developed in 1997 at JRC-ISIS in collaboration with the University
oI Brescia and the University oI Bologna, Italy. Knowledge based methodology : the method
employs Knowledge Engineering techniques, and includes explicit modelling oI the
knowledge and problem solving procedure oI the domain expert.
The approach provides structured and disciplined support to the knowledge engineer in
eliciting the knowledge and reasoning strategies oI the experts, building consistent knowledge
models, and applying these models to the solution oI the expert judgement task.
C1A-UPM methodology|1|
This methodology was developed in 1997 by the Department oI nuclear engineering,
University oI Polytechnics oI Madrid, Spain.
It was developed and adapted on the basis oI the NUREG-1150 methodology, although
there exists a very important diIIerence between them regarding the way to aggregate experts
evaluations. The CTN protocol has been developed to get estimates oI subjective probabilities
Ior unknown parameters and uncertain events. It consists oI nine steps executed sequentially.
Procedure guide for structured expert judgement|4|
This methodology was developed in 2000 by DelIt University oI Technology, The
Netherlands.
This is a European Guide Ior Expert Judgement in Uncertainty Analysis. It deals with
procedures to perIorm an expert judgement study with the aim oI achieving uncertainty
distributions Ior an uncertainty analysis. In that Iield oI application, the methods developed at
the DelIt University oI Technology have beneIited Irom experience gained with expert
judgement in the US with the NUREG-1150 methodology. The procedure guide represents a
mix oI these developments.
LCM methodology|5|
This methodology was developed by EPRI, USA as part oI the LiIe Cycle
Management/Nuclear Asset Management studies.
In order to guarantee long-term equipment reliability risk in nuclear power plants, LCM
helps managing aging degradation and obsolescence oI important systems, structures and
components. It gives an optimal solution Ior liIe cycle management based on an economical
comparison between the diIIerent possible solutions.
1RIZ-AFD methodology|6|
This methodology was developed in 1997 by KAPLAN, USA. Il allows identiIication and
analyses oI Iailures based on the TRIZ methodology. AFD ( Anticipatory Failure
Determination) was recently developed in the United States.
AFD consists oI two tools: AFD 1 and AFD 2. AFD 1 is used to analyse Iailure causes. AFD
2 completes AFD1 with a number oI steps Ior Iailure anticipation.
RIPBR, Risk-InIormed, PerIormance-Based Regulation developed at the Department
oI Nuclear Engineering, MIT. RIPBR is an evolving alternative to the current
prescriptive method oI nuclear saIety regulation. RIPBR is goals oriented while the
prescriptive method is means oriented.
RIPBR is capable oI justiIying simultaneous saIety and economic nuclear power
improvements. It includes the Iormulation oI probabilities through expert elicitation and the
review oI risk-inIormed, perIormance-based engineering analyses used to evaluate proposed
changes to existing technical speciIications.
3.2 Description of the methodologies
For each oI these methodologies, a method sheet has been prepared to provide a
summarized description oI each.
Each sheet contains :
- the date and country oI development,
- the organization which developed the methodology,
- the characteristics oI the method (presented to underscore its originality),
- the input data available to the expert, and the output data are both described.
The sheet then presents the various phases involved in the method and the existing tools.
The main applications oI the methodology are given, as well as the methodology`s weak
and strong points. Finally, the background reIerences are given.
Example oI Nureg-1150 methodology sheet.


NUREG-1150 methodology

Developed in 1987-1990
By US-NRC, USA.

Characteristics


Highly structured methodology : the approach includes training oI the
experts, review discussions, individual elicitations, composition and
aggregation oI the opinions and review by experts.

In the NUREG-1150 approach, the domain experts write reports on the
issue and their Iinal estimates are elicited individually aIter expert`s
discussions, then averaged on an equal weight basis.

Input


Missing data or inadequate models with substantial uncertainty about the
probabilities associated with the events, variables or processes that
characterize an issue.

Output


Quantitative judgement in the Iorm oI probabilities or probability
distributions that reIlect the best current knowledge oI the events,
variables or processes under study.

Phases


1. IdentiIication and selection oI the issue(s);
2. IdentiIication and selection oI the experts;
3. Discussion and reIinement oI the issue(s);
4. Probability training;
5. Preparation Ior elicitation;
6. Elicitation;
7. Aggregation and resolution oI agreements;
8. Documentation and communication.

Existing tools


Procedure guide

Applications


SaIety probabilistic evaluation oI American nuclear power plants.
Severe accident oI nuclear reactors, USA.
Environmental risks resulting Irom the disposal oI nuclear wastes:
Radioactive waste storage centres, USA (US-NRC, NUREG-1154,
1990).
Seismic saIety problems.
Health risks due to car emissions.

Strong points


Easy to implement.
Quantitative results.

Weak points


There is no speciIic Iormalised technique Ior individual elicitation.
There is no technical discussion associated to experts group
elicitation.
There is no uncertainty evaluation oI the experts knowledge.
There were no probabilistic values associated to experts opinion and
uncertainty.

References


Keeney R.L., Von WinterIeld D., On the uses of expert fudgement on
Complex Technical Problems, IEEE trans. Eng. Manag., Vol. 36, pp. 83-
86,(1989).

Keeney R.L. and al, Use of expert fudgment in NUREG-1150, Nuclear
Engineering and Design, Vol. 126, pp. 313-331, (1991), North-Holland.

Eppel T., Von WinterIeld D., Eliciting Experts Judgements to predict
the outcomes of the FARO L-21 Experiment. The NUREG-1150
methodologv. With an Appendix containing individual reports prepared
by the Iollowing substantive experts: R.M. Bilbao y Leon, G.C.
Colombo, G.P. Dobson, D. GryIIroy, P. Meyer, O. Zuchuat. EC-JRC-
ISIS/SMA 3250/97, Final Report Revised January, (1997).

Lannoy A., Procaccia H. , Lutilisation du fugement dexperts en surete
de fonctionnement, Editions TEC & DOC, December, (2001).


3.3 Classification of the methodologies
To compare the methods studied, we have classed them with respect to their
appropriateness Ior anticipation and to the eIIort which they require, in an anticipation/eIIort
diagram.




















Figure 2 : Classification of the expert judgement methodology.
Note : The eIIort/anticipation diagram represents an initial look at the various
methodologies. A more precise classiIication, based on the expertise oI a Iew major experts
will be issued in the near Iuture.

Effort required for implementation
Appropriateness for anticipation
NNC
FEJ-GRS
STUK-VTT
NUREG-1150
KEEJAM
CTN-UPM
European guide
LCM
TRIZ-AFD
RIPBR
Appropriateness
1
2
4
3
Part 1 at the top right shows those methods which are more appropriate to anticipation but
which require high elicitation eIIorts.
Parts 2 and 3 at the bottom bring together the methods which are only moderately
appropriate Ior anticipation purposes.
Part 4 at the top leIt corresponds to those methods which are appropriate Ior anticipation
and which do not require major eIIorts Ior implementation. Method NUREG-1150 is located
by expertise in this Irame.
In our context, methodology NUREG-1150 appears to provide the best basis and it would
be useIul to adapt it to our Iailure anticipation context by Iurther developing the aspects
speciIic to anticipation and reducing the elicitation eIIorts. In this respect, the experts are not
very available and the expertise time must thereIore be reduced.
3.4 Analysis of the methodologies
The comparison oI the various methodologies reveal a set oI generic phases which have
been developed to a greater or lesser extent in each depending on its objectives.
These generic phases are:
1. DeIinition oI elicitation objectives.
2. Choice oI experts to be elicited.
3. Training session in probabilities Ior experts.
4. Preparation oI a questionnaire.
5. Elicitation.
6. Aggregation oI expert replies.
7. Synthesis.
With respect to the generic phases described above, the phase concerning training oI
experts in probabilities has not been opted Ior at this time. Furthermore, this phase can be
replaced by questions adapted to the experts interviewed and by work involving translation oI
the qualitative replies into probabilities. This would lighten the load oI the expert and best
responds to the expert`s availability constraints.
4 Specifications of an expert judgement methodology well suited to failure anticipation
4.1 Objective
The objective is to allow the analyst to call on the expert to anticipate potential Iailures oI a
given equipment based on his own knowledge and on the data gathered by the analyst.
4.2 Originality
The expert is not only required to apply the knowledge which he has in tacit Iorm, but also
to provide imagination and creativity in anticipating an event.
4.3 Constraints
Limited studv time and experts which have onlv limited availabilitv.
This constraint will limit the choice with respect to the type oI elicitation to be chosen. The
accent is placed on individual interviews. However, a return to the experts, as used in the
Delphi method, should not be excluded.
Reticence of experts with respect to elicitation
The objective oI the study is to stimulate the expert`s creativity to anticipate Iailures which
may never have yet occurred. It is important Ior the expert to be able to express himselI Iree
oI any constraints or pressure which can be created by interactive groups. The Delphi
approach thereIore does not seem very well suited to our study context as it results in
systematically eliminating the most original replies |3|. This could be counter-productive in
the anticipation context.
4.4 Preliminary Inputs for elicitation input data generally available before expertise
Objectives and context oI the elicitation
Data concerning the studied component: boundaries, design, Iunctions, materials,
operating conditions, environment, procedures (saIety, maintenance,.),.
These data are generally very heterogeneous. You can Iind: operating Ieedback,
procedures (like maintenance procedures), knowledge reports (rules, reliability reports,.),
physical data,.
4.5 Outputs
1. IdentiIication oI potential degradation mechanisms and Iailures oI the component.
2. Assessment oI degradation and Iailure evolution.
3. Evaluation oI Iailure eIIects : saIety, unavailability and maintenance costs,
dosimetry.
4. Solutions to apply to avoid or postpone Iailures (eIIiciency and costs).
5. Conclusion
Through this state oI the art on expert judgement methodologies used in nuclear studies,
we have been able to compare the existing methodologies. They have been classiIied
according to their appropriateness to Iailure anticipation. This has allowed us to identiIy the
methodology that seems the most useIul. This identiIied methodology, NUREG-1150, must,
nevertheless, be better adapted to anticipation problems.
In order to build this methodology we must answer the Iollowing questions :
Which experts ? In this respect, in the anticipation context, several experts with
diIIerent backgrounds and skills must be interviewed (degradation mechanisms expert,
material expert, design expert, operation expert, maintenance expert, .). One
diIIiculty is that these experts do not respond to the same questions; their responses are
generally complementary.
Which basic inIormation to provide to each expert ? (knowing that each has a diIIerent
skill).
How to Iormulate the questions to be easily understood by experts according to their
skills ?
Another interesting point is also how to integrate the operating Ieedback oI like
components. Using this Ieedback, it is also possible to anticipate potential Iailures.

ReIerences
1] G. COJAZZI, D. FOGLI, Benchmark exercise on expert fudgement techniques in
PSA level 2, Extended Iinal report, 2000.
2] R.L. KEENEY and al, Use of expert fudgement in NUREG-1150, Nuclear
Engineering and Design, Vol. 126, pp. 313-331, (1991), North-Holland.
3] A. LANNOY, H. PROCACCIA , Lutilisation du fugement dexperts en surete de
fonctionnement, Editions TEC & DOC, December 2001.
4] R.M. COOKE, L. J. H. GOOSSENS, Procedure guide for structured expert
fudgement, Project report, Nuclear Science and Technology, European Commission,
2000.
|5| G. SLITER, Life Cvcle Management planning sourcebooks, EPRI, December
2001.
6] F. GUARNIERI, P. HAIK, AFD, une nouvelle methode pour lidentification et la
maitrise des defaillances . presentation, illustration et perspective. Proceedings oI the
conIerences oI ESREL 2002, Lamdu Mu 13, 2002.
7] M.W. GOLAY, Improved nuclear power plant operations and safetv through
performance-based safetv regulation, Journal oI hazardous materials, Vol.71, pp219-
237, 2000.