Adaptation of clinical practice guidelines

Adaptation of Clinical Practice Guidelines Perry GROOT 1 , Arjen HOMMERSOM, and Peter LUCAS Radboud University Nijmegen Toernooiveld 1, 6525 ED Nijmegen, the Netherlands Abstract. A rigorous development process of clinical practice guidelines through a systematic appraisal of available evidence is costly and time consuming. One way to reduce the costs and time, and avoid unnecessary duplication of effort of guideline development is by relying on a local adaptation approach of guidelines developed at the (inter)national level by expert groups. In this chapter we survey the work on guideline adaptation, which includes methodologies, case studies, assessment of effectiveness, and related work on guideline adaptation in the Artificial Intelligence community. Keywords. Protocol, Guideline development, Refinement Introduction The trend of the last decades has been to base clinical decision making more and more on sound scientific evidence, i.e., evidence-based medicine [78]. In practice this has led medical specialists to develop evidence-based clinical practice guidelines (CPGs) for promoting standards of medical care. Worldwide, a number of organisations, such as CBO2 (Dutch Institute for Health Care Improvement) in the Netherlands and SIGN3 (Scottish Intercollegiate Guidelines Network) in Scotland, have been founded to assist specialist groups and general practitioners in the development of guidelines. In 2002 the Guidelines International Network4 was founded to promote systematic development of CPGs through international collaboration [51]. A rigorous development process of CPGs through a systematic appraisal of available evidence is, however, costly and time consuming. One way to reduce the costs and time, and avoid unnecessary duplication of effort of guideline development is by relying on local adaptation of guidelines developed at the (inter)national level by expert groups. In this context ‘guideline adaptation’ is a process in which existing guidelines are modified to reflect the local situation so that they can be used within a different care setting. A local adaptation of one or more CPGs is often called (clinical) protocol. A protocol typically provides detailed information about duration, dose, or procedure, suited to the 1 Corresponding Author: Perry Groot, Radboud University Nijmegen, Toernooiveld 1, 6525 ED Nijmegen, the Netherlands; E-mail: [email protected]. 2 http://www.cbo.nl [accessed January 2008] 3 http://www.sign.ac.uk [accessed January 2008] 4 http://www.g-i-n.net/ [accessed January 2008] local context, that has been omitted from the original guideline. Basically, a medical protocol is a summary of the most important sections that are in a guideline, mostly recommendations, supplemented with hospital-specific details, although certain recommendations may be changed if they do not fit the local context. Several reasons may exist for adapting the recommendations of an guideline to suit a local context, e.g., cultural differences [66,65,55], constraints on resources [57], end-user involvement, etc. Legitimate changes can be made in recommendations even when the evidence they are based on is the same [17,35,7,77]. This book chapter is structured as follows. In Section 1 we discuss two methodologies that have appeared for the identification of candidate guidelines for local adaptation. In Section 2 we give an overview of case studies performed on guideline adaptation in terms of their objective for guideline adaptation, the setting, and the adaptation steps followed. In Section 3 we discuss a few randomised trials that focus on the effectiveness of the local adaptation approach on the uptake of nationally produced evidence-based CPGs. In Section 4 we discuss work done in the Artificial Intelligence community on guideline adaptation. We focus on 1) the adaptation of guidelines modelled in a formal representation language, 2) a logical representation of guidelines and theory refinement, and 3) machine learning techniques for learning and adapting guidelines from data. In Section 5 we give our overall conclusions on this chapter and Section 6 discusses some of the problems encountered in guideline adaptation that still need to be addressed. 1. Methodology Guideline adaptation should follow similar procedures used in guideline development, including making transparent any decisions and key factors that influence the modifications. Two approaches have appeared for the identification of candidate guidelines for local adaptation, which are partly overlapping. The Practice 10. Schedule Review and Revise Local Guideline 1. Identify a Clinical Area to Promote Best Practice 9. Obtain Official Endorsement and Adoption of Local Guideline 2. Establish an Interdisciplinary Guideline Evaluation Group 8. Finalize Local Guideline 3. Establish Guideline Appraisal Process 7. Seek External Review − Practioner and Policy Maker Feedback; Expert Peer Review 6. Adopt or Adapt Existing Guideline(s) for Local Use 4. Search and Retrieve Guidelines 5. Assess Guideline a) Quality b) Currency c) Content Figure 1. The Practice Guidelines Evaluation and Adaptation Cycle (PGEAC) [27]. A methodology for the evaluation and adaptation of clinical practice guidelines. Step 1 Define the clinical questions Step 2 Search for the source guidelines Step 3 Screen retrieved guidelines Step 4 Assess selected source guidelines: Quality, consistency, applicability Step 5 Adapt recommendations to context of use Step 6 External review Step 7 Adoption/endorsement and implementation Figure 2. The procedure for guideline adaptation proposed by the ADAPTE working group [19]. Guideline Evaluation and Adaptation Cycle (PGEAC) [27,29,30,31] is a ten step approach (Figure 1), which can be used to adopt a guideline with all its recommendations; adopt one guideline, but omit some recommendations that lack strong evidence or cannot be adopted locally; or take the best recommendations from several guidelines and adapt them to include them into one guideline. The other approach (Figure 2) has been developed by the international working group ADAPTE5 [19,20] and overlaps with the PGEAC approach. According to [20], the ADAPTE process was designed to create the conditions necessary to ensure the quality and validity of the resulting guideline and to foster adherence and ownership of professionals towards the adapted guideline whereas the PGEAC was designed to facilitate comparison of different guidelines and guideline recommendations on the same topic and offers a systematic way to evaluate guideline quality and clinical utility. Nevertheless, both approaches are fairly similar. We will discuss both approaches in more detail below. 1.1. Getting Started The first two steps in the PGEAC approach is the identification of a clinical area to promote best practice and the establishment of an interdisciplinary guideline evaluation group. The identification of an area in which to promote best practice can be selected based on several reasons. These include the prevalence of the condition or its associated burdens, concerns about variations in care, associated costs of different care options, effectiveness of the guideline in influencing health care practice, the desire to keep care practice evidence-based, or the knowledge of 5 http://www.adapte.org [accessed January 2008] the existence of evidence-based guidelines [29]. The establishment of the guideline evaluation group should comprise stakeholders who will be affected by the guideline recommendations that will be selected. The multidisciplinarity of the group will enhance the relevance for practice and foster broad ownership and uptake of the adapted guideline [32] (cf. Section 3). In the ADAPTE process, the need for developing a guideline and the establishment of an evaluation group are more considered as necessary conditions for starting the adaptation process and are not labelled explicitly as steps in the process. Additionally, however, the ADAPTE process focusses on defining a number of clinical questions, which is made explicit by using the PIPOH criteria: Patient population (including disease characteristics), Intervention(s) of interest, the Professionals to whom the guideline will be targeted, health Outcome(s) of interest, and the Health care setting in which the adapted guideline will be used. 1.2. Establish Guideline Appraisal Process A guideline appraisal instrument needs to be chosen such that guidelines can systematically be assessed and compared according to the same criteria. Many appraisal instruments have been developed over the years [28], but the Appraisal of Guidelines Research and Evaluation (AGREE) instrument6 is rapidly becoming the gold standard in guideline appraisal instruments [11]. The AGREE instrument was designed to assess the quality of the development process and the way it is reported. Hence, a rigorously developed guideline may still score insufficiently using the AGREE instrument when the development process is not described in detail. 1.3. Search for and Retrieve Guidelines Both the PGEAC approach and the ADAPTE process give the following advice. To make sure that the most relevant high quality guidelines are obtained, a systematic search needs to be done which should start with guideline clearinghouses, e.g., the National Guideline Clearinghouse, the Guidelines International Network, or with country-specific databases. Additionally, websites of known guideline developers or search engines can be useful. For this, the population and intervention terms made explicit using PIPOH by the ADAPTE approach could be of help in the search strategy. The PIPOH approach is in fact very similar to the PICO approach, which involves Population, Intervention, Control or context, and Outcomes of interest [37], used in the PGEAC approach, but stated less explicitly. In addition to the retrieval of guidelines, the ADAPTE process has an explicit step in which the retrieved guidelines are screened against the clinical questions defined earlier. Only those guidelines that correspond to the clinical questions are selected for a more detailed appraisal. Screening of guidelines is not part of the PGEAC approach, although it is suggested that additional criteria can be used in the search process to omit certain guidelines from the search results and only those guidelines that meet the minimum inclusion criteria will be used in the appraisal process. 6 http://www.agreecollaboration.org [accessed January 2008] 1.4. Assess Guidelines A pivotal step in the adaptation process is the appraisal of the guidelines. Both PGEAC and ADAPTE consider a number of fairly similar dimensions in the appraisal of the guidelines, i.e., the overall quality of the guideline, the consistency and currency of the guideline, and applicability of the guidelines recommendations to the context of use. The overall quality of the guideline can be used to identify the higher quality evidence-based guidelines, which can be used to restrict the number of guidelines that will follow a full appraisal when the appraisal of all retrieved guidelines is impractical. The consistency and currency of the guideline is validated by checking whether the guidelines recommendations are consistent with the cited evidence and whether these recommendations are still current or need to be updated according to newly obtained results. Finally, each guideline needs to be compared in terms of the recommendations made and level of evidence supporting the recommendations and whether they are applicable to the context of use. 1.5. Adopt or Adapt Guidelines for Local Use After the appraisal, one can adopt or adapt existing guidelines. Adopting a guideline means choosing the best guideline and accepting all its recommendations. Adapting guidelines means taking the best recommendations from several guidelines, applicable to the local context, and adapting and reformatting them into a new guideline. Strong evidence-based recommendations should only be changed when the supporting evidence has changed or when not applicable to the local context, for example, because of resource constraints [57]. The ADAPTE process points out that it is still possible to consider de novo development of a guideline. 1.6. External Review Before the dissemination and implementation of the resulting draft of local recommendations, it should be sent to local practitioners, organisational policy makers, and other stakeholders for a review. This also holds for de novo guideline development and the recommendations in the PGEAC and ADAPTE approach are, therefore, identical. 1.7. Adoption and Implementation In this phase the same issues hold for guideline adaptation as for guideline development. PGEAC and ADAPTE therefore give similar recommendations. When the guideline has been finalised, official endorsement from policy makers should be sought for those clinical care settings in which the guideline will be implemented. The formal decision making and procedural process for endorsing the guideline should be documented by the organisation and a dissemination and implementation plan should be finalised. 1.8. Scheduling Review and Revision of Local Guideline In contrast with ADAPTE, PGEAC explicitly mentions that the adaptation of guidelines is a process cycle and that revisions of the local guideline need to be scheduled. This aspect has had less attention than guideline development, but several criteria (e.g., expiry date, changes in evidence, important outcomes, availability of health care resources, new interventions, etc.) can be set to determine when and what should be reviewed and updated [9,63]. 1.9. Final Remarks Summarising, both the PGEAC cycle and ADAPTE process are fairly similar methodologies for guideline adaptation. Some differences exist, but are mainly differences in explicitness. Both groups have recently merged into the ADAPTE group7 whose main endeavour is the development and validation of a generic guideline adaptation process that fosters the validity and quality as well as the users’ sense of ownership toward the adapted guideline. This has been coined the ADAPTE framework and has also resulted in a generic manual and resource toolkit for guideline adaptation. Both are, at the time of writing, still undergoing an evaluation study. 2. Case Studies A number of publications have appeared that report practical examples and experiences with guideline adaptation. In this section we give an assessment of these publications in terms of their objectives for adaptation, the country in which the adaptation took place, and the steps followed in the adaptation process (cf. Table 1), based on a previous literature survey reported in [19]. 2.1. Alternative to de Novo Guideline Development Several publications have appeared that consider guideline adaptation as an alternative to de novo guideline development [43,30,31,76]. The goals in these publications are loosely formulated as the need for providing evidence-based care in a certain medical area, but without weighing all the pros and cons of guideline development versus guideline adaptation. The pros for guideline adaptation are more or less taken for granted. Additionally, those reports focus on the applicability of a guideline adaptation process. For example, [30,31] use the PGEAC approach (cf. Section 1) whereas [43] uses a guideline adaptation process established by the Registered Nurses Association of Ontario, Canada, in 1999. The guideline adaptation process is therefore overall well documented and covers almost all the guideline adaptation steps included in the review criteria. Each of the reported adaptation processes started with a search for relevant CPGs whereas other reports started their adaptation process from a guideline already selected. These reports are also the only ones to include a detailed assessment of the quality of 7 http://www.adapte.org [accessed January 2008] Table 1. Case-study descriptions on guideline adaptation from [19]. Reference Country A Adaptation Process Steps B C D E F Adaptation as an alternative to de novo development Graham et al., 2002 [30] Graham et al., 2005 [31] Canada Canada + + + + + + Macleod et al., 2002 [43] Canada + + + Voellinger et al., 2003 [76] Switzerland + + + + - + + + - - + + + + - Adaptation as part of an implementation process (international): Armstrong et al., 2004 [2] Canada + Croudace et al., 2003 [12] De Wit et al., 2000 [16] UK Europe - - + - +/- - + + + Glasier et al., 2003 [26] UK - - + - - - Hungin et al., 2001 [39] Peleg et al., 2006 [54] Europe US/Israel - - + + - - + Reddy et al., 1999 [56] India - - + - - + Rhineart et al., 1991 [57] Shye et al., 2000 [67] Indonesia US/Israel - - + + - + - - + - + Adaptation as part of an implementation process (national): Brown et al., 1995 [6] Capdenat et al., 1998 [8] US France - - + + Hall et al., 2000 [34] UK - - - - + + Lobach, 1995 [42] Maviglia et al., 2003 [47] US US - - +/- - + + + Silagy et al., 2002 [68] Australia - - + - + + Tomlinson et al., 2000 [73] UK + - - + + + Adaptation process steps: A) Search for and retrieve existing guidelines, B) Assess guidelines, C) Adopt or adept for local use, D) Complementary literature search, E) Seek external review, and F) Implementation. These steps correspond accordingly to steps in the PGEAC approach: A-4, B-5, C-6, E-7, and F-9. the contents of the relevant guidelines. In [43,30,31] the quality of the guidelines is established using the Appraisal Instrument for Clinical Practice Guidelines [10], which is an older version of the AGREE instrument [11] used by [76] for quality assessment. 2.2. Adaptation as Part of an Implementation Process Other publications that reported experiences on the process of guideline adaptation, were usually given in the context of an implementation process of a guideline at a local site. This was done either by adapting an international guideline developed in a different country [2,12,16,26,39,54,56,57,67] or by adapting a national guideline to a local context [6,8,34,42,47,68,73]. For example, [57] adapts a CDC guideline for the prevention of nosocomial infection in a pediatric intensive care unit in Jakarta, Indonesia. Because of limited resources, changes had to be made to the CDC guideline as well as the local environment. For example, the installment of handwashing sinks, avoiding use of critical devices, indirect quality control of sterilisation by monitoring time and temperature, etc. Whenever possible, a low-technology, common sense approach was used such that fundamental infection control principles could be preserved without straining the local resources and capabilities. All reviewed publications that focus on implementation basically follow the same adaptation steps, which include the adaptation of the guideline in question, the implementation, and, in about half the cases, an external review. The other process steps were almost never included in the report. The guideline to be adapted was already assumed to be given and no search for guidelines was performed. Also, the assessment of the guideline was lacking in all the reviewed publications although this is considered to be a pivotal step in the adaptation process by both the PGEAC and ADAPTE guideline adaptation approach (cf. Section 1). This seems to indicate that work on guideline adaptation is still very much in development and case studies are more of an empiric nature. This is also supported by the fact that out of the twenty case studies investigated, only five were performed before the year 2000. 2.3. Guideline Integration with Local Decision Support Besides the publications presented in Table 1, many other publications exist that focus on the integration of medical guidelines with a local decision support system, but that do not emphasise guideline adaptation. Nevertheless, in many of such cases technical issues are addressed when implementing guidelines at a local site, which may result in changes compared to the original guideline (e.g., the clinical information system in use, the data models of the electronic medical record (EMR), and the data actually collected). Several groups have reported their experience on guideline adaptation when implementing them at a local site. For example, Shiffman [64] investigated the validity of an asthma guideline through a logical analysis showing that the guideline under study was incomplete and ambiguous. Such logical integrity violations need to be addressed before the guideline can be operationalised and a structured data entry system can be devised through an examination of the guideline decision points. [72] investigates the practical considerations when adapting an inpatient heart failure guideline to the outpatient setting and implementing it within an EMR. As a result, about one-third of the original guideline recommendations were not included in the final implementation, because of a different setting at the local site. Additionally, some guideline data definitions had to be translated into several EMR entries as the data definitions were not directly available from the local EMR. Similar results are reported in [54], for the adaptation and implementation of an American diabetes foot care guideline at an Israelian site. Some principled approaches are being advocated that may be used to overcome difficulties when integrating guidelines with local decision support systems. For example, [60,25] advocate resorting to guidelines intentions, in order to ensure the adaptability of the procedure to different contexts, while still preserving the original intentional objectives. A setting-independent format is advocated by [4], which relies on an explicit description of dependencies between actions, and requires that they will be preserved by adaptation. Argumentation is advocated by [24] to display arguments in favour or against a certain treatment from multiple sources in order for the physician to make an educated decision. These studies show that when adapting a guideline for a local site, one should consider the implementation and integration with the local setting (e.g., a local EMR) as soon as possible, as this may have major impacts for the encoding (i.e., the computational representation). Such issues are currently, however, not yet supported by guideline adaptation methodologies (cf. Section 1). 3. Assessment of Effectiveness of Local Adaptation on Uptake of Guidelines Guideline development at a national level by multidisciplinary groups of experts with adaptations being made at a local level is already part of the process of several organisations (e.g., CBO, SIGN). Advocates of this approach argue that evidence-based guidelines can be developed at the national level by expert groups as the skills necessary are available at this level, but unlikely to be available at a local level [68]. Although several arguments can be given in favour of an approach of local adaptation of guidelines developed at a national level by clinical experts, so far this approach has had very little formal evaluation [68]. A few randomised trials have appeared in the literature that specifically focus on the effectiveness of the local adaptation approach on the uptake of nationally produced evidence-based CPGs [68,12]. For example, in [12] 30 (out of 42) practices from Bristol, UK were screened for some time before they were split into two groups of 15 practices, containing 56 and 60 GPs each. One group continued with the usual care while the other group adapted the WHO ICD-10 PHC guidelines. Both practices were then screened again using 186 patients in each group. Both randomised trials [68,12] report no significant changes in practitioner behaviour or patient outcomes, which seems to contradict earlier reports that involvement of end-users in the development process may lead to an increased uptake of CPGs [32]. It is a well known problem, however, that CPGs - without any adaptation - often fail to affect clinical practice [33,79]. Systematic research is therefore done to find out why physicians do not follow CPGs [14,13,36]. (The issue of physicians’ compliance is also discussed in detail in Chapter 9.) These complications, as well as limitations in the performed trials (e.g., a minimal rigorous assessment of the evidence and appraisal of additional literature), limit the results of the studies in resolving the effectiveness of local adaptation on the uptake of clinical guidelines. 4. Adaptation and Artificial Intelligence So far, we looked at guideline adaptation mostly from the view of the medical community. In this chapter, we relate guideline adaptation to concepts in Artificial Intelligence (AI). As not much work has yet been done in AI that specifically focusses on guideline adaptation, many of the things we discuss here will be from a possible future research perspective. Guideline development and guideline adaptation is a knowledge engineering task that can be subdivided into various phases such as knowledge acquisition, representation design, implementation, evaluation, and re-implementation [70]. Most of the modelling activities can be carried out by tools as well as an engineer, i.e., the concept of balanced cooperation [49]. Several tools have already been built for guideline formalisation based on knowledge acquisition (KA) and information extraction (IE) techniques such as Stepper, GEM-Cutter, DELT/A, Uruz, AsbruView, Protégé, AREZZO, and TALLIS [40] (cf. Chapter 8). Such tools usually take the text document of a medical guideline as starting point, from which a formal model is derived. IE is an emerging technology in natural language processing to locate facts and specific pieces of information from unstructured natural text, which can either be developed using a knowledge engineering approach or an automatic learning approach. The automatic approach takes as input a set of documents in natural language and outputs a set of extraction patterns using machine learning techniques [40]. Below we discuss these topics in more detail. Firstly, we look at formal guideline representation languages developed in the Artificial Intelligence (AI) community. Secondly, we look at adaptation from a logic viewpoint as a theory refinement problem. Thirdly, we look at machine learning techniques for developing and adapting guidelines. 4.1. Guideline Representation Language Researchers in AI have been working toward offering computer-based support in the development and deployment of guidelines by using computer-oriented languages and tools [15,53]. Examples of languages include PROforma [22,23], Asbru [59,61], EON [74,75], and GLIF [52] (cf. Chapter 2). These languages model complex clinical processes as a ‘network of tasks’, where a task consists of a number of steps, each step having a specific function or goal [21,52]. Adaptation of medical guidelines is therefore often considered a form of program refinement or program transformation in the AI community. We discuss several studies on adaptation of a guideline represented in a formal guideline representation language in more detail below. For details concerning the languages, the reader is referred to Chapter 2 of this book. As formal guideline representation languages have been evolving since the 1990s, only a few case studies [46,38,45,54] were found that report on the adaptation of a formal model written in a formal guideline representation language in the context of an adaptation process. (References [38,54] are part of this book.) The studies reported in [46,38,45] use the Asbru language, whereas the study reported in [54] uses the GLIF language. The work of [38] does not focus on the adaptation process itself, but focusses on the verification of the begin- and endproduct of the adaptation process for obtaining differences between guideline and protocol using a formal approach. The differences between the same guideline and protocol are also analysed in [46] from an informal angle for the guideline and protocol text and for the corresponding Asbru models. According to [46], the most frequent occurring differences between the guideline and protocol text are refinements of the guideline recommendations in which elements are made more specific or substituted to provide more detail about treatments. For example, the protocol may specify the therapy of choice in cases where the guideline offers different alternatives, or the protocol may include special cases not considered in the guideline. Other refinements, analysed in [46] were found to be the result of recent evidence, i.e., the protocol was more up-to-date than the guideline from which it was adapted. Overall, most of the guideline and protocol text were found to be similar, although a few sections seemed different, because of a different layout used for the protocol. An analysis of differences between the constructed Asbru models of the guideline and protocol showed similar results. Also on this level it was found that much of the Asbru model of the guideline could be reused to construct the Asbru model of the protocol. These results are in agreement with the results of [54], which concludes that a significant portion of the original guideline was also useful for the local site, although the local adaptation process also had significant effects on parts of the encoding. 4.2. Logical Modelling of Guideline Adaptation The task network modelling languages of the previous section are not suitable to define adaptation in such a way that one can reason about the adaptation process itself. Although a lot of work has already been done about formal verification of CPGs (cf. Chapter 4), such work was never done primarily in the task network modelling language, but always in some meta-language. Furthermore, the properties typically looked at (e.g., termination, reachability of plans, etc.) say nothing about the adaptation process, but merely something of the final product of guideline adaptation (cf. [38]). In this section, we look at first order logic as a meta-language for describing the guideline adaptation process. Let T be some theory that represents the formalisation of the guideline text whereas T ′ represents some adaptation from T . More generally, T ′ can be identified with a theory Ti in an adaptation process T ≡ T0 ⇒ T1 ⇒ · · · ⇒ Tn in which each theory Ti is some adaptation step of the original guideline T (with ‘⇒’ not necessarily being material implication). Furthermore, T should have an adaptable representation, i.e., the theory remains consistent when local information (e.g., a particular choice between certain resources) is added to the theory: T ∪ LI 6|= ⊥ for any piece of local information LI. Local adaptability will not hold in general for guideline adaptations as some piece of local information has already been used to adapt the guideline, i.e., ∃LI Ti ∪ LI |= ⊥ These are just some thoughts on the characterisation of the guideline adaptation process and is far from being complete. In guideline adaptation, we may consider the guideline to be a theory that provides solutions (i.e., treatment paths) for some domain and the protocol to be a revision of this theory. Many reasons may exist for revising the guideline, e.g., local restrictions are invalidated, newly obtained evidence provides new patient management options, financial costs of drug or equipment manufacturing has decreased, or additional formatting is needed to increase readability. In this light, guideline adaptation may be considered a theory refinement problem and current research on refining knowledge-based systems may offer interesting possibilities for guideline development and guideline adaptation. Theory Refinement In general, whenever a theory is built of some real-world application domain, one sooner or later is confronted with the problem of maintenance of the model. As building up a theory of a domain is very time consuming, rebuilding the theory from scratch, each time the application changes, is usually too costly. Hence, one would like to detect shortcomings of the theory and make repairs to the model. Several reasons may be distinguished for wanting to change the theory [81,70]: Revision: The theory gives wrong answers, either because the theory does not cover all cases, or some cases are covered incorrectly. Performance enhancement: The theory provides solutions to cases that are too long or too costly as they can be improved. Restructuring: The theory has become ill structured and is not transparent, because of, for example, redundancies and implicit concepts. Although the theory provides correct answers, they are not open for human inspection and explanations are incomprehensible. This is also graphically represented as part of Figure 3. The maintenance task consists of three closely related topics. Validation of a KBS is concerned with determining whether the formal model of reality, i.e., the knowledge base, does indeed correspond with reality. Revision of a KBS is concerned with modifying its answer set to deal with the inconsistency or incompleteness of the knowledge base. Restructuring of the KBS is concerned with changing the representation of the knowledge base to, for example, increase speed or readability, without changing the answer set. 4.3. Machine Learning Here, we take a closer look at machine learning techniques. In particular, machine learning techniques for learning logical theories and logical relations among concepts such as relational learning and inductive logic programming techniques. Furthermore, we do not restrict the input to a set of documents in natural language. We focus on the integration of machine learning methods into the modelling environment of the knowledge engineer to induce rules from examples, possibly hand-tailored to experts’ specifications. Besides IE, machine learning techniques are also applicable to guideline development and adaptation by building up the concepts underlying medical guidelines from medical data. Since the 1990s, machine learning techniques have been gaining importance in a knowledge engineering context [50,3,71] and have resulted in a number of tools such as MOBAL [50] and LINK [69]. Below we discuss in more detail the research and the insights gained in this research area and relate it to guideline adaptation. KBS maintenance Validation Refinement incorrect/ incomplete Revision (changes answer set) incorrect incomplete specialising (correcting revisions) generalising (completing revisions) too slow/hard to understand Restructuring (keeps answer set) hard to understand structure improving too slow performance improving Figure 3. KBS maintenance (adapted from [81,70]). Inductive Learning Algorithms Here, we give a short overview of existing inductive learning algorithms and the issues involved in developing such algorithms. The input language to the learning algorithm is an important aspect when comparing different approaches. Although many approaches use some specific input language, some general classes have been identified, i.e., propositional, structured objects (i.e., binary relations of depth one; e.g., ARCH, Induce, Cluster, KHG), and restricted first-order theories (e.g., KL-ONE, Horn Logic, FOIL, GOLEM, RDT, Inductive Logic Programming (ILP) techniques) [41,1]. Any of these domains can in principle be used for learning, however, intuitive relational connectedness between concepts is lost, for example, when using a propositional input language. Earlier learning algorithms often produce outputs that are incompatible with their inputs, however, much research has focused on finding a common ground for in- and output. Motivation for this is to have a ‘closed-loop’ learning process [18], i.e., allowing the knowledge engineering task to be done incrementally by incorporating the output into subsequent input. Examples of such efforts are attribute arrays, Prolog facts or clauses, or representation languages for interchanging information between several algorithms such as CKRL and KIF [70]. More often than not, the output language can be further restricted to a true subset of the input language. Making this explicit has been considered an important component in ILP research. Many relational learning algorithms, however, assume that their background knowledge is static, i.e., the knowledge does not change during learning [80]. This, of course, limits their applicability w.r.t. guideline adaptation or updating them on a regular basis. A few systems have been developed that do support incremental learning with changing background knowledge such as AUDREY, which can explicitly revise its domain theories, and FOCL, which can recover from incomplete and incorrect domain theories [1]. An algorithm that solves the problem of incrementally building up expertise in a rapidly changing environment is an interesting research area in developing relational learning algorithms and might greatly benefit guideline development and adaptation. Machine learning techniques and relational learning algorithms offer interesting possibilities for guideline development and guideline adaptation. Much work, however, still needs to be done in this research area to make learning algorithms effective tools for such applications. For example, supporting incremental learning with changing background knowledge is still largely an open problem. Nevertheless, some approaches are starting to appear that diverge from the current mainstream view of guideline development and formalisation of text documents using formal representations and data mining techniques (e.g., [44,82,58]). 4.4. Conclusions So far, research on guideline adaptation has mainly focussed on the adaptation of documents without considering any computational representation. Early studies show that computational representations are also likely to be adaptable and that large portions are also useful for the local site. A shortcoming of this work is, however, that it is difficult (or impossible) to give general statements about adaptation of guidelines in terms of task network modelling languages and one is therefore unable to reason about the adaptation process. Other representations such as logic, allows one to state guideline development in more general terms and guideline adaptation as a theory refinement problem. Machine learning techniques offer interesting research prospects, which diverges from the mainstream document oriented view of guidelines, for using logical or probabilistic representations for learning and adapting guidelines directly from clinical data. 5. Overall Conclusions In this chapter we have given an overview of work done by the medical guideline community on guideline adaptation. We discussed two state of the art guideline adaptation methodologies, the PGEAC approach and ADAPTE process, which were found to be fairly similar, and both are now being merged into the ADAPTE framework. We also gave an overview of case studies done on guideline adaptation. Most of these studies were of an empirical nature and adapted guidelines without any adaptation methodology. We also noted that guideline adaptation by the medical community as part of an implementation process does not take into account any computable representation although work done in the AI community has shown that this may influence the adaptation of a guideline. We finished the part on guideline adaptation by the medical community by two assessment studies of the effectiveness of the local adaptation approach on the uptake of nationally produced evidence-based CPGs. Both studies, although inconclusive, were unable to show a positive effect in uptake through guideline adaptation. Thereafter, we looked at work done by the AI community on guideline development and guideline adaptation. Recent work is very much focused on offering computer-based support in the development and deployment of guidelines by using computer-oriented languages and tools. Almost all current work takes a guideline text document as starting point for building a formal model in some task network modelling language using knowledge acquisition and information extraction techniques. Such techniques are limited for describing the adaptation process and some meta-language is therefore necessary. For this, we looked at first order logic and looked at guideline adaptation as a theory refinement problem. We went beyond the current approaches by considering the integration of machine learning techniques into the guideline development process for learning and adapting guidelines from data. There is a still a big gap between work done in the guideline community and work done in the AI community. Much work still needs to be done to bridge this gap. 6. Research Agenda With respect to guideline adaptation, some of the main problems in the guideline community is inaccessibility of CPGs and their varying quality. Not all guidelines are published or accessible through the Internet [76]. Furthermore, several studies suggest that the quality of developed guidelines is highly variable and that often many details, which are needed for assessment, are missing [7,5,35,48,62]. For example, Hart et al., [35] report that several stroke prevention guidelines provide no adequate methodologic information (panel selection, patient preferences, justification of risk stratification criteria) to permit assessment of their quality, potential bias, and clinical applicability. The management recommendations were found to be relatively consistent between guidelines, but differed in several important areas. Voellinger et al., [76] report problems with guideline adaptation, because of a lack of guidelines focussing on co-morbidities as well as the need to adapt the various or missing levels of evidence to a uniform scale. The experience of the organisation involved in guideline adaptation is one factor that may explain the varying quality of CPGs. Training and instruments such as the AGREE instrument should be further developed to improve the quality of CPGs. In the field of AI there is still a lot of work to be done as the topic of guideline adaptation is not well understood at this moment. There should be a clear characterisation of the guideline adaptation process - what kind of adaptation operations are allowed - before tools and languages can be build to support the adaptation task. Also, some people have the misguided belief that guidelines can be represented as executable plans. One might want to represent a protocol that can be executed by a local decision support system, but current evidence-based guidelines are often too incomplete to be considered an executable plan. Instead, CPGs are currently merely some constraints on executing clinical practice. How to combine general constraints with workflow management is still an open problem. Other languages than task network modelling languages may be needed. Finally, it is the belief of the authors that the adaptation process can only be rightly supported by integrating techniques earlier in the development process of CPGs. Current work, that takes the paper document as starting point, is flawed as the textual documents are currently missing too much detail to support all kinds of other tasks such as verification and adaptation. The bottleneck in developing guidelines is still the development process itself. Techniques such as machine learning offer interesting new possibilities that could help in alleviating the problems mentioned above. References [1] D.W. Aha. Relating relational learning algorithms. In S. Muggleton, editor, Inductive Logic Programming, pages 233–260. Academic Press, 1992. [2] P.W. Armstrong, P. Bogaty, C.E. Buller, P. Dorian, and B.J. ONeill. The 2004 ACC/AHA Guidelines: a perspective and adaptation for Canada by the Canadian Cardiovascular Society Working Group. The Canadian Journal of Cardiology, 20:1075–1079, 2004. [3] F. Bergadano, A. Giordiana, and L. Siatta. Machine Learning: An Integrated Approach and its Application. Ellis Horwood, Chichester, UK, 1991. [4] A.A. Boxwala, Q. Zeng, D. Tate, R.A. Greenes, and D.G. Fairchild. Applying axiomatic design methodology to create guidelines that are locally adaptable. In Proceedings American Medical Informatics Annual Symposium, page 980, 2002. [5] F. Bro and F.B. Waldorff. Guidelines – lets take a break and then move forward together! Scandinavian Journal of Primary Health Care, 22:2–5, 2004. [6] J.B. Brown, D. Shye, and B. McFarland. The paradox of guideline implementation: how AHCPR’s depression guideline was adapted at kaiser permanente northwest region. Joint Commission Journal on Quality Improvement, 21:5–21, 1995. [7] J.S. Burgers, B. Fervers, M. Haugh, M. Brouwers, G. Browman, T. Philip, and F.A. Cluzeau. International assessment of the quality of clinical practice guidelines in oncology using the appraisal of guidelines and research and evaluation instrument. Journal of Clinical Oncology, 22(10):2000–2007, 2004. [8] E. Capdenat Saint-Martin, P. Michel, J.M. Raymond, H. Iskandar, C. Chevalier, M.N. Petitpierre, L. Daubech, M. Amouretti, and P. Maurette. Description of local adaptation of national guidelines and of active feedback for rationalising preoperative screening in patients at low risk from anesthetics in a French University hospital. QHC, 7:5–11, 1998. [9] E. Clark, E.F. Donovan, and P. Schoettker. From outdated to updated, keeping clinical guidelines valid. International Journal for Quality in Health Care, 18:165–166, 2006. [10] F. Cluzeau, P. Littlejohns, J. Grimshaw, and G. Feder. Appraisal instrument for clinical guidelines. London: St. George’s Hospital Medical School, 1997. [11] The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality and Safety in Health Care, 12(1):18–23, 2003. [12] T. Croudace, J. Evans, G. Harrison, D.J. Sharp, E. Wilkinson, G. McCann, M. Spence, C. Crilly, and L. Brindle. Impact of the ICD-10 Primary Health Care (PHC) diagnostic and management guidelines for mental disorders on detection and outcome in primary care. cluster randomised controlled trial. BJP, 182:20–30, 2003. [13] K. Dahlberg. Reasons physicians do not follow clinical practice guidelines. The Journal of the American Medical Association, 283(13):1686, 2000. [14] D.A. Davis and A. Taylor-Vaisey. Translating guidelines into practice. a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. Canadian Medical Association journal, 157(4):408–416, 1997. [15] P.A. de Clercq, J.A. Blom, H.H.M. Korsten, and A. Hasman. Approaches for creating computer-interpretable guidelines that facilitate decision support. Artificial Intelligence in Medicine, 31(1):1–27, 2004. [16] N.J. de Wit, J. Mendive, B. Seifert, F. Cardin, and G. Rubin. Guidelines on the management of h.pylori in primary care: development of an implementation strategy. Family Practice, 17(suppl2.):27–32, 2000. [17] F. Eisinger, G. Geller, W. Burke, and N.A. Holtzman. Cultural basis for differences between us and french clinical recommendations for women at increased risk of breast and ovarian cancer. Lancet, 353:919–920, 1999. [18] W. Emde, C. Habel, and C.R. Rollinger. The discovery of the equator or concept driven learning. In IJCAI, pages 355–458, 1983. [19] B. Fervers, J.S. Burgers, M.C. Haugh, J. Latreille, N. Mlika-Cabanne, L. Paquet, M. Coulombe, M. Poirier, and B. Burnand. Adaptation of clinical guidelines: literature review and proposition for a framework and procedure. International Journal for Quality in Health Care, 18(3):167–176, 2006. [20] B. Fervers and I.D. Graham. Guideline adaptation - the way forward? In Guidelines International Network Conference 2005, Lyon, 2005. [21] J. Fox and S. Das. Safe and Sound: Artificial Intelligence in Hazardous Applications. AAAI Press, 2000. [22] J. Fox, N. Johns, A. Rahmanzadeh, and R. Thomson. PROforma: A method and language for specifying clinical guidelines and protocols. In J. Brender, J.P. Christensen, Scherrer. J.R., and P. McNair, editors, Medical Informatics Europe, pages 516–520. IOS Press, 1996. [23] J. Fox, N. Johns, A. Rahmanzadeh, and R. Thomson. PROforma: a general technology for clinical decision support systems. Computer Methods and Programs in Biomedicine, 54:59–67, 1997. [24] J. Fox, V. Patkar, and R Thomson. Decision support for healthcare: the PROforma evidence base. Informatics in Primary Care, 14:49–54, 2006. [25] D.B. Fridsma, J.H. Gennari, and M.A. Musen. Making generic guidelines site-specific. In Proceedings American Medical Informatics Annual Symposium, pages 597–601, 1996. [26] A. Glasier, S. Brechin, R. Raine, and G. Penney. A consensus process to adapt the world health organization selected practice recommendations for UK use. Contraception, 68:327–333, 2003. [27] I. Graham, M. Harrison, and M. Brouwers. Evaluating and adapting practice guidelines for local use: a conceptual framework, pages 213–229. Churchill Livingstone, 2003. [28] I.D. Graham, L.A. Calder, P.C. Hébert, A.O. Carter, and J.M. Tetroe. A comparison of clinical practice guideline appraisal instruments. International Journal of Technology Assessment in Health Care, 16:1024–1038, 2000. [29] I.D. Graham and M.B. Harrison. Evaluation and adaptation of clinical practice guidelines. Evidence-based nursing, 8:68–72, 2005. [30] I.D. Graham, M.B. Harrison, M. Brouwers, B.L. Davies, and S. Dunn. Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations. Journal of obstetric, gynecologic, and neonatal nursing, 31:599–611, 2002. [31] I.D. Graham, M.B. Harrison, K. Lorimer, T. Piercianowski, E. Friedberg, M. Buchanan, and C. Harris. Adapting national and international leg ulcer practice guidelines for local use: the ontario leg ulcer community care protocol. Advances in skin & wound care, 18:307–318, 2005. [32] J.M. Grimshaw and A. Hutchinson. Clinical practice guidelines – do they enhance value for money in health care. British medical bulletin, 51:927–940, 1995. [33] J.M. Grimshaw and I.T. Russell. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 342(8883):1317–1322, 1993. [34] L. Hall and M. Eccles. Case study of an inter-professional and interorganisational programme to adapt, implement and evaluate clinical guidelines in secondary care. Clinical performance and quality health care, 8:72–82, 2000. [35] R.G. Hart and R.D. Bailey. An assessment of guidelines for prevention of ischemic stroke. Neurology, 59:977–982, 2002. [36] A. Haynes, B. Haines. Barriers and bridges to evidence based clinical practice. BMJ, 317(7153):273–276, 1998. [37] C. Heneghan and D. Badenoch. Evidence-based medicine toolkit. Blackwell Publishing, Oxford, United Kingdom, 2006. [38] A. Hommersom, P. Groot, P. Lucas, M. Marcos, and B. Martinez-Salvador. Computerbased medical guidelines and protocols: A primer and current trends, chapter A Constraint-based Approach to Medical Guidelines and Protocols. IOS Press, 2008. [39] P. Hungin. Working together across international boundaries: improving the primary-secondary care network. European journal of gastroenterology & hepatology, 13(suppl2.):13–16, 2001. [40] K. Kaiser, C. Akkaya, and S. Miksch. How can information extraction ease formalizing treatment processes in clinical practice guidelines? a method and its evaluation. Artifical Intelligence in Medicine, 39(2):151–163, 2007. [41] N. Lavrač and S. Džeroski. Review of “inductive logic programming: Techniques and applications”. Machine Learning, 23(1):103–108, 1996. [42] D.F. Lobach. A model for adapting clinical guidelines for electronic implementation in primary care. In Proceedings of the Annual Symposium on Computer Applications in Medical Care, pages 581–585, 1995. [43] F.E. MacLeod, M.B. Harrison, and I.D. Graham. The process of developing best practice guidelines for nurses in Ontario: risk assessment and prevention of pressure ulcers. Ostomy/wound management, 48:30–38, 2002. [44] S. Mani, C. Aliferis, S. Krishnaswami, and T. Kotchen. Learning causal and predictive clinical practice guidelines from data. In Proceedings of MedInfo. IOS Press, 2007. [45] M. Marcos, B. Martı́nez-Salvador, M. Balser, J. Schmitt, A. Dunets, S. Bäumler, and W. Reif. A formal approach to the refinement of medical practice guidelines into protocols. Submitted to Computers in Biology and Medicine, 2006. [46] M. Marcos, B. Martı́nez-Salvador, A. Hommersom, P. Groot, P. Lucas, A. Jovell, and S. Blancafort. Case-study in transformations for protocol development from guidelines. Technical Report D5.1, Protocure II, 2006. [47] R.D. Mavliglia, S.M. adn Zielstorff, M. Paterno, J.M. Teich, D.W. Bates, and Kuperman G.J. Automating complex guidelines for chronic disease: lessons learned. Journal of the American Medical Informatics Association, 10:154–165, 2003. [48] J. Miller and J. Petrie. Development of practice guidelines. Lancet, 355:82–83, 2000. [49] K. Morik. Balanced cooperative modeling. Machine Learning, 11(2-3):217–235, 1993. [50] K. Morik, S. Wrobel, J.U. Kiets, and W. Emde. Knowledge Acquisition and Machine Learning. Academic Press, London, 1993. [51] G. Ollenschläger, C. Marshall, S. Qureshi, K. Rossenbrand, J. Burgers, M. Mäkelä, and J. Slutsky. Improving the quality of health care: using international collaboration to inform guideline programmes by founding the Guidelines International Network (G-I-N). Quality & safety in health care, 13:455–460, 2004. [52] M. Peleg, A. Boxwala, O. Ogunyemi, P. Zeng, S. Tu, R. Lacson, E. Begnstam, and N. Ash. GLIF3: The evolution of a guideline representation format. In Proceedings of American Medical Informatics Association Symposium, pages 645–649, 2000. [53] M. Peleg, S. Tu, J. Bury, P. Ciccarese, J. Fox, R.A. Greenes, R. Hall, P.D. Johnson, N. Jones, A. Kumar, S. Miksch, S. Quaglini, A. Seyfang, E.H. Shortliffe, and M. Stefanelli. Comparing computer-interpretable guideline models: A case-study approach. Journal of the American Medical Informatics Association, 10(1):52–68, 2003. [54] M. Peleg, D. Wang, A. Fodor, S. Keren, and E. Karnieli. Computer-based medical guidelines and protocols: A primer and current trends, chapter Adaptation of practice guidelines for clinical decision support: a case study of diabetic foot care. IOS Press, 2008. [55] R.C. Rasmussen, M.A. Schermann, J.M. Shutske, and Olson DK. Use of the North American guidelines for children’s agricultural tasks with Hmong farm families. Journal of agricultural safety and health, 9:265–274, 2003. [56] K.S. Reddy. Implementation of international guidelines on hypertension: the Indian experience. Clinical and experimental hypertension, 21:693–701, 1999. [57] E. Rhinehart, D.A. Goldmann, and E.J. ORourke. Adaptation of the centers for disease control guidelines for the prevention of nosocomial infection in a pediatric intensive care unit in Jakarta, Indonesia. The American journal of medicine, 91:213S–220S, 1991. [58] D. Riaño, J.A. López-Vallverdú, and S.W. Tu. Mining hospital data to learn SDA∗ clinical algorithms. In 11th Conference on Artificial Intelligence in Medicine, From Knowledge to Global Care Workshop, Amsterdam, 2007. [59] A. Seyfang, R. Kosara, and S. Misch. Asbru’s reference manual, asbru version 7.3. Technical Report Asgaard-TR-20002-1, Vienna University of Technology, Institute of Software Technology, 2002. [60] Y. Shahar, S. Miksch, and P. Johnson. An intention-based language for representing clinical guidelines. In Proceedings American Medical Informatics Annual Symposium, pages 592–596, 1996. [61] Y. Shahar, S. Miksch, and P. Johnson. The asgaard project: A task-specific framework for the application and critiquing of time-orientied clinical guidelines. Artificial Intelligence in Medicine, 14:29–51, 1998. [62] T.M. Shaneyfelt, M.F. Mayo-Smith, and J. Rothwangel. Are guidelines following guide- [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] lines? the methodological quality of clinical practice guidelines in the peer-reviewed medical literature. The Journal of the American Medical Association, 281:1900–1905, 1999. P. Shekelle, M.P. Eccles, J.M. Grimshaw, and S.H. Woolf. When should clinical guidelines be updated? BMJ, 323:155–157, 2001. R.N. Shiffman. Towards effective implementation of a pediatric asthma guideline: integration of decision support and clinical workflow support. In Proceedings of the Annual Symposium on Computer Applications in Medical Care, pages 791–801, 1994. J. Shoultz, N. Phillion, J. Noone, and B. Tanner. Listening to women: culturally tailoring the violence prevention guidelines from the put prevention into practice program. Journal of the American Academy of Nurse Practitioners, 14:307–315, 2002. J. Shoultz, B. Tanner, and R. Harrigan. Culturally appropriate guidelines for alcohol and drug abuse prevention. The Nurse practitioner, 25:50–56, 2000. D. Shye, A. Porath, and J.B. Brown. Adapting a national guideline for local use. a comparative case study in a US and an Israeli health maintenance organization. Journal of health services research & policy, 5:148–155, 2000. C.A. Silagy, D.P. Weller, H. Lapsley, P. Middleton, Shelby-James T., and B. Fazekas. The effectiveness of local adaptation of nationally produced clinical practice guidelines. Family Practice, 19:123–230, 2002. E. Sommer. Learning relations without closing the world. In Proceedings of the European Conference on Machine Learning, pages 419–422, Berlin, 1994. Springer-Verlag. E. Sommer. Theory Restructuring: A Perspective on Design & Maintenance of Knowledge Based Systems. PhD thesis, University of Dortmund, 1996. E. Sommer, K. Morik, J. Andre, and M. Usynski. What on-line learning can do for knowledge acquisition. Knowledge Acquisition, 6:435–460, 1994. W.M. Tierney, J.M. Overhage, B.Y. Takesue, L.E. Harris, M.D. Murray, D.L. Vargo, and C.J. McDonald. Computerizing guidelines to improve care and patient outcomes: the example of heart failure. Journal of the American Medical Informatics Association, 5:316–322, 1995. J. Tomlinson, R. Baker, K. Stevenson, and M. Lakhani. The leicestershire local guideline adaptation and implementation programme: a model for clinical effectiveness? Journal of evaluation in clinical practice, 6:125–229, 2000. S. Tu and M. Musen. A flexible approach to guideline modeling. In Proceedings of American Medical Informatics Association Symposium, pages 420–424, 1999. S. Tu and M. Musen. From guideline modeling to guideline execution: Defining guideline based decision-support services. In Proceedings of American Medical Informatics Association Symposium, pages 863–867, Los Angeles, CA, 1999. R. Voellinger, A. Berney, P. Baumann, J.M. Annoni, C. Bryois, T. Buclin, C. Büla, V. Camus, L. Christin, J. Cornuz, P. de Goumoëns, O. Lamy, J. Strnad, B. Burnand, and F. Stiefel. Major depressive disorder in the general hospital: adaptation of clinical practice guidelines. General Hospital Psychiatry, 25:185–193, 2003. J. Wils, P. O’Dwyer, and R. Labianca. Adjuvant treatment of colorectal cancer at the turn of the century: European and US perspectives. Annals of oncology: official journal of the European Society for Medical Oncology, 12:13–22, 2001. S.H. Woolf. Evidence-based medicine and practice guidelines: an overview. Cancer Control: journal of the Moffitt Cancer Center, 7(4):362–367, 2000. S.H. Woolf, R. Grol, A. Hutchinson, M. Eccles, and J. Grimshaw. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ, 318:527–530, 1999. S. Wrobel. Automatic representation adjustment in an observational discovery system. In Proceedings of the Third European Working Session on Learning, pages 253–262, 1988. S. Wrobel. First order theory refinement. In Advances in Inductive Logic Programming, pages 14–33, Amsterdam, 1996. IOS Press. O. Ying-Lie. Model-based guideline development for symptom-based indications. In ECAI 2006 Workshop – AI techniques in healthcare:evidence based guidelines and protocols, pages 46–51, 2006.