Evaluating community health services: conflict and controversy

zyx zyxwvut Health and Social Care in the Community 5(4), 255-260 Evaluating community health services: conflict and controversy Pam Dawson''" (MCSP) and Bob Heyman' (PhD) Division of Physiotherapy and Applied Life Sciences, Faculty of Health, Social Work and Education, University of Northumbria at Newcastle, *Division of Behavioural and Contextual Studies, Faculty of Health, Social Work and Education, University of Northumbria at Newcastle ' Correspondence Pam Dawson Senior Lecturer Division of Physiotherapy and Applied Life Sciences Faculty of Health, Social Work and Education University of Northumbria at Newcastle Coach Lane Campus Newcastle upon Tyne NE7 7XA UK Abstract The best way to evaluate community health services remains the subject of much argument and debate. This paper explores difficulties encountered in the management of a randomized controlled trial of physiotherapy in a primary health care setting, and discusses conflicts which arose when carrying out a time limited commissioned evaluation of a carer support scheme. Although randomized controlled trials are still regarded as the gold standard in medical research, they can be difficult to coordinate, and often cannot be applied when patient problems are chronic or incurable and require multidisciplinary intervention. There are many cultural and organizational barriers that have to be overcome for evaluation to be successfuI and meaningful. If these barriers are underestimated or ignored, the quality of evaluation is compromised. Keywords: carer support scheme, community physiotherapy,conflicts, ethics committees, evaluation, randomized controlled trial Accepted for publication: 26 July 1996 Introduction The need for good quality evaluation of health services has never been more apparent (Culyer 1994) and purchasers are increasingly funding evaluation research. However, it is easy to underestimate the extent to which organizational and cultural barriers limit the effectiveness of research based evaluation, particularly in areas where the standard model of clinical trials of precisely defined interventionscannot be applied. This paper will examine the role of evaluation research in the develop ment of evidence based community health care practice, taking into account the organizational context in which community health care evaluations take place. The NHS and Community Care Act (Department of Health 1990) attempted to create a 'market' for health and social care services and separated purchasers from providers of such care. Purchasers are expected to ensure value for money, and providers are expected to *PD was formerly Superintendent Physiotherapist at Newcastle Domiciliary Physiotherapy Service, Newcastle upon Tyne City Health Trust. 0 1997 Blackwell Science Ltd zyx zyx zyxw z compete in order to obtain contracts. It is not surprising that tensions have arisen in the purchaser/provider relationship, with purchasers appearing happier with the contractingprocess than providers (Applebyet al. 1994). The randomized controlled trial (RCT) provides a means of evaluating specific medical interventions, and there have been calls for more RCTs (Pollocket al. 1993). However, late twentieth century western health systems have to deal mainly with problems that are longterm, involve conditions that cannot be cured, and require multidisciplinary interventions. Such problems have been described as 'wicked', a term first coined by Rittel & Weber (1974) and applied to health by Kingsley & Douglas (1991). Those who try to manage wicked problems have to confront moral dilemmas and to balance, for example, dependency against risk for vulnerable people, or the needs of people with health problems against those of their carers. It is not possible to define clear, consensual outcomes for the treatment of wicked problems. Pope &Mays (1993)have debated the relative merits of obtaining information about outcome and output of services through RCTs as against using ethne graphic, observational methods to learn about processes 255 P. Dawson and B. Heyman zyxwvutsrqpo of service delivery. They make the point, which we will return to later, that methodological pluralism is required in health services research. User perspectives can provide an important component of community service evaluation, particularly where desirable outcomes cannot be defined easily (Heyman 1995)or when holistic or complementary therapies do not conform to the constraints of the RCT (Knipschild 1993). Organizational procedures for implementing medically managed RCTs for specific interventions, e.g. drug trials, are well-established in the National Health Service. Randomized controlled trials are widely regarded as the gold standard in medical research and are argued to be the best way to gather evidence to underpin rehabilitation practice (Pollock et al. 1993, Wade 1995).However, the RCT does have limitations. The only real conclusion it is possible to make when rejecting the null hypothesis in a clinical trial is that the interventions being compared differ on the chosen outcome. Even theories which have been tested and verified several times can only be termed corroborated, not true in the absolute sense (Senn 1991)if one adopts a Popperian position. Neither random assignment nor controls are a panacea for all threats to reliability and validity (Cook & Campbell 1979, Senn 1991). Assumptions about the objectivity of RCTs are made by many researchers, who fail to acknowledge that even with rigorous procedures, there will often be interactions between treatment and patient variables. Patients are unique individuals, who respond to treatment in different ways (Knipschild 1993)and may well have preferences about how they should be treated (Brewin & Bradley 1989), which will influence their motivation and perhaps make them behave atypically. Validity is thus threatened when trials of participative treatments are interpreted as though they were drug trials, without considering the effects of psychological or social processes. It can be much harder to evaluate services if the evaluation is not medically managed, and involves nonspecific, multidisciplinary interventions, as illustrated by the two case studies outlined below. We discuss, firstly, a large scale RCT of domiciliary physiotherapy, and, secondly, a small scale evaluation of a support scheme for carers of terminally ill patients. The discussions will concentrate on organizational, ethical and cultural problems encountered while carrying out the research, rather than on the research results. 1989 and 1992, supported by a €300 000 grant from the Sainsbury Family Trust Fund. Our discussion of the processes involved in the trial will focus on the following points. Firstly, problems can arise when control of research is shared between medical and non-medical professionals. Secondly, there are contradictions between the evaluation of services and their provision in conditions of scarcity. Thirdly, problems with sample recruitment and attrition can weaken the interpretation of results derived from RCTs, particularly where the interventions are complex, variable, long-term and social in nature. Fourthly, there are issues concerning the generalizability of demonstration projects, and the take-up of inconclusive findings of trials of complex interventions by purchasers. The methodology of the RCT, in brief, was as follows. Two randomly allocated groups of clients were compared. One group received a new domiciliary physiotherapy service (experimental group), and the other received the best existing usuaI care (control group). The trial was set in the community, and criteria for referral were that the client had difficulty travelling to hospital, or had problems linked to the home environment, or that the carer required support. The RCT design was chosen because of its status as the gold standard in medical research, and as the design most likely to yield valid and reliable results. A pragmatic model (Schwartz et al. 1980) was adopted, with interventions optimized to meet the needs of individuals. Outcome measures included the Barthel Index (Collin et al. 1988) as a measure of functional ability, the Nottingham Health Profile (Hunt et al. 1986)as a measure of health status and clients' use of other services. The evaluation (which pre-dated the 1990 NHS reforms) was planned and implemented by the physiotherapy service, in order to meet the demands of the District Health Authority for valid data on service effectiveness. It was intended to recruit a large sample (800 subjects) from all general practitioners (GI's) within the Health Authority boundary, in order to increase the power of statistical tests and maximize external validity (Altman 1991).From referral rates in an earlier pilot study it was estimated that recruitment would take approximately 6 months. At an early stage in the development of the trial methodology, it was proposed that some GP practices (chosen at random) would have access to the experimental service and others would not. This was, however, rejected by the Local Medical Committee (LMC),in favour of randomizing individual clients. The LMC thought it would be unfair to GPs who did not have access to domiciliary physiotherapy and who would therefore have to recruit clients solely for control purposes. Initially, it zyxwvutsr zyxwvu Evaluation of a new domiciliary physiotherapy service zyxwvutsrqp An RCT of community-based domiciliary physiotherapy was undertaken in Newcastle upon Tyne between 256 0 1997 Blackwell Science Ltd, Health and Social Care in the Community 5(4), 255-260 Evaluating community health care services was proposed that clients should be allocated to an intervention using the randomized consent design (Zelen 1979),where clients are not required to consent to participation in the trial, provided their treatment is as it would be under normal circumstances. Suitable subjects would be randomized into either the experimental or control group, and then the GP would seek only the consent of those receiving the new or experimental service. A fieldworker would then seek consent from both groups for data collection and interviews. The randomized consent design was rejected by the local medical ethics committee, which insisted that GPs should both recruit patients and seek their informed consent prior to randomization. Although many referrals in the pilot study had come from other primary health care team members, e.g. district nurses, the ethics committee would not allow anyone other than the GP to recruit subjects for the trial. GPs were given specific guidelines on patients suitable for the trial, but only recruited 245 subjects over a 12-month period (31% of the total required, in double the estimated time). The reasons for the poor recruitment rate were varied. Some GPs claimed that the withholding of domiciliary physiotherapy from the control group would deprive patients of clinical benefit. However, there had never been a domiciliary physiotherapy service in Newcastle, and claims about the value of such a service were based on anecdotes, speculation, and precedents derived from other health districts, where services were being developed without evaluation. Other GPs objected to the RCT on the grounds that, as doctors, they should be able to freely access any service they felt would benefit their patients. It is possible that GPs felt unwilling to admit to patients that they were unsure of the best treatment (Taylor et al. 1984). The problems discussed above are not unique to this trial of domiciliary physiotherapy. A paper published in the British Medicat journal (Tognoniet al. 1991) described the difficulties experienced by a group of Italian researchers attempting to conduct an RCT of the treatment of hypertension in general practice. They found a large discrepancy between the number of doctors who agreed to participate and the number who started recruiting, leading them to question the attitude of Italian GPs towards controlled research. Following the publication of the Tognoni paper, two letters were published in the British Medical Jouvnal reporting similar experiences. Peto & Coulter (1991) wrote to the editor about their study of outcomes of treatment for menorrhagia in Oxford, in which they had to extend their recruitment phase because of lower then anticipated recruitment of patients by GPs. Two months later, Jonker & Sumajow (1992) reported on zy zyx their surveillance study of reflux oesophagitis in The Netherlands, in which, after a year, many GPs who had promised to cooperate had referred no patients. Greenberg (1991) congratulated Tognoni et al. for raising awareness of a problem that is usually glossed over in reports of trials in general practice. Reasons for GPs’ reluctance to enter patients into trials are discussed by Taylor (1992). She surveyed general physicians involved in a multicentred clinical trial in the USA and Canada and found that poor recruitment of patients was linked more to the social process of the RCT than to any inherent resistance or reluctance to support research. Although general physicians publicly argued that experiments were ethically wrong, or that trials were too time consuming, Taylor found that there were conflicting professional roles. American and Canadian general physicians (and presumably British GPs) normally get their rewards and social status from their individual doctor-patient relationship. They are supposed to reduce uncertainty for the patient, apply their knowledge, skills and experience to individual patient problems and then receive personal appreciation from patients and families when there is a successful outcome. Physicians surveyed in this RCT of treatment of eye cancer felt uncomfortable about random allocation taking over their decision making role, and they disliked admitting to patients that they didn’t know which was the best treatment for them. They were uncertain about whether they could remain motivated to carry out administrative tasks when research was not perceived as high status work in general practice. Many physicians were also worried about shifting their allegiance from the present patient to future patients who would stand to benefit from the results of the RCT. The GPs involved in the trial of domiciliary physiotherapy must also have faced the dilemma of conflicting professional roles. However, because patients in the trial would be likely to benefit from the future service if the RCT were successful, the issue of asking patients to participate in a trial which would not benefit them directly did not really arise. In the trial of domiciliary physiotherapy, the mechanisms for obtaining informed consent also caused problems. When research fieldworkers visited patients who had been recruited to the trial, it became apparent that, despite agreed protocols, many different explanations had been given by individual GPs. Some control group patients had false expectations that they would receive domiciliary physiotherapy. Although it is possible that GPs were deliberately sabotaging the trial, it is more likely that, because of their lack of research training, they were unaware of the implications of deviations from the research protocol (Jonker & Sumajow 1992). zyxwvuts zyxwvutsrqp zyxwvut z zyxwvu 0 1997 Blackwell Science Ltd, Health and Social Care in the Community 5(4), 255-260 257 zyxwvutsrqpo P. Dawson and B. Heyman Using research fieldworkersto obtain consent may well have prevented the problems caused by GPs failing to follow the agreed protocol. Subsequent patient dissatisfaction may have had some effect on sample attrition, which although higher in the control group (38%)than in the experimental group (29%),was not significantly different (chi-square = 2.34, P = 0.13). Although patients who refused to be interviewed formed only a small proportion of total dropouts, the difference between the two groups in refusal rates was highly significant. In the experimental group there were two refusals (2%of total patients recruited) compared with 13 refusals (11% of total patients recruited) in the control group (chi-square= 8.95, P = 0.003). In both the experimental and control groups, dropouts from the trial scored significantly worse on the Barthel Index and the social isolation item of the Nottingham Health Profile, suggesting that those subjects who had the most potential for measurable improvement on the scales were lost from the trial, thus affecting the interpretation of the results. Some GPs were blatantly honest at the outset, saying that they supported the trial in principle, but were unlikely to recruit subjects due to the extra work involved in obtaining consent and collecting data. Extra workload is a predictable disincentive to GP involvement in trials (MacIntyre 1991). A financial incentive such as that given in many drug trials can compensate for extra workload (Waldron & Cookson 19931, but there is no evidence that paying GPs would have improved recruitment to the trial. The findings of the domiciliary physiotherapy trial were inconclusive, showing no significant differences between the experimental and control groups. Exploratory analysis with sub-groups has suggested, however, that those clients who had received either day hospital or respite care used significantly less of these services if they were in the experimental group (Dawson 1995). The failure to reject the null hypothesis may have been due to the smaller than intended overall sample size, the masking of real effects by differential sample attrition, or lack of sensitivity of the outcome measures. Despite these non-significant results, the study still had a marked impact on policy. The strength of feeling among GPs who didn’t want their patients to be randomized into the control group indicated that physiotherapy was seen as having a necessary and positive role in the community. Since there was no evidence that experimental patients were in any way disadvantaged, a decision was made to continue the domiciliary physiotherapy service, despite the inconclusive evaluation findings. It was soon confirmed that many GPs had withheld potential referrals to the trial, as their referral rates to the continued service immediately doubled (Dawson 1993).The domiciliary physiotherapy service has now regained its freedom to receive referrals from other health and social services colleagues. The results of the trial may now be included in a national meta-analysis, to enhance the likelihood of discovering reliable trends across multiple research projects. zyxwvutsr zyxw zy 258 Evaluation of a new carer support scheme This pilot scheme provided intensive support for carers of patients who were not suffering from cancer and were expected to live for less than a year. A district health commission had initially funded the scheme for one year (1994/95) and required the evaluation in order to decide whether to provide further funding. Our discussion of the processes involved in the evaluation will focus on the issues which can arise when evaluation is tied to the one year purchasing cycle of health commissions, and the problems of evaluating innovative schemes targeted at vulnerable client groups. As time and the research budget (€1000)were limited, the main evaluation tool used was a questionnaire given to caring relatives, to assess their perceptions of their support needs and the extent to which these were met by existing services. An RCT was ruled out because there were strong ethical objections, from the voluntary agency funding the scheme, to subjecting carers of a dying relative to randomization. Additionally, there were problems in recruiting carers for what was a new service. Therefore a supply of ’subjects’ for randomization was not available. Instead, an attempt was made to recruit a comparison group of carers who were receiving conventional support, e.g. district nursing and social work, from a neighbouring area. It was intended to recruit 30 carers into each group in order to permit statistical comparisons. The local medical ethics committee in the trial district approved the proposal, but there was a 4-month delay in obtaining approval in the comparison district, and it was not possible to obtain data from this district before the funding deadline. In the study period (May-November), only eight carers used the scheme, far fewer than had been anticipated. Recruitment was slow, firstly, because of the time required to publicize the scheme; secondly, because carers tended to stay with the scheme for several months, tying up its limited resources; and thirdly, because of the need for formal identification that someone was near to death. Some GPs refused to z zyxwvutsrqpo 0 1997 Blackwell Science Ltd, Health and Social Care in the Community 5(4), 255-260 Evaluating community health care services enter carers into the scheme as they felt there were ethical problems in divulging a terminal diagnosis. Other professionals were also put off by having to certify that the relative being cared for had less than 1 year to live. By insisting that the agency accepted only those who were terminally ill, the commission was attempting to ensure that they only funded ‘health’ and not ‘social’ care. However arbitrary this distinction may sometimes be, it is a fact of current organizational life in health and social services that one type of organization will be reluctant to fund a service which may overlap with the remit of the other. The action research format of the evaluation, which involved regular meetings between the researcher, the agency and the commissioning manager, made it fairly easy to renegotiate the terms of entry to the scheme when problems arose. Eventually the requirement, to certify that the relative being cared for had less than 1year to live, was relaxed. Of the eight carers who received intensive support, five returned questionnaires. These carers were very positive and rated the scheme more favourably than conventional services. However, with such small numbers, and in the absence of a comparison group, no statistical conclusions could be drawn. The evaluation of the scheme could only recommend that it should continue for another year for further data collection. This recommendation was accepted by the health commission. A number of lessons can be drawn from this case study. Firstly, it is difficult to undertake RCTs with small-scale, innovative health care interventions, particularly in sensitive areas. Secondly, short-term funding cycles are incompatible with the time-scales required for ethical approval and data collection. Thirdly, the incidental ’action research element of evaluations (in this case modifying the terms of reference of the carer scheme in order to improve recruitment) may be as useful as formal data collection and analysis. zyx and ethically acceptable may not be politically possible because ethical committees or gatekeeper organizations do not allow it. Organizational barriers need to be addressed, particularly with respect to ethics committees. Problems with multicentre ethical approval are widespread in the UK (Garfield 1995, Middle et a / . 1995, While 1995).A recent paper by Alberti (1995) calling for nationally coordinated ethics committees with common application forms is to be welcomed, and may pave the way for essential multicentre research. In order to foster community services evaluation, multicentre research is needed to provide controls for local initiatives. We would also recommend that researchers in primary health care consider using fieldworkers or other professionals, as well as GI‘s, to recruit patients to trials. Where appropriate, client perspectives can be used to inform and sharpen an RCT design. They may also be used as meaningful outcomes in their own right. The client’s own perspective may be the most valid source of information on service value, particularly when desired outcomes are variable and difficult to define. In the community context, where the focus of modern care is at the level of the individual, the process as well as the outcome of services are equally important. A more participative approach to research can have advantages in terms of learning quickly, from experience, about the process of service implementation. Single experiments that yield non-significant results should not necessarily be ignored. Decisions often have to be based on imperfect knowledge, especially in applied research. It must be remembered that both experimental and non-experimental research, using either quantitative or qualitative data, can be flawed. No one method is inherently better than the other. All that can be said is that one may be more suited to a particular research problem. It is our view that a broad pragmatic approach to methodology is required if research is to be relevant to holistic community- based practice. zy zyxwvu Conclusions The two case studies are at opposite ends of the spectrum, in terms of scale and methodology. However, both were problematic because of tensions between the requirements of ethical committees, health authorities, medical practitioners, clients and researchers. There is a long standing cultural divide between researchers, practitioners and managers (Roland 1995)and the difficulties illustrated by the case studies are, in our experience, endemic to non-medical health service evaluation. Systematic, replicable evaluations of complex interventions require the cooperation of a number of parties with different agendas, and may be more difficult to achieve than has been suggested by recent policy papers (Culyer 1994).What may be methodologically desirable 0 1997 Blackwell Science Ltd, Health Acknowledgements The Newcastle domiciliary physiotherapy trial was funded by the Sainsbury Family Trust Fund. The following project advisory group members provided valuable scientific and practical guidance: Prof. John Bond, Dr Barbara Gregson, Mrs Sylvia Hogarth, Dr Judith Hooper, Dr David Parkin, Dr Pauline Pearson and Prof. Rowena Plant. zyxwvuts References Alberti K.G.M.M. (1995) Local research ethics committees. British Medical rouvnal311, 639-40 and Social Care in the Community 5(4), 255-260 259 zyxwvutsrqpo zyxwvutsrqp zyxwvutsrqp P. Dawson and B. Heyman Altman D.G. (1991) Practical Statistics for Medical Research. Chapman and Hall, London. Appleby J., Smith P. & Ranade W. (1994) Monitoring managed competition. In R. Robinson and J. Le Grand (Eds) Evaluating the NHS Reforms. Kings Fund, London. Brewin C.R. & Bradley C. (1989) Patient preferences and randomised clinical trials. British Medical Journal 299,313-5. Collin C., Wade D.T., Davies S. & Horne V. (1988)The Barthel Index: A reliability study. International Disabdity Studies 10, 61-63. Cook T.D. & Campbell D.T. (1979) Quasi-Experimentation. Design and Analysis lssues for Field Settings. Houghton Mifflin Company, USA. Culyer J. (1994) Funding Research in the NHS.Centre for Health Economics, University of York. Discussion paper 125. Dawson P. (1993) A n evaluation of a new Community Physiotherapy Service in Newcastle upon Tyne. Report to Newcastle District Health Authority and the Sainsbury Family Trust Foundation. Available from Newcastle City Health NHS Trust. Dawson P. (1995) Evaluation of a new Domiciliary Physiotherapy Service in Newcastle upon Tyne. Clinical Rehabilitation 9 (2), 179-81. Department of Health (1990) NHS and Community Care Act. HMSO, London. Garfield P. Cross district comparison of applications to research ethics committees. British Medical Journal 311,660-1. Greenberg G. Clinical trials in general practice. Learning from failures. British Medical Journal 303,940. Heyman B. (1995) Researching User Perspectives on Community Health Care. Chapman and Hall, London. Hunt S.M., McEwen J. & McKenna P. (1986)Measuring Health Status. Croom Helm, London. Jonker P.L. & Sumajow C.A. (1992) Randomised clinical trials in general practice. British Medical Journal 304,508. Kingsley S. & Douglas R. (1991) Developing service strategies: the transition to community care. In A. McNaught (Ed.) Managing Community Health Services. Chapman and Hall, London. Knipschild P. Trials and errors. Alternative thoughts on the methodology of clinical trials. British Medical Journal 306, 1706. MacIntyre I.M.C. (1991)Tribulations for clinical trials. British Medical Journal 302,1099-1000. Middle C., Johnson A., Petty T., Sims L. & Macfarlane A. (1995) Ethics approval for a national postal survey: recent experience. British Medical Journal 311,659-60. Peto V. & Coulter A. (1991)Randomised clinical trials in clinical practice. British Medical Journal303,1549. Pollock C., Freemantle N., Sheldon T. & Song F. (1993) Methodological difficulties in rehabilitation research. Clinical Rehabilitation 7,63-72. Pope C. & Mays N. (1993) Opening the black box: an encounter in the corridors of health services research. British Medical Journal 306,315-8. Rittel H.W.J. & Weber M.M. (1974) Dilemmas in a General Theory of Planning. Petrocelli, New York. Roland M. (1995) Getting research into practice. Journal of Epidemiology and Public Health. 49,225-6. Schwartz D., Flamant R. & Lellouch J. (1980) Clinical Trials. Academic Press, London. Senn S.J. (1991) Falsificationism and clinical trials. Statistics in Medicine 10,1679-92. Taylor K.M. (1992) Integrating conflicting professional roles: physician participation in randomized clinical trials. Social Science and Medicine 35 (2),217-24. Taylor K.M., Margolese R.G. & Soskolne C.L. (1984) Physicians’ reasons for not entering eligible patients into a randomised clinical trial of surgery for breast cancer. New England Journal of Medicine 310,1363-7. Tognoni G., Alli C., Avanzini F., Bettelli G., Colombo F., Corso R., Marchioli R. & Zussino A. (1991) Randomised clinical trials in general practice: lessons from a failure. British Medical Journal 303,969-71. Wade D.T. (1995) Randomized and controlled trials in clinical rehabilitation. Clinical Rehabilitation 9,275-82. Waldron H.A. & Cookson R.F. (1993)Avoiding the pitfalls of multicentre research in general practice. British Medical Journal 307,13314. While A.E. (1995) Ethics committees: impediments to research or guardians of ethical standards. British Medical Journal 311,661. Zelen M. (1979) A New Design for Randomized Clinical Trials. New England Journal of Medicine 300,1243-5. zyxwvutsr zyxwv 260 zy zyxwvu zyxwvutsrqpon 0 1997 Blackwell Science Ltd, Health and Social Care in the Community 5(4), 255-260