Academia.eduAcademia.edu

Scaling up HIV/AIDS evaluation

2006, The Lancet

AI-generated Abstract

Scaling up evaluations of HIV/AIDS interventions is crucial for understanding their effects and improving global health strategies. Current evaluation efforts largely focus on process indicators rather than direct measurements of health outcomes. To effectively assess HIV/AIDS control efforts, independent, population-based evaluations must be integrated into national strategies, emphasizing the need for contextual understanding and robust research capacity. The paper argues for a coordinated, multicountry approach to evaluation, highlighting the importance of evidence-based understanding to guide intervention effectiveness and resource allocation.

Viewpoint Scaling up HIV/AIDS evaluation Sara Bennett, J Ties Boerma, Ruairí Brugha The HIV/AIDS pandemic shows little signs of abating, with an estimated 38 million infections by the end of 2004 and 3 million deaths.1 Yet an effective, expanded, prevention and treatment response could avert as many as 29 million infections and 10 million deaths by 2020 in sub-Saharan Africa alone.2 The level of external funding for HIV/AIDS is on the right trajectory to achieve the goal of reversing the epidemic by 2015, through radically scaling up prevention, treatment and care:3 US$1 billion since 2000 to the World Bank’s multicountry AIDS program (MAP), US$1·7 billion approved by the Global Fund to fight AIDS, TB and malaria (2002–05);4 and the US Congress’ appropriation of US$2·4 billion in 2004 to the US President’s emergency plan for AIDS relief (PEPFAR). In the high-prevalence, low-income countries of southern and eastern Africa, the combined commitments from these initiatives often amount to over half the countries’ total health budgets.5 The question remains, however: will the increased funding achieve the anticipated outcomes? UN agencies, with the support of the Global HIV/AIDS initiatives, have developed monitoring frameworks and methods for defining commonly accepted programme indicators.6 However, remarkably little attention has been paid to longer term, fundamental evaluation questions, and how to go about answering them. What are the population health impacts of HIV/AIDS control efforts? What are the underlying factors causing these impacts? Finally, how well are the different global initiatives working together to contribute to the overall goals? Independent evaluation research should be a fundamental component of the HIV/AIDS scale-up, principally for two reasons. First, the global commitment to scale up HIV/AIDS services is a huge experiment. Although this experiment cannot be assessed by controlled trials for political and ethical reasons and the urgent need to scale up quickly, there is a need for data to demonstrate effect and secure future funding. Second, the evidence base for designing effective packages of interventions against HIV/AIDS is thin.7 Knowledge is lacking with respect to which combinations of prevention and treatment strategies work best, and under what circumstances. Antiretroviral treatment programmes constitute a complex challenge for health systems, and virtually no evidence base exists to guide policy in low-income or middle-income countries. Synergies between prevention and treatment strategies are crucial, but the few studies conducted show mixed results.8,9 This weak evidence base cannot delay implementation, but scale-up of treatment programmes must be accompanied by adequate investment in assessment and operations research. We review current evaluation plans, summarise evidence gaps www.thelancet.com Vol 367 January 7, 2006 and obstacles to evaluation scale-up, and propose steps towards a multicountry, multi-initiative, assessment. The major HIV/AIDS initiatives have all defined country-level performance targets. Review of evaluation plans suggests that early indicators typically focus on inputs and processes (eg, number of facilities with capacity to provide antiretroviral treatment, number of peer and community educators trained); followed in later years by output (coverage) indicators, such as proportion of people in need receiving treatment. Effect, such as number of infections averted, is epidemiologically modelled rather than directly measured. The Global Fund intends a 5-year evaluation in 2006, focusing mainly on its internal organisation and processes, with epidemiological assessments to follow. The Institute of Medicine is mandated by Congress to carry out an evaluation of PEPFAR, the nature of which is under discussion.10 MAP has recently completed an independent assessment based mainly on retrospective case studies but also aiming to use some analysis of effects in sub-Saharan Africa.11 If the scale-up of HIV/AIDS control is to be driven by sound evidence of health effects, then independent, population-based assessments need to be an integral part of scale-up strategy. The assessments should be not just initiative specific, nor an afterthought; but should provide evidence of the overall effects and benefits of scale-up in different country settings. The main objectives of evaluations should be to assess the effect of global HIV/AIDS control efforts on HIV incidence, AIDS morbidity, and mortality, and to understand the mechanisms through which these effects have occurred. Successful strategies to track the spread of the epidemic, and ensure effective programmes of care, treatment, and support are highly contingent on contextual factors, such as epidemiological context, cultural practices, the status of women, governance, and health system capacity. Evaluative research is needed to determine which strategies are effective in which contexts. Trends in new HIV infections are the best indicator of effect of prevention measures. Countries with generalised epidemics (such as those in southern and eastern Africa) are the largest recipients of funding. In these countries, antenatal clinic-based surveillance systems—more recently, complemented by household surveys with HIV testing—are the prime source of information on HIV trends.12 More investment is needed to validate and implement new methods that facilitate direct measurement of HIV incidence. In countries where HIV infection is concentrated in high-risk subgroups such as intravenous drug users and commercial sex workers, biological and behavioural surveillance of populations at risk is needed. Focused behavioural studies are essential for the understanding of why biological trends have Lancet 2006; 367: 79–82 Abt Associates, 4800 Montgomery Lane, Suite 600, Bethesda, MD 20814, USA (S Bennett PhD); Measurement and Health Information Systems, Evidence and Information for Policy, World Health Organization, Geneva, Switzerland (J T Boerma PhD); and Department of Epidemiology and Public Health, Royal College of Surgeons in Ireland, Dublin, Ireland (R Brugha MD) Correspondence to: Dr Sara Bennett [email protected] For MAP see http://www.worldbank.org For the Global Fund see http://www.theglobalfund.org For PEPFAR see http://www.state.gov/s/gac/plan 79 Viewpoint arisen. The absence of good behavioural trend data to understand the Uganda HIV prevalence decline in the 1990s has allowed political interests to control a debate on the relative contributions of different components of sexual behaviour, notably abstinence, faithfulness, and condom use.13 Measurement of sexual behaviour trends in generalised epidemics is fraught with difficulties,14 and systematic investment and planning in this area are needed (panel 1). To ascertain the effect of treatment, clinical data on survival and quality of life in HIV-infected persons are needed, complemented by population estimates of AIDS mortality. These estimates may rely on population-level data from vital registration systems, where available. In view of their weaknesses in most low-income countries, evidence could be drawn from special studies where high quality data collection is possible—eg, sites with demographic surveillance systems—and complemented by modelling. To enhance understanding of how contextual factors interact with and affect the success of different intervention packages, multicountry studies are required, with countries being purposefully selected to indicate different relevant contextual factors (such as political leadership or health system capacity). Multilevel analysis could be used to investigate and disentangle how country-level, community, and individual-level factors affect or determine health outcomes, as well as the extent to which variability in outcomes between countries is Panel 1: What is needed to assess global HIV/AIDS control efforts What we need to know ● Contextual factors and their interaction with interventions (epidemiological context, sex issues, political leadership, health system capacity, etc) ● Exposure to interventions ● Trends in HIV incidence rates ● Trends in risk behaviours ● Trends in HIV/AIDS mortality ● Quality of life and survival of AIDS patients ● Interactive effects of different initiatives ● Wider health-system effects Types of data sources ● Basic data on epidemiology, socioeconomic conditions (eg, literacy, gender equality), health system assessments, and sociological or political studies as needed. ● Systematic documentation of coverage and quality of interventions ● Strengthened HIV surveillance systems ● General population-based surveys with HIV testing and behavioural data ● Clinical data from patient records ● Risk population biological and behavioural surveys ● Health system analyses 80 explained by variables defined at each level.15,16 Whereas multilevel approaches have been applied to understanding the epidemiology of HIV/AIDS,17 their application to evaluation design needs to be further developed. In addition to learning how health system capacity changes the effectiveness of intervention strategies, the effects of the new global HIV/AIDS financing initiatives on health systems, and the delivery of other (nonHIV/AIDS) services also need to be tracked. In the context of weak and over-stretched health systems in the poorest countries, the new global AIDS initiatives may have perverse effects, distorting priorities and undermining the coverage and quality of other services, such as maternal and child health, or malaria control.18 Such an evaluation component could assess effects on the delivery and utilisation of key non-HIV/AIDS services; and on the numbers, distribution, and skills of health workers,19 among other indicators. There are many obstacles to scaling up assessments of HIV/AIDS initiatives (panel 2). First, evaluations of such initiatives provide benefits that accrue not only to those at country level, but to a broad spectrum of global stakeholders.20 Neither country-level stakeholders nor any single global-level stakeholder has sufficient incentive or is willing, alone, to invest sufficiently in evaluation. For collective action to occur, stakeholders need to come together and agree to support the investment jointly. Multicountry studies are powerful ways to generate evidence from an uneven set of interventions and programmes especially in the messy and complex realities of developing country health systems.21 They enable an understanding of why certain effects have arisen in some settings but not in others, recognising that implementation strategies and interventions packages will vary between countries. Comparability in a multicountry study requires common protocols and standardised outcome indicators, the development of which also presents a collective action challenge. A second challenge is the problem posed by attribution, which is both politically sensitive and technically complex. High-prevalence countries, where assessments are most needed, are often recipients of substantial support from multiple global HIV/AIDS initiatives, which makes for complex interactions. Initiatives can play complementary parts, working through different channels: the Global Fund, through country coordinating mechanisms often dominated by ministries of health,22 MAP through National AIDS Councils, and PEPFAR Panel 2: Barriers to scaling up HIV/AIDS assessments ● ● ● ● ● Collective action needed Technical and political issues in attribution Concern about unfavourable results Weak country assessment capacity Funding constraints www.thelancet.com Vol 367 January 7, 2006 Viewpoint directly through large non-governmental organisations. Coordinated action around prevention, treatment and support at the district and facility level, which is the ideal, would preclude valid attribution of effects to individual global HIV/AIDS initiatives. However, political pressure on each initiative to demonstrate its achievements, so as to secure future revenues, could impel individual initiatives to count and claim numbers of people who are on treatment. Such narrowly focused assessments will (at best) be of little use in that initiatives are operationally tightly entwined. At worst, they could lead to absurd claims and undermine coordination, threatening global control efforts. The knock-on effect for countries (and third challenge) is that large recipients of HIV/AIDS funding are understandably often concerned about assessment findings, which they feel might affect the way their country is perceived and jeopardise future funding from initiatives. Countries need to be protected from potentially adverse consequences of participating in evaluations, through making participation a positive condition for future programme funding. Evaluators also need protection to maintain independence in the face of strong political and technical preferences on the part of the HIV/AIDS initiatives. Fourth, countries may also be hesitant to undertake the rigorous kind of evaluations needed, because of scarcity of country level research and evaluation capacity, and weak links between researchers and policymakers. This problem was the primary reason identified by Richard Feachem, executive director of the Global Fund, to explain low spending on operational research.23 Historically less than 10% of health research has been conducted in developing countries, and much of that has been undertaken by northern researchers.24 The problem is exacerbated by weak health information systems in many settings; for example, service statistics and surveillance data from health facilities are often incomplete and inaccurate, population based surveys are undertaken intermittently, and vital registration systems rarely function. Finally, there seems to be a funding constraint. Studies by the Global Forum for Health Research25 and the WHO26 propose that 5% of development assistance funds should be dedicated to operational research. If this recommendation was applied to HIV/AIDS for 2004, at least US$180 million should have been spent on operational research and assessment for HIV/AIDS control.27 Current spending is unknown but certainly far less. The multicountry evaluation of the integrated management of childhood illness initiative had a budget of $10 million over a 5-year period. The HIV/AIDS initiatives will be considerably more complicated than the integrated management of childhood illness initiative to assess, in view of the range and complexity of control strategies and types of studies required. Nonetheless, we believe that an adequate 3-year, multicountry evaluation could be mounted for around an additional www.thelancet.com Vol 367 January 7, 2006 US$3–6 million per high-prevalence country, building on planned and previous data collection efforts. This figure amounts to around 1% of current external funding commitments to HIV/AIDS: a fairly small investment relative to the potential future gains from a coordinated and rigorous assessment. The apparent funding constraint is perhaps better viewed as a lack of commitment to, or appreciation of the importance of, research that assesses HIV/AIDS initiatives. How should a coordinated evaluation be implemented? The biggest stumbling block to a coordinated assessment is the incentives that the global HIV/AIDS initiatives— Global Fund, PEPFAR, and MAP—face at present. Without changes in incentives the HIV/AIDS initiatives will continue to pursue restricted, initiative-specific assessments. Changing the incentive structures for assessment should be a priority for technical and funding agencies, and the activist community. Technical agencies such as WHO and UNAIDS need to argue forcefully for rigorous independent assessment that does not seek to attribute specific effects to different parties, but rather measures the joint effects of global HIV/AIDS control efforts and analyses how the different initiatives are working together to achieve them. Activists and funding agencies need to recognise the shortsighted nature of creating competition between initiatives, when at the country level they need to work together. Openness and support of appropriate assessments on the part of the global HIV/AIDS initiatives should be rewarded by the funding decisions of their major donors, which have the greatest leverage to bring about coordination. Ultimately the success of such strategies will depend on the political will of the wealthiest nations and their readiness to set the global good above short-term political gain. In commissioning evaluations, independence from the global HIV/AIDS initiatives is essential to protect researchers from pressures stemming from the interests of individual initiatives. Furthermore, thought should be given as to how to encourage, rather than deter, country stakeholders who are likely to be concerned about how negative findings could affect future funding. Mechanisms to support the independence of assessments could include the establishment of an intermediate partner (such as a small independent secretariat, perhaps located in a multilateral agency) to act as a buffer and support implementation; and direct funding from a consortium of the main donors currently supporting the global HIV/AIDS initiatives, to countries or the secretariat. There needs to be an international effort to coordinate the development of common assessment protocols that can be adapted to local contexts. Continued communication between assessment teams during implementation could be facilitated by an international research network. Country researchers should lead individual country studies and be engaged at all stages of the evaluation design, implementation, and analysis, with support from For further details on current global health funding see http://www.gatesfoundation.org/ globalhealth/grants 81 Viewpoint the global research community, when and where complementary research skills are required. Within countries, careful selection of settings and coordination between research activities would enable triangulation of findings, where qualitative, behavioural and health systems studies could help explain epidemiological findings. This kind of large-scale, multicountry assessment could also make a significant contribution to building research capacity, which is a major constraint to improving health services and health status in many low income countries.28 The many global stakeholders who are committed to stopping the HIV/AIDS pandemic in its tracks and alleviating its dire social consequences have been cavalier in their disregard for the importance of evidence that assesses control efforts. Global HIV/AIDS initiatives are now well into the implementation phase; yet there are currently no concrete plans to conduct rigorous multicountry evaluations of global HIV/AIDS control efforts. Assessments of their effects require baseline data, as well as necessarily lengthy consultations with stakeholders during the design phase. The window of opportunity for setting up a multicountry impact assessment is receding. A comprehensive and coordinated assessment to determine effect and understand the effects of different intervention packages, building on the comparative strengths of different initiatives is needed. Spending on assessment should not be seen as a distraction from the urgency to save lives now, but rather as an investment that has the power to save lives over the medium to long term. If the global community fails to invest now in the development of this knowledge base, in 5 years we will be no better informed than we are at present about which control approaches work and which do not, especially in countries with high HIV/AIDS prevalence and weak health systems, where AIDS control presents the greatest challenge. Such ignorance could irreparably reverse the current global commitment towards mobilising the necessary resources to bring about a world free of AIDS. Conflict of interest statement T Boerma, in his capacity as a WHO staff member, has provided advice to the technical evaluation reference group of the Global Fund. WHO has received a grant from PEPFAR for work that T Boerma is leading. All other authors declare that they have no conflict of interest. References 1 AIDS epidemic update 2004. Geneva: UNAIDS and WHO, December 2004. 2 Salomon JA, Hogan DR, Stover J, et al. Integrating HIV prevention and treatment: from slogans to impact. PLoS Med 2005; 2: e16. 3 Millennium Project. Investing in development, a practical plan to achieve the millennium development goals. New York: United Nations Development Programme, 2005. 4 Global Fund. Investing in the future, the Global Fund at three years. Geneva: the Global Fund to fight AIDS, tuberculosis and malaria. http://www.theglobalfund.org/en/about/replenishment (accessed March 18, 2005). 5 Lewis M. Addressing the challenge of HIV/AIDS: macroeconomic, fiscal and institutional issues. Working paper no 5. Washington, DC: Center for Global Development, 2005. 82 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 UNAIDS, WHO, UNICEF, FHI, Measure Evaluation and USAID. National AIDS programmes: a guide to monitoring and evaluating HIV/AIDS care and support. http://pdf.dec.org/pdf_docs/PNACX 519.pdf (accessed October, 2005). Hogan D, Salomon J. Prevention and treatment of human immunodeficiency virus/acquired immunodeficiency syndrome in resource limited settings. Bull World Health Organ 2005; 83: 135–43. Moatti JP, Prudhomme J, Traore DC, Juillet-Amari A, Akribi HA, Msellati P. Access to antiretroviral treatment and sexual behaviours of HIV-infected patients aware of their serostatus in Côte d’Ivoire. AIDS 2003; 17 (suppl 3): S69–77. Scheer S, Chu PL, Klausner JD, Katz MH, Schwarcz SK. Effect of highly active antiretroviral therapy on diagnoses of sexually transmitted diseases in people with AIDS. Lancet 2001; 357: 432–35. National Academies of Science. Committee on President’s emergency plan for AIDS Relief [PEPFAR] implementation evaluation. http://www4.nationalacademies.org/webcr.nsf/Project ScopeDisplay/BOGH-H-04-01-A?OpenDocument (accessed April 4, 2005). World Bank. Committing to results: improving the effectiveness of HIV/AIDS assistance. Washington DC: World Bank, 2005. Boerma JT, Ghys P, Walker N. HIV estimates from national population-based surveys as a new gold standard. Lancet 2003; 362: 1929–31. Singh S, Darroch JE, Bankole A. A, B and C in Uganda: the roles of abstinence, monogamy and condom use in HIV decline. Reprod Health Matters 2004; 12: 129–31. Cleland J, Boerma JT, Carael M, Weir SS. Monitoring sexual behaviour in general populations: a synthesis of lessons of the past decade. Sex Transm Infect 2004; 80 (suppl II): i1–7. Duncan C, Jones K, Moon G. Context, composition and heterogeneity: using multilevel models in health research. Soc Sci Med 1998; 96: 97–117. Diez Roux A, Schwartz S, Susser E. Ecological variables, ecological studies, and multilevel studies in public health research. In: Detels R, Mcewen J, Beaglehole R, Tanaka H, eds. Oxford textbook of public health, 4th edn. Oxford: Oxford University Press, 2002: 504. Aral SO, Padian NS, Holmes KK. Advances in multilevel approaches to understanding the epidemiology and prevention of sexually transmitted infections and HIV: an overview. J Infect Dis 2005; 191 (suppl 1): S1–6. Systemwide Effects of the Fund (SWEF) Research Network. Common research protocol. Monitoring and evaluating the health system-wide effects of the global fund to fight AIDS, tuberculosis and malaria. http://www.theglobalfund.org/en/files/links_resources/ library/studies/EF2_full.pdf (accessed October, 2005). Joint Learning Initiative. Human resources for health: overcoming the crisis. http://www.globalhealthtrust.org/Report.html (accessed October, 2005). Kaul I, Faust M. Global public goods and health: taking the agenda forward. Bull World Health Organ 2001; 79: 869–74. Bryce J, Victora CG, Habicht JP, Vaughan JP, Black RE. The multi-country evaluation of the integrated management of childhood illness strategy: lessons for the evaluation of public health interventions. Am J Public Health 2004; 94: 406–15. Brugha R, Donoghue M, Starling M, et al. The Global Fund: managing great expectations. Lancet 2004; 364: 95–100. Feachem R. Editorial: the research imperative: fighting AIDS, TB and malaria. Trop Med Int Health 2004; 9: 1139–41. Global Forum for Health Research. 10/90 Report on Health Research 2003–2004. Geneva: Global Forum for Health Research, 2004. Global Forum for Health Research. Monitoring financial flows for health research. Geneva: Global Forum for Health Research, 2004. Commission on Macroeconomics and Health. Macroeconomics and health: investing in health for economic development. Geneva: WHO, 2001. Organization for Economic Cooperation and Development (OECD). International development statistics online. http://www.oecd. org/dataoecd/50/17/5037721.htm (accessed April, 2005). WHO. World report on knowledge for better health. Geneva: WHO, 2004. www.thelancet.com Vol 367 January 7, 2006