Viewpoint
Scaling up HIV/AIDS evaluation
Sara Bennett, J Ties Boerma, Ruairí Brugha
The HIV/AIDS pandemic shows little signs of abating,
with an estimated 38 million infections by the end of 2004
and 3 million deaths.1 Yet an effective, expanded,
prevention and treatment response could avert as many as
29 million infections and 10 million deaths by 2020 in
sub-Saharan Africa alone.2 The level of external funding
for HIV/AIDS is on the right trajectory to achieve the goal
of reversing the epidemic by 2015, through radically
scaling up prevention, treatment and care:3 US$1 billion
since 2000 to the World Bank’s multicountry AIDS
program (MAP), US$1·7 billion approved by the Global
Fund to fight AIDS, TB and malaria (2002–05);4 and the
US Congress’ appropriation of US$2·4 billion in 2004 to
the US President’s emergency plan for AIDS relief
(PEPFAR). In the high-prevalence, low-income countries
of southern and eastern Africa, the combined
commitments from these initiatives often amount to over
half the countries’ total health budgets.5 The question
remains, however: will the increased funding achieve the
anticipated outcomes?
UN agencies, with the support of the Global HIV/AIDS
initiatives, have developed monitoring frameworks and
methods for defining commonly accepted programme
indicators.6 However, remarkably little attention has been
paid to longer term, fundamental evaluation questions,
and how to go about answering them. What are the
population health impacts of HIV/AIDS control efforts?
What are the underlying factors causing these impacts?
Finally, how well are the different global initiatives
working together to contribute to the overall goals?
Independent evaluation research should be a
fundamental component of the HIV/AIDS scale-up,
principally for two reasons. First, the global commitment
to scale up HIV/AIDS services is a huge experiment.
Although this experiment cannot be assessed by
controlled trials for political and ethical reasons and the
urgent need to scale up quickly, there is a need for data to
demonstrate effect and secure future funding. Second, the
evidence base for designing effective packages of
interventions against HIV/AIDS is thin.7 Knowledge is
lacking with respect to which combinations of prevention
and treatment strategies work best, and under what
circumstances.
Antiretroviral treatment programmes constitute a
complex challenge for health systems, and virtually no
evidence base exists to guide policy in low-income or
middle-income countries. Synergies between prevention
and treatment strategies are crucial, but the few studies
conducted show mixed results.8,9 This weak evidence base
cannot delay implementation, but scale-up of treatment
programmes must be accompanied by adequate
investment in assessment and operations research. We
review current evaluation plans, summarise evidence gaps
www.thelancet.com Vol 367 January 7, 2006
and obstacles to evaluation scale-up, and propose steps
towards a multicountry, multi-initiative, assessment.
The major HIV/AIDS initiatives have all defined
country-level performance targets. Review of evaluation
plans suggests that early indicators typically focus on
inputs and processes (eg, number of facilities with
capacity to provide antiretroviral treatment, number of
peer and community educators trained); followed in later
years by output (coverage) indicators, such as proportion
of people in need receiving treatment. Effect, such as
number of infections averted, is epidemiologically
modelled rather than directly measured.
The Global Fund intends a 5-year evaluation in 2006,
focusing mainly on its internal organisation and
processes, with epidemiological assessments to follow.
The Institute of Medicine is mandated by Congress to
carry out an evaluation of PEPFAR, the nature of which is
under discussion.10 MAP has recently completed an
independent assessment based mainly on retrospective
case studies but also aiming to use some analysis of effects
in sub-Saharan Africa.11 If the scale-up of HIV/AIDS
control is to be driven by sound evidence of health effects,
then independent, population-based assessments need to
be an integral part of scale-up strategy. The assessments
should be not just initiative specific, nor an afterthought;
but should provide evidence of the overall effects and
benefits of scale-up in different country settings.
The main objectives of evaluations should be to assess
the effect of global HIV/AIDS control efforts on HIV
incidence, AIDS morbidity, and mortality, and to
understand the mechanisms through which these effects
have occurred. Successful strategies to track the spread of
the epidemic, and ensure effective programmes of care,
treatment, and support are highly contingent on
contextual factors, such as epidemiological context,
cultural practices, the status of women, governance, and
health system capacity. Evaluative research is needed to
determine which strategies are effective in which contexts.
Trends in new HIV infections are the best indicator of
effect of prevention measures. Countries with generalised
epidemics (such as those in southern and eastern Africa)
are the largest recipients of funding. In these countries,
antenatal clinic-based surveillance systems—more
recently, complemented by household surveys with HIV
testing—are the prime source of information on HIV
trends.12 More investment is needed to validate and
implement new methods that facilitate direct
measurement of HIV incidence. In countries where HIV
infection is concentrated in high-risk subgroups such as
intravenous drug users and commercial sex workers,
biological and behavioural surveillance of populations at
risk is needed. Focused behavioural studies are essential
for the understanding of why biological trends have
Lancet 2006; 367: 79–82
Abt Associates,
4800 Montgomery Lane,
Suite 600, Bethesda,
MD 20814, USA
(S Bennett PhD); Measurement
and Health Information
Systems, Evidence and
Information for Policy, World
Health Organization, Geneva,
Switzerland (J T Boerma PhD);
and Department of
Epidemiology and Public
Health, Royal College of
Surgeons in Ireland, Dublin,
Ireland (R Brugha MD)
Correspondence to:
Dr Sara Bennett
[email protected]
For MAP see
http://www.worldbank.org
For the Global Fund see
http://www.theglobalfund.org
For PEPFAR see
http://www.state.gov/s/gac/plan
79
Viewpoint
arisen. The absence of good behavioural trend data to
understand the Uganda HIV prevalence decline in the
1990s has allowed political interests to control a debate on
the relative contributions of different components of
sexual behaviour, notably abstinence, faithfulness, and
condom use.13 Measurement of sexual behaviour trends in
generalised epidemics is fraught with difficulties,14 and
systematic investment and planning in this area are
needed (panel 1).
To ascertain the effect of treatment, clinical data on
survival and quality of life in HIV-infected persons are
needed, complemented by population estimates of AIDS
mortality. These estimates may rely on population-level
data from vital registration systems, where available. In
view of their weaknesses in most low-income countries,
evidence could be drawn from special studies where high
quality data collection is possible—eg, sites with
demographic surveillance systems—and complemented
by modelling. To enhance understanding of how
contextual factors interact with and affect the success of
different intervention packages, multicountry studies are
required, with countries being purposefully selected to
indicate different relevant contextual factors (such as
political leadership or health system capacity). Multilevel
analysis could be used to investigate and disentangle how
country-level, community, and individual-level factors
affect or determine health outcomes, as well as the extent
to which variability in outcomes between countries is
Panel 1: What is needed to assess global HIV/AIDS control
efforts
What we need to know
● Contextual factors and their interaction with interventions
(epidemiological context, sex issues, political leadership,
health system capacity, etc)
● Exposure to interventions
● Trends in HIV incidence rates
● Trends in risk behaviours
● Trends in HIV/AIDS mortality
● Quality of life and survival of AIDS patients
● Interactive effects of different initiatives
● Wider health-system effects
Types of data sources
● Basic data on epidemiology, socioeconomic conditions
(eg, literacy, gender equality), health system assessments,
and sociological or political studies as needed.
● Systematic documentation of coverage and quality of
interventions
● Strengthened HIV surveillance systems
● General population-based surveys with HIV testing and
behavioural data
● Clinical data from patient records
● Risk population biological and behavioural surveys
● Health system analyses
80
explained by variables defined at each level.15,16 Whereas
multilevel approaches have been applied to understanding
the epidemiology of HIV/AIDS,17 their application to
evaluation design needs to be further developed.
In addition to learning how health system capacity
changes the effectiveness of intervention strategies, the
effects of the new global HIV/AIDS financing initiatives
on health systems, and the delivery of other (nonHIV/AIDS) services also need to be tracked. In the context
of weak and over-stretched health systems in the poorest
countries, the new global AIDS initiatives may have
perverse effects, distorting priorities and undermining the
coverage and quality of other services, such as maternal
and child health, or malaria control.18 Such an evaluation
component could assess effects on the delivery and
utilisation of key non-HIV/AIDS services; and on the
numbers, distribution, and skills of health workers,19
among other indicators.
There are many obstacles to scaling up assessments of
HIV/AIDS initiatives (panel 2). First, evaluations of such
initiatives provide benefits that accrue not only to those at
country level, but to a broad spectrum of global
stakeholders.20 Neither country-level stakeholders nor any
single global-level stakeholder has sufficient incentive or is
willing, alone, to invest sufficiently in evaluation. For
collective action to occur, stakeholders need to come
together and agree to support the investment jointly.
Multicountry studies are powerful ways to generate
evidence from an uneven set of interventions and
programmes especially in the messy and complex realities
of developing country health systems.21 They enable an
understanding of why certain effects have arisen in some
settings but not in others, recognising that
implementation strategies and interventions packages will
vary between countries. Comparability in a multicountry
study requires common protocols and standardised
outcome indicators, the development of which also
presents a collective action challenge.
A second challenge is the problem posed by attribution,
which is both politically sensitive and technically
complex. High-prevalence countries, where assessments
are most needed, are often recipients of substantial
support from multiple global HIV/AIDS initiatives,
which makes for complex interactions. Initiatives can
play complementary parts, working through different
channels: the Global Fund, through country coordinating
mechanisms often dominated by ministries of health,22
MAP through National AIDS Councils, and PEPFAR
Panel 2: Barriers to scaling up HIV/AIDS assessments
●
●
●
●
●
Collective action needed
Technical and political issues in attribution
Concern about unfavourable results
Weak country assessment capacity
Funding constraints
www.thelancet.com Vol 367 January 7, 2006
Viewpoint
directly through large non-governmental organisations.
Coordinated action around prevention, treatment and
support at the district and facility level, which is the ideal,
would preclude valid attribution of effects to individual
global HIV/AIDS initiatives.
However, political pressure on each initiative to
demonstrate its achievements, so as to secure future
revenues, could impel individual initiatives to count and
claim numbers of people who are on treatment. Such
narrowly focused assessments will (at best) be of little use
in that initiatives are operationally tightly entwined. At
worst, they could lead to absurd claims and undermine
coordination, threatening global control efforts. The
knock-on effect for countries (and third challenge) is that
large recipients of HIV/AIDS funding are understandably
often concerned about assessment findings, which they
feel might affect the way their country is perceived and
jeopardise future funding from initiatives. Countries need
to be protected from potentially adverse consequences of
participating in evaluations, through making participation
a positive condition for future programme funding.
Evaluators also need protection to maintain independence
in the face of strong political and technical preferences on
the part of the HIV/AIDS initiatives.
Fourth, countries may also be hesitant to undertake the
rigorous kind of evaluations needed, because of scarcity of
country level research and evaluation capacity, and weak
links between researchers and policymakers. This
problem was the primary reason identified by Richard
Feachem, executive director of the Global Fund, to explain
low spending on operational research.23 Historically less
than 10% of health research has been conducted in
developing countries, and much of that has been
undertaken by northern researchers.24 The problem is
exacerbated by weak health information systems in many
settings; for example, service statistics and surveillance
data from health facilities are often incomplete and
inaccurate, population based surveys are undertaken
intermittently, and vital registration systems rarely
function.
Finally, there seems to be a funding constraint. Studies
by the Global Forum for Health Research25 and the
WHO26 propose that 5% of development assistance funds
should be dedicated to operational research. If this
recommendation was applied to HIV/AIDS for 2004, at
least US$180 million should have been spent on
operational research and assessment for HIV/AIDS
control.27 Current spending is unknown but certainly far
less. The multicountry evaluation of the integrated
management of childhood illness initiative had a budget
of $10 million over a 5-year period. The HIV/AIDS
initiatives will be considerably more complicated than
the integrated management of childhood illness initiative
to assess, in view of the range and complexity of control
strategies and types of studies required. Nonetheless,
we believe that an adequate 3-year, multicountry
evaluation could be mounted for around an additional
www.thelancet.com Vol 367 January 7, 2006
US$3–6 million per high-prevalence country, building on
planned and previous data collection efforts. This figure
amounts to around 1% of current external funding
commitments to HIV/AIDS: a fairly small investment
relative to the potential future gains from a coordinated
and rigorous assessment. The apparent funding
constraint is perhaps better viewed as a lack of
commitment to, or appreciation of the importance of,
research that assesses HIV/AIDS initiatives.
How should a coordinated evaluation be implemented?
The biggest stumbling block to a coordinated assessment
is the incentives that the global HIV/AIDS initiatives—
Global Fund, PEPFAR, and MAP—face at present.
Without changes in incentives the HIV/AIDS initiatives
will continue to pursue restricted, initiative-specific
assessments. Changing the incentive structures for
assessment should be a priority for technical and funding
agencies, and the activist community. Technical agencies
such as WHO and UNAIDS need to argue forcefully for
rigorous independent assessment that does not seek to
attribute specific effects to different parties, but rather
measures the joint effects of global HIV/AIDS control
efforts and analyses how the different initiatives are
working together to achieve them. Activists and funding
agencies need to recognise the shortsighted nature of
creating competition between initiatives, when at the
country level they need to work together. Openness and
support of appropriate assessments on the part of the
global HIV/AIDS initiatives should be rewarded by the
funding decisions of their major donors, which have the
greatest leverage to bring about coordination. Ultimately
the success of such strategies will depend on the political
will of the wealthiest nations and their readiness to set the
global good above short-term political gain.
In commissioning evaluations, independence from the
global HIV/AIDS initiatives is essential to protect
researchers from pressures stemming from the interests
of individual initiatives. Furthermore, thought should be
given as to how to encourage, rather than deter, country
stakeholders who are likely to be concerned about how
negative findings could affect future funding.
Mechanisms to support the independence of assessments
could include the establishment of an intermediate
partner (such as a small independent secretariat, perhaps
located in a multilateral agency) to act as a buffer and
support implementation; and direct funding from a
consortium of the main donors currently supporting the
global HIV/AIDS initiatives, to countries or the
secretariat. There needs to be an international effort to
coordinate the development of common assessment
protocols that can be adapted to local contexts. Continued
communication between assessment teams during
implementation could be facilitated by an international
research network.
Country researchers should lead individual country
studies and be engaged at all stages of the evaluation
design, implementation, and analysis, with support from
For further details on current
global health funding see
http://www.gatesfoundation.org/
globalhealth/grants
81
Viewpoint
the global research community, when and where
complementary research skills are required. Within
countries, careful selection of settings and coordination
between research activities would enable triangulation of
findings, where qualitative, behavioural and health
systems studies could help explain epidemiological findings. This kind of large-scale, multicountry assessment
could also make a significant contribution to building
research capacity, which is a major constraint to
improving health services and health status in many low
income countries.28
The many global stakeholders who are committed to
stopping the HIV/AIDS pandemic in its tracks and
alleviating its dire social consequences have been
cavalier in their disregard for the importance of evidence
that assesses control efforts. Global HIV/AIDS
initiatives are now well into the implementation phase;
yet there are currently no concrete plans to conduct
rigorous multicountry evaluations of global HIV/AIDS
control efforts. Assessments of their effects require
baseline data, as well as necessarily lengthy
consultations with stakeholders during the design
phase. The window of opportunity for setting up a
multicountry impact assessment is receding. A
comprehensive and coordinated assessment to
determine effect and understand the effects of different
intervention packages, building on the comparative
strengths of different initiatives is needed.
Spending on assessment should not be seen as a
distraction from the urgency to save lives now, but rather
as an investment that has the power to save lives over the
medium to long term. If the global community fails to
invest now in the development of this knowledge base, in
5 years we will be no better informed than we are at
present about which control approaches work and which
do not, especially in countries with high HIV/AIDS
prevalence and weak health systems, where AIDS control
presents the greatest challenge. Such ignorance could
irreparably reverse the current global commitment
towards mobilising the necessary resources to bring about
a world free of AIDS.
Conflict of interest statement
T Boerma, in his capacity as a WHO staff member, has provided advice to
the technical evaluation reference group of the Global Fund. WHO has
received a grant from PEPFAR for work that T Boerma is leading. All
other authors declare that they have no conflict of interest.
References
1
AIDS epidemic update 2004. Geneva: UNAIDS and WHO,
December 2004.
2
Salomon JA, Hogan DR, Stover J, et al. Integrating HIV prevention
and treatment: from slogans to impact. PLoS Med 2005; 2: e16.
3
Millennium Project. Investing in development, a practical plan to
achieve the millennium development goals. New York: United
Nations Development Programme, 2005.
4
Global Fund. Investing in the future, the Global Fund at three years.
Geneva: the Global Fund to fight AIDS, tuberculosis and malaria.
http://www.theglobalfund.org/en/about/replenishment (accessed
March 18, 2005).
5
Lewis M. Addressing the challenge of HIV/AIDS: macroeconomic,
fiscal and institutional issues. Working paper no 5. Washington, DC:
Center for Global Development, 2005.
82
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
UNAIDS, WHO, UNICEF, FHI, Measure Evaluation and USAID.
National AIDS programmes: a guide to monitoring and evaluating
HIV/AIDS care and support. http://pdf.dec.org/pdf_docs/PNACX
519.pdf (accessed October, 2005).
Hogan D, Salomon J. Prevention and treatment of human
immunodeficiency virus/acquired immunodeficiency syndrome
in resource limited settings. Bull World Health Organ 2005; 83:
135–43.
Moatti JP, Prudhomme J, Traore DC, Juillet-Amari A, Akribi HA,
Msellati P. Access to antiretroviral treatment and sexual behaviours
of HIV-infected patients aware of their serostatus in Côte d’Ivoire.
AIDS 2003; 17 (suppl 3): S69–77.
Scheer S, Chu PL, Klausner JD, Katz MH, Schwarcz SK. Effect
of highly active antiretroviral therapy on diagnoses of sexually
transmitted diseases in people with AIDS. Lancet 2001; 357: 432–35.
National Academies of Science. Committee on President’s
emergency plan for AIDS Relief [PEPFAR] implementation
evaluation. http://www4.nationalacademies.org/webcr.nsf/Project
ScopeDisplay/BOGH-H-04-01-A?OpenDocument (accessed April 4,
2005).
World Bank. Committing to results: improving the effectiveness of
HIV/AIDS assistance. Washington DC: World Bank, 2005.
Boerma JT, Ghys P, Walker N. HIV estimates from national
population-based surveys as a new gold standard. Lancet 2003; 362:
1929–31.
Singh S, Darroch JE, Bankole A. A, B and C in Uganda: the roles of
abstinence, monogamy and condom use in HIV decline.
Reprod Health Matters 2004; 12: 129–31.
Cleland J, Boerma JT, Carael M, Weir SS. Monitoring sexual
behaviour in general populations: a synthesis of lessons of the past
decade. Sex Transm Infect 2004; 80 (suppl II): i1–7.
Duncan C, Jones K, Moon G. Context, composition and
heterogeneity: using multilevel models in health research.
Soc Sci Med 1998; 96: 97–117.
Diez Roux A, Schwartz S, Susser E. Ecological variables, ecological
studies, and multilevel studies in public health research. In:
Detels R, Mcewen J, Beaglehole R, Tanaka H, eds. Oxford textbook
of public health, 4th edn. Oxford: Oxford University Press, 2002:
504.
Aral SO, Padian NS, Holmes KK. Advances in multilevel approaches
to understanding the epidemiology and prevention of sexually
transmitted infections and HIV: an overview. J Infect Dis 2005;
191 (suppl 1): S1–6.
Systemwide Effects of the Fund (SWEF) Research Network.
Common research protocol. Monitoring and evaluating the health
system-wide effects of the global fund to fight AIDS, tuberculosis and
malaria. http://www.theglobalfund.org/en/files/links_resources/
library/studies/EF2_full.pdf (accessed October, 2005).
Joint Learning Initiative. Human resources for health: overcoming
the crisis. http://www.globalhealthtrust.org/Report.html (accessed
October, 2005).
Kaul I, Faust M. Global public goods and health: taking the agenda
forward. Bull World Health Organ 2001; 79: 869–74.
Bryce J, Victora CG, Habicht JP, Vaughan JP, Black RE. The
multi-country evaluation of the integrated management of childhood
illness strategy: lessons for the evaluation of public health
interventions. Am J Public Health 2004; 94: 406–15.
Brugha R, Donoghue M, Starling M, et al. The Global Fund:
managing great expectations. Lancet 2004; 364: 95–100.
Feachem R. Editorial: the research imperative: fighting AIDS, TB and
malaria. Trop Med Int Health 2004; 9: 1139–41.
Global Forum for Health Research. 10/90 Report on Health
Research 2003–2004. Geneva: Global Forum for Health Research,
2004.
Global Forum for Health Research. Monitoring financial flows for
health research. Geneva: Global Forum for Health Research, 2004.
Commission on Macroeconomics and Health. Macroeconomics and
health: investing in health for economic development. Geneva:
WHO, 2001.
Organization for Economic Cooperation and Development (OECD).
International development statistics online. http://www.oecd.
org/dataoecd/50/17/5037721.htm (accessed April, 2005).
WHO. World report on knowledge for better health. Geneva: WHO,
2004.
www.thelancet.com Vol 367 January 7, 2006