Generalisability in economic evaluation studies in healthcare

Generalisability in economic
evaluation studies in
healthcare: a review and
case studies
MJ Sculpher,1* FS Pang,1 A Manca,1
MF Drummond,1 S Golder,2 H Urdahl,1
LM Davies3 and A Eastwood2
1

Centre for Health Economics, University of York, UK

Centre for Reviews and Dissemination, University of York, UK
3
Manchester Medical School, Manchester University, UK
2

* Corresponding author

Executive summary
Health Technology Assessment 2004; Vol. 8: No. 49

Health Technology Assessment

NHS R&D HTA Programme

HTA

Executive summary: Generalisability in economic evaluation studies in healthcare

Executive summary
Background
Given the increasing need for economic evidence
to inform the resource allocation decisions of a
range of decision-makers and in many
jurisdictions, there is interest in the
generalisability of economic evaluations, that is,
the extent to which the results of a study based on
measurement in a particular patient population
and/or a specific context hold true for another
population and/or in a different context. The
context which is the primary focus of this report is
the location in which the study was undertaken
and/or the decision-maker for whom the study
was undertaken. The focus of this report is
economic evaluation as applied to health
services.

Aims and objectives

The aim of the project was to review, and to
develop further, the methods used to assess and to
increase the generalisability of economic
evaluation studies.
The specific objectives were to conduct:
1. A systematic review of methods literature on
generalisability relating to economic evaluation
to identify factors causing variability in costeffectiveness between locations and over time,
and the extent of that variability.
2. A systematic review of methods literature on
economic evaluation relating to available
methods to assess variability between locations
and over time.
3. A systematic review of applied economic
evaluation studies undertaken alongside
multilocation trials to describe how studies
have assessed and reported generalisability and
variability in results between locations.
4. A series of case studies involving the secondary
analysis of cost-effectiveness analyses
undertaken alongside multilocation trials to
explore the use of multilevel modelling to
assess variability in cost-effectiveness between
locations.
5. A structured review of economic evaluations
based on decision analytic models in the field

of osteoporosis to describe how studies have

made their analyses relevant to particular
decision-makers/jurisdictions and assessed how
results might vary across locations.
6. A case study of a decision analytic model to
illustrate methods to estimate cost-effectiveness
for the NHS based on data partly collected in
non-UK locations.

Methods
For Objectives 1 and 2 above, methodological
studies relating to economic evaluation in
healthcare were searched. This included electronic
searches of a range of databases, including
PREMEDLINE, MEDLINE, EMBASE and
EconLit, and manual searches of key journals.
Similar methods were used for Objectives 3 and 5
to identify applied economic studies. The case
studies (Objectives 4 and 6) involved highlighting
specific features of previously published economic
studies related to generalisability and locationrelated variability. In the case of Objective 4, the
case-study was based on the secondary analysis of
three economic studies using data from
randomised trials.

Results
Variability in cost-effectiveness by time
and place

The factor most frequently cited as generating

variability in economic results between locations
was the unit costs associated with particular
resources.
Some of the most frequently cited factors are as
much associated with the measurement of
effectiveness as with cost-effectiveness (e.g. the
artificial characteristics of trials and patient case
mix).
No studies were identified which explicitly
considered factors causing variability
in the results of economic studies over
time.
Several authors have shown important
variations between locations in the volume
and cost of resource use and in cost
effectiveness.

Health Technology Assessment 2004; Vol. 8: No. 49 (Executive summary)

Methods to assess variability in

cost-effectiveness by time and place

In the context of studies based on the analysis of

patient-level data, regression analysis has been
advocated as a means of looking at variability in
economic results across locations. These methods
have generally accepted that some components
of resource use and outcomes are exchangeable
across locations whereas others are not.
Recent studies have also explored, in costeffectiveness analysis, the use of tests of
heterogeneity similar to those used in clinical
evaluation in trials.
The decision analytic model has been the main
means by which cost-effectiveness has been
adapted from trial to non-trial locations. Most
models have focused on changes to the cost side
of the analysis, but it is clear that the
effectiveness side may also need to be adapted
between locations.
The review failed to identify a major literature
on variability in cost-effectiveness over time,
although an emerging literature using Bayesian
decision theory may be of value.

Dealing with variability by location in

economic studies alongside
multilocation trials

There have been weaknesses in some aspects of

the reporting in applied cost-effectiveness studies.
These may limit decision-makers ability to judge
the relevance of a study to their specific situations.
There was little use of the statistical approaches
identified in the methods review to assess
variability by location.
The case study demonstrated the potential
value of multilevel modelling (MLM). Where
clustering exists by location (e.g. centre or
country), MLM can facilitate correct estimates
of the uncertainty in cost-effectiveness results.
MLM also provides a means of estimating
location-specific cost-effectiveness.
The use of location-specific covariates in MLM
can explain some of the variation in costeffectiveness.
An important policy issue is raised by this work:
the extent to which location-specific estimates of
incremental net benefit are useful to decision
makers.

Use of decision analytic models to

provide location-specific estimates of
cost-effectiveness

The review of applied economic studies based

on decision analytic models showed that few
studies were explicit about their target decisionmaker(s)/jurisdictions.

The studies in the review generally made more

effort to ensure that their cost inputs were
specific to their target jurisdiction than their
effectiveness parameters.
Standard sensitivity analysis was the main way of
dealing with uncertainty in the models,
although few studies looked explicitly at
variability between locations.
The modelling case study illustrated how
effectiveness and cost data can be made
location-specific. In particular, on the
effectiveness side, the example showed the
separation of location-specific baseline events
and pooled estimates of relative treatment
effect, where the latter are assumed
exchangeable across locations.

Key recommendations
Economic evaluation using patient-level
data

At the design stage of a study, selection of study

sites should ideally focus on those that are
representative of the jurisdiction(s) for which
economic data are required.
There is value in collecting data on the
characteristics of trial centres which could be
used as covariates in regression models.
The patients included in studies should reflect
the normal clinical caseload, but it is important
to collect a number of patient-level variables
that could be used as covariates.
Resource use data (e.g. hospital days) should be
reported separately from the unit costs of those
resources.
MLM should be considered as a means of
assessing the degree of clustering in cost and
effectiveness data within trial locations. If
clustering is extensive, MLM can reflect this
characteristic at the analysis stage and generate
location-specific estimates of cost-effectiveness.
There remains an important role for sensitivity
analysis in exploring the implications of
variation in some parameters (e.g. unit costs
and preference values).
Reporting more information on the
centres/countries in a study can assist decisionmakers in interpreting the relevance of results
to their situation.

Economic evaluation using decision

analytic modelling

Given the focus on a decision, any analysis

should be clear about the specification of
the decision problem and the relevant
decision-maker(s) and jurisdiction(s).

Executive summary: Generalisability in economic evaluation studies in healthcare

The overall analytical approach, model

structure and data inputs should be appropriate
to the relevant decision-maker(s).
Where several sources of data exist for a
particular parameter, these should be pooled in
such a way that the uncertainty relating to their
precision and possible heterogeneity (including
that related to location) is reflected in the
model.
It is important to distinguish parameter
uncertainty from variability or heterogeneity,
where the latter is concerned with how
parameter estimates vary across contexts.
Probabilistic analysis, where data inputs are
incorporated as random variables, is the
appropriate means of handling parameter
uncertainty.
When a model is targeted at more than one
decision-maker/jurisdiction, an important aspect
of the analysis is to assess the variability in
results between locations, for example, using
sensitivity or scenario analysis.

discussed extensively in the literature relating to

both trial-based and modelling studies.
Regression-based methods are likely to offer a
systematic approach to quantifying variability in
patient-level data. In particular, MLM has the
potential to facilitate estimates of cost-effectiveness
which both reflect the variation in costs and
outcomes between locations and also enable the
consistency of cost-effectiveness estimates between
locations to be assessed directly. Decision analytic
models will retain an important role in adapting
the results of cost-effectiveness studies between
locations.

Summary of recommendations
for further research
Drawing on the material in this report, it is
possible to summarise some important areas for
further research. As far as possible, these have
been placed in priority order.

Conclusions
A large number of factors are mentioned in the
literature that might be expected to generate
variation in the cost-effectiveness of healthcare
interventions across locations. Several papers have
demonstrated differences in the volume and cost
of resource use between locations, but few studies
have looked at variability in outcomes.

In applied trial-based cost-effectiveness studies,

few studies provide sufficient evidence for
decision-makers to establish the relevance or to
adjust the results of the study to their location of
interest. Very few studies utilised statistical
methods formally to assess the variability in results
between locations. In applied economic studies
based on decision models, most studies either
stated their target decision-maker/jurisdiction or
provided sufficient information from which this
could be inferred. There was a greater tendency to
ensure that cost inputs were specific to the target
jurisdiction than clinical parameters.
Methods to assess generalisability and variability
in economic evaluation studies have been

The development of methods of evidence

synthesis which model the exchangeability of
data across locations and allow for the
additional uncertainty in this process. These
methods should relate to all parameters
relevant to economic evaluation.
Assessment of alternative approaches to
specifying multilevel models to the analysis of
cost-effectiveness data alongside multilocation
randomised trials.
Identification of a range of appropriate
covariates relating to locations (e.g. hospitals) in
multilevel models.
Further assessment of the role of econometric
methods (e.g. selection models) for costeffectiveness analysis alongside observational
datasets, and to increase the generalisability of
randomised trials.

Publication
Sculpher MJ, Pang FS, Manca A, Drummond MF,
Golder S, Urdahl H, et al. Generalisability in
economic evaluation studies in healthcare: a
review and case studies. Health Technol Assess
2004;8(49).

HTA
How to obtain copies of this and other HTA Programme reports.
An electronic version of this publication, in Adobe Acrobat format, is available for downloading free of
charge for personal use from the HTA website (http://www.hta.ac.uk). A fully searchable CD-ROM is
also available (see below).
Printed copies of HTA monographs cost 20 each (post and packing free in the UK) to both public and
private sector purchasers from our Despatch Agents.
Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is
2 per monograph and for the rest of the world 3 per monograph.
You can order HTA monographs from our Despatch Agents:
fax (with credit card or official purchase order)
post (with credit card or official purchase order or cheque)
phone during office hours (credit card only).
Additionally the HTA website allows you either to pay securely by credit card or to print out your
order and then post or fax it.
Contact details are as follows:
HTA Despatch
c/o Direct Mail Works Ltd
4 Oakwood Business Centre
Downley, HAVANT PO9 2NP, UK

Email: [email protected]
Tel: 02392 492 000
Fax: 02392 478 555
Fax from outside the UK: +44 2392 478 555

NHS libraries can subscribe free of charge. Public libraries can subscribe at a very reduced cost of
100 for each volume (normally comprising 3040 titles). The commercial subscription rate is 300
per volume. Please see our website for details. Subscriptions can only be purchased for the current or
forthcoming volume.
Payment methods
Paying by cheque
If you pay by cheque, the cheque must be in pounds sterling, made payable to Direct Mail Works Ltd
and drawn on a bank with a UK address.
Paying by credit card
The following cards are accepted by phone, fax, post or via the website ordering pages: Delta, Eurocard,
Mastercard, Solo, Switch and Visa. We advise against sending credit card details in a plain email.
Paying by official purchase order
You can post or fax these, but they must be from public bodies (i.e. NHS or universities) within the UK.
We cannot at present accept purchase orders from commercial companies or from outside the UK.
How do I get a copy of HTA on CD?
Please use the form on the HTA website (www.hta.ac.uk/htacd.htm). Or contact Direct Mail Works (see
contact details above) by email, post, fax or phone. HTA on CD is currently free of charge worldwide.
The website also provides information about the HTA Programme and lists the membership of the various
committees.

NHS R&D HTA Programme

he research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly
influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE)
and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care.
HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key
component of the National Knowledge Service that is being developed to improve the evidence of
clinical practice throughout the NHS.
The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on
the costs, effectiveness and broader impact of health technologies is produced in the most efficient way
for those who use, manage and provide care in the NHS. Health technologies are broadly defined to
include all interventions used to promote health, prevent and treat disease, and improve rehabilitation
and long-term care, rather than settings of care.
The HTA programme commissions research only on topics where it has identified key gaps in the
evidence needed by the NHS. Suggestions for topics are actively sought from people working in the
NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.
Research suggestions are carefully considered by panels of independent experts (including consumers)
whose advice results in a ranked list of recommended research priorities. The HTA Programme then
commissions the research team best suited to undertake the work, in the manner most appropriate to find
the relevant answers. Some projects may take only months, others need several years to answer the
research questions adequately. They may involve synthesising existing evidence or designing a trial to
produce new evidence where none currently exists.
Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is
able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a
National Clinical Director. TARs bring together evidence on key aspects of the use of specific
technologies and usually have to be completed within a limited time period.
Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work commissioned
for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees
and editors.
Reviews in Health Technology Assessment are termed systematic when the account of the search,
appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the
replication of the review by others.
The research reported in this monograph was commissioned by the HTA Programme as project number
98/22/05. As funder, by devising a commissioning brief, the HTA Programme specified the research
question and study design. The authors have been wholly responsible for all data collection, analysis and
interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the
accuracy of the authors report and would like to thank the referees for their constructive comments on
the draft document. However, they do not accept liability for damages or losses arising from material
published in this report.
The views expressed in this publication are those of the authors and not necessarily those of the HTA
Programme or the Department of Health.
Editor-in-Chief:
Series Editors:
Managing Editors:

Professor Tom Walley

Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde,
Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Sally Bailey and Caroline Ciupek

ISSN 1366-5278

Queens Printer and Controller of HMSO 2004

This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journals provided
that suitable acknowledgement is made and the reproduction is not associated with any form of advertising.
Applications for commercial reproduction should be addressed to NCCHTA, Mailpoint 728, Boldrewood, University of Southampton,
Southampton, SO16 7PX, UK.
Published by Gray Publishing, Tunbridge Wells, Kent, on behalf of NCCHTA.
Printed on acid-free paper in the UK by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk.