A Guide To Health

Economics For Those

Working in Public Health
A concise desktop handbook

Prepared for Public Health Wales by

Dr Joanna Charles and Prof Rhiannon Tudor Edwards, Bangor University
Contents

Page
1. Purpose of this handbook 2
2. Definitions of key health economics terminology 2
3. What is health economics? 5
4. Purpose of economic evaluation 6
5. Methods of economic evaluation 6
6. Social Return on Investment (SROI) 14

7. Critical appraisal of economic evaluations and

decision analytical models 17
8. Useful health economics resources 27

1
1. Purpose of this concise desktop handbook
This handbook, written by health economists at Bangor University is
intended to provide an introduction and define key economic terms, so
that those without health economics expertise can better understand
and appraise economic evidence. In particular, we are aware that you
may need to find and interpret economic evidence; this handbook is a
quick reference guide to key methods and terminology.
Bold key terms in the handbook are described in detail in the
“Definitions” section below.

2. Definitions of key health economics terminology

List compiled from Berger et al., (2003); Pass et al., (1993) and the BMJ
(2012).
Allocative efficiency – allocation of resources between types of health
services in a way that results in maximum gain to all parties.
Cost-benefit analysis – compares the costs and benefits of an
intervention, procedure or programme in monetary terms.
Cost-consequence analysis – describes the costs and outcomes of an
intervention in a disaggregated form.
Cost-effectiveness acceptability curve – the curve illustrates the
probability of ‘intervention A’ being more cost-effective than
‘intervention B’ given a range of values that a decision-maker may attach
to an additional quality adjusted life year to reflect uncertainty in the
estimates.
Cost-effectiveness analysis – costs are compared with a treatment’s
common therapeutic goal, expressed in terms of one main outcome
measured in natural units (e.g., improvement in blood pressure or
cholesterol level).
Cost-effectiveness plane – a graphic representation of the Incremental
cost-effectiveness ratio. See Incremental cost-effectiveness ratio
definition below.
Cost-minimisation analysis – a method of evaluation utilised when the
intervention, procedures or programmes are expected to have exactly
the same outcome. The analysis then identifies the less costly option.

2
Cost-utility analysis – a method of evaluation that measures health
benefits in preference-based non-monetary units such as Quality
Adjusted Life Years (QALYs) or Disability Adjusted Life Years (DALYs).
Decision analysis – a structured way of thinking about how an action
taken in a current decision would lead to a result, constructed as a logical
model describing the relationships between inputs and results.
Decision analytic modelling – a modelling technique used to estimate
the costs, outcomes and cost-effectiveness of different interventions and
programmes in health care and public health.
Disability Adjusted Life Years (DALYs) – used to generate health related
measures of utility for those living with a disability measured in terms
of time lost due to premature death (mortality) and time lived with a
disability (morbidity).
Discounting – a method of incorporating positive time preference (higher
value given to costs and benefits that occur now, compared to those
occurring in the future) into the evaluation when the costs and benefits
do not occur in the same time period.
Discount rate – the rate chosen to express the strength of preference
over timing of costs and benefits. Since 2003 the Treasury recommends
a 3.5% discount rate. To check the discount rate consult the HM Treasury
Green Book.
Direct medical costs – associated with the service/programme under
consideration. These are organisational and operational costs borne by
the health sector (e.g., health professionals’ time, supplies, equipment,
power etc).
Direct nonmedical costs – incurred by patient/families in the course of
treatment (e.g., transport costs, parking).
Health capital - defined by Grossman (1972) as the present value of a
person’s lifetime health.
Herd immunity – a form of indirect protection from infectious diseases
that occurs when a large percentage of the population becomes immune
to an infection and consequently provides a measure of protection to
those who are not immune.
Incremental cost – the difference between the costs of one intervention
and the costs of its comparator/alternative.

3
Incremental cost-effectiveness ratio (ICER) – obtained by dividing the
difference between the costs of the two interventions by the difference
in the outcomes, i.e. the extra cost per extra unit of outcome.
Indirect costs – losses in production due to absence from work. Indirect
costs can also fall on people other than the person receiving treatment,
for example, other members of the family may need to take time off
work to take a family member to the local GP clinic or A&E department.
Intangible costs – non-physical costs to the patient and their families
from ill health such as pain and anxiety.
Marginal costs – the additional cost increases or savings arising as a
consequence of small output changes within a health care programme.
Important to consider as part of resource allocation alongside wider
considerations such as capacity, staffing and equipment.
Opportunity cost – the value of benefits foregone by not using resources
in their next best alternative use.
Perspective – the point of view from which an analysis is conducted (e.g.,
public sector or societal).
Positive externalities – a benefit that is enjoyed by a third-party as
a result of an economic transaction (e.g., herd immunity through
vaccination programmes).
Quality Adjusted Life Years (QALYs) – calculated by aggregating the
number of years gained from a drug or health care intervention,
weighted by a proportion that represents the relative value attached to a
given health state of quality of life in those years.
Sensitivity analysis – analysis that tests the robustness of an economic
model by examining the changes in results when adjusting key
parameters.
Social Return on Investment (SROI) – analysis that results in a ratio of
benefits to costs, estimating the value created for every £1 invested.
Technical efficiency – the use of health care resources in such a way that
maximises output from given resources or minimises resource use for a
given level of output.

4
3. What is health economics?
Health economics is a sub-discipline of economics, which is the study of
how society uses scarce resources to meet its wants and needs.
Health economics views health and health care as an economic good (as
in goods and services) and is predominately concerned with how society
uses scarce health care resources to meet these wants and needs.
There are three basic economic questions:
1. What goods and services to produce?
2. How can we produce goods and services?
3. How should we distribute goods and services between members
of society?
Economic principles are applied to health and health care because;
- Resources are finite (e.g. there are only so many doctors and local
community services such as GP surgeries and pharmacies that are
open generally during business hours)
- In contrast, demand for health and health care is infinite
- To create a balance between finite resources and infinite wants
and needs, choices are necessary and consequently costs and
benefits must be compared
- Prioritisation is also required for investment and disinvestment
(e.g. do you utilise your scarce resources to implement
interventions to increase physical exercise or reduce tobacco
consumption?)
Public health and health care is different to how other goods and services
operate in a market because:
- Individual ill-health is unpredictable (individuals are not able to
control when they will fall ill, how long it will take them to recover
or how serious the illness is)
- There are indirect consequences to public health and health care,
these include positive externalities (whereby a benefit is enjoyed
by a third-party as a result of an economic transaction) such as
herd immunity through vaccination programmes
- Consumers have limited knowledge. They rely upon providers
(e.g. doctors, nurses, social workers, etc) to provide health care,

5
 medicines, information, interventions and referrals to specialist
services if required
- Doctors, nurses, social workers, etc act as “gatekeepers”, deciding
who requires and receives treatment, and what type of treatment
they receive
- There are educational and financial barriers to entry into
the medical profession, thus maintaining limited consumer
knowledge in the general population
- The demand for health care is a derived demand, created from
the demand for health. Consumers typically want more health
capital and in order to achieve this, individuals allocate resources
in order to both consume and produce health (e.g. by engaging in
health promotion activities such as lifestyle changes)

4. Purpose of economic evaluation

Health economics is interested in the interplay between costs and
outcomes/benefits. Economic evaluation techniques provide a
framework for identifying the costs and benefits of different health
interventions. We conduct economic evaluations, as we need to consider
scarcity of resources and opportunity costs - the value of benefits
foregone by not using resources in their next best alternative use.

5. Methods of economic evaluation

There are five main methods of economic evaluation:
• Cost-Minimisation Analysis (CMA)
• Cost-Effectiveness Analysis (CEA)
• Cost-Utility Analysis (CUA)
• Cost-Benefit Analysis (CBA)
• Cost-Consequence Analysis (CCA)

There are also alternative techniques which include the consideration

of costs and outcomes, but generally the five methods listed above are
most commonly used. A range of methods are listed in Table 1.

6
Method Full or Partial Economic Evaluation
Cost-Minimisation Analysis (CMA) Full Economic Evaluations
Cost-Effectiveness Analysis (CEA)
They compare alternative services/
procedures/interventions in terms
Cost-Utility Analysis (CUA) of both their costs and outcomes/
Cost-Benefit Analysis (CBA) effects/benefits. It is worth noting
that each method measures
outcomes differently
Cost-Consequence Analysis (CCA) Partial Economic Evaluations
Cost-analyses They focus solely on costs or
resources used
Cost-description studies
Cost-outcome descriptions
Table 1. Methods used in health economics categorised by whether or
not they would be considered full or partial economic evaluations.
Each method of economic evaluation is described in turn below, with
limitations highlighted and an example paper listed demonstrating the
method.

5.1 Cost-minimisation analysis (CMA)

Cost-minimisation analysis is a form of economic evaluation used when
an intervention or service and its alternative (e.g. usual care or current
practice) achieve outcomes that are the same (Brazier et al., 2007;
Robinson, 1993a). Under these circumstances, cost-minimisation analysis
aims to identify the least costly option (Brazier et al., 2007, Robinson,
1993a).
Limitations of cost-minimisation analysis
In practice, it is difficult to find interventions or services with the same
outcomes, as there is often uncertainty around the outcome measure
of choice (Brazier et al., 2007). The use of cost-minimisation analysis
highlights questions about the gathering of cost data such as, what
perspective should be chosen. For example, a public sector perspective
would include costs accrued by primary care and NHS secondary care,
personal social services and local government. A societal perspective
would include costs such as provider costs of equipment and staff,

7
individual costs of lost wages, travel and costs to relatives such as
childcare costs or transport. Other questions to consider include should
costs reflect opportunity costs and should one take account of the effects
of inflation and discounting. Discounting is a method of incorporating
positive time preference (higher value given to costs and benefits
that occur now, compared to those occurring in the future) into the
evaluation when the costs and benefits do not occur at the same time
period.
Example of cost-minimisation analysis in practice
Jones J, Wilson A, Parker H, Wynn A, Jagger C, Spiers N, Parker G. (1999).
Economic evaluation of hospital at home versus hospital care: cost
minimisation analysis of data from randomised controlled trial. BMJ, 319
(7224), 1547–1550.

5.2 Cost-effectiveness analysis (CEA)

Cost-effectiveness analysis compares the costs of alternative procedures,
services or interventions with a treatment’s common therapeutic goal,
expressed in terms of one main outcome measured in natural units (e.g.,
improvement in blood pressure or cholesterol level) (Berger et al., 2003).
An incremental cost-effectiveness ratio (ICER) is used in this method.
An ICER calculates the difference in costs between one intervention
and an alternative, divided by the difference in outcomes (OHE, 2008).
Effectiveness data is typically collected from economic evaluations
alongside clinical trials or randomised controlled trials (Robinson, 1993b).
An ICER can also be illustrated graphically using a diagram named the
cost-effectiveness plane, shown in Figure 1.
The horizontal axis represents the difference in effect between the
intervention (I) and the alternative (A). The vertical axis represents the
difference in cost between the intervention and the alternative.
If point I falls in the North East (NE) quadrant, the intervention is more
effective and more costly than the alternative (point A).
If point I falls in the South West (SW) quadrant, the intervention is less
effective and less costly than the alternative (point A).
If point I falls in the South East (SE) quadrant, the intervention is more
effective and less costly than the alternative (point A). In other words it
dominates, (i.e. prevails over) the alternative and would be considered
cost-effective.
8
If point I is in the North West (NW) quadrant, the intervention is less
effective and more costly than the alternative (point A).
If point A falls in either the NE and SW quadrants, the choice of whether
to implement the intervention or the alternative depends upon the
maximum cost-effectiveness ratio one is willing to accept. The slope of
the line IA gives the cost-effectiveness ratio.
Figure 1. Example cost-effectiveness plane adapted from Black (1990).
An amended version of the cost-effectiveness plane diagram from
Black, W. C. (1990)” . The CE Plane: A Graphic Representation of Cost-
Effectiveness. Medical Decision Making Vol. 10 (3) pp. 212-214. Copyright
© 1990 by Society for Medical Decision Making. Reprinted by permission
of SAGE Publications, Inc

Cost-Effectiveness Acceptability Curves (CEACs)

CEACs illustrate the uncertainty surrounding the estimate of cost-
effectiveness. A CEAC (Figure 2) shows the probability that an
intervention is cost-effective compared with the alternative for a range of
ceiling ratios or thresholds that a decision-maker might be willing to pay
for a particular unit of outcomes. Care must be taken when interpreting
the information provided by a CEAC. It simply presents the probability
that an intervention is cost-effective compared with the alternative for a
range of values. The CEAC should not be used to make statements about
the implementation of the intervention (Fenwick & Byford, 2005).

9
Figure 2. Example cost-effectiveness acceptability curve (CEAC).
Figure 2 demonstrates at £7,500 per QALY, the probability of the
intervention being cost-effective is 50%.

A word of caution regarding cost-effectiveness analysis

It is worth noting that the perspective of an analysis is key when
performing cost-effectiveness analysis.
If the perspective is restricted and does not cover all the relevant
stakeholders/payers, then this can lead to cost shifting from one part
of the system to another or from one agency to another, rather than
providing a cost-effective solution. To reduce the likelihood of this
occurring, researchers should use as comprehensive a perspective as
possible (Berger et al., 2003). For public health evaluations, the National
Institute for Health and Care Excellence (NICE) recommend a public
sector perspective (NICE,2012).

10
Example of cost-effectiveness analysis in practice
Edwards RT, Céilleachair A, Bywater T, Hughes DA, Hutchings J. (2007).
Parenting programme for parents of children at risk of developing conduct
disorder: cost effectiveness analysis. BMJ, 334 (7595), 682.
Owen L, Morgan A, Fischer A, Ellis S, Hoy A, Kelly MP. (2012). The cost-
effectiveness of public health interventions. Journal of Public Health, 34 (1),
37-45.

5.3 Cost-utility analysis (CUA)

Cost-utility analysis is an extension of cost-effectiveness analysis. It
is a form of economic evaluation in which health benefits are usually
measured in preference-based non-monetary units such as Quality
Adjusted Life Years (QALYs) or Disability Adjusted Life Years (DALYs).
QALYs are calculated by aggregating the number of years gained from
a drug or health care intervention, weighted by the proportion that
represents the relative value attached to a given health state (utility)
(Robinson, 1993c). DALYs are calculated by aggregating the time lost due
to premature death and time lived with a disability.
QALY = length of life x quality of life
DALY = years lived with disability + years of life lost

Health utility scores typically range between 0 (death) and 1 (perfect

health). One QALY is equal to one year of life lived in perfect health.
There are health states with negative values, which would be considered
worse than death. There are many measures available to produce utility
scores, (e.g. EQ-5D, EuroQol Group 1990; SF-36, Brazier et al., 1992; HUI,
Horsman et al., 2003). The choice of measure is based upon the research
question, suitability for the population under study and previous
literature. An individual may choose to use a particular measure in order
to allow comparability with previously published studies.
Limitations of cost-utility analysis
There are equity issues associated with QALYs. As length of life is used in
the equation, it is argued that the young gain more QALYs due to the fact
they have more life left to live than the elderly. Thus, resource allocation
based upon the maximisation of QALYs would cause a redistribution
of health care resources away from the elderly, favouring the young
(Wagstaff, 1991). Kelly et al., (2005) and Weatherly et al., (2009) have

11
argued that the QALY approach may be too narrow to capture the full
range of benefits from public health interventions. Though multiple
measures exist to generate utility scores, it is worth noting that different
measures produce different utility scores, making comparisons with
other studies sometimes difficult. Consideration of the population and
plans to compare the study with other published studies should be
undertaken before choosing a particular utility measure.
Example of cost-utility analysis in practice
Edwards RT, Linck P, Hounsome N, Raisanen L, Williams N, Moore L
Murphy S. (2013). Cost-effectiveness of a national exercise referral
programme for primary care patients in Wales: results of a randomised
controlled trial. BMC Public Health, 13 (1), 1021.
Use of ICERs in Economic Evaluations such as CUA and CEA
In the UK, governing bodies such as NICE use ICERS applying a ceiling
ratio/threshold to determine if an intervention, programme or service is
cost-effective. NICE typically use cost per QALY equations to determine
cost-effectiveness. NICE (2008) states the QALY should be used because
it is a standardised and internationally recognised method to compare
and measure clinical effectiveness and cost-effectiveness across different
treatments and patient groups. In the UK, the ceiling ratio has been
suggested at £20,000-£30,000 per QALY (NICE, 2008). In the USA, the
ceiling ratio is set at $50,000 per QALY, and between A$42,000-A$76,000
per QALY in Australia (Eichler et al., 2004).

5.4 Cost-benefit analysis (CBA)

Cost-benefit analysis places monetary values on both costs and
outcomes. It aims to answer the question is the benefit worth the cost
(Morris et al., 2007). However, it can only value tangible outcomes
e.g. money. It struggles to value intangible outcomes, which are yet to
be quantified e.g. happiness, relief from pain. In evaluations of health
care services or procedures, the use of monetary values allows you to
determine if a service or procedure offers an overall gain to society if
its total benefits surpass its total costs (Robinson, 1993d; Brazier et al.,
2007; McIntosh et al., 2010).
Benefits in this method can be valued using the human capital approach,
an approach that values benefits in terms of productivity gains or by
individual’s preferences using willingness to pay (Robinson, 1993d) or
willingness to accept (Drummond & McGuire, 2007).

12
Willingness to pay requires asking individuals how much they would be
prepared to pay to obtain the benefits or avoid the costs (e.g., money
or negative effects) of illness (Robinson, 1993d; Brazier et al., 2007;
McIntosh et al., 2010). Willingness to accept requires asking individuals
how much they would accept to be paid to abandon a good or put
up with something negative (e.g., side-effects from a medication that
reduced other symptoms) (Drummond & McGuire, 2007). Willingness
to pay and willingness to accept are often dependent upon how the
individual values money itself, as well as their valuation of benefits and
negative effects (Robinson, 1993d).

Limitations of cost-benefit analysis

By using the same outcome measure (costs/money), the method allows
you to compare interventions that can be unrelated (e.g. smoking
cessation intervention and a physical activity intervention as both aim
to benefit population health) (Berger et al., 2003). However, in order to
convert non-monetary outcomes into costs, assumptions are required.
Depending on the outcome, the evaluation could be based on rather
large assumptions. In order to perform the analysis, these assumptions
will be required. Researchers should be transparent in their assumptions
and test them through sensitivity analyses. Cost-benefit analysis has
also been criticised for giving greater weight to the preferences of the
wealthy (Berger et al., 2003). It is worth noting that asking individuals to
apply monetary values to outcomes will be rooted in their circumstances
and relative to their own earnings - what is expensive to one person
would not necessarily be considered expensive by another.
Example paper of cost-benefit analysis in practice
Reynolds AJ, Temple JA, Robertson DL, Mann EA. (2002). Age 21 Cost-
Benefit Analysis of the Title I Chicago Child-Parent Centers. Educational
Evaluation and Policy Analysis, 24, 267-303.

5.5 Cost-consequence analysis (CCA)

Cost-consequence analysis collects, categorises and lists the cost
components of a chosen intervention (Brazier et al., 2007). This type
of analysis lists the components of an intervention in a disaggregated
format, without making judgements of their relative importance. The
verdict is left to the decision maker (Brazier et al., 2007). By providing
the information in this format, the decision maker can focus upon the

13
outcomes that are most important or salient to them (Berger et al.,
2003). The biggest criticism of cost-consequence analysis is that it has
no weighting system to appraise the results. Costs and consequences
are presented disaggregated, requiring the decision makers themselves
to devise a system to appraise the results (Berger et al., 2003). These
decisions, made at an individual level, may not always be in the best
interest of the patients or society.
Example paper of cost-consequence analysis in practice
Gage H, Kaye J, Owen C, Trend P, Wade, D. (2006). Evaluating
rehabilitation using cost-consequences analysis: an example in
Parkinson’s disease. Clinical Rehabilitation, 20, 232-238.

5.6 General considerations for all evaluation methods

When reading the results of the economic evaluation methods in
Section 5, there are few key points to note. The Drummond Checklist
(Drummond et al., 2015) highlights key points to be aware of when
reading an economic evaluation - in addition, we wish to emphasise the
following:
Perspective – The perspective should be stated and you - as the reader -
should critically appraise if this perspective covers all relevant costs and
outcomes.
Sensitivity analysis – Sensitivity analyses should be conducted to explore
the extent to which assumptions made in the analysis are upheld, whilst
adjusting key variables (e.g., dosage of intervention received).
Discounting – Discounting incorporates positive time preference –
meaning benefits that occur now are valued higher than those that occur
in the future. The Treasury recommends a discount rate of 3.5%. Costs
and outcomes that occur after 1 year should be discounted by applying
the discount rate to account for positive time preference.

6. Social Return on Investment (SROI)

Social Return on Investment (SROI) analysis, which is common in
the USA, is becoming of interest to UK policy makers, local service
commissioners and charities. The method results in a ratio of benefits to
costs, estimating the value created for every £1 invested.
The central purpose of SROI is to address the challenge of measuring a

14
wider concept of value, capturing aspects across the triple bottom line of
economic, social and environmental value. SROI involves a stakeholder
consultation from the outset and throughout the process to help
establish boundaries, indicators and verify assumptions made in the
analysis.
The Cabinet Office (2011) have published ‘A Guide to Social Return
on Investment’. This guide provides step-by-step instructions on how
to conduct an accurate SROI. It was produced to help third sector
organisations better communicate their impact to the public, funders and
the Government.
The guide outlines six key stages to conducting an SROI analysis, which
are presented below.
The six stages to SROI as stated by The Cabinet Office (2011).
Stage Brief description of activity undertaken during that stage
1 Establishing scope and identifying key stakeholders
- Be clear about who is directly impacted by the intervention or
service
- What impact has the intervention had?
- How will these stakeholders be consulted during the process?

2 Map outcomes
- Engage with stakeholders to develop an impact map or theory
of change which is a diagram that shows the relationship
between inputs, outputs and outcomes
- Inputs are what a stakeholder puts into an intervention (e.g.
time to attend an exercise class as part of an intervention)
- Outputs are evidence that an activity has taken place (e.g.
number of hours conducting exercise or sessions attended)
- Outcomes are evidence that a change has taken place (e.g.
improvements in physical health from the intervention)

15
Stage Brief description of activity undertaken during that stage
3 Evidencing outcomes and giving them a value
- Find data from published sources or collect own data to show
whether outcomes have happened and value outcomes
4 Establishing impact
- After collecting evidence on outcomes, assign a monetary
value to these outcomes
- Also establish what changes would have happened without
the intervention (i.e. what happened in a control condition or
as a result of usual care)
- Are there changes from outside factors or other activity
that are not directly attributable to the intervention? (e.g.
is the improvement seen in physical health just from the
intervention or has the participant been engaging in exercise
outside the intervention – for example daily walks)
- How long are effects likely to last?

5 Calculating the SROI

- This stage involves adding up all the benefits, subtracting any
negatives and comparing the result to the investment. This is
also where the sensitivity of the results can be tested

6 Reporting, using and embedding

- Share findings with stakeholders and respond to any
comments or suggestions
- Be clear on your audience
- Create a technical appendix detailing assumptions and
calculations
- Verify results through an assurance process or work with an
expert to improve creditability

16
7. Critical appraisal of economic evaluations and decision
analytical models

7.1 The Drummond et al., (2015) Checklist for a Sound Economic Evaluation
In order to assess the results of a published economic evaluation, Drummond
et al., (2015) developed a checklist to identify elements they considered to
demonstrate a sound economic evaluation. It is worth noting that it is unlikely
that every study will satisfy all the points raised in the checklist. However, the
checklist provides a guide to the types of questions one should be asking when
reading published economic evaluations, so they can assess the strengths and
weaknesses of the study and make their own judgement of the usefulness
and relevance of the findings for their purposes. The checklist questions are
presented below.
The Drummond Checklist comprises of 10 main questions (adapted from
Drummond et al., 2015)
The Drummond Checklist adapted from Drummond et al (2015) Methods for
the Economic Evaluation of Health Care Programmes pp. 42-44. Copyright ©
2015 Oxford University Press. Reprinted by permission from Oxford University
Press. This reprinted Oxford University Press content is excluded from the
Handbook’s Creative Commons license. Anyone wishing to use the material
outside of this handbook needs to contact Oxford University Press for
permission (http://global.oup.com/?cc=gb)

1. Was a well-defined question posed in an answerable form?

- Were both costs and effects examined?
- Were alternatives considered?
- Was the perspective of the analysis stated? Is the analysis
embedded in any decision making context?

2. Was a comprehensive description of the competing alternatives given?

- Were any alternatives that were relevant to evaluation omitted?
- Was a do-nothing alternative considered or should it be?

17
3. Was the effectiveness of the programmes or services established?
- Was this done through a randomised controlled trial? Did the
trial reflect what happens in usual care or routine practice?
- Was this done though a systematic review of evidence from
clinical studies? If so, was the search strategy including inclusion
and exclusion criteria clearly described?
- Were observational data or assumptions used when establishing
effectiveness? If so, are there any potential biases in the results?

4. Were all the important and relevant costs and consequences for each
alternative identified?
- Was the range wide-enough for the research question at hand?
- Were all relevant perspectives covered (e.g., community, NHS,
patient)?
- Were capital costs as well as operating costs included?
Capital costs are one-time expenses typically incurred to set up a service
Operating costs are the recurrent delivery costs of a service, e.g. staff

5. Were costs and effects measured accurately in appropriate physical

units (e.g., QALYs)?
- Were sources of service utilisation described and acceptable?
- Were any items omitted? If so, what effect does this have on the
analysis?
- Were there any special circumstances that made measurement
difficult? Were these difficulties addressed?

18
6. Were costs and effects valued credibly?
- Were all sources of the values clearly identified?
- Were market values employed for changes involving resources
gained or depleted?
- Where market values were absent (e.g. volunteer labour) or
market values did not reflect actual values (e.g. equipment
given at a reduced rate), were adjustments made to
approximate market values?
- Was the valuation of effects appropriate for the question
posed? Was the appropriate type of analysis/analyses (e.g.
cost-effectiveness, cost-benefit or cost-utility analysis)
undertaken?
Market value is the price an asset would fetch in the marketplace

7. Were costs and effects adjusted for differential timing?

- Were future costs and effects discounted to their present
value?
- What was the discount rate used and was the justification for
this rate specified?

8. Was an incremental analysis of costs and effects of alternatives

performed?
- Were the additional (incremental) costs generated by one
alternative over another compared to the additional effects,
benefits, or utilities generated?

19
9. Were allowances made for uncertainty in the estimates of costs and
effects?
- Were appropriate analyses undertaken on patient-level data of
costs and effects?
- If sensitivity analyses were undertaken, were the justification
for the ranges and distribution of values chosen (for key
parameters) specified and explained?
- Were conclusions drawn sensitive to uncertainty from the
statistical and/or sensitivity analyses?

10 Did the presentation and discussion of study results include all issues of
concern to users?
- Were conclusions of the analysis based on an index or ratio
(e.g. cost-effectiveness or cost-benefit ratio)? Was this ratio
interpreted intelligently or in a mechanistic fashion?
- Were the results compared with those of others who have
investigated the same question? If so, were allowances made
for potential differences in methodology?
- Did the study discuss the potential of generalisability of the
results to other settings or patient/population groups?
- Did the study take in account other important factors in the
choice or decision under consideration (e.g. ethical issues,
limited staff numbers or wider policy context and relevance)?
- Did the study discuss issues of implementation (e.g. feasibility
of adopting recommendations)? Are there any potential issues
regarding finance and resources? Could resources be relocated
from other areas to assist the implementation?

20
7.2 Philips et al., (2004) checklist for critically appraising decision analytic models
Decision analytic modelling is widely used in the field of health economics as
a means of estimating the costs, outcomes and cost-effectiveness of different
interventions and programmes in health care and public health.  A logical model
is presented with mathematical representation of the relationships between
inputs and results. These methods are often used to predict health outcomes
and costs when the intervention cannot be evaluated directly or the scope
of the evaluation falls outside of the existing evidence base. The checklist
questions are presented below.

The Philips et al., (2004) checklist for critically appraising decision analytic
models (adapted from Philips et al., 2004)
Permission to reproduce the Philips checklist has been granted by the authors
and the publishers of the checklist the National Institute for Health Research
(NIHR)

Dimension of Questions to ask

quality
Statement - Is there a clear statement of the decision problem?
of decision
problem / - Is the objective of the evaluation and model
objective specified and consistent with the stated decision
problem?
- Is the primary decision-maker specified?

Statement - Is the perspective of the model stated clearly?

of scope /
perspective - Are the model inputs consistent with the stated
perspective?
- Has the scope of the model been stated and
justified?
- Are the outcomes of the model consistent with
the perspective, scope and overall objective of the
model?

21
Dimension of Questions to ask
quality
Rationale for - Is the structure of the model consistent with a
structure coherent theory of the health condition under
evaluation?
- Are the sources of data used to develop the
structure of the model specified?
- Are the causal relationships described by the model
structure justified appropriately?

Structural - Are the structural assumptions transparent and

assumptions justified?
- Are the structural assumptions reasonable given
the overall objective, perspective and scope of the
model?

Strategies/ - Is there a clear definition of the options under

comparators evaluation?
- Have all feasible and practical options been
evaluated?
- Is there justification for the exclusion of feasible
options?

Model type - Is the chosen model type appropriate given the

decision problem and specified causal relationships
within the model?

Time horizon - Is the time horizon of the model sufficient to reflect

all important differences between options?
- Are the time horizon of the model, the duration
of treatment and the duration of treatment effect
described and justified?

22
Dimension of Questions to ask
quality
Disease states/ - Do the disease states (state transition model) or
pathways the pathways (decision tree model) reflect the
underlying biological process of the disease in
question and the impact of interventions?

Cycle length - Is the cycle length defined and justified in terms of

the natural history of disease?

Data - Are the data identification methods transparent and

identification appropriate given the objectives of the model?
- Where choices have been made between data
sources, are these justified appropriately?
- Has particular attention been paid to identifying
data for the important parameters in the model?
- Has the quality of the data been assessed
appropriately?
- Where expert opinion has been used, are the
methods described and justified?

Data modelling - Is the data modelling methodology based

on justifiable statistical and epidemiological
techniques?

Baseline data - Is the choice of baseline data described and

justified?
- Are transition probabilities calculated appropriately?
- Has a half-cycle correction been applied to both cost
and outcome?
- If not, has this omission been justified?

23
Dimension of Questions to ask
quality
Treatment - If relative treatment effects have been derived
effects from trial data, have they been synthesised using
appropriate techniques?
- Have the methods and assumptions used to
extrapolate short-term results to final outcomes
been documented and justified? Have alternative
assumptions been explored through sensitivity
analysis?
- Have assumptions regarding the continuing effect
of treatment once treatment is complete been
documented and justified? Have alternative
assumptions been explored through sensitivity
analysis?

Costs - Are the costs incorporated into the model justified?

- Has the source for all costs been described?
- Have discount rates been described and justified
given the target decision-maker?

Quality of - Are the utilities incorporated into the model

life weights appropriate?
(utilities)
- Is the source for the utility weights referenced?
- Are the methods for derivation for the utility
weights justified?

24
Dimension of Questions to ask
quality
Data - Have all data incorporated into the model been
incorporation described and referenced in sufficient detail?
- Has the use of mutually inconsistent data been
justified (are assumptions and choices appropriate)?
- Is the process of data incorporation transparent?
- If data have been incorporated as distributions, has
the choice of distribution for each parameter been
described and justified?
- If data have been incorporated as distributions, is it
clear that second order uncertainty is reflected?

Assessment of - Have the four principal types of uncertainty been

uncertainty addressed?
- If not, has the omission of particular forms of
uncertainty been justified?

Methodological - Have methodological uncertainties been addressed

by running alternative versions of the model with
different methodological assumptions?

Structural - Is there evidence that structural uncertainties have

been addressed via sensitivity analysis?

Heterogeneity - Has heterogeneity been dealt with by running the

model separately for different subgroups?

Parameter - Are the methods of assessment of parameter

uncertainty appropriate?
- If data are incorporated as point estimates, are the
ranges used for sensitivity analysis stated clear and
justified?

25
Dimension of Questions to ask
quality
Internal - Is there evidence that the mathematical logic of the
Consistency model has been tested thoroughly before use?

External - Are any counterintuitive results from the model

Consistency explained and justified?
- If the model has been calibrated against
independent data, have any differences been
explained and justified?
- Have the results of the model been compared with
those of previous models and any differences in
results explained?

7.3 Standards of Reporting for Economic Evaluations

The Consolidated Health Economic Evaluation Reporting Standards
(CHEERS) statement was developed to provide consistency in the
reporting of economic evaluations. The checklist is similar to the
Consolidated Standards of Reporting Trials (CONSORT) format and
provides consistency with other approaches. The CONSORT statement
and checklists are an evidence-based, minimum set of recommendations
for reporting randomised trials. They were developed to create
standardisation in reporting, to facilitate transparency and aid critical
appraisal and interpretation. The 24 item CHEERS checklist describes
the key recommendations of the information that should be included
under headings for example: title, abstract, background/objectives,
target population, study perspective, comparators, time horizon,
study parameters, incremental cost and outcomes, limitations and
generalisability. For the full checklist, see Husereau et al., (2013).

26
8. Useful Health Economics Resources
Health Knowledge
An online resource for anyone working in health, social care and well-
being. The website offers a broad range of learning materials, divided
into four different learning styles:
• A Public Health Textbook organised in relation to the Faculty of Public
Health Part A syllabus.
• Text courses with text, questions, answers and feedback on a range of
topics.
• Podcasts and Video PowerPoints with supporting resources.
• Management training with PowerPoint slides, workbooks and trainer
notes in four clinical areas: diabetes, coronary heart disease, stroke and
child health.
This online resource has a specific section on health economics.
www.healthknowledge.org.uk/public-health-textbook/medical-sociology-
policy-economics/4d-health-economics

Service Utilisation and Costs

Database of Instruments for Resource Use Measurement (DIRUM)
DIRUM is an open-access database of resource-use questionnaires for
use by health economists involved in trial-based economic evaluations.
Funded by the Medical Research Council Network of Hubs for Trial
Methodology Research, DIRUM offers a unique (and permanent) web
address for each resource use measure for citation in papers and reports.
DIRUM also provides a repository of methodological papers related to
resource use and cost measurement.
http://www.dirum.org/

Unit Costs of Health and Social Care

A downloadable PDF containing unit costs of health and social care
contacts and care developed by the Personal Social Services Research
Unit at the University of Kent at Canterbury and the London School of
Economics and Political Science. This document is used in economic
evaluations, quantifying and applying a cost to GP appointments and
community care.
http://www.pssru.ac.uk/project-pages/unit-costs/

27
NHS reference costs
A downloadable Excel file containing average unit costs to the NHS of
providing secondary health care to NHS patients. It is used in economic
evaluations to apply a cost to treatment received by participants to
calculate the cost of secondary care.
https://www.gov.uk/government/collections/nhs-reference-costs

Systematic Reviewing
Shemilt and colleagues (2013) reflect on the value and desire for the
consideration by end users for coverage of an economic perspective
in a Cochrane review and outlines two potential approaches and
future directions. This paper provides a good introduction to economic
perspectives and considerations when conducting systematic reviews.
http://www.systematicreviewsjournal.com/content/2/1/83

Useful Databases containing economic evidence

The Health Technology Assessment Database
http://www.crd.york.ac.uk/CRDWeb/
NHS Economic Evaluation Database
http://www.crd.york.ac.uk/CRDWeb/
RePEc (Research Papers in Economics)
http://www.repec.org/
EconLit
https://www.aeaweb.org/econlit/

28
References
Berger ML, Bingefors K, Hedblom EC, Pashos CL, Torrance GW. (2003).
Health Care Cost, Quality, and Outcomes: ISPOR Book of Terms.
Lawrenceville, NJ: ISPOR.
BMJ (2012) Clinical Evidence Toolbox
http://clinicalevidence.bmj.com/x/set/static/ebm/toolbox/678253.html
Brazier J, Ratcliffe J, Saloman JA, Tsuchiya A. (2007). Measuring and
Valuing Health Benefits for Economic Evaluation. Oxford University Press:
Oxford, UK
Brazier JE, Harper R, Jones NM, O’Cathain A, Thomas KJ, Usherwood T,
Westlake L. (1992) Validating the SF-36 health survey questionnaire: new
outcome measure for primary care. BMJ. 305(6846): 160–164.
The Cabinet Office’s Guide (2011). A Guide to Social Return on
Investment.
http://www.bond.org.uk/data/files/Cabinet_office_A_guide_to_Social_
Return_on_Investment.pdf
Drummond MF, Sculpher MJ, Claxton K, Stoddart GL and Torrance GW.
(2015). Methods for the economic evaluation of health care programmes
(4th Ed.). Oxford University Press: Oxford, UK
Drummond MF, McGuire A. (2007). Economic Evaluation in Health Care,
Merging Theory with Practice. Oxford University Press: Oxford, UK
Eichler HG, Kong SX, Gerth WC, Mavros P, Jonsson B. (2004). Use of
Cost-Effectiveness Analysis in Health-Care Resource Allocation Decision-
Making: How Are Cost-Effectiveness Thresholds Expected to Emerge?
Value in Health, 7, 518-528.
The EuroQol Group (1990). EuroQol- a new facility for the measurement
of health-related quality of life. Health Policy 16(3):199-208.
Fenwick E, Byford S. (2005). A guide to cost-effectiveness acceptability
curves. British Journal of Psychiatry 187:106-108.
Glick HA, Doshi JA, Sonnad SS, Polsky D. (2007). Economic Evaluation in
Clinical Trials. Oxford University Press: Oxford, UK
Grossman M. (1972). On the concept of health capital and the demand
for health. The Journal of Political Economy 80(2):223-255.
Horsman J, Furlong W, Feeny D, Torrance G. (2003) The Health Utilities
Index (HUI®): Concepts, Measurement Properties and Applications.
Health and Quality of Life Outcomes (electronic journal), 1, 54.

29
McIntosh E, Clarke P, Frew E, Louviere J. (2010). Applied Methods of Cost-
Benefit Analysis in Health Care. Oxford University Press
Morris S, Devlin N, Parkin D. (2007). Economic Analysis In Health Care.
Wiley and Sons, Chichester, UK
National Institute for Health and Care Excellence (NICE) (2008).
Measuring effectiveness and cost effectiveness: the QALY.
http://publications.nice.org.uk/how-nice-measures-value-for-money-
in-relation-to-public-health-interventions-lgb10b/nices-approach-to-
economic-analysis-for-public-health-interventions
National Institute for Health and Care Excellence (NICE) (2012). Methods
for the development of NICE public health guidance (3rd edition).
http://www.nice.org.uk/article/pmg4/resources/non-guidance-methods-
for-the-development-of-nice-public-health-guidance-third-edition-pdf
Office of Health Economics (OHE) (2008) Glossary.
Owen L, Morgan A, Fischer A, Ellis S, Hoy A, Kelly MP. (2012). The cost-
effectiveness of public health interventions. Journal of Public Health, 34
(1), 37-45.
Phillips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R,
Woolacoot N, Glanville J. (2004). Review of guidelines for good practice
in decision-analytic modelling in health technology assessment. Health
Technology Assessment 8 (36), 1-158.
Robinson, R. (1993a). Economic Evaluation in Health Care. Costs and cost-
minimisation analysis. BMJ, 307, 726-728.
Robinson, R. (1993b). Economic Evaluation in Health Care. Cost-
effectiveness analysis. BMJ, 307, 793-795.
Robinson, R. (1993c). Economic Evaluation in Health Care. Cost-utility
analysis. BMJ, 307, 859-862.
Robinson, R. (1993d). Economic Evaluation in Health Care. Cost-benefit
analysis. BMJ, 307, 924-926.

Acknowledgements
The authors wish to thank Dr Christopher Johnson, Public Health Wales;
Liz Green, Wales Health Impact Assessment Support Unit; and Dr Sam
Groves, Swansea Centre for Health Economics, Swansea University who
provided feedback on drafts of the handbook.

30
 Joanna Charles is a Research Fellow at the Centre for Health
Economics and Medicines Evaluation at Bangor University. She
has an interest in the use of health economic techniques for
evaluating public health interventions, evidence synthesis and
micro-costing methodologies.

Rhiannon Tudor Edwards is Professor of Health Economics

and Co-Director of the Centre for Health Economics and
Medicines Evaluation at Bangor University. She is an
Honorary Member of the Faculty of Public Health, a Fellow
of the Learned Society of Wales and a Health and Care
Research Wales Senior Investigator. Her interests centre on
the economics of prevention and the application of health
economics to the evaluation of public health interventions.

Funded by Public Health Wales

Public Health Wales is an NHS organisation providing professionally independent
public health advice and services to protect and improve the health and wellbeing
of the population of Wales. Production of this report was funded by Public Health
Wales. However, the views in this report are entirely those of the authors and
should not be assumed to be the same as those of Public Health Wales.

This Handbook is published under a Creative Commons Attribution-Non

Commercial-No Derivatives 4.0 International License and
may be downloaded and shared as long as the authors are credited, but adapting
the handbook’s content or using it for commercial purposes is not permitted.

Centre for Health Economics and Medicines Evaluation

Ardudwy Hall, Normal Site, Bangor University, Bangor, Gwynedd, LL57 2PZ
Phone: 01248 382153
ISBN Number 978-1-84220-144-2
October 2016