Public Disclosure Authorized

Public Disclosure Authorized

Public Disclosure Authorized
84080

Value-for-Money Analysis-
Practices and Challenges:
Public Disclosure Authorized

How Governments Choose When to Use PPP to

Deliver Public Infrastructure and Services
Value-for-Money Analysis—Practices
and Challenges:

How Governments Choose When to Use PPP

to Deliver Public Infrastructure and Services
 Value-for-Money Analysis—Practices
and Challenges:

How Governments Choose When to Use PPP

to Deliver Public Infrastructure and Services

Report from World Bank Global Round-Table

28 May, 2013, Washington DC

World Bank Institute (WBI) and

Public-Private Infrastructure Advisory Facility (PPIAF)
© 2013 International Bank for Reconstruction and Development /
International Development Association or
The World Bank
1818 H Street NW
Washington DC 20433
Telephone: 202-473-1000
Internet: www.worldbank.org

This work is a product of the staff of The World Bank with external
contributions. The findings, interpretations, and conclusions expressed
in this work do not necessarily reflect the views of The World Bank, its
Board of Executive Directors, or the governments they represent.

The World Bank does not guarantee the accuracy of the data included
in this work. The boundaries, colors, denominations, and other
information shown on any map in this work do not imply any judgment
on the part of The World Bank concerning the legal status of any
territory or the endorsement or acceptance of such boundaries.

Rights and Permissions

The material in this work is subject to copyright. Because The World
Bank encourages dissemination of its knowledge, this work may be
reproduced, in whole or in part, for noncommercial purposes as long
as full attribution to this work is given.

Any queries on rights and licenses, including subsidiary rights, should

be addressed to the Office of the Publisher, The World Bank, 1818
H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail:
[email protected].

Design and layout: Sara Tejada Montoya

Table of Contents

Acknowledgements ........................................................................................ 7

1 Introduction .............................................................................................. 9

2 What is VFM Analysis? ........................................................................... 12

3 PPP Decision-Making: How VFM Analysis Is Used .............................. 15

4 Quantitative VFM Analysis: the PSC and Key

Methodological Issues ........................................................................... 23

5 Conclusions and Interesting Questions ............................................... 28

Annex A. Global VFM Roundtable 28 May 2013:

Agenda and Participants .............................................................................. 31
Acknowledgements
This report has been drafted by Helen Martin, Consultant in the Latin
America and the Caribbean Sustainable Development Department of the
World Bank, based on presentations delivered to a “Global Roundtable
on Value for Money Analysis” held on 28 May, 2013, at the World Bank
Headquarters in Washington, DC, and the ensuing discussion at that event.
Roundtable presenters are listed in Annex A to this report, and included
PPP practitioners from the United Kingdom, France, the United States of
America, Chile, the Republic of Korea, India, Canada, and South Africa.
Comments and feedback on the report were provided by Clive Harris,
Daniel Benitez, Lincoln Flor, Jay-Hyung Kim, Rui Monteiro, and Satheesh
Sundararajan, all of the World Bank, as well as by the roundtable presenters.
The roundtable was jointly convened by the Public-Private Infrastructure
Advisory Facility (PPIAF), the Latin America and Caribbean region of the
World Bank (LAC) and the World Bank Institute (WBI). The task team leader
of this initiative was Mr. Lincoln Flor, Senior Public-Private Partnerships
Specialist of WBI.

7 Acknowledgements
1. Introduction
A growing number of governments are using Public-Private Partnerships
(PPPs) to deliver infrastructure. A PPP is a long-term contract between a
private party and a government agency, for providing a public asset or
service, in which the private party bears significant risk and management
responsibility1. Such partnerships can help make the best use of the
resources of both the public and private sectors—including finance,
experience, expertise, and focus on delivery—to expand and improve
public infrastructure assets and services.

Governments enter into PPPs for a variety of reasons, as described further

in Section 3 below. Nonetheless, for most governments the potential to
achieve greater “value for money” than other procurement and delivery
models is an important, if not the primary factor in the decision to implement
a project as a PPP. Definitions of “value for money” vary; the UK Treasury,
for example, defines the concept as follows:

“Value for Money (VFM) is the optimum combination of

whole-of-life costs and quality (or fitness for purpose) of the
good or service to meet the user’s requirements”.

Broadly speaking, a PPP may provide value for money compared to

1 This report adopts the broad definition of PPP set out in the WBI and PPIAF (2012) PPP Reference
Guide. This definition encompasses performance-based, long-term contracts for new or existing
assets and services; including contracts paid for by service users (sometimes called concessions),
a government agency, or a combination of the two. For more details and examples of different
PPP types, see the PPP Reference Guide, available at http://www.ppiaf.org/sites/ppiaf.org/files/
publication/Public-Private-Partnerships-Reference-Guide.pdf.

9 Introduction
traditional procurement models if the advantages of risk transfer combined
with private sector incentives, experience and innovation—in improved
service delivery or efficiencies over the project life-time—outweigh the
increased costs of contracting and financing. This raises challenges for
policy-makers: how to assess the value for money of different procurement
and delivery options—that is, carry out “value for money (VFM) analysis”—
and how to use the results of this analysis in PPP decision-making2.

VFM analysis plays an important role in many PPP programs: a recent OECD
study found that 19 of 20 surveyed countries apply some kind of value for
money assessment to proposed PPPs3. However, even in countries with
well-established PPP programs, the approach to and use of this analysis is
evolving, and is often the subject of controversy and debate. Meanwhile,
many of the World Bank Group (WBG)’s clients with some PPP experience—
for example, in the Latin America and Caribbean region—are trying to
move towards a more systematic approach to VFM analysis and PPP
project selection, but facing challenges in developing and implementing
appropriate methodologies.

To inform this debate, the World Bank convened a global “roundtable”

of PPP practitioners to discuss VFM and how it can be assessed. The aim
of the roundtable was to draw lessons from countries that have relatively
well-developed approaches and tools for VFM analysis: with respect to how
this analysis has evolved, what are the on-going and new challenges, and
how the approaches might apply in countries with less well-established
PPP programs. The presenters at this roundtable—which included PPP
practitioners from the United Kingdom (UK), France, the United States
of America (USA), Chile, the Republic of Korea (hereafter Korea), India,
Canada, and South Africa—are listed in Annex A4.

2 As defined, VFM analysis is only a part of a typical PPP project appraisal process. Other PPP appraisal
criteria typically include the feasibility and economic viability of the project (that is, does the project
provide VFM, irrespective of its implementation as a PPP or other contractual model); its commercial
viability (that is, whether the project is likely to be able to provide adequate return to attract good-
quality investors); and its affordability, or fiscal responsibility. These criteria and the overall PPP
appraisal process are described in more detail in the WBI-PPIAF PPP Reference Guide. In some PPP
programs—such as the Republic of Korea—“VFM analysis” is used to refer to PPP project appraisal
as a whole; throughout this report it is used to refer only to the part of that appraisal that compares
PPP to other options for project procurement and delivery.
3 Philippe Burger and Ian Hawkesworth (2011) “How to Attain Value for Money: Comparing PPP and
Traditional Infrastructure Public Procurement”, OECD Journal on Budgeting Volume 2011/1
4 Unless otherwise noted, references to specific governments’ PPP programs in this report are taken
from the respective roundtable participant’s presentation, and the ensuing discussion.

Value-for-Money Analysis—Practices and Challenges 10

This report presents some of the key issues in assessing VFM that arose
during the roundtable discussion, based on the experience of the
participants. The content of this report is as follows:

• Section 2 provides an overview of VFM analysis

• Section 3 discusses how VFM analysis is used in the PPP decision-
making process
• Section 4 describes some methodological challenges with VFM
analysis
• Section 5 concludes, and summarizes the key lessons from the
roundtable.

11 Introduction
2. What is VFM Analysis?
As defined above, the purpose of Value for Money (VFM) analysis is to
inform governments’ decision on whether to implement proposed projects
as PPPs, or through other more “traditional” forms of public procurement
(although in practice, the contribution of VFM analysis to that decision varies
between PPP programs, as described further in the following section). To
that end, VFM analysis typically involves a combination of qualitative and
quantitative analysis: these are briefly described in turn below.

This report focuses on ex-ante analysis of the VFM of a potential PPP. This is
closely linked with ex-post VFM assessment—reviewing whether a particular
PPP, or the PPP program as a whole, has achieved value for money in
practice—in that experience with PPP can and should influence future PPP
decision-making. As discussed further in subsequent sections, in practice
few governments systematically carry out ex-post VFM assessments of PPP
projects—in turn creating challenges in availability of data to inform ex-
ante VFM analysis.

Most of the PPP programs represented at the roundtable carry out VFM
analysis for each proposed PPP project (in some cases, only for projects
above a certain size). However, participants noted that VFM analysis may not
be necessary for multiple, similar projects—and could instead be applied
to a “test case” for the first project of a given type. For example, the road
development agency of the State of Madhya Pradesh, India undertook
VFM analysis when considering new types of road PPP models involving
availability payments. Their conclusions were then checked ex-post, by
comparing the performance of the new PPPs with other road projects.

Value-for-Money Analysis—Practices and Challenges 12

Qualitative analysis

Qualitative VFM analysis typically involves sense-checking the rationale for

using PPP—that is, asking whether a proposed project is of a type likely to
be suitable for private financing. This often takes place at a relatively early
stage of PPP development, as described in Section 3.

Some jurisdictions have clearly-defined criteria for this analysis. For

example:

• The UK Treasury has defined criteria for assessing suitability, and

unsuitability, for a Private Finance Initiative (PFI—the UK’s PPP
model). Suitability criteria include the long-term, predictable need
for the service; the ability to allocate risk effectively—including
through performance-related payments and ensuring sufficient
private capital at risk; the likely ability of the private sector party
to manage risk and take responsibility for delivery; presence of
stable and adequate policy and institutions; and a competitive
bidding market. “Unsuitability” criteria include projects that are
either too small or too complicated; sectors where needs are
likely to change or there is a risk of obsolescence (for example,
PFI projects are no longer used in the ICT sector in the UK); or
where the contracting authority is inadequately skilled to manage
PPP.
• In France, “preliminary analysis” of a PPP includes checking
against several criteria under three categories: PPP relevance—
for example, appropriateness of an integrated, whole-of-life
approach to managing a project; commercial attractiveness; and
the potential for optimal risk allocation.
• In the Commonwealth of Virginia, USA, assessment of a
potential PPP at “high level” and detailed screening stages
also considers proposed road projects against specific criteria
to determine if the project is delivered under the Public-Private
Transportation Act (PPTA)—that is, as a PPP. These criteria
include whether a project is sufficiently complex to benefit from
private sector innovation; whether a PPP can achieve appropriate
risk transfer; and the degree of stakeholder support. The extent
to which a project can generate revenues from tolls is also taken
into consideration when assessing possible PPP structures.

13 What is VFM Analysis?

In other cases, such as in Chile, qualitative analysis of the suitability of
a project for PPP plays an important part in PPP decision-making, albeit
without being guided by documented criteria.

Quantitative analysis

Quantitative VFM analysis involves comparing the value for money of a

proposed PPP (or actual bids received) with a “Public Sector Comparator”
(PSC)—that is, a model of the project if implemented through traditional
public procurement. The scope of this analysis varies, as described further
in Section 4 below:

• Some governments (such as Chile) simply compare the estimated

fiscal cost of the PPP (that is, the payments that would be made
to the private partner) and of implementing the project under
traditional public procurement.
• Most governments (such as British Columbia, Canada, Korea,
and South Africa) adjust the fiscal cost comparison for the
government’s risk exposure in each case—that is, build into the
“PSC” the cost of bearing those risks that would be transferred to
the private partner under a PPP model.
• Finally, in a few cases (such as France) differences in socio-
economic benefits between the procurement and delivery
models are included.

Quantitative VFM analysis has typically involved comparing two options:

a “preferred” PPP model, against a PSC. However, governments are
increasingly expanding the analysis. In Virginia, USA, for any proposed road
project that passes the initial PPP screening, a range of possible contractual
structures are assessed: including pure tolled concessions, different levels of
availability payments, or design-build-finance models. The UK Treasury—
having withdrawn its previous PSC model, as described further below—is
in the process of considering whether and how quantitative VFM analysis
could be applied to a broader set of procurement and delivery options.

Value-for-Money Analysis—Practices and Challenges 14

3. PPP Decision-Making: How VFM Analysis
Is Used
Different governments use VFM analysis in different ways. The role of VFM
analysis in PPP decision-making is a subject of spirited debate, as evidenced
by the roundtable discussion. This debate centers on two main questions.
The first concerns the relevance of VFM analysis, and whether or not VFM
is really the driving consideration when governments decide to implement
projects as PPPs. The second relates to the limitations of VFM analysis:
even where VFM is an important consideration, is VFM analysis—or at least,
the approaches to VFM analysis that have been used to date—the right tool
for informing this decision? Concerns about the limitations of quantitative
VFM analysis led to a recent decision by the UK Treasury, pioneer of the
PSC, to withdraw its quantitative VFM analysis model for a major revision.

Several aspects of the use of VFM analysis in PPP decision-making, and its
relevance and limitations, were raised at the PPP roundtable:

• Why governments do PPPs, and implications for the relevance of

VFM analysis.
• Timing and role of VFM analysis in the project selection and
development process, and achieving the right balance between
qualitative and quantitative approaches.
• Pitfalls of the Public Sector Comparator—in particular, the risk of
quantitative VFM analysis appearing “overly-scientific.”
• Role of PSC in procurement—whether and why VFM results are
communicated to bidders.

15 PPP Decision-Making: How VFM Analysis Is Used

These issues, and the differing perspectives put forward by PPP practitioners
during roundtable presentations and discussions, are described in turn below.

Why Governments do PPPs, and the relevance of VFM

analysis

As noted by round-table attendees, many governments turn to PPPs not

necessarily as the best value for money approach for implementing a
project after analyzing all the options, but rather because there is seen to
be no realistic public alternative in the face of financial or implementation
capacity constraints.

In particular, one of the most common reasons for governments to turn to

PPPs is the perception that PPPs create “fiscal space” to enable accelerated
implementation of infrastructure projects. This is particularly the case for
PPPs involving user charges, which can raise additional revenue for funding
infrastructure investment (as well as, in some cases, contributing to more
economically efficient use of services). While governments could also
introduce user fees for public projects, charges may be seen as politically
or socially easier to introduce under a PPP.

Under many PPPs, however, the full cost of the project is ultimately paid
by government—that is, over the long term no additional funding or fiscal
space is created. However, the nature of the expenditure changes: with
upfront capital expenditure often replaced by the recurrent cost of meeting
availability payments. Depending on how PPP commitments are treated in
fiscal reports and accounts, this can also create “space” in the short term,
for example in the face of deficit or debt targets—and hence an impetus
to implement projects as PPPs irrespective of whether doing so will create
better value for money5. This effect can be exacerbated where PPPs involve
transfers from one level of government to another—for example, in the UK,
where the availability of “PFI credits” were often the driver for contracting
authorities choosing to do PPP (indeed these credits were introduced

5 Evolving norms in public sector accounting appear likely to erode this perceived advantage of PPP
over time—at least for government-pays PPPs—as the equivalence of PPP obligations and other
public liabilities are increasingly recognized. For further discussion and resources see for example the
WBI/PPIAF PPP Reference Guide (2012); Katja Funke, Tim Irwin, and Isabel Rial (2013) “Budgeting
and Reporting for Public-Private Partnerships”, International Transport Forum Discussion Paper
No. 2013-7, OECD.

Value-for-Money Analysis—Practices and Challenges 16

as an incentive for authorities to use PPP); these credits have since been
abandoned.

Equally, the decision to introduce PPPs—or in many cases not to introduce

PPP, for example in particular “sensitive” sectors, or in the face of influence
by public sector unions—may be influenced by political or social attitudes
over fiscal or value for money considerations.

In such cases VFM analysis may appear less relevant as an input to decision-
making. For example, as discussed at the roundtable and described in Box
1 below, approaches to VFM analysis for PPPs involving user fees vary, with
some governments choosing not to apply it. On the other hand, even in
the face of limited alternatives, roundtable participants noted the value to
be gained from carrying out VFM analysis: to sense-check the decision to
pursue the project as well as the proposed PPP structure. Moreover, as
described in Section 4 below, some governments explicitly build into VFM
analysis the benefits of earlier implementation of proposed PPPs.

Box 1: VFM Analysis for User-Pays PPPs

Governments vary in their approach to VFM analysis for projects

involving user charges. In particular, where charging users is perceived
to be more feasible under a PPP than for publicly-run infrastructure,
VFM analysis may be seen as less relevant. For example:

• VFM analysis in Chile is limited to social sector PPPs that will

be paid for entirely by Government availability payments.
In economic sectors such as transport, user charging is
seen as the more economically efficient way to pay for
infrastructure, and as more politically and socially feasible
under a PPP—the decision to implement a project as a PPP
in these sectors is therefore driven by the financial viability
of the proposed project.
• Similarly, in France VFM analysis is only required for
“partnership” contracts; concessions (that is, user-fee
projects) have a different legal framework that does not
involve VFM analysis.

17 PPP Decision-Making: How VFM Analysis Is Used

 On the other hand, as some participants noted, VFM analysis can
equally be applied to all types of PPPs, and applied to different
contractual options—both on the basis that a user fee-funded
project could be done as either a Government project or as a
PPP; and as a helpful sense-check on the proposed PPP structure.
For example, In Virginia, USA, the Department of Transportation
(VDOT) undertakes VFM analysis for all proposed concessions. To
date this analysis has largely involved comparing the public and PPP
options for implementing toll roads—going forward the analysis will
be used to compare different possible PPP contract types, including
concessions based on availability payments.

Timing and role of VFM analysis in the project selection

and development process

VFM analysis could in theory be carried out for all public investment projects
to determine the best procurement and delivery option, as a systematic
component of a broader project cost-benefit and options analysis6. In
practice, few countries currently take this approach—although some are
moving in that direction.

More often, VFM analysis is formally applied only to projects already

earmarked as PPPs. That is, the initial decision to propose a project
for the PPP “route” is based on a qualitative assessment by contracting
authorities, which may be supported by PPP agencies, and often not based
on any specific methodology (although this decision may be influenced by
the likelihood of subsequently passing a formal VFM screening). Round-
table presenters agreed that experienced practitioners “know what makes
a good PPP project”. For example:

• In France, VFM analysis is currently only applied to projects

envisaged as PPP, “more as a technical study pre-implementation
of a project as a PPP than a tool to determine the best contracting

6 Presentation by Ian Hawkesworth to VFM Roundtable; for more detailed analysis and
recommendations on procurement options analysis, see Philippe Burger and Ian Hawkesworth (2011)
“How to Attain Value for Money: Comparing PPP and Traditional Infrastructure Public Procurement”,
OECD Journal on Budgeting Volume 2011/1

Value-for-Money Analysis—Practices and Challenges 18

 mode”. However, a recent review by the national audit entity
recommended that VFM analysis should be extended to all large,
complex public investment projects.
• Similarly, in Korea, it is the decision of the contracting authority
whether to propose a project as a PPP (alternatively, an
unsolicited PPP proposal may be received from a potential
investor). PIMAC—which is responsible for appraising both PPP
and non-PPP projects—applies VFM analysis only to projects on
the “PPP route”. If a project is found not to provide VFM as a PPP
but has a positive benefit-cost ratio, it reverts to the traditional
procurement route.
• VFM analysis in Chile—an assessment of the fiscal cost of PPP or
traditional public procurement—has to date been undertaken by
the finance ministry relatively late in the process, and considered
as a “sense check” of the earlier decision by the contracting
authority to put the project forward as a PPP.

Moreover, the timing of VFM analysis in the process of developing a

project as a PPP presents a trade-off: between availability and accuracy
of information—limited in early stages—and impact of the analysis, which
is typically limited later on in the process as it becomes more difficult
to “change route”. Many countries iterate the analysis: typically with
qualitative analysis taking place earlier in the process, while quantitative
analysis comes later.

In the UK’s current review of its PPP program, the timing of VFM analysis
has been an issue of concern. VFM analysis is done at four key stages: at
the overall program level, at project inception, prior to launch of public
procurement, and prior to contract signature. There has been concern that
the UK has “not got the balance right yet”: earlier stages of analysis are
more crucial for decision-making, but often the VFM analysis at this stage
receives less scrutiny. Moving forward, the UK Treasury intends to put in
more thought (if not necessarily more detail) to VFM analysis at earlier
stages, and focus in later stages on double-checking earlier conclusions as
more information emerges.

Similarly, the PPP Unit in Virginia, USA’s Department of Transportation

submits all proposed PPPs to a two-stage screening process, which
includes qualitative VFM analysis. It is this screening that determines

19 PPP Decision-Making: How VFM Analysis Is Used

whether a project is accepted by the unit for development as a PPP. When
the quantitative VFM analysis is done later in the process, this creates a
pressure to “deliver the right result”, given the cost of changing approach
at later stages—that is, quantitative VFM analysis is used more to rationalize
an earlier decision, than as an actual decision tool.

Possible pitfalls of the PSC and quantitative VFM analysis

In addition to challenges of timing, most round-table attendees agreed

that a major risk of quantitative VFM analysis is that the results are seen
as “overly-scientific”. This was the primary concern of the UK Treasury
in withdrawing its PSC model: in the experience of the UK, “too much
emphasis has been given to the quantitative analysis—as if it provided
mathematical proof of VFM”.

In practice, as described further in Section 4 below, methodological

challenges and limited information means that quantitative VFM analysis
is highly subjective—as one roundtable participant put it, “at best,
a hypothetical analysis on average reference project based on many
(unrealistic) assumptions”. However, the apparent exactness of a quantitative
VFM “result” can belie the subjectivity of the process. While the specificity
and simplicity of a number can be useful—several roundtable participants
noted that quantitative analysis “can be helpful with political and public
perception” of a proposed PPP—it can also tempt officials to over-rely on
quantitative results at the expense of “real judgment”; or worse, be open
to manipulation.

Despite these significant limitations, most attendees viewed quantitative

VFM analysis as a valuable part of the PPP development process: albeit
as much for the process itself—a systematic examination of the structure
and risk allocation of a proposed PPP—as for its input into the decision to
implement a project as a PPP. All agreed that a clear understanding of risk
is crucial to achieve value from a PPP, and avoid poorly-structured projects
that provide fiscal surprises down the road. In this sense, the VFM test can
be seen “as a project management process, rather than a highly statistical-
based, rigorous analysis”.

Value-for-Money Analysis—Practices and Challenges 20

Role of VFM analysis in procurement, and communicating
PSC to bidders

As well as informing the decision to pursue a PPP, some governments use

the results of VFM analysis explicitly in the tender process for a PPP—with
a view to ensuring the government’s assumptions on VFM are borne out
in the final bids. In some cases the VFM analysis provides a hard limit on
acceptable bids, in others more as a “guide” to bidders on the government’s
expectations. For example:

• In British Columbia, Canada, a firm “affordability ceiling” is

announced in bid documents for each PPP. This ceiling is set at the
value of the PPP “shadow bid” as modeled by the Government,
rather than the PSC value—that is, the idea is to push bidders
to achieve the projected value for money savings in practice. A
“scope ladder” is also defined, defining how and in what order
of priority certain specifications could be removed or reduced,
in case no bidder can come in below this affordability ceiling
without adjusting the scope of the project.
• In Korea, the results of the VFM assessment are used in the
bid process in two ways. First, in the case of an unsolicited
proposal, the results of VFM test (broadly speaking—including
economic viability) inform the bonus awarded to the proponent
in the subsequent competitive bidding process7. Secondly, the
government usually uses the VFM analysis results to set a “bottom
line” for price bidding, with a view to achieving VFM.
• In France, in the absence of an official doctrine, the results of VFM
analysis can be communicated to candidates as part of project
documentation, to let them know what to expect with respect to
overall cost assumptions. This approach is favored by the national
PPP Unit as it ensures all participants have equal access to the
information contained in the VFM analysis, and helps avoid initial
offers that differ greatly in scope from the envisaged project.

Roundtable participants also noted that communicating VFM analysis to

bidders can help ensure a level playing field—recognizing that even if this

7 For more on Korea’s approach to dealing with unsolicited PPP proposals, see Hodges and Dellacha
(2007) “Unsolicited Infrastructure Proposals: How Some Countries Introduce Competition and
Transparency”, PPIAF Working Paper No. 1

21 PPP Decision-Making: How VFM Analysis Is Used

analysis is not formally shared as part of bidding documents, it may reach
some potential bidders through more indirect means.

Whether or not the VFM analysis is communicated to bidders, many

governments—such as the UK, Korea, and France—carry out a final “VFM
check”, in which final bids are compared with the final version of the PSC
(which may have been updated through the course of the tender process,
as contract details and specifications are finalized) prior to signing a PPP
contract.

Value-for-Money Analysis—Practices and Challenges 22

4. Quantitative VFM Analysis: the PSC and
Key Methodological Issues
The overall process of quantitative VFM analysis is common to many
countries. As illustrated in Figure 1 below, it typically begins with comparing
the estimated cost to government of a project under a PPP model (or a
range of possible contractual models) to that under a traditional public
procurement model. These costs are often then adjusted to take into
account various differences between the options—such as tax implications
or risk allocation—before discounting the cost streams to reach a present
value figure. Nonetheless, several challenges with quantitative VFM
analysis—and methodological questions on which different PPP programs
diverge—were raised during the PPP round-table. These key issues are
highlighted in Figure 1, and described in turn below.

Key Issue 1: Cost and revenue assumptions

A significant driver of the results of VFM analysis is the assumptions made
on project costs under public and private options—in particular, the extent
to which a PPP is assumed to achieve lower costs, through efficiency or
innovation. For countries with longer histories of PPP such assumptions
can be informed by actual historic project outturns, although recognizing
changing circumstances remains an issue; the challenge is greater for
countries with less PPP experience. For example:

• In France, earlier VFM analyses typically assumed a percentage

reduction in capital costs under a PPP, compared to public
procurement. However, a 2013 report by the national audit entity

23 Quantitative VFM Analysis: the PSC and Key Methodological Issues

Figure 1: Overview of Quantitative VFM Analysis and Key Methodological
Issues
Initially estimated by building “shadow” financial
Government
model of PPP project−based on assumed costs,
payments under PPP
revenues, and financial structure; later
Model(s)
confirmed through bidding process
vs 1. Cost and revenue
assumptions:
Capital, O&M, and financing cost under nature & extent of
“Raw” Public Sector “traditional” Government procurement (or a assumed private
Comparator (PSC) range of possible procurement models)−offset sector efficiencies
by estimated revenues

• Procurement and transaction costs

2. Scope of analysis:
Adjusted for “fair” • Competitive neutrality
extent of adjustments
comparison • Cost of risk 3. Approach to valuing
to “raw” PSC approach
• Non-financial benefits risk

Discounted to 4. Discount rate used

present value terms

VFM Expressed in absolute terms, or as a percentage of PSC

recommended that such an assumption should only be made if

based on data on actual project outturns in a particular sector.
Increasingly such data is available in France, as the PPP program
develops—for example, PPP schools have been found to achieve
on average a 15 percent saving in capital expenditure compared
to traditionally-procured schools.
• In the UK, a large database of more than 700 PFI projects provided a
good basis for PFI cost estimates—although assumptions had also
been revised for changes such as the increased cost of long-term
private sector finance over recent years. On the other hand, public
procurement has been improving over time, making historical
cost differentials between PPP and traditional public procurement
relatively less useful in projecting costs going forward.
• In Chile, an assumption of private sector efficiency over public
procurement is currently built into their VFM analysis—a
“significant assumption” that is based on very limited experience
in practice with social sector PPPs.

Differences in project revenue assumptions between PPP and public

procurement may also affect VFM results. In Korea, the treatment of toll
revenue in VFM analysis has been a point of concern. Currently the same toll
revenue is assumed under PPP and public procurement options—although

Value-for-Money Analysis—Practices and Challenges 24

this may be unrealistic, as experience suggests tolls on public roads are
typically set lower than on PPP roads, resulting in higher traffic, and most
likely differing revenues (as well as economic benefits). Also in Korea, the
revenue from ancillary uses of assets is assumed to be the same under both
PPP and public options; whereas in France such additional revenue sources
(and associated investment) are typically assumed only to apply in the PPP
case—if only because administrative law makes it difficult for a government
entity to engage in commercial activities that are not core to its function.

Key Issue 2: Scope of analysis

As described in the overview in Section 2, and highlighted in Figure 1, the

scope of quantitative VFM analysis varies. From a starting point of simple
cost estimates, most countries make adjustments to capture additional
costs or benefits, and “level the playing field” between PPP and public
procurement options. Table 1 on page 12 describes these adjustments, and
their implications for the scope of VFM analysis in different countries.

Table 1: Adjustments to VFM analysis

Adjustment Description and Country Approaches

Management The cost to government of project management and transaction implementation
and may differ between a PPP and traditional public procurement. Treatments of these
transaction costs vary: some governments, such as France, adjust both PSC and PPP cost
costs accordingly; while Korea excludes contract management costs from both options.
“Competitive Most governments adjust the PSC estimate to level out apparent cost
Neutrality” advantages of implementation by a public body. These can include adjusting
to compensate for differences in cost of land acquisition by public and private
entities (in Korea, for example, the same land acquisition schedule and cost is
applied under both models), and in tax liabilities.
Cost of One of the key differences between a PPP and traditional public procurement is how
bearing risk risk is allocated—and hence the riskiness, or variability of the cost to government of
the project. Approaches to capturing the cost of risk in VFM analysis vary:
Many governments (such as Korea, and South Africa) adjust for the
government’s risk exposure by building into the PSC the cost of bearing key
risks that would be transferred to the private partner under a PPP model (with
risks retained by government in both cases assumed to cancel out). The cost of
risk-bearing in the PPP model is assumed to be built into the cost of financing,
plus contingencies in construction and operating budgets.
In France, British Columbia, Canada, and Virginia, USA, cash flows under
both the PSC and shadow PPP model are adjusted for risk, but with different
probabilities and risk preferences to reflect the different apt itude and cost of
risk-bearing of government and the private party.

25 Quantitative VFM Analysis: the PSC and Key Methodological Issues

 In Chile, on the other hand, no risk adjustment is made—data on risks,
particularly cost overruns, under PPPs and public projects are simply considered
too scarce to make useful assumptions.
Adjusting for risk raises its own methodological issues, as described further
below.
Non-financial Most governments’ quantitative VFM analysis assumes that implementing a
benefits project as a PPP leaves project benefits unchanged—any implication for quality
or timeliness of service is left to qualitative analysis.
One exception is France, where the higher benefit associated with expected
earlier completion of a PPP project is included in the analysis, in part to
offset the implications of faster capital expenditure. This benefit is currently
approximated by MAPPP, by using total project cost as a proxy for project
benefits, and calculating the value of bringing forward that benefit by x years
at the social discount rate—a simple approach that is expected to be refined
shortly, as part of a broader review and reform of the VFM approach that will
place greater emphasis on non-financial benefits.

Risk quantification methodologies and assumptions

As noted in Table 1 above, most governments incorporate into VFM analysis

the cost of bearing risk—recognizing that risk allocation is an important
distinction between PPP and more traditional models of procurement.
Generally, the approach taken is to add back to the analysis the cost of any
significant risks that will be transferred under the PPP model—that is, risks
that are retained by government under both public sector and PPP models
are assumed to cancel out.

However, this raises its own methodological challenges: how to quantify

the cost of risk-bearing. For example, in France two different approaches
are used. For smaller projects (less than €50 million), a “mean value” is
used, and calculated by estimating the probability and impact of any
significant risk events. Larger projects use Monte Carlo simulations, in
which distributions are assumed for the likelihood and impact of each risk,
to calculate a distribution of possible costs. The number and level of detail
of risks assessed vary by country—France has developed risk distributions
for a range of project risks; whereas in Korea the focus is on the overall risk
of project cost and time over-runs.

This type of quantitative risk analysis is complex, and requires sophisticated

financial modeling, as well as data on risk outcomes from previous projects
to inform assumptions on probability distributions. This has been a
challenge in many countries. For example, the probability distributions used

Value-for-Money Analysis—Practices and Challenges 26

in France have yet to be backed by more evidence from PPP experience—
the current approach was developed with advice from an insurance broker.
Moreover, historical experience may not necessarily be an indicator of
future performance—for example, in the UK, improvements over time in
public procurement (in particular efforts to address optimism bias) have
significantly reduced the risk of cost over-runs of public projects.

As described above, Chile does not adjust for risk given the lack of data
on project risk outcomes under PPPs or public procurement. Elsewhere in
Latin America, governments have struggled to implement in practice the
VFM analysis methodologies set out in guidance material—due both to a
lack of capacity to implement the complex analysis, and a lack of data to
inform assumptions8.

Discount rates

The final step in calculating the relative VFM of PPP or traditional government
procurement options is typically to calculate the net present value (NPV)
of government payments under each option. Since government cash flow
profiles vary significantly between PPP and traditional procurement models,
the discount rate applied can have a significant impact on the result of the
VFM analysis.

Most governments represented at the round-table (for example, France,

Chile, Korea, and Virginia, USA) use the appropriate government (that is, risk-
free) borrowing rate to discount cash flows under both procurement options.
The justification is that the cost of risk-bearing is built in to the analysis explicitly
through its cash-flow impact and the cost of financing in the case of the PPP
model; moreover, risk-reflective discount rates are more typically used to
capture riskiness of income streams, rather than payment streams.

The government of British Columbia, Canada, on the other hand, uses a

risk-adjusted “project Internal Rate of Return (IRR)”—set by PPP Unit staff
based on their previous project experience—to discount cash flows under
both public and PPP models. Bidders are then required to use the same
discount rate when calculating the value of bids in NPV terms, allowing
direct comparison of bids with the PSC and shadow PPP calculations.

8 Presentation by Daniel Benitez to PPP Roundtable on experience in Latin America with VFM analysis.

27 Quantitative VFM Analysis: the PSC and Key Methodological Issues

5. Conclusions and Interesting Questions
The use of value for money analysis to inform PPP decision-making is
difficult, and can be controversial. Practitioners face some significant
methodological challenges—particularly given very limited ex-post VFM
information or other data on PPP project outturns—that mean conclusions
of VFM analysis can be misinterpreted or worse, manipulated.

Nonetheless, governments and infrastructure users benefit from

having VFM at the center of PPP decisions. Sometimes PPP is seen as
the only option to deliver a project—because of implementation capacity
constraints, or the perceived creation of “fiscal space” by PPPs, whether
genuine or not. Even in these cases, there is much to be gained from doing
VFM analysis to sense-check the decision to pursue the project, and the
proposed PPP structure—it is worthwhile to governments to understand
whether or not implementing a project now as a PPP comes at a cost, and
if so, to weigh this cost against the associated benefits.

In this light, there is much to be gained in strengthening VFM analysis going

forward. To that end, interesting lessons from the round-table included the
following:

• Governments need to strike the right balance between

qualitative and quantitative approaches—particularly in new
PPP programs, where there is very limited data available to
inform assumptions for quantitative analysis; and in some cases
a lack of capacity to implement complex risk analysis. Generally
speaking, this will involve greater emphasis and scrutiny on
qualitative aspects of PPP decision-making—for example through

Value-for-Money Analysis—Practices and Challenges 28

 clear guidelines and criteria for picking potential PPPs—and
developing simplified approaches to quantitative VFM analysis.
• Governments should be realistic about the nature of
quantitative VFM analysis. Quantitative analysis can be useful
to inform decision-making, but should be understood and
communicated more as a tool to consistently and systematically
assess the combined result of a set of assumptions, than as a
scientific process that provides “proof” of VFM.
• Thorough risk analysis is crucial to successful PPPs. Many
participants saw VFM analysis as important in part because it
requires thorough and systematic risk analysis. Whether or not
quantitative VFM analysis is carried out, this highlights that sound
risk analysis is crucial—to achieving value from a project both in its
design (through sound project structuring) and implementation
(through effective risk management), and to avoid fiscal surprises.
• Better data is needed on PPP and major infrastructure
investment project outturns. Quantitative approaches to VFM
analysis—and risk analysis more generally—could be improved
significantly by more systematic collection of data on actual PPP
project outturns, and ex-post assessment of VFM achieved in
practice. Round-table participants noted this as an area where
the World Bank Group could make a valuable contribution—in
collecting new information on PPP performance, as well as in
identifying and creating mechanisms by which existing country-
level data can be effectively pooled and shared.
• Ultimately, VFM analysis should be integrated with overall
public investment planning. Some governments are moving
towards application of VFM analysis (both qualitative and
quantitative) to assess a range of possible project structures.
Going forward, several round-table presenters noted that VFM
analysis could and should be systematically applied to all major
infrastructure projects, to assess the best procurement option9.

Much as for other aspects of developing, appraising, and implementing a

PPP project, value for money analysis—particularly quantitative risk analysis—

9 For a more detailed discussion of a “procurement option test” to be applied to all investment
projects, see the final chapter of Philippe Burger and Ian Hawkesworth (2011) How to Attain Value
for Money: Comparing PPP and Traditional Infrastructure Public Procurement, OECD Journal on
Budgeting Volume 2011/1

29 Conclusions and Interesting Questions

can be time and resource-intensive. As is the case for the PPP programs
represented at the roundtable, most governments benefit from establishing
dedicated teams to oversee the PPP development and appraisal process,
and rely on the support of experienced advisors in doing so.

Finally, participants agreed that VFM analysis is just the start of the process
of achieving value through a PPP. The best-structured and assessed PPP still
requires careful shepherding over the project lifetime—with well-defined
contract management structures, attentive management of emerging risks,
and an appropriately flexible approach to dealing with change—to achieve
value for money in practice.

Value-for-Money Analysis—Practices and Challenges 30

Annex A. Global VFM Roundtable 28 May
2013: Agenda and Participants
This report presents and summarizes the discussion and conclusions of a
Global Roundtable on Value for Money analysis, held at the World Bank on
May 28, 2013. The aim of the roundtable was to bring together experienced
PPP practitioners from a range of countries, to discuss experience with VFM
analysis and its use in PPP decision-making. The event was jointly presented
by the World Bank Institute (WBI) and the Public-Private Infrastructure
Advisory Facility (PPIAF).

Lincoln Flor, Senior Public-Private Partnerships Specialist, WBIPP, was the

task team leader and John Saville, Program Assistant, WBIPP, organized
the logistics and administrative arrangements.

The roundtable was introduced by Abha Joshi-Ghani, Director of Knowledge

Exchange and Learning, WBI. Lincoln Flor made an introduction to Value
for Money methodology and the objectives of the roundtable, followed by
Satheesh Kumar Sundararajan, Infrastructure Finance Specialist, PPIAF, who
provided opening remarks and introduced some core themes on Value for
Money Analysis.

The full-day event comprised three sessions, described below.

Session 1: Findings on VfM practices in OECD countries

The first session on VFM practices in OECD countries was moderated by

Clive Harris, Practice Manager, WBI. The presenters were:

31 Annex A
 • Ian Hawkesworth, Coordinator, OECD PPP Network, OECD
• James Ballingall, Head of Assurance Team, Infrastructure UK,
United Kingdom
• Francois Bergère, Director, Mission to Support Public-Private
Partnerships (MAPPP), Ministry of Economy and Finance, France

Session 2: Country Experiences and Lessons Learned I

The second roundtable session, focusing on country experiences with VFM
analysis, was moderated by Aurelio Menendez, Transport Sector Manager,
World Bank. The presenters were:

• Morteza Farajian, Public-Private Partnerships Program Manager,

Department of Transportation, Commonwealth of Virginia, USA
• David Duarte, Head of Contingent Liabilities and Concessions,
Budget Office, Ministry of Finance, Chile
• Hyeon Park, Executive Director, Public and Private Infrastructure
Investment Management Center (PIMAC), Korea Development
Institute, Republic of Korea

Session 3: Country Experiences and Lessons Learned II

A third session, also on country experiences with VFM analysis, was

moderated by Adriana de Aguinaga, Manager, PPP Advisory, International
Finance Corporation. The presenters were:

• Daniel Benitez, Senior Transport Economist, Latin America and the

Caribbean (LAC) region Sustainable Development Department,
World Bank
• Mark Liedemann, Assistant Vice-President, Partnerships BC,
British Columbia, Canada
• Vivek Aggarwal, Managing Director, Madhya Pradesh Road
Development Corporation, Madhya Pradesh, India
• William Dachs, Senior Executive Manager, Gautrain Management
Agency, South Africa.

A wrap-up discussion was chaired by Clive Harris, Manager, WBIPP, and the
workshop was closed by Jose Luis Irigoyen, Director, and Head of Global
Expert Team on PPP, World Bank.

Value-for-Money Analysis—Practices and Challenges 32