Selecting The Wrong Person

Choosing the Wrong Portfolio of Projects
5 Reasons Organizations Choose the Wrong Projects (and What to Do About It)
Lee Merkhofer, Ph.D.
This 6-part paper is an expanded version of a paper originally prepared as a guest article for Max Widemans Project Management website, http://wwwmaxwideman.com. Versions of the paper have since been republished several times, including in the Project Management Institutes newsletter, The Measured, as chapters in the books A Management Framework for Project, Program and Portfolio Integration, by Trafford Publishing, and Readings in Information Technology Management, by Thomson Learning.
c 2003-2009 Lee Merkhofer Consulting
Abstract
There are five major reasons that organizations choose the wrong projects: (1) biases and errors in judgment, (2) failure to establish an effective framework for project portfolio management, (3) lack of the right metrics for valuing projects, (4) inability to assess and value risk, and (5) failure to identify project portfolios on the efficient frontier As a consequence of these shortcomings, organizations underestimate the value derived from projects; conduct too many small, low-value projects; don't kill failing projects soon enough; unknowingly take on high-risk projects; make project choices based on political considerations not in the best interest of the enterprise; and use inefficient decision-making processes that discourage trust and honesty. By my estimate, such problems cause organizations to obtain only about 60% of the value that could be derived from their project portfolios. Methods are available for overcoming the reasons organizations choose the wrong projects. Success requires cultural change, organizational re-engineering, improving decision-making competencies, and institutionalizing best-practice analytic tools that fit the organization and its needs. Organizations that address the reasons for choosing the wrong projects can dramatically improve performance and create a significant, competitive business advantage.
www.prioritysystem.com
Table of Contents
Abstract ................................................................................................................................................... i Introduction .............................................................................................................................................1 Errors and Bias in Judgment ...................................................................................................................2 Heuristics and Judgmental Biases .................................................................................................. 3 Estimating and Forecasting Biases............................................................................................... 11 Bounded Awareness and Decision-Making Heuristics ................................................................ 15 Debiasing...................................................................................................................................... 20 References for Part 1 .................................................................................................................... 26 Failure to See the Forest for the Trees ..................................................................................................28 Problems with Project-by-Project Decision Making.................................................................... 28 Combine Projects into Project Portfolios ..................................................................................... 33 Establish a Portfolio Management Office .................................................................................... 37 Improve the Prioritization Process ............................................................................................... 44 References for Part 2 .................................................................................................................... 48 Lack of the Right Metrics......................................................................................................................49 Defining Project Value................................................................................................................. 50 Project-Selection Decision Models .............................................................................................. 57 Is It Hard to Create a Project Decision Model? ........................................................................... 63 The Scientific Method .................................................................................................................. 68 Project Valuation Metrics............................................................................................................. 73 References for Part 3 .................................................................................................................... 78 Inattention to Risk .................................................................................................................................79 Risk and Risk Management.......................................................................................................... 80 Project Selection Risk .................................................................................................................. 84 Characterizing Risks with Probabilities ....................................................................................... 88 Risk Tolerance.............................................................................................................................. 94 References for Part 4 .................................................................................................................... 97 Inability to Find the Efficient Frontier ..................................................................................................98 The Efficient Frontier................................................................................................................... 99 Efficient Frontiers versus Ranking Curves ................................................................................ 103 A Real Efficient Frontier............................................................................................................ 107 Importance of the Efficient Frontier........................................................................................... 110 Achieving Best-Practice ......................................................................................................................113 Address the Reasons Organizations Choose the Wrong Projects .............................................. 114 Five Levels of Project Portfolio Management ........................................................................... 116 Best-Practice Project Portfolio Management ............................................................................. 122 Why Isn't Everyone Doing this Already?................................................................................... 126 Competing on Analytics............................................................................................................. 133 References for Part 6 .................................................................................................................. 134
ii
Introduction
If you've attended recent business conferences, you may have heard business leaders and management consultants speak of the "60% solution." The phrase refers to the belief that organizations obtain only about 60% of the value that could be derived from their businesses. The remaining 40% of available value is lost, reportedly, due to errors in decision making and weaknesses in business systems. I believe 60% is an accurate characterization of the way most organizations select and manage their project portfolios. As demonstrated in Part 5 of this paper, it is possible to compare, using models, the value generated by an organization's current set of project choices with the value that would be generated under a value-maximizing set of choices. My comparisons (similar results have been obtained by others) show that organizations can typically increase value by 20-40% without increasing costs, or decrease costs by 20-40% without decreasing value, each budget cycle, by making better project decisions. Yes, 40% is BIG, and you may not be convinced by a claim based solely on models. Therefore, this 6-part paper offers further evidence by explaining the 5 major reasons that organizations mismanage their project portfolios: (1) errors and biases in judgment, (2) failure to see the forest for the trees, (3) lack of the right metrics, (4) inattention to risk and (5) inability to find the efficient frontier. As you will see, each of these reasons can by itself produce significant losses. It should not be surprising, therefore, that the combined effect could be a 40% loss of value. As you read the 5 reasons, consider the degree to which each is a factor in your organization. Understanding organizational dysfunctions is useful even if your intention is not to invest in a stateof-the-art project priority system. By taking steps to mitigate problems that affect your organization, you may be able to reduce costs while obtaining more value from your projects. Given the importance of project decisions, obtaining even small improvements will be worth the effort.
Errors and Bias in Judgment

Why do we need decision aids? Can't people make good choices on their own? Like many decision analysts, I was first attracted to the science of decision making as a result of reading about the errors and biases that affect people's judgments. Remarkably, it appears that our brains have been hard-wired to make certain errors. Hundreds of different biases have been identified and categorized, including biases that distort our judgments, that introduce errors into the estimates and forecasts that we produce, and that cause us to make the wrong choices. If you're not already familiar with the major results from this fascinating field of research, this introduction should help you to appreciate the value of formal decision-aiding tools. Without the assistance of such tools, the decisions made within organizations, including choices of which projects to conduct, will be systematically biased. Errors and bias in judgment is the first reason that organizations choose the wrong projects.
Choosing the Wrong Portfolio of Projects Part 1: Errors and Bias in Judgment
Heuristics and Judgmental Biases

The fact that people's intuitive decisions are often strongly and systematically biased has been firmly established over the past 50 years by literally hundreds of empirical studies. Psychologist Daniel Kahneman received the 2003 Nobel Prize in Economics for his work in this area. The conclusion reached by Kahneman and his colleagues is that people use unconscious shortcuts, termed heuristics, to help them make complex decisions. "In general, these heuristics are useful, but sometimes they lead to severe and systematic errors" [1]. Understanding heuristics and the errors they cause is important because it helps us find ways to counteract them. For example, when judging distance people use a heuristic that equates clarity with proximity. The clearer an object appears, the closer we perceive it to be. Although this heuristic is usually correct, it allows haze to trick us into thinking that objects are more distant than they are. The effect can be dangerous. Studies show people often drive faster in fog because reduced clarity and contrast make going fast appear slower. Airline pilots are similarly tricked, so pilots are trained to rely more on instruments than on what they think they see out the cockpit window.
Figure 1: Haze tricks us into thinking objects are further away.
Some of the dozens of well-documented heuristics and related errors and biases are summarized in the table below.
Comfort Zone Biases People tend to do what's comfortable rather than what's important.
Perception Biases People's beliefs are distorted by faulty perceptions.
Motivation Biases People's motivations and incentives bias their judgments.
Errors in Reasoning People use flawed reasoning to reach incorrect conclusions.
Groupthink Group dynamics create additional distortions.
People: Become attached to the status quo. Value things more highly if they already own them. Seek information that confirms what they already suspect. Ignore information inconsistent with their current beliefs. Are more likely to choose an alternative if it is not the most extreme one considered. Make choices or predict outcomes based on what is pleasing to imagine rather than on rational evidence. Fail to learn and correct their beliefs despite strong evidence that they should. Keep doing the same things, even if they no longer work. Distort their views of reality in order to feel more comfortable.
People: See things according to the conventions of their profession, ignoring other views. Overlook and ignore unexpected data. See patterns in data where none exist. Anchor on information that is readily available, vivid or recent. Make insufficient adjustments from their initial anchors. Ascribe more credibility to data than is warranted. Overestimate what they know. Underestimate the time/effort to complete a difficult task. Perceive recent events as more distant and very distant events as less distant. Give different answers to the same question posed in different ways.
People: Behave differently when they think they are being observed. Unconsciously distort judgments to "look good" and "get ahead." Remember their decisions as better than they were. Take actions as if concerned only with short-term consequences. Attribute good decisions to skill, bad outcomes to others' failures or bad luck. Escalate commitments to avoid questioning earlier decisions. Favor actions that shield them from unfavorable feedback. May have the urge to do the opposite of what is suggested to resist a perceived constraint on freedom of choice.
People: Believe they can control outcomes that they cant. Simplify inappropriately. Are persuaded by circular reasoning, false analogies, and other fallacious arguments. Are surprised by statistically likely "coincidences." Base the credibility of an argument on its manner of presentation. Abhor risk but seek bigger risks to avoid a sure loss. Disregard probabilities when making decisions. Prefer eliminating a small risk to reducing a large risk. Think an unusual string of random outcomes (e.g., heads) makes that outcome more likely in the future Cannot solve even simple probability problems in their heads.
Groups: Give preferential treatment to those perceived to be group members. Reinforce their beliefs via the bandwagon effect. "Dive in" when solving problems without having all the necessary information. Are excessively cautious in sharing data. Assume they agree when they dont. Avoid expressing inconsistent, opposing views. Jump to conclusions prematurely or get bogged down trying to reach agreement. Discount solutions not invented here. Create illusions of invulnerability and ignore external views of the morality of their actions.
The following summary, including ideas for countering biases, is derived from some of the many excellent papers on the subject, especially the popular 1998 Harvard Business Review article by Hammond, Keeney, and Raiffa [2]. Status Quo Bias Status quo bias refers to the tendency people have to prefer alternatives that perpetuate the status quo. Psychologists call this a "comfort zone" bias based on research suggesting that breaking from the status quo is, for most people, emotionally uncomfortable. It requires increased responsibility and opening oneself up to criticism. For example, if you are a company executive considering introducing a lowercost/lower-quality version of an existing product to your product line, you face the risk of adversely affecting the perceptions of customers who choose your high-quality products. If things work out badly, you could be accused of making a bad choice. Just considering the change forces you to confront the trade-off between increased profits and the risk of damaging your brand image. Sticking to the status quo is easier because it is familiar; it creates less internal tension. There are often good reasons, of course, for leaving things unchanged. But, studies show that people overvalue the status quo. In one experiment, each participant in a group of students was randomly given a gift consisting of either a coffee mug or a candy bar. When offered the opportunity to trade, few wanted to exchange for the alternative gift. It is not likely that students who naturally preferred the coffee cup to the candy bar received their preferred gift by chance. Apparently, "owning" what they had been given made it appear more valuable.
Would you trade?
The power of this bias was quantified in a related experiment. Students were randomly chosen to receive mugs. Those with mugs were asked to name the minimum price at which they would sell their mugs. Those without were asked to name the maximum price at which they would buy. The median selling price was more than twice the median offer price. Again, ownership increased perceived value. Sometimes referred to as the endowment effect, this bias may help explain why investors are often slow to sell stocks that have lost value. Likewise, it might be a factor for explaining why companies have trouble disposing of bad business assets. Social norms tend to reinforce preference for the status quo. For example, courts (and many organizations) view a sin of commission (doing something wrong) as more serious than a sin of omission (failing to prevent a wrong). As another example, government decision makers are often reluctant to adopt efficiency-enhancing reforms if there are "losers" as well as "gainers." The status quo is seen as fair; any change is seen as unfair. The burden of proof is on the side of changing the status quo. Lack of information, uncertainty, and too many alternatives promote holding to the status quo. Thus, many organizations continue to support failing projects due to lack of solid evidence that they've failed. Killing a project may be a good business decision, but changing the status quo is uncomfortable for the people involved.
What causes status quo bias? According to psychologists, when people face the opportunity of changing their status quo, the loss aspects of the change loom larger than the gain aspects. Losses represent the certain elimination of visible, existing benefits. Gains, in contrast, are prospective and speculative. We know what we have, who knows what we will get? We fear regret, and this fear is amplified by our desire to maintain the respect and approval of others. In business, the key to success is often bold action, but for many CEOs, the only thing worse than making a strategic mistake is being the only one in the industry to make it. Sticking with the status quo is safer. The best advice for countering the bias toward the status quo is to consider carefully whether status quo is the best choice or merely the most comfortable one: When you hear comments like "let's wait and see" or "let's meet next month to see how the project is going," ask yourself whether you're hearing status quo bias. Think about what your objectives are and whether they are best served by the status quo or a change. Ask yourself whether you would choose the status quo alternative if, in fact, it wasnt the status quo. Avoid overestimating the difficulty of switching from the status quo. Actively manage migration away from the status quoensure dissatisfaction with the status quo and communicate the same message repeatedly about the need for the change. Identify who might be disadvantaged by changing the status quo, and look for ways to mitigate or compensate for those disadvantages. Note that a change becomes the status quo over time. Evaluate alternatives in terms of how they will be viewed in the future as well how they are perceived in the present. Sunk Cost Bias We know rationally that sunk costspast investments that are now irrecoverableare irrelevant to decision making. Sunk costs are the same regardless of the course of action that we choose next. If we evaluate alternatives based solely on their merits, we should ignore sunk costs. Only incremental costs and benefits should influence our choices. Yet, the more we invest in something (financially, emotionally, or otherwise), the harder it is to give up that investment. For example, in a telephone call, being on hold and hearing the recording, "Your call is important to us...Please stay on the line," often means that you've got a lot longer to wait. Still, it's hard to hang up and call another time. There is a great deal of research demonstrating the influence of sunk costs. In one study, students were shown to be more likely to eat identical TV dinners if they paid more for them. Another study arranged to have similar tickets for a theater performance sold at different pricespeople with the more expensive tickets were less likely to miss the performance. A third study found that the higher an NBA basketball player is picked in the draft, the more playing time he gets, even after adjusting for differences in performance.
The Concorde supersonic airplane is often cited as an example of sunk cost bias. It became obvious early on that the Concorde was very costly to produce and, with few seats, was unlikely to generate adequate revenue. Few orders for planes were coming in. Still, even though it was clear that the plane would not make money, France and England continued to invest.
An example of sunk cost bias?
Sunk cost reasoning shows up frequently in business. For example, you might be reluctant to fire a poor performer you hired in part because you may feel to do so would be an admission of earlier poor judgment. You might be inclined to give more weight to information you paid for than to information that was free. You might find it harder to terminate a project if youve already spent a lot on it. Why is it so difficult to free oneself from sunk cost reasoning? Many of us appear to be born with strong feelings about wasting resources. We feel obligated to keep investing because, otherwise, the sunk cost will have been wasted. We would then need to admit (at least to ourselves if not to others) that we made a mistake. It has even been suggested that sunk cost reasoning may be a kind of self-punishment. We may unconsciously force ourselves to follow through on commitments that no longer seem desirable in order to tell ourselves to be more careful next time. Techniques for countering sunk cost bias include: Ask yourself what another manager would do in your place, one without a prior history in the investment. Seek the opinions of people who were uninvolved in the original choice. Be alert to sunk cost bias in the decisions and recommendations made by others. Comments like weve invested so much already and we dont want to waste those resources are signals. Consider re-assigning responsibilities. Avoid creating a mistake-fearing culture within your organization. Set an example by admitting when you are wrong. Change course quickly without regard to the sunk costs of investments that have gone bad. Remember that even smart choices (taking into account what was known at the time the decision was made) can have bad outcomes. Cutting your losses does not necessarily mean that you were foolish to make the original choice. Try thinking of the sunk cost not as an investment that was wasted, but as an investment that (perhaps indirectly) led to valuable information indicating that a course change is needed. Follow the advice of Warren Buffet: When you find yourself in a hole, the best thing you can do is stop digging. Supporting Evidence Bias Supporting evidence bias is our tendency to want to confirm what we already believe and look for facts to support it. This bias not only affects where we go to collect information, but also how we interpret the evidence that we receive. We avoid asking tough questions and discount new information that might challenge our preconceptions.
Suppose, for example, that you are considering an investment to automate some business function. Your inclination is to call an acquaintance who has been boasting about the good results his organization obtained from doing the same. Isnt it obvious that he will confirm your view that, "It's the right choice"? What may be behind your desire to make the call is the likelihood of receiving emotional comfort, not the likelihood of obtaining useful information. Supporting evidence bias influences the way we listen to others. It causes us to pay too much attention to supporting evidence and too little to conflicting evidence. Psychologists believe the bias derives from two fundamental tendencies. The first is our nature to subconsciously decide what we want to do before figuring out why we want to do it. The second is our tendency to be more engaged by things we like than by things we dislike. Despite our inclination to look for supporting evidence, it is usually much more informative to seek contradictory evidence. Confirming evidence often fails to discriminate among possibilities. To illustrate, in one study students were given the sequence of numbers 2, 4, 6 and told to determine the rule that generated the numbers. To check hypotheses, they could choose a possible next number and ask whether that number was consistent. Most students asked whether a next number "8" would be consistent with the rule. When told it was, they expressed confidence that the rule was, "The numbers increase by 2." Actually, the rule was, "Any increasing sequence." A better test would have been to check whether a next number incompatible with the hypothesis (e.g., 7) was consistent with the unknown rule. Supporting evidence bias can cause us to perpetuate our pet beliefs. For example, if a manger believes people are basically untrustworthy, that manager will closely monitor their behavior. Every questionable act will increase suspicions. Constantly watching employees will make it impossible to develop mutual trust. Studies show that when people are placed in situations where authority figures expect them to cheat, more of them do, in fact, cheat. The behavior pattern reinforces itself to everyones detriment. Changing what we believe takes effort. When first encountered, data that conflicts with our preconceptions is often interpreted as being the result of error, or to some other externally attributed factor. It is only after repeatedly being exposed to the conflicting information that we are willing to make the effort to change our beliefs. Some advice for avoiding supporting evidence bias: Check to see whether you are examining all the evidence. Avoid the inclination to accept confirming evidence without question. Get in the habit of looking for counter arguments. In meetings, consider appointing someone to serve as devil's advocateto argue against the prevailing point of view. If that seems too uncomfortable, at least appoint a devils inquisitor someone with responsibility to ask tough questions. When you encounter something that conflicts with your beliefs, dig deeper. Resist the temptation to dismiss data that makes you uncomfortable.
Framing Bias The first step in making a choice is to frame the decision, but it is also where you can first go wrong. The way a problem is framed strongly influences the subsequent choices we make. People tend to accept the frame they are given; they seldom stop to reframe it in their own words. A frame that biases our reasoning causes us to make poor decisions. Edward Russo and Paul Shoemaker [3] provide a story to illustrate the power of framing. A Jesuit and a Franciscan were seeking permission from their superiors to be allowed so smoke while they prayed. The Franciscan asked whether it was acceptable for him to smoke while he prayed. His request was denied. The Franciscan asked the question a different way: In moments of human weakness when I smoke, may I also pray? Of course, this frame elicited the opposite response. Whether outcomes are described as gains or losses influences people's choices. In one experiment, people were asked to express their preferences among alternative programs impacting community jobs. They were told that due to a factory closing 600 jobs were about to be lost. However, if program A is adopted, 200 jobs will be saved. On the other hand, if program B is adopted, there is a 1/3 probability that 600 jobs will be saved and a 2/3 probability that none of the 600 jobs will be saved. Most people preferred program A. Another group was given a rephrasing of the choice. If program C is adopted, they were told, 400 people will lose their jobs. If program D is adopted, there is a 1/3 probability that nobody will lose their job and a 2/3 probability that 600 will lose their job. This group mainly favored program D. Similar effects occur in less-controlled, everyday circumstances. For example, it sounds more positive to say that a new product launch has a "1-in-10 chance of succeeding" compared to the mathematically equivalent statement that it has a "90% chance of failing." If people are rational, they should make the same choice in every situation in which the outcomes are identical. It shouldn't matter whether those outcomes are described as "gains" or "losses" or "successes" or "failures." But, the choice establishes different frames, and decisions often differ because of it. Another example, described by Hammond, Keeney and Raiffa [2], involves automobile insurance laws voted on in New Jersey and Pennsylvania. Each state gave voters a new option: By accepting a limited right to sue they could lower their insurance premiums. New Jersey framed the initiative by automatically giving drivers the limited right to sue unless they specified otherwise. Pennsylvania framed it by giving drivers the full right to sue unless they specified otherwise. Both measures passed, and in both cases large majorities of drivers defaulted to the status quo. But, because of the way Pennsylvania framed the choice, drivers in that state failed to gain about $200 million in expected insurance savings. Advice: Ask yourself if you are working on the real problem. Look for implicit assumptions or unnecessary constraints in the way that you perceive your problem. To promote objective reasoning, avoid framing alternatives with value-laden terminology (e.g., labeling a proposed resource allocation as "fair").
Try posing problems in a neutral way that combines gains and losses, adopts alternative reference points, or promotes objectivity. Look at the problem from other perspectives. For example, reverse the context. If you are the seller, how would you see things if you were the buyer? Choose a frame that captures all of what's important. For example, ask, "What's the total cost of ownership?" not "What's the price?" Watch out for leading questionsquestions or phrasing designed to create a frame intended to elicit a particular response. Choose a high-level perspective for framing. For example, What is the aggregate risk of the project portfolio? Looking only at project-by-project risk may result in a portfolio of overly conservative projects.
10
Estimating and Forecasting Biases

People are notoriously poor at estimating and forecasting. They interpret statistical correlation as implying cause-and-effect. They tend to naively extrapolate trends that they perceive in charts. They draw inferences from samples that are too small or unrepresentative. They make overly optimistic forecasts that cannot be justified by available information. They routinely overestimate their abilities and underestimate the time and effort required to complete difficult tasks. Estimating and forecasting biases represent a special class of biases important to project-selection decision making. Misestimating Likelihoods Uncertain situations are particularly troublesome. Studies show that people make systematic errors when estimating how likely uncertain events are. As shown in Figure 2, likely outcomes are typically estimated to be less probable than they really are. And, outcomes that are quite unlikely are typically estimated to be more probable than they are. Furthermore, people often behave as if extremely unlikely, but still possible, outcomes have no chance whatsoever of occurring.
Figure 2: People systematically over- or under-estimate probabilities.
11
Overconfidence Overconfidence is another powerful bias. We believe we are better at making estimates than we are. I've often repeated a well-known demonstration to illustrate what I call the "2/50 rule." People are asked to provide confidence intervals within which they are "98% sure" that various uncertain quantities lie. The quantities for the questions are selected from an Almanac, for example, "What's the elevation of the highest mountain in Texas?" Give me low and high values within which you are 98% sure that the actual value falls. When the true value is checked, up to 50% of the time it falls outside of the specified confidence intervals. If people were not overconfident, values outside their 98% confidence ranges would occur only 2% of the time. Overconfidence is also demonstrated by the many examples of people expressing confidence about things that are subsequently proved wrong. For example, British Mathematician Lord Kelvin said, Heavier-than-air flying machines are impossible. Thomas Watson, founding Chairman of IBM, reportedly said, "I think there is a world market for about five computers." The Titanic was the ship that couldn't sink. Likewise, surveys show that most drivers report that they are better than average, and most companies believe their brands to be of "above-average" value. Anchoring A related bias is anchoring. Initial impressions become reference points that anchor subsequent thoughts and judgments. For example, if a salesperson attempts to forecast next year sales by looking at sales in the previous year, the old numbers become anchors, which the salesperson then adjusts based on other factors. The adjustment is usually insufficient. Anchors can be set through any mechanism that creates a reference point. For example, in one study, groups of consumers were shown credit card bills that either did or did not contain minimum payment requirements and asked how they would pay the debt off given their real-life finances. The payments for those who indicated they would pay over time were 70% lower for the group who saw information on minimum payments compared to those who did not. Apparently, the minimum payment works as an anchor, causing the card holder to pay a smaller amount than would have been paid in the absence of the anchor.
Titanic The ship that couldnt sink
Minimum payment anchor
Recent events are easy to recall and often become anchors. Thus, investors tend to believe that whats happening currently to the price of a stock will continue to happen into the future (thus, anchoring contributes to stock price volatility since it prolongs up- and downswings) Knowing that recent job performance has a more pronounced affect on impressions, workers naturally give more attention to performance in the 3 months just prior to reviews than in the previous nine months. Dramatic events can become strong anchors. For example, the vividness of the horrible events of September 11 caused many to view airline travel as too risky, but many experts believe that air travel has never been safer.
12
Motivational Biases Motivational biases can affect estimates and forecasts whenever estimators believe that the quantities expressed may affect them personally. For example, managers may have an incentive to overstate productivity forecasts to reduce the risk that the capital dollars allocated to their business units will be reduced. More subtle biases also affect estimates provided from managers, and the effect can depend on the individual. For example, project managers who are anxious to be perceived as successful may pad cost and schedule estimates to reduce the likelihood that they fail to achieve expectations. On the other hand, project managers who want to be regarded (consciously or unconsciously) as high-performers may underestimate the required work and set unrealistic goals. Most managers are overly optimistic. When companies collect data on the financial returns from projects, they almost always find that actual returns are well-below forecasted returns. Motivational biases can also cause experts to minimize the uncertainty associated with the estimates that they provide. They may feel that someone in their position is expected to know, with high certainty, what is likely to happen within the domain of their expertise. Likewise, I have found that managers sometimes become defensive when asked to estimate the potential losses associated with a proposed project, even in environments where it is well-known that projects can fail. They may feel that admitting to downside potential would suggest deficient risk management practices or the fallibility of their project management controls. Poorly structured incentives, obviously, can distort decisions as well as estimates. For example, any company that rewards good outcomes rather than good decisions motivates a project manager to escalate commitments to failing project, since the slim chance of turning the project around is better from the managers perspective than the certainty of project failure. Base-Rate Bias Base-rate bias refers to the tendency people have to ignore relevant statistic data when estimating likelihoods. For example, most people believe they are more likely to die from a terrorist attack than from colon cancer, even though statistics indicate otherwise. Variations of this bias are important in business environments, including the tendency people have to be insufficiently conservative (or "regressive") when making predictions based on events that are partially random. Investors, for example, often expect a company that has just experienced record profits to earn as much or more the next year, even if there have been no changes in products or other elements of the business that would explain the recent, better-than-anticipated performance. The tendency to underestimate the effort needed to complete a complex task has been attributed to baserate bias. Instead of basing estimates mostly on the amount of time it has taken to do previous similar projects, managers typically take an "internal view" of the current project, thinking only about the tasks and scenarios leading to successful completion. This almost always leads to overly optimistic forecasts. One manager I know says he always multiplies the time his programmers say will be required to complete new software "by a factor of two," because "that's what usually happens."
13
Small Sample and Conjunctive Event Bias Small sample bias is another example of inaccurate statistical reasoningpeople draw conclusions from a small sample of observations despite the fact that random variations mean that such samples have little real predictive power. Conjunctive events bias refers to the tendency for events that occur in conjunction with one another to make the result appear more likely. For example, the possibility that you may die during a vacation (due to any cause) must be more likely than the possibility that you will die on vacation as a result of a terrorist attack. Yet, one study showed that people are willing to pay more for an insurance policy that awards benefits in the event of death due to terrorism than one that awards benefits based on death due to any cause. Lack of Feedback Forecasting errors are often attributed to the fact that most people don't get much opportunity to learn through feedback about the accuracy of their forecasts. We are all fairly good at estimating physical characteristics like volume, distance, and weight because we frequently make such estimates and get quick feedback about our accuracy. We are less experienced (and get less verification) when making forecasts for things that are more uncertain. Weather forecasters and bookmakers have opportunities and incentives to maintain records of their judgments and see when they have been inaccurate. Studies show that they do well in estimating the accuracy of their predictions. Advice for improving forecasts and estimates includes: Think about the problem on your own before consulting others and getting anchored to their biases. Be open-minded and receptive. Seek opinions from multiple and diverse sources. Tell them as little as possible about your own ideas beforehand. Tell people you want "realistic" estimates. Ask about implicit assumptions. Encourage the estimation of a range of possibilities instead of just a point estimate. Ask for low and high values first (rather than for a middle or best-guess value) so as to create extreme-valued anchors that counteract the tendency toward overconfidence around a middle value. Require project proponents to identify reasons why what they propose might fail. Give people who provide you with estimates knowledge of actual results as quickly as possible. Use network diagrams, scenario building, and similar techniques to identify and define the sequencing of component activities. A major value of such techniques is that they reduce the likelihood that necessary project activities will be overlooked (e.g., procurement and training), with the result that time and costs are underestimated. Routinely use logic to check estimates. As a simple example, if you have 2 months to complete a project estimated to require 2000 hours, verify that you have a sufficient number of FTE's available.
Trip insurance pricing illustrates conjunctive bias
14
Bounded Awareness and Decision-Making Heuristics

In addition to the biases that degrade judgments, research shows that problems arise from the processes that people use to make decisions. Problems include bounded awareness and the use of flawed decision-making heuristics. Bounded Awareness Bounded awareness, described by Max Bazerman and Dolly Chugh [4], refers to the well-documented observation that people routinely overlook important information during the decision-making process. One cause is our tendency to become overly focused. Focus limits awareness, and important information outside the range of focus can be missed. Thus, an airplane pilot attending to his controls can overlook the presence of another airplane on the runway. Cell phones can distract drivers and contribute to car accidents. George Day and Paul Schoemaker [5] observe that companies are often slow to recognize important developments at the periphery of the business that turn out to be strategically important. Examples include the music industrys failure to foresee the threat of Napster-type services, Polaroids bankruptcy resulting from the rapid rise of digital photography, and Wal-Marts surprise at social concerns Unforeseen resistance leading to community resistance to new stores. Studies show that unexpected information is particularly easy to miss. In one experiment, participants were shown a video of an exciting basketball game. There is a 5-second interval during which a person in a gorilla costume walks through the game thumping his chest. Few people recall seeing the gorilla. Changes that occur slowly are often not recognized until it is too late. This may explain, in part, business scandals. At Enron, for example, accounting irregularities were adopted slowly. Like the boiling frog, executives may have been lulled into a sense of comfort, unaware that they were in hot water. The Challenger space shuttle disaster provides an example of a situation wherein important information existed, but was not sought out. On the day of the launch, decision makers argued whether the low temperature would be a problem for the shuttles O-rings. Seven prior launches with some sort of O-ring failure were examined, but no pattern between failures and temperature emerged. After the fact, data were analyzed for all 24 previous launches. The expanded data indicated that the Challenger had more than a 99% chance of malfunctioning [6]. Sometimes information is known, but not acted upon. For example, the drug company Merck did not withdraw its pain relief drug Vioxx (Figure 4) from the market until 2004. Yet, evidence of the drugs risks was reported in a medical journal as early as 2000. The delay in taking action cost Merck dearlyover $200 million has so-far been awarded in claims. Why didnt Merck withdraw the drug sooner? The warning was available, but it appears to have been ignored.
Was data ignored?
15
Group dynamics hinder information sharing. Teams may discuss information, but the discussion usually conveys information that is widely known, not information that is uniquely known to a single team member. Psychologists say that this is because sharing something that others already know produces positive reinforcement in the form of the agreement of others. However, introducing into the discussion something that only you know typically doesnt generate positive feedbackPeople just sit there, and it is unclear what happens next. Organizational structures can institutionalize bounded awareness. Organizational silos and multiple layers of management hinder information flow. Information doesnt get transmitted well across silos, and management layers can filter out messages management doesnt want to hear. Advice: Stay alert to peripheral threats and opportunities. For example, watch for regulatory, technological, and market-oriented changes and trends and be prepared to modify your strategies. Instead of using the information that happens to be in the room at the time, identify what information is needed to make the decision and then get it. In meetings, assume everyone has unique information and ask for it. Meeting agendas should request individual reporting. Get outsiders perspectives. Outsiders may help you to see critical information that you could easily overlook when immersed in day-to-day activities. Break down organizational silos and other barriers to information flow. Decision-Making Heuristics Just as people use heuristics to make judgments, executives often rely on simplified reasoning to make difficult decisions. According to decision theorist Herbert Simon, complexity, limited time, and inadequate mental computational power reduce decision makers to a state of bounded rationality. Decision-making shortcuts can save time and ease the psychological burden of decision making. However, like other judgmental biases, there are circumstances where the use of such heuristics may result in bad choices. Reasoning by Analogy Reasoning by analogy means choosing a course of action based on an approach found effective for an analogous situation. For example, the business model for Toys R Us was supposedly developed based on an analogy with supermarkets. Similarly, Intel reports that it moved into low-end chips to avoid U.S. Steels mistake in not pursuing the market in low-end concrete reinforcing bars. Dangers arise when analogies are built on surface similarities only. For example, numerous companies have unsuccessfully attempted to copy Dell Computer's build-to-order business model. Steep declines in microprocessor prices is what gives Dell the advantage (Dell saves money due to the very short
Copying Dell
16
time between when it needs to purchase a processor and when the computer is sold). Thus, the analogy fails for industries that arent experiencing rapidly declining supply costs. Rules of Thumb Business rules of thumb include the processing power of computer chips doubles every 18 months, a new business needs to have 4 consecutive quarters of profits before launching a public offering, and a public health risk is acceptable if the probability of fatality is less than one-chance-in-a-million. The problem with rules-of-thumb is that they ignore specifics and can become outdated with changing circumstances. For example, if you used a rule of thumb that a fast-food restaurant is worth 2 to 3 times its annual revenues, you might end up paying too much for a restaurant with declining sales, old equipment, and health code violations. Mortgage lenders who stuck with the old rule of thumb that borrowers can spend no more than 25% of income on mortgage payments lost out in areas of rapid price appreciation to more aggressive lenders. Seeking Means Rather Than Ends If it is difficult to determine the choice that best achieves the end goal, choosing based on some presumed means to the end can seem like a reasonable approach. Means-based decision making is especially likely if the end goal is difficult to measure. For example, organizations often fund learning initiatives based on the goal of improving the productivity of their workers. Because it is difficult to estimate the impact of a training course on peoples productivity, many corporate learning departments simply report the number of hours employees spend in the classroom. A program that manages to get 80% of employees to spend 40 hours in class isnt necessarily providing the best learning for the company. Seeking means rather than ends leads to decision errors when the selected means is only a characteristic of the desired end, not the cause of it. With regard to choosing projects, the end goal is to produce the greatest possible value for the organization, subject to the constraint on available resources. Because value is difficult to measure (see Part 3), a popular heuristic is to prioritize projects based on strategic alignmentprojects are ranked based on how well they fit elements of stated corporate strategy. Translating strategy to action is important, however, being well-linked to an element of strategy doesnt guarantee that a project will be successful or that it will generate the most value for the organization. Incrementalism Incrementalism is an approach to decision making wherein a large change is made slowly though a series of small adjustments. The idea is to be cautious, to experiment and learn. For example, an organization might avoid a high-risk, big investment decision by making a series of small investments. The first step might be to conduct a low-cost, low-risk pilot project. If the pilot appears successful, a bolder move might then be taken. Or, an expanded pilot might come next. Home Depot, for example, reportedly uses this approach. The companys policy is to test proposed changes to retail strategy in individual stores before making large-scale changes. A major advantage of incrementalism is that it preserves flexibility while providing useful information. However, there are two potential problems. One is that experimenting takes time, and the window of opportunity may close before the organization is prepared to fully act. Another problem relates to sunk
17
cost bias. It may be hard to abandon a plan as costs accumulate because decision makers dont want to admit that earlier investments were wasted. Over-Reliance on Experts Because decisions often involve complex technical considerations, experts frequently play a necessary role in decision making. Experts have credibility and command respect from others, both inside and outside the organization. There can be problems, however, with relying on experts. Experts can easily dominate discussion and discourage constructive dissent. Over-reliance on experts can create bad choices. Experts may be biased towards solutions that rely more heavily on technology, and they may not fully understand important non-technical considerations that impact the decision. The Bush Administration, for example, has been criticized for relying too much on experts in the Central Intelligence Agency in its decision to take down the government of Iraq. Due to the risks associated with using decision-making heuristics, you should: Be careful using solutions based on analogies. Look for differences in the respective situations and ask how those differences might cause the analogy to fail. Dont make a decision based on a rule-of-thumb without thinking carefully about whether the rule applies in the current situation. Have things evolved such that rule of thumb no longer makes sense? Think about whether your decision logic is confusing means with ends. Will what you intend to do guarantee that you get what you want? Or, are you merely choosing something that is only associated with (does not cause) the desired result? Watch out for the natural attractiveness of incremental solutions and the subsequent power of sunk cost bias. Is the incremental approach popular solely due to reluctance to upset the status quo? Is what will be learned that valuable, compared to the incremental costs including the costs of delay? Be sure to get the judgments of experts, but dont allow them to dominate the decision-making process. Institute controls that ensure the objectivity of experts and allow others play a role and share responsibility for choices. Consider steering clear of the heuristic methods in favor of formal decision analysis when the stakes are high and the probability of success must be maximized. Decision Errors Figure 3 shows the 3 essential steps to benefiting from decisions: (1) recognize the opportunity to choose, (2) make a good choice, and (3) implement.
18
Figure 3: Essential steps for benefiting from decisions.
Missing the opportunity to make a decision means that the "do nothing" or status quo option is chosen by default. Warren Buffet has reportedly said that he considers his worst investment failures to be missed opportunities to take action, "The things I could have done but didn't do have cost us billions of dollars" [7]. Taking advantage of an opportunity to decide doesn't, of course, ensure success. There are many reasons why a conscious business choice may be poor, including misinterpreting the problem to be solved (poor framing), failing to identify good alternatives, not being clear about objectives, and using heuristics or unaided intuition rather than reasoned logic. Finally, there is nothing to be gained from making a good choice if it is not implemented. A study by an Ohio State University researcher found that about one-third of nearly 400 business decisions studied were never implemented by the organizations involved. [8] It is often assumed that a bad decision will be evident to us after the fact. However, this is not necessarily the case. In one experiment, researchers used a card trick. They showed participants pairs of pictures of female faces and asked them to choose the face they thought more attractive. Later, they switched some of the cards and asked the participants to explain their choices. In the majority of cases where the cards were switched, participants failed to recognize the switchthey simply justified the choices they thought they had made. [9]. Be alert to the opportunity or need to make decisions. Consciously frame the decision to be made. Be clear about decision objectives. Identify distinct and creative alternatives. Use analysis and logic to make a good choice. Follow through and implement what you decide.
19
Debiasing
The problem of bias is of critical importance, given that judgment pervades human experience and is crucial to decision making: Should I accept this job? Should we develop a new product? For whom should I vote? Is the defendant guilty? Decision-making errors, obviously, can be extremely costly at the personal, professional, and societal levels. Not surprisingly, there has been a fair amount of effort invested in looking for ways to reduce bias. Unfortunately, there does not appear to be an easy fix. In 1982, decision scientist Baruch Fischhoff reviewed 4 strategies for reducing bias: (1) warning subjects about the potential for bias, (2) describing the likely direction of bias, (3) illustrating biases to the subject, and (4) providing extended training, feedback, coaching, and other interventions. Fischhoff concluded that the first 3 strategies yielded limited success, and even intensive, personalized feedback and training produced only moderate improvements in decision making. [10]. In the succeeding 25 years since Fischhoffs study, much additional research has been conducted, but the basic conclusion remains the same. One continuing line of research involves investigating whether biases can be reduced by encouraging subjects to put more effort into forming judgments. Asking students to show their work, for example, has been shown to slightly increase the chances of obtaining a correct answer (it is more helpful for pinpointing where lack of understanding may occur). In general, however, the limited success of such techniques suggests that most biases are not very sensitive to the amount of effort one applies. Encouraging people to take an outsiders perspective has been shown to somewhat reduce the tendency for overconfidence to bias estimates (What do you think someone not directly involved would say?). The idea is to reduce personal biases by removing oneself mentally from the specific situation. Some studies show that the technique can improve estimates of the time it would take to complete a task and the odds of success [11, 12]. Increasing accountability for decisions has been shown to lead to better choices. [13]. Likewise, enhancing accountability for opinions that people express can help in some circumstances. For example, it has been suggested that, when obtaining critical information from someone, it may be useful to take notes (or to appear to take notes). If people believe you may quote them to others, they may be more careful in what they say. Similarly, to support project-selection decisions, it is useful to have project proponents document precisely why they believe their Note taking may encourage more proposed projects should be conducted. Going on record encourages managers to thoughtful responses be more careful in their logic, and the fear of being proved wrong helps counter over-optimism. Training (in biases) has been shown to help people address some biases. However, the effect is generally short lived and does not produce an overwhelming improvement in performance. One problem is that it is often hard to get people to appreciate that bias is something that affects them personally, not just others.
20
Thus, in situations where time permits, it helps to demonstrate biases. For example, if you are obtaining judgments from a group of individuals and are concerned about overconfidence bias, dont just tell them about the 2/50 rule (described above). Instead, run them through an exercise that demonstrates that the rule applies to them. Not surprisingly, improving cause-effect understanding of the relevant systems and processes has been shown to improve the quality of estimates and decisions. For example, studies show that when people are encouraged to look for common principles underlying seemingly unrelated tasks they are able to better create solutions for different tasks that rely on the same underlying principles [14]. There is evidence that groups reach better decisions when alternatives are evaluated simultaneously as opposed to having each alternative evaluated sequentially and potentially rejected. The presumed explanation is that people initially react emotionally when considering an alternative; they think mostly about how it will affect them personally. If alternatives are evaluated simultaneously side-by-side, group members are less susceptible to this reaction [15]. Strategies for Reducing Biases The usual strategy for reducing a bias is to address the mental processing error that is believed to produce the bias. For example, in one study, researchers assumed that hindsight bias, the tendency to exaggerate the extent to which one could have anticipated a particular outcome, results from the difficulty people have in appreciating the limited information available at the time and the distinct inferences that could be made from that information. By providing evidence that argued against the actual outcome, they found that their subjects could be made more resistant to the bias [16]. Similarly, it has been hypothesized that peoples tendency to over-claim credit for a group accomplishment is due in part to the tendency to be more aware of ones own efforts. Researchers showed that when people are asked to estimate not only their own contributions but also those of others, they attribute less credit to themselves. [17]. Figure 4 (derived from Wilson and Brekke [18]) illustrates a view for how judgmental biases are created and suggests a general strategy for reducing biases. According to the figure, the first step is to create awareness of the flawed mental processes involved. The subject must be motivated to correct the bias, and understand the direction and magnitude of the errors produced. Finally, the bias must be removed or countered. The technique used to mitigate the bias of concern is often the application of a countering bias, for example, countering overconfidence by encouraging subjects to describe (and therefore anchor on) extreme possibilities. Many of the recommendations provided earlier in this section under Advise are based on this logic.
21
Figure 4: General strategy for debiasing.
Decision Aids Hundreds of decision aids have been developed or recommended to reduce the distortions in decisions caused by errors and biases. I developed the list of sample aids below in the context of a chapter I prepared for a book on aids for environmental decisions [19]. (Note that many entries in the table represent a category or type of aid rather than a specific aid.). As indicated, there are at least 5 categories of aids: (1) checklists for promoting a quality decision process, (2) thinking aids intended mainly to improve perspective or create insights, (3) models and optimization methods for recommending choices, (4) aids for promoting group consensus, and (5) voting methods. As an example of the first category, Figure 5 is a checklist aid for scoring the decision-making process relative to the components of a quality decision-making process.
22
Sample Decision Aids

Checklists Thinking aids DeBonos six hats SWOT analysis Theory of constraints Models and analysis Comparison methods PMI (listing pluses, minuses and interesting) Paired comparison methods Weighted comparison methods Analytic hierarchy process ELECTRA Force field analysis Grid analysis Goal programming Kepner-Tregoe methods Multi-attribute utility analysis PROMETHEE SMART (simple multi-attribute rating technique Cost-benefit analysis Decision analysis Social choice theory Expert systems Optimization methods Linear programming Integer programming Quadratic programming Dynamic programming Probabilistic optimization methods Decision trees Stochastic programming Fuzzy logic Game theory methods Zero-sum games N-person games Cooperative games Non-cooperative games Group facilitation methods Delphi technique Nominal group method Voting methods Approval voting Borda count Copelands method Plurality voting Plurality with run-off elections
Figure 5: Checklist diagram for evaluating deficiencies in the decision-making process [20].
23
Models and analysis, in my opinion, represent the most effective way to address errors and biases in decision making. Essentially, the concept is to replace flawed intuitive reasoning with a formal, deliberative process. Much evidence has accumulated indicating the effectiveness of models and analysis. For example, in situations where past data is available on the inputs and results of a decision making process, simple models can be created using regression analysis. Such models are being used to help graduate schools to decide which students to admit, clinical psychologists to diagnose neuroses and psychoses, and credit card companies to decide whether to accept loan applications. For a very wide range of applications, researchers have found that such models produce better and more reliable decisions than those made by people, including the experts who made the original decisions from which the models were derived [21]. Another argument for using models comes from research that shows that experts appear to be less subject to biases when addressing issues that are within their areas of expertise. Models break the problem of evaluating alternatives down into individual pieces such that different experts with specialized knowledge can be selected to focus on each piece. Thus, models dissect a complex problem in a way that makes the required judgments less likely to be biased. A Strategy for Decision Making Approaches to making good decisions differ greatly in terms of the amount of time and effort required. Intuitive decisions can be fast, automatic and effortless, while analysis is slower and requires considerably more effort. Figure 6 illustrates that the appropriate approach to decision making depends on the significance of the decision, the challenges involved, and the time available for analysis. It is useful to develop a quiver of decision making strategies, and to select the approach that makes most sense for the given circumstance.
Figure 6: The appropriate decision aid and decision-making approach depends on the nature of the decision.
24
The best protection from bias comes from training, using formal techniques for obtaining important judgments, utilizing well-founded decision models and analysis, and instituting rigorous decision processes that document the judgments and assumptions upon which choices are based. The subsequent parts of this paper describe such a process specific to decision-making for project portfolio management. As stated by Ken Keyes, "To be successful in the business world we need to check our bright ideas against the territory. Our enthusiasm must be restrained long enough for us to analyze our ideas critically" [22].
25
References for Part 1

1. Tversky and D. Kahneman, Judgment Under Uncertainty: Heuristics and Biases, Cambridge Press, 1987. 2. As indicated in the text, much of what appears in this section of the paper, and in similar summaries published elsewhere, comes from the The Hidden Traps in Decision Making, by J. Hammond, R. L. Keeney, and H. Raiffa, Harvard Business Review, pp 47-55, Sept-Oct 1998. The paper has been republished several times, most recently in the Jan 1, 2006 issue of HBR 3. Russo, J.E., and P. J.H. Shoemaker, Winning Decisions: Getting it Right the First Time, Doubleday, 2001. 4. Bazerman, M and D. Chugh, Decisions without Blinders, Harvard Business Review, Jan 2006, pp 88-97. 5. Day, G. S. and Schoemaker, P. J. H., "Scanning the Periphery" Harvard Business Review, Nov. 2005, pp. 135-148. 6. Bazerman, M and D. Chugh, Bounded Awareness: What You Fail to See Can Hurt You, HBS Working Paper #05-037, Revised 8/25/2005. 7. The quote appears on www.numeraire.com, Global Value Investing, which cites a speech given by Warren Buffet in Boston in 1997. 8. P.C, Nutt, Why Decisions Fail, Berrett-Koehler, 2002. 9. P. Johansson, L. Hall, S. Sikstrom, and A. Olsson, Failure to Detect Mismatches between Intention and Outcome in a Simple Decision Task, Science, Vol 310, pp 116-119, 2005. 10. Reported in M. H. Bazerman and D. Moore, Judgment in Managerial Decision Making (7th ed.), Hoboken, N.J., Wiley, 2008. 11. D. Kahneman and D. Lovallo, Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk and Risk Taking, Management Science, 39, 17-31, 1993. 12. A. C, Cooper, C. Y. Woo, and W.C. Dunkelberg, Entrepeneurs Perceived Chances for Success, Journal of Business Venturing, 3(2), 97-109, 1988. 13. R. P. Larrick, Debiasing, in D. J. Koehler and N. Harvey (eds.) Blackwell Handbook of Judgment and Decision Making, Oxford, England, Blackwell Publishers, 2004. 14. S. Moran , Y. Bereby-Meyer and M. Bazerman, Stretching the Effectiveness of Analogical Training in Negotiations: Learning Core Principles for Creating Value, Negotiation & Conflict Management Research, 1(2), 99-134, 2008.
26
15. M. H. Bazerman, S. B, White and G. F. Loewenstein, Perceptions of Fairness in Interpersonal and Individual Choice Situations, Current Directions in Psychological Science, 4, 39-43, 1995. 16. P. Slovic and B. Fischhoff, On the Psychology of Experimental Surprises, Journal of Experimental Psychology: Human Perception and Performance, 3, 544-551:23, 31, 1977. 17. K. Savitsky, L. Van Boven, N. Epley, and W. Wright, The Unpacking Effect in Responsibility Allocations for Group Tasks, Journal of Experimental Social Psychology, 41, 447-457, 2005. 18. T.D. Wilson & N. Brekke (1984). Mental Contamination and Mental Correction: Unwanted Influences on Judgments and Evaluations. Psychological Bulletin, 116, 117-142, July 1994. 19. Merkhofer, M. W., Assessment, Refinement, and Narrowing of Options, Chapter 8, Tools to Aid Environmental Decision Making, V. H. Dale and M. R. English, eds., Springer, New York, 1998. 20. This figure is similar to the approach and corresponding figure in R. Howard, The Foundations of Decision Analysis Revisited, Chapter 3 in Advances in Decision Analysis, W. Edwards, R. Miles, and D. von Winterfeldt, eds., Cambridge University Press, 1999. 21. R. J. Jagacinski and J Flach, Control Theory for Humans: Quantitative Approaches to Modeling Performance, CRC Press, 318-319, 2003. 22. K. Keyes, Jr., Taming your Mind, Love Line Books, 1991.
27
Choosing the Wrong Portfolio of Projects Part 2: Failure to See the Forest for the Trees
Failure to See the Forest for the Trees

Most organizations put ample effort into making individual projects successful, but not enough effort into making the entire portfolio of conducted projects as successful as it could be. The fault does not lie with project managers. Most project managers are highly motivated. If their projects are funded, they do their best to deliver on time and within budget. However, just because an organization has a portfolio of mostly on-time and on-budget projects does not mean that it has the best possible project portfolio, or that it is effectively allocating resources among the projects that it is doing. The problem is that project managers are focused on their individual projects, not on the success of all projects. Few project managers, for example, suggest terminating their projects if things are going badly. Likewise, the typical project manager does not look for opportunities to redirect his or her resources toward the projects of other project managers who may need them more. Senior executives, unlike project managers, do not judge success on a project-by-project basis. What does (or should) matter to them is the aggregate cost, value, and risk of the overall project portfolio. Likewise, the chief executive's concerns are mainly about achieving the objectives of investors. Whether or not individual projects succeed is, generally, of secondary importance. Although it should be obvious that it is the performance of the project portfolio as a whole that really matters, many organizations do not manage the aggregate cost, value and risk of their total project portfolio. Either no one has responsibility for managing the project portfolio, or the efforts to do so are not as effective as they could be. Project managers tend to the individual "trees," but no one is caring for the "forest." Failure to see the forest for the trees is the second reason organizations choose the wrong projects.
28
Problems with Project-by-Project Decision Making

Failure to manage at the level of the project portfolio creates several closely-related problems, including too many projects, poor project choices, inconsistent and inappropriate priorities, and misallocated resources. The Downward Spiral that Produces Poor Project Portfolios If project selection decisions are not made at the portfolio level, by default the project portfolio is the end result of numerous individual project choices made one at a time with little regard for the impact that one choice has on the next. The typical result, illustrated in Figure 7, is a dysfunctional feedback loop, or downward spiral, that produces too many projects, inadequate project information, inability to kill failing projects, and too many low-value projects.
Figure 7: The downward spiral resulting in poor project portfolios.
Too Many Projects The reasons that organizations generally undertake too many projects have to do with the way projects are generated and the way that project go/no-go decisions are made. In many organizations, projects are initiated by functional executives and others without regard to whether resources are available to do the work. Because there is little or no collaboration or coordination among project requestors, many more projects are proposed than there are resources to conduct them. Then, regardless of whether tools are used to formally evaluate project proposals, the basis for a "go" in project-by-project decision-making is necessarily whether or not the proposed project is judged to achieve some hurdle or threshold of acceptability. Projects look good, especially in their early stages, so many projects pass the hurdle.
29
Later, when people are assigned to projects, it becomes apparent that they are committed for more than 100% of their time. A typical hurdle for large projects imposed by many organizations is calculation of financial net present value (NPV). If the NPV is positive, the acceptable hurdle is achieved and the project is deemed a go. For most projects, it is easy for project proponents to make assumptions that ensure that the project NPV will be positive. Thus, few projects get screened out based on financial criteria. Poor Project Performance Resource managers feel they have no choice but to undertake approved projects, so resources are multitasked in the attempt to satisfy all. Too many projects means that resources are spread too thinly. . Key resources become overloaded and tasks are delayed. Stress levels go up, morale suffers, and the team concept starts to break down. Quality of execution drops and project schedules slip. More time is spent reviewing project budgets and cost aspects of projects than on shortening schedule. Inadequate Project Information Surveys routinely report that management complains about insufficient information for decisionmaking. For example, R. G. Cooper reports that a study of product development projects at 300 firms "revealed major weaknesses in the front-end of projects: weak preliminary market assessments, barely adequate technical assessments, dismal market studies and marketing inputs, and deficient business analyses" [1]. The information that is available is typically scattered across different departments and business units, making it difficult to execute informed decisions about where to invest scarce resources, how to prioritize initiatives and balance project demands. Doing project work competes with time spent developing project data, so too many projects means that insufficient time is devoted to providing better quality data. Failure to Kill Projects Poor information gives management an insufficient basis for making tough decisions. No one wants to be the one to kill a questionable project, "It's like drowning puppies!" As a result, failing projects don't get terminated soon enough. Failure to kill failing projects compounds the problem. Nothing sucks up resources and exhausts project teams more than trying to save a failing project. Resources are spread even more thinly. Bias Toward Small, Low-Risk, Short-Duration Projects Not only does inadequate information lead to an inability to kill projects, it also tends to create a bias toward small, low-value, low-risk, short-duration projects (e.g., extensions, modification, and updates). High-value/high-risk projects aren't viewed as feasible (given the constraint on resources). Even if bigger and riskier projects are proposed, management usually isn't prepared to take the risk due to the lack of quality information. When a large project does get started, available resources get sucked into the big one, often leaving other projects high and dry.
30
Other Symptoms of the Downward Spiral Not sure if your organization is suffering from the downward cycle? Other symptoms include: No standardized way to document or compare projects. Politics plays a bigger role in project selection than business value. Excessive project delays due to not enough resources. Frequent changes in project status, from active to on hold to top priority and back again. Competition, rather than cooperation, among departments and business units when staffing and funding projects. No clear executive-level view of the project portfolio and its business impacts. No link between the portfolio and executive-level concerns, such as share price. Frequent difficulty finding enough of the right people to adequately staff a project. Completion of projects that, after the fact, dont seem to have contributed much to the business. Inconsistent and Inappropriate Priorities A related problem is that the failure to establish organizational priorities causes inefficiencies in the dayto-day allocation of peoples time. Consider the following example from an article by the Product Development Institute, Inc. [2]: Imagine that you are a member of a project team in a typical organization. Like most project team members, you are working on several projects simultaneously. You are under pressure from all your project managers to make faster progress. Suppose you finish a milestone on an important project ahead of schedule. Do you use the opportunity to get an early start on your next task on this project? More likely, you turn your attention to one of your other projects. Your day-to-day priorities are based on your desire to minimize the pressures you feel from your various project managers. Although the organization may benefit more from your taking the opportunity to put your important project ahead of schedule, that project is unlikely to obtain the benefits of your early finish. The point being made by the authors is that work is always prioritized. It is only a question of who does it and how. In the absence of established priorities, people use their own prioritization methods. One common prioritization method is first in first out (FIFO). "Whatever task shows up in my in-box first, I do first." Another common prioritization method is the "squeaky wheel." Whoever complains to me the loudest gets the work done first. Others include doing first the work for the manager that you like the best, has the most interesting projects, or writes your performance reviews. None of these methods is likely to result consistently in the best allocation of resources for the organization. Prioritization is a leadership task. Unless management makes the effort to prioritize projects, the allocation of project resources will be left to individual project managers, and the results will likely not be in the best interests of the organization.
31
Misallocated Resources Too often, the approach to allocating a limited budget is something like this. Individual business units identify projects, compute their funding needs, and submit budget requests. When the total funding requirement is computed, executives decide that spending must be reduced. Rather than consider carefully how the shortfall should be distributed across organizational units, the decision is made to cut each unit's request by the same percentage. Taking the same cut from each unit's project portfolio is regarded to be "fair." "Fair Share Allocations" Arent Fair The problem with this approach is that all organizational units do not have the same ability to absorb cuts in spending. A 10% reduction in project funding may result in more than or less than a 10% reduction in the value delivered from the project. In some extreme cases, such as that illustrated in Figure 8, eliminating a fraction of funding can eliminate all of the value of the project!
Figure 8: Cutting the budget may eliminate all of the value of a project portfolio.
32
Combine Projects into Project Portfolios

Solving the problems created by project-by-project decision-making requires shifting focus from the project to the project portfolio. The enabling step is to collect information about individual projects into a common database. With project portfolio management tools, the database is electronic. This ensures that the information is readily accessible and facilitates keeping the data up-to-date. In addition, tabular and graphical displays can be provided that make it easy to aggregate project data and to create summary reports (Figure 9).
Figure 9: Combine project data into a single database.
The project portfolio provides a big-picture view. It enables managers to become aware of all of the individual projects in the portfolio, and provides a deeper understanding of the collection as a whole. It facilitates sensible sorting, adding, and removing projects from the collection based on real-time project information. Single or Multiple Portfolios A single project inventory can be constructed containing all of the organization's ongoing and proposed projects. Alternatively, multiple project inventories can be created representing project portfolios for different departments, programs, or businesses. Since project portfolio management can be conducted at any level, the choice of one portfolio versus many depends on the size of the organization, its structure, and the nature and interrelationships among the projects that are being conducted. It might not, for example, make sense to force a centralized project portfolio on an organization that practices decentralized decision-making.
33
The key is to group similar projects that use common resources so as to leverage the knowledge and expertise needed for portfolio management. If multiple project portfolios are defined, related projects should be organized into common portfolios. Portfolios of highly interrelated projects are often referred to as programsgroups of projects that must be selected and managed in a coordinated way in order to maximize success Figure 10 illustrates the relationships that may exist among portfolios in a large organization.
Figure 10: Sample relationships among portfolios, programs, and projects.
Portfolio groupings should be organized so as to be as independent of one another as possible. Decisions about what projects to conduct within one portfolio should not depend in a significant way on the projects that are conducted within any other portfolio. The decision of how to allocate resources among the various project portfolios can then be made at a higher level, based on estimates of the how the value of each portfolio depends on the funding that it receives. Project Classification Schemes Since it is useful to be able to monitor and control the mix of various types of projects within the project portfolio, a project classification scheme should be established. Projects can be classified in many different ways. Examples include: size; type/purpose (e.g., maintenance, growth, productivity, innovation); geographic location; skills or technologies required; sponsor, client or market served; asset class addressed (e.g. infrastructure, IT); and stage of the project life cycle (e.g., R&D, commercialization). Multiple schemes can be used so that each project is classified in several different ways. No one approach is best for every organization. The key is to choose classification schemes that will yield information most useful for decision making. Knowing the various categories to which a project belongs helps to characterize that project and enables the construction of charts indicating how spending is distributed (see Figure 11 for an example).
34
Figure 11: Sample chart for investment mix based on project classification.
Project Template The information entered into a portfolio database should, at minimum, include the project name, type, and a brief description; internal and external requirements; number and skills of people required; estimated time to completion; and estimated cost. Importantly, the recorded information must also include some level of business justification and value assessment. What, exactly, is the need that is being addressed? What benefits are expected from doing the project? When will these benefits begin to accrue? Also, risks associated with successfully completing the project or securing the benefits should be identified. Finally, in situations where change is rapid, the time urgency of the project should be indicated. If the project is delayed, what will the consequences be? If project information is standardized, a template can be provided for submitting project proposals. The template may be a paper form, or an electronic from. Using a standardized project information template encourages complete proposals and more consistent proposal evaluations. Benefits of the Project Portfolio Once opportunities have been identified, with relevant information normalized and made easily accessible, individuals throughout the organization with broad understanding of the business can provide reality checks. Summary measures conveying data related to cost, risk, and benefit can be used to create graphics and comparative analyses that allow decision-making teams to collaborate on project-selection decisions. Organizations invariably report that creating their first inventory of ongoing and proposed projects is revelational. "I didn't know we had so many things going on, no wonder we can't get anything done!" Counting projects produces instant value: If you schedule 130% of your human resources to projects,
35
for example, you can be assured that some things wont be done. Reducing the number of projects eases the strain on common resources, giving remaining projects the resources they need and eliminating time spent by managers in negotiations over the people and other resources. The initial project inventory often uncovers significant duplications and mismatches. For example, CIO Magazine [3] reports that when Schlumberger first grouped IT projects, they found that 80% overlapped. Duplicate efforts should be eliminated, obviously, and similar projects combined into a single project. Schlumberger reportedly saved $3 million just by eliminating project redundancies. Looking at projects from the perspective of the portfolio makes projects look less like discrete efforts and more like an interconnected suite. Information and understanding is improved. Interdependencies among projects can be noted. New requirements can be evaluated against current commitments. Portfolio analysis allows investigating questions like, How are resources for Project A impacted if completion of Project B is delayed? The portfolio extends the focus beyond project management and highlights objectives and outcomes. Recognizing project portfolio management as an ongoing activity creates a shift away from typical oneoff, ad hoc approaches to project management. The portfolio establishes a philosophy and culture that enables visibility, standardization, and measurement as a means for process improvements. Figure 12 summarizes these and other benefits (discussed elsewhere in this paper) typically observed from implementing project portfolios.
Figure 12: Some benefits of the project portfolio.
36
Establish a Portfolio Management Office

The function of the project portfolio management office is to manage the organization's project portfolio, which includes prioritizing projects, allocating resources to projects, and tracking the performance of the project portfolio. A key focus is ensuring that the overall collection of projects maximally supports the objectives of the enterprise. In addition, the office collects and distributes data for reviewing assessing, and managing individual projects to ensure that they are meeting their expected contributions to the portfolio. As illustrated in Figure 13, portfolio management provides the necessary link between project management and enterprise management.
Figure 13: Portfolio management links project management to enterprise management.
Participants, Authorities, and Responsibilities The first step for establishing a project portfolio management function is to decide who will participate as active managers of the project portfolio. A project portfolio manager, typically a senior manager, should be appointed with accountability for the success of the entire project portfolio. The table below lists important skills of the portfolio manager.
37
Important skills and capabilities for the portfolio manager

Good understanding of the organizations vision, mission, and strategy. Understanding of project and program management, including ability to assess project health based on high-level reporting documents. Leadership skills, including communication, presentation, and team building. Broad understanding of the business, including markets, customer base, partners, and applicable regulations. Able to interact effectively with senior executives. Good analytic skills, with understanding of the basic concepts and techniques for project valuation and risk assessment. Understanding of process development, change management, and continuous learning.
The portfolio manager should be given an estimate of the total funding to be made available, but it should then be up to the portfolio manager to determine how to allocate the funds within that cost constraint. It should be possible for the portfolio manager to suspend at any time further commitment of investment dollars due to cost increases, failure to make anticipated progress, changing economic climates, or shifts in business conditions. At the very least, the portfolio manager should have responsibility for recommending resource allocations for final approval by a committee of senior executives. In either case, senior executives should be enlisted to serve as a steering committee responsible for providing and updating (e.g., in response to changing business objectives) the value judgments and policy decisions needed to guide portfolio management. A portfolio management team should support the portfolio manager. The team, oftentimes, includes department heads from sub-organizations that generate request for projects, provide project resources, and/or use project deliverables. The team should have responsibility for verifying cost, value, and risk estimates provided in support of project proposals and requests for resources. This team is then responsible for evaluating project proposals, accepting or rejecting proposals, accelerating and decelerating projects, allocating resources, and otherwise managing the project portfolio over time. One member of the team should be designated as the primary contact person for each project manager. Adopt a Successful Portfolio Management Process The successful project portfolio management process includes four major components (Figure 14). First, a structured process is used to acquire key information about all projects and to organize the data into one or more project portfolios. Second, consistent and objective methods are employed to analyze projects and compare their risks and benefits. Third, resource demands are compared with capacity so that the subset of projects that make best use of available resources can be selected. The final component consists of tracking portfolio performance for ongoing assessment and adjustment. The management of these components is the responsibility of the project portfolio management team.
38
Figure 14: Components of project portfolio management.
More detailed responsibilities of the team are illustrated in Figure 15. On a regular basis tied to the planning and budgeting cycle (i.e. annually, biannually, or quarterly), the portfolio team reviews all projects that are seeking funding, including ongoing projects and new project concepts. Projects are screened to determine which proposals require formal evaluation. Projects exempted from formal evaluation include obvious "non-starters," very small projects, and projects that are more appropriately funded from other budgets. Ongoing projects and mandated projects (i.e., projects that the organization is legally required to conduct) may or may not be exempted (see below).
Figure 15: Typical portfolio management cycle.
39
Project managers with projects that pass the initial screening are authorized to use resources to complete additional analysis necessary to provide the data required for entry into project proposal templates. In some instances, the necessary resources can be fairly large. For example, a proposed construction project may require some minimal level of engineering analysis to determine feasibility, set project scope, provide cost estimates and estimate project effectiveness at addressing needs. Alternatively, a feasibility study for a large project might itself be a project that competes for funding within the project prioritization process. Be aware that, in most cases, project portfolio management forces an increase in the effort spent on analyzing project opportunities. The extra effort is justified by the need for better information to support decision making. To be as efficient as possible, screening systems may be used to select a different level of data requirements and analysis for different types or sizes of projects. Mandated and Ongoing Projects Although some organizations exempt mandated and ongoing projects from formal evaluation, my recommendation is that both types of projects be routinely evaluated along with new projects. In some organizations, "mandated" projects consume most of the budget. Very strict criteria should be provided for mandatory projects to ensure that they are defined with minimum possible scope and cost. Add-ons that go beyond what is strictly required should be defined as separate projects. Formally evaluating mandated projects promotes consistency, provides useful information, and helps the organization ensure that all of the benefits of mandated projects are, in fact, achieved. Evaluating ongoing projects for continued funding is useful to ensure that struggling or obsolete projects do not prevent funds from being available to meet more-pressing needs. Where possible, large, longduration initiatives should be reframed as a series of smaller projects. For example, in the case of a software project, it may be better to make incremental funding decisions to develop improved versions rather than deciding whether to create one big Version 1.0 that does everything. The concept is to identify small "chunks" of work of the minimum size and scope necessary to generate some measurable benefit. Although project managers often object to re-evaluating their projects, requiring ongoing projects to be re-evaluated is not unfair or overly burdensome to the managers of those projects. Not as much effort is required to re-evaluate an ongoing project because template inputs need only to be updated rather than generated "from scratch." Estimates tend to get less uncertain as a project proceeds in its life cycle, and providing up-to-date inputs allows portfolio information to be kept current. Also, ongoing projects normally have an advantage in the competition; the relevant costs are the remaining costs needed to obtain the anticipated project benefits, exclusive of costs already spent. Furthermore, when evaluating on-going projects, the costs of terminating a project must be considered. Avoiding contract termination costs, staff reassignment costs, etc., are legitimate benefits of continuing the project. Thus, achieving the required bang-for-the-buck usually isn't difficult for ongoing projects. If an ongoing project is eliminated, the decision should be regarded by all as an instance of successful resource reallocation, not as a personal failure on the part of a project manager.
40
Gate Reviews Note that the evaluation of on-going projects for the purposes of project prioritization is not meant to replace other project progress evaluations. A large, multi-stage project may need to be evaluated at specified phases or "gates" to ensure that it is on track to meeting its milestones and deliverables (Figure 16). For example, a complicated project might include development of an initial concept, a feasibility study, a design phase, an engineering phase, and so forth. At each stage, managers may decide to revamp the project. Project gate reviews involve in-depth evaluations based on real-time information, but they are made in relative isolation to the decisions made on other projects. Nevertheless, a gate review can signal a project that is "in trouble," and this could trigger a re-evaluation (prior to the next portfolio review) or even result in immediate project termination.
Figure 16: Project gate reviews must be coordinated with project portfolio management.
Prioritizing Projects between Major Reviews As demonstrated by the above example, projects may sometimes have to be evaluated between the major budgeting cycles. This is true even in the absence of formal gate reviews. For example, a project may need to be re-evaluated if there is a major, adverse change in the project scope, a risk event occurs, an assumption is found not to hold, or there is a realization after completing a portion of the work that earlier cost or schedule estimates were overly optimistic. Also, urgent new projects may need to be evaluated between planning periods. Evaluations between major reviews involve comparing such projects against the most-recently established project priorities. A contingency fund may need to be established to accommodate projects that may be added outside the normal budget cycle. Multiple Project Versions Although it is not common to do so, it is often a good idea to require multiple versions of proposals for large projects. For example, if a project exceeds some specified size, some organizations will require project proponents to submit enhanced, decremented, and minimum-cost versions of the project, in addition to a base-case version. By providing project alternatives, organizations can avoid "all-ornothing" choices for important, but resource-demanding projects. Proposal Evaluation and Process Management Based on formal evaluations, projects are prioritized and "go," "no-go," kill, and "hold" recommendations are made with the goal of creating a value-maximizing project portfolio. Project recommendations are reviewed by the executive committee and project funding for approved projects is
41
authorized. Projects are phased based on critical paths to fit people and other resource constraints, with the most urgent projects starting first. Projects designated as "hold" may be resubmitted later (oftentimes project managers redesign such projects with the goal of increasing benefits and decreasing costs). The portfolio management office monitors the status of on-going projects to ensure that projects stay on track to achieve the anticipated benefits that motivate their being included in the selected project portfolio. Once the project is completed, the products (assets, services, etc.) delivered by the project should be similarly monitored and evaluated to ensure that the forecasted benefits are realized. The performance of activities that utilize project outputs is typically beyond the responsibility of the project portfolio manager. However, it is important for responsible managers to provide the project portfolio manager with feedback on the extent to which actual benefits are in fact achieved. Without such feedback on actual performance, there is no real accountability. In addition, the portfolio management office should: Ensure consistent and accurate progress reporting on the costs and other critical resources consumed by projects. Define and develop the detailed, continuous process by which projects are evaluated, prioritized, selected, and managed. Enforce a collaborative effort that enables senior executives (the steering committee) to reach agreement on portfolio goals and objectives. Provide coaching and training to project managers to help them to understand project evaluation criteria and to enable them to efficiently generate inputs for the project template. Communicate to project proponents and other stakeholders which projects are approved and project priorities. Provide stakeholders with timely assessments of portfolio progress, with early identification and correction of portfolio-level issues that may impact performance. Adjudicate resource conflicts between projects. Maintain visibility of key project information across the enterprise. Ensure that the project portfolio remains in tune with changing business objectives and strategy. Identify lessons-learned and continually refine the portfolio management process.
Resource Assignments and Staffing Although resource balancing tools support project staffing, with the exception of a few highly-focused custom applications, such tools do not provide capability to automate staff assignments. Mathematically, assigning staff to candidate projects is a constrained optimization problemin theory it should be possible to identify staff assignments that maximize a specified objective function. However, solving such a problem is complicated, especially since the optimal allocation must continually change over time. More importantly, specifying the assumptions for a mathematical analysis would be very difficult. For example, among other things, it would be necessary to know or estimate the relationship between the number of people assigned and the time required for completing project tasks. It would also be necessary to know how the project duration and the phasing of work affects performance.
42
Although some have tried to automate the resource assignment process, no commercially viable tools have yet been developed with such capability. Thus, rather than automate staff assignments, nearly all project portfolio management tools are designed to support the traditional four-step process wherein (1) candidate projects that could be conducted using available resources are identified, (2) the cost of such projects are estimated, including the labor costs, (3) project choices are identified that create project portfolios ject selection and resource assignment process.
43
Improve the Prioritization Process

The most challenging step in the above description of the project portfolio management office is choosing which projects to fund. In order to continually make the best use of limited organization resources, the portfolio office must decide which projects to initiate, which on-going projects to continue to fund, and which projects to terminate. As illustrated in Figure 17, the portfolio management office manages the project pipeline.
Figure 17: The Portfolio Management Office manages the project pipeline
For most organizations, making project selection decisions involves the forced-ranking of projects. By forced ranking, I mean that managers get together and force each project into either a strict priority ordering or into to a few priority groups. Projects are added until the business runs out of resources. Below that point, projects are put on hold or killed outright. Considerations that apply at the portfolio level, such as project synergies and portfolio risk, may be used as final modifiers to the ranking. Needless to say, forced ranking, as well as the final choice of which projects to conduct, are difficult decisions. In the absence of a more formal approach, politics can play a major role in project selection. In some instances, experienced, unbiased individuals who are well-informed about the needs to be served can do a good job of ranking projects. However, project ranking becomes more difficult and time consuming if there are: Many project proposals. If there are more than twenty or so requests, decision makers may not be sufficiently informed about the projects or their underlying needs to make consistent choices. Different motivations for projects. Prioritizing projects is much harder (and more contentious) when the motivations for doing projects can be very different. How do you compare a project intended to improve worker safety with one that makes a product more attractive to customers?
44
Project complexities. If projects are technologically complex or involve difficult-to-judge considerations (such as risks or critical timing factors), most experienced individuals will be hard pressed to provide reliable priorities without reference to formal criteria. Multiple decision makers. The more people involved in priority setting, the less likely it is that they will agree without first agreeing on the objectives, principles, and criteria to guide project prioritization. Pressures to justify project choices. There may be stakeholders not involved in setting priorities, including those who provide project proposals, who require an explanation for the funding choices that are made. If decision makers arrive at their choices intuitively, they may have trouble explaining priorities when questioned or challenged.
Although creating a portfolio management office and a project inventory helps, the key to reaping the true benefits of portfolio management is following through and implementing a formal, organized, and logical process for prioritizing proposals and optimizing the project portfolio. Improve Project Data Even the best project prioritization process will be worthless without adequate project data. As one manager put it, A micrometer wont help you measure a cloud. Steps must be taken to address the problems of poor project data that plague most organizations. The first step to getting better data is to make sure that information requirements are well-defined. If project proponents are clear about what information is needed to enable their proposals to be considered, they are much more likely to supply that information. Thus, the templates for collecting data on project proposals must be complete and precise. Second, there must be a culture and expectation that rigor is required to generate project proposals Estimates must be backed by reason and analysis. Project proponents need to do their homework before the project gets proposed up the management chain. Third, the organization must be prepared to allocate increased resources to project planning. Skill, experience, and true cross-functional collaboration are often needed to generate solid project proposals. Inevitably, increasing the effort devoted to preparing project proposals detracts from the resources available for actually doing projects. However, as previously asserted, the tradeoff in improved decision making will be worth it. Note that a lack of adequate systems for collecting relevant project data should not rule out attempts to implement project portfolio management. In my experience, project evaluation systems based mostly on data generated subjectively through best professional judgment can perform quite well, so long as those providing the judgments are knowledgeable and techniques are used to counter gaming and judgmental biases. Invariably, project portfolio management acts as a forcing function that causes the organization to improve its ability to collect important business data.
45
Be Careful Using Scoring Models Many organizations use scoring rules as aids to project prioritization, for example, with regard to a given criterion, 1 = unfavorable, 2 = neutral, and 3 = favorable. The scoring rules are meant to systemize the process used by the managers who collectively rank projects and to record their judgments. A score above a certain level is judged a must do. Alternatively, projects may be grouped into priority categories. Projects in critical categories, for example, projects that address safety issues or projects related to regulatory requirements, are designated priority one. Point scoring and priority categorization systems have significant limitations. Frequently, too many projects get labeled must dos. One reason is that the project proponents may designate as mandatory projects that have only a statistical influence on safety, as opposed to just projects that eliminate ongoing harm or remove clear code violations. Assigning a number to something doesn't necessarily make for a more accurate method of measurement. If scores are subjectively defined without clear criteria, the process can be as arbitrary as a beauty contest. Different people will assign wildly different scores to the same project, and the same person may assign different scores on different occasions. Regardless, middle scores are common for most projects, especially when numerous scoring criteria are used. High scores on some criteria cancel out low scores on others. Most scoring models aren't sufficiently precise to trust small differences in total scores. Furthermore, ranking projects by their project scores is generally incorrect anyway. Projects are undertaken to produce value for the organization. Thus, as noted previously, the proper ranking metric (for independent projects) is value added per unit of cost. Most scoring systems dont claim to measure value. Even when they do, they often fail to scale results to project cost and, therefore, dont estimate "bang for the buck." (Methodology for developing defensible, value-based scoring methods as well as problems with simplistic methods are described in the next part of this paper.) Estimate Cost, Value and Risk Correctly prioritizing projects requires being able to estimate the costs, value, and risks of alternative project portfolios. However, each side of the equation is difficult. Project costs include not just the funding request, but also any funding provided from other sources plus the opportunity costs of using equipment, personnel, raw material and any other "non-costed" resources that will be employed by the project. Also, all future costs necessary to obtain project benefits, including future O&M costs, should be identified, estimated, and included in the calculation. Some companies still do not track costs at the project level, relying instead on the general ledger system to impute approximate project costs. Tracking project costs is essential to encourage accurate estimating and to provide budget data needed to make, monitor, and update project decisions. The foundation for effective project portfolio management includes a finance system that tracks labor costs using fully burdened labor cost rates for roles and individual resources.
46
Estimating value is even more difficult than estimating costs. Systems must be in place to help managers measure the benefits of projects as well as to determine the value of alternative project portfolios. This leads to the hardest partdeveloping the metrics for measuring project and portfolio value. See the next part of this paper for information on how to this can be accomplished.
47

1. P. Belliveau, A. Griffin and S. Somermeyer (eds.), The PDMA ToolBook for New Product Development, Wiley, 2002. 2. The TOC Multi-Project Management Method, The Product Development Institute, Inc., Whippany, NJ , 1999. 3. S. Berinato, "Do the Math," CIO Magazine, October 1, 2001.
48
Choosing the Wrong Portfolio of Projects Part 3: Lack of the Right Metrics
Lack of the Right Metrics

The metrics that an organization uses for evaluating projects have a big impact not only on the projects that get chosen but also on the projects that get proposed. "Tell me how you will measure me, and I will tell you how I will behave" [1]. Even if the metrics aren't used to create incentives, managers interpret them as indicating what the organization regards as important. Lack of the right metrics is the third reason organizations choose the wrong projects.
Metrics = Incentives
49
Defining Project Value

The goal when choosing projects should be to maximize the total value to be derived from those projects. But, how do you measure project value? More fundamentally, how do you define value? A Definition of Value People have been arguing about the definition of value for centuries. The relevant concept of value for our purposes is termed the "utilitarian concept of value: The value of a project is a measure of the degree to which that project enables the organization to achieve its objectives. This view of value was first articulated in the fourth century BC by Aristotlethe value of something is not an intrinsic property of that thing, but rather is determined by its usefulness to those that want it. Organizations conduct projects because they believe that the consequences or outcomes of the projects will be useful. Thus, we define the value of a project to be the worth, to the organization, of obtaining the consequences of the project. The project consequences that are desired depend, of course, on the organization's fundamental objectives. Suppose, for example, that an organization's fundamental objective is to create value for its shareholders (more discussion of shareholder value is provided below). Suppose further that the organization is considering a hypothetical project that provides only one consequence: an immediate, one-time cash flow to the company of one million dollars (to be more precise, suppose that the magnitude of the cash flow from the project is such that, after all tax and accounting considerations are addressed, increases the net worth of the company by $1 million). Under the specified assumptions, the value of that project would logically be $1 million. More precisely, I define the value of a project to be the maximum amount the organizations leaders would be willing to spend in order to obtain the consequences of doing the project (including consideration of risks). In the literature on valuation, this value is termed a buying indifference priceit is a buying price because it is an amount the organization would pay to buy the project consequences, and it is an indifference price because it is the price point such the organization is indifferent between passing on the project versus paying the price and obtaining the project consequences. Note that our definition of project value has an important property: An organization will prefer Project A to Project B if and only if our measure of project value is higher for Project A than it is for Project B. Many common approaches to project prioritization dont use a definition of value that has this essential property (see, for example, the discussion of strategic alignment in a later section). Unless project value is defined in a way that maps to the true preferences of the organization, that measure cannot correctly prioritize projects. Our definition of project value is a true measure of the relative attractiveness to the organization of its project alternatives.
50
Project value is what the organization would be willing to pay to obtain project consequences:
Project A is preferred to Project B if an only if: Value[Project A}>Value[Project B]
How do we determine how much an organization would be willing to pay to obtain the anticipated consequences of conducting various projects? Before addressing that question we need to consider more carefully why organizations conduct projects. Projects Determine the Evolution of the Business The business of an organization is always evolving, and the projects that the organization chooses affect that evolution. For example, a new technology might become available that would allow the organization to reduce its costs. If a project is conducted to install the technology, the organization would incur lower operating costs than it would have had it chosen not to do the project. It is also true that the projects that an organization chooses not to do affect the evolution of the business. For most organizations, standing still means falling behind. There are many reasons for this, including increasing competition, changing customer preferences, and the aging of organizational assets. Thus, to take one example, if the organization chooses not to do projects that maintain or replace aging assets, the service provided by those assets will decline. Figure 18 illustrates a useful way to think about project value. At the point in time when an organization is considering a new project, it is really facing a choice between two alternative future states. If the project is conducted, that project will, presumably, transform the business to some more desirable state. If the project is not conducted, some other, presumably less-desirable, state will result.
51
Figure 18: Project choices determine the future state of the business.
The perspective of Figure 18 provides a basis for computing project value. The value of a project is the difference between the values of the two potential future states of the business: Project value = Value with the project Value without the project Some Observations Regarding Project Value Based on Equation 1, we can conclude some important things about project value. To estimate project value, it is necessary to consider what would happen if the project is not conducted as well as what would happen if the project is conducted. Many project scoring systems ignore the implications of not doing projects, which causes some types of projects to be grossly undervalued. The same project can have different values to different organizations. For the purposes of estimating project value, therefore, it is not sufficient to consider just the characteristics of the project itself. It is also necessary to consider characteristics of the business that determine how useful that project will be to that business. The value of a project can change depending on what other projects are conducted. Because of synergies and economies of scale, for example, the costs and benefits of a project can change based on the other projects that are also conducted. If such dependencies exist, optimizing the project portfolio requires estimating the value of conducting different groupings of the interdependent projects. (Equation 1)
52
For the purpose of identifying metrics for measuring project value, the most important implication of Equation 1 is this: Project value is the contribution that the project makes to the total value achieved by the organization. Because project value is the difference between the values that the organization attains in two alternative states, the methods for estimating project value are essentially the same as the methods for estimating organizational value. This is good news, since management scientists have devoted considerable effort to determining how best to measure the value created by organizations. We can use the concepts and methods that they have devised to enable us to quantify project value. Inadequacy of Financial Metrics So, how can you measure the value created by an organization? Interestingly, most businesses get it wrong. Most businesses use financial metrics computed from cash flows to measure value, for example, return on investment (ROI), return on assets (ROA), internal rate of return (IRR), net present value (NPV), pay-back period, etc. Using these metrics to evaluate candidate projects requires forecasting the cash flows that would be produced by the project. Some impacts on cash flow may be relative easy to forecast, in particular, the costs to conduct the project and any direct impacts the project will have on the firms future costs and revenues. However, it is difficult to translate many project consequences into impacts on cash flow. For example, how would a project designed to collect better information about customer preferences impact future cash flows? From a practical standpoint, cash flow analysis will not capture many project impacts. Thus, at best, financial metrics provide only a partial representation of what is important to a business. According to a study by Research Technology Management, companies that rely mostly on financial metrics obtain unbalanced portfolios that are not well matched to the strategy of the firm [2]. The limitations of financial metrics are even more obvious when it comes to evaluating projects in the public sector. Public-sector organizations have social missions and may not even sell goods and services that generate cash flows. Even it they do, earning a financial return may not be a primary objective. For example, a water utility has a mission that includes serving community water needs. A public school has a mission that includes educating students. Cash flow analysis will not do a good job of indicating how well these organizations are accomplishing their missions. Financial metrics fail to measure the value of projects intended to achieve non-financial objectives. Two Views on Organizational Value If financial analysis is inadequate, how then can we measure value? There are two competing views on how to measure the value created by an organization, based on the answer to the question, Value to whom? The first view is that the purpose of organizations is to produce value for owners (shareholder value). The second view is that the purpose of organizations is to create value not just for owners, but for other stakeholders as well (stakeholder value). Shareholder Value As indicated above, most management scientists and U.S. business leaders argue that the fundamental objective of investor-owned organizations is to maximize value for the owners of the business. In the case of a publicly traded company with one type of stock, shareholder value is roughly the number of outstanding shares multiplied by current share price. Dividends, of course, increase the return to
53
shareholders while simultaneously decreasing share price. For both theoretical and practical reasons, increasing market value can be argued to be the ultimate goal of many private-sector businesses. One argument for shareholder value is the objectivity of market valuations. Market prices are objective and applied consistently across all publicly held companies and markets. If they were not there would be opportunities for arbitrage; that is, one could buy shares of a company at a lower price in one market and simultaneously sell them at a higher price in another market. Irrespective of theory, seeking higher stock value is clearly a practical objective for senior management. Compensation for top executives is often tied to stock performance. Poor stock performance is a common reason for a companys board of directors to consider replacing its chief executive. A company whose stock is depressed because it is not effectively managing for shareholder value is a candidate for investor takeover. It is important to understand that managing for shareholder value and managing for profitable cash flows are two very different things. The figure below compares the market values and risk-adjusted discounted values (NPV) of projected future earnings for four companies in the same industry. In each case, the discount rate for computing the NPV is the companys weighted average cost of capital.
Figure 19: Market value is not the NPV of projected earnings (data for 4 companies in same industry).
As illustrated, a companys market value is usually higher than the NPV of its earnings. In the literature on real options, the additional value is termed option value. In addition to projected future cash flows, market value depends on how buyers and sellers in the marketplace perceive, among other things, the ability of the business to obtain and respond to future opportunities and avoid future threats. Also as shown in the figure, it is not uncommon to find two companies in the same industry such that one has a higher NPV of projected returns but the other has a higher total market value. Indeed, some dot com companies have substantial market values yet cannot convincingly forecast any profits. Option value does eventually translate into earnings. However, because it is generally difficult or impossible to forecast the exact mechanisms by which this will occur, option value and other indirect sources of value will not be represented in accounting forecasts of cash flows.
54
The implication of the above is that impacting cash flows is only one of several paths by which projects can influence shareholder value. The alternative paths that may be relevant depend on the nature of the organizations business. For example, a project that generates new knowledge that improves consistency of execution can reduce investment risk. This can have the effect of lowering the cost of capital and increasing the returns on the capital invested. Investors seeking investments likely to increase in value will buy more shares, which will increase shareholder value. As another example, a service company might conduct projects that enable it to meet commitments or promises to customers. This improves customer perceptions and can influence the views and reports of Wall Street analysts, resulting in higher stock prices. According to the perspective of shareholder value, the appropriate metrics for evaluating projects in the private sector are those relevant to forecasting impacts on the market value of the business. Real options analysis and related methods are designed to do this. Companies that evaluate projects by estimating impacts on profits alone ignore a significant component of market value. Stakeholder Value The alternative view, more popular outside the U.S., is that firms should create value for all stakeholders, not just shareholders. Stakeholders are groups of people who have an interest in the business or organization (Figure 20). Typically, important stakeholders are those groups whose actions can have an important impact on the success of the organization. Stakeholder value may include creating value for employees, suppliers, customers, the local community, and society at large.
Figure 20: Some potentially important stakeholders.
Proponents of narrower shareholder value argue that organizations pursuing self-interest and economic efficiency do, in fact, serve all stakeholders. However, this argument carries the implicit assumption
55
that the markets within which the firm operates are "perfect," so that stakeholders other than shareholders are unharmed by the firm's focus on benefiting shareholders alone. For example, if employees are laid off, they can immediately get equivalent jobs elsewhere. Similarly, if suppliers or consumers are mistreated, they can take their business to other firms. If there is any impact on stakeholders, the argument goes, it is beneficial. The most successful companies create the products and services that customers want most at the lowest price. Their rising stocks attract more investors, which enables them to satisfy even more customers and hire more employees. The tax system, the argument goes, ensures that successful companies create tax revenues for social ends. The U.S. economy is well diversified, and many have argued that this makes the "perfect market assumption a pretty good approximation. However, well-publicized accounting and investment scandals, declines in the U.S. housing market, volatile oil prices, and the global environmental impacts of carbon emissions are raising new questions about the wisdom and acceptability of absolute reliance on shareholder value. Some critics of shareholder value have claimed that the current recession was largely caused by excessive risk taking fueled by shareholder demand for ever higher levels of short-term gain. Still, defenders of shareholder value have responded by blaming the principal-agent problemthe providers of capital and the manager-agents they employ to make decisions inevitably have different incentives and information. The best way to serve the long-term investor, they say, is to tie decisions more closely to shareholder value. Regardless of ones stance in the debate on shareholder versus stakeholder value, prudence suggests that private-sector companies should give consideration to impacts on stakeholders when evaluating projects. Companies in regulated industries are often required to serve the interests of customers, citizens, and other stakeholders. In any case, it would be foolish for a company to ignore significant concerns of its stakeholders. Recognizing the importance of stakeholder value further weakens the argument for using financial metrics as the sole basis for project prioritization. Stakeholder value is almost certainly the more appropriate definition of value for non-profit and publicsector organizations. However, even those companies that care only about shareholders need to consider how their actions affect other stakeholders. If they dont, they may fail to conduct projects needed to prevent adversely impacting the attitudes and perceptions of workers, customers, regulators, and local citizens and/or public officials. Careful investors factor considerations such as employee relations, customer satisfaction, regulator attitudes, and community perceptions into their investment decisions. Thus, significant impacts on stakeholders inevitably affect shareholder value. As is the case with shareholder value, methods are available for quantifying stakeholder value. The appropriate project valuation metrics are those that result from applying these methods. As shown in the next sub-section, metrics may be found by constructing a project-selection decision model that evaluates projects and project portfolios based on the goal of maximizing organizational value (shareholder or stakeholder value).
56
Project-Selection Decision Models

So, what kinds of metrics reflect impacts on value? Many organizations have trouble answering this question. Organizations tend to measure whats easy to measure, not necessarily whats important. Most organizations use a bottom-up approach. They define interesting metrics, but then can't come up with the algorithms for computing project value based on those metrics. They end up using arbitrary aggregation equations, such as weighted summations, or vague and unjustified concepts, such as balance or strategic alignment. Unless there is a logical way to combine metrics to determine the value added by projects, the metrics will not be of much help in identifying go/no-go decision rules or in choosing value-maximizing project portfolios. How can you determine the value added by projects? Create a Decision Model The answer isyou need to reverse the process, use a top-down approach and create a project-selection decision model [3]. A decision model is an analytic representation, typically implemented in software, that captures and derives recommended choices based on representing the components of a decision (Figure 20). The 3 decision components of a decision are: (1) the alternatives that are available to you, (2) what you want, and (3) what you know and believe about how the choices that you make affect the outcomes that you will receive.
Figure 20: A decision model derives recommended choices based on the 3 components of any decision
The decision model is structured using logic and incorporates principles for promoting clear and unbiased thinking. From a mechanical standpoint, the model that you create takes as input characteristics of the project, the projects anticipated impacts, and other factors, and produces as output a dollar measure of the value added by the project (Figure 21).
57
Figure 21: A project-selection decision model values projects in two steps.
As indicated by Figure 21, a decision model values projects in two steps. First, the model estimates project consequences relevant to the achievement of the organizations fundamental objectives (e.g., creating shareholder or stakeholder value). Project consequences are estimated based on cause-effect reasoning and inference (in the figure, this step is labeled simulation). Then, those consequences are translated into the worth to the organizations of obtaining those consequences (the valuation step). The consequences are valued based on a formal theory of valuation. Defining Objectives Project selection decision models are constructed from the top down because the first step is structuring objectives. Structuring objectives means creating a hierarchy of well-defined objectives by identifying the organizations fundamental objectives, identifying sub-objectives that specify or clarify how those fundamental objectives may be achieved, and continuing to define additional sub-objectives as needed or to the extent that it is helpful. For example, suppose the organizations fundamental objective is creating value for shareholders. As shown previously (Figure 19), shareholder value may be regarded as the NPV of the companys projected cash flows plus the companys option value. Thus, a shareholder value objective could be split into two sub-objectives: (1) maximizing the NPV of future cash flows and (2) maximizing option value. Relevant sub-objectives for increasing NPV would include (1) increasing revenue and (2) decreasing costs. Sub-objectives for maximizing option value depend on the organization and the nature of its business, but generally such sub-objectives involve mechanisms for increasing the ability of the organization to identify and take advantage of future opportunities and avoid future risks, as well as other sub-objectives that, if achieved, would enhance investor expectations and, therefore, increase the market value of the company.. If, on the other hand, the fundamental objective of the organization is maximizing stakeholder value, then the objectives hierarchy may be established by identifying who the stakeholders are and then identifying and structuring their objectives. An example of an objectives hierarchy for a case wherein the organizations fundamental objective is to create value for stakeholders is provided in the next subsection.
58
Much has been written about how to define, characterize, and structure decision objectives. Basically, the goal is to specify objectives that are (1) fundamental (a common error is to specify as an objective something that is only one means for achieving what is really desired), (2) well defined (such that people readily agree on the meaning of each objective), (3) measurable (it is possible to know and measure how well each objective is being achieved), (4) comprehensive (the set of objectives is sufficiently complete to capture all of the strengths and weaknesses of available alternatives), and (5) non-redundant (at any specified level of the hierarchy, there is no overlap or double counting among the objectives at that level). Characterizing Distinctions Once objectives have been structured, the next step for constructing a decision model is specifying measures for characterizing the distinctions that determine how well the objectives are being achieved. The distinctions that we seek to identify are the attributes or characteristics that we care about. For example, suppose the context is decision making by a utility that distributes electric power. Customers (and therefore the utility) care about power outages, so one distinction that matters when thinking about electricity delivery is whether or not power is available to the customer. Any measure we define must have at least two possible states, for example, yes power or no power, but we can define measures to have many more states if it helps us to differentiate alternatives and measure how well we are achieving our objectives. For example, the measure for electric power to the customer could alternatively be defined as the voltage available from the customers power outlet, which could be zero, 110 volts, 109 volts, 108 volts, etc. Oftentimes, multiple measures are required to characterize with adequate precision the achievement of a decision objective. For example, another distinction that would matter in the context of customer outages would be how long it takes to get the customer back on line. Thus, outage duration could be characterized by the number of minutes that the customer is without power. In general, measures should be defined whenever (1) the degree to which the objective is achieved depends significantly on performance against that measure, (2) performance against the measure doesnt track or exactly correlate with performance against some other measure that has already been defined (i.e., you need the additional measure to distinguish possibilities), and (3) available understanding is sufficient to conclude that the level of performance against the measure will differ depending on the choice that is made (i.e., the measure differentiates among the alternatives). Measures must be defined for each decision objective. For example, customers care about the accuracy of their bills. Thus, the percentage of customer invoices with errors might be a relevant measure for an organization concerned about improving customer billing. As an example relevant to measuring option value, customer brand loyalty might affect investor expectations. A possible measure for impact on brand loyalty might be the percent of customers making repeat purchases or the companys ranking in customer satisfaction surveys. The selection of measures is one of the more creative aspects of model building. In addition to choosing measures that quantify the achievement of the objectives, measure selection should be based in part on the types of data the organization does or can collect to assess performance. Influence diagrams (illustrated in the next sub-section), are often used to identify useful performance measures.
59
Describing Possibilities and Likelihoods Measures that characterize the distinctions that matter provide the framework needed to capture project consequences. To illustrate the concept, imagine that we combine the measures that weve defined into a tree. The tree starts with a set of branches indicating the possible states for the first measure, which are connected to branches for the second measure, and so forth. The paths through the tree define all of the possibilities, and that set of possibilities is the same regardless of the order in which the tree has been drawn. Some of the possibility scenarios will, of course, be more desirable than others, and it will be the job of the valuation model to sort this out (see below). The job of the consequence simulation model is to capture how the choice of project alternatives affects the likelihood of the various scenarios (paths through the tree). Sometimes, a project choice that we make will ensure that we end up on a specific branch (reflecting a specific outcome or level of performance) for a measure. For example, if the electric utility from the example above conducts a project to add a distribution line that will reach 100 new homes, and the number of residential customers receiving service is a measure of performance, then, assuming the project is successfully completed, the number of customers will be incremented by 100. If there is uncertainty about project consequences, a choice merely affects the likelihood of the various branches for that uncertain measure (e.g., the number of new customers that will be added is somewhere between 0 and 100, depending on how many of the new homes in the housing development that the line will serve will be built and sold to homeowners). The above illustrates that one way to capture the relevant beliefs about the consequences of choices is to assign probabilities to the possible performance outcomes that correspond to the paths through the tree of possibilities. We could do this by assigning probabilities to the branches for the first measure, conditional probabilities to the branches for the second measure (that indicate how likely each level of performance is for the second measure given each possible level of performance for the first measure), and so forth. If we then multiply all of the probabilities along any path, we will obtain the probability of that scenario. Since the goal of the consequence model is to express our beliefs about how choices affect our ability to achieve the distinctions we care about, a modeling approach based on assigning probabilities to the branches in a tree of possibilities would provide a fully adequate consequence model. Creating a Consequence Model The purpose of above description is not to suggest that developing a tree structure and assigning probabilities is always the best way to construct a project consequence model. Rather, it is to demonstrate the basic problem that must be solved and to illustrate that there is at least one solution approach that will always work. Consequence models built on tree structures are, in fact, quite common. For example, for evaluating pharmaceutical projects, tree models are typically used to lay out the likelihood of success and timing and to simulate the outcomes of the various tests and regulatory approvals that are required. There are, however, many other model forms that may be used to infer or simulate project consequences. The simplest approach is direct estimation (What do you think will happen if the project is conducted?). A common approach for the case where data are available on the outcomes of similar projects is to use statistical analyses to establish relationships between project characteristics and project consequences.
60
Importantly, there are many existing models for estimating various types of business consequences. For example, perhaps the most common sub-model used in project evaluation is the financial, or business case model. The typical project financial consequence model computes the NPV of cash flows from the project (free cash flows), accounting for the timing of incremental costs and revenues, taxes, the time value of money, and other considerations that affect the equivalent, current financial worth of the cash flows to the business. Thus, project consequence models typically include a cash-flow NPV model as the sub-model for measuring the projects direct financial benefits. Other models have been developed for capturing understanding regarding many different processes, activities, and impacts relevant to evaluating project impacts on business objectives, and these models can likewise be utilized. For example, there are well-established models for estimating R&D success, the impact of project characteristics on customer choice, the rate at which new products penetrate exiting markets, the effectiveness of projects to reduce pollution, the productivity and reliability of business assets, and so on and so forth. Selecting, refining, and piecing together various sub-models that have worked well in other applications is the most common approach for creating project selection decision models. Valuing Consequences Project consequence must be valued in a common unit of measurement; otherwise they cant be combined to obtain a total project value. Since financial gain is an expected consequence for many projects, dollars (or another unit of currency) is the obvious choice for valuing all project consequences, including non-financial benefits. Also, unless we express project value in terms of dollars, we wont be able to determine whether project benefits justify project costs. Money is the clear unit for measuring value. One approach for determining the monetary value of potential project consequences would be to ask the organizations leaders to provide the numbers. However, the task would be difficult and it would be hard to obtain consistent answers. Thus, methods based on formal analysis are used. Various theories and associated models exist for determining how much decision makers should be willing to pay for something, assuming certain assumptions apply. For example, the discounted cash flow model computes a current value for a project that generates a time stream of future payments based on discounting. The discounted cash flow model can be derived from two assumptions: (1) that investors wish to maximize their wealth and (2) that a market exists that will provide a return from investing cash. Other valuation methods exist for valuing projects that produce non-financial as well as financial project consequences. The three best known methods for assessing the equivalent dollar value of decision outcomes are (1) cost benefit analysis, which derives the dollar value of outcomes based on peoples willingness to pay and data on how outcomes are priced in the marketplace, (2) multi-attribute utility analysis (and related multi-criteria methods, such as AHP), which assign values based on modeling peoples preferences and their willingness to make tradeoffs, and (3) real options analysis, which includes in the valuation the flexibility made available or destroyed by project decisions. These methods, which have long been used to quantify shareholder and stakeholder value, are described in detail in many articles and books. Each includes capability to include considerations of risk as part of the valuation (see Part 4 of this paper).
61
For potentially controversial dollar conversions, such as for health and environmental impacts, rather than assign values directly, most organizations find it convenient and less controversial to leverage results from other sources, including academic research, government recommended values, and values adopted by other organizations. Sensitivity analysis; that is, varying assumed values across a range to see whether it changes decision recommendations, is commonly used to demonstrate that most project decisions are not overly sensitive to assumed dollar conversions.
62
Is It Hard to Create a Project Decision Model?

The prospect of creating a model that quantifies the dollar value of projects no-doubt sounds hard. Building a decision model requires developing and documenting understanding of what your organization does, how it does it, and how the choices that are made determine the value that is created. This understanding is critical to knowing what to do to create organizational success. Thus, one could argue that using decision modeling to identify project evaluation metrics merely forces the organization to do what it should be doing anyway. Actually, building a decision model is not as difficult as it may sound. In my experience, the basic design for even a sophisticated decision model can be structured in a 3-day framing workshop (using techniques based on value modeling, influence diagramming, and cause-effect reasoning). The resulting qualitative model can then be quantified fairly quickly by refining and piecing together pre-existing submodels and using well-established relationships for estimating various types of benefits. As noted previously, in addition to project financial value, standardized sub-models are available for quantifying health, safety, and environmental value; customer value derived from products and services; brand image value, learning value; and so forth. The combined model captures business understanding in relevant areas such as R&D, engineering, manufacturing, marketing, sales, customer relations, legal counsel, regulatory affairs, etc. The result establishes an explicit connection between the project, the impacts the project would create on the business, and the value of those impacts. An Example Figure 22 (next page) provides an example. The figure is a graphical representation of a portion of a project-selection decision model developed by a team at an electric utility. The figure provides detail only for that portion of the model for measuring project value attributed to improving electric service reliability. The upper part of the figure is a hierarchy of objectives for selecting projects. As indicated, the utility adopted stakeholder value as its overall objective. Key stakeholders were identified to be shareholders, customers, workers, citizens, and others (e.g., business partners, elected officials, some state and federal agencies, etc.). The sub-objectives for stakeholder value represent the fundamental concerns of the various stakeholders: (1) the utilitys financial performance; (2) health, safety and the environment; (3) satisfying customers (both existing customers who want high-quality service and new customers who will desire electric service); (4) satisfying other stakeholders (e.g., responding to regulator concerns and maintaining a good image with the citizens of local communities), and (5) building a platform for future success (providing learning, improved capability, flexibility, ability to respond quickly, etc.). The middle part of the figure is the influence diagram constructed for the service reliability objective. It identifies the factors, relationships and metrics assumed to determine the level of satisfaction derived by customers based on the reliability of the electric service provided. As indicated by the nodes and arrows in the diagram, the utility believes that a customers level of satisfaction with service reliability depends primarily on the frequency and duration of the outages that the customer experiencesthe fewer outages experienced and the shorter the duration of those outages, the happier the customer will be.
63
Figure 22: Portion of a project selection decision model for an electric power delivery company.
The bottom portion of the figure indicates how the reliability component of the decision model was quantified. As indicated, the service reliability sub-model takes as input the metrics identified in the influence diagram and produces as output its measure of reliability performance: total weighted annual
64
customer minutes out. This performance measure assumes that the level of customer dissatisfaction is roughly proportional to the total amount of time that the customer is without power (some utilities use a scaling function to indicate that particularly long outages, for example, outages that would cause food in refrigerators to spoil or companies to send workers home is proportionally more costly). Finally, through weights assigned to different customer types, the utility reflects the assumption that outages are more harmful or costly for some types of customers than for others. This utility chose the weights for residential, commercial and industrial customers based on industry studies estimating the actual dollar costs of outages to these various types of customers. In addition, the utility defined critical customer to serve as a special category with a very high weight for customers, such as a hospital, for reliability of service is judged to be especially important. The sub-model for measuring reliability benefit works as follows. If a proposed project will impact service reliability, the project proponent estimates the scope of impact (How many residential, commercial, industrial, and critical customers will be impacted?). These numbers can be derived based on the configuration of the utilitys distribution network and the specific components of the network that the project will address. In addition, forecasts are provided for the number and duration of outages experienced by these customers (1) if the project is not conducted and (2) if the project is conducted. These forecasts are made by extrapolating data on historical outages within the project scope and based on estimates for how the proposed project will affect system performance. The measure of project reliability benefit is total weighted customer outage minutes avoided, and the value is expressed in dollar terms based on the weights assigned to the customer types To illustrate another aspect of this decision model, Figure 23 shows how the value of serving unmet energy needs was quantified. The assumption is that providing power to new customers creates value for those customers because the value derived is greater than the price charged. (This principle is termed consumer surplus by economists.) To estimate the value of projects that provide this benefit, the following logic is used. First, the expected shortfall in energy service is estimated based on forecasts of growing energy demands relative to the capacity constraints of the existing electric distribution network. In the utility industry, the shortfall is referred to as expected unserved energy (EUE), and it is measured in kilowatt hours.
Figure 23: The utilitys sub-model for new energy delivery.
65
The black curve in Figure 23 shows the results of such a forecast of EUE (the jumps in the curve in the early years correspond to the completion of known residential and commercial development projects; the curve increases smoothly in later years because in this time frame the forecast is based only on estimated average annual growth rates). The second step is to forecast the EUE that would exist if the project is conducted. The EUE with the project (the red curve shows a sample forecast) is obtained based on the schedule by which the project would bring new capacity on line (this determines the shape of the curve in the initial years), the total amount of new capacity provided by the project, and the point in time when the new capacity would be fully utilized (this determines the gap between the two curves in the later years). The difference between the two curves represents the EUE avoided (in kilowatt hours). To convert EUE avoided to a dollar value, the utility again uses data from willingness-to-pay surveys. Too Complicated? If this sounds a bit complicated, consider that the utility using the above-described decision model spends hundreds of millions of dollars each year on new projects. Yet, limitations on resources means that many project proposals must be delayed or killed. If the improved decisions that result from the formal decision model and associated priority system only increase the value derived from projects by a few percentage points, the required effort is easily justified by the value gained in just the first year of use. Furthermore, a decision model can be a very effective way of explaining decisions and justifying the need for project resources. The utility in the above example has presented its decision model to its regulators and used model results to help explain and defend its project choices as well as proposed rate increases that would enable it to conduct additional projects. Less complicated decision models are typical for organizations with smaller project budgets, less complicated decisions, and less need to defend choices to outsiders. However, the basic principles for building the decision model are the same. Although organizations may be initially uncertain about many of the relationships that must be specified to define their project decision models, understanding these relationships is key to success. Again, the management science and industry science literature contains numerous, often business-specific submodels for estimating and measuring performance. Thus, as noted previously, the process of defining the relevant computations typically involves selecting and combining elements of previously developed and accepted approaches. Organizations that create explicit project decision models document bestunderstanding. As understanding improves, they revise their models and thereby further improve their ability to optimize their project portfolios. Decision Model Uses Creating a decision model takes work, but it is worth it. Having a decision model is critical to making intelligent choices. Knowing project value allows you to determine whether the project should be done at all, and whether, after it has been started, it should be continued. Knowing the value of your various project portfolios tells you whether you are allocating too much or tool little to each, and enables you to determine the right allocation of resources across your various organizational units and business functions.
66
A decision model has other uses as well. For example, a decision model provides a way to estimate the value of a day of additional effort, the value of a project feature, or the value created by expending a dollar more of project cost. The project team or portfolio manager can use the decision model to illustrate how a marginal change in resources, say plus or minus 10%, might affect the overall value to be generated. A decision model is a means for explaining and justifying the resources required for doing projects. Finally, the decision model sends important signals to those who propose and manage projects. The decision model tells engineers and others what project characteristics and attributes are valued in the funding process. It tells managers who execute funded projects what performance is expected if the project is to create the value that motivated its funding.
67
The Scientific Method

My advice that organizations create a project-selection decision model in order to define project evaluation metrics is nothing more than a recommendation to follow the "scientific method." The scientific method refers to a multi-step process that forces a careful, deliberate approach to building and validating new knowledge. It is the process scientists use to develop accurate (that is, logical, reliable, and non-arbitrary) answers to tough questions. The scientific method was formalized in the early 1600s by the lawyer and philosopher Francis Bacon. Bacon, who was heavily influenced by Copernicus, was critical of the then common scholastic approach to learning wherein accepted authorities would deduce and then disseminate broad scientific truths based on limited observations. For example, casual observations of the night sky led people to conclude that the planets circle the earth, not the sun. Copernicus had a difficult time reversing this belief. Bacon, in his book The New Organon (Greek word meaning tool or instrument) argued that whatever the mind seizes and dwells upon with particular satisfaction is to be held in suspicion. He proposed an inductive form of reasoning committed to empirical evidence that would build understanding through a series of steps. Figure 24 provides one representation of the scientific method.
Figure 24: Developing a decision model is an application of the Scientific Method.
68
The scientific method embodies several important principles. The first is that the problem to be addressed should be formulated in terms of the specific question you want solved, not in terms of some other question that you think might be easier to answer. Since the goal is to select projects that create maximum value, the relevant question for project selection is, What is the value of a project? In contrast, none of the following questions about which much has been written address what we really want to know: How do we measure balance? How do we measure project alignment with strategy? What are some potentially useful project metrics? Second, because the scientific method is fact based, information relevant for solving the problem is collected. In this case, such information includes understanding the relevant theories for measuring value. Third, a model (hypothesis) is constructed to explain the relevant phenomenon. For our purposes, the necessary model is one that predicts the consequences of doing versus not doing proposed projects and assigns a value to those consequences. The model provides a causal explanation for what is observed (that some projects produce more highly-valued consequences than others). Finally, central to the scientific method is the concept of testing and refining the model. This can involve "thought experiments" or the comparisons of real-world observations with model predictions. Do the projects that the model predicts will produce highly-valued consequences actually produce those consequences? Whenever inconsistencies are observed, understanding is refined and the model is revised accordingly. A decision model is not much different from the types of conceptual models developed in other areas addressed using the scientific method. Because the model is quantitative, it allows statistical measures of the reliability of the results to be established. Initially the decision model may be constructed based on judgment, but it is tested and refined based on facts and observations. The key is that the model is available for scrutiny by people other than those who developed it originally. Individuals with special expertise, experience, or knowledge can identify questionable assumptions and suggest refinements. Thus, decision models have a built-in tendency to get better. This is why scientists claim to "stand on the shoulders of the giants who came before them." The key to progress is defining explicit and rigorously logical relationships that can be quantified with estimates and that can be tested and refined over time. Scorecards and Strategic Alignment Are Not Decision Models The majority of currently available, project prioritization and portfolio management software tools provide capability for defining both financial and non-financial metrics. The tools are often based on a balanced scorecard approach. Scoring scales are developed to measure project contribution to various non-financial objectives. The scores are weighted and added to financial metrics to provide a balanced measure of project performance. This aggregate measure is then used to prioritize projects. Scorecards are useful in some contexts, but the way that they are defined means that they cannot properly prioritize projects The following example, taken from a case study used in a Stanford professional development course, illustrates the problems experienced by those who use balanced scorecards to prioritize projects [4]: When Inspire Pharmaceuticals, a North Carolina-based biotech company, first started managing its pipeline of products for diseases of the eyes and lungs, it used an approach familiar to many: the Balanced Scorecard. Eight key criteria were ranked by importance on a scale of 0 to 10. Each
69
project in the pipeline -- discovery, development, or commercialization -- was classified by how well it met these key criteria. With a potential range from 0 to 100, in theory this approach could provide guidance as to which projects to fund and which to drop. In reality, though, the top ranking project differed from the bottom ranking project by just 9 points on the 100-point scale, and nothing scored higher than 35. Not only that, but the projects at the top of the list were in late-stage development, and future expenses to commercialize were likely to be high. The projects at the bottom of the list were found to be those in early research; did it make sense to kill those low-scoring projects when the investment at this stage was so minimal? Hardly the clear insight managers were seeking. According to the case study, when Inspire replaced their scorecard priority system with a value-based approach using the same methods described in this paper, they were able to obtain logical and meaningful prioritizations of project alternatives. Scorecard Problems There are four problems for using balanced scorecards to prioritize projects. First, and most fundamentally, the goal for selecting projects should be maximizing value, not creating a balanced project portfolio. Basic scorecard guidance advises that, "The measures represent a balance between external measures for shareholders and customers and internal measures of critical business processes, innovation, and learning for growth" [5]. Assigning weights to measures of this type implies a willingness to accept lower performance in one area (e.g., lower performance for shareholders) in return for better performance in another area (e.g., better business processes). Why would an organization want to accept less value (e.g. lower shareholder value) in order to obtain a higher score (i.e. better "balance") on some internal business process? Value maximization, and not balance, is the goal. The second problem is that, contrary to typical scorecard mathematics, it is generally not correct to weight objectives that represent means for achieving more fundamental objectives. For example, suppose we include scorecards for measuring the impacts of a project on the quality of business processes as well as scorecards for the impacts on operating costs, customer service, etc. But, improving business processes is merely a means for achieving more fundamental objectives, including reducing operating costs and improving customer satisfaction, etc. Thus, a project might get a favorable score on process improvement, but zero weight should be assigned to this score if the value of that process improvement is completely reflected in the scores assigned to metrics that represent the fundamental objectives that explain why business process improvements are important. If the weight is not zero, there will be double counting. Failure to account for the hierarchical nature of objectives (including the fact that means objectives contribute to the achievement of fundamental objectives) is a serious error being made by many who are designing tools for project portfolio management. For example, several websites advise, There are four goals for portfolio management, value maximization, balance, strategic direction and the right number of projects. There is only one goal, value maximization. The proper balance, strategic direction and number of projects are whatever is required to maximize value. A decision model shows how lowerlevel objectives relate and contribute to the achievement of higher-level objectives (Figure 22 provided
70
level objectives that enable the top-level objective (value maximization) to be achieved. A third problem is more basicScorecards do not implement any defensible calculus for project valuation. Contrary to the weight-and-add scorecard approach, it is generally not correct to add different types of value. This statement, which is well established by value measurement theories such as multi-attribute utility analysis, often comes as a surprise to people accustomed to adding and subtracting money values. In fact, being able to weight and add sources of values is an exception. It requires the condition in which the value of achieving any level of performance on any one objective does not depend on the degree to which any other objective is achieved, a condition sometimes referred to as preferential independence. Multi-attribute utility analysis provides tests for determining whether different types of values may be added, or whether more complicated aggregation equations are needed. Scoring methods are being advocated that involve weighting and adding scores for criteria such as project risk, internal rate of return, time-to-complete, urgency, and many other criteria that fail to pass the test of preferential independence. It makes no sense, for example, to weight and add a project's score for time-to-complete to weighted scores for other criteria that indicate the value added once the project is completed. Being quick is more valuable if the project adds a lot of value than if the project adds little or no value. Weight-and-add could only make sense, in this case, if the weights are not constants; that is, if the weight assigned to time-to-complete is a function of the ultimate value of the project. A sound decision model addresses these issues by specifying a logically correct way of quantifying value. Prioritizing projects using a balanced scorecard approach will distort project decisions unless the weights and mathematical form of the aggregation equation are derived consistent with a defensible theory of valuation. Strategic AlignmentThe Most Popular Scorecard Approach Many providers of project portfolio management software and even some of the recent books on project portfolio management promote an easy way to prioritize projects: Prioritize projects based on alignment with strategy. If your budget doesnt have room for the high-end software, you can always fall back on alignment as a sound way to rank your projects. The approach needs to be really simple, so that small projects can come up with a strategic alignment rating in about 5-10 minutes. [6] Strategic alignment is probably the most popular scoring method being used today by organizations to rank projects. Strategic alignment sounds great in principle. An organizations projects should be consistent with and implement its strategy. But, how do you measure alignment? This is where it gets shaky. Typical advice goes something like this [7]: 1. Identify the elements of your organizations strategy, for example, build brand value, invest in core technologies, build organizational effectiveness, and revitalize quality. 2. Rate each project based on the degree of impact you expect it to have on each strategy. 3. Add the individual ratings to provide an overall strategy score. 4. Combine this score with ratings on other criteria, such as net present value or risk.
71
This approach has all of the flaws identified typical of most scorecard approaches, plus some additional problems: Given the subjectivity involved, it would be hard to get consistent and repeatable scores from different people for something like the degree of impact a project has on a vaguely defined concept like revitalize quality. How do you combine an alignment score expressed in points with a financial metric like NPV, which is expressed in dollars? Isnt the goal of strategic alignment to get the portfolio of projects that best implements strategy? The scoring approach does something different: It favors projects that impact the most elements of strategy. Suppose there are individual projects that are each extremely well-aligned with a single element of strategy. Weighted scoring will not rank these projects highly (if there are other projects that have even a mediocre impact on many elements of strategy). Thus, strategic alignment does not produce the portfolio of projects that best promotes strategy. How do we know that the organizations expressed strategy is optimal? How do we know that choosing projects that align with the strategy will produce the best outcomes for the organization? The fundamental problem is that strategic alignment is not a decision model. To review, a decision model is an analytic model that generates as output a measure of the value of a project; that is a number with the property that Project A is preferred to Project B if and only if the computed measure is higher for Project A than it is for Project B. We can easily show that strategy alignment is not a decision model by applying the four steps of the scientific method: 1. Theory: Is there any theory to support the idea that scoring projects based on alignment with strategy provides a measure of the value of those projects? You can find discussions of project valuation theories in many management science books and professional, peer-reviewed papers. Strategic alignment is not considered a theory of valuation. 2. Hypothesis: Suppose we adopt the tentative hypothesis that projects that align the most closely to the most elements of strategy will also provide the most value to the organization. 3. Thought experiments: It is easy to debunk this hypothesis through thought experiments. Suppose the projects that align most closely with strategy are not very effective at addressing needs, produce high risks, or cost too much. Conducting such projects is not likely to provide the most value for the organization. 4. Empirical evidence: Is there empirical evidence indicating that strategic alignment produces high-value project portfolios? I begin my training seminars on project portfolio management by asking, Of those of you who are currently using a scoring tool to prioritize projects, how many think the recommendations being made by your tool are wrong? On average, about 60% of the attendees raise their hands. Most users have only recently put their tools in place, so expect this number to go up.
72
Project Valuation Metrics

Once you have a project-selection decision model, it is easy to specify metrics. The desired metrics are "observables" (discussed below) that influence the model's value drivers; that is, those project characteristics and impacts (i.e., model parameters) that have the greatest influence on value. These typically include forward-looking financial metrics, like NPV, but also factors and considerations on value paths that don't directly impact cash flows. Building a decision model leads to metrics that capture the variety of ways that projects contribute value. In my experience, a well-designed decision model will identify metrics for measuring some or all of the following types of value: Financial value. Metrics are needed to account for any increases in revenues or reductions in costs that may result from conducting a project. Typically, the appropriate financial metrics are the incremental cash flows attributable to conducting the project, or standard financial metrics that are derived from such cash flows (see the subsection below for more discussion). Revenue generated from new products is often estimated with the aid of sub-models that simulate the various development and commercialization stages, in which case metrics indicating the likelihood success at each stage are also included. Health, safety, and environmental (HS&E) value. If some projects impact the health and safety of workers or the public, or the natural environment, metrics may be needed to account for such impacts. The field of risk assessment provides many well-established metrics for this purpose. Typically, the desired metrics indicate the scope, nature, likelihood, and seriousness of the health, safety, or environmental impacts. Customer value. For organizations that sell or otherwise provide products and services, metrics for estimating the impacts of projects on customers are needed. The field of economics provides various metrics for measuring customer value, including the concept of consumer surplus. The field of market analysis has developed numerous customer-satisfaction metrics. Another common approach is to adopt metrics that describe specific product or service characteristics that customers care about (e.g., attributes of product and service quality and price). Stakeholder value. In additional to customers, organizations typically have other stakeholders whose attitudes and perceptions are important and should be managed. Examples include organized labor, regulators, business partners, etc. The relevant metrics typically depend on the type of stakeholder and ways in which the attitudes of those stakeholders are typically expressed or impact the interests of the organization. Oftentimes, metrics for stakeholder value consist of those metrics needed to identify and characterize the importance of the concerned stakeholder groups and metrics that measure the anticipated reactions of those groups. Mission value. Public sector organizations arent the only ones that have missions. Many private sector organizations have adopted mission statements that identify goals beyond maximizing shareholder value. The appropriate metrics in this case measure the contribution of proposed projects to the achievement of these goals.
73
Community socio-economic quality. Some organizations conduct projects that significantly impact local communities. For example, a project to build a new manufacturing plant might have a significant impact on community jobs. The fields of sociology and economics provide numerous metrics potentially useful for such situations. Option value. Consistent with the theory of real options, projects that contribute to the organizations platform for future success provide a source of value. For example, an IT (information technology) project may give the organization new capability. An R&D project may provide new knowledge or understanding important to the business. Also, a project may have a distinct, strategic value. Metrics may be defined that capture these additional sources of value. The key is not to double count. As suggested by the above, numerous metrics may be needed to capture all of the potential components of project value. If metrics representing some sources of value are omitted, the value of projects will be underestimated. Furthermore, there will be a bias against doing those projects that provide the types of value that are not captured due to the omitted metrics. Note that metrics need to represent timing, that is, when the project benefits are likely to occur and how long they will persist, and, oftentimes, risks (e.g., the likelihood that the project will actually produce its anticipated benefits). When all such metrics are specified, the decision model defines the algorithm that allows the value of a project to be expressed in dollar terms. Obviously, there is a limit to how many project evaluation metrics should be used. The 80/20 rule applies. The goal should be to include the minimum number of metrics necessary to roughly capture every significant source of project value, not numerous metrics that more completely capture just a subset of components of value. In other words, dont make the mistake of defining multiple, potentially redundant metrics for capturing things that are relatively easy to address (like financial value), while omitting metrics for something that may be important but hard to address (like impact on learning and capability). Since few if any projects will provide significant contributions under each type of value, having lots of metrics doesnt necessarily create a significant burden for evaluating proposed projects. Estimates need only be provided for the subset of metrics that are relevant to capturing the specific motivations for doing that project. Metrics as "Observables" and the Clairvoyant Test To the extent possible, metrics should be observables; that is, characteristics of projects or project outcomes that can be observed and measured in the real world. Since estimating project value requires forecasting the future, metrics don't, obviously, all have to be things we can observe today. Metrics can, for example, include a projected future state of some observable, for example, an improvement in a reliability-of-service statistic important to customer satisfaction. A useful device for checking whether a metric is observable is the so-called "clairvoyant test" devised by my college mentor, Professor Ron Howard. Before accepting what appears to be a good metric, consider whether a clairvoyant could give an unequivocal value for that metric given that a project decision is made in a specific way. Oftentimes, the clairvoyant test points out inexactness of what initially appears to be a well-defined metric. For example, "customer satisfaction" doesn't pass the clairvoyant test. However, "percent reduction in
Clairvoyant test
74
recorded customer complaints" and "company ranking in the next industry customer satisfaction survey" are metrics that do pass the test. Metrics that don't pass the clairvoyant test are vague. They create inconsistency and imprecision when used for estimating. More importantly, if the metrics are not observables, they cannot be monitored so that actual values can be compared against estimates. Financial Metrics The traditional financial metrics should be used to determine the direct financial components of project value. Project investment cost is, of course, an important financial metric for any project. Projects that impact operations (e.g., projects that create new revenues or that affect future operating costs) produce downstream financial impacts that must also be evaluated. Thus, any and all significant, incremental, period-by-period cash flows that are anticipated to result from projects should be estimated, either as a most-likely or average case or in the form of alternative scenarios. The organization's standard accounting model may then be used to determine the resulting after tax, or unencumbered "free" cash flows, which may be used to compute a project's financial NPV. Some important principles for estimating financial value in support of project prioritization include: Ignore previously paid, sunk costs. Include opportunity costs (the opportunity cost of a resource is the value of the net cash flow that could be derived from it if it were put to its best alternative use). Include overhead expenses (e.g., administrative expenses, managerial salaries, legal expenses, and rent) that are directly related to a project. Indirect overhead can, if necessary, be prorated across proposed projects. Include "spill over" effects. For example, if a project introduces a new product or service that draws sales from existing products, include such lost revenue in cash flow estimates. Interpret expected project cash flows submitted in support of a project proposal as commitments to be achieved by the project manager. If there are cash flow components that are more speculative or for which the project manager cannot be held accountable (e.g., because they are contingent on events beyond the control of the project manager), specify such cash flows separately and assign probabilities. Identify and include any terminal cash flows, for example, cash flows expected from the disposal of assets at the conclusion of the relevant product or service lifecycle. Be consistent in accounting for inflation. For example, using an inflation-adjusted discount rate while ignoring inflation in estimating cash flows would result in a bias against accepting projects. For the purposes of prioritizing projects, remember that the project's financial benefit is its NPV exclusive of its current-period costs.
Be suspicious of long-term, positive NPVs. Keep in mind the economic axiom that excess profits (the source of positive NPV) must be zero in a perfectly competitive market. A long-term, positive NPV requires some sustainable competitive edgebeing first, being the best, or being the only. Retaining
75
that edge indefinitely would require a barrier to the entry of competitors. Consider carefully how long it will take competitors to catch up and drive profits back down. Each Organization Needs Its Own Metrics Different organizations conduct different types of projects. The metrics for evaluating new product investments by a software vendor, for example, will be different than the metrics needed to evaluate process improvements for a company operating an oil pipeline. Also, different organizations create value in different ways. An electric utility, for example, creates value differently than does a ballet school. Some organizations will seek to maximize shareholder value, while others will want to value impacts to other stakeholders as well. Thus, each organization will have a different model for how its projects create value and, therefore, will want to use different metrics. There is no one set of project metrics that works for every organization. However, in all cases, good metrics provide a means for computing the value added by projects. Good metrics are observables. And, they are sensitive to project decisions so that they may be used to differentiate the value of alternative project portfolios. Metrics Provide Justification for Tough Choices One of the most under-appreciated benefits of having good metrics linked to a defensible decision model is improved justification for decisions. Author Anthony O'Donnell quotes a portfolio manager at an insurance company that implemented a portfolio management tool: "People would come to me and ask me to do a particular project...I would tell them I couldn't fit it in, but had a hard time articulating why." Metrics now allow him to give concrete reasons for turning away projects. "Their satisfaction immediately went up, and I still didn't do their projects![8]. The Right Metrics Turn Project Proposals into Performance Contracts Deriving metrics from a decision model ensures that the organization is seeking the right information about proposed projects; namely, the information necessary to estimate the value to be derived by the organization if the project is conducted. The value of the project can then be compared with its cost, and the resulting bang for the buck compared with similar estimates for other candidate projects. This provides a sound basis for making project-selection decisions. If, in addition, the metrics are observables, the organization further benefits in that project proposals serve as performance contracts. In return for a chance at obtaining a share of the organizations limited resources, project proponents indicate in the clearest and most relevant terms what they expect the project will accomplish. Project results and impacts can then be tracked and compared with the original estimates. Performance contracts document the terms of the agreement, protecting both parties to the contract. Framing the project as a performance contract creates a healthy shift in perspective. Instead of choosing which projects to cut, the focus is on deciding what project opportunities to purchase. Due to uncertainty, project outcomes may not exactly match forecasts. Thus, what the implicit contract requires is not that project managers invariably be held responsible for achieving all of the performance indicated by their estimates, but that any significant deviations between estimates and actuals be explained. Over time and on average, some projects should exceed expectations while others will fall short. In the meantime, the organization can learn to improve forecasts by tracking and better understanding the uncertainties that are involved. In situations where the uncertainty is considerable, it may be useful to separate metrics for indicating the
76
benefits that can be expected from benefits that are more speculative and uncertain. The expected benefits then become the basis for the performance contract and the speculative benefits can be appropriately discounted based on risk (see Part 4).
77

1. E. Goldratt, The Haystack Syndrome, North River Press, 1991. 2. R. Foti, Priority Decisions, PM Network, 16, April, pp. 24-29. 3. The recommendation to develop a decision model and, more generally, the views and ideas expressed in this and the next part of this paper are shared by many decision analysts. See especially "Choosing the Right Metrics for Measuring, Monitoring, and Maximizing Shareholder Value," C. Spetzler and R. Arnold, www.sdg.com, May 2003. The book Value Focused Thinking by R. Keeney describes many of the concepts and techniques for building decision models. 4. Stanford Center for Professional Development, How Value-Based Management Delivers Vastly Improved Portfolio Results, Strategic Decision and Risk Management Webinar XSDR001, Stanford University, April 1 2009 5. R. Kaplan and D. Norton, The Balanced Scorecard, Harvard Business School Press, 1996. 6. These quotes were found by searching strategic alignment on Google, and were taken from a Project Management Institute project portfolio management seminar brochure, an article entitled From Crisis to Control: New Standards for Project Management, and a presentation from a consulting firm. You can find similar statements all over the web. 7. This description of how to rank projects based on strategic alignment is from pages 140-143 of Chapter 4.1: Linking Strategy and Project Portfolio Management, by K. C. Yelin, in H. Levines book Project Portfolio Management, Wiley, 2005. 8. ODonnell, "Worth the Effort," Insurance Technology, March 4, 20.
78
Choosing the Wrong Portfolio of Projects Part 4: Inattention to Risk
Inattention to Risk
The crises in financial markets, terrorism, political upheavals, weather-related disasters, court liability rulings, and other causes are forcing organizations to give risk management more attention. Yet, while nearly all organizations are focusing more on security, quality assurance, liquidity, and insurance, when it comes to selecting projects many still don't adequately address risk. Inattention to risk is the fourth reason organizations choose the wrong projects.
79
Risk and Risk Management

There are important reasons why more attention to project risk is needed. The increasingly competitive economic environment is putting ever more pressure on managers to produce results quickly. Meanwhile, projects are becoming more complex due, for example, to new technologies, more regulatory requirements, increased product liability, and the greater dependencies organizations have with external business partners. Finally, uncertainty in world markets and government interventions create external risks that can doom an otherwise sound project. Organizations are being held to higher standards by shareholders, customers, regulators, and the public. Executives are much less tolerant of budget overruns and inferior project outcomes. A major project mishap can seriously damage the reputation and profitability of the organization. Coming in on time, on budget, and to project specifications is no longer good enough. What is Risk? The first step toward better addressing risk is to understand it. Risk, according to Webster, is "a possibility of loss." Risks arise from uncertainty, our inability to foresee the future. If an uncertainty creates the potential for loss, we refer to it as a risk. The opportunity to quantify risk is provided by the language of probability. A probability distribution (sometimes called a risk profile) characterizes a risk by describing the range of possible consequences and their probabilities of occurrence, as shown in Figure 25.
Figure 25: Risk is quantified by providing a probability distribution over possible consequences.
Risk is not an additive propertythe risk of a portfolio is not the sum or average of the risks of the individual projects within the portfolio. In the case of projects, like financial investments, portfolio risk is determined by the underlying statistical relationships among the uncertainties that contribute. If these underlying statistical relationships are identified and modeled, they can be exploited to find optimal
80
risk-based tradeoffs. Conversely, if they are ignored, large risks may be masked and opportunities to avoid them missed. Types of Risk The most common project risks are: Cost risk, typically escalation of project costs due to poor cost estimating accuracy and scope creep. Schedule risk, the risk that activities will take longer than expected. Slippages in schedule typically increase costs and, also, delays the receipt of project benefits, with a possible loss of competitive advantage. Performance risk, the risk that the project will fail to produce results consistent with project specifications.
There are many other types of risks of concern to projects. These risks can result in cost, schedule, or performance problems and create other types of adverse consequences for the organization. For example: Governance risk relates to board and management performance with regard to ethics, community stewardship, and company reputation. Strategic risks result from errors in strategy, such as choosing a technology that can't be made to work. Operational risk includes risks from poor implementation and process problems such as procurement, production, and distribution. Market risks include competition, foreign exchange, commodity markets, and interest rate risk, as well as liquidity and credit risks. Legal risks arise from legal and regulatory obligations, including contract risks and litigation brought against the organization. Risks associated with external hazards, including storms, floods, and earthquakes; vandalism, sabotage, and terrorism; labor strikes; and civil unrest.
As indicated by these examples, project risks include both internal risks associated with successfully completing each stage of the project, plus risks that are beyond the control of the project team. These latter types include external risks that arise from outside the organization but affect the ultimate value to be derived from the project. In all cases, the seriousness of the risk depends on the nature and magnitude of the possible end consequences and their probabilities. In addition to project risk, project deferral risk can be important. Project deferral risk refers to the risks associated with failing to do a project. Like project risk, project deferral risk can arise from any of the bulleted risk sources listed above (the second list). Project deferral risk can also occur if there is only a limited window of opportunity to conduct a projectif the project is not conducted now, there may be a risk that it might never be possible to effectively do it later. Oftentimes, external risks contribute more to portfolio risk because they impact multiple projects simultaneously. For example, a pharmaceutical companys R&D project is affected by the uncertain
81
outcomes surrounding the specific compound involved, however many projects could be impacted by a change in regulations. Similarly, a petroleum firms exploration project depends on uncertainty over whether oil is present at the given location, but uncertainties over the market price of oil affect many projects. Likewise, a construction company might have many projects threatened by the external risk of an increase in steel or concrete commodity prices. Project Risk Management Project risk management, as defined by Max Wideman, is "an organized assessment and control of project risks." Figure 26 shows the general, 3-step approach to risk management. Step 1 is to identify the risk. Empirical data, recent events, and new regulations (which often signal regulator concern over new risks) are inputs to the risk identification process, and brainstorming and risk scenarios (see below) are examples of techniques that can be used to define and clarify risks. Step 2 is to analyze the risks, which means characterizing risk in terms of likelihood and consequences. Step 3 is to manage the risk, taking into account the resources available and the organization's willingness and ability to accept risk.
Figure 26: The basic steps of risk management.
The appropriate level of detail required for risk management depends, obviously, on the level of risk. Riskier projects, such as new product launches, global initiatives, projects involving new technology, some major regulatory-driven projects, and so forth, tend to have complex interacting elements and involve high stakes. A poor track record on similar projects is an indicator or risk. Likewise, more attention to risk is required when there is project deferral risk. Such situations arise for organizations responsible for managing systems whose failure might produce serious, large-scale health, safety, environmental or financial consequences (such as a large electric transmission network or an oil refinery). While sophisticated risk management is most needed for the most risky project environments, some level of project risk management should be provided in all cases.
82
An organization can practice risk management in several different contexts. Projects are proposed throughout the organization in response to perceived needs, threats, and opportunities. Sometimes, the identified need is reducing a risk. For example, an organization operating a hazardous facility may invest in projects to reduce health, safety, and environmental risks. In such cases, the project is itself an investment in risk management (in which case there may be project deferral risk). Regardless of the need or opportunity the project is intended to address, there are three main contexts for project risk management. As shown in Figure 27, these are project planning, project selection, and project execution.
Figure 27: Opportunities for risk management.
Many organizations have instituted risk management processes within project planning and project execution. However, risk management in project selection is often little more than a yes/no answer to "Should we accept the risk?" Risk is often viewed as an "intangible" and described using qualitative terms such as "likely" versus "unlikely" and "significant" versus "insignificant." Such words are insufficiently precise and mean different things to different people. For example, a lower-level manager might have a very different notion of what qualifies as a significant risk compared to that of the CEO. Failure to describe and understand project risk and project deferral risk, coupled with project-by-project decision-making, creates problems for risk management.
83
Project Selection Risk

Important types of risk are best addressed by project selection because they are outside the scope of project managers. A paper by the accounting firm Ernst & Young provides this example [1]: A company conducted a project to install new equipment to increase capacity. However, the project planning team failed to evaluate whether the market could absorb the increased supply made available by the added capacity. Narrowly defined, the project was a success because the new equipment was installed successfully, on time and on budget. However, because there was insufficient demand, the company could not sell its extra output at its prevailing price. It ultimately had to shut down some of its production lines. As illustrated, risk management within project planning and project execution often fails to address external risks. Project portfolio management provides an opportunity to account for external risks and to get senior executives to take some ownership of project risks before the project commences, and understanding of project deferral risk before making the decision to defer the project.. Risky Projects May Be Good Projects For many organizations project risk is simply something to be avoided. But, as Alan Greenspan has stated, "Risk-taking is indeed a necessary condition for the creation of wealth" [2]. Successful organizations deliberately take risks when it is to their advantage. According to Suzanne Labarge, Vice Chairman of the Royal Bank of Canada, "Risk in itself is not bad. What is bad is risk that is mismanaged, misunderstood, mis-priced, or unintended" [3]. Failure to recognize, understand and accept risk often leads to project portfolios skewed toward low-risk projects with little upside potential. It can also lead to an occasional, unrecognized, high-risk project that endangers the enterprise. Characterizing Risks The first step to managing project-selection risk is to identify and characterize the risks associated with doing and not doing projects. Typically, a risk involves a source or cause (e.g., some possible event), a mechanism by which the risk source could impact the objectives of the organization, and some level and type of potentially adverse consequences. A risk can only be evaluated in relation to the objectives that it impacts. Thus, characterizing risks requires understanding the objectives that are threatened by the risk. Part 3 of this paper argued that the fundamental objectives of organizations are creating shareholder and stakeholder value, and that these overall objectives can only be achieved if certain sub-objectives are achieved. The relevant subobjectives depend on the organization (as described in Part 3, sub-objectives, along with appropriate metrics, are identified by creating a project decision model). Important sub-objectives typically include some or all of those shown in Figure 28.
84
Figure 28: (Sub)-objectives (relevant to value creation) that may be impacted by risk.
Risks should be identified and documented. Many tools for project portfolio management provide templates for this purpose. A statement of risk should encompass the cause and impact to the objectives which might occur. For example, a project cost risk might arise if there is some question about whether the project team has all of the necessary skills for some task. Skills that do not match the job, therefore, is a potential source of risk. Cost, obviously, is the objective potentially impacted. If other objectives might also be impacted (e.g., customer satisfaction) then those objectives should likewise be identified and an explanation provided for how and what sort of impacts might occur. Characterizing the risk includes providing an indication of the nature and magnitude of the potential impact on objectives (e.g., costs could increase by $X, customer Y would be dissatisfied as a result of a two-week delay in obtaining service). Risks should be identified with a level of detail where a specific impact can be identified and specific actions to address the risk can be considered. If the risk is identified and understood, ideas may be generated for modifying the project plan to mitigate the risk. Also, identifying project risks provides a database of project risks useful when considering similar projects in the future. Scenario Building One of the simplest tools for exploring risk is scenario building (Figure 29), a technique originated by the military. In the context of understanding project-selection risk, scenario building involves hypothesizing plausible events or futures that significantly impact the value or success of a project or set of projects. Envisioning the scenarios as mental movies helps to stimulate thinking. Not only does scenario planning help uncover real possibilities, it encourages managers to come up with ways of avoiding potential disasters and ensuring that things turn out reasonably well regardless of which future actually takes place. Large oil companies have long been big users of scenario building. The popularity is often attributed to an early success. In the 1970s, a planning group at Shell Oil generated scenarios that could affect the price of oil, an uncertainty important to many company projects. One scenario was that prices would remain stable. Another was that OPEC would demand much higher prices. As the latter scenario was developed, it became increasingly clear to the team that the scenario was not just plausible, it was highly likely. However, when the team warned upper management, no changes in company decisions could be observed. So, the team went one step further. They described the logical ramifications of the scenario in terms that leadership would understandit meant slow growth for the industry and the possibility that OPEC countries would take over Shells oil fields. When the Arab oil embargo did occur in 1973, only Shell was reasonably prepared. To manage risks highlighted by the scenario, the company had slowed refinery expansions and adapted their refineries to better accommodate alternative types of crude
85
oil.
Figure 29: Scenario building.
Qualitative Methods for Addressing Project Risk The project risks that remain after project plan based on project risk-management the residual risks that must be considered when selecting projects. For most organizations, factoring risk into project portfolio management involves merely flagging project risks so that they may be considered when decisions to select or reject projects are made. Typically, qualitative methods are used. For example, identified risks might be categorized as high, medium, or low with respect to likelihood and consequence. As shown in Figure 30, this results in a 3x3 risk matrix defining potential responses. Color codes (e.g., green, yellow, and red traffic lights) are often used to help summarize risk judgments.
Figure 30: Risk scoring checklist and risk matrix.
86
Qualitative methods may be adequate for screening purposes or in situations where risks are not very serious. The limitation of qualitative methods is that they provide no help for determining the impact of risk on project or portfolio value. In many situations, risk can have a major impact on the value of projects and project portfolios. Using analysis to optimize the project portfolio requires employing quantitative methods to assess and value risk. Quantitative Methods for Valuing Risk There are two alternative approaches for valuing risk: 1. Compute the decrement (or increment) to the worth of the project caused by project and projectdeferral risk. 2. Adjust the discount rate (cost of capital) to reflect the specific level of project risk for each project. The former approach, which requires that risks be quantified using probabilities, is described in the next two subsections of this paper. The latter approach involves using project hurdle rates. Hurdle rates are simpler, but as explained below, they have serious limitations. Be Careful Using Hurdle Rates Most organizations that account for risks when valuing projects do so using hurdle rates. The hurdle rate is a risk-adjusted cost of capital used to discount future project costs and benefits. Increased hurdle rates are applied to projects considered to be more risky. Using hurdle rates may be preferable to ignoring risk or treating it as an intangible. However, hurdle rates have limitations. For one thing, hurdle rates only address project risk, they cant account for project-deferral risk. Also, organizations are frequently unclear about what hurdle rate should be applied based on project risk. Studies have shown that the rates used by firms vary considerably. According to finance theory, the "correct" hurdle rate is the "opportunity cost" of the investment, which is the return available from investing in securities equivalent to the risk of the project being evaluated. Most companies don't adjust the hurdle rate for risk nearly enough. A more fundamental problem is reflected in research on real options showing that the discount rate needs to vary with the project management strategy (e.g., an irreversible project investment would call for a higher hurdle rate), as well as with time (the discount rate is not a constant, but changes depending on when the future discounted outcomes are expected to occur), and with changing information. Using a constant hurdle rate for a project implicitly assumes that uncertainty increases over time in a specific way (geometrically). Hurdle rates tend to create a bias toward short-term, quick-payoff projects because they severely penalize project benefits that occur in the longer term.
87
Characterizing Risks with Probabilities

The best way to understand project risk or project deferral risk is to characterize the risk by describing the range of possible outcomes, estimating when they could occur (risk timing), and assigning probabilities. If relevant data are available (e.g., as might be the case for system failure probabilities for evaluating reliability maintenance projects), probabilities for characterizing risks can be derived using statistical analysis. In the absence of such data, probabilities must still be assigned, and it makes sense to do so directly based on professional judgment. Although quantifying risks requires more inputs to describe proposed projects, note that the additional inputs need not be very complex. If some aspect of a project's performance is uncertain, instead of obtaining only a middle-value, point estimate, get a range of possible values (e.g., a 90% confidence interval) as well as a mean or most-likely value. (As I described in Part 1, techniques should be used to guard against overly narrow ranges caused by overconfidence.) With practice, it takes no more time to specify a range than it does to generate a single point estimate. The necessary probabilities can be roughly estimated from the range and a mean or most-likely value. The easiest uncertainties to quantify are those associated with random events whose mean rates of occurrence can be measured, such as weather, accident rates, and commodity prices. Limiting risk quantification in this way, however, can create a sense of lack of urgency. For example, a week before hurricane Katrina struck, New Orleans hosted an offshore-drilling conference that included a panel discussion entitled, What Has the Industry Learned From Ivan (Hurricane Ivan had struck the previous September). The lesson was that rigs needed to be much better secured. The industry, however, had not yet made any changes prior to Katrina. The engineering approach to risk assessment told them that hurricanes of this size occur infrequently, which led them to believe that there was plenty of time before the next major storm occurred. Based on best-professional judgment, probabilities can be assigned to uncertainties for which no frequency data exists. If you can imagine an event, you can assign a probability to it, if not in absolute terms0.01%, 1%, 10% then relative to another event whose probability can be measured. (Before you dismiss the predictive value of subjective probabilities, read the section in Part 7 on predictive markets). Libraries of pre-generated probability distributions can be created to describe commonly encountered risks. For example, Chevron is developing a library of accident probability distributions for analyzing capital investment projects at oil terminals, and Shell has developed a library of distributions for hydrocarbon volumes for oil exploration projects. Generating such probability libraries helps ensure consistency and facilitates auditing the assignments over time. The cover story for an issue of the Journal ORMS Today argued that companies need to consider the prospect of a Chief Probability Officer, someone with the job of managing the probabilities distributions assigned to support project portfolio management [4].
88
Once probabilities have been assigned, mathematical reasoning can be used to avoid many of the errors and biases described in Part 1 of this paper. For example, you can calculate how the number of observations affects the accuracy of estimates (to avoid small sample bias) and how the conditions required for an event affect the events probability (to avoid conjunctive bias). Also, a method known as Bayes Theorem can be used to calculate how a probability should be revised or updated as new information becomes available. The amount of uncertainty caused by project risks and the specific project outcomes that are impacted may be used to better estimate what hurdle rates should be used and the types of benefits to which they should be applied. Quantifying uncertainty also allows more sophisticated methods (such as risk tolerance, explained in the next sub-section of this paper) to be employed to account for risk aversion. Project Managers Benefit from Using Probabilities to Describe Uncertainties Although project managers may initially feel uncomfortable with probabilities, my experience is that this group can benefit significantly from moving away from using artificial point estimates. The following is a summary of an example devised by Mark Durrenberger of Oak Associates making this point [5]. Imagine that a project manager is asked to complete a project in 3 weeks. Suppose the project manager feels that this estimate is unrealistically optimistic; that everything would have to go just right to make the deadline. The project manager may feel apprehensive about going to the project sponsor to address the problem. It may not be easy to explain why an optimistic, aggressive project schedule isn't a good one. Suppose, instead, that the project manager estimates as a range the time required to complete each project step. Those ranges can be combined (by adding the means and variances) to determine the probability of completing the effort within any specified time interval. Rather than feeling "at the mercy" of the sponsor, the project manager can now say, "I understand your desire to complete the project within three weeks. However, my calculations suggest that we have less than a 5% chance of meeting that deadline." The sponsor will want to know more, including how the probability estimate was obtained. This gives the project manager the opportunity to discuss the realities of the job and to negotiate tradeoffs (like providing more resources or eliminating some project deliverables so as to increase the likelihood of meeting the desired schedule). Note that specifying probability ranges is not a license for a project manager to make baseless claims. Over time, performance can be compared with range estimates. A project manager whose performance routinely beats the means of his specified uncertainty ranges, for example, will be exposed as one who pads estimates. Quantifying Uncertainty Over Value Importantly, if probabilities have been assigned to the uncertainties associated with key risks, including scenarios, those probabilities can be propagated through the decision model (described in the previous part of this paper) to derive the uncertainty over the various project outcomes and total value of the project. This can be done using Monte Carlo analysis or decision trees.
89
Monte Carlo analysis is a form of simulation for investigating the uncertain behavior of a physical system. Typically, the system is represented by a mathematical model with uncertain parameters or inputs. The uncertainties are described by probabilities. Monte Carlo analysis involves selecting sets of inputs in accordance with the specified probabilities, executing the model, and recording the model output. Since the specific inputs that are selected for any trial are generated randomly (according to the specified probabilities), the process is a little like rolling dice (hence the name). If enough trials are conducted, a frequency plot of the model output shows the shape of the probability distribution over the combined outcome. By applying Monte Carlo analysis to the project decision model, a probability distribution, or risk profile, can be generated (Figure 31). You can do sensitivity analyses wherein only one die is rolled (only one uncertainty is allowed to vary) while keeping the others fixed. In this way, you can see which uncertainties (risks) have the biggest influence on project value (and focus energies accordingly). The results can be used to investigate changes that might be made to reduce risk. For example, if the distribution says theres a 50 percent probability the project will run a month late, you might decide to build an extra month into the schedule. Using the decision model and Monte Carlo analysis, you can generate a risk profile for the value of each of your proposed projects.
Figure 31: Using Monte Carlo analysis to quantify uncertainty in the value delivered by a project.
Decision trees and event trees represent another means for generating probability distributions over project value. Whereas Monte Carlo analysis excels at simulating what happens when many risks and other uncertainties are in play at once, decision trees and event trees are more effective in mapping either-or situations and the sequential uncertainties that follow decisions. As illustrated by the example in Figure 32, a decision tree is a graphic tree structure composed of decision nodes and chance nodes. The order of the nodes (from left to right) in the tree corresponds to sequence in which uncertain information is anticipated to be revealed and decisions must be made. A decision tree without decision nodes is sometimes called an event tree.
90
Figure 32: Decision tree for evaluating a project with uncertainty.
Branches emanating from decision nodes correspond to alternatives available at points of decision, and branches from chance nodes represent the possible outcomes to risks and other uncertainties, with associated probabilities. If the decision model is used to compute a project value corresponding to each path through the tree, the values can be displayed at the trees end points (the probabilities of each end point value are the product of the probabilities along the path to the end point). The tree can then be analyzed to determine the risk profile (specifically, by computing a cumulative probability distribution over value by computing the probabilities of all values less than each possible value). Decision trees can also be used to compute a risk-adjusted value for each project, using the method or risk tolerance described later.
91
An advantage of decision trees and event trees is that they are convenient for exploring how uncertainties evolve over time (as in Figure 33).
Figure 33: Characterizing risks shows how uncertainties evolve over time.
Risks of the Project Portfolio Another important reason to consider quantifying project risks is it that the overall risk of the project portfolio can then be determined. Conducting a portfolio of projects reduces risks through risk diversification (hedging) in the same way that an individual can reduce financial investment risks by investing in a portfolio of diversified stocks. In a stock portfolio there is a limit to how much diversification can reduce risk. This limit is determined by the degree to which stock prices tend to move together; that is, the degree to which the prices of the stocks in the portfolio are statistically dependent or correlated with overall market movements. To understand the risks of a stock portfolio, it is necessary to measure these correlations (this is typically done using the correlation statistic called "beta"). In a project portfolio, as illustrated previously, there are risks (e.g., external risks) that impact multiple projects simultaneously. So, in exactly the same way as with stocks, a project portfolio is not as effective at reducing correlated risks. The only way to estimate accurately the risks of alternative project portfolios, and thereby choose projects that collectively produce maximum value at minimum risk, is to quantify these project risks, including statistical dependencies. Unlike the case for many financial portfolio risks, there is typically no direct way to measure the statistical dependence among the risks of conducting potential projects. However, models can be constructed to represent the relationships. The key is to identify which risks simultaneously affect which projects, and to use the model to appropriately relate the projects to one another. Failure to account for risks that simultaneously impact numerous investments can have devastating consequences. The 2008 financial crisis provides many illustrations. For example, insurance giant American International Group (AIG) used models to assess the risks associated complicated contracts
92
called credit-default swaps, which totaled more than $400 billion. According to an article in the Wall Street Journal, AIG knew their models left out certain market forces and contract terms, but neglected to expand the models [6]. In retrospect, it was clear that the failure to address the common threats caused AIG to vastly underestimate risk and to continue to purchase the dangerous contracts. Were it not for the government bailout, AIG would have collapsed. Fully characterizing project and project-deferral risks shows whether the assumptions required for using hurdle rates are satisfied and supports the selection of project-specific hurdle rates. It also allows the use of another approach involving the concept of risk tolerance.
93
Risk Tolerance
An individuals or companys aversion to risk taking can be quantified and measured. The concept works as follows. If decision makers did not care about risk, they would want to "go with the odds;" that is, they would want to make decisions so as to maximize expected value. The expected value is defined as the probability-weighted sum of the distribution of possible uncertain outcomes. Decision makers unconcerned about risk would want to maximize expected value because the expected value is the amount that they would obtain on average each time the uncertainty is faced. As an example, the expected value of a coin flip that pays $1 on "heads" and zero on "tails" is 50 cents. If you participated in a thousand such coin flips, your winnings would be very close to $500, so that the average value of each flip would be 50 cents. For substantial risks, organizations as well as individuals tend to be risk averse, meaning that they value uncertainties at less than their expected values. The "certain equivalent" is defined as the amount of money for which a decision maker would be indifferent between receiving that amount for certain and receiving the distribution of uncertain payoffs represented by the gamble. For example, a risk-averse decision maker might assign a certain equivalent of $500,000 to a risky project with equal chances of yielding $0 and $2,000,000, even though the expected value for this alternative is $1,000,000. Note that this same logic means that a gamble with negative expected value (large downside risk) has a certain equivalent that is even more negative than its expected value (which is why individuals and organizations are willing to pay more in insurance premiums than the expected loss that they are eliminating). The goal of a risk averse decision maker is to maximize the certain equivalent. For risks with complex payoff distributions, it is generally difficult to determine the certain equivalent. However, the certain equivalent can be estimated for a simple gamble and the results used to infer the certain equivalents of more complicated risks. The approach involves constructing a utility function that represents the degree of aversion to taking risks. Figure 34 illustrates the form (exponential) often chosen for the utility function. The horizontal axis shows possible values or certain equivalents. The vertical axis shows the corresponding utility, where utility is a numerical rating assigned to every possible value. The shape of the utility function determines the degree of aversion to taking risks. The more the plot curves or bends over, the more risk aversion is represented. With the exponential utility function, the degree of curvature is determined by the parameter R, known as the risk tolerance. Thus, risk tolerance is an indicator of a decision makers or organizations willingness to accept risk. Risk tolerance, as defined here, is not the maximum amount that the decision maker can afford to lose, although decision makers and organizations with greater wealth generally have larger risk tolerances.
94
Figure 34: The exponential utility function is often used to model risk aversion.
Once the risk tolerance is set, the utility function may be used to compute a certain equivalent as follows. First, locate each possible payoff x on the horizontal axis and determine the corresponding utility U(x) on the vertical axis. For example, if risk tolerance is $1 million and the risk is 50% chance of $0 or $2 million, the corresponding utilities (from Figure 10) are 0 and approximately 0.8. Second, compute the expected utility by multiplying each utility by its probability and summing the products. For the example, the expected utility is roughly 0.5x0 + 0.5x0.8 = 0.4. Third, locate the expected utility on the vertical axis and determine the corresponding certain equivalent on the horizontal axis. The result for the example is approximately $500,000. There are several ways to determine the risk tolerance for an organization. One is to ask senior decision makers, ideally the CEO, to answer the following hypothetical question. Suppose you have an opportunity to make a risky, but potentially profitable investment. The required investment is an amount R that, for the moment, is unspecified. The investment has a 50-50 chance of success. If it succeeds, it will generate the full amount invested, including the cost of capital, plus that amount again. In other words, the return will be R if the investment is successful. If the investment fails, half the investment will be lost, so the return is minus R/2. Figure 35 illustrates the possible outcomes of the opportunity. As indicated, the expected value of the investment is R/4.
Figure 35: What is the maximum amount (R) you would accept in this gamble?
95
If R were very low, most CEOs would want to make the investment. If R were very large, perhaps close to the market value of the enterprise, most CEOs would not take the investment. The risk tolerance is the amount R for which decision makers would just be indifferent between making and not making the investment. In other words, the risk tolerance is the value of R for which the certain equivalent of the investment is zero. Various studies have been conducted to measure organizational risk tolerances. The results show that risk tolerances obtained from different executives within the same organization vary tremendously. Generally, those lower in the organization have lower risk tolerances. As a rough rule of thumb for publicly traded firms, typical risk tolerances at the CEO or Board level are often equal to about 20% of the organization's market value. Once risk tolerance has been established, the certain equivalent for any risky project or project portfolio can be obtained via the utility function. The effect, as illustrated in Figure 35, is to subtract a risk adjustment factor from the expected value. The risk adjustment factor depends on the risk tolerance and the amount of project risk. If the projects are independent (i.e., their risks are uncorrelated), then the certain equivalent of the project portfolio will be the sum of the certain equivalents of the individual projects. If project risks are correlated, the certain equivalent for the portfolio can be obtained once the distribution of payoffs for the portfolio are computed (accounting for correlations as described above).
Figure 35: Adjusting project value for risk.
An advantage of this approach is that a single risk tolerance can be established for the organization. Use of the common risk tolerance ensures that risks are treated consistently, thus avoiding the common bias in which greater levels of risk aversion tend to be applied by lower-level managers. For a demonstration of the importance in the context of project prioritization of addressing risk and risk tolerance, see the Risk Demo at www.prioritysystem.com\riskdemo.html.
96

1. Project Risk Mitigation: A Holistic Approach to Project Risk Management, Assurance & Advisory Business Services, Ernst & Young, 2002. 2. L. Kahaner and A. Greenspan, The Quotations of Chairman Greenspan, Adams Media Corporation, 2000. 3. S. Labarge, "Valuing the Risk Management Function," Presentation at the Risk Management Association's Capital Management Conference, Washington DC. April 10, 2003. 4. S. Savage, S Scholtes and D. Zweidler, Probability Management, ORMS Today, February 2006, 20-4. 5. M. R. Durrenberger, True Estimates Reduce Project Risk, Oak Associates, Inc., 1999. 6. Behind AIGs Fall, Risk Models Failed to Pass Real-World Test, The Wall Street Journal, p. 1, November 3, 2008.
97
Choosing the Wrong Portfolio of Projects Part 5: Inability to Find the Efficient Frontier
Inability to Find the Efficient Frontier
The goal for selecting projects is to pick project portfolios that create the greatest possible riskadjusted value without exceeding the applicable constraint on available resources. Economists call the set of investments that create the greatest possible value at the least possible cost the "efficient frontier." Most organizations fail to find the best project portfolios and, therefore, do not create maximum value. Inability to find the efficient frontier is the fifth reason organizations choose the wrong projects. If the problems discussed in the previous parts of this series are addressed, value-maximizing project portfolios can be found. Specifically, if the organization is managing the project portfolio, has the right metrics and models in place, including the ability to measure and value risk, and has taken steps to minimize errors and biases in inputs provided to those models, the capability exists to estimate the value that would be created by conducting any proposed project portfolio. It is a relatively easy last step, then, to find the best combination of projects. The concept of the efficient frontier is highly useful in this regard.
98
The Efficient Frontier

Suppose that an organization plots some of its available project portfolios on a graph relating total value to total cost, as shown in Figure 37. Economists would describe Portfolio A as inefficient because there is another project portfolio, Portfolio B, that produces more value for the same cost. Similarly, there is also a Portfolio C that produces the same value for less cost. Furthermore, there is a Portfolio D with a combination of these two characteristics.
Figure 37: Different project portfolios have different costs and values.
Now suppose that we consider all of the alternative portfolios that can be constructed from a set of candidate projects. Typically there are many, as suggested by the realistic example of Figure 38. In this case the organization had 30 project proposals under consideration in one budget cycle. Four of those projects were considered mandatory (3 process fixes and a new initiative required by regulators) leaving 26 discretionary projects.
99
Figure 38: Portfolio value versus cost
In general, if there are N potential projects, there are 2N possible project portfolios because there are a total of 2N subsets within a set of N items. (For more explanation see the paper Mathematics: Methods for Solving the Capital Allocation Problem, available on my website). Thus, this application required evaluating 226 or approximately 67 million portfolios, far more than shown in Figure 38. The best portfolios define the efficient frontier. Portfolios along the curve at the frontier of the plot are said to be efficient because they allow the organization to obtain the greatest possible value from any specified available budget. Finding the Frontier It is relatively easy for a computer, with an efficient optimization engine, to try various combinations of projects and locate the efficient frontier, provided the right algorithms for calculating portfolio value are in place. Essentially, the optimization is run multiple times, each time with a different specified cost constraint. Even though there may be too many possible portfolios for even the fastest computers to try all combinations, approximate methods are available that can ensure sufficient accuracy for practical purposes. The optimization identifies the highest-value portfolio obtained for each cost, and the result is plotted. The curve obtained in this way defines the efficient frontier (Figure 39).
100
Figure 39: The efficient frontier
The Characteristic Curve of the Efficient Frontier Notice how the efficient frontier is curved, not straight. This is because the frontier is made up of the best possible projects, i.e. those portfolios that show up first on the left side of the curve. Such portfolios create the greatest "bang-for-the-buck," and therefore the curve is steepest here. As the cost constraint is relaxed and more projects can be added, the new projects provide less incremental value compared to those included earlier. The slope of the curve encompassing these projects is flatter because the incremental bang-for-the-buck is not quite as high. Thus, there is a declining return in the value obtained with each additional increment of cost. This is what causes the curve to bend as shown in Figures 39. (As described later, however, the efficient frontier will typically not be completely smooth, but will have some bumps in it.) The 80/20 Rule Vilfredo Pareto, an Italian economist, was the first to report what has become recognized as a common rule describing how dissimilar objects are often distributed. Specifically, Pareto observed that approximately 20% of the people owned 80% of the wealth. Since then, a similar relationship has been observed in many other areas, including business contexts, for example, 80% of profits come from 20% of customers, 80% of results come from 20% of the effort, and 80% of the value can be achieved from just 20% of the activities. The relationship is not exact, of course, but it is close in a surprising number of situations. It has become known as the "law of the trivial many and the critical few," or, more simply, as the 80/20 rule. The curvature of the efficient frontier is such that it often corresponds closely to the 80/20 rule. Roughly 80% of the value available from doing all projects may be achieved by doing just 20% of those projects (assuming, of course, that the best projects are chosen). The lesson is similar to other instances where the rule appliesManagers should concentrate on identifying and doing the few things that are critical rather than wasting effort on the many things whose impacts are trivial.
101
The Efficient Frontier Depends on the Quality and Quantity of Project Options The efficient frontier improves if project alternatives improve. Figure 40 shows what happened when project proponents were asked to submit 3 alternative versions for each project proposal (the original or base-version proposal, a minimum cost - reduced scope version, and an enhanced cost - enhanced scope version).
Figure 40: More (and better) project options improve the efficient frontier
Additional project options allow better project portfolios to be constructed. Thus, the efficient frontier moved up and to the left. By adjusting the spending levels for projects, portfolio value was increased by 14%. The Efficient Frontier Moves over Time Regardless of the number of project alternatives analyzed, the efficient frontier tends to improve over time. Organizations continually face the challenge of identifying investment opportunities and finding project alternatives that advance the frontier. As project managers better understand the link between their project designs and the value derived by the organization, they create better project proposals. Also, better technology creates new opportunities that create more value for less cost. This causes the efficient frontier to move up. The fundamental goal, though, remains the samecreate as much value as possible using as little capital as possible. To do this, you must find the efficient frontier.
102
Efficient Frontiers versus Ranking Curves

As indicated in the previous section, generating the efficient frontier requires a software tool with an optimization engine; that is, the ability to solve mathematical optimization problems with constraints. Many project portfolio management tools lack optimization engines and therefore lack the ability to generate efficient frontiers. As an alternative, such tools often create a graph that looks similar to an efficient frontierthey rank projects based on the ratio of project value (or some measure related to project value) to project cost and then plot cumulative value versus cumulative cost while adding projects to the portfolio in rank order. Although many authors erroneously refer to the result as the efficient frontier, it is more accurately referred to as a ranking (or productivity ranking) curve. The reason that the efficient frontier and a ranking curve are often confused is due to the fact that there is an important special case for which the two curves will be very nearly the same. In particular, if projects are independent and risks are either independent or do not matter, the costs and value of the project portfolio are basically just sums of the costs and values of the individual projects that make up the portfolio. In this case, the value-maximizing portfolios can be obtained by ranking projects based on the ratio of project value to project cost, and the efficient frontier can be plotted by adding projects to the portfolio in the order of benefit-to-cost ratios. Figure 41 illustrates.
Figure 41: A project ranking curve.
Although ranking produces portfolios on the efficient frontier for the special case where the choice is among independent projects, in general the curve produced by ranking projects will not be the efficient frontier. As explained below, there are several reasons for this difference.
103
The Ranking Curve Misses Opportunities to Use Unspent Budget At best, a ranking curve will only match the efficient frontier at precisely the cumulative budget levels that result when projects are added to the portfolio in rank order. Ranking fails to identify the portfolios on the efficient frontier that lie between these points (e.g., the portfolio colored red in Figure 42).
Figure 42: The efficient frontier identifies higher-value portfolios not on the ranking curve.
The yellow points in Figure 42 represent portfolios constructed by ranking. If the budget is A, the remaining funds are insufficient to include Project e, so Project d is the last project added to the portfolio. An optimization engine is needed to identify a better project combination (e.g., the portfolio designated by the red point) that comes closer to using the total budget. Such portfolios create more value, typically by replacing one or more higher ranked projects by a combination of lower ranked projects so as to use a greater fraction of the available budget. The error inherent in using the ranking curve rather than the efficient frontier tends to be more significant the more available projects differ in cost. The Ranking Curve Misses Opportunities to Adjust Project Funding Based on Available Budget As shown previously, providing multiple versions of proposed projects (i.e., different levels of project effectiveness based on different project scopes, approaches, etc.) can significantly shift the efficient frontier. Two versions of the same project are not independent project options. For example, if you choose a low-cost version of a project, that decision would obviously reduce the attractiveness of simultaneously funding a higher-cost version of the same project. This violates the basic requirement for ranking, meaning that a simple ranking approach cannot be used to approximate the efficient frontier when there are multiple project options available. Optimization is needed to select project versions that together best utilize the available budget.
104
Because adjusting project funding is often a better option than the all-or-nothing choice implied by project ranking, the efficient frontier generally lies above the ranking curve if there are multiple versions available for projects. Figure 43 provides an illustrative example. In this case, 3 versions are assumed to be available for each project, a low-cost, mid-cost, and high-cost version. Project benefits are assumed to increase with project costs, and the project costs and the incremental costs of moving to more expensive project versions are assumed to be the same for all projects. These assumptions ensure that the efficient frontier will have the sort of smooth curve that is shown in the figure. Notice that higher budgets result in more expensive project versions being selected.
Figure 43: The efficient frontier varies project funding to increase portfolio value (illustrative data).
105
The Ranking Curve Fails to Account for Multiple Resource Constraints A major limitation of the ranking curve compared to the efficient frontier is that the ranking curve cannot account for constraints other than the funding constraint. Only a single cost constraint can be addressed through the ranking approach. If, for example, some projects require funding over multiple budget cycles, the constraints that may exist on funding for the various years cannot be addressed. Furthermore, in such cases, it is not entirely clear, when ranking projects based on the ratio of benefits to costs, what cost should be used in the denominator and what benefits should be used in the numerator. For example, should projects be ranked based on the ratio of total benefit (the benefit generated if the project continues to be funded until completion) to total cost (including the remaining, out-year costs required to secure those benefits)? Or, should projects be ranked based the ratio of total benefits less remaining costs to budget year costs? Or, should projects be ranked based on budget year benefits (the portion of benefits that are attributed to budget-year spending) to budgetyear costs? Each approach will yield a different ranking. With the efficient frontier, additional constraints can be established for the optimization. Thus, for example, the optimization engine can be used to generate efficient frontiers that show the value maximizing project portfolios under alternative budget-year costs subject to various specified constraints on out-year funding as well as subject to constraints on people and other resources needed for projects. Establishing additional constraints to be achieved by the resource allocation will change project recommendations (to ensure that the constraints are met), which will tend to lower the efficient frontier. A simple ranking curve cannot under these circumstances identify the optimal project portfolios.
106
A Real Efficient Frontier

Real efficient frontiers tend to be more complex than the simple examples above. To illustrate, the business units within an organization proposed a total of 30 projects. Budget-year costs ranged from $40,000 to over $8 million. Some of the larger projects were multiyear in duration, and about a third of the projects were proposed in multiple versions (e.g., low cost, base, and enhanced versions). If all projects were funded (with multi-version projects funded at their highest funding levels), a total budget of roughly $33 million would be required. The organizations target budget was $17 million. Obviously, prioritization was necessary. Figure 44 shows the efficient frontier, and Figure 45 shows how the projects included in the portfolios on the frontier depend on the total budget. Although the efficient frontier bends over in the typical fashion, there are the bumps in the curve. The labels in Figure 45 are unreadable, but what is important is the pattern.
Figure 44: A real efficient frontier.
107
Figure 45: Optimal funding decisions for budgets between $1 M and $35 M in $1 M increments.
As in the previous simplified example, the rows in the table of Figure 45 correspond to projects and the columns (left to right) represent increasing funding levels. The colors in the cells indicate funding recommendations, with darker cells indicating recommendations to fund more expensive project versions. Although an increase in the budget is generally associated with an increase in funding for some project, sometimes increasing the budget causes the funding recommendation for a project to be reduced. This is typical behavior for real efficient frontiers. The bumps in the efficient frontier are caused when the optimization engine identifies opportunities for moving projects into and out of the portfolio in ways that significantly increase total value. To take a dramatic example, notice the jump in the curve that occurs for a budget just above $5 million. The cause of the jump is a high-cost project with a minimum spending level of $5.05 million (it is the project in the 5th row in the table in Figure 45). At this spending level, the project provides a relatively high benefit-to-cost ratio of 8.78. Figure 46 shows the detailed portfion of the funding table corresponding to budgets near $5 million (the increments in this table are $25,000 per column, compared to $1 million in the previous table).. If the total budget is less than $5.05 million, then, obviously, the $5.05 million project cannot be accomodated. However, when the budget hits 5.05 million, all of the smaller projects that were funded earlier are eliminated from the portfolio, which frees the entire budget to fund this one highvalue project and results in the jump in the curve (an increase in portfolio value of $8.62 million!).
108
Figure 46: Optimal funding decisions for budgets between $5 M and $5.1 M in $25 K increments
Although large jumps in the efficient frontier often occur at highly constrained budgets, such jumps are not of practical significance unless they occur near the organizations actual budget constraint (which typically happens when there are high-cost projects with high benefit-to-cost ratios that are in danger of not being funded). In such situations, the efficient frontier can identify opportunities for small increases in the total budget that could produce relatively large increases in total value.
109
Importance of the Efficient Frontier

Suppose we construct the efficient frontier as described in the previous sections. The result is useful for several reasons. The Efficient Frontier Answers Key Questions The efficient frontier allows us to answer four important questions: 1. What are the best project choices given a specified constraint on the available budget and/or other resources? 2. If, for political or other reasons, we choose a specific, non-optimal set of projects (a project set not on the efficient frontier), by how much are we over-spending and how much potentially achievable value are we losing? 3. Based on the total value obtained for the total dollars spent, are we over- or under-spending on projects? 4. How should we allocate the total budget across organizational units or to different types of projects? The best project choices (Question 1) are, of course, those contained in the portfolio that lies on the efficient frontier at the cost level at, or just below, the cost constraint. The losses resulting from choosing a non-optimal portfolio (Question 2) are the horizontal and vertical distances from the selected portfolio to the efficient frontier (see Figure 48 below). With regard to Question 3, an organization is overspending on projects if the chosen portfolio lies on that portion of the curve where the slope is less than 1. Question 4, how to allocate resources among organizational units can be answered if a separate efficient frontier is constructed for the projects proposed by each business unit. As shown in Figure 47, the optimal allocation funds portfolios on the respective curves where the slopes are equal. This result provides a useful approach for developing tiered priority systems for organizations with decentralized project prioritization processes.
110
Figure 47: The optimal allocation matches slopes on the efficient frontiers.
. Evidence that Finding the Efficient Frontier Adds Considerable Value In real-world applications, it is sometimes possible to compare the performance of a current portfolio with optimal portfolios that are on the efficient frontier. Figure 48, derived from an actual application, shows that an alternative portfolio was found that increased value by over 30% without increasing costs. Similarly, an alternative portfolio was found that reduced costs by 40% without decreasing value. This result is typical for organizations conducting projects that are difficult to value. Application of the efficient frontier approach shows that current project portfolios are often well below their potential.
111
Figure 48: Project portfolio enhanced by applying the efficient frontier.
Better project portfolios are not the only benefit of the efficient frontier. Calculating the efficient frontier creates a new perspective, one that helps managers throughout the organization to fully appreciate the reality that resources are limited, to better understand the relationship between value created and costs incurred, and, when the opportunity is great enough, to find ways to break the constraint on costs.
112
Achieving Best-Practice
Project portfolio management is a tool-supported process for effectively organizing and managing the multi-project environment. It is an ongoing, dynamic process wherein projects and project proposals are regularly evaluated, prioritized, and selected based on the goal of obtaining the greatest possible value from the available limited resources. Making the best project choices is the core of project portfolio management, but project selection includes deciding how to allocate and apply resources to projects, project timing, and when to accelerate, slow, or kill projects. Almost any project selection process will separate "must-do" projects from clear "losers." How far organizations go beyond this depends on the effort they put into it. In order to get much beyond a "60% solution," however, organizations need to address the fundamental reasons that organizations choose the wrong projects.
113
Choosing the Wrong Portfolio of Projects Part 6: Achieving Best-Practice
Address the Reasons Organizations Choose the Wrong Projects

In summary, organizations need to: Address the errors and biases that affect human judgment. Increase awareness of prevalent errors and biases, including comfort zone, perception, and motivation biases, as well as errors in reasoning, and "group think." As Daniel Kahneman advises, use the knowledge to create "human error detectors" within your organization. Consider incentives and the effects of framing when evaluating your own and other people's judgments. Remember that an estimate from a disinterested but knowledgeable party may be more reliable than that of a better-informed but involved expert. Provide feedback to people on the accuracy of their forecasts. Require that forecasts of project performance be expressed in terms of observables that pass the clairvoyant test. Then, collect data from funded projects to help calibrate people and keep their estimates honest.
Get control of the project-selection process. See the forest as well as the trees. Collect projects and project proposals into a common database. Look for duplications and interdependencies. Establish common format and content requirements for project proposals. Move from project-by-project decision-making to decision-making aimed at producing optimal project portfolios. Create a project portfolio management office with responsibility for managing the organization's portfolio of projects. Expose hidden discretionary spending (e.g., mandatory projects that could be delayed, done with reduced scope, miscellaneous spending) Adopt a systems perspective that explores the chain of consequences produced by the choices that may be made. Understand the options that are created and destroyed by project choices. Understand and measure how project investments translate into business performance (e.g., service, reliability, efficiency, and productivity).
Make Value Creation the Organizational Goal. Promote a culture focused on creating the greatest possible value for the organization. Value creation is a compelling argument, one that can overcome inertia and the barriers against positive change. Create a project-selection decision model for your business that documents bestorganizational understanding about how projects create value. Use the decision model to select metrics and to systematically evaluate proposed projects. Engage senior executives in the process of establishing objectives, defining how value tradeoffs should be made, sharing ownership of project decisions, and co-developing project expectations. Understand and measure how performance impacts translate into economic (e.g., shareholder/stakeholder) value within businesses and across the enterprise. When choosing projects, consider project urgency as well as project value.
114
Be proactive in addressing risk. Create a culture that insists on facing up to risk. Accept the fact that the world is volatile, that things are changing rapidly, and that bad things might happen. Establish processes for identifying internal and external project and project-deferral risks. Assess those risks and communicate them. Implement risk-mitigation action plans. Think in terms of probabilities. Dont just ask what might happen. Ask how likely it is. Avoid the bias toward doing too many, mostly low-risk, low-return projects. Remember that risky projects often create learning and increased capability, values that aren't readily captured in financial metrics. For "big bet" investments, quantify risks and consider establishing an organizational risk tolerance to guide decision-making.
Build decision-making competencies. Empower decision makers. The old command-and-control structure no longer works. It is too slow and creates information overload for leaders. Remove barriers to the free flow of information. If people hoard information as a source of power, others cant make informed choices. Promote and attend training workshops on the principles and techniques of decision analysis, project prioritization, and project portfolio management. Learn the best techniques for articulating objectives, expressing value tradeoffs, assessing probabilities, and establishing risk tolerance. Recognize and reward people based on the quality of their decisions, not just based on the quality of their outcomes.
Be smart about institutionalizing new tools. Start at the top and generate executive buy-in. Leaders must be visible champions for change. Create awareness, build consensus, and motivate stakeholders at all levels. Involve stakeholders in the design, testing, and roll-out of new tools. Establish formal agreements on roles and responsibilities across the organization. Provide training and support appropriate to peoples roles. Dont overwhelm the organization. Match the pace of change to the organizations capacity to evolve. A gradual roll-out based on proof-of-concept is usually more successful. Use gap analysis to target the initial application on a critical need. Publicize successes throughout the organization. Develop the governance process. Organizational structure that supports project portfolio management is essential.
115
Five Levels of Project Portfolio Management

Figure 49 summarizes five levels of project portfolio management maturity [1]. Each level represents the adoption of an increasingly comprehensive and effective subsets of related solutions discussed in the previous parts of this 6-part paper for addressing the reasons that organizations choose the wrong projects. Understanding organizational maturity with regard to project portfolio management is useful. It facilitates identifying performance gaps, indicates reasonable performance targets, and suggests an achievable path for improvement. The fact that five maturity levels have been identified is not meant to suggest that all organizations ought to strive for top-level performance. Each organization needs to determine what level of performance is reasonable at the current time based on business needs, resources available to engineer change, and organizational ability to accept change.
Figure 49: Five levels of project portfolio management.
The detailed definitions of the levels, provided below, are not precise. Real organizations will tend to be more advanced with regard to some characteristics and less advanced relative to others. For most organizations, though, it is easy to identify one of the levels as roughly characterizing the current maturity of project portfolio management.
116
Level 1: Foundation Level 1 organizes work into discrete projects and tracks costs at the project level. Project decisions are made project-by-project without adherence to formal project selection criteria. The portfolio concept may be recognized, but portfolio data are not centrally managed and/or not regularly refreshed. Roles and responsibilities have not been defined or are generic, and no value-creation framework has been established. Only rarely are business case analyses conducted for projects, and the quality is often poor. Project proposals reference business benefits generally, but estimates are nearly always qualitative rather than quantitative. There is little or no formal balancing between the supply and demand for project resources, and there is little if any coordination of resources across projects, which often results in resource conflicts. Over-commitment of resources is common. There may be a growing recognition that risks need to be managed, but there is little real management of risk. Level 1 organizations are not yet benefiting from project portfolio management, but they are motivated to address the relevant problems and have the minimum foundation in place to begin building project portfolio management capability. At this level, organizations should focus on establishing consistent, repeatable processes for project scheduling, resource assignment, time tracking, project tracking, and general project oversight and support. Level 2: Basics Level 2 replaces project-by-project decision making with the goal of identifying the best collection of projects to be conducted within the resources available. At a minimum this requires aggregating project data into a central database, assigning responsibilities for project portfolio management, and force-ranking projects. Redundant projects are identified and eliminated or merged. Business cases are conducted for larger projects, although quality may be inconsistent. Individual departments may be establishing structures to oversee and coordinate their projects. There is some degree of options analysis (i.e., different versions of the project will be considered). Project selection criteria are explicitly defined, but the link to value creation is sketchy. Planning is mostly activity scheduling with limited performance forecasting. There are attempts to quantify some non-financial benefits, but estimates are mostly
117
guestimates generated without the aid of standard techniques. Overlap and double counting of benefits between projects is common. Ongoing projects are still rarely terminated based on poor performance. The PPM tools being used may have good data display and management capabilities, but project prioritization algorithms may be simplistic and the results potentially misleading to decision makers. Portfolio data has an established refresh cycle or is regularly accessed and updated. Resource requirements at the portfolio level are recognized but not systematically managed. Knowledge sharing is local and ad hoc. Risk analysis may be conducted early in projects but is not maintained as a continual management process. Uncertainties in project schedule, cost and benefits are not quantified. Schedule and cost overruns are still common, and the risks of project failure remain large. Level 2 organizations are beginning to implement project portfolio management,, but most of the opportunity has not yet been realized. The focus should be on formalizing the framework for evaluating and prioritizing projects and on implementing tools and processes for supporting project budgeting, risk and issues tracking, requirements tracking, and resource management. Level 3: Value Management Level 3, the most difficult step for most organizations, requires metrics, models, and tools for quantifying the value to be derived from projects. Although project interdependencies and portfolio risks may not be fully and rigorously addressed, analysis allows projects to be ranked based on "bang-for-the-buck," often producing a good approximation of the value-maximizing project portfolio The principles of portfolio management are widely understood and accepted. The project portfolio has a well-defined perimeter, with clear demarcation and understanding of what it contains and does not contain. Portfolio management processes are centrally defined and well documented, as are roles and responsibility for governance and delivery. Portfolio management can demonstrate that its role in scrutinizing projects has resulted in some initiatives being stopped or reshaped to increase portfolio value. Executives are engaged, provide tradeoff weights for the value model, and provide active and informed support. Plans are developed to a consistent standard and are outcome- or value- based. Effective estimation techniques are being used within planning and a range of project alternatives are routinely considered. Data quality assurance processes are in place and independent reviews are conducted. There is a common, consistent practice for project approval and monitoring. Project dependencies are identified, tracked, and managed.
118
Decisions are made with the aid of a tool based on a defensible logic for computing project value that generates the efficient frontier. Portfolio data are kept up-to-date and audit trails are maintained. Costs, expenditures and forecasts are monitored at the portfolio level in accordance with established guidelines and procedures. Interfaces with financial and other related functions within the organization have been defined. A process is in place for validating the realization of project benefits. There is a defined risk analysis and management process, with efforts appropriate to risk significance, although some sources of risk are not quantified in terms of probability and consequence. Level 3 organizations demonstrate a commitment to proactive, standardized project and project portfolio management. They are achieving significant return from their investment, although more value is available. Level 4: Optimization Level 4 is characterized by mature processes, superior analytics, and quantitatively managed behavior. Tools for optimizing the project portfolio correctly and fully account for project risks and interdependencies. The business processes of value creation have been modeled and measurement data is collected to validate and refine the model. The model is the basis for the logic for estimating project value, prioritizing projects, making project funding and resource allocation decisions, and optimizing the project portfolio. The organizations tolerance for risk is known, and used to guide decisions that determine the balance of risk and benefit across the portfolio. There is clear accountability and ownership of risks. External risks are monitored and evaluated as part of the investment management process and common risks across the whole portfolio (which may not be visible to individual projects) are quantified and in support of portfolio optimization. Senior executives are committed, engaged, and proactively seek out innovative ways to increase value There is likely to be an established training program to develop the skills and knowledge of individuals so that they can more readily perform their designated roles. An extensive range of communications channels and techniques are used for collaboration and stakeholder management High-level reports on key aspects of portfolio are regularly delivered to executives and the information is used to inform strategic decision making. There is trend reporting on progress, actual and projected cost, value, and level of risk.
119
Assessments of stakeholder confidence are collected and used for process improvement. Portfolio data is current and extensively referenced for better decision making. Level 4 organizations are using quantitative analysis and measurements to obtain efficient, predictable and controllable project and project portfolio management. They are obtaining the bulk of the value available from practicing project portfolio management. Level 5: Core Competency Level 5 occurs when the organization has made project portfolio management a core competency, uses best-practice analytic tools, and has put processes in place for continuous learning and improvement. Portfolio management processes are proven and project decisions, including project funding levels and timing, are routinely made based the value maximization value. Processes are continually refined to take into account increasing knowledge, changing business needs, and external factors. Portfolio management drives the planning, development, and allocation of projects to optimize the efficient use of resources in achieving the strategic objectives of the organization. High levels of competence are embedded in all portfolio management roles, and portfolio management skills are seen as important for career advancement. Portfolio gate reviews are used to proactively assess and manage portfolio value and risk. Portfolio management informs future capacity demands, capability requirements are recognized, and resource levels are strategically managed. Information is highly valued, and the organizations ability to mitigate external risks and grasp opportunities is enhanced by identifying innovative ways to acquire and better share knowledge. Benefits management processes are embedded across the organization, with benefits realization explicitly aligned with the value measurement framework. The portfolio is actively managed to ensure the long term sustainability of the enterprise. Stakeholder engagement is embedded in the organizations culture, and stakeholder management processes have been optimized. Risk management underpins decision-making throughout the organization. Quantitatively measurable goals for process improvement have been established and performance against them tracked. The relationship between the portfolio and strategic planning is understood and managed. Resource allocations to and from projects are intimately aligned so as the maximize value creation. Level 5 organizations are obtaining maximum possible value from project portfolio management. By fully institutionalizing project portfolio management into their culture they free people to become
120
more creative and innovative in achieving business success. Building Project Portfolio Management Maturity Experience shows that building project portfolio management maturity takes time. As suggested by Figure 50, significant short-term performance gains can be achieved, but making step changes requires understanding current weaknesses and the commitment of effort and resources.
Figure 50: Step changes can be made, but achieving high levels of maturity typically takes years.
121
Best-Practice Project Portfolio Management

Although the specifics of project portfolio management necessarily differ from organization to organization, the fundamentals of best practice are nearly universal. Project portfolio management is an ongoing activity, not just an annual event. It is driven by the fundamental objectives the organization, and it demands and enables broad organizational involvement. The goal in all cases is to select and manage project portfolios so as to create the greatest possible value for the organization. Organizations that have achieved best practice have a clear understanding of the value creation process. They have identified and structured their objectives and have established metrics for forecasting and tracking the degree to which they achieve those objectives. They use a projectselection decision model to evaluate project proposals, and update and refine that model as their knowledge and understanding improves. The model allows them to estimate the impact on the business of doing versus no doing individual projects as well as conducting various combinations of interdependent projects. The project consequences estimated include not only impacts on financial performance, such as future cost savings and increases in revenue, but also include non-financial impacts, such as improved customer service and organizational learning. Best-practice organizations translate these performance impacts into equivalent monetary value based on the worth of the project benefits to the organization. Project risks are addressed, including both project risk and project deferral risk. Portfolio-level risks are likewise understood, quantified, and managed. The project portfolio management process is supported by a quality software tool with smart, template-driven data collection and flexible graphic and reporting capabilities. The outcomes of portfolio decisions are monitored, and there is a significant focus on mining experience and identifying opportunities for improvement. Participants The table on the next page identifies the key players and responsibilities.
122
Business Unit/Sponsor
Any organizational component that requests or consumes a portion of the budget for the purpose of conducting projects. Each business unit Identifies projects, assists project managers in constructing business cases for justifying projects, and champions its projects and project portfolio. The business unit is responsible for providing quality assurance for data related to its projects.
Project Manager
Project Portfolio Manager

Manager with responsibility for the project portfolio. Usually supported by a team. Team may be composed of directors of the business areas. The project portfolio manager establishes the rules, and procedures for making portfolio decisions. The portfolio manager analyses projects and portfolios proposed by business units and recommends the overall project portfolio.
Executive Team
Select corporate officers who guide and provide inputs to the project portfolio management process.
Individual with overall responsibility for successful planning and execution of a project.
Project managers work closely with business units/sponsors to provide good data for the portfolio management process. Project managers are responsible for ensuring that approved projects perform according to plan.
The executive team provides policy inputs for the process, including weights for trading off different types of project benefits. The team sets targets, approves the budget and project portfolios, and ensures that portfolio decisions are enforced.
Phases The three phases of the project portfolio management cycle are preparation, execution, and performance management. Figure 51 shows the typical process timeline and Figure 52 provides more detail on typical process flows.
Figure 51: Typical project portfolio management process timeline.

123
Figure 52: Process flow diagram for project portfolio management. 124
Preparation A calendar is established and responsibilities are assigned. Corporate executives set high-level strategy. The executive team establishes financial and performance targets for each business unit and provides guidelines and a schedule for completing the budgeting process. With the assistance of the portfolio manager, the team establishes basic assumptions for analysis (including value weights, discount rates, and risk tolerance). Execution Within each business unit, projects are identified, classified, and appropriately grouped. Alternative project solutions are explored. Decision units for each project are defined (e.g., go vs. no-go, alternative project solutions, alternative funding levels and scopes for the preferred project solution). Project descriptions are documented, including timelines, resource needs, etc., and project proposal templates generated by the tool are completed. Technical analyses are conducted to estimate the impacts of each project or project grouping on the achievement of corporate and business unit objectives (using external models or the models inherent in the tool). Financial analyses are similarly conducted. Risks are identified and described. With the aid of the tool, project and portfolio values are estimated and the projects within each business unit are prioritized. Based on the results, project plans are revised. Business unit priorities are established and feed corporate-level prioritizations and the allocation of resources across business units. A value prioritized budget is established and spending approvals are granted. Performance Management A performance management plan monitors and manages the success of the project portfolio. The plan specifies project performance indicators that allow comparing forecast and actual performance, plus monitoring and reporting schedules. Project shortfalls are analyzed, including causes, approaches for correcting variances, and decisions for ongoing investments. Lessons learned are derived and the project portfolio management process is refined prior to the next budget cycle.
125
Why Isn't Everyone Doing this Already?

As this paper has described, methods are available for addressing all of the reasons organizations choose the wrong projects, including quantitative methods for identifying projects on the efficient frontier. Some organizations that regularly make high-stakes, project-selection decisions (e.g. some financial institutions, oil and gas exploration companies, pharmaceutical companies, some high-tech companies, the military and some other government agencies) are already using sophisticated versions of these methods. But, most organizations are not. A fair question is, "If what youve described is so great, why isn't everybody already doing it?" One practical reason why relatively few organizations are using these techniques is that the relevant models and mathematical operations require sophisticated computer operations. It is only recently that high-powered computer capability has become widely available, and more recent still that managers have begun to feel comfortable using computers directly in support of their work. In some organizations, resistance to formal, analytic methods may be politically motivated. Some executives may fear any change that threatens the power systems they have worked long and hard to create. A system that causes decisions to be based more on fact and reason will mean that political pressures and false urgencies will be less effective tools for steering the organization. From the personal perspective of a politically savvy executive, particularly one that believes he or she is more able than others to discern what is best for the organization, it may make perfect sense to resist attempts to adopt analytic methods. Most organizations, however, are under increasing pressure to find ways to do more with their limited resources, and many managers are facing growing demands to explain the basis for their decisions, Thus, it would seem increasingly unlikely that a few self-serving individuals could successfully discourage their organizations from adopting proven analytic solutions. Yet many organizations continue to be slow in adopting best-practice methods. There must be more fundamental reasons that explain the resistance. Indeed, if what Ive said in this paper is truethat organizations may be obtaining only 60% of the value that could be derived from using best-practice project portfolio managementthe barriers to adopting such methods must be significant. I believe that there are 5 impediments that cause people to resist best-practice analytic methods (Figure 53): (1) the prevalence of misinformation, (2) the fact that leaders may take misplaced pride in being intuitive decision makers, (3) fear of complexity, (4) a belief that there may be too much uncertainty to justify sophisticated methods, and (5) under appreciation of the predictive power of judgment. Understanding these barriers can help you to develop effective strategies for bringing best practice methods into your organization.
126
Figure 53: Barriers to using best practice, analytic, project portfolio management.
Prevalence of Misinformation In a recent article, Jeffrey Pfeffer and Robert Sutton address the question of why more companies arent using evidence-based management, which they define as the conscientious, explicit and judicious use of current best evidence in support of decision making [2]. Instead, companies seem to repeatedly adopt, then abandon, one ill-supported business fad after another. There exists, they point out, a huge body of peer-reviewed studiesliterally thousands...that although routinely ignored, provide simple and powerful advice about how to run organizations. If found and used, this advice would have an immediate positive effect. The failure of most businesses to use sound management science, they argue, is largely due to misinformation. They offer managers these warnings: People are trying to mislead you. Because its so hard to distinguish good advice from bad, managers are constantly enticed to believe in and implement flawed business practices. A big part of the problem is consultants, who are always rewarded for getting work, only sometimes rewarded for doing good work, and hardly ever rewarded for evaluating whether they have actually improved things. You are trying to mislead you. People want easy solutions, but applying practices based on sound theory generally takes more effort. It requires] a willingness to put aside convenient half-truthsand replace these with an unrelenting commitment to gather the
127
necessary factors to make more informed and intelligent decisions. Simplistic approaches are popular because they do not force you to think very hard or to think in new ways. Unless you are an expert in the relevant theories, it is tough to distinguish sound methods from snake oil. Managers dont have the time to research all of the claims that land on their desks. Partial truths are the most difficult to defend against. Thus, concepts like balance and strategic alignment catch on, even though there is no defensible argument or evidence for claiming that they result in better project choices or improved performance for the organization. To quote Ronald Howard, In decision-making, as in many other pursuits, you have a choice of doing something the easy way or the right way, and you will reap the consequences[3]. Checking the claims of those who are trying to sell you something may take effort, but it can save you from making serious mistakes. Check references. Be suspicious. Ask to see evidence to support claims. Get an objective opinion from an independent expert in the field. Misplaced Pride in Intuition In an article entitled A Brief History of Decision Making, Leigh Buchanan and Andrew OConnell characterize the current era in which leaders seem to take pride in making decisions based on intuition as a romance of the gut[4]. The popular view is that real leaders dont need analysis, Intuition is one of the X factors separating the men from the boys [5]. Recent years have seen two best-selling books on this theme. Psychologist Gary Kleins book, Intuition at Work, advises managers to trust your gut [6]. Similarly, columnist Malcolm Gladwells best-selling book Blink argues that instantaneous decisions are often better than those based on lengthy rational analysis [7]. Behavioral psychologists Robin Hogarth and Paul Schoemaker provide a critical review of Blink [8]. They observe that Gladwells faith in intuition ignores the vast literature on biases (see Part 1 of this paper). They provide counterexamples to Gladwells examples and demonstrate that some of Gladwells own case studies seem to imply the opposite of his conclusions. Gladwell himself acknowledges that intuition is not always the best approach, and his book provides an example of how an analytic model is proving more effective than un-aided judgment in an urban hospital. Author Barry Anderson observes that it takes courage to be rational [9]. Analysis can show that what we believe is not consistent with evidence and logic. Also, analysis is a great leveler of hierarchy. If the decision is going to be made by the facts, then everyones facts, assuming they are relevant and accurate, are equal. Unlike intuition, analysis can be learned. Analysis changes power dynamics, replacing formal authority, reputation, and intuition with data. This means that senior leaders, often venerated for their wisdom and decisiveness, may lose some stature if their intuitions are replaced by analysis. Basing decision on intuition may be easy and attractive, but leaders need to decide what course is really in their best interests. Do they want to use intuition and avoid analysis that can prove them wrong, or do they want to use all effective means to ensure that their organizations actually perform well?
128
Fear of Complexity For some managers, trying to select optimal project portfolios may seem too complicated to tackle. Psychologists identify fear of complexity as one of the key pitfalls that prevent people from overcoming important problems. They point out that the perception of complexity is reduced, however, when people use information processing structures (what some psychologists call description languages) that provide a good fit to the complexities they encounter. Systems modeling, the foundation for the methods described in this paper, is a language for describing and understanding complex problems. Models break a complex problem down into its individual pieces. The critical components are identified, sorted out, and analyzed separately. Computers perform the required synthesis at the end. As long as the concepts are understood, the fact that the math may be difficult is not really an issue; computers can handle the math. Admittedly, systems modeling and the related methods that are described in this paper including multi-attribute utility analysis, probabilistic analysis, and risk tolerance, can themselves seem complex. Remember, though, that the most sophisticated tools need not be applied in all situations. If projects do not involve significant risks there is no need for probabilities and concepts like risk tolerance. More critical and difficult decisions require more sophisticated methods. I urge organizations to follow the often quoted advice of Albert Einstein, "Seek the simplest possible solution, but no simpler." In particular, use methods that get the basic concepts right (e.g., the end goal is to create value, not balance). With learning and familiarity that come from experience, the appropriate methods will no longer seem complex. Discomfort with Uncertainty People would much prefer analysis to tell them what will happen, not what might happen. Thus, there is the perception that analysis is less useful in situations that are highly uncertain. Although there are well-established theories and techniques for optimizing decisions involving uncertainty (e.g., decision analysis), these are not topics about which most members of the public, or even most managers, are very familiar. Many managers assume that the great uncertainty over the costs and benefits of projects means that there is little value to applying sophisticated techniques to prioritize them. My experience is that it doesnt take much education for skeptics to agree that sophisticated methods based on probability can significantly improve decisions involving uncertainty. For example, roughly 40 years ago, probabilistic analysis was used to "beat" the game of blackjack. The sticking point seems to be whether the same methods have merit when probabilities must be based on models and judgment rather than purely on "objective data." The following checklist of questions helps me decide whether I need to quantify judgmental uncertainty: Is the uncertainty significant? If so, assessing a range of values for the uncertain quantity will make more sense, and be easier, than specifying a single-number best guess. 2. Does the uncertainty make a difference? Try varying the uncertain quantity across the range of possibilities. The uncertainty only matters if you would want to make decisions differently depending on the actual value.
1.
129
3.
Do experts believe some possibilities are more likely than others? If so, a probability distribution can be chosen to quantify those beliefs. I've never encountered a situation where the experts didn't have relevant beliefs, but if I did, there would be good reason to quantify the uncertainty using a uniform probability distribution, which assumes that all possibilities are equally likely.
Once uncertainties have been expressed as probability distributions, a probabilistic analysis is usually not much harder than the corresponding analysis without probabilities. It only becomes more difficult if the experts know a great deal, in which case such knowledge may need to be represented in more complex models that necessitate more sophisticated forms of analysis. The existence of uncertainty does not undermine the usefulness of probabilistic methods. On the contrary, it enhances their usefulness. When significant uncertainties are present, only a systematic and rigorous approach can produce an accurate understanding of risk and support a sound logic for making risky decisions. Under-Appreciation of the Power of Judgment "But," people say, "Probability obtained in this way are subjective!" Actually, everything associated with decision-making is subjective, but in the interest of space I won't get into those arguments! People sometimes confuse subjectivity with bias. As demonstrated in Part 1 of this paper, estimates based on judgment are often biased. However, formal methods are available for mitigating most biases. In the area of probability assessment in particular, considerable effort has been devoted to developing techniques for eliciting probability judgments that accurately reflect the beliefs of those who provide them. Organizations that eschew analysis because of its dependence on subjective judgment are, in my opinion, underutilizing what may be their most important resource. Consider the concept known as the knowledge-based theory of the firm [10]. The theory argues that knowledge is the only strategically important business resource. Other resources, like raw materials and electric power, are available at essentially the same prices to all competitors. Knowledge is the only resource that really distinguishes a company and can provide a real advantage. According to the theory, fundamentally what firms do is "apply knowledge to the production of goods and services." Since knowledge is held by individuals and not the organization, "the central role of the enterprise and its management is to integrate distributed knowledge and make it usable" [10]. Models and probabilities provide the best-known means for doing this. Models capture in a clear and useful fashion peoples understanding about what makes the business successful. Probabilities encode beliefs about key uncertainties represented within the models in a precise, transparent way that can be understood by others and that can be readily used by the models to develop optimal decision strategies. Although judgmental probabilities are indeed subjective, it is important to appreciate that they are not arbitrary. If a manager is using probabilities property and says there is a 25% chance that the project will go over budget, that manager is saying that the degree of confidence in achieving the budget is
130
the same as randomly selecting a red ball from an urn containing one red ball and three white balls. Thus, subjective probability is related to an objective reality. Expressing uncertainty as a probability gives a much more precise and useful statement than saying "it's uncertain." Judgmental probabilities can be processed to derive logical inferences. Furthermore, judgmental probabilities can be calibrated to experience. If there is ample evidence that only one-fourth of projects come in on budget, then, presumably, others will have more confidence in the 25% probability judgment. Predictive Markets Are subjective probabilities any good? Undoubtedly, the main reason that subjective probabilities arent used more often is the mistaken belief that they have little merit. But, awareness of the power of judgment is now gaining a significant boost based on the remarkable and well-publicized success of predictive markets. Around for years, these markets attracted attention after correctly calling close presidential elections (and, also, when they were rejected by the government as a method for predicting terrorist attacks). In case you havent heard, predictive markets are betting markets wherein participants buy and sell shares of financial assets whose final values are tied to the outcomes of specific uncertain events (e.g., the outcome of an election). In effect, the market generates a price for the uncertain event that reflects a consensus probability that the event will occur. One of the oldest and most famous is the University of Iowas Iowa Electronic Market, which, since 1988, has been predicting the results of American presidential elections with more accuracy than polling services. A similar market correctively predicted all of the 2005 big category Oscar winners [11]. Even game markets, where players bet bragging rights rather than money, appear to result in good forecasts. According to one study, web-based betting competitions accurately predict movie box office returns, the winners of formula one racing events, and future developments in science and technology [12]. Predictive markets work because they produce forecasts in a way that effectively combines the knowledge of many participants while avoiding many of the biases identified in Part 1 of this paper. Seeing opportunity, companies are beginning to create their own predictive markets, focusing on specific uncertainties that they want quantified. Hewlett-Packard, for example, has used internal markets to forecast sales and Eli Lilly has used them to predict the success of drug research [13]. Google recently reported that it uses predictive markets internally for generating strategic insights. The company compared the prices for events with the frequency with which the events occurred (if the prices are correct, events priced at 10 cents should occur about 10 percent of the time). The results were remarkably close [14]. The criteria for establishing a successful predictive market are the same as conditions that decision analysts have established for assessing subjective probabilities: 1. The uncertain quantity must be precisely and unambiguously defined (see the discussion on the clairvoyant test in Part 3). 2. Participants must have some knowledge of the relevant subject matter. (It has been observed, for example, that a predictive market that allowed the general public to bet on the veracity of String Theory in particle physics would likely not be a very accurate predictor.)
131
3. A motivation must exist for people to express what they really believe will happen (not what they would like to happen or what they think others want to hear). Perhaps awareness of the success of predictive markets will cause more organizations to consider using analytic methods that properly rely heavily on judgment to evaluate and prioritize projects.
132
Competing on Analytics
There are reasons to be optimistic. As sophisticated methods gain use, evidence of their value is beginning to surface. For example, an article in the journal Oilfield Review reported a study of 20 oil exploration companies that "established a strong positive correlation between the degree of sophistication in the companies' use of decision and risk analysis and the success of their project decisions." The same article also described another oil company study that found that "Companies that integrated workflow and used decision and risk analysis saw their performance improve shortly after the introduction of this methodology" [15]. Of the numerous ways that analytics can improve organizational performance, project portfolio management, in particular, is receiving good press. CIO Magazine reports a survey of portfolio management in product development applications that found that companies that achieve project portfolio management excellence experienced 50% faster revenue growth [16]. A similar study released in 2006 by the AberdeenGroup reports that companies that are best in class at product portfolio management are four times more likely to achieve margin premiums of 75% or higher from new products [17]. A study focused on R&D portfolios found that companies performing in the top 20 percent had previously installed an explicit, established method of project portfolio management across the organization [18]. In 2008, a market research firm estimated that project portfolio management software delivers over 500% ROI [19]. Also, something new is happeninga few highly successful companies are making no secret of the fact that they have adopted superior analysis as a competitive business strategy. Explaining better than expected 2009 3rd quarter earnings, Ciscos CFO said, We have continued our emphasis on operational excellence, portfolio management, and customer focus, all of which we believe positions us for future success [20] . John Wilder, CEO of the utility TXU, claims to have cut costs by more than $1 billion through a series of initiatives that included bringing analytic rigor to our portfolio decisions [21]. In an HBR article entitled, Competing on Analytics, Thomas Davenport identifies eBay, Google, Amazon, and Dell, among others, as a new breed of companies that are oriented to a much higher level of analytics: predictive modeling, optimization techniques than weve been used to in the business world [22], They hire employees for their analytic expertise, provide them with the best available information, and arm them with the best quantitative tools. As a result, they make the best decisions: big and small, every day, over and over and over. Beat the 60% solution The introduction to this paper described the concept of the 60% solution, the belief held by many that organizations only obtain about 60% of the value that could be derived from their businesses. As I have explained, I believe that choosing the wrong portfolio of projects is a major reason for lost business value. The 60% solution can be beaten by doing a better job of choosing and managing project portfolios. It may not be easy, but it can definitely be done. The fact that optimizing project decisions is hard to do, but doable, is why organizations that address the problems identified in this paper can create for themselves a significant competitive business advantage.
133

1. 5-level maturity models have been developed for many organizational processes. The UK Government is developing a similar model as a guide for public sector organizations: P3M3 (Portfolio, Programme and Project Management Maturity Model), Public Consultation Draft, v2.0, Office of Government Commerce, United Kingdom, June 24, 2008. 2. J. Pfeffer and R. I. Sutton, Evidence-Based Management, Harvard Business Review, January 2006. 3. R. Howard, The Foundations of Decision Analysis Revisited, Chapter 3 in Advances in Decision Analysis, W. Edwards, R. Miles, and D. von Winterfeldt, eds., Cambridge University Press, 1999. 4. L. Buchanan and A. OConnel, A Brief History of Decision Making, Harvard Business Review, January 2006. 5. A. Hayashi, When to Trust Your Gut, Harvard Business Review, February 2001. 6. G. Klien, Intuition at Work: Why Developing Your Gut Instincts Will Make You Better at What You Do, Currency Doubleday, 2002. 7. M. Gladwell, Blink: The Power of Thinking Without Thinking, Little, Brown and Co., New York, 2005. 8. R. Hogarth and P. J. H. Schoemaker, Beyond Blink: A Challenge to Behavioral Decision Making, Journal of Behavioral Decision Making, March, 2005. 9. B. F. Anderson, The Three Secrets of Wise Decision Making, Single Reef Press, 2002. 10. R. M. Grant, "Toward a Knowledge-Based Theory of the Firm," Strategic Management Journal, Vol. 17, 1996. 11. J. Wolfers and E. Zitzemwitz, Predictive Markets, NBER Working Paper No. W10504, May 2004. 12. D. M. Pennock, S. Lawrence, F. A. Nielson, and C. L. Giles, Extracting Collective Probabilistic Forecasts from Web Games, Proceedings of the 7th ACM SIGKDD International Conference on Knowledge and Data Maining, pp. 174-183, ACM, New York, 2001. 13. M. Einbinder, Information Markets: Using Market Predictions to Make Administrative Decisions, Virginia Law Review, Vol. 92, p. 149, 2006. 14. Reported by J. Nocera, The Future Divined by the Crowd, The New York Times, March 11, 2006. 15. W. Bailey, B. Couet, F. Lamb, and P. Rose, "Taking a Calculated Risk," Oilfield Review, Autumn 2000. 16. Pittiglio, Rabin, Todd & McGrath (PRTM), Product Development Benchmarking Series, PRTM, Performance Measurement Group, 2000, as reported in L. Cosgrove, Analyzing Product Development, CIO Magazine, March 1, 2002.
134
17. The Aberdeen Group, The Product Portfolio Management Benchmark Report, August 2006 18. R. G. Cooper, S. J. Edgett, and E. J. Kleinschmidt, Portfolio Management for New Product Development: Results of an Industry Practices Study, R&D Management, pp 381-389, October 2001. 19. IDC White Paper (sponsored by PPM vendor CA), How Project and Portfolio Management Solutions Are Delivering Value to Organizations, Doc #213980, September 2008. 20. Cisco, Cisco CFO Frank Calderoni Offers Commentary on the Quarter (video), www.youtube.com/watch?v=gCNwGG2YKHM. 21. An Interview With TXUs CEO: C. John Wilder Describes How Economic Thinking Helped him Lead TXU Out of a Regulated Mind-Set Toward Competitive Success, The McKinsey Quarterly, Feb, 2007. 22. T. Davenport, Competing on Analytics, Harvard Business Review, January 2006.
135

Selecting The Wrong Person

Uploaded by

Copyright:

Available Formats

Selecting The Wrong Person

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Selecting The Wrong Person

Uploaded by

Copyright:

Available Formats

Choosing the Wrong Portfolio of Projects

Lee Merkhofer, Ph.D.

c 2003-2009 Lee Merkhofer Consulting

Choosing the Wrong Portfolio of Projects

c 2003-2008 Lee Merkhofer Consulting

Choosing the Wrong Portfolio of Projects

c 2003-2009 Lee Merkhofer Consulting

Choosing the Wrong Portfolio of Projects

c 2003-2008 Lee Merkhofer Consulting

Choosing the Wrong Portfolio of Projects

Errors and Bias in Judgment

c 2003-2009 Lee Merkhofer Consulting

Heuristics and Judgmental Biases

Figure 1: Haze tricks us into thinking objects are further away.

c 2003-2009 Lee Merkhofer Consulting

Perception Biases People's beliefs are distorted by faulty perceptions.

Motivation Biases People's motivations and incentives bias their judgments.

Errors in Reasoning People use flawed reasoning to reach incorrect conclusions.

Groupthink Group dynamics create additional distortions.

c 2003-2009 Lee Merkhofer Consulting

Would you trade?

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

An example of sunk cost bias?

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

Estimating and Forecasting Biases

Figure 2: People systematically over- or under-estimate probabilities.

c 2003-2009 Lee Merkhofer Consulting

Titanic The ship that couldnt sink

Minimum payment anchor

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

Trip insurance pricing illustrates conjunctive bias

c 2003-2009 Lee Merkhofer Consulting

Bounded Awareness and Decision-Making Heuristics

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

Figure 3: Essential steps for benefiting from decisions.

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

Figure 4: General strategy for debiasing.

c 2003-2009 Lee Merkhofer Consulting

Sample Decision Aids

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

References for Part 1

c 2003-2009 Lee Merkhofer Consulting

c 2003-2009 Lee Merkhofer Consulting

Failure to See the Forest for the Trees

c 2003-2009 Lee Merkhofer Consulting

Problems with Project-by-Project Decision Making

Figure 7: The downward spiral resulting in poor project portfolios.