Legg Mason Capital Management Thought Leader Forum 2011

Daniel Kahneman
Professor, Psychology and Public Affairs, Emeritus, and Senior Scholar Woodrow Wilson School, Princeton University 2002 Nobel Prize in Economics
Michael Mauboussin: Well, I hope you all enjoyed the lunch session. It's my honor to introduce our final speaker of the day, Danny Kahneman. Danny is the Eugene Higgins Professor of Psychology at Princeton University and a recipient of the 2002 Nobel Prize in Economic Sciences. In the last couple of decades there's been a burgeoning area of work in behavioral economics or behavioral finance, and this whole movement can be traced directly back to the seminal work done by Professor Kahneman and his collaborator, Amos Tversky, from the 1970s. Kahneman and Tversky laid the groundwork for what is now known as the heuristics and biases camp, which is essentially the study of the limits of judgment and decision-making under uncertainty. This work has been extraordinary and has earned Professor Kahneman numerous awards and honors too many for me to list, but obviously the most visible of those being the Nobel Prize. As I was writing my last book, Think Twice, I had to do a great deal of research, and what struck me as I moved from topic to topic was that I kept running into the unbelievable contributions from Professor Kahneman. He's truly a towering figure in the world of psychology and certainly one of my intellectual heroes. Professor Kahneman is the co-author of several academic works, including Heuristics and Biases and Judgment Under Uncertainty, and he has a new book that will be out shortly called Thinking Fast and Slow, and I certainly have pre-ordered it and I highly recommend it. Please join me in welcoming Professor Danny Kahneman. [applause] Professor Daniel Kahneman: Thank you. Well, there is a growing agreement, I think, and it's been very clear in the talks today, that we don't understand the world very well. Nassim Taleb, who's been mentioned a lot and is one of my heroes, is writing a book now, and what I really like is the subtitle of the book, and the subtitle is How to Live in a World That We Do Not Understand. A very good question. We systematically underestimate the amount of uncertainty to which we're exposed, and we are wired to underestimate the amount of uncertainty to which we are exposed. It is actually extremely difficult to accept how much uncertainty there is. You can do an exercise on yourself. When you think about "Harry Potter" really, you still think it must be exceptional. When you think of Mozart, was it luck that Mozart is what Mozart is, or could it have been Salieri? What we really learned today, what we could have learned from Matthew Salganiks presentation, was that there are hundreds of books that could have been just as important as Harry Potter. There is nothing special about Harry Potter within the class of books that are not failures. And the choice, and this is what Matthew was telling us, the choice is random, it is unpredictable. There is no system to it, there is no logic to it, that's just the way it happens. Very difficult to accept. And part of the difficulty of understanding how much luck, the role that luck plays in our lives and in the determination of these events, is that as soon as something happens, we understand why it happened. And this is

Legg Mason Capital Management

Page 1

Thought Leader Forum 2011 Daniel Kahneman

one of the things that Nassim went into. He has learned quite a bit of psychology, actually, and that is a very important bit of psychology, which is that we are really not as surprised as we ought to be by surprises. And the reason we are not as surprised is that as soon as something happens that we really had not anticipated, we understand it. We work it out. That's a mistake we'll never do again. Our view of the world immediately changes, and furthermore we are systematically mistaken about what we used to think earlier. A very simple thought experiment will convince you of that. There are two football teams, and it's the beginning of the season. Make them college teams. So far as you know, they're well-matched. Now, they play a game and one of them destroys the other. Now they're no longer equal. Now one of them is much stronger than the other. You will not be able to undo in your mind the thought that one of them is stronger, and somehow that you knew it was stronger. You will forget that you thought they were even. You will forget that there was no particular reason. So now that is was stronger, the fact that it won by so much is no longer surprising. That's the mechanism. So the mechanism is that by wiping out the surprises as we go along, we create an illusion of the world that is much more orderly than it actually is. One of the major influences on my thinking in that domain is Phil Tetlock. And there is the question of why do those pundits and CIA analysts do so badly? And notice that we are inclined to think that the CIA analysts do badly. We're inclined to think that the television stations or chains that rejected American Idol missed something. They made a mistake. If you think that, you have not assimilated the lesson of this morning. It's not that the pundits do badly. It's not that the television chains made a mistake. They didn't make a mistake. The world is incomprehensible. It's not the fault of the pundits. It's the fault of the world. It's just too complicated to predict. It's too complicated, and luck plays an enormously important role. In thinking about Phil's research, I came up with a thought experiment that sort of, for me at least, dramatized the amount of luck there is. Now, think of Adolf Hitler and his role in the history of the 20th century. Now, that was an important figure in the history of the 20th century. Now, at the moment of conception, it could have been Ms. Hitler. There was a fifty-fifty chance that that fertilized egg would be female. Looking back there is a one-eighth probability of a 20th century that doesn't have Hitler in it, or Stalin, or Mao. That wouldn't be the same 20th century. So you can see the role that sheer luck plays. And, you know, fewer things are more lotterylike than the fertilization of an egg. Sheer luck plays an enormous role, and we can't accept it. We cannot accept the extent to which luck is a determinant. Our mental machinery is designed to make sense of the world. Our mental machinery is designed to tell us stories, and those are stories we believe, and the stories tend to be simple. They tend to be causal, and yet, internally coherent. And the quality of those stories plays a very significant role in our mental life. So I'll talk about that a bit. I'll talk about the difficulty in integrating statistics with thinking about single cases, and I will talk of the phenomenon of overconfidence, which I think is an important phenomena in the psychology of judgment. Let me tell you a story that I've often told before, but it's a useful story because it brings together several themes that are important, to me at least. Many years ago, when I was still teaching at Hebrew University, but already working on the topic of judgment under uncertainty, I had the idea of developing a curriculum for judgment and decision making under uncertainty for high schools. And it was to be without mathematics.

Legg Mason Capital Management

Page 2

Thought Leader Forum 2011 Daniel Kahneman

So we had that idea, talked to the ministry of education, which provided a small grant so that we could work on that. I made up a team, and we went to work. And it really went quite well for a while, I think probably for about a year before the incident that I'm going to tell you about. You know, we developed a few chapters. We had an outline. We'd given one or two practice lessons, and it was a team that... I was leading it, but there was another important professor there, who was the dean of the School of Education and an expert on curriculums. And there were some teachers and some of my graduate students. And on a Friday afternoon, I don't know what possessed me, I had the idea of doing an exercise of the kind of forecasting exercises that we were thinking about. And the exercise was considering how long it was going to take us to complete our book. It seemed like a good topic. Now, we hadn't thought of that before, which is rather, you know, I'm not proud of it, but we had not asked ourselves that question. And it looked like an interesting question to ask. Now, I conducted that, and I think that's about the only thing I did right, actually, but I conducted that meeting properly. And the proper way to deal with a question like that is not to have a debate. The proper way is to ask everybody to write down their answer on a slip of paper. That's how you get independent judgments, and the quality of the average of these judgments is going to be probably better than the quality of what comes out of a discussion. And then anyway, you can discuss it, but you learn a lot by doing it that way. So we did it that way. And I put down the answers on the board, and we had a distribution, and it was all between 18 months and 30 months, between one-and-a-half and two-and-a-half years. All of us, myself included, and Seymour, the dean of the School of Education, was in that range, too. And then, I had an idea. I asked Seymour whether he knew of other teams like ours that had developed a curriculum where none existed before, a truly original curriculum. And he said he knew several. This was a period of intense ferment, actually, in the world of education. Lots of people were generating new curricula in the late '70s. And so, I asked him, "Do you know enough about these teams so that you could locate them, approximately where we are, in terms of the progress that they have made?" And he said yes, he could. It was obvious what to do next, so I asked him, "Well, how long did it take them to complete the book?" It took him a while, actually, to generate the answer because he was embarrassed by the answer. He said, "You know, actually, it had never occurred to me before, but not all of them completed the book." He said about 40 percent never finished. Now, that was a completely new thought for us, the idea that we would fail just hadn't occurred to us. And it was something that was under control, clearly a manageable task. We were going to do it. Now, 40 percent of teams had failed. Then, I asked him, "Well, and those who finished? How long did it take them?" And he said, "I can't think of any that took less than seven years. And I can't think of any that persevered more than 10. So, somewhere between seven and 10 years." Now, this is a very rich story, in terms of what you can pull out of it. But one of the things you can pull out of it is about Seymour. First of all, he had the information in his head about the statistics. And he didn't use it. It's not that he decided not to use it, it didn't seem relevant. When he tried to forecast how long it would take us to complete our task, he did what people do. He extrapolated, you know, he imagined, he looked into his crystal ball. It was all about us and about our project. He was dealing specifically with our project. I call that the inside view, looking at our problem, at our forecasting problem from the inside.

Legg Mason Capital Management

Page 3

Thought Leader Forum 2011 Daniel Kahneman

What I had him do, by asking him the question of the other team, was to take what I call the outside view, that is, view our case as a specific instance of a broader category. This is statistical thinking. It's a very different kind of thinking. And there are several observations to be made about that. The first is, the inside view comes much more naturally. The natural way to think about things is what Seymour did and what all of us did. We thought about how long it would take us to complete the work and we imagined, we used our imagination to forecast the outcome. Turns out, that's a miserable way of doing it. The outside view is clearly the correct approach to a case like that. I should add, in case you are curious, there was a book. The book was finished. The book was finished eight years later. I was no longer there. Nobody could have forecast all the vicissitudes that caused that miserable project to take eight years. Furthermore, by the time the book was finished, the Ministry of Education had lost interest in the project. It was never used. So, it was a complete waste of time. But the point is that all the other teams that had been in Seymour's mind, I'm sure they all made the same mistake. They didn't know the odds they were facing. They didn't know that their probability of failing was 40 percent and that it might take them between seven and 10 years if they were lucky and successful. They had no idea, because they were all thinking the way we were thinking. They had a plan, they had an idea of how things should work, and they were using their plan as an anchor to make their forecasts. We call that the planning fallacy, by the way. The planning fallacy, which is endemic, is, you have a plan, which tends to be, by and large, a best case scenario. Then you adjust it. What happens, you have to think about how plans fail, and the failures of plans are not predictable. I mean, it's clear that something will go wrong, but you don't know, usually, what will go wrong. There are many, many reasons that can cause a project like that to take eight years. You can't anticipate all these reasons. To some extent, you could think of them as luck, they are noise in the system. They are unpredictable. Well, let me first tell you that there have been developments. The outside view that practice in which you have a forecasting problem, and you look at the statistics that now has a name, an official name, it's called "reference class forecasting." It's got a champion. His name is Bent Flyvbjerb and he is a professor at Oxford. And it is now, actually, the recommended practice, by, I think, the American Planning Association there is such a thing which passed a resolution endorsing reference class forecasting. That is, when you make a plan, try to take the outside view into consideration, and see if the plan has any realism to it. Flyvbjerb has collected a lot of information about plans and their realization. And of course, we are not surprised to hear that the plans are typically wildly optimistic. And we now have numerical information about certain classes of plans. He has studied, in particular, transportation plans. And the forecast of both utilization and cost and time are systematically wrong. Now, this is not always innocent. I mean, some people deliberately make promises or make optimistic plans in order to suck the resources of the organization. But even when people do their best, they are going to underestimate the role of luck and uncertainty in their outcomes. The story suggests a way of making predictions, which sometimes can be useful. Oh, I forgot the detail, an important detail, in my story about Seymour. When he had told us the bad news, seven years, 10 years, and so on, I was grasping at straws. And so, I asked him, "When you think of us, in comparison to the other teams that you just told us about, how do we stack up? Are we stronger, are we weaker?" And that I will never forget, because he was very quick this time. He said, "I would say we're below average. But not by much."

Legg Mason Capital Management

Page 4

Thought Leader Forum 2011 Daniel Kahneman

Now, I hope you won't ask me why we continued, because that is why we went on with the project, which obviously, we should have stopped that day. But I can't help myself but tell you, this is truly the most idiotic part of the story. Of course we should have quit. And to try to explain why we didn't quit, is again, to go back to this illusion of understanding and to the difficulty we have with statistical thinking. I mean, I believed it. It's not that I didn't believe what he told me. It just didn't seem all that urgent to quit the project just because of some statistical facts. You have the sense that the statistical facts are not germane, are not pertinent to you. Base rates just don't matter. Now, if you want to make a prediction, and there are many cases where this is going to apply, you can take the outside view. And what the outside view tells you, if you do it right, is it generates what I would call a baseline prediction. The baseline prediction is what you would say if you only knew that the case belongs to that category, and nothing else. So, that's the baseline. And in this case, clearly, the baseline was closer to the mark than our best estimate. Once you have the baseline, adjust. So, if we had been a lot stronger or a lot weaker, the rational estimate would have moved a bit from the parameters that he proposed. And that notion of a baseline forecast is a very important notion, in which we try to ignore the information we have about the case, because it is intrusive, because we are likely to overweight the information that we have. That's a very important part of the story. The information we have makes a story. And as I indicated earlier, the brain is wired to make up stories and believe them. The statistics do not make a good story. And for an interesting reason, by the way. The statistics are not causal. You know, it's just numbers. A story is causal, there are causes, there are effects, there are things that cause, bring about other things. That's how we make sense of the world. Statistics just doesn't compete. And so, I look back at that incident and ask how could we ignore what he was telling us, but we did. We actually ignored it. And it was because our sense of progress, the way we felt about that particular incident seemed so much more compelling than cold statistics, that we couldn't bring ourselves to follow the statistics. And this happens a lot, in the difficulty that people have in integrating statistics with causal stories. OK. I will tell you a riddle. It's better if I project it, but I think you can follow me. And anyway, you're not supposed to get it right, so it really doesn't matter. There is a town in which 85 percent of the cabs are green and 15 percent are blue. And there was a hit-and-run accident involving a cab at night, and there a witness. Conditions of visibility were so-so, and the witness basically said, "I'm 80 percent sure that it was a blue cab, one of the smaller company." And people are asked, presented that problem, and they're asked what is your probability that the cab involved in the accident was blue. Hundreds of people have been asked this question, and the most frequent answer is 80 percent. You know, there was a witness there. He was tested, and actually, you know, we say that under the visibility conditions, he was 80 percent accurate when he said green or when he said blue. They go with the witness. Actually, this is the wrong answer. The correct answer is slightly less than 50 percent that it's blue, because the base rate continues to be relevant. The number of cabs continues to be relevant. Very hard to see it. You don't see it. I'm not going to explain it now. I mean, some of you do, but that's because you studied base theorem somewhere else, but if you didn't have the mathematics, why not trust the witness. Now, let me tell you a variation on that story, and all of you, I think, will immediately feel the difference. There are two cab companies in the city: 50 percent of the cabs are green and 50 percent of the cabs are blue. But 85 percent of the accidents involve green cabs. Now, there was a witness, and the rest of the story is the same. Do

Legg Mason Capital Management

Page 5

Thought Leader Forum 2011 Daniel Kahneman

you feel the difference? Nobody wants to ignore the fact that 85 percent of the accidents are caused by green cabs. I mean, the drivers in that company must be insane. This is the way that people see it. You immediately infer a causal propensity. You make a causal inference from that statistic, and now that is used. When people combine that with a witness, they get roughly the correct answer. So there is a real profound difference between the way our mind deals with arbitrary statistics and with causal stories. And sometimes statistics enable us to make a causal story, because, as in this case, you immediately felt that something must be wrong with the green drivers, and then you use it. Now, let me talk about an error of prediction, a way of prediction. I'll give you a detail, a fact. So, it's about Julie, she's a graduating senior. I'm going to ask you to guess her GPA, and I'm going to tell you one fact about her. She read fluently when she was four years old. That's all I'm going to tell you. All of you have a GPA in mind, and I could do magic, not only Apollo Robbins. I know what GPA you have in mind. I mean, it's high. It's really quite high. It's probably is a 3.7, somewhere there. There is not much variability either, because I think I also know how you do it, and this is how you do it. You take the information that she read fluently at age four, and because of the capability of our intuitive mind, that immediately translates because we know the world, into a sense of how extreme is it, how precocious is it to read fluently at age four. Where did that put her, Julie, on the distribution? What percentile is she at, for precociousness of reading? And you have an idea. Furthermore, my guess is that your idea is pretty good, because this is something we do learn about the world. We learn frequencies. We're quite good at learning frequencies, so we know the age at which children learn to read. We know how exceptional it is, not as exceptional as reading fluently at age two-anda-half, but it's good to read fluently at age four. Now, what did you do to get the GPA? Very easy. You picked a GPA that is as extreme as her reading ability. That's what people do. There's a lot of evidence that this is what people do. This is crazy. This is absolutely wrong. That's intuition. You know, you didn't deliberately do it. There was a GPA that came to your mind when I told you about Julie. And that GPA is the GPA that matches, because there is another facility in our brain, and in mine, which is... I call that intensity matching. I'm not the first one to deal with it, but the idea is that you can take any dimension, which is an intensity dimension, and sort of match how intense it is to almost any other dimension. So I could ask you, if among the incomes of teachers, how high an income is as extreme as Julie's reading ability? You'll give me an answer. I mean, something will come to your mind. Among children who are 10 years old, how tall a child would have to be, you can do it. People really do it quite well. They match across intensity dimensions, which enables you sometimes to answer a question with a lot of confidence, when it's the wrong question. That is, you've been asked about a GPA, but you really answered a question about her precociousness, without knowing that you had switched from one to the other. Now, this is too extreme. You're not supposed to predict, to make extreme predictions on the basis of weak evidence, and this is weak evidence. I mean, the correlation between reading fluency at age four and GPA, at best it's 0.3 or 0.4. My guess is it would be lower, but people predict as if the correlation was perfect. And that is another characteristic of the stories that we tell ourselves. The quality of the story determines how we predict, and it determines the confidence we have in our predictions. We judge it by the quality of the story, but you can take a wonderful story on the basis of evidence that is false or unreliable or very, very sparse.

Legg Mason Capital Management

Page 6

Thought Leader Forum 2011 Daniel Kahneman

It doesn't take a lot of information to create. You created a story out of the fact that Julie read fluently at age four. It's something that happens automatically in our brain. And the best you could do was to match it to the question that was asked and that's the answer that people give. We are not wired properly for statistics. We're really good at telling stories, but we're not wired properly for statistics. Now, I've been going on a theme of confidence and the confidence that people have. And it's obvious from what I say that we can expect people to be way overconfident, because they have that ability to tell good stories, and because the quality of the stories is what determines their confidence. The extent of that overconfidence is actually quite remarkable. There was a study reported by my friend, Dick Thaler, in a column, I think, in the New York Times, but you may not know it. The business school at Duke University conducts a survey of the CFOs of Fortune 500 companies, and they have a substantial sample. And they ask them every year to state their confidence interval, an 80 percent confidence interval for I think it's the S&P [500] Index for the next year. So they state their confidence interval. Now, that goes back to something that Phil was talking about, to the issue of calibration. If they were properly calibrated, then 80 percent of the time the true value would fall inside their confidence interval, and 20 percent of the time it would fall outside their confidence interval. That's not what happened. Actually, instead of 80 percent falling inside the confidence and 20 outside, it's 36 percent for inside and 64 percent for outside. The confidence intervals are ridiculously narrow, if you compare it to what people know. Now, that's not the only kind of information we have. We also know that the CFOs have no idea what they're talking about. When you look at the correlation between their predictions about the S&P 500 and what actually happens to the S&P 500, the correlation is actually not zero. It's slightly negative. I mean, they're a little worse than chance, not by much. They don't know a thing. So I asked the people who did that study to carry out a computation and to figure out the correct confidence interval that you should give about the S&P 500 when you know as much as these CFOs do. And the answer is there is an 80 percent probability that the S&P 500 outcome will be between minus 10 percent and plus 30 percent. You are meant to smile when I say this, because this is ridiculous. I mean, a CFO who would say that would be kicked out of the room. You're supposed to say something, to say a little more than that. I find it astonishing, you know, the width of that confidence interval. Clearly, it is much, much wider than I expect, and I've been studying that problem for a long time. So there is a vast amount of overconfidence. CFOs have it. All of us have it. And it's related to our ability, to a storytelling ability, which in turn is related to the belief that we have that the world can be understood and that outcomes can be forecast. What I really think is a question you should ask yourself, what did you really learn about Harry Potter today? What did you really learn about Mozart? I can't believe that Harry Potter is not exceptional. I don't know about you, but I can't believe it. I can't believe that Mozart is not exceptional, but Mozart, I know that Mozart is exceptional. I've listened to him so much and I love him so much the idea that it might have been Salieri is scandalous to me. I find it shocking. The world retrospectively makes more sense than it should given how little we understand it. I found it remarkable, actually, the extent to which we seem, all of us today, the last three speakers, to converge on this conclusion. To my surprise Phil seems to be the more optimistic among us. He's searching for a pattern. I, under his influence, turned into a radical pessimist. I think you probably can forecast short and medium term. Long term, I think, is completely hopeless because long-term I think the world is chaotic and random. Many phenomena that look to us highly regular are, in fact, chaotic and random.

Legg Mason Capital Management

Page 7

Thought Leader Forum 2011 Daniel Kahneman

It's impossible for a publisher. I'm publishing a book so I've been exposed to a lot of forecasts. It's absolutely impossible for your editor not to feel that there is a true answer about whether this is a good book or not and that if the book is good, it will be recognized as good. No such luck. At least I don't believe it. So I hope I'm lucky, because you really can't tell. You can predict failure, but that was a profoundly important statement. You can predict failure, but you can't predict success. We can discard the bad, but how much we can do with the good? Much less than we think. On this happy note I'll close. [applause] Audience Member: If you don't mind me asking, do you have any active investments? Do you invest in individual stocks or funds? All index? Professor Kahneman: No. You know the answer. [laughter] I'm inconsistent and incoherent, but not to that extent. No, I don't. [laughter] Audience Member: I want to just ask one on the prediction of failures. I'm not quite sure what that means. So when you say you can predict failure, but not success, since most books are rejected, most films are not made, I'm not quite sure. The things that were rejected, the Harry Potters and all, they succeeded. How do we understand that notion? Professor Kahneman: There was that graph that [was shown by Matthew Salganik that] demonstrates that when you look at an independent assessment of the quality of in that instance, songs the songs that are panned in the initial judgment are panned by everybody. There is agreement across people in populations on the bad songs. There is much less agreement on the good songs. Because there is less agreement on the good songs, the selfreinforcing, the snowball effects that Matthew was talking about, which amplify the role of luck, are much, much larger in determining whether a book or a song that was good could turn out to be marvelous or truly inferior. That is determined by luck and that is really very hard to accept. I have such strong opinions about the songs I hear and the books I read. I know if they're good or they're bad. Well, it turns out that's not the way it works. Audience Member: Just a follow up or maybe ask Matt [Salganik] on that. So the independent thing, that was independent against the ones where there was social stuff, but you didn't run a bunch of independents to see if they all agreed on that, right? Just a one off instance? Matthew Salganik: There was only one independent world. Because the behavior of those people is independent, you don't need to do multiple replications of it, because it would be the same in each of those. Professor Kahneman: You could split the independent group and do a split half reliability. Salganik: Yes. That's actually how we got the confidence intervals around the unpredictability from the independent condition was this repeated splitting, because they're independent. But it's also the case, as you point out, a lot of the things don't ever get made. In this experiment all 48 songs got a chance and in a lot of cases a lot of things don't ever get a chance to appear. So how good are we at saying the things that never appear, whether those are going to be good or not? We don't know because we never get to see that data. So our ability to understand that could be really biased, because we have never seen outcomes for those things. Professor Kahneman: This is, by the way, a very general effect. That is, you predict success in graduate school among applicants, but the applicants are selected and self selected, so they are a very selective group. Tests are not going to be very successful within that group. If you included all the population, you would do a lot better, but the relevant test is among the candidates. You accept whatever selection process is in place, and within that, when you are in the top range, that's where luck plays an enormous role, and a deeply counterintuitive role. Yes.

Legg Mason Capital Management

Page 8

Thought Leader Forum 2011 Daniel Kahneman

Audience Member: What about the one that's way outside the norm? The one that comes to mind is Rocky. [Sylvester Stallone] was rejected 1,013 times before somebody finally said, "Do it," and most of the rejection was because they didn't want him to play Rocky. Where does determination by the party that's pushing, where does that play into this? Professor Kahneman: The amount of determination is part of the uncertainty that you're anticipating. It makes a lot of sense after the fact. Look, every success story after the fact makes sense, that's the problem. There's something I'd like to add about hindsight, just to strengthen the point. It's something that many of us are thinking about. You hear some people say that so-and-so knew that the crisis was going to happen. Some people have read Michael Lewis' The Big Short, and they say there were those people, and they knew the crisis was going to happen. I think that's a scandal. The scandal is the use of the word "know." They didn't know the crisis was going to happen, they thought the crisis would happen. Very different. When we use the word "know," we use the word "know" on something that is true. It wasn't true before it happened, it was just a thought that they had with a certain amount of confidence. What makes it a scandal? When we use the word "know," we create the illusion that it was knowable. See, they knew, so it was knowable. No, it was not knowable. Equally intelligent people didn't know that. And Michael Lewis, I don't think, makes that mistake. "The Big Short" is about a few people who got rich, because they thought that and, indeed, it happened. It doesn't make them smarter than the rest of the world that didn't know it. It makes them, in my opinion, luckier. They happened to be right in a way that paid off in a big way. Audience Member: Given that we've been hearing now that it's much easier to predict failure than it is to predict success, should we be better at shorting stocks or knowing that stocks are going to go down? Is there any data that might suggest that managers who have the ability to short may outperform long only managers? Professor Kahneman: I don't know of any such data. It's an interesting question, actually. I actually don't know. The problem with stocks is that there is a confusion between predicting what the firm will do, or what the company will do, and what the stock will do. Predicting what companies will do is much more like predicting success and failure, if you have the data. And my guess is that failure is easier to predict than success at the level of companies. At the level of stocks, that is to justify shorting, you're asking me whether the stock is properly valued now, so that it's worth shorting, and I have no idea. We really have to keep those two apart, and by the way, it's a very common mistake. It's very difficult to separate in our own thinking stocks from companies. What we believe we know about companies seems plausible to transfer to our knowledge about stocks, and I don't think that's... there's a key element missing. Audience Member: I wanted to go back to the gentleman's question that incorporated Rocky, and we talked earlier about Harry Potter. Intuitively, it would seem that the amount of determination is part of the uncertainty. But if you could find that author, who after getting rejected seven times, is going to go and do it the eighth but you're saying you can't do that. As we look at companies, not stocks, over many years, we come to know the executives, the CFOs who know nothing that you talked about. But we see CEOs and CFOs make good capital allocation decisions, lie to us, observed them over a period and number of cycles. Isn't there some edge to be gained by figuring out, and again, we don't know, but based on our observations, the hardest working CEOs, and people that are doing what we would believe is the right thing, that leads to value creation? Professor Kahneman: I was surprised. Actually, I researched that thing. Not myself, I looked at the literature on that, and being a radical pessimist about prediction, I was a bit surprised. You can learn a fair amount about companies from the CEOs. CEOs have a style, and the style transfers, and if the style is appropriate, the company is likely to benefit. Now, a couple of remarks about that. The first is it's a small effect. I mean, the correlation between the quality of the CEO and the quality of the company is about like the correlation between

Legg Mason Capital Management

Page 9

Thought Leader Forum 2011 Daniel Kahneman

precociousness of reading and GPA. It's about I would think 0.3. I'll tell you what that does, what 0.3 does, just to bring it to life. If you had pairs of companies and you looked at their CEOs, and the companies were ranked one is better than the other, and the CEOs were ranked, one is better than the other, in what proportion of cases does the ranking match the better CEOs with the better companies? 60 percent of the time is a rough estimate. So you do better than 50 percent, but you don't do enough to justify what the business press does with successful CEOs, which is to give them, you know, you might think Steve Jobs did it all by himself. Audience Member: Hi. You said very directly and very forcefully that there's kind of more luck in the system than we generally acknowledge. Does that mean that from a financial point of view, or maybe other points of view, that luck is kind of underpriced? Have you seen instances of that, either in society or in financial markets or anything else? Professor Kahneman: I'm not sure I understood you. Audience Member: OK. If there's more luck than most people acknowledge, then luck happens more often, but would that mean that then in terms of price, if you're going to price luck, it's kind of underpriced. Do you see that at all in society or in financial markets or anything of that sort? Professor Kahneman: No, and it's not clear to me how you would price luck. I mean, luck could be good or bad, so I'm not exact... Audience Member: Volatility maybe. Professor Kahneman: Nassim Taleb draws a direct inference, and maybe that's what you mean. He says insurance for extreme events is still cheap. You should buy it. That's his line. Insurance against intermediate events is too expensive. That is his belief on the basis of his analysis. So that may correspond to what you have in mind. Audience Member: You said earlier that you believe that long-term predictions are less possible than short-term predictions. Now, in our business, there's some long-term direction of random events, like the growth of the economy. So if I invest in the economy and have a 10-year time horizon, aren't I always sure that I will be worth more in the end? Professor Kahneman: I should've added a qualification, something that Phil Tetlock and I've been discussing for more than 20 years, I think. When I mean that experts cannot predict, I mean experts cannot predict better than the average reader of the New York Times. And as the average reader of the New York Times knows a lot about the world, including that, until the world self-destructs in one way or the other, the economy will grow by and large, yes. Now, the question is what is the value added of experts in long-term predictions beyond that background? And on that I thought that Phil's answer, the answer of his 2005 book [Expert Political Judgment] was conclusive none. So that's the background. It's not that the world is completely unpredictable. It is that some of it is trivially predictable, and to get beyond the trivial, that seems to be impossible. Audience Member: How much of your success do you attribute to luck? Professor Kahneman: I can honestly say I'm on record of saying that, because I got a big prize a couple of years ago, not the Nobel. I got one that to me means more. It's the career achievement from the American Psychological Association, and I had to give a speech of thanks. And I was thinking of speeches of thanks at the Oscars, and people thank all sorts of things and all sorts of people. They thank their mothers, they thank their friends, and I had a mother to thank and I had friends, but nobody ever thanks luck and they should, and I did. So I consider myself an extraordinarily lucky person.

Legg Mason Capital Management

Page 10

Thought Leader Forum 2011 Daniel Kahneman

Audience Member: I asked you that question because there's a famous golfer, I can't remember if it's [Ben] Hogan or who it was, who said, "The harder I practice the luckier I get." So you have individuals that create an environment for themselves, whether they're managing money or they're running a company, where they rely less on luck and more about creating an environment for success. I wouldn't say that luck doesn't exist, but I think that it's less a factor. Professor Kahneman: It's very easy to confuse what the effects are, especially when you're discussing correlations. What is almost certainly true is that if he practiced less, he would do less well. How well he's going to do when he practices a lot, that is where that graph with enormous variability comes in. Lots of people practice a lot, and that's where luck comes in. So yeah, you can generate your own bad luck easily. Generating your own good luck, lots of people are trying to do that and not all of them succeed. Mauboussin: Yes, exactly. One of the challenges of the inside/outside view is selecting a proper or an appropriate reference class, and this is one of the things where you can slice and dice reference classes. So from a practical point of view, how would you think about reference classes, and how fine do you parse them, how broad are you allowed to accommodate and so forth? Professor Kahneman: Yes, for reference class forecasting, obviously the key thing is what class do you select? And it's not always obvious. So actually in the first article that we wrote about that, Amos Tversky and I, we discussed what is the reference class for a book? So is it books of the same genre, or is it books by the same author? And there are many possibilities. And there is no definitive answer, except to try it multiple ways. That is, if the different outside views do not agree, then you know less than you think. That is very useful information. But if there is some convergence between different ways of looking at the problem from the outside, you ought to have some more confidence in it. And there is never any perfect reference class. Mauboussin: I have a bunch of other questions, but one of them... You've probably given talks like this many times to many audiences like this. If you had to offer a few bits of advice for people who are trying to navigate the world as we've described it, to allow them to improve or maybe make fewer mistakes, are there a few things that they might do to...? A journal, for example? Professor Kahneman: I have an excellent bit of advice. You should read, I think it's the March issue of the Harvard Business Review in which I have an article on this problem, with a couple of collaborators, which talks about the outside view and it talks about Gary Klein's premortem, and about other ways of testing for biases. And that article includes a little more than I know, so it certainly has everything I know. And it was written with two people at McKinsey who know more than I do on the practice of it. Yes? Audience Member: You just mentioned Gary Klein. I wonder if you could comment on his work. I think Michael's been critical of some of Gary Klein's work. Professor Kahneman: Gary Klein is a psychologist who is a proponent of intuition, and he has those wonderful stories about the cardiac ward nurse who comes home and sees her father-in-law, and she says, "We're going to the hospital." And he says, "Why? I feel fine." "We're going to the hospital." She doesn't know why, but she grabs him, they go to the hospital, he goes directly into the operating room, and she has saved his life. Those are stories we love, so clearly, and they are true stories. I mean, chess players have intuitions about the chess board, so intuition exists. Valid intuition exists. Now, mistakes exist. So Gary Klein has spent his professional life studying good intuitions. I've spent a lot of my professional life studying bad intuitions. And so about 10 years ago, I approached him and I said, "Let's write a piece together where we'll try to see when intuition is good and when it is bad." And actually we did. We thought it was easy, but it took us seven years. [laughter]

Legg Mason Capital Management

Page 11

Thought Leader Forum 2011 Daniel Kahneman

Professor Kahneman: And we quarreled a lot and we negotiated a lot and we ended up really quite good friends. And the question of when you can trust an expert who is making a claim intuitively, and when you should not trust an expert, I think we have an answer that both of us agree on, although he still loves intuition and I'm still kind of skeptical about it. And he hates biases. You mention the word "bias," he starts shaking. But we agreed on some tests that you can apply for when intuition is likely to work. Mauboussin: I'm curious, I know you've been working on writing your book, but are there any books you've read in the last year or so that you've found to be fun or instructive and have helped your thinking? Professor Kahneman: Well, your book is pretty good. I would say I liked it. Mauboussin: Only pretty good? Im just kidding. Professor Kahneman: Yes, it's good. I think. Yes, I read that with profit. But I know that's not the answer you were looking for. What have I read that influenced me? Mauboussin: What about the Josh Foer book? Professor Kahneman: Yeah, I can mention that. That I loved. I recommend it to everybody, [Foers] Moonwalking with Einstein. It's a very nice book about the functioning of human memory, and it's highly relevant to the book I have written that is coming out. So, yes, that book I read with profit and enjoyment. Mauboussin: With that, thank you very much, Professor Kahneman. [applause]

Transcription by CastingWords The comments, opinions and any forward predictions expressed in this commentary reflect those of the speaker as of the date of this commentary. These views are subject to change at any time based on market or other conditions, and LMCM disclaims any responsibility to update such views. These views may not be relied upon as investment advice and, because investment decisions for clients of LMCM are based on numerous factors, may not be relied upon as an indication of trading intent on behalf of the firm. The information provided in this commentary should not be considered a recommendation by LMCM or any of its affiliates to purchase or sell any security. The information contained herein has been prepared from sources believed reliable but is not guaranteed by us as to its timeliness or accuracy, and is not a complete summary or statement of all available information. This commentary is intended solely for our clients, is for informational purposes only, and may not be publicly disclosed or distributed without our express written consent. Legg Mason Capital Management ("LMCM" or "the Firm") consists of two legal entities doing business under the LMCM name: Legg Mason Capital Management, LLC ("LMCM"), and LMM LLC ("LMM"). Each is a subsidiary of Legg Mason, Inc.

Legg Mason Capital Management

Page 12