Superforecasting

The Art and Science of Prediction

By: Philip Tetlock, Dan Gardner

Narrated by: Joel Richards

Length: 9 hrs and 45 mins

Publisher's Summary

From one of the world's most highly regarded social scientists, a transformative book on the habits of mind that lead to the best predictions.

Everyone would benefit from seeing further into the future, whether buying stocks, crafting policy, launching a new product, or simply planning the week's meals. Unfortunately people tend to be terrible forecasters. As Wharton professor Philip Tetlock showed in a landmark 2005 study, even experts' predictions are only slightly better than chance. However, an important and underreported conclusion of that study was that some experts do have real foresight, and Tetlock has spent the past decade trying to figure out why. What makes some people so good? And can this talent be taught?

In Superforecasting, Tetlock and coauthor Dan Gardner offer a masterwork on prediction, drawing on decades of research and the results of a massive, government-funded forecasting tournament. The Good Judgment Project involves tens of thousands of ordinary people - including a Brooklyn filmmaker, a retired pipe installer, and a former ballroom dancer - who set out to forecast global events. Some of the volunteers have turned out to be astonishingly good. They've beaten other benchmarks, competitors, and prediction markets. They've even beaten the collective judgment of intelligence analysts with access to classified information. They are "superforecasters".

In this groundbreaking and accessible book, Tetlock and Gardner show us how we can learn from this elite group. Superforecasting offers the first demonstrably effective way to improve our ability to predict the future - whether in business, finance, politics, international affairs, or daily life - and is destined to become a modern classic.

Download the accompanying reference guide.

©2015 Philip Tetlock Consulting, Inc., and Connaught Street, Inc. (P)2015 Audible, Inc.

Great for Experts

This is not a book for laymen. This is a book for people who already think they are a pretty good forecaster and want to review ideas which may improve forecasts. Some reviewers seem to think this book did not have much in the way of specifics. I really disagree! I found it totally packed with useful ideas to improve forecasting. Here is my summary of some of the ideas presented:

Personalities:

Enjoy solving puzzles
Enjoy working with numbers
Enjoy deeply learning the viewpoints with which you disagree
Growth Mindset
Grit
Easily change your mind when facts change
Enjoy understanding current events
Embrace Uncertainty
Think probabilistically
Intelligence
Intellectual Humility
Embrace cognitive dissonance

Baseline:

Always make a testable prediction
Always specify a date range
Always specify a confidence range
Examine the question and all assumptions
Examine your own potential biases
Predict to the finest scale reasonable
Research the question (so you can identify the pertinent)
Review opposing viewpoints
Formalize development of a baseline probability.
Decompose prediction into prerequisites
Adjust baseline with prerequisites and details
Examine various scopes to avoid scope invariance

Adjust:

Set Alerts to keep informed of changes
Adjust predictions when facts appear
Adjust predictions over time even if facts don't change
Understand how new knowledge adjusts confidence (Bayes)

Combine:

Use prediction markets and other wisdom of crowds
Combine predictions with diverse others
Encourage Sharing
Extremize combined predictions
Mission Command Leadership

Learn:

Measure with something like Brier Score
Keep Records
Review errors and correct process
Monitor and Avoid belief perseverance

Now, I do have a very few nits to pick.
Firstly the author seems to have a view about uncertainty that I don't share, and I think subtlety clouds his thinking. He believes in randomness. He points out that most superforecasters don't believe in fate and instead use probabilistic thinking. He then seems to reject determinism and he believes science has shown that reality is fundamentally random. Actually reality is at least largely deterministic and maybe completely so (see deBroglie Bohm theory). I don't believe in fate or destiny, but I think the world is (at least mostly) deterministic, but there are limits to our knowledge. Probability, then, is a technique to deal with ignorance, not randomness. It seems to me the author's beliefs about randomness leads him to misunderstand the idea of the Black Swan. Black Swans are rare, but that is not the point. The point is one is ignorant of Black Swans, and does not even consider them, until they find one. In another passage the author implies that an invasion of the earth by aliens has a low probability. How does he know...because it has not happened before? In 1491 what was the probability of discovering a new continent by 1496? Experts and Experience says 0%. Reality says 100%. I say, the probability was very low (based on experience) with a very, very, large variance (almost complete lack of knowledge).

I really enjoyed and learned from this book, but it is really not for everyone. If you do forecasting for a living (or serious hobby) I highly recommend this book.

The narration is very clear and perfect for this material.
There is a PDF that is mildly interesting but not essential.

Not as valuable as the cover promises

Important nuggets of wisdom are buried in an avalanche of bloated a anecdotes, personal gloating and thinly veiled, politically biased preaching.

For those of you who just want the essence, let me summarize the only parts of the book that are actually valuable
Continuously improve, see what works, what doesn't, and adapt(duh).
Use clearly defined measurements and numbers when assessing your precision.
If you are able to spend more time in forecasting, you will do better than those who don't have as much time (again, duh)
Read the news a lot to keep up with current events
Read up on and research whatever you are forecasting
Seek out opposing views, especially those that contradict your own view
Start from a reasonable base line (called an outside view) and wok from there

And that's pretty much all that was to be gained from the book. The other lessons are explained far better by superior authors such as Kahnemann.

The rest is mostly noise and can be safely skipped over.

Terrific book

In a world increasingly dominated by sensationalized headlines, this book could not be a more refreshing or timely read. We are spiraling into idiocracy, and Tetlock/Gardner are doing their best to encourage people to keep score and learn. This book is filled with wisdom, not just about forecasting or process, but about human tendencies. Readers will feel encouraged to become more intellectually honest, to challenge the loudest pundits making vague claims, and to strive for facts that support or refute opinions instead of vice versa.

Not totally pointless

I recently enrolled in a sports prediction ring with some friends and I chose this book because I wanted to see if it might give me some insights into the art/science of prediction in general. Clearly the focus of this book is not sport at all, but I thought there must be some generalizable, transferable aspects – and there are. But on the whole I’d say the book mainly matched my existing perceptions of how to predict the future, although I did pick up a few nice nuggets along the way and there was some value in the book’s confirmation of some things I felt already knew.

The author states that there is no such thing as fate and that everything is the result of happenstance and probability. For example, I know that my being born was the random product of the circumstances of my parents’ meeting, the fact that none of their parents were killed in World War 2, the arbitrary time and place where I was conceived and the incredible odds against me winning a sperm race. I've never believed that life is predetermined by fate or destiny. So this was not news to me at all.

So how do you make good predictions? A few things help: It helps to be numerate, to diligently study the subject matter in question, to update your predictions as circumstances change, to keep an open mind.

There were a few things I hadn’t realised, such as the fact that lay people predict the future just as well, if not better, than experts - as long as they do the necessary research – and also, that groups fare better than individuals. There is a kind of ‘emergent property’ of groups whereby the totality is greater than the sum of the parts, as long as the group members interact cooperatively.

Overall I’d say that this book is worth the read, but I predict with 73% certainty that I’ll have forgotten it in 6 months’ time.

overwhelming exposition

I have a lot of respect for this author and all of the hard work that he put into this book. However I cannot recommend it. I was overwhelmed with how much exposition was in this book. It included example after example after example and contained fewer "takeaways" than what I was hoping for.

excellent work. less than compelling read. just not for me.

Research overview -- a general outcome

Imagine that you spend a dozen years working on putting together the best chorus group. It's labor intensive, you have auditions, experiment with music, and the exact mix of singers to make it great. You succeed, beating the standard chorus groups. Now that you have that trophy, you want to tell the world about it and so you do -- you write a book. Enter Superforecasting by Tetlock et. al.

Superforecasting is an account of a government funded research approach for forecasting short term world events. The author, and his colleges, successfully created a process that aggregates amateurs individuals into teams that have an exceptional accuracy at predicting world events in the next 18 months. Individuals with exceptional accuracy are assembled into super teams and their performance further improve beating paid analysts.

Tetlock readily admits that no forecasts of world events three years out have any better chance of happening better than a chimpanzee with a dart -- he even authored a study on the subject. He does an excellent job describing the process and the results. Also, to his credit, he details criticism of his process by others.

As a research overview, I give it high marks. It will appeal to those who like to understand how forecasting geopolitical forecasting study was approached by one specific, successful team -- a team that won the competition. Also, Tetlock is clear on how this was accomplished. This book is not about the general topic of forecasting. The book rated a 3 because it is stuck. It did not go full-metal-jacket statistical, nor did it show the impact of the approach. For me, it's a meh.

As Compelling as can be Expected

it is a book about forecasting so there's only so much excitement one can expect. what I appreciate about this book is the author's deliberate effort to relay the state of forecasting and offer prescription about how the science could move forward. All the while explaining why it might never get beyond chimpanzees throwing darts at a board.

combines history, practical advice and novelty

This book takes the historical problem of poor forecasting of future events and analyzes how we all might do better by looking into the practices of a few individuals who get it right much more often than the loud, public media figures do. Great read.

Powerful and understated

At first the book appears to be an examination of a novel a group of "Super forecasters", but beneath its surface it is actually an excellent commentary on the common flaws in judgment and perception that pervade human thought and society. The author uses lessons learned from his research to suggest how society can overcome these shortcomings and move past the culture of knee-jerk opinions and punditry. Overall, a very though-provoking and understated book.

Good book but for a different reason

I had to listen to this book 3 times to build a good review. The title "Superforcasting, The Art and Science of Prediction" gives the impression the reader will learn how to be a predictive genius-or at least a learned novice. Well, not so much. The authors describe a cadre of people who are numerate, curious, and have no hard biases. Pretty much describes about 2% of the population. At least in the US, probably lower in the rest of the world. Strongly held beliefs, the inability to understand logarithms and statistics and how to use them negates the unlearned. Could a person learn these things, sure. In a perfect world we would all have time to delve into these heady practices. But, cognitive proclivities (yes the word fits, look it up) and the 'get up in the morning go to work' world outside academia, the world of retired computer programmers, and the news junkies makes it unlikely your average Jane and John can master these skills.
On the positive side (see headline) as a book to build skills as a critical thinker this one should be in the library. That is, in essence, what the art of prediction is about. Looking at a situation and thinking hard about it to divine a possible outcome. As a tutorial to develop those critical thinking skill this book accomplishes that. Beware though, to much critical thinking can lead to inertia. Gotta run with a decision sooner or later. To enhance those critical thinking skills, read someting by Nassem Talib after, or before, you read 'Superforcasting'

Not much substance.

Plenty of anecdotes but little in the way of superforecasting guidance. This book didn't achieve the goal in my opinion.

book presents a new way of thinking

I enjoyed the audiobook. very easy to grasp the key points on how to reach good judgement. how our approach to forecasting and analysis needs to change to include more aggregation and more variables.

gives insight on the good judgement Project and the outcomes.

few key messages for professionals:
1) consult across sectors
2) aggregate
3) use a base score
4) incrementally improve forecasts and prediction and regularly review works
5) test the validity of data
6) don't fall for pundits :)

Bit wooly

This book took a very long time to go going and when it did I thought I might as well have got Thinking Fast And Slow as it covered so much of the same ground. There were very few worked examples really and a lot of waffle.

The book was literally read word for word from the printed version and used terms like "the figure" a few times. The course material was pathetic and very half-hearted, a few figures in a PDF with no context or explanation.

The appendix was one of the most interesting parts and I realised that the whole purpose of this book seems to be to advertise the author's organisation and to pay for his team's "super forecasts".

A practical guide on reducing cogntive biases

What did you like most about Superforecasting?

To me, the superforecasting is a perfect practical guide to exercise some of the principles laid out in Kahneman's book, Think fast and slow. In the book, the author has laid out a set of concrete strategies to make effective forecast and i especially like the idea of constant beta, which is so true for my work.

Feels like common sense

Doesn’t offer anything revolutionary and the methods offered all seems like common sense, despite linkages to other research.

Look into my crystal ball

This is the strange story of a group of people who spend large proportions of their lives "predicting" what they consider to be the probability of certain events. They are asked to ponder such unknowns as the probability that unemployment rates in certain a certain country will be more or less than a certain level on a certain date, or the likelihood that exchange rates between certain currencies will fall between a certain range. Many of the people who partake in this pastime analyse every known piece of data to collate algorithms which they input into their models to provide their estimates. Naturally, some of them produce answers that are marginally better than average and also better than mere random predictions and these people are given the title of "Superforecasters". This all seems harmless enough until you hear that the US government takes the predictions of these Superforecasters very seriously indeed and is likely to use this "intelligence" to implement major policy initiatives. This book is intriguing and chilling in equal measure. I wonder how many of them predicted the coronavirus outbreak? I suspect not many, but if their crystal balls really did give advance knowledge of this pandemic then these people really should have their palms crossed with silver.

Incitfull and entertaining

Extremely interesting and full of useful information. Although it is a bible of forecasting, it is so much more and has major sections about team work, medical and military history and much more. Highly recommended.

Extremely thoughtful, thorough explanation

The world I live in is one dominated by spitballing in preference to deliberating, boldness serving as the unworthy proxy of worthiness, and self-reflection at a level where personal change only happens when life smack one with a metaphorical brick. This book outlines all the things that are the opposite of that. I will be re-reading several chapters. At least, 7, 9, 10, and 12.

Thought provoking throughout!

A thoroughly enjoyable listen. The authors very clearly lay out what it means to be a critical thinker. The book focuses on how to use this for prediction but obviously using these life edicts you stand to make much better decisions in pretty much everything. It's certainly not an easy process but definitely worth the effort.

Hard to get through but with some interesting bits

Struggled to finish this. Some interesting facts about what makes a good forecast but it was too long and I found it difficult to keep my attention on what was being said.