Friday, February 20, 2009

The good ideas of Nassim Nicholas Taleb

The bad news: I have just stopped reading one of the worst books of my life since I joined the ranks of the literate some fifty years or so. I hate this book as much as I hated Superman #149, and I have already made clear what a terrible experience that was last month.

The good news: I have a week's worth of blog posts out of the reading this awful book AND recommending a book written about a quarter of a century ago that made many of Taleb's more cogent points, but in a way that was funny, insightful and tremendously entertaining.

The even better news: The much better book, which I will reveal on Sunday, is still in print, available at ridiculously cheap prices used online, and may even be in your library or local used book store. I will spend most of next week talking about the ideas from the much better book.

The back story: My dear friend from high school, Steve Lilley, recommended The Black Swan by Nassim Nicholas Taleb. Steve, who is in the business of predicting system failure for folks like NASA, found the book very interesting, and because the book is mathy, he thought I might enjoy it.

I didn't.

The good news from the back story: Steve Lilley and I are still good friends.

The reason for writing this positive piece about Taleb's book before I beat it mercilessly with a big ugly stick: As regular readers know, yesterday was my mom's birthday, and I write this in honor of something she used to do.

My mom is not a spiteful person and I don't remember her holding longstanding grudges. Sometimes she would make negative comments about someone, sometimes extremely negative, but she would end any such statement, whether small quibble or extended diatribe, with "He (or she) is probably a very nice person."

I am merely reversing the order, saying some nice things first and then launching into a soul restoring beatdown of the shoddiest piece of professional writing I have ever seen, possibly in my lifetime, and this from someone who is a blogger, a regular reader of blogs, and sees excerpts of Richard Cohen in The Washington Post quoted by Princess Sparkle Pony.

Just a thought. If a reader comes to the blog after I finish everything, this will be the last thing they read, so the order will be the same as what my mom does for those readers.

Are we ready to start? Do you have some cocoa or other comforting beverage? Very well, then, let's begin.

Good Idea #1: The future is hard to predict. Clearly, humans desperately want to believe that someone actually knows the future. Whether it's interpreting Revelations or Nostradamus or reading the astrology column or checking in with your favorite stock market advisor or sports gambling tipster, it is a universal desire in people to want to know what happens next.

The thing is, even with advances in modern science, there are only a few topics where prognosticators today are much better than people who told the future by chopping open a chicken and reading the poor creature's entrails. For a positive example, five day forecasts of the weather are better now than they were when I was a kid, because there is more satellite data and better predictive models due to lots more data and lots faster computers and a knockout tournament of predictive models until the best were adopted, adapted and improved and the less good ones were cast aside.

Good Idea #2: Normal distribution is not always your friend. There is an important mathematical idea called The Central Limit Theorem that says that if a set of data is normally distributed, you can have a pretty good idea how often different types of results will show up. The normal curve on the right in blue and white says that if you get a z-score of 1.18, about 88.1% of data will get a lower z-score and (100-88.1)% = 11.9% of data will be at 1.18 or higher. Most of the measurements you get from normally distributed data will be pretty close to average, z-scores are found by an easy formula and the percentages that correspond to the z-scores can be found on a lookup table. So far, so good.

There is a second mathematical idea, the Fuzzy Central Limit Theorem, that says lots of sets of data, both found in nature and man-made, are going to be very close to being normally distributed.

Taleb's main hypothesis is that people who predict are expecting normal distribution too much of the time, and there exist many things we try to predict that can have huge outlying data, unexpected results Taleb calls "black swans", and instead of these rare occurrences being unlikely and forgettable anomalies, they are instead the most important data in the set and have disproportionate power to affect the outcome of the future event we are trying to predict.

Example: How much will the total monetary damage from hurricanes in the United States be in 2009? How about earthquakes or wildfires? These numbers can be changed massively by one big event, and no one knows when or where the big event will happen. A large wildfire in Wyoming might not add up to one tenth or even one hundredth the cost of a large wildfire near San Diego, for example.

(Are you still holding a grudge against Wyoming, Matty Boy? Okay, yes, hypothetical. Just a little bit.)

Good Idea #3: The past is hard to understand. Compared to Good Idea #1, this one is a little less intuitive. Taleb's idea is that we rely too much on narrative, turning the past into a story. In science, this idea is stated either as "Correlation is not causation" or "Sequence is not consequence". As humans, it is our nature to look at what happened, focus on a few factors we consider important, and either ignore data that does not fit with our narrative or bring data to the forefront that confirms our narrative. Armed with that narrative, we feel confident to predict the future.

See Good Idea #1.

These are the best ideas from Nicholas Nassim Taleb's book. Absorb them well and avoid the hours you might spend, or more precisely waste, reading his book, hours which you can never bring back. This weekend, I will give specific reasons why you should not read it, effectively vivisecting it, because it deserves to die.

Given all the well-deserved fuss there is about the cartoon in The New York Post, let me be clear. I want this book to die, to become yet another bestseller thrown on the scrap heap of history. Let me wish Mr. Taleb himself a long and happy life.

No comments: