We live in a strange world when the most uplifting news concerns what we don’t know. Russ Roberts at George Mason University posted this gem yesterday. It comes from two weather researchers at Colorado State University who have for two decades attempted to predict the number of hurricanes we will see during each upcoming season.
We are discontinuing our early December quantitative hurricane forecast for the next year … Our early December Atlantic basin seasonal hurricane forecasts of the last 20 years have not shown real-time forecast skill even though the hindcast studies on which they were based had considerable skill.
Whoa! Say that again? As someone who works with statistics, I can tell you that the hindsight problem is under-appreciated. For example, in my field, business, we like to know things like which applicants will make the best employees. This is not as easy as it sounds, and any number of models have been proposed to explain the connection between personality, experience, age, IQ, and any number of other variables. Some models work better than others, but none are completely satisfactory.
Along the way, someone got the bright idea to chuck all of the variables of interest into a regression equation and weight them according to their contribution in predicting the success of a new hire. This is a perfectly logical way to proceed, kind of like a weather researcher seeing if his/her model would have predicted hurricanes in the past. The method actually works–for a while.
It turns out that the predictive value of such a model deteriorates over time. It will eventually become useless. You see, the computer will always find a model that explains the past–it just optimizes the relationship between the independent variables (predictors) and the dependent variable (the one we want to predict) based on past data. If the relationships do not hold over time, neither does the predictive value of the model.
There can be any number of reasons for this dismal account of weather prediction and hiring, but the simplest is a problem from physics–the Third Body Problem. Newtonian mechanics do an outstanding job of predicting how two bodies in space affect each other through gravity. In short, they will rotate around each other in a precise and stable path.
Add a third body to the problem and things get messy. The pattern becomes chaotic. No, the laws of physics have not changed, but the model describing the interaction becomes completely unmanageable. Here is a great animation demonstrating this curiosity.
Weather may be “predicted” only through modest probabilistic statements beyond about three days. This does not mean long-range predictions are completely flawed, just that we can’t hope to predict precise things like the number of hurricanes in a season. We know for a certain region about how much rainfall to expect in a month well in advance, but we can’t predict which days during the month it will rain.
The thing that makes this kind of admission notable is the courage and integrity it took for Gray and Klotzbach to make it publicly. Don’t expect a lot of media coverage on this one, by the way. Those who benefit from the global warming industry have too much at stake to let humility get in the way. For the rest of us, knowing what we don’t know can save us from a world of needless trouble.