Expert: All Predictive Models Are Wrong

The headline of this post is somewhat misleading but the point is that predictive models are too-even when used properly. Says catastrophe risk management expert Karen Clark: "Model users frequently forget that all models are based on simplifying assumptions, and therefore all models are wrong."Clark's comment appeared in a release by her eponymous firm consulting firm, Karen Clark & Company, announcing a report on the performance of so called "near-term" hurricane models. These models have suffered heavy criticism from Florida government spokesmen, as we reported early last year. The near- or medium term models were designed to replace standard models that failed to predict the anomalous 2004 and 2005 hurricane season.

In late 2006, Florida Governor Charlie Crist said RMS's medium-term model was based on "unscientific" assumptions and amounted to "big insurance" taking advantage of the people of Florida.

Karen Clark & Company's statements avoid such rhetorical flourish, but the conclusions of the consultant's report might offer some vindication of Governor Crist's pique. The near-term models, introduced by AIR Worldwide, EQECAT and RMS in 2006, predicted cumulative insured losses of 37.2 billion, $40.8 billion and $42 billion respectively-seriously short of the actual cumulative losses of $13.3 billion during the specified period.

"With the close of the 2008 hurricane season, and three years into the application of near term hurricane models, it is a good time to evaluate the models' performance," commented Karen Clark. "While it is still too early to make definitive conclusions about the near term models, with insured losses significantly below average for the cumulative 2006 through 2008 seasons, initial indications are there is too much uncertainty around year-to-year hurricane activity and insured losses to make credible short term predictions."

While Clark's critique appears fundamentally to be about the statistical limitations of models, last week members of a panel at the Casualty Actuarial Society's (CAS) annual meeting inveighed against the abuse of risk modeling as an important contributing factor to the financial crisis. The actuaries criticized both over-reliance on the models and the use of faulty or incomplete assumptions.

According to a CAS report, David Ingram, senior vice president, Willis Re, "observed that the models missed that differences in the size of down payments would impact experience; increasing the pool of subprime borrowers would alter the characteristics from the select group of the best subprime borrowers of a few years ago to the riskier group of more recent years; changing the terms of mortgages from fixed rates to gimmicky rates would have a major impact; and there could be a national risk from subprime mortgages."

The panel's moderator, Thomas Hettinger, managing director, EMB America LLC, commented that opinions about the effectiveness of models would polarize into two camps, the "naysayers," who would opine that models simply don't work, and those who will embrace approaches to modeling as the methodology continues to evolve.

Both the Karen Clark & Co. and the CAS commentators contribute a worthwhile message of caution about predictive modeling: it has its uses but those uses are limited, by the quality of input data, assumptions and the simple fact that they represent nothing more than probability. As Ms. Clark notes, "Models are an attempt to replicate reality, but they are not reality."Near-term hurricane models introduced by AIR Worldwide, EQECAT and RMS in 2006, predicted cumulative insured losses of 37.2 billion, $40.8 billion and $42 billion respectively-seriously short of the actual cumulative losses of $13.3 billion during the specified period.

Anthony O'Donnell has covered technology in the insurance industry since 2000, when he joined the editorial staff of Insurance & Technology. As an editor and reporter for I&T and the InformationWeek Financial Services of TechWeb he has written on all areas of information ... View Full Bio