Effective model selection starts with model appropriateness

When I started my career in catastrophe risk management, model selection was often treated as an operational rather than technical decision.

The focus was on finding a provider that fitted existing systems, with less attention given to whether the model was the most appropriate for the decisions it would inform.

That mindset can still be seen today, with some organisations viewing catastrophe models primarily as tools for satisfying regulatory or rating agency requirements.

What’s exciting though is that over the last few years we’ve seen the rise of technical experts who, driven by rapidly changing trends in exposure, technology, human behaviour and climate, have shifted the conversation. Increasingly, organisations are asking harder questions about scientific credibility, predictive skill and whether models are genuinely fit for purpose.

I’ve spent much of my career looking at how organisations evaluate catastrophe models and I’ve watched this shift first hand. However, while catastrophe models have evolved significantly, evaluation practices haven’t kept pace.

A Shift in Emphasis

Most organisations still build evaluation frameworks from scratch, developing bespoke question sets, reviewing extensive documentation and repeatedly asking vendors many of the same questions. The emphasis remains on gathering information rather than determining whether a model is appropriate for the decision being supported.

That diversity was understandable in the absence of accepted standards. Over time, it has generated a wealth of practical experience across organisations, perils and use cases. The opportunity now is not to replace them, but to bring together what they have collectively taught us and develop a more consistent way of thinking about model appropriateness.

A model selected to support capital assessment may provide the most credible view of aggregate portfolio risk, while an underwriting team may place greater value on location-level differentiation and hazard detail. As a result, no single catastrophe model will necessarily be optimal for every use case.

Cat Modelling “Off Ramps”

A colleague of mine recently described different use cases as taking different “off ramps” along the catastrophe modelling pipeline. It’s a useful way of illustrating that different decisions draw on different parts of the modelling process and therefore require different model capabilities, but I’ve increasingly come to think that even this analogy is still too linear.

I see catastrophe models less as a single end-to-end process and more as a collection of interconnected components, each with uniquely valuable information that can provide valuable and different insights in isolation and as part of the whole. The use case determines which pieces of the puzzle matter and how they should be brought together. Rather than asking every user to travel the same road, we should be assembling the information around the decision we’re trying to support.

Flood Risk Modelling

Take flood modelling as an example. One provider may incorporate highly detailed local terrain and flood defences, while another prioritises methodological consistency across countries. Both are technically valid; they are simply answering different questions. This thinking became the foundation of our work with Oasis LMF. Over the last 18 months we’ve partnered with practitioners across the market to develop practical guidance for assessing model appropriateness.

One of the more interesting outcomes was that even those of us involved in leading the project evolved our thinking as the work progressed. What started as a narrow exercise focused on a few core questions, such as reinsurance layering and risk selection, became much more nuanced as we considered the risk appetites of different organisations, the complexities of specific perils and regions, and the growing range of decisions catastrophe models are now being used to support. It strengthened our belief that model appropriateness and model evaluation is always context dependent.

And if model appropriateness depends on context, then we need to be very deliberate when communicating what we mean by a standardised approach to model selection.

Consistent Processes

Standardisation should not mean a fixed set of questions applied repeatedly across every evaluation. Instead it should provide a consistent process for defining the questions different decision-makers are trying to answer, and the model capabilities required to answer them. Only then can we meaningfully evaluate whether a model is appropriate.

This challenge reminds me of an observation by the Roman playwright Terence: quot homines, tot sententiae, or “as many people, so many opinions.” The same is true of catastrophe modelling. Every team is asking different, but most often equally valid, questions of the same model. Our challenge isn’t to reduce that diversity, but to help users navigate it by collapsing an almost infinite set of possibilities into the handful of model capabilities that matter for the decision at hand. Only then can model selection focus on the question that matters most - can my model provide appropriate information to improve my decision?

By Tom Philp, founder and CEO, Maximum Information