Policymaking in an uncertain world

Policymaking is difficult: uncertainty is inevitable. There is imperfect information regarding the consequences of decisions, limited capacity as human beings to process complex information, and the policy arena is a contested process, with competing objectives and value judgements. But, nonetheless, decisions still need to be made. Modelling (if done well) can help us cope with complexity and guide us through the evaluation and decision making process.

What is a model?

Models are computer simulations, designed to help do the math when our brain can’t cope. They are intentionally a simplification of reality designed to focus on the essentials of a particular situation. Models vary in sophistication, from our sat nav, which has programmed geographical maps, to give us the best route home, to macroeconomic models, which programme economic theory about how the economy works, to estimate the impact of changes in interest rates on house prices. So how can modelling in health economics help us to make policy decisions in an uncertain world?

Benefits of modelling I:  Help overcome short term evaluation

Evaluation periods are usually quite short, whereas the impacts of interventions may only be fully realised over the longer term. The danger is that if we limit evidence gathering on benefits and costs to a “within-trial” evaluation, we run the risk of misrepresenting an intervention and drawing erroneous conclusions. Modelling can help project our best estimate about long term impacts.

In health technology assessment, economists build models by simulating an underlying disease process and learn the relevant epidemiology and statistics, or collaborate in relevant multidisciplinary teams. For instance, in cardiovascular disease (CVD), we can model from CVD risk factors (such as blood pressure and cholesterol) to estimate future CVD events (such as heart attacks and stroke) and estimate the consequences for life expectancy. So, for instance, a statin trial might estimate the impact on reducing cholesterol. But models can then convert this information into projections of the change in (quality adjusted) life expectancy and health service costs. More on this in the next blog, when we give detailed examples.

Benefits of modelling II: Help with choosing between intervention options

A nice problem to have is when there is lots of evaluation evidence about different kinds of interventions. But given scarce resources we need to choose. This may be a complicated task, if different evaluations have collected information on different outcome measures or over different time periods etc. Models can help guide us by: first, encouraging decision makers to make explicit their objectives and budget; second, collating the information on different intervention options; and third, taking us through an “option appraisal” process to point to the interventions promising best value for money. This process requires larger generic models to allow many different interventions to be compared. So for instance, we ideally want a CVD model that can simulate the likely impacts of both individually targeted (such as, statins or exercise regimes) and population-wide interventions (such as a smoking ban). These kinds of models permit policymakers to develop and evaluate packages of interventions.

“All models are wrong, but some are useful” (George Box).

There are various guidance documents regarding how to build models. The key issue is whether they are valid! This can be deconstructed into two elements: (i) “face validity” – where the model has the agreement of experts in the subject area (e.g. those that know about CVD); (ii) predictive validity – that it has been demonstrated that the model has accurately predicted outcomes of interest (or can be recalibrated to do so).  The latter can be difficult, if data aren’t available. But sometimes policy cannot (or does not want to) wait for the evidence – judgments are often taken given that the risks of not intervening may be greater than waiting for perfect evidence. The importance of explicit modelling is equally, if not more, important to those situations where we have some evidence but we need to project long term consequences. Models, as simulations, of a local system can be a way to harness expert opinion, make explicit value judgements and project ahead the different possible scenarios. In this way models can be used to make the case for intervening in the absence of perfect information. Climate change policy, defence policy, and fiscal policies are all live examples of this.

Modelling in public health – a rarity!

Economic evaluation of public heath interventions is rare, especially for upstream interventions such as welfare reform, housing improvements or urban regeneration.  However, the need for modelling here can be even more important than for clinical evaluations. For instance, the impact of upstream interventions will only be fully realised in the distant future; and perhaps intergenerationally, while the upfront costs can be considerable.  

But how can we model long run impacts? This is a major challenge. We face a similar situation to the early days of health technology assessment. Economists then broadened their horizons and learned epidemiology to help build models to be used in economic evaluation (e.g. the statin example). Equally, the challenge is to work with other social scientists and interventionists to develop testable intervention theories to be used as a basis for modelling. For example, it is common for social interventionists to create “logic models” that harness expert opinion and map out the intended impacts of interventions. The collective challenge is to turn these into formal causal models to be tested using evaluation data. In application, models can convert intervention impacts seen in short run evaluations into longer term projections.  This process will require real innovation, but is feasible, practical and promises to be useful. If health economists are really interested in public health, then we need to do some social science for a change!

So use models, get involved and make them better!

In short, models are there to compensate for the fact we lack perfect information.  Models can help get the most out of short term evaluations, and can help us think through the possible consequences of different policy options. Ideally, a model should be as simple and user friendly as using a sat nav. Decision making and evaluation in an uncertain world is difficult; and models are there to help make the best choices given the available data.