What would happen if…? What would have happened if…? These recurring questions in our life allow us to fantasise about future scenarios or imagine alternative endings to past events. Wondering about the consequences of your actions or how things could be different if you made different decisions are important reflections as they help you to assess the consequences of your future or past decisions. Similar types of questions make up the basis of the health impact assessment, a group of tools, procedures and methods that are used to evaluate the changes caused by the implementation of a public health intervention.
The preliminary assessment
In order to assess the possible effects of a policy you need to first identify the initial conditions of the population group that will be impacted by the intervention. Initially you need to have a picture of the pre-intervention context in terms of social and health demographics, and then you should identify one or more measurable epidemiological indicators (outcome variables) that can track any potential post-intervention change, whether this was predicted (preliminary assessment) or yet to be observed (retrospective assessment).
During the planning phase you will come across the concept of “prediction of policy impact”, which is a preliminary assessment of the potential impact based on the available evidence. In some cases you would create predictive scenarios through which you can obtain a relatively detailed and quantitative indication of the changes that will occur in the system based on a series of assumptions. This type of analysis aims to provide elements that help the decision-making process by identifying and assessing the possible consequences of the actions that you are about to take.
The impact assessment
When an intervention was already implemented, instead, you would use the counterfactual approach to assess its effect, which is defined as the difference between what happened after the intervention (factual scenario) and what would have happened if this had not taken place (counterfactual scenario).
There are different study designs that can quantify the impact associated to an intervention under assessment and they are divided into experimental or quasi-experimental depending on the selection of the counterfactual group. In an experimental design the intervention is applied randomly between the two groups under study while in a quasi-experimental design the groups are not equivalent or there is just one group that is studied before and after the intervention. The latter is particularly helpful to assess unplanned events such as floods, nuclear contaminations or pandemics [1-3].
The time factor
The analysis of interrupted time series is particularly interesting in this regard. This is a quasi-experimental study design that is ideal to assess the impact- on a population group- of an intervention that starts at a defined moment in time after which you hypothesise that there will be complete discontinuity compared to the previous time [4]. The basis of this method is to utilise the trend of the outcome variable, which is measured in equally distanced intervals during the pre-intervention period, to predict the post-intervention trend, thus reconstructing the counterfactual scenario [5]. This way the effect is estimated as the difference between what is observed in the post-intervention period and the projection obtained through the historical series, keeping in consideration any change to both the level and the slope of the trend.
Checking for distortions
Sometimes it helps to create a counterfactual scenario with the use of a control group and not just the historical series- this would be the case for the controlled interrupted time series study design. When you insert a control series that was not exposed to the intervention you can then create a more complex counterfactual scenario where comparisons are made within the group itself (before and after the intervention) and between the groups (the intervention group and the control group).
The essential prerequisite for the selection of a control group is that it should be as similar as possible to the intervention group. The control series should be exposed to any event or co-intervention, aside from the intervention itself, which might also influence the intervention group. A large range of different controls can be utilised to limit distortions and improve the validity of controlled interrupted time series studies. Keeping in consideration the availability of data and the possible confounding factors, the most adequate control group can be chosen based on the geographic area, specific demographical or clinical features, and individual behaviours or by selecting historical cohorts or controlled outcomes or time periods [6].
The main advantage of this approach is that it can help you to check for distortions caused by confounding factors that are part of the intervention you are studying. If you detect an effect in the intervention group but not in the control group this would suggest that the observed change is probably a consequence of the intervention; and vice versa, if you detect an effect both in the intervention and in the control series then it would suggest that it was caused by some confounding event.
The correct interpretation of the results is strictly dependent on the knowledge of the phenomenon that produced them.
Despite the presence of some methodological challenges to the assessment of both the intentional and unintentional effects of interventions, interrupted series analysis is an extremely helpful tool due to its flexibility, simplicity of interpretation and validity.
Nonetheless, you should remember that this type of design allows you to assess the direction and nature of a specific change, but does not provide answers around the mechanisms that generated it. The correct interpretation of the results is strictly dependent on the knowledge of the phenomenon that produced them.
References
[1] Milojevic A, et al. Epidemiology 2012;23:107-15.
[2] Scherb HH, et al. Medicine 2016;95:e4958.
[3] Scortichini M, et al. Excess mortality during the covid-19 outbreak in Italy: a two-stage interrupted time-series analysis. Int J Epidemiol 2021;49:1909-17.
[4] Bernal JL, et al. Int J Epidemiol 2017;46:348-55.
[5] Hategeka C, et al. MJ Global Health 2020;5:e003567.
[6] Bernal JL, et al. Int J Epidemiol 2018;47:2082-93.
© 