Developing methods for systems-informed Health Impact Assessment (System-HIA)

Patient and Public Involvement (PPI)

Our advisory group included a public representative, recruited via the NIHR People in Research website, who was able to provide guidance on study design and delivery in general as well as the recruitment and support of public representatives for study workshops. Given the technical nature of the topics discussed at workshops for this methodological research project, we recruited public participants from an existing public panel which had previously taken part in a Health Impact Assessment exercise and were therefore familiar with the methods. We recruited three public participants though due to ill-health, only two were able to participate. All public representatives were offered preparatory pre-meetings with the joint principal investigator prior to each event to and were also encouraged to make contact should questions or issues arise at any other time. Public representatives were compensated for their time (including preparation and pre-meetings) according to NIHR guidelines.

Equality, Diversity and Inclusion (EDI)

With regard to our research topic, equity is a core value underpinning HIA practice, as identified in the International Association for Impact Assessment HIA Best Practice Principles¹. How systems approaches might help further realise and support this value was therefore an important consideration as part of the research: for instance, the ability of existing systems tools to capture potential differential impacts of a given proposal on different population groups was a key topic for discussion during the stakeholder workshops.

With regard to research conduct, we sought to ensure diversity and minimise bias in our recruitment to the workshops by issuing open invitations to existing communities of practice as well as using professional networks and ‘word-of-mouth’. The online format reduced participation burden and enabled a much greater geographical diversity and international representation than would otherwise have been the case. Our public participants were recruited either by open advertisement on the NIHR People in Research portal (advisory group member) or a public panel convened for an existing HIA (workshop participants): together they represented diverse identities and life experiences relating to economic precarity, chronic illness, minority ethnic status, and the asylum system. Our recruitment materials emphasised our commitment to meeting accessibility needs and all participants – both professional and public – were asked about any such requirements in advance of events to enable accommodations.

The project had three phases: a scoping review, a set of stakeholder workshops, and two follow up co-investigator workshops. We had initially planned to conduct pilot studies during the third phase, but this turned out not to be feasible for reasons described below.

Phase A: Scoping review

We undertook a rapid scoping review of existing peer-reviewed and grey literature to identify examples of the use of systems approaches in impact assessments, including those not explicitly including health. The research question for the review was: how have systems approaches been applied in impact assessment activities to date?

The review protocol – including detailed search strategies and inclusion criteria – is provided in in the OSF repository (https://doi.org/10.17605/OSF.IO/U76VW). We searched five bibliographic databases (Medline, Scopus, Social Science Citation Index, Science Citation Index, and ASSIA) on 18^th April 2023, using terms relevant to impact assessment and systems approaches. No limit was applied by date or geographical setting, though full texts had to be available in English. Eligible article types included primary research, literature reviews, editorials, position papers, and commentary. To identify grey literature, we searched selected websites relevant to impact assessment using terms relevant to systems approaches and invited suggestions for specific examples from our co-investigators and study advisory group. Potentially relevant citations from eligible studies were also screened for inclusion, using a snowballing approach.

To be eligible for inclusion, studies had to engage with systems approaches either by making a claim about their application; engaging with at least two of the core characteristics of complex systems from 9; or using one of a list of methods characteristics of systems approaches (as defined in the protocol). They additionally had to discuss or apply these elements as part of impact assessment or impact appraisal.

Citations retrieved in searches were de-deduplicated and imported into Rayyan for screening. Screening was carried out by all reviewers (RP, ET, VW) via a two-stage process based on (1) title and abstract and (2) full text. An initial set of 30 citations was screened at each stage to pilot and refine the inclusion/exclusion criteria. Following this pilot stage, 10% of citations at each stage were screened by two reviewers to check reliability and identify any further issues requiring protocol refinement: the remainder were screened by a single reviewer. All included articles were extracted by one reviewer using a structured template in Excel and the extraction then checked by a second reviewer. Where systematic or scoping reviews were identified, the included studies were screened for potential inclusion using the above criteria. Quality assessment was not undertaken given that this was a scoping review focused on understanding the extent to which these techniques have been applied in practice.

Phase B: Stakeholder workshops

During this phase, we held a series of online workshops with stakeholders from the HIA community to consider how systems tools could be applied to the HIA process. The workshops were intended as the primary ‘test-bed’ for exploring potential applications of systems approaches and methods within HIA, supplemented by pre- and post-workshop activities undertaken by participants and project researchers.

Each of the four workshops was held on two separate occasions to maximise the possibility of the same people attending all three workshops and accommodating different timezones, resulting in a total of eight completed workshops. Workshops were held online via Microsoft Teams and recordings subsequently professionally transcribed. Each of the first three workshops was 2.5 hours; the final session was 1 hour. Workshops were intended to be sequential, with participants expected to attend at least the first three, with the fourth being optional.

We recruited participants for the workshops via a range of different routes and with an international scope, outlined in Table 1. Our focus was on recruiting:

The final set of participants comprised 27 individuals from across these categories. Consent was obtained by email before the workshops.

In advance of the workshops, project staff reviewed a range of sources to collate practical tools relevant to HIA that would serve as the basis for engagement with workshop participants. We identified the UK Government toolkit on Systems Thinking for Civil Servants as a key source for this purpose, since it was designed for use in the kinds of policy contexts where HIA may be deployed but supplemented this with additional tools identified by co-investigators and our advisory group. Briefing materials for participants on the key concepts of systems thinking and a set of ten core tools which we would explore in detail in the workshops were circulated in advance. The tools which formed the basis for engagement at the workshops are listed in Table 2.

We also identified three key vignettes of real-world HIA which participants could use when testing the tools of interest: these are detailed in Table 3. The use of vignettes was to enable workshop participants to contextualise how different systems tools could be used in the different stages of HIA and to provide a proposal against which the tool could be tested. The choice of vignettes was based on capturing the diversity of policy sectors (food/education; energy; and spatial planning/local economic development); levels of governance (national; regional; local); and of intervention types (introduction of new universal policy; withdrawal of existing infrastructure; change in land use) to which HIA can be applied. Details of the vignettes were circulated in advance alongside the briefing materials for participants.

The purpose and activities of each workshop are described in Table 2.

For the first three workshops, there were typically three breakout groups in each, with each group assigned a vignette to use for the duration of that workshop. Participants were allocated to breakout rooms in advance to ensure a balance of professional backgrounds and geographical areas of practice, and to ensure that participants had the opportunity to test tools against different HIA vignettes over the course of the three workshops. We initially planned to allow participants to choose between systems tools for each exercise: however, it was time-consuming and less informative than expected so from the second iteration of the first workshop onwards, each breakout group was assigned a tool. During workshops, we used the online whiteboard tool Miro to structure group exercises.

We used the online note-making tool Padlet to seek reflections from participants prior to the first workshop (on the challenges they experienced in HIA and their experience of systems thinking to date) and subsequent to each of the first three workshops (on their experience of testing the tools and their views on the application of systems approaches to HIA).

After each workshop, facilitators completed a structured debriefing template to capture their reflections and experiences of the workshops and identify learning for subsequent workshops and phases of the project.

Together, the transcripts; completed Miro boards; Padlet outputs; and facilitator reflections formed the basis of the collated data from workshops: we analysed these using thematic analysis in NVivo.

Phase C: Post-workshop engagement

During this phase, we had originally anticipated piloting systems tools on a prospective basis as part of real-world HIA undertaken by our practice partners. This proved not to be appropriate for both scientific and practical reasons. Firstly, the workshops had identified a number of reservations about directly applying existing systems tools to HIA practice (as detailed below under results), and had highlighted that systems tools were likely to require significant adaptation prior to use in HIA. Secondly, despite scoping a range of pilot opportunities from the beginning of the project, we did not identify any prospective HIA which would be feasible to use as pilot studies – either due to a mismatch with project timescales or other sensitivities around delivery.

Instead, we undertook the following:

These workshops were transcribed and analysed thematically alongside the stakeholder workshops.

Advisory group

An advisory group was convened for the project, with a membership spanning expertise in systems science; public health; and health impact assessment, as well as a public representative. The group met twice during the duration of the project with email updates on a bimonthly basis. The purpose of the group was to provide advice on all aspects of the study by incorporating perspectives from a range of relevant stakeholders and scientific experts, to maximise the usefulness and impact of the project for policy, practice, and research.

Ethics and study registration

Ethical approval for the study was obtained from the University of Glasgow College of Social Sciences Research Ethics Committee for Non-Clinical Research involving Human Participants/Data (reference number 400220331). Date of approval 19/07/2023.

The study was registered with Research Registry (reference: researchregistry9504).

Scoping review

The scoping review identified 25 sources for inclusion: 16 from bibliographic databases; four from grey literature searches or stakeholder feedback; and five via snowballing (Figure 2). These sources are summarised in the Systems-HIA Characteristics of studies document (Systems-HIA Characteristics of studies), with a more detailed description of findings provided in the Systems-HIA Scoping Review document (both documents are available in the OSF Systems-HIA repository https://doi.org/10.17605/OSF.IO/U76VW). Publication dates spanned 2004 to 2022, with no clear trend in frequency over time. Policy sectors of interest included environment; agriculture; housing; tourism and leisure; and transport. Twelve countries were identified in those sources with a specific geographical focus, with all inhabited continents represented at least once. The most common forms of impact assessment were environmental impact assessment or strategic environmental assessment; three studies examined the application of systems approaches to health impact assessment¹².

Figure 2. Flow chart for scoping review.

Of the included sources, 14 reported case studies of the application of systems approaches to impact assessment; nine described theoretical or conceptual considerations in integrating the two disciplines; and two were examples of guidance on impact assessment which acknowledged systems approaches. Many sources explicitly advocated for greater application of systems approaches to impact assessment^12–23, including two examples of guidance documents^16,17.

The most common systems approach described in the sources was systems thinking in general terms (featuring in twelve sources), followed by systems dynamics modelling (featuring in nine sources), soft systems approaches (four sources), and causal loop diagrams or other mapping approaches (three sources). Relatively few sources described the use of participatory approaches.

The most common motivation described for using systems approaches in impact assessment was the identification of a policy proposal or problem as part of a complex system and therefore the perceived limitations of conventional linear approaches to understanding it^12–30. Several papers highlighted the commonalities between resilience thinking and systems thinking^14,20: the influence of the former within environmental and sustainability assessment traditions may explain why these account for the majority of examples identified. Other motivations included: testing assumptions and/or different policy scenarios^{10,13,19,21,24,28}; integrating potentially competing considerations within the impact assessment process^14,21,27; identifying characteristics of causal relationships including factors which promote or hinder change^23,24,31;21; enabling integrated consideration (and mitigation) of cumulative impacts from 13,17,18,22,31; concordance with key concepts in impact assessment including socio-ecological models of human environmental impact and resilience^{13,14,17,20,32}; communicating^13,17,27; or modelling the interactions between stakeholders²⁵. Where participatory systems approaches were used, authors highlighted the value of these in exploring multiple perspectives, organising collective knowledge, and ensuring stakeholder ownership^{22,24–28,30}.

Among the applied case studies, the way in which systems approaches were used in impact assessment varied: some advanced novel approaches to impact assessment which integrated systems approaches²⁵ or included systems tools at specific points in the process (e.g., 26); whereas others used systems methods as a stand-alone analytical tool (e.g., 29) or, more rarely, made claims about the use of systems approaches which were not borne out by the data presented³³.

Few sources explicitly considered how systems tools might be applied to the distinct stages of impact assessment. Where this could be ascertained, most applied systems approaches at the a[ppraisal stage?]^{12,21,22,26–28,31,34}, though some sources mentioned boundary-setting or problem definition as part of scoping^17,27,30. Only one source identified impact assessment practice itself as part of a complex system to which systems insights could be applied²⁰.

Challenges highlighted included: lack of suitable data for quantitative systems modelling and the robustness of resultant findings^10,19,22; negative perceptions of systems thinking by IA practitioners as overly abstract or theoretical¹³; limited familiarity or skills in systems thinking among IA practitioners^14,15,21,22; real or perceived increase in time and budget required for systems-informed assessments^15,22,26; failure to connect systems thinking with other conceptual traditions relevant to impact assessment such as resilience thinking¹⁴; difficulties among practitioners and/or stakeholders in adopting a systems thinking ‘mindset’²⁶; and potential conflict with established impact assessment methods where scoping tends to result in narrower and reductionist approaches to issues¹³

Stakeholder workshops

Workshop participants showed high levels of engagement with the practical tasks of applying tools to HIA vignettes as well as reflecting on their usefulness.

In general terms, participants reacted positively to tools where they could identify immediate benefit but resisted where tools were perceived as adding to complexity to the HIA. In general, participants preferred more structured tools, such as the Perspective Diagram, Context Diagram and the 12 Question tool. Systems tools were seen as beneficial where a clear purpose could be easily discerned, e.g. building an understanding of stakeholder perspectives (eg.12 Questions tool), or as a framework for writing up findings (e.g. CATWOE).

The most common reason for rejecting a tool was that it was difficult to apply. This was either because a direct HIA focus was missing from the tool, or applying it was perceived as complicating things that were already routinely done in HIA practice (e.g. considering population subgroups). HIA practitioners commonly use checklists of populations and determinants of health to prompt consideration of potential impacts, including differential impacts, on health. Whilst the tools were often focused on eliciting the same outcomes, the absence of such prompts was a seen as a challenge in the application of the tools and there was agreement that it would take time and training to gain confidence in using the tool. Departure from existing approaches to HIA – which provide structure and transparency – was also seen as potentially worsening existing challenges in engaging stakeholders in the process. On the other hand, at least one participant mentioned that increasing policymaker familiarity with systems approaches might be an asset and enhance stakeholder buy-in to the process.

A recurring theme from participants was that significant adaptation would be required before any of the systems methods presented could be integrated into HIA practice. There were also some concerns voiced about how evidence could be sourced and integrated with the tools in a systematic way, given that ethical use of value is a core value of HIA¹.

Participating HIA practitioners worked in a range of contexts, from large public health teams which prioritised community engagement to consultancy firms where one person could be carrying out the HIA with no public engagement. Broadly, those who worked in established public health teams which used established HIA guidance and checklists were less keen to adopt the tools and more likely to perceive them as overly complicated than independent practitioners, who approached them more flexibly and found them useful for providing a structured broad perspective.

Participants’ familiarity with systems approaches was also relevant, with those who had used systems tools before being much more inclined to see their benefits. Similarly, participants responded most positively to the Theory of Change tool, because they were already familiar with the theory of change flow charts from other contexts, and found it easier to see how they could potentially use the tool. The Participant Feedback document (OSF Systems-HIA repository, https://doi.org/10.17605/OSF.IO/U76VW) provides detailed feedback on each of the specific tools examined in the workshops.

When participants were asked to consider at what stage of the HIA process each tool might be applied, the most common responses were scoping and recommendations. The choice of the scoping stage, often mentioned along with screening, was linked to the capacity of the tools to explore boundaries, for example CATWOE, Context Diagram, Rich Picture. The choice of recommendations tended to reflect the power of systems tools to communicate or help structure complex situations, for example Context Diagram, Theory of Change, Causal Loop Diagram. Participants frequently identified more than one stage where a tool could be useful though were also often ambivalent and unable to provide detail on how the tool could be used. This was linked to a general lack of confidence on how tools could be applied within the HIA process. Answering this question was also complicated by a lack of standardised language in HIA relating to stages and therefore a lack of shared understanding among the participants: for example, UK participants often spoke of screening and scoping in one phrase, not making strong distinctions, while others noted that they used different terms, which included identification, appraisal, assessment or evaluation.

Our choice to present the systems tools in their existing form – i.e., not tailored to HIA and agnostic as to how they may be used – was motivated by a wish to avoid biasing participant feedback, but was identified by participants as particularly challenging. Facilitators reflected that participants struggled with how to apply the tool to the agreed purpose of identifying potential health impacts. A key part of this challenge was defining the system of interest: for instance, was the system the HIA process itself, the proposal, or the policy sector in which the policy was being introduced? More explicit guidance on how to identify the system of interest was therefore seen as important if these tools were to be useful in practice.

At the final participant workshops, several participants reported having introduced systems tools to their own HIA practice as a result of the study: these comprised the perspective diagram, the context diagram, the critical systems heuristic tool, and participatory systems mapping.

Co-investigator workshops

In our co-investigator workshops, we explored how applying systems tools may have made a difference to two completed HIA, with the involvement of the lead practitioner for each assessment. We selected the rich picture tool to apply to the scoping stage of the HIA of the affordable homes policy, and participatory systems mapping to apply to the appraisal stage of the HIA of Brexit in Wales.

Some similar themes emerged to the participant workshops: in particular, while it was felt that many experienced HIA practitioners would naturally adopt a systems perspective, there was concern that the systems tools added additional complication to the HIA process without clear benefits.

However, the co-investigators also [identified?] a number of other key lessons. First, we identified that applications of systems approaches were most likely to be beneficial when problem-led, rather than tool-led: i.e. starting with a defined problem or challenge for the HIA and applying a systems tool (as is or adapted) rather than starting with the tool and trying to identify a potential application. Second, we identified that some explicit discussion of systems approaches as part of HIA training might help less experienced practitioners to develop their understanding and practice. Finally, we identified a number of critical points during the HIA process where systems approaches were most likely to be useful: these are discussed further below under ‘Synthesis’.

Synthesis

Combining results from all three strands of the project, we identified the following as the most fruitful future avenues for applying systems approaches to HIA practice.

1. HIA focus for systems tools: a consistent finding was the need for systems tools to be adapted to HIA if they were to be useful, and used, in practice. This was felt to be crucial given the time constraints and variable stakeholder buy-in which characterises most HIA practice. In particular, participants often returned to the need for tools to incorporate prompts which would allow participants to identify potential health consequences (including unintended ones), affected population groups, and opportunities to address these consequences. Changing the terminology used by tools to reduce jargon from systems science was also felt to be important, as was guidance on how to identify the system of interest. An alternative approach we identified was to incorporate systems prompts into HIA tools (such as checklists for identifying population groups or potential health impacts): for instance, suggestions to look for feedback loops.

2. Better understanding the system to inform boundary-setting: boundary setting at the scoping stage was identified as a consistent challenge in HIA, as well as an area where there was commonality with systems approaches, which typically place significant emphasis on clarifying the boundaries of the system of interest. The boundaries of the system were often a major focus of discussion when participants attempted to apply systems tools to specific vignettes, and scoping was among the most commonly identified stages at which tools were felt to be useful. A key application of systems tools may therefore be in supporting initial characterisation of the system in which the proposal will be implemented, at the outset of an HIA, in order to set appropriate boundaries in the scoping stage (for instance, those relating to the time frame or geographical reach of potential impacts).

3. Better understanding the system to inform appraisal of impacts: there was widespread acknowledgement across all three strands of the project that proposals subject to HIA do represent interventions in complex systems and that the processes that give rise to impacts display characteristics such as feedback, adaptation, and path-dependence. There is therefore a case for the use of systems approaches to enhance identification of potential impacts at the appraisal stage by incorporating these properties: for instance, by understanding feedback loops between different potential impacts and between the proposal and the wider system. However, based on our experience in the co-investigator workshops – through applying PSM to the appraisal stage of the scenario on Brexit in Wales – this is likely to require further adaptation of the tools as discussed above.

4. Application to the institutional and political context of HIA: a key theme across all three strands of work was that many of the challenges of HIA were institutional and political, rather than methodological. For instance, challenges identified include: gaining buy-in from stakeholders to support and participate in HIA; constraints on time, budget, and staff capacity for undertaking HIA; timing the HIA process optimally in order to influence decision-making; managing stakeholders’ responsibilities and expectations; and willingness to implement recommendations. Workshops with both participants and co-investigators identified that systems approaches could help provide a lens for analysis of the institutional and political context in which HIA takes place, in order to help enhance the uptake and impact of HIA.