A scoping review of horizon scanning approaches used to identify emerging research methods

Patient and Public Involvement

Patients and the public were not involved during this project, but the findings from this piece of work will be used to inform future involvement and engagement days to shape the next stages.

Eligibility criteria

Appendix 1 provides full details of the eligibility criteria. In summary, the research question was formulated using the Population, Concept, and Context (PCC) framework, and any information source that used horizon scanning to identify research methods or methodologies was eligible for inclusion. There were no restrictions on the type of research field, place or setting where the research was conducted, or the research sponsor¹⁴.

Types of evidence to be included

Peer-reviewed published articles, conference abstracts, and reports from grey literature were eligible for inclusion. Editorials, letters, comments, notes, erratum, economic evaluations, cost and resource use studies and trial protocols were excluded.

Search strategy

A pragmatic search strategy was developed by an experienced information specialist (CE) using concepts for ‘horizon scan’ and ‘method’. The search strategy was developed in Embase (OVID) and peer-reviewed by a second information specialist (SGGM) prior to being translated to Scopus and ProQuest (Social Science Premium Collection). Search results were combined and deduplicated using EndNote (version 20.2 Clarivate Analytics, PA, USA). No limitations or restrictions were placed on the searches. The searches were conducted on the 6 June 2024, and the full search strategies can be found in Appendix 2.

Screening process

Two reviewers (NO’C or CE or JL) independently reviewed the title, abstracts, and then full-text articles for each record. Disagreements were resolved through discussion and consultation with a third reviewer, with a fourth reviewer (GN) consulted if needed. Reasons for exclusion at the full-text stage are in Appendix 3, and the search results and inclusion process are presented in Figure 1.

Figure 1. A PRISMA flowchart was created using the Shiny app.

It details the number of studies identified, screened, and included in this scoping review¹⁷.

Data extraction

Following a pilot phase with two articles (NO’C, CE), one reviewer (NO’C) extracted data from the remaining articles. This data was comprehensively checked by one review team member (CE), with further accuracy checks by two other reviewers (GN and JL). Disagreements were resolved through discussion. In the interest of time, authors were not contacted for additional information.

The data extraction file can be viewed here https://osf.io/ (DOI 10.17605/OSF.IO/4XQV6). Appendix 4 (V1.1, 30.08.2024) provides a complete list of extracted data items. In brief, data was extracted across four categories: study characteristics, horizon scanning, methods and reporting. The EuroScan International Network’s early awareness and alert system (also called horizon scanning systems) is an internationally agreed framework used in this study to categorise horizon scanning approaches¹⁶. There is variation in how the framework is described and applied, for the purposes of this study, we have considered all stages of the framework as the horizon scanning system. A brief description of this system, including its stages and processes for detecting healthcare technologies, is described in Table 1.

Critical appraisal

In line with methodological standards for scoping reviews, critical appraisal of the individual studies was not undertaken¹⁴.

Data analysis and presentation

Results are presented alongside a narrative summary. Reporting for this manuscript adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR) Checklist available in Appendix 5 on the Open Science Framework repository (https://doi.org/10.17605/OSF.IO/4XQV6)^15,19. Deviations from the protocol are documented in Appendix 6.

Selection

After duplicates were removed, 727 citations were identified. At the title and abstract stage, 680 records were excluded, with 47 full-text articles retrieved and assessed for eligibility. Of these, 29 were excluded for the following reasons: 16 did not present primary horizon scanning research; 11 used horizon scanning but not to identify research methods; and two were excluded because they were not written in English. A full list of excluded articles is available in Appendix 3 and Figure 1 shows the flow of records through the identification and eligibility assessment stages.

Characteristics of included evidence

Five studies fully met the eligibility criteria to be included in this review^20–24. Thirteen additional studies partially met the eligibility criteria and have been categorised as included studies of secondary relevance because they aimed to identify emerging issues, opportunities, knowledge gaps, or challenges that are likely to be important in the future; while these studies did not use horizon scanning to identify research methods, all reported research methods as a finding^25–37. This was considered an important secondary finding relevant to the aims of this scoping review, and these studies are discussed further under ‘secondary findings’. Although consensus was reached on the full inclusion of the five studies and inclusion of the thirteen studies of secondary relevance, this was only achieved after extensive discussion and consultation between the review team. The difficulty in applying inclusion and exclusion criteria was primarily attributable to the limited reporting in identified studies and the scientific fields represented by the studies.

The five included studies were all peer reviewed studies that were published between 2017–2024, they all belonged to the environmental, ecological, and earth science disciplines. Two authors specified the remit of their research; one was relevant globally, and the other had a European focus. All authors reported funding information, details of this and other characteristics of the included studies are described below in Table 2.

1. Horizon scanning approaches used to identify methods

Horizon scanning was broadly conceptualised in the included studies as a methodology to gather emerging insights to address future challenges, see Table 3 for individual descriptions.

Table 4 presents the horizon scanning methods that were used to identify needed or emerging research methods in each of the included studies.

Table 4. Horizon scanning approaches used in the included articles categorized against the horizon scanning system developed by the EuroScan network and later adopted by Hines et al.¹⁸.

Author (date)citation	Detection	Filtration	Prioritisation	Assessment	Dissemination	Updating	Evaluation of process
Abdelhady (2024)20	Time Horizon: NR Eligibility Criteria: NR Method(s): Systematic literature review Restrictions: Field: Earth Science; Date: 2000–2022. Database: Scopus. Journals: Paleontology.	NR	Bibliometric analysis: Topics were ranked based on keyword frequency over time to highlight historical trends using AntConc v4.2. Four groups of emerging techniques were assessed but the methods to determine are unclear	Narrative /opinion piece	Not formally reported/ publication	NR	NR
Novak (2020)23,38	Time Horizon: NR Eligibility Criteria: NR Method(s): Interviews with multi-disciplinary experts (n100 from 60 institutions) and a Literature review (>250 articles).	NR	NR	Narrative/ opinion piece Qualitative framework to display technological maturity and socio-political/ economic factors which influence readiness, deployment and acceptability of applications.	Not formally reported/ website/ publication	NR	NR
Musche (2019)22	Time Horizon: NR Eligibility Criteria: NR Method(s): Multi- disciplinary expert consultation (n=6) was conducted with a random sample (n=28) to generate initial questions. Following this, online surveys were distributed using snowball sampling to the core groups personal networks in order to collect emerging research questions.	Methods and criteria are unclearly reported; however, questions were consolidated, de-duplicated, and eliminated by the leading team (n4).	Questions grouped into four themes and ranked by the core team for relevance to future research infrastructure. The top four questions from each theme (16 total) were selected for discussion and refined through re-formulation, merging, reinstating, or scope adjustment. Final mean scores were calculated and alternative questions presented, the five highest-ranking questions per theme formed a list of 20 priority research questions.	Narrative and visual/opinion piece A conceptual visualisation to demonstrate the increasing complexity of research infrastructure, themes and ecosystems alongside policy relevance.	Not formally reported/ publication	NR	NR
Tomsett (2019)24	Time Horizon: Near future Eligibility Criteria: NR Method(s): Literature review	NR	NR	Narrative/ opinion piece	Not formally reported/ publication	NR	NR
Hofmann (2017)21	Time Horizon: NR Eligibility Criteria: NR Method(s): Literature review	NR	NR	Narrative/ opinion piece	Not formally reported/ publication	NR	NR

Abbreviations: NR, not reported.

When mapped against the horizon scanning system, the most well-reported stage of horizon scanning was signal detection, which was reported in all studies. A literature review was the most common method (n=4) used to identify signals^20,21,23,24. Two studies used a combination of approaches, such as expert consultation alongside online surveys, or interviews with experts together with a literature review^22,23. The criteria and methods used for assessing eligibility of information (filtration) were not clearly reported in any study. Two authors reported prioritisation criteria; the methods used to prioritise the identified signals included expert engagement for consensus-deriving ranking and quantitative ranking techniques using bibliometric analysis. Assessment methods included author opinion, which was reported narratively, and qualitative frameworks to indicate wider determinants of readiness such as socio-political and economic factors. No studies conducted formal assessments of the prioritised innovations to ascertain the potential for impact, for example to examine the existing and ongoing evidence base and related areas of uncertainty, costs, ethical, political, legal or cultural impact (such formal methods may include rapid evidence synthesis, driver analysis, scenario planning or modified-Delphi study¹⁸). Although a dissemination plan was not explicitly reported by any of the studies, the publications themselves are a form of dissemination and are classified under this category. Additionally, none of the studies or related materials document any intention to update or evaluate the horizon scanning process or findings. There was no reported public involvement in the planning or conduct of these studies. Overall, there was heterogeneity in the reporting of horizon scanning methodology used and its presentation across the studies.

2. Methods identified

All five studies reported using horizon scanning to identify research methods within the environmental, ecological, and earth science disciplines. Table 5 provides a summary of the methods identified across different fields.

Identified methods or techniques included digitisation, computational techniques, genomics, next-generation sequencing, statistical methods to facilitate dataset comparability and the harmonisation of measurements and methods across sites.

3. Reporting

Only two studies cited references to support their horizon scanning method, and none referenced the use of any reporting guidance. Abdelhady cited guidance on conducting bibliographic analysis^20,39, while Musche referenced previously published work to justify the use of a modified Delphi method and the omission of a workshop^22,40–42.

In addition to narrative reporting, a range of visualisations (n=19) were presented across the five included studies (see Box 1 for a description of visualisation types and examples). Two studies used visualisations in their assessment stages as conceptual tools to illustrate awareness of technologies or methods in relation to broader considerations, such as sociopolitical and economic acceptability^22,23.

4. Secondary finding

The thirteen studies that partially met the eligibility criteria are summarised below and in Table 6. They used horizon scanning approaches to explore broad aspects of future importance, such as issues or opportunities, and then discussed research methods as a finding. A full list of references is available in Appendix 7^25–37.

Six of the studies were interested in emerging issues related to health and social care, specifically in relation to genomics, rare diseases, ionizing radiation, and health policy, one of these studies explored plastic pollution and had a transdisciplinary remit that included health^{25,26,28–30,33}. The remaining seven studies contained research relevant to environmental and ecological sciences.

To summarise the horizon scanning approaches used, seven of the thirteen studies used a combination of approaches. The reported approaches, either in isolation or combination, include: one case study on the utility of citation network analysis, five used the Delphi method or a modified version, five used surveys, and three reported workshops. One article described a four-step foresight approach with horizon scanning as the identification stage, although reporting was limited due to it being an abstract. The horizon scanning approach in a second abstract is also unclear, but it appeared to involve expert consultation.

To highlight some of the methods identified as an area of focus in health and social care, Provencher et al., used survey-based horizon scanning to prioritise research questions for plastic pollution policies, ultimately highlighting the need for standardised sampling and reporting methods³³. Similarly, Giuffre et al., used foresight methodology to identify emerging trends in rare diseases to guide future policy development, they identified trends in 1) citizen empowerment 2) leveraging big data and artificial intelligence to improve data standardisation and interoperability, automated data extraction techniques, and the creation of disease registries using existing clinical data^28,43. Other research methods and tools identified as a need include artificial intelligence (AI) solutions for medical radiation applications and the requirement for further work to effectively implement genomics into health and social care via national policies^25,26.

Examples of emerging areas where focus should be given includes improving methods for predicting skills demand, improving scientific literacy, and developing tools to assess national metrics such as population health³⁰. Additionally, emerging areas include diversifying policy evaluation methodologies and incorporating public opinion into early policy development by using crowdsourcing and social media. Methods for assessing the value of non-scientific evidence in policymaking were also identified as well as a need for methods to determine when public engagement is appropriate, how best to monitor social intelligence and identify emerging diseases³⁰.

Summary of evidence

In this scoping review, five studies were identified that reported using horizon scanning to identify research methods. While the search did not have date limits applied and was broad enough to identify literature from multiple disciplines, all the studies included belonged to the environmental, ecological, and earth science disciplines and were published relatively recently between 2017 and 2024. Horizon scan methodology used research approaches, such as literature reviews, interviews, and surveys, highlighting 29 research methods that are likely important for the future advancement of research in those areas.

It has been previously reported that there is variation in the aims of horizon scanning, the methods used to conduct it, and the terminology used to describe it. This variation likely reflects the wide-ranging disciplines that rely on its use to generate early intelligence to answer different policy and research questions, together with a lack of agreed standardised methodology for the planning, conducting and reporting of horizon scanning. As described in the methods section, the stages outlined in the historical 2014 EuroScan horizon scanning system were used as a framework to map the horizon-scanning findings. As far as can be determined, it is the only internationally agreed framework that describes the stages of horizon scanning, and using it helped to categorise and present the results. However, there are major drawbacks to taking this approach, it was originally developed for a specific use in the healthcare and biomedical fields as an early awareness system to support the health technology assessment of medicines, and this may have limited its applicability to other disciplines and where the use of horizon scanning is for different aims and purposes. Additionally, none of the authors appear to have followed pre-defined, systematic methods nor reported them explicitly. Considering this, best judgment and consensus were used to categorise the identified horizon scanning methods.

Within the findings, variability in the definitions and aims of horizon scanning was noted. While the EuroScan early awareness and alert system is referred to as a horizon scanning system by both EuroScan and Hines et al., they also highlight one stage of the process (i.e., detection), as horizon scanning and consider separate subsequent stages (i.e., filtration, prioritisation, assessment, dissemination, updating and evaluation) under foresight methodology. There is also variation in the definitions used in the EUnetHTA 2020 recommendations, where the authors describe the later stages (topic identification, selection and prioritisation) as horizon scanning only if it is proactive. This variation suggests a lack of consensus on whether horizon scanning should be regarded as a systematic method solely to identify emerging signals, with subsequent stages (filtration, prioritisation, assessment, dissemination, updating and evaluation) considered separately as foresight methodology. In practice, horizon scanning centres take a tailored approach to meet the needs of individual research question and often consider stages beyond detection as part of the horizon scanning process⁴⁴. There are ongoing efforts to make the planning, conduct and reporting of horizon scanning more methodologically robust⁴⁵; variation or ambiguity in how expansive the definition of horizon scanning should be, means that careful delineation is required to determine the scope and reach of all considerations of horizon scanning activity.

As previously mentioned, many studies were poorly reported. This, combined with the varying norms across different research areas, broadly defined aims (e.g., horizon scanning to identify issues, threats, or opportunities), and terminology for findings that could be considered methods (e.g., AI and big data), made several scoping review stages challenging. One particularly difficult aspect was identifying research methods due to the lack of a cross-disciplinary classification framework. The Methods Map developed by Sage may be a useful framework for organising various research methods, approaches, and techniques for researchers looking to undertake similar work, but it is limited in its applicability to transdisciplinary areas and also in some areas of evidence-based medicine, like systematic review and health economics⁴.

Using horizon scanning as a scientific method to explore emerging trends and research likely to impact the future can increase early awareness, evaluation, and the subsequent adoption of the most promising and innovative methods. Similarly, using horizon scanning or demand signalling to identify a need for research methods, rather than identifying emerging ones, could drive innovation by encouraging cross-pollination of ideas and adoption of approaches from adjacent fields, ultimately accelerating patient access and improving patient outcomes.

Limitations

The search strategy was designed to be targeted, aiming for specificity using the keyword ‘horizon scanning’. Horizon scanning is often described as a methodology that falls under the broader umbrella term ‘foresight’, and so it is possible that by omitting this, some relevant records may not have been identified. However, testing the impact of this during an earlier search development stage found that the term foresight generated many irrelevant records, which would not be considered at the full-text stage. As the aim of this work was to map what has been conducted and reported in this area rather than exhaustively identify detailed information relating to each data item, authors were not contacted for additional information. Lastly, only studies written in English were eligible for inclusion, studies excluded based on language are transparently reported in Appendix 3. It is difficult to assess the impact of excluding both articles on the results given translation issues; However, it appears one was relevant to National security in the Netherlands, and the other appears to be a review of horizon scanning methods used for HTA worldwide and in Germany.

The lack of standardised reporting led to difficulties in assessing eligibility, extracting data, and categorising the results. To mitigate this, multiple team discussions to derive consensus were relied upon. To ensure robust data extraction, one person extracted the data, and three other reviewers checked each data item for agreement. As mentioned, reporting standards are being developed and this should support future mapping work⁴⁵.

Despite following current guidance for conducting a scoping review, including the use of recognised measures to reduce bias and error, as with any secondary research, it is possible that relevant studies may not have been identifiable in the databases searched or were indexed in ways that our search would not have detected.