Gaining insights into a funding portfolio through publication tracking

Data sources

The REF 2021 impact case study dataset. The REF 2021 impact case study dataset was downloaded from the REF 2021 website⁸. At the time of download, there were 6,361 publicly available impact case studies. Each impact case study contains information on the summary of the impacts, the underpinning research, the details of the impacts and list of publications and sources of evidence of the impacts.

NIHR infrastructure publication data. Each year, NIHR infrastructure award holders submit annual reports to the NIHR. These contain information about publications that have been supported by the infrastructure. The publication data provided include a publication reference in the form of a publication citation as well as the associated Digital Object Identifiers (DOI). In some cases, Pubmed or Pubmed Central identifiers are provided instead of the DOI. Where an identifier is missing, the publication reference is used to query Pubmed to systematically source the relevant information. Publication identifiers are validated using Dimensions, Europe Pubmed Central and Pubmed.

Linking up REF data with our publication data. The list of publications section of the REF 2021 impact case studies could also be optionally used by HEIs to include information about funding for the underpinning research but we found that this option was inconsistently used across the HEIs. When used, we found that this optional element typically covered project funding for which a specific amount of funding was directly attributable to the project, which is not a feature of NIHR infrastructure funding. Therefore, we used the list of publications from the REF 2021 impact case studies to link the case studies to NIHR infrastructure support.

Additional citation analysis. Using forward and backward citation analysis⁹, as shown in Figure 1, we also traced the movement of research as follows:

Figure 1. This is a diagrammatic representation of how we have used forward and backward citation analysis to trace the movement of research from one piece of NIHR infrastructure to another but also from NIHR infrastructure into the policy space in order to develop our impact case studies.

Backward citations provide an indication of where research has been ‘pulled’ into the NIHR infrastructure from and forward citations provide an indication of where research from the NIHR infrastructure has been ‘pushed’ along the innovation pathway into further research or real-world settings.

We then ran ARC publications that had been underpinned by BRCs, PSRCs or HRCs through Overton¹⁰ to search for policy citations. The policy papers were manually coded to the following NIHR priority areas of interest:

NIHR case study development and validation. The REF 2021 impact case studies were coded to NIHR priority areas based on key word searches, for example ‘health inequalities’, ‘digital’, ‘digital health’, ‘artificial intelligence’, ‘workforce’ within the titles, impact summaries and details of the impact section. These were manually cross-checked. Case studies in our priority areas that had been supported by collaborative working across more than one infrastructure scheme were then selected for adaptation to create NIHR impact case studies to help us to understand how different pieces of infrastructure work together. We also developed impact cases studies in the four areas of policy interest based on ‘bodies of work’ involving multi-disciplinary researchers and research teams from different parts of NIHR research infrastructure. Semi-structured interviews were conducted with researchers or representatives of research teams and thematically analysed using inductive theming in order to understand how different pieces of NIHR infrastructure work together, enablers, barriers and challenges and evidence of impact. Any impact claims were validated using data sources that were independent of the researchers involved.

Patient and Public Involvement. Patients and the public were not involved in this research because it is not health research. The authors of this article work for the NIHR and have conducted this research on research to understand what is funded and supported by the NIHR, to provide transparency and accountability for the public money that the NIHR spends on research.

NIHR infrastructure support for REF case studies

Of the 6,361 REF impact case studies that are publicly available, the NIHR infrastructure has supported 327 of which 59 are supported by more than one scheme. Figure 2 shows the numbers for how many REF impact case studies were supported by each type of NIHR infrastructure and where impact case studies were supported by just one infrastructure scheme. Figure 3 breaks the total number of REF impact case studies supported by NIHR infrastructure i.e 327 into the different types of impact, as classified by the REF dataset.

Figure 2.

This graph shows (a) how many REF impact case studies have been supported by each type of NIHR infrastructure and (b) how many REF impact case studies have been supported by just one infrastructure scheme.

Figure 3. This graph shows the total number of REF impact cases that have been supported by the NIHR infrastructure, broken down by impact type.

Insights from researcher interviews

The interviews that formed part of the case study development process explored how different pieces of NIHR infrastructure work together, enablers and barriers for impact. This revealed five areas of strength in terms of how NIHR infrastructure delivers impact and areas for future work to better understand some of these mechanisms.

The first is strategic partnerships and collaborations between different pieces of NIHR infrastructure and how these help to move projects down the innovation pathway¹¹. An example of this is a strategic partnership between the MindTech HRC¹² and the SLaM BRC¹³. The BRC brings expertise in the early stage investigation of mechanisms that underpin new interventions or technologies. The HRC brings expertise in the evaluation of these interventions or technologies in NHS settings. By working together, these two pieces of infrastructure can ensure that minimal time and money is lost in developing products that won’t work in the NHS. This way of working is also called ‘fast failure’¹⁴ and enables the movement of interventions from bench to bedside quicker than the typical 10–15 year time lag between research and impact¹⁵.

The ability to embed research in the NHS is a key strength of NIHR infrastructure. An example of this is research¹⁶ by the Yorkshire and Humber PSRC which looked at the effects of wellbeing and burnout of healthcare professionals on patient safety. Prior to this research, the interest in NHS workforce resilience was primarily on retention and skills rather than wellbeing. The direct involvement of clinical and academic staff in research projects also means that clinical data that are not routinely collected can be monitored to help understand unmet needs but also build the evidence base for novel interventions.

The case study development process revealed close working relationships between the NIHR infrastructure, particularly the ARCs, and commissioners through the Health Innovation Networks (HINs)¹⁷. The HINs support the evaluation and implementation of innovations in the NHS to address specific challenges and foster economic growth. This allows NIHR infrastructure to build the evidence base for and push new interventions and technologies into the NHS but also pull ideas and interventions from the HINs to evaluate in various ways. An example of this is ARC North Thames which evaluated the benefits of co-locating welfare advice services in GP settings, demonstrating significant improvements in patient mental health and well-being, reaching those most in need and supporting healthcare staff. This directly led to services being retained in primary care settings in specific parts of England and roll-out nationally in Scotland¹⁸. Understanding the push/pull mechanisms between the NIHR infrastructure and the HINs and wider health and care system is an area for future work.

The next area of strength is core funding. NIHR infrastructure contracts tend to be at least five years in length which means that infrastructure centres have well developed patient and public involvement and engagement (PPIE)¹⁹, business development, and intellectual property management functions which enable early and seamless input but also the ability to quickly pivot towards addressing new or emerging health and care system needs. In the absence of these, external funding would need to be sought for specific projects which could cause administrative delays or lead to the ‘death’ of projects considered high risk or not in line with specific funder priorities. The infrastructure also has established mechanisms for sharing and linking with various databases and sources which enrich the research. Another area for future work is understanding why this ‘valley of death’²⁰ exists for projects and what additional support can be provided to help.

The NIHR infrastructure supports PhD students, post-docs and other researchers from a range of disciplines and at different career stages, recruited to support specific projects. Core funding enables a greatly reduced reliance on external, short-term, precarious funding²¹ to attract and retain research talent. The buy-out of their time allows them to develop and explore new ideas, sometimes using seed funding from the infrastructure, to get them to a stage where external funding can be sought. Early career researchers also have opportunities to co-lead various initiatives in the PPIE or knowledge mobilisation space, for example, which helps with career development for succession planning within the infrastructure. The NIHR Academy²² provides training and support to health and care researchers at all stages of their careers to build national health and care research capacity and capability. Evaluating our capacity building programmes helps us to understand their value and further develop them to meet the dynamic needs of our workforce.

The interviews also revealed a specific need for more join up across the social care system which is fragmented due to the fact that is it made up of several private organisations, commissioned to deliver NHS or community care services. This poses a challenge for adoption, implementation and downstream impacts on patients, due to the amount of time and costs of working with multiple stakeholders across the care system. Targeted funding to support researchers with policy- or implementation-ready research findings to work with multiple companies within the care system could help with this.