Translational framework for implementation evaluation and research: Protocol for a qualitative systematic review of studies informed by Normalization Process Theory (NPT)

Aim and objectives

This qualitative synthesis of NPT studies aims to better understand how NPT explains observed and reported implementation processes, and to explore the ways in which its constructs explain the implementability, enacting, and sustainment of complex healthcare interventions. In this way, the review aims to make further progressive steps in theory development. The synthesis will answer five key research questions (RQ).

Registration

Because this study reviews the development and application of an implementation theory it is not eligible for inclusion in the PROSPERO register of systematic reviews.

Searches and citation analysis

Our searches will update those of our earlier review¹³. We will search three bibliographic databases (Scopus, PubMed, and Web of Science), and a search engine (Google Scholar) to find citations of papers that developed or expounded the main constructs of NPT^3–8,22; papers that developed NPT related methods or tools^9–11; and citations of the NPT web-enabled online toolkit²³. The same search string will be used across all searches: “Normali* Process Model” OR “Normali* Process Theory” OR “general theory of implement*” OR "general model of implement*" OR “NPT Toolkit” OR “www.normalizationprocess.org”. Preliminary searches were completed from 13^th December to 22^nd December 2021 and the research team are in the process of finalising the included studies.

Screening

In our earlier review¹³, we discovered that combining searches using bibliographic databases (e.g., Web of Science, PubMed, and Scopus) and search engines (i.e., Google Scholar) generated not only a significant number of duplicate citations, but also significant numbers of broken or ‘page not found’ hyperlinks. Because of this, deduplication and the disposal of broken hyperlinks will be combined. One reason for this is that ‘normalization’ is also a term commonly used in a number of disciplines and is also used to describe a procedure used in the construction and reconciliation of relational databases.

Covidence © 2022 (https://app.covidence.org), a systematic review automation tool, will be used for the screening process. AH and CRM simultaneously conducted the title and abstract screening on Covidence. Of the authors, AH has the least knowledge regarding the historical development of NPT, which will minimise bias and balance CRM’s expertise of NPT. Any studies which are eligible (i.e., they meet the criteria set out below) or which may be eligible (i.e., where the content is unclear or reviewers disagree) will be obtained in full text. Disagreements about inclusion will be resolved by discussion. The same process will be conducted for full text screening. Papers selected for inclusion will be stored as portable document format (.pdf) files in secure Endnote Libraries with automatic back up. We will characterise papers included in the review using the typology developed for our earlier review¹³. This divided included studies into seven domains: service organisation and delivery; diagnostic and therapeutic interventions; e-Health and telemedicine; screening and surveillance tools; decision support and shared decision-making; change in professional roles; and guideline implementation. The screening results will be presented in a PRISMA flow diagram.

Inclusion and exclusion criteria

We will include English language peer-reviewed health and healthcare-related journal articles published between 1st June 2006 and 31st December 2021 that employ NPT either solely or in combination with some other theory to report on (a) primary studies using qualitative or mixed methods, (b) qualitative evidence syntheses (including for example qualitative systematic and scoping reviews; meta-ethnographies; and realist and hermeneutic reviews). We will exclude editorials or commentaries; protocols and other study designs; research monographs, theses or dissertations; books and book chapters; conference proceedings and abstracts; or webpages, blogs, or other social media. We will also exclude peer-reviewed studies that solely report on quantitative study designs; that contain only nominal or passing references to NPT; that are restricted to methodological or theoretical discussions or make theoretical or methodological recommendations; and reports of the application of NPT in settings other than those related to health, healthcare, and social care.

Quality appraisal

All papers that meet the inclusion criteria outlined above will be included in analysis using Leximancer® topic modelling software by CRM. However, these inclusion criteria will generate too many papers for manual analysis. To identify papers for the latter we include papers that score ‘high’ when their quality, bias, and confidence are assessed using the Critical Appraisal Skills Programme (CASP) checklist²⁴, and that also meet the definitions developed by Kislov et al.,^17,18 of ‘theoretically informed’ (i.e., papers that offer a rigorous non-descriptive analysis), and ‘theoretically informative’ (i.e., papers that develop relationships between theoretical constructs or challenge theoretical propositions). This will be conducted by AH with support from CRM.

Data extraction

For all included publications, descriptive information will be extracted by AH with support from CRM, including authors, year of publication, health care problem addressed, study type and methods, data collection procedures, how NPT was used in the study, and whether this had been pre-specified in the study protocol. The extraction instrument is provided at Table 1. Procedures for the extraction of data for analysis are described below.

Data preparation

This review will combine two approaches to data analysis. We will use: (a) conventional ‘manual’ qualitative content analysis²⁵ — sometimes called ‘directed content analysis’²⁶ — informed by the approach recommended by MacFarlane and O’Reilly de Bruin²⁷; and (b) conduct partially automated semantic and relational text searches using Leximancer® text analytic software.

Data preparation for qualitative content analysis is uncontentious. Portable document format (.pdf) copies of all included papers will be uploaded into a single NVivo® project directory (QSR International (1999) NVivo® Qualitative Data Analysis Software, release 12.0). No special data preparation is necessary for qualitative content analysis. All content analytic operations, (e.g., case identification, coding, annotation, and memoing), can be carried out using .pdf files. The results, discussions, and conclusions of included papers will be treated as formal data for the review.

Data preparation for qualitative content analysis using text analytic software modelling is not uncontentious. This is because it calls for the modification of the texts that are treated as data²⁸. This is because .pdf files present problems when they are interrogated by Leximancer®. Leximancer® software cannot easily distinguish between different forms of data and metadata, for example tables, diagrams and formatting instructions. To convert the content of .pdf files into usable data for textual analysis, text from the results, discussion, and conclusion sections of included papers will be extracted from .pdf files and converted into a separate set of Microsoft Word (.docx) files using Microsoft 365 version 2203. All metadata must be identified and eliminated from these files and text contained in tables must be extracted and reassembled as discrete paragraphs to enable Leximancer® to ‘read’ them properly²⁹.

Qualitative data analysis

Qualitative content analysis using manual coding. Formal data for analysis will be the results, discussion and conclusion sections of included papers. This will combine abductive analysis (searching for unexpected phenomena of interest in the data), and deductive analysis using an NPT coding manual developed for this review¹². A synopsis of the coding manual is given in Table 2. It will be integrated into NVivo 12® Software. CRM and AH will lead the coding exercise, and all co-authors will each independently read, and check coding of included papers. Where disagreements about coding occur, they will be arbitrated by a third member of the team.

Qualitative content analysis using Leximancer® text analytic software. Leximancer® is topic modelling or text analytic software that partially automates coding processes in qualitative content analysis²⁹. The algorithms that drive Leximancer® are a commercial secret, but topic modelling software generally uses Latent Dirichelet Allocation algorithms to create statistical models of the distribution and proximity of words in a text or group of texts³⁰. This produces information about associations between them (called ‘concepts’ in Leximancer®) within a natural language corpus³¹. It identifies empirical regularities and presents these using maps, graphs, and extracted examples. It is important to emphasise that the software undertakes no interpretive activity; it only establishes the relationships between words. Some authors have claimed that these relationship sets, or concepts, are analogous to the categories that ‘manual’ qualitative analysis produces³², even though they require interpretive selection. To undertake analysis using Leximancer® we will follow the procedures set out by Haynes et al.²⁹. We will run Leximancer® across the whole data set of included papers³³ to identify empirical regularities in natural language data, and ways in which they may be connected³⁴. Searches will be informed by terms from our own coding framework as well as unsupervised Leximancer® coding of a qualitative data set.

Theoretical interpretation

In both text analytic and manual analysis, we will explore how constructs of the theory have been employed across the qualitative dataset and their contribution to explaining implementation processes. Our approach here will be to perform integrative interpretation of the two bodies of data. We will identify and chart the presence of NPT constructs across the corpus of included papers. We will characterise the ways that these constructs are used to explain core elements of interventions in in practice.

An important problem in directed content analysis is that its results may be restricted by the coding frame in use (i.e., we will only discover that which we are already predisposed to finding). Because an important purpose of this review is to be theoretically informative rather than merely theoretically informed, we will seek insights from the application of existing theory (in this case NPT) to the corpus of data, but also seek to develop and extend these insights. The value of Leximancer® is that it will identify unexpected empirical regularities in natural language data rather than the theory-determined regularities that will be identified in qualitative content analysis.

Assessment of confidence

AH and CRM will use the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach through the iSoq (Version 1.0) online tool, to assess our confidence in each finding³⁵. GRADE-CERQual assesses confidence in the evidence, based on the following four key components: methodological limitation of included studies, coherence of the review finding, adequacy of the data contributing to a review finding, and relevance of the included studies to the review question. After assessing each of the four components, AH and CRM will make a judgement about the overall confidence in the evidence supporting the review finding. The confidence will be judged as high, moderate, low, or very low, and will be presented in a ‘Summary of Qualitative Findings table’ as per Lewin et al. (2018).

Limitations of our approach

This is an ambitious project, and because of this it brings with it a number of risks and limitations. The first of these is the risk of being overwhelmed with qualitative data. We propose to mitigate this risk by using two kinds of criteria to reduce the number of papers eligible for inclusion in manual analysis: (a) a score of ‘high’ using CASP, and (b) evaluation as theoretically informed or theoretically informative contributions to the literature^17,18. Second, there is the risk that using a published coding framework to support manual analysis will lead to a mechanistic analytic approach. We propose to mitigate this by using additional abductive approaches that actively seek out and identify within the data surprises³⁶, empirical irregularities³⁷, and deviant cases³⁸, that characterise these in relation to NPT, and explain them. A third potential risk is that there are too few theoretically informed or informative papers to give critical mass to a theory-focused review. The experience of our earlier review¹³ suggests that this is not likely to be a problem. Finally, an important risk is that using Leximancer® to review the results and discussion sections of all papers included in the review will not tell us anything meaningful or useful. Using Leximancer® does not mean that human interpretation is suspended and previous published studies have not suggested that this is an important risk. However, that may be an effect of publication bias. We will therefore treat this as an empirical question, test the data using the software, and critically explore the results. Because this will be done in parallel with the manual analysis, the possible discovery that Leximancer® does not deliver will not pose a threat to the whole project.

Dissemination and implementation

The primary outcome of this work will be a qualitative evidence synthesis, presented in papers that conform to established reporting standards^24,39,40. This will lead to peer-reviewed publications and conference and seminar presentations. The workflow and outcomes are described in Figure 2, and will include:

In addition to peer-reviewed journal articles, we will exploit different dissemination pathways: working through two NIHR Applied Research Collaborations in the UK (North Thames, and North-East and North Cumbria), and internationally through the European Implementation Collaborative. Finally, this review will contribute to a programme of work in NIHR North Thames ARC that is intended to lead to the development of a Translational Framework for Implementation Research (the TRIPR study). As findings become available, we will explore their implications with patients and carers, patient and public involvement (PPI) representatives, clinicians, commissioners, and service managers.