A multiple criteria decision analysis to establish the use cases and candidate point of care tests to enter into a platform trial of multiple in vitro diagnostic point of care tests in the prehospital environment

Patient and Public Involvement

This work benefitted from a PPI co-applicant who had prior experience of relevant high-quality point of care testing research and who was involved in all aspects of the research. A wider group of Public Advisory Group members were involved in the following ways:

Public Contributors (PCs) helped us to understand the range of variation in service provision and patient circumstances. This included looking at the potential alternative care pathways that can be accessed by testing and describing the facilitators and barriers to pre-hospital testing from a patient/carer perspective. They also advised on the pre-requisites that make pre-hospital testing and alternative care pathways acceptable from a patient and carer perspective.

PCs were involved in the development of the Stakeholder Survey 1 to create the longlist of ‘criteria’ for assessment and the Stakeholder Survey 2 to define the weighting of criteria. PCs also participated fully in both stakeholder workshops.

We conducted an MCDA to prioritise candidate in vitro POC tests and use cases for inclusion in a platform trial of POC testing in UK ambulance services. The MCDA was conducted in accordance with the principles described in the good practice guidelines for the conduct of MCDA published by the International Society for Health Economics and Outcomes Research (ISPOR)¹⁰.

The MCDA was conducted between March and September 2023 and the process is illustrated in Figure 1.

Figure 1. Multiple Criteria Decision Analysis flow diagram.

Scoping the use cases

Identification of potential use cases

The options (use cases) were identified through systematic literature search and selection process following principles of scoping review¹¹. We did not assess the bias in the studies identified, nor did we attempt any formal synthesis of the evidence as these are outside of the methods used for scoping reviews¹².

Searching for relevant studies

To identify POC tests used within, or with potential for use in the EMS we searched three electronic bibliographic databases (Medline and EMBASE on Ovid and CINAHL on Ebscohost). The search was developed by an information specialist (SD) and systematic reviewers (TM,JS) in liaison with the rest of the team. The strategy combined terms for point of care tests with terms for EMS. We did not apply a filter to restrict types of study design. The search strategies are available in an online data repository¹³. We searched reference lists of key systematic reviews. We restricted to English language and research published after 2000 and focused the search on adults. Following an initial broad search for POC tests in the EMS, additional supplementary searches for specific potential use cases were completed between April and August 2023. A table summarising the searches is presented in Extended Data File 1. Titles and abstracts of records were screened by one author (TM) with all excluded records checked for exclusion by a second member of the wider team (see acknowledgements). Full-text versions of the records marked “retrieve” by either reviewer were screened by one author (TM) with all excluded papers checked for exclusion by a second member of the wider team. Strict inclusion and exclusion criteria were used to guide selection, criteria included: reporting on a POC test used in the EMS, setting was the EMS, POC test was using bodily fluids (saliva, blood, serum, breath, sweat). We excluded studies where the focus was use of equipment to obtain a diagnosis from a body scan, such as ultrasound. We excluded tests used in specialist services such as mobile stroke units because of the reliance on specialist personnel and scanning equipment in conjunction with any POC tests. Search data were managed using Rayyan and Endnote software^14,15. Screening was managed using the Microsoft software Access and Excel^16,17.

Preparation of stakeholder summaries

Data extraction from studies identified in the scoping exercise was done by one researcher (TM) and checked by a second (KM,RP). Data extracted included: Type of test(s), test device name and manufacturer, study design, condition the test was being used for, biomarkers being tested. Information for potential use cases was organised into summaries for stakeholders into 6 types of information, 1) systematic review of primary research, 2) Economic evaluation of use of POC tests in the EMS, 3) ability to make causal inference for a change in patient pathway, from a) experimental studies with interventions assigned at random, or b) from non-randomised studies with a comparator group or controlled before and after studies 4) demonstration of the POC test being used in the EMS from observational studies describing the use of a POC test (single arm observation studies) 5) Views of paramedics or clinicians or patients or hospital staff on the use of POC tests in the EMS to change the patient pathway (qualitative studies or surveys). Studies investigating the diagnostic test accuracy of POC tests were documented and listed but not summarised as this information is not sufficient to ascertain if a change in patient pathway has potential. The number of records for each use case, and the type of information identified for each were organised into tables and stakeholder summaries that were provided at two workshops. The summaries presented at Workshop 2 are listed in (Extended Data File 1).

Stakeholder recruitment and patient contributor engagement

Stakeholder recruitment

The MCDA focused on UK EMS care, but recruited national and international stakeholders with the expertise to meet the inclusion criteria described in Figure 1. Stakeholders were identified and recruited during June and July 2023. Eligible stakeholders, who had been identified as lead authors of research found during the scoping review, or whose work and expertise was familiar to the study team, were contacted. Snowball sampling was then employed to recruit additional stakeholders. Initial contact was via an email providing information about the study. Where stakeholders expressed an interest in participating, they were provided with a participant information sheet and a consent form and asked to consent to participate in the study. In addition to seeking topic experts, we also recruited a sample of patient and public stakeholders from an established patient and public research advisory group.

Public contributor workshops

To support the full engagement of public contributors in the MCDA process we convened two virtual public contributor workshops. The initial workshop was designed to provide contributors with training on background information about the topic, the proposed platform trial, and the MCDA methodology. A member of the study team (KK) and an expert in Patient and Public Engagement (AG) presented information to the group and answered questions. Participants were also provided with written information. A subsequent workshop was convened and supported by JC, HN and AG to inform public contributors about the process that would be used in Workshop 2, and to ensure there was adequate support to enable patient and public contributors to engage fully in all activities.

Exploring the criteria and mapping the use cases

Stakeholder Survey 1

Stakeholder Survey 1 (Extended Data File 2) was disseminated to non-public contributor stakeholders to explore candidate criteria that should be considered when deciding whether a POC test is suitable for use in EMS. The survey contained questions focused on ‘a priori’ themes of promise, plausibility, risk, costs, the nature of the condition being investigated, the technology, the adopter system, the wider institutional and societal context, the evidence quality and other criteria which we selected from reviews of pertinent research by Campbel and Knox¹⁸ and Greenhalgh and colleagues¹⁹. Stakeholders were asked to rate the overall importance of each of the criteria using a five-point Likert scale but were also asked to provide free text responses to elaborate upon their answers.

Responses to the survey were collated and reviewed by the research team. Likert scale responses were summarised using frequencies. Survey 1 results informed an update of a priori themes, producing a new shortlist of criteria to take forward into Stakeholder Workshop 1.

Stakeholder Workshop 1

Prior to the workshop summaries of the evidence available for different use cases were provided to the stakeholders. Responses to Survey 1 were visually summarised using an online interface (Miro board²⁰) and formed the basis of a subsequent discussion. During the online workshop, stakeholders were invited to map clinical pathways for candidate use cases (mapping both the current care pathway and the proposed amendments to the care pathway that introduction of a POC test would allow), discuss the importance of each shortlisted criterion and make suggestions or alterations to the shortlist. All ideas were captured on the Miro board and displayed to the stakeholders, who were asked to agree on the final content of the Miro board by the end of the workshop. The workshop was recorded using Microsoft Teams.

We then compared the refined shortlist of criteria against the Evidence and Value: Impact on Decision Making (EVIDEM) framework²¹, which has previously been used for MCDA involving health technologies²². Following an iterative process of coding and refinement within the study team, we reached a final consensus that the list of criteria identified was truly representative of stakeholder opinions, while being sufficiently specific and measurable to use in the MCDA process. The study team also prepared a draft rating system with which to score each criterion. We searched for published examples of established scoring systems and used these wherever possible. Where that was not possible, the team agreed three- or four-point scoring systems to put to the stakeholders.

Evaluating the criteria

Stakeholder Survey 2

To validate and finalise the list of criteria to be used for scoring and to determine the weightings that would be used to score those criteria, we disseminated a second survey (Extended Data File 2) to non-public contributor stakeholders. Each of the shortlisted criteria was presented back to the stakeholders. They were asked three questions about each criterion. First, they were asked to score the importance of the criterion on an 11-point scale. This would later be used to assign weightings to the final criteria. Second, they were asked to provide feedback on the proposed rating system. They were asked if they felt that the rating system was appropriate. If not, we asked stakeholders to explain why not and to propose alternatives. Finally, we asked the stakeholders if any of the categories should be used to exclude a POC test from being considered in a future platform trial. This allowed us to identify the ‘deal breakers’ or satisfice criteria, which would exclude a POC test from further consideration.

The responses to the survey were collated and analysed within the study team. The list of criteria was finalised and stratified into two sections: criteria for prioritising use cases; and criteria for prioritising individual POC tests or assays within those use cases. Scoring systems and satisfice criteria were finalised based on consensus of the stakeholders, with review within the study team for every instance of disagreement. Outputs were taken forward into Stakeholder Workshop 2.

Measuring the use cases against the criteria and achieving consensus on scoring POC tests

Stakeholder Workshop 2

Both expert and public contributor stakeholders were invited to participate in the online Stakeholder Workshop 2. Prior to the workshop, all stakeholders were provided with the final list of options (use cases) identified alongside their associated care pathways (current and POC test-driven), which had been mapped earlier in the project (Extended Data File 4). A summary of the evidence identified in the updated systematic scoping exercise was also provided for each use case (Extended Data File 1).

During the first half of the second workshop, the final list of criteria for scoring use cases was presented back to all stakeholders, who were given an opportunity to make any final adjustments. Once consensus had been achieved, we proceeded to the second stage of the workshop, during which the options (candidate use cases) were presented and scored. The relevant care pathways were explained to stakeholders, and further comment was invited. The summary of evidence for each use case was then presented. The summary was provided in language that was comprehensible to a lay person, and there was an opportunity for all stakeholders to ask questions.

Once we had confirmed that the stakeholders had understood the information and after answering any questions, we proceeded to invite the stakeholders to score the options against each of the agreed criteria using the scoring systems developed earlier in the project. This was done via Mentimeter²³. Average scores from Survey 2 provided the weighting figure assigned to each criteria. The pre-determined weightings and satisfice criteria were used to create a template scoring matrix prior to the workshop. Once stakeholders had provided a score, this was averaged (mean) and added to the matrix by a study team member during the workshop. The weighting and average figures were multiplied for each criteria, and this provided a final score for each use case.

By collating the scores contemporaneously, we were then able to present the final results back to the stakeholders and ensure that consensus had been achieved about the final scores and the ranking of use cases. Having ranked the use cases, we selected the top three scoring use cases for inclusion in a future platform trial.

Achieving consensus on scoring POC tests

Prior to completing Stakeholder Workshop 2, we also presented the agreed criteria for scoring in vitro POC tests and scoring systems for rating individual tests/assays within the relevant use cases. Consensus was achieved for each prior to the end of the workshop. We then identified all eligible commercially supplied, CE- or UKCA-marked assays for the relevant test. This list will be updated and POC tests scored prior to a platform trial of in vitro POC tests in EMS to allow for industry advancements and for the study team to systematically review each device before scoring.

Ethics

Ethics committee approval was granted by the University of the West of England, Bristol Faculty Research Ethics Committee reference: UWE REC REF No: HAS.23.04.101.

Scoping the use cases

Options (use case) - Identification

Systematic scoping reviews of POC tests in EMS supplemented by surveys of industry organisations, identified the candidate use cases shown in Table 1.

Stakeholder recruitment and patient contributor engagement

Stakeholder recruitment

We recruited 32 stakeholders to participate in the study and their expertise and country of origin is shown in Table 2.

Exploring the criteria and mapping the use cases

Stakeholder Survey 1

Fifteen “expert” participants (54% response rate) completed Stakeholder Survey 1 in May and June 2023.

Charts displaying the results from the Likert questions are shown in Figure 2. Free text responses to Survey 1 are included in Extended Data File 3.

Figure 2. Survey 1 charts.

Stakeholder Workshop 1

Stakeholder Workshop 1 was conducted on June 7^th 2023 with 13 stakeholder participants. The findings of Survey 1 were discussed during the workshop and a list of criteria to continue into Survey 2 for scoring and weighting was discussed and developed. These are shown in Table 3.

Use cases were mapped and discussed in Stakeholder Workshop 1 and through consensus the most appropriate use cases for further analysis were identified, based principally on available technology and pathway readiness. Identified use cases were ketone testing in diabetic ketoacidosis, NT-proBNP testing in heart failure, lactate testing in trauma, lactate testing in sepsis, bacterial or viral testing in respiratory tract infection, biomarker testing in traumatic head injury and troponin testing in chest pain. Use case mapping is included in Extended Data File 4.

Evaluating the criteria

Stakeholder Survey 2

Fifteen expert participants (54% response rate) completed Survey 2 in August 2023. The results are included in Extended Data File 6. Table 4 details the scoring and satisfice results of Survey 2.

Options meeting satisfice criteria (coagulation testing in trauma, blood gas testing in COPD, NT-proBNP in heart failure and sepsis, C-reactive protein in sepsis, Beta-HCG in pregnancy, biomarkers of traumatic brain injury) were automatically excluded from entering the platform trial because they met the satisfice criteria. This left five use cases to be scored during Stakeholder Workshop 2.

Measuring the use cases against the criteria and achieving consensus on scoring POC tests

Stakeholder Workshop 2

There were 14 participants in Stakeholder Workshop 2. Participants used Mentimeter²³ to score each of the criteria for the five included use cases, this is detailed further in Table 5. Weighting was applied to the criteria: Potential to alter the clinical pathway; Certainty of evidence for diagnostic accuracy; Certainty of evidence for clinical effectiveness; Size of population affected. The weighting applied was the average ‘importance’ scoring given to the criteria by participants in Survey 2. The full Mentimeter²³ results are included in Extended Data File 7 and Table 6 illustrates the results of the use case scoring.

The three highest scoring use cases; POC troponin testing in possible Acute Myocardial Infarction (AMI), POC lactate testing in suspected sepsis and POC lactate testing in trauma, were prioritised for evaluation within a future platform trial of POC testing in EMS.

Achieving consensus on scoring POC tests

The satisfice criteria for evaluating the POC tests detailed was agreed with the stakeholders during Stakeholder Workshop 2 and this is detailed within Table 4.