Digitally Enabled Care in Diverse Environments (DECIDE): protocol for a programme of rapid evaluation of technology-enabled remote monitoring in health and social care

Context for technology-enabled remote monitoring in UK health and social care

Technology-enabled remote monitoring involves the use of technology, devices or apps to support patients to actively monitor and manage their health or long term conditions (e.g. asthma, heart failure). It enables the remote exchange of information, primarily between a patient or service user and a health or care professional, to assist in diagnosing or actively monitoring health or care status or promoting good health and care. Remote exchange is typically supported by consultations or other interactions (e.g. messaging) that facilitate sense-making of that information, enabling systematic review and monitoring, accompanied by (where needed) changes in the care being received.

Care enabled by technology and the allied use of remote monitoring is a key part of the policy vision of the NHS for improving value and access for patients, and recovery of the health and care system^1–3. There is a significant policy and service-level push to develop technology-enabled care, and to use remote monitoring in ways that are meaningful and effective. Remote monitoring services range from use of apps, personal or medical devices (e.g. pulse oximeters, activity trackers) to support care, through to use of smartphones to share vital sign readings (e.g. blood pressure) with care providers, and use of technology to enable patients to get hospital standard care safely while remaining in their own home. Remote monitoring is being tested and used across the UK for a range of conditions, such as cardiovascular and respiratory diseases⁴.

The COVID-19 pandemic generated much increased need and motivation to make use of such tools, and prompted the reduction of implementation barriers (e.g. approvals for remote monitoring technology)⁵. Various interventions launched across settings and service user populations during the pandemic focussed on protecting vulnerable and multimorbid patients (e.g. use of implantables for remote management of people with heart failure⁶, monitoring real time prevalence of COVID in patients with multiple sclerosis⁷ or diabetes⁸, and developing remote neurorehabilitation services⁹) or monitoring spread of the disease itself. However, systematic use of remote monitoring interventions is fairly new. Published evidence is nascent and heterogeneous, the latter reflecting a breadth in service development and implementation practices (see Box 1). To date technology-enabled remote monitoring interventions have typically been small scale, limited to a specific clinical or geographic focus, and provided limited evidence about outcomes that are meaningful for care providers and patients. This leaves those making decisions about the wider scaling and adoption of such interventions unclear about the potential benefits and costs of doing so.

Growing interest in technology-enabled remote monitoring is accompanied by rising concerns over inequalities between groups in the population and concerns about potential digital exclusion^1,2,51,52. Such inequalities are deep-rooted and widening, leading to disparate outcomes, varied access to services, and poor experiences of care, particularly for certain groups (e.g. sharing specific characteristics such as those related to race, gender, ethnicity, disability, socioeconomic status/deprivation, geography^53–55). This is compounded by issues of digital exclusion, where some people (e.g. with learning or physical disabilities) have unequal access and capacity to use technologies that are increasingly essential to fully participate in health and social care⁵¹.

Rapid evaluation of technology-enabled remote monitoring

Rapid evaluation can provide timely, rigorous and evidence-based insights to inform decision-making on the adoption, implementation and use of technology to support remote monitoring^56–59. A number of rapid evaluation teams exist. These largely focus on service innovations in health and social care, with a direct policy focus (often linked to a centrally-steered remote monitoring intervention) and the aim of rapidly foregrounding evidence that can inform timely decision-making⁶⁰. There has already been some evaluation and learning related to technology-enabled remote monitoring by these rapid evaluation teams. Examples include evaluations on the use of pulse oximetry at home⁵⁶, and in care homes⁶¹; the development of remote monitoring care models, including virtual wards^18,56 and the use of remote AI monitoring in social care⁴⁸.

Across existing rapid evaluations to date, assessment of technology-enabled remote monitoring intended to support health and social care has reported mixed results. Use of oximeters during the COVID-19 pandemic provided care homes with a level of reassurance but knowledge of the additional support available through the NHS was limited⁶¹. Evaluation of the wider use of oximeters to monitor patients at home⁵⁶ indicated lower mortality but this was not statistically significant. Low numbers and high variability across the service meant that cost-effectiveness and long term impact were not assessed. Exploration of the use of in-home sensors to support social care was low on recruitment, exacerbated by COVID-19, and found that limitations in digital infrastructure, system complexity, and general service pressure led to challenges with implementation and disappointing outcomes⁶².

In sum, while there is growing potential to use technology-enabled remote monitoring to benefit more people and services, there has been limited high quality, peer reviewed published evaluation to date. There are opportunities to bring what evidence there is together in the context of the current policy push for technology-enabled remote monitoring. For instance, there is a lot we do not know about whether different types of remote monitoring work as intended; when they work and for whom, why and how; how they can be appropriately spread; and what is needed to make sure they are helpful and do not disadvantage access to and quality of care for some people. There is a need for process, outcome and impact evaluations of interventions at various stages of development and implementation to address questions around adoption, spread, and sustainability and inequalities. There is also a need to consider the varied contexts in which technology-enabled interventions are intended to land, and to pay attention to the often ‘rugged’ landscapes⁶³ of health and social care characterised by emergence, system complexity and unpredictability and requiring attention to the rich and complex factors associated with good outcomes.

Funded by the National Institute for Health and Care Research, Health and Social Care Delivery Research (NIHR HSDR) programme, DECIDE has been established as a centre for rapid evaluation with a dedicated focus on technology-enabled remote monitoring. A collaboration across the University of Oxford and RAND Europe, it brings together topic expertise in health and care innovation, with a track record of delivering cross-disciplinary and methodologically robust rapid evaluations. The aim is to generate a strong evidence base on the potential and limitations of technology-enabled remote monitoring in health and care, and support patients, service users, carers and those who commission remote monitoring services to enable high quality care and ensure decision-makers can make better-informed decisions.

DECIDE’s operational objectives are to:

Patient and public involvement and engagement

Meaningful involvement of service users, carers, patients and members of the public is central to DECIDE, and will enable focused and relevant research and user-friendly outputs. We plan to involve service users throughout and in three key areas. Firstly, via a User Advisory Group with a lay chair and representation from Carers UK (carersuk.org/), National Voices (nationalvoices.org.uk/) and People Street (peoplestreet.net/). The Group is deliberately small, enabling rapid engagement and ensuring that DECIDE evaluates technology-enabled remote monitoring solutions that matter, asks the right questions, collects the right information appropriately, engages the right people, and interprets, communicates and shares learning effectively. To ensure that DECIDE holds issues of health and digital inequality as a central thread, User Advisory Group members co-produced the underpinning principles in Box 3.

Secondly, we will involve services users in dedicated PPIE activities related to individual evaluations. We will invite at least one User Advisory Group member to join each evaluation PPIE group and expand this to ensure diverse representation, include specific areas of expertise/experience in health and social care, and ensure that we conduct evaluation ‘with’ or ‘by’ people who use services (rather than ‘to’, ‘about’ or ‘for’ them). The involvement team will vary in size and scope depending on the design and requirements of the evaluation. Evaluation specific involvement activities are supported by a dedicated budget and are likely to include (but aren’t restricted to) advising on approaches to engage service users, design and delivery of participant activities and materials, and interpretation of findings to maximise dissemination across the whole community.

Finally most, if not all, evaluations will involve consenting service users as participants, and where relevant including both those who use technology-enabled remote monitoring and those who do not. Participant activities will generally involve some form of generating data which may include, for instance, participation in interviews, workshops and/or observations within each evaluation. Guided by DECIDE’s cross-cutting theme of inequalities (see below) we aim to ensure robust and (where relevant) diverse sampling and recruitment strategies to ensure that evaluations engage with a range of participants and voices.

Rapid evaluation

DECIDE uses rapid evaluation to enable timely evidence that can inform policy and practice relating to technology-enabled remote monitoring. Our approach is to provide rapid evaluation of selected interventions using robust methods and a theoretically informed approach, underpinned by a set of guiding principles (see Box 2), and with issues of equality, diversity and inclusion remaining central throughout.

DECIDE principles build on the wider rapid evaluation literature, which emerged from responses to humanitarian emergencies and unfolding crises^65,66. The emphasis in rapid evaluation is on the ‘4Rs’ – rapid, responsive, relevant and rigorous^57,67 – with the aim of maximising likelihood of impact of actionable findings in a short time frame (compared to traditional evaluation methods^68–70). As is the case with DECIDE, use of the term ‘rapid’ can mean different things depending on the rationale, scope, and available timeline for each evaluation (e.g. fast set up, condensed data collection, accelerated dissemination⁵⁷).

Rapid evaluation uses established methods, combined with focused, efficient and collaborative work processes, to rapidly assess interventions, programs or policies and so provide timely feedback and insights that can support decision-making⁵⁷. This typically involves adapting standard qualitative, quantitative and mixed methods approaches, for instance through compressed analysis or extrapolating findings from limited data, or use of larger teams allowing larger scale data collection and/or complex analysis. Careful attention to process and awareness of limitations support the trustworthiness and credibility of reporting, and ease tensions (e.g. between rigour and rapidity) that can come with working at speed⁶⁵.

Setting and context

DECIDE has a UK-wide focus, with a remit to conduct up to 6 rapid evaluations over three years (June 2023 to July 2026) and with one of these potentially offering longitudinal, comparative evaluation across the UK. This is important given the varied policy, system and infrastructural contexts in which each of the four UK nations operate, which have potential to shape technology-enabled remote monitoring and from which there is potential for cross-national learning^71,72. For example, the NHS in Scotland has partnered with an independent sector provider to support national roll-out of remote monitoring in a number of pathways, including blood pressure monitoring and chronic pain management^73,74. Other nations currently aim to support remote monitoring through national and regional funding and support programmes.

Identification of topics for rapid evaluation

We envisage an engaged relationship with our funder that is supported by good governance via our Steering Committee (see below) and User Advisory Group, and informed by discussion with potential policy stakeholders. We plan early engagement with these key groups to inform both topic selection and subsequent topic specification, scoping and protocol development, as well as quality control and dissemination (Figure 1).

Figure 1. Input from stakeholders to evaluation of technology-enabled remote monitoring.

Selection of topics will happen via three routes (Figure 2). First, the DECIDE team will respond to priority evaluations identified by the NIHR team through the funder’s formal prioritisation processes. Second, the team will engage Steering Committee members, other stakeholders and potential policy customers to identify key topics for rapid evaluation as they relate to priority interventions and services (e.g. technology-enabled remote monitoring to support seasonal planning). Finally, and complementing existing routes, the team will draw on early rapid horizon scanning work, conducted at the outset of the funding period. This involved a combination of a rapid scoping review of the literature with stakeholder consultation to identify evidence gaps and potential areas for evaluations, and was informed by principles of good practice for health research priority setting⁷⁵ (e.g. in terms of information gathering, inclusiveness, transparency).

Figure 2. Overview of the DECIDE process for identifying, specifying, conducting and disseminating rapid evaluations.

Horizon scanning facilitated identification of the following five broad topic areas for potential evaluation which have shaped discussion about DECIDE’s work plan to date:

Example research questions

Exact questions to be addressed will vary according to the focus and scope of each evaluation. However, we will work closely with relevant stakeholders (Figure 1) in designing and delivering evaluations, and will be guided by a set of co-produced questions (see Box 4 for examples). Exact questions will depend on the focus, scope and scale of each evaluation, as well as the readiness of a technology-enabled care pathway for specific types of evaluation. For instance, some pathways may lend themselves to exploratory evaluation to identify the nature of implementation processes and early experiences, while more mature interventions or those where the local data infrastructure is in place may be ready for more impact focused evaluation work.

Following selection of a topic, each potential evaluation will then require a brief topic specification to be co-produced with the identified policy customer (e.g. Department of Health and Social Care, NHS England, NHS Scotland). On confirmation by the funder, we will then scope and produce a full protocol for formal funder approval. Depending on the focus of each evaluation this is likely to involve rapid, scoping desk research and stakeholder consultation to understand patient journeys and service processes related to the remote monitoring innovation, familiarisation with key stakeholders, liaison with participating sites (Figure 1) and consideration of key process and/or outcome measures.

Study design

We will employ a core set of tried and tested methods across evaluation projects, while retaining flexibility for rapid tailoring of our approach depending on the needs of individual cases. The primary research method adopted for each project will depend on the time horizon agreed with the funder, input from the policy customer and other stakeholders, diffusion of the technology being evaluated, availability/accessibility of data (e.g. within electronic health records), and the results of scoping and literature reviews. We will proactively consider how to address inequalities in designing and undertaking evaluations, and use relevant theory that enables explicit consideration of digital inclusion (see below).

Following the process outlined in Figure 2, exact activities for each evaluation will be guided by the stage of adoption and nature of the evaluation but will include familiarisation discussions with patients/PPIE representatives, staff and other stakeholders to establish initial programme theory; rapid review of the available literature (e.g. using restricted/rapid review⁷⁶ or living systematic review⁷⁷); exploring outcome measures that are meaningful in each context, and agreement of data collection protocols with sites. PPIE contributors will be involved in shaping evaluation plans.

Once the formal protocol is approved by the funder, we will immediately progress evaluation. Data collection will focus on five core evaluation domains which will be adapted or extended depending on the needs of each project and accessibility/availability of (particularly quantitative) data:

Economic research methods will vary. Rapid evaluations of 9-12 months are likely to include economic evaluations based on patient-level observational studies and quasi-experimental designs (e.g. regression adjustment, difference-in-difference, propensity score matching) and economic evaluations based on decision-analytic models including value of information analyses. For more rapid evaluations, we will use alternative methods to facilitate that rapidity, including assessments of economic costs (using provider surveys, patient/client surveys, time/motion studies, other bottom-up costing approaches), budget impact analyses, and stated preference methods (e.g. discrete choice experiments, best-worst scaling). Where data is available and accessible, we anticipate drawing upon routine data sources in health care and complementing this with primary locally collected data in both health and social care settings. We will also apply an equity lens within each evaluation, seeking to ensure that disparities in care provision and outcomes by geography, socioeconomic status, ethnicity and other protected characteristics are adequately addressed by the evaluation (e.g. including consideration of the potential for inequalities as an outcome of the technology). For example, within economic evaluations, distributional cost-effectiveness methods will be used to assess trade-offs between efficiency and equity concerns.

We will work closely with sites to set up data collection (qualitative and/or quantitative as appropriate), and use options for telephone/video interviews as well as face-to-face. We will seek to use maximum variation sampling to recruit a diversity of patients and carers, for instance sampling across sociodemographic and ethnic backgrounds, health needs and conditions, and with different levels of self-assessed digital literacy/confidence/access.

In each evaluation we will conduct data analysis in parallel with data collection and feed findings directly into iterations of programme theory. We will establish the summative impact of the technology-based remote monitoring service across relevant qualitative and quantitative metrics.

Evaluation timelines will vary depending on scope and scale of projects. We will continue to revisit plans iteratively to ensure we remain well-placed to respond to shifting priorities and emerging learning. Evaluation findings will be documented in narrative form, describing how the technology was implemented.

Engagement with theory and development of cross-cutting themes

DECIDE evaluations take as a starting point that the success of an intervention (e.g. new technology) is intimately related to its interactions with the conditions and actions in the broader adoption context and wider health system⁷⁸. Evaluations are informed by relevant theory, including the Non-adoption, Abandonment and challenges to Scale-up, Spread and Sustainability (NASSS) framework, an evidence-based framework consisting of 7 interacting domains, that guide thinking on sociotechnical implementation, roll-out and embedding of technology-supported innovations in health and care^64,79. NASSS will inform study design and data collection on the multiple interacting influences affecting implementation and sustainment of technology-enabled remote monitoring. The NASSS framework has already proven useful in previous research on remote monitoring (e.g. concerning heart failure and other cardiovascular conditions)^23,80,81. Work is underway by the team to adapt the NASSS framework to the specific context of social care (to be reported separately).

In addition, evaluations will draw on a number of theoretical strands of work from science and technology studies, medical sociology and organisational theory (see Box 5) to underpin data analysis and synthesis. Where feasible and relevant, this theoretical engagement will enable development of cross-cutting themes (including, but not limited to, DECIDE’s focus on digital equity and equality) and comparative analysis across evaluations.

Programme management, ethics and governance

All evaluations will have senior oversight and draw on the Centre’s pool of experienced research staff from its collaborating organisations.

The programme will be overseen by an independent Steering Committee with diverse representation from policy, clinical care, social care, the commercial sector, and people with experience on patient advocacy groups or regulatory bodies; and with voices from across the four UK nations. The Steering Committee will advise on strategic and operational aspects of DECIDE activity, oversee independence and quality, and feed into topic identification and prioritisation (Figure 1). Additional input will be sought at the evaluation level, including (but not limited to) from service providers and health and care professionals. A cross-partnership Internal Advisory Group consisting of senior Oxford, RAND and Health Innovation Network representation, will provide topic and method-relevant advice and contribute to the design, planning, implementation and delivery of DECIDE’s work.

We will submit a classification request to the Research Governance Ethics and Assurance (RGEA) team at the University of Oxford to confirm the appropriate approval process for each rapid evaluation: (a) service evaluation that does not require research ethics approval, (b) research not requiring NHS HRA ethical approval, or (c) research requiring NHS HRA approval. We will follow the confirmed approvals process for each evaluation. All evaluations will be conducted in accordance with the Health Research Authority guidance (UK Policy Framework For Health and Social Care Research)¹⁰³ and with the ‘The Concordat to support Research Integrity’¹⁰⁴.

Dissemination, outputs and anticipated impact

We will support diverse dissemination and engagement channels, engage early on with a range of stakeholders, and enable real-time formative feedback on emerging learning with those directly involved in each evaluation in a timely way to support decision-making and learning cycles, with final learning disseminated more widely.

The focus will be on tailored dissemination and outputs specific to individual evaluations, as well as synthesis/learning across evaluations (e.g. on topics, methods, effective PPIE), including:

Internal quality assurance processes will ensure that all outputs are peer reviewed (see Figure 1). To extend potential reach and impact, we will work with an applied design agency throughout to develop high quality materials and resources and tailor messaging across evaluations.