Keywords
Digital Health Intervention, Cost-effectiveness, Value of Information, Osteoporosis
-
Views
43
Downloads
How to cite this article
Siciliano M, Bathers S, Bentley I et al. Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention [version 1; peer review: 1 approved with reservations]. NIHR Open Res 2024, 4:15 (https://doi.org/10.3310/nihropenres.13575.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
Export Citation
Sciwheel
EndNote
Ref. Manager
Bibtex
ProCite
Sente
▬
✚
Study Protocol
Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention
[version 1; peer review: 1 approved with reservations]
PUBLISHED 03 Apr 2024
OPEN PEER REVIEW
REVIEWER STATUS
Abstract
Plain Language Summary
Corresponding author:
Cynthia Iglesias-Urrutia
Competing interests:
ZP reports grants from the NIHR Clinician Scientist Award (CS-2018-18-ST2-010), Royal Osteoporosis Society and the Haywood Rheumatology Research and Development Foundation to develop the iFraP intervention. ZP has received consultancy fees from UCB Pharma and Keele University has received sponsorship from UCB Pharma. TON is supported by the NIHR Manchester Biomedical Research Centre. CM is funded by the NIHR Applied Research Collaboration West Midlands and the NIHR School for Primary Care Research. CJ is part funded by NIHR Applied Research Collaboration West Midlands Other authors declare no other competing interests
Grant information:
This trial was funded by the National Institute for Health and Care Research (NIHR) [Clinician Scientist Award (CS-2018-18-ST2-010)/NIHR Academy] to Professor Zoe Paskins. The views expressed are those of the author(s) and not necessarily those of the NIHR, NHS or the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Copyright:
© 2024 Siciliano M et al.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
How to cite: Siciliano M, Bathers S, Bentley I et al. Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention [version 1; peer review: 1 approved with reservations]. NIHR Open Res 2024, 4:15 (https://doi.org/10.3310/nihropenres.13575.1)
First published: 03 Apr 2024, 4:15 (https://doi.org/10.3310/nihropenres.13575.1)
Latest published: 03 Apr 2024, 4:15 (https://doi.org/10.3310/nihropenres.13575.1)
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Background
In recent years, digital transformation of health has been gaining considerable importance in the policy space. Digital health has been identified as a priority in national health system programmes1, continental projects2 and global initiatives3. Digital Health encompasses a broad set of medical technologies and interventions designed to improve health4. Different types of products fall under the category of digital health interventions (DHI), ranging from mHealth devices, Artificial Intelligence applications, genomics and, computerised decision support tools (CDSTs).
Digital health technologies (DHT) - and the DHIs as part of which DHT are delivered - have been associated with a number of potential advantages, e.g., improving effectiveness, efficiency, accessibility, safety, and personalization5. However, DHIs also present some peculiarities that make their clinical and economical evaluation more nuanced than traditional health innovations from pharma or medical devices5–11. Some challenges include:
Digital health’s novelty and heterogeneity, which have contributed to a lack of established methodologies for the economic evaluation of such tools8.
The key role of user involvement in determining the efficacy of the intervention, which can critically determine the efficacy of a DHI (e.g., user / health technology interaction)6.
The interventions can have a complex impact, i.e., one that simultaneously affects multiple outcomes6–10. In contrast, the traditional tools of cost-utility and cost-effectiveness analysis are meant to capture one dimension at a time.
The consequences associated with the use of DHI can fall outside the targeted population. This can question the appropriateness of a National Health Service perspective to capture changes brought about by a Digital Health Tool (DHT) and associated intervention6,8.
Often DHI are subject to several changes and updates throughout their development phase (i.e., they are incrementally developed). This makes traditional evaluation tools such as RCT not particularly suited for an evaluation and arguably, unable to match the more dynamic nature of digital health innovations7,10.
Decision Support Tools (DSTs) – also called decision aids or conversation aids - are a family of digital health interventions which are key to support shared decision-making in pursue of personalised medicine, as highlighted in the National Institute for Health care and Excellence (NICE) guidelines on Shared Decision Making12. CDSTs go beyond being simple repositories of information, as they have the potential for individualized content, a high degree of interaction, and scalability13. Moreover, the information contained can be specifically curated for patients with varying degree of health literacy. Consequently, the patient is able to engage more with the clinician, moving from a traditional principal agent relationship to something closer to a partnership14. Nevertheless, the challenges connected to the evaluation of DHI also affect DSTs, as highlighted by Trenaman et al.15.
In this paper we describe the protocol for the within-trial healthcare economic evaluation of a complex intervention including a digital DST: the iFraP intervention16. To the best of our knowledge, our protocol is the first one to illustrate how a within-trial cost-effectiveness and value of information (VoI) analysis will be conducted alongside a randomised controlled trial (RCT) of a DHI. This paper aims to contribute to the literature on how to design a healthcare economic evaluation study to rigorously estimate the value for money of a DHI.
Methods
Illustrative case: iFraP intervention rationale and development
Osteoporosis is a disease that compromises bone structure, making bones weak and more likely to break17. The drug treatments for osteoporosis revolve around fracture risk reduction through bone strengthening. There are different types of osteoporosis medicine given either by tablet or injection (including bisphosphonates, RANK ligand inhibitor; and anabolic agents such as recombinant parathyroid hormone and anti-sclerostin monoclonal antibodies) with calcium and/or vitamin D supplements acting as treatment adjunct18. Despite the considerable burden of the disease and the numerous treatment options, there is a considerable treatment gap19 (i.e. proportion of people in whom osteoporosis medicine is recommended by clinical guidelines, but do not receive treatment). The treatment gap arises because people at risk are either not identified or offered treatment, or are offered treatment, but decide not to take it. The care gap is a newly proposed term that emerged during the development of the iFraP intervention to address the implicit assumption that all patients recommended medicine should take it regardless of their preferences19. This term highlights the gap (and possible solutions) as a clinician problem, not a patient problem, and puts emphasis on the importance of clinician behaviours to facilitate informed shared decision-making between healthcare professionals and patients. The reasons behind osteoporosis undertreatment are multiple: concerns about long term bisphosphonates efficacy18; safety20; uncertainty about what medicine actually achieves21; and, potential overconfidence with routinely used risk calculators known -under certain circumstances- to be associated with a risk of underestimating the risk of fractures22. To an extent, promoting better clinician-patient communication on bone health and treatment options, could contribute to address some of these challenges.
Some osteoporosis decision tools have been developed to improve clinician-patient communication. However, these have not met the required international quality criteria and have as yet, failed to demonstrate effectiveness at improving drug initiation and/or persistence (collectively described as adherence23)24. The Improving uptake of Fracture Prevention drug treatments (iFraP), a complex digital health intervention, features a DST with an interactive representation of individual fracture risk, information about risk factors for osteoporosis, bone density, and accessible explanations of bone health and treatment risk/benefit profile, and has been co-developed to achieve this need.
A detailed description iFraP’s development is provided elsewere16. Briefly, the iFraP intervention includes a CDST to support the patient-clinician discussion, a training package for the clinician on how to use the tool and consultation skills enhancement, and additional information resources for the patients, along with a print-out from the CDST; most elements of the complex intervention support the use of (training), or are a by-product (print-out), of the CDST as a DHI. All the resources have been conceptualized and developed following the MRC complex intervention development framework25, comprising i) an evidence synthesis including an evaluation/assessment of the available online patient information26; ii) an assessment of existing decision tools in osteoporosis24; iii) a Delphi study to inform the model content of the FLS consultations27 and iv) focus groups and individual interviews with patients and health professionals28. iFraP was developed following a collaborative approach including bone specialists, nurses, people with lived experience, GPs, behavioural psychologists, representatives from Health Literacy UK, the Royal Osteoporosis Society, academics, and health economists.
iFraP trial outline
The iFraP team designed the study to evaluate the effect of the iFraP intervention on patient-reported ease in decision-making about osteoporosis medicine, to examine the experience of care and effectiveness, and to assess the within-trial cost-effectiveness and VoI of the iFraP intervention compared with usual Fracture Liaison Service (FLS) practice. FLSs are services which enact secondary fracture prevention, and assess people who have low trauma fractures for osteoporosis. This is a two sided multicentre individual randomised controlled trial (RCT), with parallel process evaluation and health economic evaluation. Participants are randomised in a 1:1 ratio, using blocked randomisation stratified by FLS. The planned sample size is 380 patients who have sustained a fragility fracture (following a fall from standing height or less); have been referred to FLS; and, are set to receive a consultation from an FLS29.
Study population
Adult patients aged 50 years and over who are eligible for and FLS consultation based on having a previous fragility fracture(s).
Intervention and comparator
The intervention is a consultation delivered by FLS clinicians with the aid of a dynamic, interactive, and patient-tailored CDST to communicate individual fracture risk, bone health and treatment recommendations to patients. The clinicians in the intervention arm will receive a dedicated training course and will partake in a 3-hour role play session with experts to familiarize with the new consultation. Lastly, additional information resources will be given to both the patient and the GP after the consultation. The comparator is the usual FLS NHS care, which does not involve the use of CDST to support patient-clinician discussion. The clinicians in the comparator arm will not have access to the iFraP training and resources.
Start and end dates
The data collection started on April 2023 and it is planned to end in July 2024, in line with a three-month follow-up duration for the trial.
Type of economic evaluation
Within trial cost-effectiveness analysis of iFraP intervention compared with usual FLS practice and VoI analysis.
Study perspective
The perspective for the analysis will be that of the English National Health Service and Personal Social Service.
Costing year
The costing year will be 2023. Main sources will be the 85th volume of the British National Formulary (BNF) (Covering the March-September 2023 period) and the 2023 edition of the Unit Costs of Health and Social Care manual (PSSRU).
Time horizon
Due to the within-trial nature of the analysis, we will employ a three-month time horizon, equivalent to the length of the trial follow-up.
Discount rate
Given that our time horizon will cover less than a year, we will not discount costs or health outcomes.
Health outcomes
The EQ5D-5L questionnaire is a standardised tool developed by the EuroQoL group, used to estimate generic health-related quality of life (HRQoL)30. In the questionnaire, respondents are asked to rate their severity level in the five areas of life (dimensions) of mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. In this version of the questionnaire, there are 5 possible answers to each question, equating to (1) no problems, (2) slight problems, (3) moderate problems, (4) severe problems, or (5) inability in the domain. The final profile of the respondents is a sequence of 5 digits (e.g. 12322), corresponding to the answers in the 5 dimensions. Patient participants will self report their health status at baseline, 2-week, and 3-month follow-ups.
Resource use and costs
There are four main categories of costs: health care (services) utilisation, medications use, iFraP intervention, and medication side effects. Self-reported healthcare utilisation and medication use data will be collected via a patient questionnaire administered at baseline and 3 months. Osteoporosis medication uptake and adherence will be cross-referenced with hospital medical record review data. Further details of this can be found in the clinical protocol (submitted to NIHR Open Journal).
Interventions cost
Micro-costing will be used to estimate the unit cost associated with the development and delivery of the iFraP intervention. This will include decision support tool development cost and training costs. Consultation costs for iFraP and usual FLS consultations will not be estimated as this is the same for both study groups.
Bayesian Value of Information (VoI)
This analysis will evaluate the Expected Value of Perfect Information (EVPI) which captures the hypothetical value of simultaneously eliminating all uncertainty from the economic evaluation. Our VoI analysis will be conceptualized as rapid VoI, in a framework similar to the one used in research prioritization31.
Analysis
Handling missing data
After examining the data, we will evaluate eventual missing data strategies to implement, following the recommendations presented in Faria et al.32. More precisely, we expect to see a relatively low amount of missing data at baseline, making it possible to impute the missing values by using the group mean for any missing covariate. We will test the assumption of missing-at-random mechanism as described in Cro et al.33 and then inspect the missingness pattern in covariates and outcomes. If a monotonic pattern is found in the missing data over time, we will rely on Inverse Probability Weighting (IPW). If not, we will use multiple imputation (MI) methods.
From EQ5D-5L to Utilities
Currently, NICE recommends the conversion of EQ5D-5L into EQ5D-3L using one of the available mapping functions34. In line with this, we will perform our analysis employing the methods described by van Hout et al.35. In the sensitivity analysis, the mapping function used will be taken from Hernandez et al.36.
Within-trial cost-effectiveness analysis
The cost-effectiveness analysis will compute the intervention Incremental Cost effectiveness Ratio (ICER) and compare it with appropriate cost-effectiveness thresholds. The utility values will be described using beta-based regression models to account for the multimodality and limited acceptable range of values37. The costs will be modelled using GLM models, to account for the characteristic skewness of cost data, as suggested in Mihaylova et al.38. In order to account for potential correlation between costs and effectiveness, the ICER will be estimated by bootstrap techniques, which will also be used for the corresponding 95% Confidence Intervals (CI)39,40.
Analyses will be conducted on the statistical software R.
VoI
In short, the rapid VoI evaluation revolves around describing the intervention(s) net benefit(s) without specifying a full economic model, but only comparing the relative effectiveness of the intervention and its costs. Then, the uncertainty in relative effect and baseline event rates are described by suitable statistical distributions, constructed either by referring to the literature or by expert elicitation. Afterwards, a large number of individual values (i.e., realisations) are sampled from said distribution to estimate the health benefit distributions. Analysing the curves enables to estimate the consequences of uncertainty for the intervention. Following the procedures described in Glynn31, we present in Table 1 the minimum evidence required to perform a rapid VoI analysis.
Table 1. Minimum data requirements to perform a VoI analysis on iFraP trial’s decision problem.
| Primary outcome measure | Treatment adherence* rate at 3 months. |
|---|---|
| Relative effectiveness | Variation in treatment adherence* rate at 3 months. |
| Baseline event rate | Treatment adherence* at 3 months |
| Annual incidence | Number of individuals subject to frailty fractures |
| Minimum clinical difference (MCD) | Minimum variation in treatment compliance rate to produce a clinically relevant change. |
| Cost of the study | In GBP £ |
| Duration of the study | In months |
| Length of time for which the new evidence is expected to be valuable | In months |
| Discount rate | Yearly discount rate for costs and utilities |
Discussion
Osteoporosis impacts around 3.75 million individuals in the UK. The estimated prevalence in the over 50 age group is 21.9% among women and 6.7% among men. In 2019, the cost of osteoporotic fractures in the UK accounted for approximately 2.4% of public health national expenditure29. Bisphosphonates are an effective and cost-effective osteoporosis treaetment41. To date, limited uptake and adherence has prevented realising the promise associated with this bone therapy more fully. A computerised decision tool and consultation model were developed to improve shared decision making to enhance bisphosphonate uptake and adherence (iFraP). This manuscript describes the protocol for the economic evaluation of the iFraP intervention. Its findings will illustrate how a robust evaluation of the value for money of digital health interventions can be conducted in the context of randomised controlled studies.
Strengths
The paper shows how to go about designing a within trial economic evaluation of a digital health intervention (i.e. a decision support tool) and supporting resources. This trial-based evaluation is expected to be associated with high internal validity and minimise the impact of selection bias on our estimates of uptake of bisphosphonate therapy. To explore the decision uncertainty associated with the cost-effectiveness of iFraP to enhance bisphosphonate adherence we will conduct a VoI analysis. An electronic search of trial-based economic evaluation of digital health interventions study protocols identified only three study protocols and neither of these included a trial based VoI analysis42–44. No VoI analysis was found when searching the published literature for protocols of cost-effectiveness analyses of digital support tools.
Limitations
Evaluation of the cost-effectiveness of improving bisphosphonates uptake and adherence present both a short term and a life-time decision problem. The within trial cost effectiveness analysis described here will allow us to robustly explore the short-term impact of the iFraP intervention on osteoporosis medicine uptake, principally bisphosphonates. Estimating the value associated with improving adherence with bisphosphonate therapy over a life time horizon in England would require designing a model based economic evaluation. Funding restrictions limited our ability to design and implement a model based evaluation. Future research may focus on re-evaluating an existing osteoporosis decision analytic model to explore the long-term impact of the iFraP intervention on adherence with bisphosphonate bone health therapies.
Ethics and dissemination
Ethical approval was obtained from East of Scotland Research Ethics Service (EoSRES) (22/ES/0038). Following initial approval from the Research Ethics Committee (REC), they will continually be informed of all substantial changes to the management of the study. Routine reporting will take place in line with REC requirements. Dissemination and knowledge mobilisation will be facilitated through national bodies and networks such as the ROS, journal papers and conference presentations. The results of this study will be made widely and freely available to all stakeholders; a summary of the results will be published on the Keele University and ROS website. Patient Advisory Group (PAG) members will advise on how to translate these into easily understandable messages and on how best to disseminate the results to the wider public. In addition to publications in open-access peer-reviewed journals, we will use NHS networks and links to professional bodies to support dissemination of the findings to all stakeholders and will use social media to promote the findings via our dedicated Twitter and Facebook feeds.
Patient and public involvement
The osteoporosis Research User Group (RUG) at Keele University comprises people with experience of osteoporosis and/or fragility fractures, or carers. These RUG members had substantial involvement in a previous study to identify patient and public priorities for research in osteoporosis, which provided the starting point for iFraP. Furthermore, the study-specific Patient Advisory Groups (PAG) informed and agreed how public contributors will be involved throughout the iFraP programme at the outset. PAG meetings facilitated the development of the iFraP intervention and PAG members specifically commented on the importance of evaluating economic effectiveness of decision support tools and shared decision making interventions16. Furthermore, PAG members informed the design of the iFraP randomised controlled trial including choice of outcome measures and piloting questionnaires, although they did not directly inform the methods of economic analysis. Future meetings with the PAG will contribute to the analysis and interpretation of the iFraP trial results.
Download
Export To
CITE
how to cite this article
Siciliano M, Bathers S, Bentley I et al. Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention [version 1; peer review: 1 approved with reservations]. NIHR Open Res 2024, 4:15 (https://doi.org/10.3310/nihropenres.13575.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.
Open Peer Review
Current Reviewer Status:
Key to Reviewer Statuses
VIEW
HIDE
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations
A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 03 Apr 2024
Views
3
How to cite this report:
Neal D. Reviewer Report For: Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention [version 1; peer review: 1 approved with reservations]. NIHR Open Res 2024, 4:15 (https://doi.org/10.3310/nihropenres.14736.r36795)
The direct URL for this report is:
https://openresearch.nihr.ac.uk/articles/4-15/v1#referee-response-36795
https://openresearch.nihr.ac.uk/articles/4-15/v1#referee-response-36795
NOTE: it is important to ensure the information in square brackets after the title is included in this citation.
Reviewer Report 23 Aug 2025
Approved with Reservations
VIEWS 3
Thank you for inviting me to review this protocol for a trial-based cost-effectiveness evaluation of the iFraP intervention, an example of a complex hybrid digital health intervention. This study addresses important methodological issues around the evaluation of (complex) digital health
... Continue reading
CITE
HOW TO CITE THIS REPORT
Neal D. Reviewer Report For: Protocol for a trial-based economic evaluation analysis of a complex digital health intervention including a computerised decision support tool: the iFraP intervention [version 1; peer review: 1 approved with reservations]. NIHR Open Res 2024, 4:15 (https://doi.org/10.3310/nihropenres.14736.r36795)
The direct URL for this report is:
https://openresearch.nihr.ac.uk/articles/4-15/v1#referee-response-36795
https://openresearch.nihr.ac.uk/articles/4-15/v1#referee-response-36795
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations -
A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Competing Interests Policy
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Examples of 'Non-Financial Competing Interests'
- Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper.
- You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors.
- You are a close professional associate of any of the authors (e.g. scientific mentor, recent student).
- You work at the same institute as any of the authors.
- You hope/expect to benefit (e.g. favour or employment) as a result of your submission.
- You are an Editor for the journal in which the article is published.
Examples of 'Financial Competing Interests'
- You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements.
- You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors.
- You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on.
Stay Updated
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Register with NIHR Open Research
Already registered? Sign in
Comments on this article Comments (0)