Inclusivity of the target population in orthopaedic surgical randomised trials: a review of high impact journals

Introduction

Orthopaedic surgical randomised controlled trials (RCTs) are often pragmatically designed to evaluate whether surgery is an effective intervention in a realistic clinical setting^1,2. The evaluation of surgery, lends itself to a pragmatic approach, by its complex nature¹. A critical aim of a pragmatic approach is to be inclusive of a broad sample of participants that will reflect the target population in clinical practice and maximise the generalisability of findings.

Differences in recruitment across sites, however, is common in pragmatic RCTs³. This can result in recruited patients differing from those who are not recruited across characteristics such as age, sex, ethnicity, severity of disease, educational status, social class, and place of residence⁴. Reviews show that trials consistently fail to report participant flow accurately, particularly before informed consent and randomisation^5,6. As it is not always clear how many patients were screened for inclusion and why they were not randomised, the results may not be accepted by the surgical community. The National Institute for Health and Care Research (NIHR), the United Kingdom’s (UK) largest public funder of trials, has begun to focus on the inclusion of under-represented groups in health care research. Various grouping have been suggested for consideration and include the following: demographic factors (e.g. age, sex, ethnicity); social and economic factors (e.g. employment, socio-economic status, geographic location, language, digitally excluded); and health status (e.g. mental health condition, cognitive impairment, physical disabilities)^7,8. Inclusivity in trials is important for improving representation of the target population so important findings specific to different populations are not missed and to avoid potential discrimination towards historically under-served populations. More money, time and effort may be required to be more inclusive, but may lend itself to research that is representative of the whole patient population and as a result, is more informative for patient and clinical decision-making⁸.

For orthopaedic surgical trials, the focus has been on improving internal validity^1,9 rather than on their applicability to practice¹⁰. A criticism of such trials has been that the screening, choice, and application of eligibility criteria has meant many patients are excluded. This may affect whether clinicians accept the results of a trial if not considered to be reflective of their usual patients. Treatments may then not be used that could benefit patients and optimise the efficient use of NHS resources, or alternatively, are continued to be used when they have limited clinical and/or cost-effectiveness. This lack of representation has been interpreted as limiting the applicability of the findings of orthopaedic surgical trials^11,12 and can delay their translation into practice and increase research waste¹³.

In the absence of literature exploring the applicability of orthopaedic surgical trials, that such trials have been criticised for a lack of patient representation, and the emerging policy to be inclusive of underserved groups we judged it was timely to conduct a review on this topic. We chose to focus on orthopaedic trials that are published in high impact medical journals that are likely to have high visibility and potential to influence key stakeholders and clinical practice¹². The aim of this review of high impact journals was to examine whether published findings of RCTs of surgery in orthopaedics are inclusive of their target populations and suggest practical recommendations for encouraging inclusivity by design in future orthopaedic surgical trials.

Methods

Results

Study selection

3,030 potentially eligible studies were identified. After screening the title and abstract, there was full retrieval of journal articles for 27 studies; one was subsequently excluded that did not include surgery²². Therefore 26 studies were included^17,23–47. Figure 1 summarises the study selection process. Table 1S in the extended data gives a detailed description of the eligibility criteria and patient population.

Figure 1. Flowchart of included studies.

Discussion

This review of orthopaedic surgical trials published in high impact journals illustrates considerable variation in how patients are recruited that could affect clinical applicability and acceptability of trial findings. There is marked variation in patients initially screened, who meet all the eligibility criteria, provide consent and are randomised into the study. Limited data were collected about key baseline characteristics of patients who pass through the different phases of patient selection. Notably only four studies (15%) reported on ethnicity which is similar to a recent review that found only 9.3% (38 of 407) of NIHR trials demonstrated exactly how they both recorded, and reported, ethnicity⁴⁸. Critical to understanding the selection of patients into RCTs is describing their enrolment in the flowchart of the CONSORT statement¹⁸. Most studies reported the different steps of enrolment but there is considerable selectivity of patients from the screened target population to who were randomised into the study. Within included studies we found limited data about ethnicity, education or employment status of patients. The methods did not explain how language barriers were addressed, and what alternative methods of data collection and enrolment of patients without capacity to consent were used. Studies were mostly pragmatic in recruitment of patients, which by definition should have clinical applicability. However, whilst judged to be pragmatic in design²⁰, these findings suggest the need to think beyond what is traditionally considered to be pragmatic and truly be inclusive of all eligible patients and that of under-served populations.

Recently, a lack of patient representation in health care research has become the focus of the NIHR, the largest public funder of trials in the UK. Consequently, there has been an emphasis on including under-served groups. It is known, for example, that for musculoskeletal conditions some minority and ethnic groups are disproportionately represented because of risk factors such as levels of physical activity, vitamin D deficiency, poverty, and pre-existing long term conditions such as diabetes⁴⁹. A recent national survey from a representative sample of 5,030 people from across the UK found nine in ten people (88%) think a diverse mix of participants in health care and research is important even if the research costs more money (70%) or takes more time (74%)⁵⁰. Both leading funding bodies and the public expect to have inclusivity in research.

Sometimes trial teams may deliberately widen their screening to ensure every possible patient is considered for the study. Although there may be legitimate reasons for this, several included studies specified an upper age limit of ≤75 years which could proactively exclude eligible patients. When designing studies around the inclusive selection of patients and optimising the flow of patients careful consideration should be given to: defining the target population, the choice of eligibility criteria, who is involved in the screening of patients and the training they have and methods used to screen⁵¹, methods to minimise patient and/or surgeon preferences⁵² and optimise patient recruitment⁵³ and involvement of patient and public collaborators⁵⁴.

Reporting key characteristics of patients who are screened, excluded because of eligibility criteria and who are not approached or do not consent may help reassure clinicians and policy makers about the representativeness of the trial sample. The General Data Protection Regulation in the UK provides the lawful basis for processing such data and the common law of confidentiality allows the collection of data without a legal basis as long as the patient cannot be identified⁵⁵. As an example, age could be collected in years (or age bands) or only the first part of a postcode to inform measures of deprivation. This allows the lawful and feasible collection of key characteristics of the screened population without the need for consent. A consistent approach from Research Ethics Committees/Health Research Authority and subsequent Information Governance professionals undertaking local site review is required as to what is acceptable to collect that ensures anonymity but permits reporting about inclusivity. This type of data capture may not be permissible, however, due to restrictions of different jurisdictions.

In the UK and the NIHR focus on improving inclusivity in research, frameworks are being or have been developed as to how this may be achieved^7,8,56. This is part of the NIHR workstream called “Innovations in Clinical Trial Design and Delivery for the Under-served” (INCLUDE) project. This includes a roadmap that defines under-served groups and barriers to their inclusion⁸. The Ethnicity Framework launched on 1 October 2020 (https://www.trialforge.org/trial-forge-centre/include/) aims to help trial teams think about the inclusion of ethnic groups in their trial⁷. Multiple approaches to address the barriers to inclusive participation in research include: translation of recruitment and patient questionnaires subject to appropriate validation; and provision of materials in braille, audio-recorded, or animation and apps to help those with low literacy, learning or sensory difficulty. For these tools to be universally adopted into standard trial practice, a coordinated and consistent approach is required to their implementation with a greater understanding of their resource implications to be considerate of the workforce and pressures facing the NHS.

A strength of this review was applying the PRISMA guidelines^14,15. The protocol was registered prospectively. It was conducted by a multi-disciplinary team of methodologists, orthopaedic surgeons and trial coordinators. The review is limited to RCTs of orthopaedic surgical trials in high impact journals that are amongst the most cited medical journals and known to the authors as having published large-scale orthopaedic surgical trials. We chose to focus on these RCTs as they are likely to be the best resourced to deliver research and influence key stakeholders and clinical practice. The review team focussed on reporting what was presented in the main publication and supplementary material available on the journal website. It is possible that details not published by the journal are available in a full monograph, in trial registries or in the published protocol; although being described in a registry or protocol does not necessarily mean it was implemented. Moreover, different journals may have different word count policies and what is permitted to be uploaded alongside an article. This review focused on the journal publication and supplementary material as that is most likely to be read by surgeons and to influence decisions in clinical practice. Several reviewers checked study inclusion and undertook data extraction that could contribute to variability in decision-making. This was to make the review feasible with the lack of resources to support it. Maintaining the standard of a second reviewer checking a first reviewer with recourse to always the same third reviewer should mitigate this limitation. Finally, identifying studies with simple search terms were undertaken of the journal website rather than an electronic database such as PUBMED as this was more feasible with the latter lacking specificity in the searches⁵⁷. Whilst this search strategy may not meet the standards of a systematic review of effectiveness we have nevertheless undertaken a comprehensive review with the resources available to us and makes an original and timely contribution to the literature.

Conclusion

Patient selection and recruitment is a key challenge for RCTs. Different clinical pathways and differences between participating sites and resources available add to the complexity of achieving this. However, the enrolment of a highly selective sample of patients may impact on the clinical applicability and acceptability of study findings. Trials often purport to be pragmatic in design. The limited data available about who and how patients are included in these studies, questions whether they truly are pragmatic and inclusive of the target population. This review is not a criticism of existing high impact orthopaedic surgical trials that are an important contribution to the evidence-base as only recently has there been this attention towards inclusivity and improving external validity. The challenge now is to address this and ensure every person eligible to take part has the same opportunity and are not excluded whether consciously or not. This is a requirement of leading funding bodies and an expectation of the public. This may be difficult and complex to implement as it requires time, resources and funding for which there can be an opportunity cost and needs to be integrated into efficient trial design and delivery⁵⁸. Change will also not happen on its own and needs initiatives that provide training and education on inclusivity in clinical trials⁵⁹ and practical guidance about how to implement strategies to achieve this⁶⁰. The NIHR is starting initiatives to provide training in inclusive research design⁶¹ and regulatory bodies are developing guidance on increasing diversity of people taking part in clinical trials⁶². The promotion of decentralised clinical trials away from trial sites could also improve inclusivity in recruitment allowing participants to overcome geographical, financial, family and work constraints⁶³.

Finally, the following practical guidance could improve inclusivity in the screening and recruitment of patients into orthopaedic surgical trials:

(i) screening and eligibility criteria – including collection of data to allow complete reporting of the CONSORT flowchart¹⁸, careful consideration in the definition of eligibility criteria⁶⁴, and efficient data capture methods to record data on those patients screened, eligible, approached and randomised⁵¹.

(ii) collection and reporting on attributes to ensure no section of the eligible population is inadvertently excluded – including, for example, collecting data during screening on age, sex, ethnicity and (first part of) postcode to inform measures of deprivation which are often not reported yet known to influence patient outcomes^4,65,66.

(iii) embedding mechanisms to allow all eligible patients the opportunity to participate – including making information accessible in a variety of formats such as the translation of recruitment materials; provision of materials in braille, audio-recorded, or animation (that allows captions in different languages); and direction to apps to help read printed materials.

Ethics and consent

This study did not require any form of ethical approval or consent