The consultation open and close study: A feasibility study of a complex intervention

1.1 Background for study

Patients often leave GP consultations with unaddressed concerns^1,2. This can lead to high rates of re-consultation and increased morbidity in the population. Previous research shows that approximately 27% of patients consulting in primary care have seen a doctor or nurse for the same problem in the last four weeks³, and up to 50% of consultations in primary care are followed by another consultation within two weeks⁴. Although there are no estimates of re-consultation for unaddressed concerns in primary care, we know that problems are missed in up to 50% of primary care consultations², and that reducing consultation rates by just 1% in 2016 could have saved the National Health Service (NHS) over £100 million⁵.

The increased levels of telephone triage since the start of the coronavirus disease 2019 (COVID-19) pandemic may have made it more difficult for patients to communicate their concerns. Telephone consultations tend to narrowly focus on presenting symptoms^6,7 and may direct GP consultations down a more transactional route. For example, GPs may miss mental health problems or multiple patient concerns and lose opportunities for health promotion. Increased levels of non-face-to-face consultations may even lead to delayed diagnoses⁸. This might particularly affect patients who find it harder to communicate by telephone, further entrenching existing health inequities.

Patients seen in primary care often present with multiple complex problems, many of which are unrelated to physical symptoms, and include informational needs on symptom-management or self-care, emotional problems, health concerns or social problems⁹. In the context of multiple presenting problems, GPs tend to focus on physical symptoms¹⁰. While this prioritisation is entirely appropriate to ensure patient safety, any missed opportunities to improve patient understanding and ability to self-care is also costly: a study in 2015 found that increasing patient engagement in their own health could save the NHS £2 billion by 2020¹¹. Small changes to improve the ability of GPs to address patients’ presenting problems, concerns and questions could therefore have considerable impact on the overall NHS budget.

Opportunities to address patients’ problems are commonly missed at consultation opening (when the GP should elicit the patients reason for attendance)². Problems can remain unaddressed at consultation closure, if advice given is unclear, particularly with regards “safety-netting”: i.e. advising patients what to do if the problem does not resolve, or gets worse¹².

This study involved testing the feasibility of an intervention aimed at more comprehensively addressing patients’ concerns in general practice. The intervention focused on consultation opening and closing, incorporating use of an individual-level patient-reported outcome measure (PROM) at consultation opening and written information at consultation closure. It was named the Consultation Open and Close, or COAC intervention.

1.2 COAC intervention development study

The COAC intervention development study was carried out immediately prior to this study and is published in two linked papers^13,14. Both parts of the intervention (the pre-consultation form at opening and the summary report at closing) were developed and tested separately, according to Medical Research Council (MRC) guidance¹⁵.

The “opening” part of the intervention uses a form completed by patients before the consultation which gives their reasons for attending and highlights other common concerns (the pre-consultation form). This includes individualised information (a list generated by the patient of their reasons for attending, and the key issues they would like to discuss) and standardised questions (a short list of questions on common problems, with tick-box answers). The standard questions were based on the Primary Care Outcomes Questionnaire (PCOQ); a validated generic questionnaire developed to capture the main outcomes which can be influenced by primary care, including physical and emotional symptoms and function, self-care, health behaviour, adherence, and a sense of support^16,17. This was adjusted using a person-based approach in three rounds¹⁸.

The “closing” part of the intervention involves a summary of the consultation being handed or sent to the patient. This summary is generated through a programmed coded set of automated actions (known as a “protocol”) which was developed within EMIS Health^®19, the electronic patient records system used in 57% of GP practices in the UK²⁰. The protocol allows GPs to load a clinical template (which is a structured form) to allow them to input information, with the most common types of information generated by tick-boxes (e.g. tests, referrals and generic safety netting). When the GP saves the template, a Microsoft Word report is automatically generated and saved to the record. This can be printed or sent to patients via email or an SMS (short message service), which is a text message sent to a mobile phone. The protocol, clinical template and document template to enable this were developed by One Care, the Federation for GP Practices in Bristol North Somerset and South Gloucestershire (BNSSG). It was developed in the One Care test EMIS system. This system uses Read codes and is used by the GP federation to test protocols before publishing to practices. The patient-facing summary report was developed in consultation with a patient and public involvement (PPI) group and tested iteratively using a person-based approach. The final summary report had two sub-headings as follows:

An example of the pre-consultation form and summary report are shown in Figure 1 and Figure 2 (reproduced from the two linked papers on development and testing of the pre-consultation form¹⁴ and summary report¹³ respectively.

Figure 1. Pre-consultation report.

Figure 2. Consultation summary report.

1.3 Implementation during the COVID-19 pandemic

This study was conceived before the COVID-19 pandemic but began recruitment during the pandemic. The NHS long-term plan had committed practices to offer online consultations from April 2020 and video from April 2021²¹. The COVID-19 pandemic required accelerated adoption of these tools²²; in March 2020, to reduce contagion, the UK government instructed general practices to conduct all consultations remotely unless there was urgent need otherwise²³. Online consultations were implemented in UK general practices throughout 2020. Telephone or online consultation triage models were introduced²⁴, with most triaged consultations done via telephone/video²⁵.

Although patient satisfaction with general practices in 2021 was at its highest level in 3 years²⁶, media reports suggested some patients found triage frustrating and access increasingly challenging²⁷. General practices had a workforce crisis for several years and 2021 was widely acknowledged as being the most pressured time ever²⁸. Many GPs felt that media coverage reporting poor patient access was unfair and demoralising²⁸. The study was implemented in this very challenging context. There was an imperative for the study to be manageable within GP workload, beneficial to the consultation and working to resolve rather than exacerbate these issues in general practice.

1.4 Use of electronic PROMs in primary care at an individual level

PROMs were originally designed for use at aggregate level, to compare the scores of groups of patients receiving different care²⁹. However, PROMs are increasingly being used at an individual-level to inform a consultation, set priorities or aid diagnosis²⁹. Prior to the COVID-19 pandemic, feedback of individual-level PROMs information to clinicians had been used most widely in oncology³⁰. PROMs feedback was found to have a positive effect on patient experience and patient care by promoting patient self-reflection thereby helping patients remember their main concerns³¹, by improving patient-clinician communication³² and by making it easier for patients to share information which they find it difficult to express verbally³³. Since the COVID-19 pandemic, PROM feedback to clinicians in secondary care has become much more widespread. Secondary care clinicians in specialities such as rheumatology³⁴ and cardiology³⁵ have reduced the number of face-to-face appointments and the overall number of follow-up appointments while maintaining quality and patient safety through “remote monitoring”, which includes collecting symptom and/or PROMs information from patients in between appointments. Recognising that the wholescale shift to telephone consultations risked serious problems being missed in some specialities, some secondary care clinicians also collected PROMs information from patients immediately prior to appointments to aid communication and help identify possible hidden problems^36,37.

Collection of symptom information from patients before appointments has also increased substantially in primary care since the start of the COVID-19 pandemic. However, in primary care this is dominated by electronic triage. Electronic triage forms were mandated by the NHS Long Term Plan²¹ and rolled out across general practice during this study. Electronic triage forms have features that are common to ePROMs completed before consultations; they both collect clinical information from the patient which is shared asynchronously with a clinician. However, they differ in purpose and content. Electronic triage forms collect information on symptoms and are primarily used to assess whether and what type of consultation a patient needs and with whom. The patient may not receive a consultation after completion of an electronic triage form, but can be advised to self-care, go to a pharmacist or Emergency Department (ED) or receive advice from the GP through email or the triage portal. In contrast, the primary purpose of ePROMs shared with clinicians before consultations is not triage, as the patient already has a booked appointment; it can serve multiple purposes, include raising clinicians’ awareness of patient concerns or providing more detail on a patient’s problems²⁹.

The current widespread digitisation in general practice³⁸ offers a timely opportunity to integrate an ePROM into clinical practice for use at an individual-level to help identify patient concerns.

1.5 Feasibility studies of electronic PROMs

The NIHR draws a distinction between a “pilot” study (a full trial in miniature, including assessment of outcomes) and a “feasibility” study (research designed to investigate whether a randomised control trial (RCT) will be feasible, which does not include assessment of the primary outcomes)³⁹.

Cluster trials are most appropriate for interventions using PROMs feedback to clinicians, because contamination at the level of clinician or practice is a common problem with such RCTs; clinicians who are trained to make use of certain techniques at consultation opening and closure do not readily “forget” this training for control arm patients in an individually randomised trial⁴⁰. Trials of PROMs feedback to clinicians which have shown effects on patient outcome tend to use randomisation at the level of physicians or practices, rather than individual patients⁴⁰. Randomisation at the clinician or practice level also offers the potential for minimising the potential confusion that individual-level randomisation could cause for patients. In low-risk contexts, a cluster design in which physicians or practices are randomly assigned to prescribe an alternative treatment can be implemented without obtaining individual patient consent for randomisation⁴¹.

The objective of this feasibility study was to test the COAC intervention in a cluster-randomised framework to establish the feasibility both of the intervention and of a cluster RCT of the intervention.

2.1 Objectives

This was a cluster-randomised feasibility study focused on whether the intervention was acceptable and whether the trial was feasible in terms of recruitment and retention. Since the focus was on feasibility, we did not include formal comparison of the outcomes between control and intervention arms. The key research questions were:

2.2 Setting

This study was based in Bristol, North Somerset and South Gloucestershire in six primary care practices with a range of socioeconomic deprivation levels as well as urban, suburban and rural areas.

2.3 Intervention

The intervention comprised the following:

1.    Patients with an upcoming GP or nurse appointment were sent a text with a pre-consultation form to complete this before their consultation. This was done by practice administrators who ran a search on the patient record daily for patients with upcoming appointments and sent them a batch SMS invitation.

2.    Patients received the SMS invitation with a link to the form, configured in the software system REDCap^®42. Patients clicked on the link to complete the form.

3.    Preparation step: Administrators generated a colour-coded report from each patient form and manually upload each report to the patient record before the consultation. The GP reviewed the report before the consultation.

4.    Opening: At the start of the consultation, the GP made it clear that they had read the report then gave the patient a reasonable length of time to elaborate before interrupting, redirecting or closing down.

5.    Consultation: The GP carried out the consultation according to their normal practice.

6.    Closure: The clinician provided a sub-set of patients with a written summary report (given on paper for face-to-face patients or sent by SMS or email for telephone patients) of what was agreed in the consultation. This was only provided to the sub-set of patients who have either had tests ordered, safety netting advice given a referral made or another specific follow-up.

A proposed initial programme theory for how this intervention was intended to work is shown Figure 3.

Figure 3. Proposed initial programme theory of COAC.

2.4 Randomisation

Randomisation was done at the practice level to avoid contamination⁴⁰. We selected three practices in the top two deprivation quartiles and three practices in the bottom two. To achieve a balance on deprivation, the three most deprived practices were randomised one to control and two to intervention and similarly with the three least deprived practices. One or two GPs and administrators per intervention practice were trained in the intervention, including use of the pre-consultation form and consultation summary report. Control practices received a shorter training as the process was simpler for control practices (see workflow in Figure 4).

Figure 4. Feasibility study workflow: intervention and control arms.

Each practice was asked to recruit 30 patients, resulting in 120 in the intervention and 60 in the control (see Table 1). An estimated 1,200 texts were expected to be sent to recruit 180 patients. Figure 5 shows this in an anticipated CONSORT flowchart of recruitment.

Figure 5. Feasibility study: recruitment process and targets.

2.5 Sampling methodology

It was estimated that at least 115 patients (64%) of 180 patients responding to the initial text would provide follow-up data and agree to data sharing from the patient record. This sample size would enable an estimation of the follow-up rate within two-sided 95% confidence limits of ± 14%. An improved follow-up rate would generate a narrower confidence interval. A sample size of 180 was also enough to allow a sufficient pool of participants for interview, assuming 20% would consent to this.

2.6 Recruitment and consent

Practices who had participated in the intervention development study were approached by the study chief investigator (CI) and new practices were approached by the National Institute of Health Research (NIHR) Clinical Research Network (CRN) for the West of England. Practices received a Research Information Sheet for Practices (RISP) and interested practices then contacted the CI.

All selected practices already used SMS software (MJOG® or accuRX®) and the patient records system EMIS.

Patients were included who were:

▪    Aged 17 or over (on date of SMS invitation to participate)

▪    Had an upcoming appointment with a recruiting GP within the next week

Patients were excluded if they were:

▪    Housebound

▪    Had not given permission to receive SMS messages from the practice

▪    Had a recent diagnosis of life-limiting or life-threatening illness,

▪    Were deemed by the GP to be at serious suicidal risk,

▪    Were unable to complete questionnaires in English even with the help of carers.

General practice administrators searched their practice database using an electronic search strategy which identified patients with upcoming appointments who met the inclusion criteria. Batch SMSes were sent to patients with a link to the baseline questionnaire hosted on REDCap. The SMSes contained the patient EMIS number and the patient was required to input this so their questionnaire could be identified.

Administrators received an alert when a patient completed a questionnaire. On a regular basis, administrators downloaded the summary report from REDCap to PDF and attached it to the EMIS patient record system. The baseline questionnaire included an information screen explaining the purpose of the study and how the data would be used. Return of the questionnaire indicated consent. Patients were asked to consent to their contact phone number being shared with the University of Bristol for the purposes of sending a follow-up questionnaire. Consent for use of that phone number to contact the patient for interview and for access to the patient’s record for demographics and re-consultation rates was requested in the follow-up questionnaire⁴¹. A similar approach has been taken for a number of other cluster trials^41,43,44. The researcher then took full informed consent from patients who agreed to be interviewed prior to their interview. This consent was written for face-to-face interviews and audio-recorded for telephone interviews.

2.7 Data collection / measures

Feasibility study data collected included clinician questionnaire data, interview data, and quantitative patient data.

Clinician and administrator questionnaire data: The GP questionnaire included information for each consultation (new/review), modality (face-to-face, telephone or video), whether the pre-consultation questionnaire was useful, and why a summary report was used. Administrators completed a questionnaire indicating the number of SMS invitations sent each day, the number of reports attached and any technical issues.

Interview data: Interviews in the feasibility study (up to 30 patients and 16 practice staff) were conducted by the CI and the project research associate. Patients and practitioners were interviewed to the point of achieving “information power”, i.e. when the data analysis has yielded one or more coherent theories which are relevant to the study aims⁴⁵. We additionally had carried out 26 GP and patient interviews in the intervention development phase. For some of our research questions it was helpful to compare the two phases, so these were also used to inform the analysis.

Quantitative data: Quantitative data included patient-reported outcomes and data from the patient record. The patient-reported baseline data was collected in the initial questionnaire sent to patients as part of the intervention. This questionnaire collected a combination of pre-consultation form data (included as part of the intervention to inform the consultation) and data collected for research purposes. There is overlap between these since some of the PCOQ questionnaire items are used both for the intervention and as a research outcome measure. At the start of the intervention development study the pre-consultation form had 18 questions based on three domains of the PCOQ. Through the intervention development process, five questions were dropped and the two “support” questions reworded to better identify potential social prescribing needs. As a result, 11 of the 13 questions from the pre-consultation form overlap with the PCOQ, but there was a need to ask seven more questions from the PCOQ for research purposes. The quantitative/questionnaire data items which were collected in the feasibility study are listed in Table 2.

2.8 Analysis

2.8.1 Quantitative data analysis:

As this was a feasibility study, outcomes in the intervention and control groups were not formally compared. Instead, the analysis focused on reporting data for planning and for assessing the feasibility of the full trial. The analysis answers research questions 1 and 2.

Recruitment and retention

A CONSORT flow diagram⁴⁶ was produced. Proportions with 95% confidence intervals were calculated using the exact binomial method for the number of patients recruited, retained and completing outcome data.

Patient reported outcome measures

Patient-reported outcome measures were scored using STATA 16. The EQ-5D downloadable STATA scoring package was used⁴⁷ and published scoring rules were used for the CARE measure and the PCOQ^48,49. The COAC pre-consultation questionnaire contained some items common to the PCOQ. Additional PCOQ items were included sufficient to score three domains; the domain “confidence in seeking healthcare” was omitted as it is similar to a measure of satisfaction with the GP and including it in a questionnaire which was going to be shared with the GP may have affected the way patients responded. Means and standard deviations were calculated for each measure. Missing data was not imputed, only complete data was analysed.

Patient record data

The data extracted from the patient record included patient characteristics; consultations in the 12-week period after and including the date of the index consultation; and clinical codes added on the day of the index consultation.

In the EMIS® patient records system used in our recruiting practices, a record is added to the consultations database for different kinds of staff activity, some of which are not actually consultations. To identify the proportion of patients with at least one follow-up appointment within four weeks and twelve weeks, the consultations data extracted from EMIS® was cleaned to:

▪    Only include records added by clinicians, and exclude administrator-added records,

▪    Only include records where the consultation type was face-to-face, telephone, video or online communication,

▪    Treat consultations of the same modality and same clinician type within 30 minutes of one another as the same consultation.

The proportions of patients with repeat consultations within four and twelve weeks of the index consultation were then calculated.

2.8.2 Process evaluation

As well as informing feasibility, the qualitative data collected informed a process evaluation. The process evaluation was carried out using MRC guidance on process evaluation of complex interventions and based within a realist evaluation framework. Realist evaluation is a theory-driven approach which aims to identify core theories about how a programme is supposed to work and test them out to see if they are plausible, practical and valid. The full realist evaluation answers research question 5 and is reported in two linked papers: one for the realist evaluation of the pre-consultation form¹⁴ and one for the realist evaluation of the summary report¹³.

In line with MRC guidance, the process evaluation questions included implementation factors (recruitment and response of practice teams, recruitment and drop-out of patients, fidelity to the intervention, adaptations, acceptability). The analysis reported here answers research questions 3 and 4.

The chief investigator (MM) read and re-read the initial interview transcripts from both patients and practitioners in order to gain an overall view of the accounts given, to identify patterns in the data. MM identified themes against the MRC framework of implementation factors. The analysis was reviewed with the qualitative team (GW and AS) and finalised.

2.9 Success criteria

Based on the evaluation, a set of pre-agreed success criteria (see Table 3) were evaluated, to decide whether to continue (i.e. apply for funding for an RCT), stop (do not progress to RCT), or modify the intervention.

2.10 Patient and public Involvement

This research was informed by patient and public involvement (PPI) both before the study commenced and during the study. PPI contributors received expenses and reimbursement in line with INVOLVE guidance⁵⁰.

The PPI group were heavily involved in development of the intervention and met four times during the intervention development study to inform intervention design. Their input and the impact of this is described in the two linked papers^13,14.

The PPI group also met at the end of the feasibility study to comment on the overall interpretation of the data and discussed how the results could be used in the future either for additional research or how to benefit patients and clinicians in the future. The group assisted with drafting and approving the plain English summary for this paper and other publications.

2.11 Sponsorship, funding and ethical arrangements

This study was sponsored by the University of Bristol. Ethics approval was granted by Frenchay Research Ethics committee⁵¹ and the Heath Research Authority (HRA). BNSSG Clinical Commissioning Group Clinical Effectiveness and Research Team provided research and development approval. The study was NIHR funded and supported by the CRN who liaised with centres on the researchers’ behalf.

Insurance was provided by the University of Bristol as research sponsor. The study sponsor and funders did not have any role in study design; data collection, management, analysis, and interpretation of data; writing of the report; or the decision to submit the report for publication.

The feasibility study was registered in the ISCTRN registry (ISRCTN13471877) and on the CRN portfolio (42005). The study protocol was published before recruitment completed⁵².

3.1 Participants and response rates

Table 4 shows the sites randomised and the patients recruited in these sites. The sites recruited covered a wide range in terms of levels of area deprivation. List sizes were larger than the English average; the smallest list size was just under 10,000 patients and the highest a “super practice” with more than 40,000 patients. The ethnic mix was varied; one practice had over 40% of patients from Asian, mixed, black or other non-white ethnic backgrounds and two had less than 5%.

Figure 6 and Figure 7 show patients’ route to recruitment in both intervention and control arms.

Figure 6. Patient response rates in the intervention arm.

Figure 7. Patient response rates in the control arm.

Overall, the practices exceeded the recruitment target of 120 for intervention practices and 60 for control. From the patients contacted, 27% were recruited. Neither the control nor intervention arms met the target for completion of follow-up questionnaires or data sharing (43% intervention and 25% control versus a target of 80%). Of these, 39/52 (75%) in the intervention arm and 12/18 (67%) in the control arm agreed to data sharing from the patient record. As shown in Figure 6 and Figure 7, some patients were lost because they did not consent to follow-up and some because they consented but did not complete the questionnaire.

Table 4 shows how these recruitment rates varied across the sites. Rates across the two control sites were very similar. The intervention site rates varied more widely, with between 17% (site 2) and 40% (site 4) of patients contacted being recruited, and of those recruited, between 32% (site 2) and 64% (site 4) providing some follow-up data.

3.2 Other quantitative Findings

Table 5 shows the number and characteristics of the patients for whom patient record data was extracted. Because of the low follow-up rates, these figures are not necessarily representative of the patients recruited to this study. Most of the patients were female (67% intervention and 70% control). The average age was 55.6 in the intervention group and 47 in the control group. Based on the patient record, 62% of intervention patients and 40% of control patients had at least one long-term condition. In the intervention sites, 85% were white. In the control sites 50% were white. Ethnicity information was not available for four (8%) patients.

Table 6 shows the patient-reported outcome measures analysis. Of the 122 intervention patients recruited to the study, 117 (96%) completed enough PCOQ items to score at least one PCOQ domain, and 115 (94%) completed the EQ-5D. In the control arm, 72 (100%) of recruits completed sufficient PCOQ items to score at least one domain and 70 (97%) completed the EQ-5D index.

Of those who completed follow-up questionnaire single items, 49/52 (94%) also completed the PCOQ and 48/52 (92%) also completed the CARE measure. In the control arm 16/18 (89%) completed the PCOQ, and 15/18 (83%) completed the EQ-5D and index.

Table 7 shows re-consultation rates. Re-consultation rates were calculated using consultations data extracted from EMIS. 72% (95% CI = 55%, 85%) of intervention patients and 90% (95% CI = 56%, 100%) of control patients re-consulted within twelve weeks. The data necessary to carry out this analysis was consistently coded in the patient record, and it was a relatively simple process to establish replicable data rules to calculate the consultation rates.

Table 8 shows the GP questionnaire results. GPs reported on 111 (91%) of recruited patients. They found the summary report useful in 69% of patients, varying from 59% in site 1 to 78% in site 3. The summary report was used in 37 patients.

4.1 Summary

The process evaluation is presented in the following sections. section 4.2 gives a summary of the interview participants and section 4.3 gives a brief summary of the realist evaluation, which is presented in more detail in the two linked papers^13,14. section 4.4 presents some contextual factors affecting the implementation from the GP and patient perspectives. In section 4.5, fidelity to the intervention and adaptations made are presented. section 4.6 describes technical and process issues. In section 4.7 and section 4.8 the acceptability of the pre-consultation form and summary report are reported from the GP, patients and administrator perspective. Finally, section 4.9 presents the factors which may have affected recruitment and follow-up rates.

4.2 Interview participants

A total of 45 interviews were carried out: 30 patients, nine GPs and six administrators. In addition, the 26 GP and patient interviews from the intervention development phase were also used to inform the analysis. The number of interviews at each site in each phase are shown in Table 9. In the qualitative analysis which followed, patients 1 to 20 were from the intervention development study and patients 30 to 50 from the feasibility study. So that the evolution of their views can be compared, the same identifier is used across the studies for GPs who were in both studies.

4.3 Realist evaluation and intervention outcomes

A key finding from the GP interviews was that the pre-consultation form and the closure report were useful in different circumstances and for different types of patients. The closure report was completed for 30% of patients who completed a pre-consultation form (Table 4). GPs said this was because completing the summary report took a few extra minutes and the time trade-off was not worth it for the other patients. They also felt that there were several patients they would have liked to give a closure report, who were not eligible for one, because they had not completed the pre-consultation form. One GP summarised this as follows:

This GP had participated in the intervention development study when each part of the intervention was tested separately. She felt the technologies should still be available separately. The pre-consultation form was useful for low-risk patients, but the summary report was most useful for frail and elderly who might need a memory aid after complex instructions, yet these patients may not have completed the pre-consultation form.

Because of this finding, separate programme theories were developed for the pre-consultation form and the closure report. These are presented separately in the two linked papers^13,14. The separate programme theories resulted in two sets of theorised outcomes for each part of the intervention and two programme theories to replace the initial programme theory, shown in Figure 3.

For the pre-consultation questionnaire, six outcomes were identified. This included issues being raised that might not have been otherwise, particularly when patients had a concern they found difficult to voice. It also included a wider range of support offered to patients, partly because hidden issues were uncovered and partly because having the information in writing emphasised the importance to GPs and enabled them to quickly focus on what mattered. GPs and patients felt time was used more effectively because the dialogue began before the consultation. Some patients felt their health and wellbeing was improved, partly because they were offered more support and partly through the therapeutic act of feeling more listened to. Feeling more listened to also made some patients more confident in seeking healthcare in the future. Finally, most patients were more satisfied with their consultation. A revised programme theory showing these outcomes and mechanisms is shown in Figure 8. This has been reproduced from the pre-consultation form development and realist evaluation paper¹⁴ but with the addition of how each of the outcomes could be measured, and whether the data to measure it was captured in this study.

Figure 8. Pre-consultation form: revised programme theory and proposed measured from the theorised outcomes.

For the summary report, five outcomes were identified. The key outcome was that patients and their family were clearer on the follow-up required because the report acted as a memory aid that patients could share and discuss with their family. Patients were reassured and empowered by the information. GPs felt that they reflected more on how to plan and communicate the follow-up and this, combined with the report being available on record for other GPs to see, led to a more appropriate care pathway for the patient. Finally, some GPs thought the audit trail on the record would be useful for medicolegal purposes in case of a legal dispute. A revised programme theory showing these outcomes and mechanisms is shown in Figure 9. Again, this is reproduced from the summary report development and realist evaluation paper¹³ but with the measurement aspect added.

Figure 9. Summary report: revised programme theory and proposed measured from the theorised outcomes.

4.4 Contextual factors affecting implementation

GP factors

General practice was forced to rapidly change to a remote consulting model at the start of the COVID-19 pandemic. Although face-to-face consultations increased in 2021, there were still fewer than ever before. GPs also used SMS much more frequently than before⁵⁴. In March 2021, one GP commented:

This meant that the technological environment in general practice was vastly different between the time of the study design and the conduct of the feasibility study. Practice booking procedures and policies had also changed. Many of the feasibility study sites had allocated most of their slots to telephone triage, and most released these appointments on the day. From the point of view of the COAC Study, this meant that the study had to fit in around these new procedures. How this was done depended on an individual agreement with the study CI and each practice principal investigator (see fidelity and adaptations).

The feasibility study also coincided with patient volumes increasing to pre-COVID levels and the roll-out of the vaccination programme. This put GPs under unprecedented pressure. As one GP said:

In the context of the COAC study, this meant that practice recruitment was slightly delayed, and the intervention had to be as time effective as possible for GPs. The availability and experience of administrative staff was also a factor. Some of the practices we recruited were short of administrative staff during the period of recruitment and keen for the administrative input to be kept to a minimum.

Patient factors

Some patients had perceived a deterioration in access to general practice following the switch to remote consulting. Patients described both finding it difficult to get through on the phone and finding it difficult to get their needs met by phone when they did finally get through. Two patients explained this as follows:

Patient 28 felt that it was difficult to contact the practice even before COVID, but that COVID had made it worse. Patient 26 felt frustrated they could only get an appointment by telephone and felt that this did not meet their needs. This is particularly relevant for the COAC study, as the intention was that it would make telephone consultations less transactional and patients feel more listened to, so GPs did not seem “disinterested”. A minority of patients preferred the switch to telephone:

Many patients found that the lack of continuity of care combined with the very limited time in GP consultations made it difficult to explain their problems to the GP:

This patient struggled to get continuity of care but had identified that they would benefit from continuity of care and felt time was wasted at the start of the consultation repeating the history. This was a common complaint from patients. Many of the patients we interviewed said they often found communication difficult with their GP and in some cases this is why they opted to complete the pre-consultation form. There were, however, many exceptions to this, where the patient either did not have difficulties with access or continuity or managed to receive a good service despite having issues.

4.5 Fidelity and adaptations

Fidelity

Because of the COVID-19 pandemic regulations, the researchers were not able to directly observe consultations to assess fidelity. From the patient interviews it appeared that GP fidelity to the “Open” part of the intervention was strong. Most patients felt that the GP had read the form, let them know they had read it at the start of the consultation, often by raising issues before the patient did. Patients also reported that GPs listened at the start of the consultation rather than diverting it down the route of the issues on the form. GPs confirmed that they followed this part of the intervention as it was explained in the training:

There were two or three exceptions where patients said that the GP did not mention the form and they thought it had not been read. Is seems possible this was due to a process failure whereby the report was not uploaded to the patient record in time (see technical and process issues).

Patient and GP interviews also suggested a high level of fidelity to the consultation summary report. Overall, the summary report was used with 30% of patients, but there was a wide variation within practices from 15% in one practice to 56% in another. As covered in the training, GPs completed the summary in patient-friendly language, summarised the advice in the consultation and sent to patients on the same day.

Administrators mostly followed the process described in the technical guide. Part of this process was to check the list of patients for exclusions that would not be picked up by the EMIS query (for example, patients with a recent life-limiting diagnosis). Some administrators extended the exclusion criteria, for example one practice did not send the invitation to patients who were booked for an intra-uterine contraceptive device (coil) clinic, as the recruiting GP felt it would not be useful for these patients to complete a form. Other administrators did not have time to go through the list for exclusions, so sent the SMS to all patients with booked appointments.

Practice-level customisation

The generic procedure documented in the protocol was adjusted for each practice to fit with local booking procedures and administrator capacity. Some sites sent the SMS invitations out in the morning to patients with same day appointments, and administrators attached the reports as they were received through the day. One site lifted the embargo on same-day telephone triage so that the SMS invitations could be sent the day before and the reports attached as a single task the following morning. Another site sent the SMS invitations once a week to patients with pre-booked appointments.

The free-text in the consultation summary report was customised for each practice to reflect local practice and procedures, e.g. on safety-netting advice for fast-track referrals, or procedures for test booking. Any minor changes like this were agreed after the training session and implemented between the GP training session and the first day of recruitment.

Adaptations

GPs were provided with a broad outline for the intervention but, within this, were encouraged to adapt the process to fit their consultation style. In letting the patient know they had read the form, some GPs simply said “Thank you for completing the form, is there anything else you want to add before we start”. Other GPs raised issues at the start of the consultation “I see the pain in your side is really bothering you and I’d like to support you”.

GPs gathered information from the pre-consultation form in slightly different ways. Some GPs used the form as a prioritisation tool by summarising the patient’s problems at the start of the consultation agreeing which to focus on. All GPs read the more detailed information written by the patient as highlighted in blue. Some used the traffic light system to pick out the aspects where the patient was more bothered. Some GPs ignored the traffic-light system for the individual lines but used them as a whole to gauge the patients’ overall well-being and mood (for example a report that was mainly green was likely to be a quick consultation with no hidden agenda whereas a report that was mainly red and amber indicated that the patient had a poor perception of their own health and wellbeing). Some used the traffic lights for mood and health concerns but paid less attention to them for pain and other physical symptoms.

With the consultation summary report, GPs varied in the level of detail they provided and much of the summary they wrote up during the consultation and how much afterwards. For phone consultations, most GPs entered at least some information into the EMIS template as they were consulting with the patient, then generated and formatted the word document afterwards.

This GP wrote up the patient summary in the consultation but waited until the patient had left before running the word macro to format the summary (referred to as “Alt-F” in the quote) so they could concentrate on getting this right. Some GPs read the report aloud to the patient as they typed to engage them in the process and create a shared decision-making plan. One GP also sent the reports via SMS even for face-to-face consultations:

Some GPs modified the way they took notes in EMIS so they did not duplicate information in the template:

This GP modified her consultation notes; putting them in a very summarised format before launching the COAC template and then using the template to store the notes that they might previously have put into EMIS without the template. This got easier with practice, and one GP was just starting to do that towards the end of recruitment.

Other GPs pointed out that there is a need for some technical language in the patient record, so the COAC template would never entirely replace the consultation notes.

4.6 Technical and process issues

Pre-consultation form

The pre-consultation report required use of REDCap, MJOG and a macro-enabled spreadsheet. There were some technical issues with all these systems. This meant that the process required much more support from the CI than had been anticipated. The macro-enabled spreadsheet generated some errors in practices using Office 365. A REDCap server failure at one point meant administrators were unable to generate reports and recruitment was briefly paused. A REDCap server update on another occasion meant the links no longer worked and needed to be replaced. The process adopted in the feasibility study involved patients keying in their own EMIS numbers (as opposed to having individualised links which was used for the intervention development study). A few patients keyed the EMIS number in incorrectly. Administrators were normally able to recognise the typo and attach the correct report to the record.

Some administrators were un-used to the SMS system used (MJOG). Because of this, one made an error when sending the messages such that the patients received a message without an EMIS number within it. This meant that those patients were unable to complete a report, as the first screen asked for the EMIS number. Nearly all administrators commented that they would prefer to use a different SMS system (accuRX) as they were more familiar with it. However, at the time of the feasibility study, accuRX had just developed its batch SMS functionality and this did not include the ability to customise the message with a different EMIS number for each patient.

Finally, there were some process issues which meant that the pre-consultation report did not get uploaded onto the patient record in time for the GP to see it:

This GP was in one of the sites where the on-the-day triage system meant that the SMS invitations were sent out on the same day as the appointment, so the window of time for the patient to complete the report and the administrator to upload it to EMIS was short.

Consultation summary report

Using existing software for the consultation summary report worked relatively well. The process was designed so that, once the GP had completed the template, the production of the report and process of sending to patients was quick and easy. However, it did involve three separate actions (pressing save on EMIS to generate the report, running the macro to format the report, attaching to accuRX from EMIS). If any of these steps failed, this was frustrating for GPs. In one practice, the button to attach a document to accuRX directly from EMIS was not enabled. This meant that the GPs in that practice had to save the document to the desktop:

In another practice, the GP moved to a different computer which did not have the word macro installed on it.

Both GPs seemed unconcerned about the technical glitches; GP 2 said it “just took a bit longer” and GP6 said it was done “within a few seconds”. However, even if these technical faults only add slightly to the time, this is still additional time which GPs do not have and may dissuade GPs from completing the summary report.

GP practices all required customisation of the prefilled text to reflect local practice; for example, phone numbers to call if referrals were not received. Because the summary report was developed in a Read code system, the practices were not able to make direct changes to the free text, One Care created and republished an adjusted version for each practice. Although this did not impact the practices, it would be difficult to administrate if rolled out more widely.

4.7 Acceptability – pre-consultation form

Administrator perspective

Of the four intervention sites, one administrator felt an RCT would be easily manageable and the other three had reservations. The extent of the administrative burden depended on the practice appointment booking procedure and policy. In sites 1 and 3 appointments were booked on the day, and the administrator had to send out SMS invitations daily with a short window (sometimes only 1 hour) between the patient completing the pre-consultation form and the appointment.

Both these administrators found it difficult to send out the SMS invitations in the morning when the appointment was on the same day, as they had to remain alert throughout the day for completed forms, which was more difficult than uploading the reports as a single task.

Site 4 had adjusted their booking processes so that the forms could be sent out the day before. This administrator found the task much more manageable:

This administrator found the task straightforward and expressed interest in extending the study. However, the task was only acceptable to this administrator because a change was made to the embargo on appointments for the recruiting GP. If more GPs in the practice were recruiting, this change may not be possible without disrupting practice policy. The administrator also pointed out that she might need to reduce her levels of checking the patient list if more GPs were involved.

Site 2, in theory, should have had a similar experience to Site 4, as this site also sent the SMS invitations out in advance. However, this site was beset by technical difficulties:

This site had been involved in the intervention development study and the administrator explained that she had not found the process difficult then. However, because of three different technical problems coinciding, the feasibility study was more difficult, and this site had the longest recruitment period, but lowest number of recruits in the study. When asked if an RCT was feasible the administrator said:

The two control practices both found the process relatively easy, although each had a minor technical problem; in one practice, a patient entered the EMIS number incorrectly so they needed to correct it and in the other, the REDCap link stopped working. If these problems were resolved the administrators felt an RCT would be feasible provided they were given the time to send the message daily.

The study CI was available throughout the recruitment period to assist with technical problems at short notice. This was possible because of the small number of recruiting sites but may be more difficult in a larger RCT with multiple sites recruiting simultaneously.

GP perspective

The pre-consultation report was straightforward and acceptable to GPs who felt it fitted well into their mode of operating, particularly with the shift to more telephone consultations.

Most GPs were aware of the difficulties administrators had, so caveated the level of acceptability when asked if they would do an RCT or incorporate the pre-consultation form in routine use:

When asked the direct question of whether they would be happy to use the pre-consultation form in clinical practice this GPs said yes, but only if it was automated so that it did not require so much administrative input. Other GPs agreed with this; ignoring administrative difficulties, the form was useful and simple:

GP 2 explained that, even though the process was straightforward, it felt unfamiliar at first. Training and practice were both important factors in improving acceptability. GP 3 explained that, before their training session they had been concerned that the pre-consultation element of the intervention would be time-consuming as they had to read the document before speaking to the patient but, in fact it did not take too much time. The CI was aware from previous research that GPs often over-estimated the time taken and underestimated the value added in these situations. She therefore timed the GPs reading the pre-consultation report in the training sessions. GPs were then asked to estimate how long they thought it took, and often overestimated. Through this process, the GPs realised that reading the pre-consultation form was a time-efficient way to gather information. One GP commented on this:

The training session highlighted to this GP that the document did not take as long to read as they thought, and that the colour-coded format enabled them to obtain relevant information very quickly. Other GPs commented that they felt slightly awkward the first few times, as they were un-used to greeting the patient with a synopsis of their problem, but this got easier over time.

Patient perspective

In considering acceptability from the patient perspective, it is necessary to make a distinction between the acceptability of the intervention, and the acceptability of the research. The first half of the baseline questionnaire was the pre-consultation form completed as part of the intervention. The second half was additional questions completed to inform the research. Some feasibility study patients commented that the questionnaire felt too long and repetitive.

Patient 6 felt the questionnaire length should be reduced. However, in making this assessment they did not distinguish between the part of the questionnaire which was necessary for the intervention and the part necessary for research. Of the patients interviewed in the final round of the intervention development phase, all said that it was very straightforward to complete. No-one mentioned questionnaire length and three explicitly mentioned that it was quick to complete:

These quotes suggest that it was the research part of the questionnaire that the patients found long and repetitive because patients did not find the questionnaire too long in the intervention development phase (when it only included the questions required for the intervention), but some did in the feasibility study (when in included the questions for research purposes).

The pre-consultation form had relatively good face validity. When asked if they thought the questions were relevant and informed their consultation, most patients said yes.

This patient was pleased to receive the form and felt when completing it that it would certainly help her consultation.

Other patients found the form more face valid for the first half of the questions (on pain, physical symptoms, life effects, mood and health concerns) than on the second half (on health knowledge, support, adherence, healthy lifestyle and confidence in plan). One patient suggested the form was “mixing two things”; the short-term that the patient wanted to deal with in that consultation and the longer-term which the patient may or may not want to raise in the consultation, and this hampered face validity when they were completing the form. A few patients with long-term chronic conditions found it difficult to complete the form when they had ongoing issues they didn’t want to discuss in that consultation:

This patient was not sure whether to indicate pain on her form as it wasn’t the reason for her attendance.

Some patients had problems with specific questions on the form. One patient did not like being asked about whether they needed support with leading a healthy lifestyle:

Some patients were confused by the final question “how confident are you that you are on the right path to dealing with your health problems?”. Mainly referring to this question, one patient said:

Despite these problems with face validity, all patients who completed the form found it sufficiently face valid to complete; all patients said they would be happy to complete it again; and in the majority, face validity was strong.

Comparison with routinely used electronic triage forms

Both patients and GPs drew comparisons with the practice electronic triage forms, which were mandated by the long-term plan. Some GPs commented about this without being prompted:

These two GPs commented that the COAC pre-consultation form was easier to process and more useful than the practice electronic triage forms. Some GPs felt the colour-coded fixed format traffic light system could be usefully transferred to the triage form.

Patients also commented that the form was easier to complete than the practice online triage form:

Patient 3 commented they were initially “cynical” because of their experience of completing the practice online triage form but found the COAC form much more helpful. Another patient agreed with this and pointed out that the triage element of the other form made it difficult to fill in:

These quotes suggest that, despite having some issues with face validity in a minority of patients, the COAC form was still more acceptable to the patients who complete it than the practice’s online triage form. This may be not just due to the format, but because the patients were given a choice to complete the COAC pre-consultation form, whereas the electronic triage form was used by some practices as gateway to patients seeing a GP⁵³.

4.8 Acceptability – consultation summary report

GP perspective

Generating the summary report was straightforward and acceptable to GPs. GPs liked the result, both in terms of content and formatting and apart from minor technical problems, found it simple to produce.

As with the pre-consultation form, GPs felt their use of the summary report improved with practice.

The action of writing the report in patient-friendly language, sending it to the patient and reflecting more on the follow-up in order to do this takes time. One GP pointed out that, although this was only a few minutes, this could add up if used for lots of patients:

These GPs both felt that although production of the report took very little time, in the current time-pressured context this was still significant. Another GP pointed out that it was important to select the patients who would benefit most from receiving the report:

Only one GP (GP 2) thought completion of the summary report did not necessarily take any more time. This GP explained they typed the report as they were going along and used it in consultations that were complex. They also pointed out that, had they not completed a summary they may have spent more time verbally summarising to the patient and ensuring they understood. However, this GP tended to write shorter summary reports than the other GPs and at least one of their patients found the summary report they generated too brief to be useful. It therefore seems clear that, if the summary report is detailed enough to be useful to patients that it will take GPs a few minutes more time per consultation.

Patient perspective

The summary report was very acceptable to patients. GPs asked patients if they would like to receive this, and the majority of patients readily accepted. Patients were unanimously confident in opening the text, because they had already been informed by their GP that they would receive it and because it came in an SMS from the practice. Patients had to put in their date of birth to access the summary report, which felt secure to them, and left them in no doubt that the message had come from the practice.

Out of all the patients interviewed, only two did not find the summary report useful. Both these patients felt they had a poor consultation and were generally unhappy with their GP practice. They felt there was a lack of detail in the report and this did not compensate for the poor consultation:

Although the majority of patients were happy with the content of the report, patients varied as to how technically easy they found it to access the content. Some patients who accessed their patient record online suggested that the summary report should be accessible via Patient Access. Some patients were disappointed that the link was only valid for two weeks and some would have preferred to receive the summary by email; most GPs did not offer this option as they could not send it directly from the patient record via email, but could do this via SMS. The accuRX functionality to send the report to email was rolled out during recruiting and the recruiting GP in site 4 offered patients the choice between email and SMS.

4.9 Factors affecting recruitment and follow-up rates

Recruitment rates:

This study exceeded the target recruitment rate of 15% of patients responding to the SMS. Some intervention arm patients completed the form to help communication in the consultation and others because they wanted to help the research study:

Patient 22 above completed the form because they thought the intervention would be beneficial. Patient 21 completed it to help the research. Patient 13 also completed it because they thought the intervention would be helpful, although they were not fully confident that the GP would read the form in advance.

The SMS method of recruitment may have put off some patients. We were unable to interview non-responders to confirm this, but some patients who completed the form had initially hesitated:

This patient explained that they read the message carefully before completing it as they were initially unsure about whether it was “rogue”. This may have also affected the non-responders.

Follow-up rates

As shown in Figure 6 and Figure 7, the follow-up rates were 43% in the intervention arm and 25% in the control arm. This was lower than had been anticipated. In later interviews, patients were asked about this. Patients who completed a follow-up normally did so because they wanted to give feedback about a good experience, or they understood that completing the questionnaire was important for the research study:

These three patients all explained that they completed the follow-up questionnaire because they had a good experience of the intervention and wanted to inform this intervention. They all understood the purpose of the follow-up questionnaire.

Patients who did not complete the follow-up gave their reasons as: 1) didn’t understand the reason for it, 2) too lengthy and participant feels they have “done their bit” already, 3) they forgot, 4) they were not sure the follow-up text was genuine, and 5) they did not remember receiving a text.

Most patients who had a good experience of the consultation did not complete the follow-up because they forgot or didn’t understand its importance, not because they were unwilling. For example, patient 18 said it simply “slipped down the to do list”. Patient 33 did not understand why they received it:

When the purpose of the follow-up questionnaire was explained to patient 33 by the interviewer, they immediately completed it.

Patients who had a less positive experience of the intervention were sometimes less positive about completing the follow-up questionnaires. For example, one patient who thought their form had not been read was asked about the follow-up questionnaire and said:

Patient 48 was one of the few who did not feel their form had been read. They therefore felt that the pre-consultation form took up their time without giving them any direct benefit and therefore did not feel inclined to complete the follow-up.

We do not have data on why control arm patients were less likely to complete follow-up questionnaires. However, control arm patients did not receive an intervention so may, like patient 48, have felt less inclined to complete a follow-up or less likely to understand the importance of completing the follow up.

6.1 Main findings

This feasibility study did not meet all of the criteria for progression to an RCT. Despite this, both the pre-consultation form and the summary report were feasible and useful for patients and GPs. However, because they were useful for different types of patients, they are best considered as two separate interventions and evaluated independently. The additional time needed to generate summary reports meant GPs preferred to use it selectively in patients most likely to benefit. The process of sending the SMS invitations to patients and sharing the pre-consultation questionnaire with GPs was technically challenging for administrators and required support from the study CI. This technology needs further development to allow closer integration with routine IT systems before further evaluation is carried out, which should be possible, given current technical solutions available.

Follow-up rates were low; only 36% of patients completed the follow-up questionnaires and 26% agreed to share data from the patient record. It is likely this is due to the online method of recruitment which, while efficient, may have meant the patients were unclear about the importance of returning the follow-up information. Alternative data collection approaches would therefore be required before an RCT is feasible.

6.2 Strengths and limitations

This study was successfully implemented during a challenging time for UK General Practice. GP Principal Investigators in the practices perceived it as a highly successful study and recruitment levels were high. There was no obligation on practices who had participated in the intervention development study to continue to the feasibility study. Despite this, all practices who participated in the first phase continued to the feasibility study, indicating the engagement of GPs with the intervention. We recruited a representative mix of practices across different deprivations, sizes and ethnicity mixes. The training of GPs and administrators was very successful and we conducted a high number of interviews which contained ample information power for our analysis⁴⁵. For the patients who consented, we were able to successfully extract the data from the patient record and successfully count repeat consultations within four and twelve weeks.

The study had some limitations. We were unable to interview non-responders. This meant that we did not get the perspective of patients who did not complete the questionnaire. We found that the pre-consultation form was very acceptable to the patients we interviewed, however; there was a 26% response rate and the remaining 74% may have found it either unacceptable (for example, were unable to complete it, or found it too long) or lacking face validity (not relevant to the reason for their attendance). We did not interview patients in the control arm, although we did interview GPs and the administrator in the control arm. Although we aimed for practices with a mix of deprivation status, rurality, age and ethnicities, all practices had similarities based on their being in the same clinical commissioning group, for example all used the same online triage platform. Some of the implementation and context findings may, therefore, be less transferable to other areas of England. We had substantial attrition in the study but, because we did not collect baseline characteristics of patients responding, we are only able to compare the characteristics of patients who agreed to record sharing (and thus completed the follow-up data).

6.3 Pre-consultation form feasibility

The pre-consultation form was acceptable to patients and GPs but less acceptable to the practice administration. Although a minority of patients had problems with face validity of certain questions, all said they would be happy to complete the form again, indicating a base level of face validity. Both patients and GPs mentioned that the pre-consultation form was more useful than the practice online triage form. Other studies have noted that patients can perceive electronic triage forms as a barrier⁵⁵, so this may be not just due to the format, but because the patients were given a choice to complete the COAC pre-consultation form whereas the electronic triage form was seen in some cases as a gateway to their seeing a GP.

We found that fidelity to the intervention was high, but that GPs made adaptations to fit with normal practice. In common with other studies, we found that training and ongoing support⁵⁶ was an essential part of ensuring fidelity. GPs referred to the training session in their interviews, commenting that it helped them appreciate the benefits of the intervention, the relatively short time commitment of the pre-consultation form and gave them the opportunity to practice in advance.

The pre-consultation form proved most useful for patients with complex problems, mental health issues, health concerns, a concern they find difficult to voice, or who find consultations nerve-racking. It was also useful for patients who sometimes feel that the GP doesn’t listen to them or understand their problems. It was less useful when patients who completed it had a quick problem, or when they had underlying chronic problems that were unrelated to their consultation.

We identified six possible outcomes of the pre-consultation form in a new programme theory. The previous programme theory (Figure 3) had included reduction in re-consultation rates as an outcome. However, interviews did not show any evidence that this is a likely outcome. The six outcomes, with possible methods of data collection, are shown in Table 11. The outcomes with most qualitative evidence, likely to be the primary outcomes in a larger study are shown at the top of the table.

6.4 Summary report feasibility

The summary report was highly valued by most of the patients who received it provided the GP completed a sufficient level of detail. It was acceptable to GPs provided that they could choose when to complete it, and most said they would only issue it when they had given important safety netting advice or complex follow-up steps to patients who had memory problems, health anxiety, language problems.

Some GPs found that they adapted the way in which they wrote their consultation notes to fit the new template. As Greenhalgh has pointed out, such adaptations are an essential feature of embedding a new intervention⁵⁷ and GPs who adapted found it easier and more useful than GPs who did not; for example when GPs adapted to complete part of the template during the consultation, this meant they discussed it with patients and it became a mechanism for engaging the patient in their care plan. The task of sending the report at the end of the consultation then took only a minute or two. GPs who carried out their consultation as normal and completed the template at the end found it took longer and did not obtain these additional benefits.

We identified five possible outcomes of the summary form in a new programme theory. These five outcomes, with possible methods of data collection, are shown in Table 12. The outcomes with most qualitative evidence, likely to be the primary outcomes in a larger study, are shown at the top of the table (increased patient knowledge and empowerment and patient and family clear on follow-up).

6.5 Recruitment and follow-up rates

Recruitment rates (recruits per SMS invitations sent) were 26%, higher than the target of 15%. Rates varied among practices. The practice with the lowest recruitment rate was site 2, with 17%. This site had numerous technical difficulties, which resulted in one batch of SMS invitations not having the EMIS number in the message for the patients to input and this will have affected the rate of patients able to complete the pre-consultation form.

Follow up rates were 36% against a target of 80%. We asked patients about this in later interviews. Some patients in the intervention arm did not understand why they had been sent a follow-up form. One of the key reasons for recruitment and retention problems in RCTs is communication⁵⁸. Patients in this study were recruited without any verbal communication prior to recruitment. The short duration of the intervention (over a single consultation) meant there was little time for patients to absorb the fact that they were part of a research study. A systematic review of similarly brief interventions which sought to manipulate patient-perceived levels of clinician empathy reported reasonable follow-up rates for most studies⁵⁹. However, most of these studies captured the follow-up data immediately after the intervention while the patient was present. In our study, patients received the follow-up questionnaire 10 days after the consultation which may have affected their response.

The proportion of patients who agreed to share data from the record was also low. Patients were asked if they were willing to share data at the end of the follow-up questionnaire. Given the low rates of follow-up questionnaire completion, in retrospect this may not have been the best design; requesting for consent for data sharing at end of the baseline questionnaire rather than at the end of the follow-up would have given every recruit the opportunity to consent to this and potentially increased the proportion of patients consenting to data sharing. Many people are happy to share fully anonymised and de-identified health data for research purposes⁶⁰, so making this request at the outset could have substantially increased the proportion of recruits agreeing to data share. However, it could also have led to lower initial recruitment rates.

The patient-reported questionnaires had low levels of missing data. Although follow-up response rates were low, most patients who responded completed the entire questionnaire. Scores were within the range expected for patients in primary care for the PCOQ⁶¹, EQ-5D and CARE measure. The data extracted from the patient record was complete in terms of patient characteristics, apart from ethnicity, which had 8% missing data (this is common in routine datasets)⁶². The consultation record data was comprehensive and we were able to calculate repeat consultations.

The point estimates of intervention and control (43% intervention vs 25% control) were not sufficiently precise to establish if there was differential attrition. Systematic reviews of interventions carried out over an episode of care comprising more than one interaction, with an interactive recruitment process, have tended not to show indication of differential attrition^63,64. In contrast, COAC is a brief intervention, delivered in a single consultation. Intervention arm patients had an opportunity to ask questions in an interview and gained a greater understanding through the form being used in their consultation but control arm patients did not have this opportunity, so might have been expected to have a relatively lower rate of follow-up completion.

There is some evidence of differential attrition in studies with similar designs. For example, Little et al., (2015) carried out a cluster RCT where GPs were trained in empathetic non-verbal communication. Patients were given the follow-up questionnaire immediately after the consultation and encouraged to complete before they left with an option to post it. Follow-up rates were 92% in the intervention arm and 73% in the control arm⁶⁵. Although the authors did not report this as differential attrition, the confidence interval overlap is similarly small to this study and intervention arm patients may have been more engaged in the short duration of the intervention than control arm.

6.6 Spread and adoption

Further design improvements may be required for either the pre-consultation form or closure report to be adopted more widely. The extent to which interventions are taken up depends to a great extent on relative advantage: whether the individuals involved in the interventions perceive that the new intervention is better than what has gone before⁶⁶. Both GPs and patients perceived a relative advantage in the case of both the pre-consultation questionnaire and consultation closure report; but this advantage was higher for patients; GPs saw advantages for their patients rather than direct advantages to themselves as clinicians. Non-adoption and abandonment of technological interventions in healthcare is often high, even when relative advantage is initially perceived. This is often explained by technology (such as usability or occurrence of technical errors)⁵⁷. The relative difficulty of using the pre-consultation form from an administrative perspective suggests abandonment would be likely without automation of the SMS sending and report attaching. Through partner companies, EMIS web allows for an automated message to be sent to patients on appointment booking⁶⁷. This could be investigated further to see if the survey link and EMIS number could be added to the appointment reminder message, negating the need for a daily SMS batch invitation to patients. The attaching of reports was identified as equally important for automation but may be more difficult to implement. Some GP electronic triage systems are not integrated with EMIS, and require an administrator to manually copy and paste, or attach the online consultation to the patient record. General practice allows a 24-hour window to respond to an online consultation, so a practice administrator can upload or copy the online consultations as a single task in a day. The shorter window for the COAC form made this administrative task too difficult. Private companies are evolving their functionality rapidly, with approved Application Programming Interface (API) partner organisations able to interface with the EMIS Web clinical system⁶⁸. Automation of this element of the COAC intervention should, therefore, be possible but it may require substantial input from private providers.

For the consultation summary report, GPs found the process relatively simple, provided there were no technical malfunctions. Use of this form hinges on the decision of each individual GP, and any spread of it would require local champions to persuade their peers that this technology is effective, safe, and professionally appropriate^69,70. Any future roll-out of this should include identification of such champions and thorough testing to ensure all technical problems are resolved before starting recruitment. The main improvement suggested by patients was that they should receive the report through the medium of their choice, which was often email. This was difficult for GPs at the start of the recruitment period, but by August / September 2021, accuRX had added email to their functionality, so email was as easy for GPs as SMS. Some patients who accessed their patient record online also suggested that the summary report should be accessible via Patient Access. There is a need to make the medical record useful and understandable to patients^71,72 while ensuring it contains sufficient technical medical information⁷¹. The COAC summary report may facilitate this shift in recording information in the medical record as it allows a patient-friendly report to be generated and stored alongside more technical notes.

6.7 Future modifications to study design

Based on the findings in this feasibility study, a number of changes to study design are recommended before a larger evaluation or RCT is carried out.

Firstly, COAC is not a single intervention, but rather two different interventions. These may be used together, but they may not both be useful in all circumstances, so they should be offered separately to patients and evaluated separately.

Secondly, the summary report is almost ready to share, but needs to be configured in SNOMED CT⁷³ first, with the user guide updated so that practice staff can make their own customisation to free-text. If the summary report were to be evaluated in its own right, patients would be recruited directly by GPs in the consultation, and the option of traditional postal follow-up patient-reported questionnaires would be required, as the group of patients who benefit most from this may not be highly digitally literate.

Thirdly, the pre-consultation form requires substantial technical modifications to automate the sending of SMS invitations and integrate the pre-consultation form with the patient record. To maximise recruitment rates in a future RCT, communications should be sent out before commencing recruitment to inform patients about the intervention. Patients should be asked to consent to sharing their phone number and data from the record at the start of the pre-consultation questionnaire as a pre-requisite to entering the study. This should substantially reduce attrition, especially if patients are followed-up on this phone number by researchers. As identified in Table 11, two possible outcomes of the pre-consultation form could be assessed using data extracted from the patient record (“issues discussed that might not have been otherwise” and “wider range of tailored support offered to patients”). Using patient record data negates the need for long follow-up questionnaires and removes the problem of attrition entirely. Customised code lists should be developed in SNOMED CT to measure these outcomes. However, given we were unable to demonstrate that GPs code sufficiently consistently for these code lists to work, patient-reported outcomes should be captured as well.

Finally, although a cluster RCT is still a potentially appropriate design for each study, any future RCT should commence with an internal pilot to ensure that the changes recommended above are feasible, with stop-go criteria set based on acceptability, recruitment and retention.

Underlying data

The underlying data is stored in three repositories: one for the quantitative data, one for the qualitative data and one for the extended data as follows:

Qualitative data (restricted)

Researchers can apply for this data via a form on the repository:

https://doi.org/10.5523/bris.1ljvagu1sigje2duqj3ube527y (restricted access)⁷⁴.

This project contains the qualitative data transcripts for the COAC feasibility study, where participants agreed that these could be shared with bona fide researchers outside the Bristol research team. Information about each transcript is listed below, as follows:

Transcript ID: The name of the transcript in the folder. The name consists of:

* a participant identifier

* the type of participant (patient, clinician or administrator)

* the site (1 to 4 – this was not reported in the paper for reasons of anonymity)

* The date of the interview

Participant identifier used in papers: This is the identifier used in this paper.

The folder also contains the consent form. All patients in this study consented to point 7 in this form: "I understand that after the study my anonymised data will be made available to bona fide researchers for future research studies, and it will not be possible to identify me from these data. If I agree to this, my data will be held for twenty years."

This dataset has an access level Restricted, which means it is not available via direct download but must be requested. Research participants did not give explicit consent to share this data as open data but agreed that it should be made available to approved bona fide researchers only, after their host institution has signed a Data Access Agreement. In order to request access to this data please complete the data request form available from the link above. We will consider any application from any organisation where an established research governance process is in place.

Data are available under a Non-Commercial Government Licence for public sector information.

Quantitative data (restricted)

Researchers can apply for this data via a form on the repository:

https://doi.org/10.5523/bris.3tncqi96uvlkg2451y8qfpjlfx⁷⁵.

The data contains the patient-reported outcomes data and the data extracted from the patient record for those patients who consented to data sharing beyond the research team. It also contains the CONSORT checklist for the study⁷⁶.

This dataset has an access level Restricted, which means it is not available via direct download but must be requested. Research participants did not give explicit consent to share this data as open data but agreed that it should be made available to approved bona fide researchers only, after their host institution has signed a Data Access Agreement. In order to request access to this data please complete the data request form available from the link above. We will consider any application from any organisation where an established research governance process is in place.

Data are available under a Non-Commercial Government Licence for public sector information.

Extended data

University of Bristol: COAC Study Extended Dataset, https://doi.org/10.5523/bris.386dsq2e4iii225ms7du8pd5jq⁷⁷.

This project contains the following extended data:

1. COAC-pre-consultationForm.doc

This file contains screenshots of the pre-consultation form which patients responded to in the COAC Study.

2. COACStudy-pre-consultationform-TableOfChanges.doc

This file contains a detailed table of changes made to the pre-consultation form in the COAC Intervention Study. Patients who are quoted in this table all consented to the first six points in the consent form included in this folder.

3. COACStudy-SummaryReport-TableOfChanges.doc

This file contains a detailed table of changes made to the summary report in the COAC Intervention Study. Patients who are quoted in this table all consented to the first six points in the consent form included in this folder.

4. COACStudy-TopicGuides.doc

This file contains the interview topics guides for the COAC Study.

5. PatientConsent-Interviewsv1.3.doc

This is the patient consent form used for the COAC Study

6. PatientInfoInterviewStudy2v1.4.doc

This is the patient information leaflet given to patients interviewed for the COAC Study

7. COREQ checklist - pre-consultation form

This is a checklist for the COREQ reporting guidelines which demonstrates how they were following in collecting and analysing data about the pre-consultation form

8. COREQ checklist – summary report

This is a checklist for the COREQ reporting guidelines which demonstrates how they were following in collecting and analysing data about the summary report

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).