The Safe Assessment Form to Evaluate Risks (‘SAFER’) chart – a clinical practice evaluation study following introduction of electronic risk identification in pregnancies in Scotland

Patient and Public Involvement

There was no Patient and Public involvement in this research study. This paper describes a retrospective, cross-sectional audit of a set of 300 case notes.

Overall design

We carried out a retrospective, observational study over three epochs. We sampled three sets of case notes in November and December of years 2011, 2014, and 2015, corresponding to points prior to paper checklists, after implementation of the paper checklists, and after implementation of the electronic (SAFER) system. Each set of case-notes was examined to determine the rates of adherence to established standards of care – thus we carried out three cross-sectional sets of observations or ‘audits.’

The interventions

Standards

The clinical service decided to implement four sets of standards. They were selected because they had national endorsement (see below). We, therefore, examined these standards. Each standard addressed one of the four clinical areas described below. The standards were implemented verbatim from published checklists with no adaptation to the data item collected or contingent action recommended by the Boolean (if/then) algorithm. Each standard contained two components.

(1) Risk assessment component and (2) contingent action component. The risk-assessment should be carried out on all patients and the ‘contingent action’ should be carried out in those cases where it is indicted on the basis of the risk-assessment – in other words contingent on the case being high risk. Risk assessment was based on a checklist of questions and observation of body mass index (BMI). The system algorithm then issues prompts based on certain data, either singly or in combination. A corollary of this sequence (assessment and action suggestion) is that, while all cases sampled would be eligible for the assessment/screening step, only a subset (those who screen positive) would be eligible for the contingent action step, a point to which we shall return.

The four standards, all based on national guidelines, were:-

Phasing of intervention

The VTE prompt is invoked in three stages in the patient pathway (above). The postnatal VTE prompt was implemented many years prior to the initial observation at the end of 2011. Thus, there was no pre-implementation period for this component of the VTE prompt. Antenatal VTE prompts, however, were implemented soon after 2011 and before the second observation period in 2014. The IUGR prompt was implemented after 2011 (in 2013) and before 2014. However, the BMI and GDM prompts were only implemented in 2015.

The phasing of interventions (prompts) and observation periods (audits) are shown in Figure 1.

Figure 1. Time chart – interventions and observations.

We also provide a summary of the prompt media used across each observation point in Table 1.

Data collection points and missing data

The observations available for analysis were as follows:

1)   Postnatal VTE. Collected at baseline and midline but implemented before the 2011 baseline.

2)   Antenatal VTE. Collected at baseline, midline and endline.

3)   IUGR. Not collected at baseline (the intervention was not implemented until 2013 and, unfortunately, data were missing for the effect of moving from no prompts to paper prompts. Collected at midline and endline.

4)   BMI and GDM. Collected at baseline, midline and endline.

Observations

Audits were carried out in the form of cross-sectional observations at the above three time points. At each observation time point, we sampled 100 sequential, consecutive case notes from the singleton pregnancies with a pre-defined starting date. The case notes included the paper checklists from the earlier epoch and the electronic print-outs from the later epoch. These documents were then scrutinized to observe whether the required data had been collected. The case notes were then examined to see whether any contingent actions prompted by the system had been carried out. Thus, the data collected were based on clear boxes on the paper or electronic checklist, while the contingent actions were extracted from the case notes. For example, in a set of 100 consecutive case notes, we first observed whether the risk factors for fetal risk were collected. Then, if a prompt was issued on the basis of this information, we recorded if that contingent action (in this scenario, case-serial scans) was taken. The denominator for this contingent action would be the number of cases where an action was required according to “opportunity for error” principles¹⁴. We represent this dual quality assessment process in Figure 2 in relation to the IUGR risk factor. The sample size of 100 in each group provides 90% power to identify a 20% difference in compliance with a screening standard across any two time periods. Precision was lower with respect to contingent actions.

Figure 2. Representation of the small-for-gestational-age guideline.

Analysis

To investigate the relationship between the different media used for screening algorithms in maternity care and the prompt completion proportion, we used logistic regression modelling. We used descriptive statistics to summarize the proportions of the prompts completed successfully for each clinical area, split into each time point, where the denominators are the total number of participants at each time point. We summarized two different proportions for each clinical area: the proportion of correctly completed risk assessments, and the proportion of contingent actions completed successfully. All analyses were performed using R 4.4.1.

Two logistic mixed-effects regression models were used to estimate the association between the medium of the prompt and how successfully it was completed. One model was used to examine the proportion of successfully completed risk assessments and the other to examine the proportion of successfully completed contingent actions. We pooled the clinical areas to estimate the mean difference in the proportion of risk assessments and contingent actions completed successfully across patients and clinical areas. The dependent variable was coded as 1 if the relevant part of the prompt was completed successfully and 0 if it was not. This model aims to find a more succinct estimate of whether there is an association between the medium of the prompt and the chance of the prompt being completed successfully. We included the following explanatory variables: type of prompt (the explanatory variable of interest), clinical area (VTE, IUGR, BMI, GDM), and year (2011, 2014, 2015). There is collinearity between the year 2015 and the medium type of a SAFER prompt, as in 2015, SAFER prompts were implemented across all clinical areas. We also included a mixed-effect random variable in one of our models to account for the possibility that each person could have multiple records within each clinical area related to different contingent actions. Using the values in Table 2, we adjusted for the average age and BMI each year. We also tested the collinearity of the variables included in the models (Table 3), where the generalized VIF values were < 3, showing low to moderate collinearity.

Descriptive statistics results

As stated, to standardize, we simplified the questions to the same two questions for each clinical area: “Was the correct risk assessment made?” and “Was the correct contingent action applied when appropriate?”. The question related to risk assessment contained a value for all 100 participants over each of the three time points. For this value to be positive for each participant, the risk assessment must have been correct. If either of these criteria was not met, it was recorded as not having been completed when calculating the proportion. The question relating to contingent actions only contained values where contingent actions were needed, with the proportion being the total number of contingent actions that were completed correctly for each clinical area out of the total number of actions that should have been completed. Note that there are some areas where participants can have multiple treatments, causing the total number of contingent actions to be larger than the number of eligible people. The results for all the four conditions are presented in Table 4.

Antenatal VTE prophylaxis

There was a significant improvement from 2014 to 2015 when there was a change from using a paper checklist to an electronic SAFER prompt. In 2014 34.5% of the risk assessments were completed correctly, compared to 96.5% of SAFER prompts completed correctly in 2015. However, there appeared to be little difference in the proportion of correct contingent actions applied when the SAFER prompt was introduced.

Postnatal VTE prophylaxis

However, the implementation of postnatal VTE preceded the first observation period (Figure 1). There appears to be a ceiling effect with how often postnatal VTE risk assessments are completed correctly, as 93%, 97%, and 98% of the risk assessments were completed correctly in 2011, 2014, and 2015, respectively. The same is the case for treatment of postnatal VTE, where 96.8%, 100%, and 96.5% of the indicated treatments were completed successfully. This contrasts with the pattern for antenatal VTE above, where high compliance was delayed until after its guideline came into operation.

IUGR

There were no guidelines in 2011 regarding IUGR risk assessments nor were data collected at this time point. In 2014 and 2015, 93% and 97% of IUGR risk assessments, respectively, were completed correctly. There is some suggestion of an association between improvement with respect to this prompt type when considering whether the correct contingent actions were applied, where 70/81 (86.4%) contingent actions were applied successfully in 2014 and 81/87 (93.1%) in 2015 when the SAFER prompt was implemented.

BMI

When observing both the proportion of BMI risk assessments completed correctly and the proportion of contingent actions applied correctly, there appears to be an increase from 2011 to 2014, although there are no formal systems (checklists or SAFER electronic systems) used in either year. Contingent action, for example, increased from 31% to 85%. This may suggest that there is a temporal association, net of any intervention, towards improved surveillance over those years. There appears to be no significant difference between 2014 and 2015 once SAFER prompts had been implemented, with differences of 0 and 2 percentage points for assessment and contingent action, respectively.

GDM

Similar to BMI, no checklist intervention was implemented before the SAFER intervention between 2014 and 2015. Again, the proportion of GDM risk assessments that were completed correctly was very high in all of our samples. Although the proportion of correctly completed risk assessments is lower post-SAFER implementation in comparison to 2014 before prompts were implemented (100% (2014) and 99% (2015), the difference in these proportions is too small to support any conclusions. This is another example of the ceiling effect that appears when looking at how often risk assessments are correctly completed. We also observed no improvement with respect to the proportion of contingent actions completed correctly when comparing 2014 to 2015; however, there was a large increase from 2011 to 2014, reinforcing the suggestion that there may be an association between the time in which the risk assessment was used and an improvement in completing the risk assessments successfully, that is, a temporal trend.

Logistic regression results

Risk assessment model

Using the coefficients from Table 5, we can calculate the difference in the absolute probabilities of risk assessments being completed correctly between different media of prompt. We begin by examining the differences between non-formal screening methods and paper checklists. There was an absolute difference in probabilities of 0.078 (7.8%) between risk assessments being completed correctly using non-formal versus paper prompts.

After this, we looked at the difference between the proportion of completed paper checklists and SAFER prompts, of which there was an absolute difference of 0.006 (<1%). This value is too small to be significant given our sample size, and is affected by the ceiling effect stated earlier.

The use of SAFER prompts versus non-formal screening techniques was associated with a higher proportion of correctly completed risk assessments. There is an absolute difference in probabilities of 0.08 (8%) between risk assessments being completed correctly using either non-formal prompts or SAFER prompts.

Contingent action model

The first comparison of contingent actions was made between non-formal screening techniques and paper checklists using the values from Table 6. There was an absolute difference in probabilities of 0.05 (5%) between the probability that a contingent action was completed correctly using paper-checklists versus using non-formal screening techniques.

When comparing the proportion of successfully completed contingent actions between the prompt medium of paper and SAFER prompts, we obtained a log odds estimate of 0.2058, which corresponds to a difference in probabilities of -0.007 (-<1%) between paper and SAFER prompts. This interpretation suggests that there is a slightly smaller chance that a SAFER prompt will have a successfully completed contingent action; however, this value is too small to be significant. As shown in the results, there is a very large increase in the absolute difference between informal screening techniques and the other two prompt media.

After adjusting for confounders in our model, the use of SAFER prompts over non-formal screening techniques was associated with a higher proportion of successfully completed contingent actions. There is an absolute difference in probabilities of 0.10 (10%) when comparing the probability of a contingent action being completed successfully between non-formal screening techniques and SAFER prompts.

One limitation of these models is that because all SAFER prompts were implemented in 2015, we were unable to account for the confounding effect of time; therefore, the odds ratios and absolute differences in probability include any confounding effect of time from 2011 to 2015. Even with this in mind, we are still seeing an increase in the proportion of prompts being completed correctly, which is a positive sign that risk screening in maternity care is improving, even if we cannot isolate the cause of this specifically.

Trends in the data

A clear trend of improvement is seen across epochs: the proportions screened and proportions where the correct action was taken increased progressively from one epoch to the other. Only in the case of postnatal VTE is there no trend; this results from high baseline performance, leaving almost no headroom for further improvement. A checklist for postnatal VTE assessment preceded that of 2011; hence, our first set of observations. Thereafter, an antenatal VTE assessment was implemented, with subsequent improvement in assessment and contingent action. This pattern is certainly consistent with an intervention effect, but other explanations are plausible. One such explanation is the so-called rising tide phenomenon¹⁵ where concern across a social system prompts general improvement and specific interventions. The headroom for improvement from the latter is reduced because of spontaneous improvements across the system.

Limitations and causal inference

The elicitation of data collected by clinical staff was based on clear check boxes. However, the contingent actions had to be ascertained from case notes. While this exercise is not complex, we cannot exclude the possibility that some of the actions were overlooked (leading to underestimation of effects) and we did not have resources to duplicate observations. However, the finding of ceiling effects for some observations suggests that few, if any, of the contingent actions were missed. There is little doubt that real improvement occurred over time; precision is low with respect to contingent actions, but the pattern in the data is highly consistent. The question is whether these changes were causal or resulted from other factors that created a secular trend. The fact that changes in the medium and message did not correspond to the dataset provides some clues. The improvement in adherence with respect to GDM and BMI over both earlier epochs, despite not using paper checklists, provides some evidence for a secular trend. On the other hand, as stated, the VTE data point towards a causal explanation since antenatal and intrapartum performance improved only after the checklists were introduced. The rapidity of change - over only one year - for standards where ceiling effects were not in evidence may provide at least some evidence that the effects were (at least partially) causal. However, the data presented here have limitations in the sense that no data were collected for IUGR in 2011. The 1992 randomized trial of prompts quoted in the introduction found that both paper and computer prompts caused a sharp increase in compliance with quality standards compared to no prompt at all. The results regarding antenatal VTE were consistent with these earlier findings.

Relevance

The decision actions (clinical process) that increase dramatically following the introduction of the checklists and prompts reflect best practice guidelines which are based on evaluation studies linking process to outcome. Thus, we can assume that care was safe as a result – the logic is laid out more fully elsewhere¹⁶. While our study has significant limitations, it is one of only a few studies evaluating prompts in antenatal care. A recent systematic review summarised the world’s literature on decision support in maternity care¹⁷. This literature described evaluation of many different types of decision support, covering decision aids for patient choice, prescribing systems, risk scores and decision prompts (covered here). The decision prompts covered postnatal and intrapartum care as well as single and multiple issues. We found seven studies (all from low- and middle-income countries) that evaluated multiple antenatal prompts, analogous to the study cited here (Table 7). The findings, taken in the round, confirm our findings that antenatal decision support in the form of prompts is associated with improved compliance with criteria of effective antenatal care. The prompt systems replaced pre-existing paper records. A much neater solution is to integrate prompts into an electronic record system. Since our study, electronic maternity records have been widely adopted in the UK. Thus, there is a need to evaluate the performance over the years following our observations, during which electronic records, some of which incorporate prompts, have been introduced. To gain an insight into the causal relationship between the improvement of completion and correctness of risk assessments and the contingent actions that follow, there should be a more in-depth analysis of a larger dataset that is able to account for all the confounding factors.

The introduction of maternity records is a significant implementation that has taken place in the absence of any evaluation of which we are aware. Therefore, it is critical to evaluate these new systems. While the data presented here suggest that incorporating the above prompts in new electronic maternity systems is a good idea, there is no certainty that they were optimally configured for maximal impact. Electronic notes provide the basis for examining different formats and a wider set of prompts than we have examined, including blood test results (e.g., rhesus disease/anemia), medical disorders, medicine prescriptions, and mental health problems. There is a real risk of information overload with modern electronic systems^25,26 and it will be important to study the effect of further formal systems and design systems to suit human users, not the other way around.