Protocol for a natural experiment study of a physical activity programme aimed at children and young people living in an English city (Wolverhampton)

Study design

The study was designed as an eight-year natural experiment using a controlled before-and-after design. The pre-intervention period consists of 6.5 years (September 2017 to December 2023; 75 months), providing baseline data for the trend analysis. The post-intervention period will span 18 months (January 2024 to June 2025), during which the effects of the intervention will be assessed.

Intervention site

The intervention site is Wolverhampton City, a metropolitan city with a total population of 263,700 residents located in the West Midlands region in England. Wolverhampton has a fast-growing population and is ethnically diverse.²⁰ Twenty five percent of the resident population are aged less than 19 years old,²¹ resulting in an eligible population of an estimated 67,404 CYP who have the potential to be impacted by the intervention.

Control sites

The control sites are three LAs with similar population size and demographics, and to the best of our knowledge, are not providing similar programmes aimed at removing the cost of PA activities for CYP. These control sites are also located within the West Midlands region in England and include Dudley, Sandwell and Walsall, which together with Wolverhampton form the Black Country region. Specifically, these comparator sites have a total population of 341,900 (Sandwell), 323,496 (Dudley) and 284,100 Walsall residents, and a population of 92,313(27%),²² 77,639 (24%),²³ and 71,025 (25%)²⁴ CYP, respectively.

Patient and public involvement

Patient and public involvement (PPI) has been embedded throughout the development of this study. The research is led by the Centre for Economics of Obesity, which is supported by an established Public Advisory Group (PAG) comprising members of the community. The PAG meets several times a year and provides advice across the Centre’s wider research programme, including this project. Early engagement with the PAG helped the research team to understand the relevance and importance of the research topic from a public perspective.

The research questions were developed and refined through regular face-to-face meetings with the PAG. During these meetings, initial ideas were presented and discussed, and PAG members were invited to share their views based on their lived experiences. Some PAG members have direct experience of obesity, and their insights were particularly valuable in shaping research priorities and ensuring that the questions addressed issues of relevance to affected populations. These discussions also informed the team’s understanding of potential inequalities and how they might be considered within the analysis.

PAG members contributed to the design of the study by providing feedback on the proposed research approach and methods. While the choice of primary and secondary outcome measures was largely determined by the availability of secondary data, the PAG helped the research team to identify and reflect on the limitations of this approach, which will be acknowledged in the interpretation and reporting of results. As the study uses secondary data, the PAG was not involved in participant recruitment.

Patients and the public have also contributed to the development of dissemination plans. The PAG has advised on how findings should be communicated to non-academic audiences and on identifying relevant community groups and stakeholders who may be interested in the results, including in relation to the Wolverhampton study. This input will help ensure that outputs are accessible, meaningful, and relevant beyond academic audiences.

The intervention (Yo!Active)

The wider Yo! programme is designed to provide CYP living in Wolverhampton with the best possible start in life. It is aimed at building confidence, creating opportunities, and giving children the tools they need to succeed. Yo! is a brand and a place that promotes young opportunities across the city and is delivered by both the Council and local providers. Yo! Active is a new addition to the wider Yo! offer and focuses on offering a range of PA opportunities to CYP in Wolverhampton.

Table 1 outlines the range of activities that are included within Yo! Active together with how it compared to what was previously on offer across the city.

Yo! Active programme theory of change

The programme focuses on improving the affordability and accessibility of leisure centres and other community facilities and encouraging CYP to engage in different forms of PA. The Yo! Active programme will address these cost-related barriers to promote PA. The theoretical underpinning of the programme is that CYP engagement in PA can be modified using influences such as financial (subsidies) and social incentives. In this case, the financial incentive is a free-cost offer for sport and PA access that will lower the financial burden for young people and families participating in PA. In consideration of how financial incentives (or subsidies) may affect health behaviour, behavioural economics is increasingly providing insightful evidence for how population-level health programmes can be implemented.^25,26

Using the TIDIeR (Template for Intervention Description and Replication),²⁷ the Yo! Active programme will potentially impact population behaviours as detailed in Table 2.

In addition to providing the right incentives for sports and PA, it is important to implement policies that help increase people’s motivation and abilities to become more PA. The decision to engage in PA does not occur in isolation and is influenced by socio-economic, environmental, psychological and contextual factors. One important challenge of the programme will be, given the wider economic and social context, to ensure the intervention is sustainable over the long term. Due to several factors such as fiscal constraints, incentives may be withdrawn in the future. The question about the sustainability of PA after incentives are removed will need to be considered.

A logic model indicating the pathway for PA behaviour change during the implementation of Yo! Active is presented in Figure 1.

Figure 1. A logic model for assessing PA change behaviours following the introduction of Yo!Active.

The data

Anonymised individual-level data from a nationally representative repeated cross-sectional survey will be used for data on PA levels of CYP in Wolverhampton and the control sites, before and after the introduction of Yo!Active. The survey, the Active Lives Children and Young People Survey (CYP), measures the involvement of CYP in PA and different sports and is conducted annually by Sport England.²⁸ The online survey is conducted among schools in England and depending on the young person’s age, the survey is completed by pupils or pupils with parents/guardians. The survey captures responses among pupils during the academic year from September to June.

The CYP survey dataset is available in two versions:

Within the CYP surveys, moderate and vigorous PA is defined as follows:

Table 3 illustrates how the data from the CYP survey is recorded as PA parameters.

The CYP survey was first conducted by Sport England during the academic year 2017-2018 (September 2017 to July 2018). During the COVID-19 pandemic, the survey was altered slightly to make completion easier at home and at school, and some additional questions were added to reflect PA during the lockdown periods.

As an illustrative example, Figure 2 reports the proportion of less active CYP, i.e., not reaching at least 30 minutes per day of moderate or vigorous PA across the Black Country LAs between 2017 and 2023 using data from CYP Active Lives Survey (before YoActive!).

Figure 2. Percentage of less active CYP, i.e., less than an average of 30 min. of moderate or vigorous PA per day (all Black Country Local Authorities).

Source: CYP Active Lives Survey.

Data characteristics and timelines

To evaluate the impact of Yo!Active, we will use population annual measures of PA starting from the academic year 2017–2018 (five years before implementation). CYP data from the academic years 2017–2018, 2018–2019, 2019–2020, and 2022–2023 will therefore represent the pre-intervention period, while data up to June 2025 will represent the post-intervention period. Data from 2021–2022 will be excluded due to unusually low response levels during the COVID-19 pandemic.

Table 4 outlines the survey timelines, including data collection, data release from Sport England, and the point at which the research team will receive access to the data. As the dataset becomes available to researchers several months after each school year ends, the evaluation will cover the first 18 months of implementation. Outcomes will be compared before and after the intervention to account for temporal trends and isolate its effects.

Primary outcome

The primary outcome will be the percentage of less active children defined as those engaging in less than 30 minutes of PA per day.²⁹ This aligns with the objective of LAs to reduce the proportion of less active children. From a public health perspective, societal gains from converting inactive children into more physically active individuals are higher than just making active children more active. This can include activities such as physical education, active travel, activities taking place after school hours, play and different types of sports. So, the primary analysis will be to understand how Yo! Active is impacting the proportion of CYP not reaching the minimum of 30 minutes of moderate or vigorous PA in a day within Wolverhampton, when compared to the control sites.

Secondary outcomes

Additional PA outcomes will be explored in the analysis such as the propensity to not engage in PA. Other PA measures such as the number of days in a week being PA and the number of PA sessions in the last 28 days will be used as secondary outcomes. Categorical variables using the World Health Organisation (WHO) recommendations of at least 150 minutes of moderate PA, or 75 minutes of vigorous PA, or the combination of both intensities per week, will be used to explore the effects of the intervention on PA.²⁹ Moderate and vigorous physical activities can be combined to estimate total activity levels using a standard equivalence rule, in which minutes of vigorous activity are weighted double those of moderate activity. In other words, one minute of vigorous activity is considered equivalent to two minutes of moderate activity. For example, an individual engaging in 60 minutes of vigorous activity and 30 minutes of moderate activity would accumulate 150 moderate-equivalent minutes of physical activity.²⁹

The evaluation will also consider reported attitudes towards PA as these are fundamental drivers of the propensity and quantity of PA. The CYP survey includes questions about attitudes towards PA, such as finding sports enjoyable, satisfying, and easy. Responses to these attitudinal questions will be analysed using thematic analysis to understand if increasing access to leisure and sports activities affects attitudes towards PA, which in turn influences overall PA levels.

Statistical analysis

The average effect of Yo!Active, on levels of PA in Wolverhampton will be estimated using a synthetic difference-in-differences (DiD) approach. This method aligns with a quasi-experimental study design particularly useful to estimate the impact of exposure to an intervention when observations are recorded in a repeated cross-sectional or panel data.³⁰ We will use this approach to estimate the effect of the Yo! Active programme, by comparing the variation in PA outcomes over time in Wolverhampton (intervention group) in comparison to Sandwell, Dudley and Walsall (control groups). This approach is suitable when randomisation of individuals is not possible, and when data is recorded pre/post-intervention. The synthetic DiD approach overcomes some constraining assumptions in the classic DiD method such as the constant parallel trends assumptions that is required, in the absence of treatment, for the difference in the treatment and control group to be constant over time.^31,32

For the synthetic DiD modelling approach, the dependent variable (main outcome) will be the percentage of CYP not reaching at least 30 minutes of moderate or vigorous PA in a day. Confounding factors will include characteristics of the CYP and household/school (school year, gender, age, ethnicity, disability, family affluence score (FAS), free school meals, urban/rural, measures of deprivation (IMD), and pupil-teacher ratio).

Wellbeing indicators such as life satisfaction, happiness and worthiness will also be included in the model as covariates. These variables are presented in the dataset as mean scores out of ten and are asked in the survey as follows:

Interaction terms will be included to assess if there are differential effects of the Yo! Active according to age, sex and deprivation levels.

To control for variation in PA levels throughout the academic year, if the data allows, we will control for seasonal effects.³³

Statistical power and sample size

Sport England specifies that annual CYP survey data at the LA level can only be used if the total annual sample exceeds 200. Given this, the analysis will include only the “main” survey data conducted with pupils in years 3-11 (age 7-5/16). Data collected from pupils in years 1-2 (aged 5-7) will not be considered since the number of observations are too small.

Table 5 illustrates the number of observations available for the main survey (years 3-11). Data for the academic year 2020-21 will therefore be excluded due to low number of observations (during the COVID pandemic).

Using the PA levels for the four LAs (Wolverhampton, Dudley, Sandwell and Walsall) recorded annually from 2017 to 2022, the following method was applied to determine a sample size.

In total across years 2017 to 2022 (excluding 2020-2021 due to COVID), 7337 physical activity questionnaires were completed on school age children across all local authorities. Furthermore, 2640 (36%) of these completed questionnaires on children were deemed to be “less active”.

After the intervention has been implemented within the Wolverhampton area, it is anticipated that a total of 2752 responses to the questionnaire on children’s activity will be collected over an 18-month period. Accordingly, approximately one quarter of the responses (n = 698) are expected to come from the intervention area (Wolverhampton), with the remaining responses (n = 2,054) coming from the control areas (Dudley, Sandwell and Walsall).

With a power of 80% and alpha set to 0.05, we can expect a minimal difference in detection of 5.79% between the intervention and the control groups, i.e. a minimally important decrease to 30.19% in the intervention group for children being deemed “less active”. This is a conservative estimate as we assume independent observations rather than paired comparisons in this calculation. Additionally, we have 4 years of baseline data that is to be used in the analysis and so providing conservative measures within this calculation.

Sensitivity analysis

We will assess robustness to outcome definition and model choice by (a) using alternate PA thresholds (e.g. <20 min/day, WHO weekly equivalent), and (b) analysing continuous measures (days active/week; sessions in last 28 days). Consistency across these specifications will strengthen inference and divergence will be explored and reported.

In addition, instead of pooling Dudley, Sandwell, and Walsall into a single control group, we will re-estimate effects using each comparator LA separately (Wolverhampton vs. Dudley; Wolverhampton vs. Sandwell; Wolverhampton vs. Walsall). This will test whether results are being driven by one particular comparator and ensure robustness to differences in baseline characteristics or unobserved shocks across areas.