Detecting and responding to fake responses to an online infant care survey in the UK

Background

The Baby Sleep Project is a series of studies to support vulnerable families and prevent Sudden Unexpected Death in Infancy (SUDI).¹¹ We launched a national survey on 14th June 2022 to understand infant care practices related to safer sleep. The study formed part of an NIHR Fellowship and was conducted via a respected academic online survey platform. The survey collected data on infant care practices relating to safer sleep advice given nationwide in England. As recommended by our public involvement group, an incentive for completing the survey was offered, where respondents could opt-in to be entered into a prize draw to win one of three £50 vouchers. The survey was intended to be conducted face-to-face but due to restrictions imposed by the COVID-19 pandemic at the time, the survey was distributed online.

Survey distribution

The survey was promoted through printed postcards in children’s centres, health professionals in contact with families, and via two well-known children’s charities on Instagram, Facebook and X (formerly Twitter). Posts were public and shareable, allowing the survey link to be widely distributed.

To reduce the risk of survey fraud, participants were informed that prize winners would be contacted by email and asked to verify some data before receiving vouchers. Although multiple fake entries were possible, the risk was considered low due to the topic’s low controversy and lack of other incentives to interfere. The survey platform reported regular security testing, but lacked CAPTCHA protection, support for preventing multiple entries, or the ability to provide IP addresses.

Target respondents

The survey aimed to recruit parents or carers of babies under 1 year of age who lived in England. Survey completion was anonymous, however respondents could leave their email address or phone number at the end of the survey if they wanted to be contacted for an interview, receive the results of the survey or entered into the prize draw.

Patient and public involvement

A group of 14 parent advisors (The Baby Sleep Project Family Advisors) met online to guide the study team on the project as a whole. They contributed to question design, recruitment approaches, and how to interpret the study findings. Members of the group also provided advice about spotting fake responses, and sense checked our ideas about inconsistencies in answers. Their recommendations shaped how we spotted implausible clothing combinations and discrepancies in co-sleeping answers. They also pointed out that completing a survey between midnight and 6 am should be an amber flag, not a red flag, as many parents are up in the night with infants and may have completed the survey genuinely at this time.

Once the survey closed, the researchers noticed unusual patterns in the data. Some answers didn’t make sense, some text responses were just random characters, and some entries came in within seconds of each other. This raised concerns that automated “bots” or individuals trying to win the prize might have submitted fake responses.

To deal with this, the team developed a five-step process:

In total, about 6% of responses were removed as likely fake. The team found that fake entries tended to give random or highly unusual answers that did not match real-world patterns.

Online survey fraud is increasingly common but rarely discussed. The article recommends that researchers build checks into their study design, use tools like CAPTCHA, and analyse data carefully to spot suspicious patterns.

Strategies for detecting fraudulent responses

We defined survey fraud using Lawlor and colleagues framework,⁴ including unique participant fraud, alias fraud and suspicious submissions. Unique fraud involves one individual submitting multiple responses, either intentionally (e.g., to claim incentives) or unintentionally (e.g., forgetting they had already participated). Alias fraud involves more sophisticated attempts to mask identity. Suspicious submissions refer to any entries potentially falling into either category.

Our identification process was iterative; checking all responses for expected and potentially deviating responses, reviewing the impact of each criteria on the overall dataset. The identification of genuine responses allowed us to test if we were omitting potentially genuine responses. It was important that we retained as much of the genuine data as possible, whilst still ensuring that suspicious submissions were identified.

We were guided by the Reflect, Expect, Analyse, Label (REAL) framework, developed by Lawlor and colleagues (2021) which asks: (1) Based on your recruitment and distribution practices, how might your survey be vulnerable?; (2) What are the patterns you would expect to see in survey data?; (3) How do expected patterns related to patterns in reality?; (4) What level of suspicion is sufficient to exclude data from your survey? Table 1 shows how we adapted this framework for our purposes.

Key ‘red flags’ spotted

Red flags were developed based on literature, expert advice, and technical support from the University of Bristol and the survey platform. Some flags, such as identical nonsense text entries, strongly indicate fraud, while others (e.g., long completion time) may have benign explanations. We also accounted for legitimate use of anonymous email addresses by some parents participating in paid surveys.

Our approach balanced the need to exclude clearly fraudulent responses (core red flags) while identifying combinations of amber flags that could indicate fraud, aiming to preserve genuine data wherever possible.

Step 1 - Verifying genuine responses

Some responses were verified as genuine and excluded from further checks:

Total verified = 37; remaining unverified = 3,372.

Step 2 – Removing duplicate responses

89% (3049/3409) of responses included email addresses, which were used to identify duplicates. Duplicate responses from parents of twins were retained. For others, only the first response was kept. Most duplicates were not considered suspicious, with 100 non-twin duplicate responses removed.

Step 3 – Assessing red and amber flags

We excluded all responses with any core red flags and those with 2 or more amber flags, based on their frequency and association with suspicious patterns.

Core red flags

Three core red flags identified 100 responses within our dataset as suspicious and were removed:

Amber flags

Individually, these flags were not enough for exclusion, but 2+ suggested higher suspicion. We derived them by analysing patterns in the 100 red-flagged responses. Table 2 demonstrates the number of responses highlighted as suspicious with the number of red and amber flags present.

In total, we identified 230/3272 (7.0%) of the survey submissions as suspicious, 101 with any of the core red flags, 129 with two or more amber flags.

Step 4 – Adding back in validated responses

We contacted 223/230 participants flagged as suspicious but with valid emails, offering the chance to confirm eligibility. Of these, 21 (9%) were verified and reinstated. Seven could not be contacted (no email), leaving 209 permanently excluded and a final dataset of 3,100 responses for analysis.

Step 5 – Analysis of included and excluded responses

We compared the ‘kept’ and ‘removed’ datasets to assess the impact of fraudulent submissions. Table 4 shows differences across three groups: responses included in the final analysis, those excluded for having two or more amber flags, and those excluded for any core red flag. As responses became more suspicious, answer distributions became more uniform—suggesting random selection.

For example, among responses excluded for core red flags, 50% reported neonatal unit admission, compared to 10–15% in population estimates and among included responses. Excluded responses also more often listed someone other than the mother as the respondent and reported multiple births, with 46.5% of core red flag responses selecting twin or triplet births. A similar pattern appeared for social work involvement: 61% of red-flagged responses reported this, compared to an expected national rate of ~3%. Differences also emerged in sleep environment questions, with excluded responses more frequently reporting non-supine infant sleep positions. Reports of changes to the infant’s usual sleep routine “last night” were also more common in excluded groups, again suggesting more random or inconsistent answers.

Reasons for engaging in survey fraud

Suspicious submissions in this survey fell into two categories: unique participant fraud and alias fraud. About 3% of responses were legitimate or accidental duplicates, clearly identified by email address and removed in step 2. These were often submitted months apart, likely by individuals who forgot they had already responded.

Most suspicious entries were alias fraud, likely automated or bot-generated, as they were submitted within seconds of each other and took an unusually long time to complete. Notably, 10% of those flagged did not opt into the prize draw, suggesting the incentive was not always a motivation for fraudulent responses.

Advice for other researchers

Survey fraud is rarely discussed in academic research, but deserves more attention. Online surveys already require cautious interpretation due to limited respondent information, and fraudulent responses can make analysis misleading. Those who conduct online surveys may find our processes helpful when considering interrogation of their own datasets.