Interventions for improving the design and conduct of scientific research: A scoping review protocol

Andrew Mott; Catriona McDaid; Catherine Hewitt; Jamie J Kirkham

doi:10.3310/nihropenres.13252.2

Home Browse Interventions for improving the design and conduct of scientific research:...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Study Protocol

Revised

Interventions for improving the design and conduct of scientific research: A scoping review protocol

[version 2; peer review: 2 approved]

Previous Title 'Stage 1 Registered Report. Interventions for improving the design and conduct of scientific research: A scoping review'

Andrew Mott ¹, Catriona McDaid¹, Catherine Hewitt¹, Jamie J Kirkham²

PUBLISHED 14 Jun 2022

Author details Author details

¹ York Trials Unit, University of York, UK, York, North Yorkshire, YO10 5DD, UK
² Centre for Biostatistics, The University of Manchester, Manchester, Manchester, M13 9PL, UK

Andrew Mott
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Catriona McDaid
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Catherine Hewitt
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Jamie J Kirkham
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background

Research waste is prevalent in many scientific fields despite a number of initiatives to improve research practices. Interventions to improve practice are often implemented without evaluating their effectiveness. It is therefore important to identify the interventions that have been evaluated, assess how they have been evaluated and to identify areas where further research is required.

Objectives

A scoping review will be undertaken to assess what interventions, aimed at researchers or research teams, to improve research design and conduct have been evaluated. This review will also consider when in the research pathway these interventions are implemented; what aspects of research design or conduct are being targeted; and who is implementing these interventions.

Methods

Interventions which aim to improve the design or conduct of research will be eligible for inclusion. The review will not include interventions aimed at hypothetical research projects or interventions implemented without evaluation.

The following sources will be searched: MEDLINE, EMBASE, ERIC, HMIC, EconLit, Social Policy and Practice, ProQuest theses, and MetaArXiv. Hand searching of references and citations of included studies will also be undertaken. Searches will be limited to articles published in the last 10 years.

Data extraction will be completed using a data extraction template developed for this review.

Results will be tabulated by type of intervention, research stage, and outcome. A narrative review will also be provided addressing each of the objectives.

Plain Language Summary

It has been shown that due to the poor way in which some research is done the results either cannot be trusted or are not useful. Examples of this include: not publishing research that is completed so the results cannot be used by others; using outdated or unreliable research methods; measuring outcomes that are not useful in the real world or not important to patients; having too small a sample of participants in a study to get a reliable answer to the research question. This means that the money that has been spent on the research and the time that any participants have given to that research has been wasted.

There have been a number of initiatives to try and stop this happening. It is important to evaluate these to test whether they actually work. Some initiatives that have been evaluated include: reminders to publish research once it is complete, providing researchers with tools for better research design and public recognition for good research practices.

We intend to search for and summarise all the initiatives that have already been tested. To do this we will do a comprehensive search for all studies that have tested these initiatives. By doing this we will be able to see what further initiatives could be developed and how we can better test them.

Keywords

Meta-Research, Research-on-Research, Research Design, Research Conduct, Research Waste, Scoping Review

Corresponding author: Andrew Mott

Competing interests: Catriona McDaid is a member of the NIHR HTA and Efficacy and Mechanism Evaluation Editorial Board; NIHR Academy Pre-Doctoral Fellowship Panel; NIHR PGfAR Sub-Committee and ORUK Scientific Advisory Committee. She holds research grants from NIHR. Catherine Hewitt is a member and currently deputy chair of the NIHR HTA commissioning committee. Jamie Kirkham and Andrew Mott have no competing interests to disclose.

Grant information: This project is funded by the National Institute for Health Research (NIHR) under its Pre-Doctoral Fellowship (NIHR301991) to Andrew Mott. The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2022 Mott A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Mott A, McDaid C, Hewitt C and Kirkham JJ. Interventions for improving the design and conduct of scientific research: A scoping review protocol [version 2; peer review: 2 approved]. NIHR Open Res 2022, 2:4 (https://doi.org/10.3310/nihropenres.13252.2) First published: 01 Feb 2022, 2:4 (https://doi.org/10.3310/nihropenres.13252.1) Latest published: 14 Jun 2022, 2:4 (https://doi.org/10.3310/nihropenres.13252.2)

Revised Amendments from Version 1

Clarification has been made regarding the eligibility criteria in response to a reviewer. Specifically regarding the types of study design included and the definitions of interventions and evaluations.
This has been amended from a registered report protocol to a standard protocol to allow the review to be progressed in a timely manner, doing this has not changed the planned scoping review methods.

See the authors' detailed response to the review by Anisa Rowhani-Farid

Introduction

Estimates have suggested that as much as 85% of research is waste¹. With over $100 billion spent annually on medical research this amount of wastage is extremely costly. Glaziou and Chalmers¹ identified four overarching stages in the research process where waste can occur: question and priority setting; appropriate design and methods; availability of results; unbiased and usability of results.

Solutions to all these aspects of research waste have been suggested including the development of guidelines, policies, laws, incentives, and accepted standards^2,3. However, research waste persists despite the development of better standards and practice, with the coronavirus disease 2019 (COVID-19) pandemic exacerbating poor practices due to the speed at which research was required^3,4.

Hardwicke et al.⁵ proposed a framework for improving research practice that has four stages: 1) identify problems; 2) investigate problems; 3) develop solutions; 4) evaluate solutions. They note that many solutions are developed without any consideration for evaluation and that unintended consequences or lack of anticipated benefits may go undetected. They suggest that as a result of this many existing evaluations of these interventions are retrospective observational studies. Previous reviews in this area have looked at individual interventions or individual problems, such as interventions to improve the reporting of research⁶, incentives for data sharing⁷, and interventions for increasing the publication of research⁸.

There has however been no overarching assessment of interventions to improve the design and conduct of research. Therefore, the aim of this scoping review is to map the evaluations of interventions which aim to improve the design or conduct of research. We will use this review to identify knowledge gaps and to assess how this type of research could be improved. Some examples of interventions that we were aware of prior to starting this research include: awarding badges for data sharing⁹; legal requirements for publication¹⁰; funder recommendations for design choices¹¹; email reminders for increasing publication¹²; providing training and tools to researchers to allow them to design better studies¹³.

Objectives

The aims of this scoping review are to address the below questions.

What interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?
- ○ Where in the research pathway are these interventions implemented?
- ○ What specific aspects of research are being targeted?
- ○ Who is implementing these interventions? (funders/ethics committees/research institutions/publishers/etc.)

Protocol

Registration

The methodology of this scoping review has been registered with the Open Science Framework registry¹⁴.

Eligibility criteria

This review will include any evaluations of interventions that aim to improve the design or conduct of scientific research by targeting researchers or research teams.

Population. Researchers or research teams undertaking or developing research projects. This will not include studies where the intervention is aimed at those participating in research (such as Studies Within A Trial).

Concepts. For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns¹⁵. This could include policies, laws, and guidelines provided there has been a reported evaluation of the effectiveness of the intervention.

Interventions aimed at any aspect of design and conduct will be included from initial design (e.g. question setting, protocol development, etc.), through undertaking the research (e.g. registration, changes to research design, etc.), and publishing (e.g. timely publication, reporting standards, data sharing, etc.).

Context. This review will include any interventions aimed at scientific research. This review will not include interventions designed to improve hypothetical research projects, such as those that a research student may develop for a research methods course.

There will be no other limitation on the context that the interventions are applied in.

Types of Studies. This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions.

Information sources and searching strategy

Literature searches of the following electronic databases will be made: MEDLINE, EMBASE, EconLit, ERIC, and Social Policy and Practice. The following grey literature sources will be searched: HMIC, ProQuest Dissertations and Theses Global and all articles submitted to the MetaArXiv preprint server.

Searches were developed with previously identified studies using the Yale Mesh Analyser and the PubReminer tools and guidance from an information specialist. The full search strategy is available as extended data¹⁴. Hand searching will be undertaken of references within and citations to included articles to identify any further relevant articles.

To aid the identification of relevant research we will contact authors, researchers and other stakeholders (funders, publishers, research organisations, etc.).

No limits will be applied relating to language or source of information. Where translation is required, Google Translate will be used. The searches will be limited to the last 10 years.

Selection of sources of evidence

All records identified will be de-duplicated using Endnote v20 then imported into Covidence Systematic Review software. Free alternatives that perform similar functions are also available.

A two-stage screening process will then be used, first screening the titles and abstracts of all identified studies and then screening the full-texts of any that appear eligible at the first stage. Both stages will be completed independently by two researchers with any disagreements resolved by discussion.

This process will be piloted initially using a random selection of 25 items and will be continued until 75% or higher agreement is achieved.

Data charting process

Data extraction will be completed using a data extraction template. This process will be done by one researcher with a second independently checking with any disagreement being resolved through discussion. This process will be piloted on a small number of initial studies to ensure consistency and that all pertinent data is being captured in the data extraction template.

Data items. The below items of data are the initial items to be extracted, any further items identified during the extraction process will be reported.

- Author(s) or Organisation(s)
- Year of publication
- Country
- Population
- Sample size

- Evaluation Methodology
- ○ Evaluating organisation

- Intervention details
- ○ Intervention description & rationale
- ○ Intervention aim
- ○ Duration of intervention
- ○ Who is implementing the intervention
- ○ Where in the research pathway is this implemented

- Comparator (if applicable)

- Outcomes
- ○ Outcomes measured
- ○ Timeframe of outcomes

- Author’s conclusion as verbatim text

As well as this a type of intervention will be assigned according to the intervention function categories in the behaviour change wheel framework as below¹⁵.

- Education - Knowledge of practices
- Persuasion - Communication for improving actions
- Incentivisation - Expectation of reward
- Coercion - Expectation of punishment
- Training - Imparting skills
- Restriction - Rules or regulation
- Environmental restructuring - Changing physical or social context
- Modelling - Showing example of good practice
- Enablement - Reducing barriers/increasing means to practice
- Other - Anything not captured by framework

Analysis and synthesis of results

Results of the searching process will be presented in a flowchart as per the PRISMA-SCR guidance¹⁶.

Extracted data will be presented in tables, the following items will be tabulated and this will be done separately for aspects of design and conduct.

- Type of Intervention as per behaviour change wheel
- Stage of research pathway
- Outcome

Any further tables or visualisations of the data that will be informative for the objectives of this review will be considered at the time of analysis. A narrative review will be presented alongside this addressing each of the review objectives which will summarise the key findings and any gaps in the research. Whilst no formal assessment of research quality will be made in this review the research methodology used to evaluate these interventions will be presented and discussed.

Discussion

This scoping review will summarise the interventions targeted at researchers and research teams that have been evaluated for improving the design and conduct of scientific research. We will assess the characteristics and the expected outcomes of these interventions and look at when in the research pathway these are utilised. We then intend to use the results of this review to identify any gaps in this research and to inform the development and testing of an intervention.

Study status

Completion is anticipated to be by October 2022.

Dissemination

Following completion of the synthesis, the results of this project will be submitted for publication as a peer reviewed journal article. Data related to the results will be made available.

Ethics

As this research is a literature-based review no ethical approval is required.

Data availability

Underlying data

No underlying data are associated with this article.

Extended data

Open Science Framework: Interventions for improving the design and conduct of scientific research: A scoping review. https://doi.org/10.17605/OSF.IO/257XV

This project contains the following files:

- Scoping Review - Full Search Strategy.pdf
- Protocol - PRISMA- SCR checklist.pdf

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors would like to acknowledge Helen Fulbright, PhD, MA, PG Dip LIS, BA (Hons), MCLIP, Information Specialist, for her contribution to the development of the search strategy.

Faculty Opinions recommended

References

1. Chalmers I, Glasziou P: Avoidable waste in the production and reporting of research evidence. Lancet. 2009; 374(9683): 86–9. PubMed Abstract | Publisher Full Text
2. Ioannidis JP, Greenland S, Hlatky MA, et al.: Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014; 383(9912): 166–75. PubMed Abstract | Publisher Full Text | Free Full Text
3. Glasziou P, Chalmers I: Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ. 2018; 363: k4645. Publisher Full Text
4. Van Calster B, Wynants L, Riley RD, et al.: Methodology over metrics: current scientific standards are a disservice to patients and society. J Clin Epidemiol. 2021; 138: 219–226. PubMed Abstract | Publisher Full Text | Free Full Text
5. Hardwicke TE, Serghiou S, Janiaud P, et al.: Calibrating the Scientific Ecosystem Through Meta-Research. Annu Rev Stat Appl. 2020; 7: 11–37. Publisher Full Text
6. Blanco D, Altman D, Moher D, et al.: Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019; 9(5): e026589. PubMed Abstract | Publisher Full Text | Free Full Text
7. Devriendt T, Shabani M, Borry P: Data Sharing in Biomedical Sciences: A Systematic Review of Incentives. Biopreserv Biobank. 2021; 19(3): 219–27. PubMed Abstract | Publisher Full Text
8. Thaler K, Kien C, Nussbaumer B, et al.: Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol. 2015; 68(7): 792–802. PubMed Abstract | Publisher Full Text | Free Full Text
9. Rowhani-Farid A, Aldcroft A, Barnett AG: Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial. R Soc Open Sci. 2020; 7(3): 191818. PubMed Abstract | Publisher Full Text | Free Full Text
10. DeVito NJ, Bacon S, Goldacre B: Compliance with legal requirement to report clinical trial results on ClinicalTrials.gov: a cohort study. Lancet. 2020; 395(10221): 361–9. PubMed Abstract | Publisher Full Text
11. Hughes KL, Kirkham JJ, Clarke M, et al.: Assessing the impact of a research funder’s recommendation to consider core outcome sets. PLoS One. 2019; 14(9): e0222418. PubMed Abstract | Publisher Full Text | Free Full Text
12. Maruani A, Boutron I, Baron G, et al.: Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014; 349: g5579. PubMed Abstract | Publisher Full Text | Free Full Text
13. Weeks KL, Henstridge DC, Salim A, et al.: CORP: Practical tools for improving experimental design and reporting of laboratory studies of cardiovascular physiology and metabolism. Am J Physiol Heart Circ Physiol. 2019; 317(3): H627–39. PubMed Abstract | Publisher Full Text
14. Mott A: Interventions for improving the design and conduct of scientific research: A scoping review. 2022. Publisher Full Text
15. Michie S, van Stralen MM, West R: The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011; 6: 42. PubMed Abstract | Publisher Full Text | Free Full Text
16. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018; 169(7): 467–73. PubMed Abstract | Publisher Full Text

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 01 Feb 2022

Author details Author details

¹ York Trials Unit, University of York, UK, York, North Yorkshire, YO10 5DD, UK
² Centre for Biostatistics, The University of Manchester, Manchester, Manchester, M13 9PL, UK

Andrew Mott
Roles: Conceptualization, Funding Acquisition, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Catriona McDaid
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Catherine Hewitt
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Jamie J Kirkham
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Competing interests

Catriona McDaid is a member of the NIHR HTA and Efficacy and Mechanism Evaluation Editorial Board; NIHR Academy Pre-Doctoral Fellowship Panel; NIHR PGfAR Sub-Committee and ORUK Scientific Advisory Committee. She holds research grants from NIHR. Catherine Hewitt is a member and currently deputy chair of the NIHR HTA commissioning committee. Jamie Kirkham and Andrew Mott have no competing interests to disclose.

Grant information

This project is funded by the National Institute for Health Research (NIHR) under its Pre-Doctoral Fellowship (NIHR301991) to Andrew Mott. The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 14 Jun 2022, 2:4

https://doi.org/10.3310/nihropenres.13252.2

version 1

Published: 01 Feb 2022, 2:4

https://doi.org/10.3310/nihropenres.13252.1

© 2022 Mott A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

VIEWS

801

downloads

Citations

SEE MORE DETAILS

CITE

how to cite this article

Mott A, McDaid C, Hewitt C and Kirkham JJ. Interventions for improving the design and conduct of scientific research: A scoping review protocol [version 2; peer review: 2 approved]. NIHR Open Res 2022, 2:4 (https://doi.org/10.3310/nihropenres.13252.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 14 Jun 2022

Revised

Views

Reviewer Report 27 Feb 2023

Olalekan Lee Aiyegbusi, Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK

Approved

https://doi.org/10.3310/nihropenres.14405.r29021

This is a very relevant study which could potentially assist with the minimisation of research waste. Well written protocol.

Comments

Page 2 of methods: “A narrative review will also be provided…” this is

This is a very relevant study which could potentially assist with the minimisation of research waste. Well written protocol.

Comments

Page 2 of methods: “A narrative review will also be provided…” this is a bit confusing. Do you mean ‘narrative synthesis’? The phrase ‘narrative review’ was used again in the final paragraph of the section on Analysis and synthesis of results (page 5). A narrative review is a different type of review – so please check and revise.
Page 4: I understand that the focus of the review is on evaluated interventions. However, it would be helpful if this review captures what is out there whether they are evaluated or not. Such information itself is important. These unevaluated interventions could just be mentioned in a Table without having to provide extra details.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Evidence synthesis including systematic reviews, scoping and narrative reviews

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 28 Jun 2022

Anisa Rowhani-Farid, Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, MD, USA

Approved

https://doi.org/10.3310/nihropenres.14405.r28683

The authors have addressed the questions that I had raised in my original review. The only suggestion I have is ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 01 Feb 2022

Views

Reviewer Report 01 Mar 2022

Anisa Rowhani-Farid, Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, MD, USA

Approved with Reservations

https://doi.org/10.3310/nihropenres.14368.r28468

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’. Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful. On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

Have the authors pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks?

Partly
Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests: The Laura and John Arnold Foundation funds the RIAT Support Center, which supports my salary.

Reviewer Expertise: Meta-research (with a focus on incentives, reproducibility, reporting)

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 14 Jun 2022

Andrew Mott, York Trials Unit, University of York, UK, York, YO10 5DD, UK

14 Jun 2022

Author Response

Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, ... Continue reading Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’.

Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.

"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful.

We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs. We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date. To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.

"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions."

On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

We developed the searches with an information specialist and tested our key search terms on a number of databases. We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases. I provide our reasoning for each of these sources below.

Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy. Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchable-through-embase/

Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.

We do not feel that Google Scholar would be appropriate for this review. Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner. Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623
Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’.

Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.

"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful.

We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs. We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date. To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.

"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions."

On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

We developed the searches with an information specialist and tested our key search terms on a number of databases. We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases. I provide our reasoning for each of these sources below.

Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy. Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchable-through-embase/

Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.

We do not feel that Google Scholar would be appropriate for this review. Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner. Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623
Competing Interests: NA Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 14 Jun 2022

Andrew Mott, York Trials Unit, University of York, UK, York, YO10 5DD, UK

14 Jun 2022

Author Response

Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, ... Continue reading Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’.

Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.

"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful.

We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs. We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date. To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.

"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions."

On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

We developed the searches with an information specialist and tested our key search terms on a number of databases. We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases. I provide our reasoning for each of these sources below.

Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy. Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchable-through-embase/

Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.

We do not feel that Google Scholar would be appropriate for this review. Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner. Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623
Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’.

Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.

"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful.

We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs. We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date. To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.

"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions."

On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

We developed the searches with an information specialist and tested our key search terms on a number of databases. We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases. I provide our reasoning for each of these sources below.

Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy. Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchable-through-embase/

Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.

We do not feel that Google Scholar would be appropriate for this review. Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner. Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623
Competing Interests: NA Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 01 Feb 2022

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 14 Jun 22	read	read
Version 1 01 Feb 22	read

Anisa Rowhani-Farid, University of Maryland School of Pharmacy, Baltimore, USA
Olalekan Lee Aiyegbusi, University of Birmingham, Birmingham, UK

Comments on this article

All Comments(0)

Add a comment

Back to all reports

Reviewer Report

15 Views

27 Feb 2023 | for Version 2

Olalekan Lee Aiyegbusi, Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK

15 Views Cite this report Responses(0)

Approved

This is a very relevant study which could potentially assist with the minimisation of research waste. Well written protocol.

Comments

Page 2 of methods: “A narrative review will also be provided…” this is a bit confusing. Do you mean ‘narrative synthesis’? The phrase ‘narrative review’ was used again in the final paragraph of the section on Analysis and synthesis of results (page 5). A narrative review is a different type of review – so please check and revise.
Page 4: I understand that the focus of the review is on evaluated interventions. However, it would be helpful if this review captures what is out there whether they are evaluated or not. Such information itself is important. These unevaluated interventions could just be mentioned in a Table without having to provide extra details.

Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Evidence synthesis including systematic reviews, scoping and narrative reviews

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

16 Views

28 Jun 2022 | for Version 2

Anisa Rowhani-Farid, Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, MD, USA

16 Views Cite this report Responses(0)

Approved

The authors have addressed the questions that I had raised in my original review. The only suggestion I have is that it would be helpful if they could indicate that they will also be including pre-prints in their study sample.

Competing Interests

My salary is supported by the Laura and John Arnold Foundation now called Arnold Ventures.

Reviewer Expertise

Meta-research (with a focus on incentives, reproducibility, reporting)

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

38 Views

01 Mar 2022 | for Version 1

Anisa Rowhani-Farid, Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, MD, USA

38 Views Cite this report Responses(1)

Approved With Reservations

Have the authors pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks?

Partly
Is the rationale for, and objectives of, the study clearly described?

Yes
Is the study design appropriate for the research question?

Yes
Are sufficient details of the methods provided to allow replication by others?

Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable

Competing Interests

The Laura and John Arnold Foundation funds the RIAT Support Center, which supports my salary.

Reviewer Expertise

Meta-research (with a focus on incentives, reproducibility, reporting)

Respond to this report

Responses (1)

Author Response

14 Jun 2022

Andrew Mott, York Trials Unit, University of York, UK, York, YO10 5DD, UK

Thank you for reviewing our article, we hope that the our responses outlined below address your points.

This study has an interesting and unique study question – ‘what interventions, aimed at researchers or research teams, to improve the design and conduct of research have been evaluated?’

My only feedback and why I checked ‘partly’ for questions 1 and 4 above is that they have not defined what they mean by ‘interventions’ and by ‘evaluated’.

Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.

"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions? Or will they limit their search to randomized controlled trials of interventions? Further, what does it mean for an intervention to be evaluated? Do they mean ‘tested using the gold-standard’? There are plenty of interventions to improve research conduct but there are few randomized controlled trials. As such, more clarification/definitions will be very helpful.

We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs. We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date. To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.

"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated. No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions."

On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?

We developed the searches with an information specialist and tested our key search terms on a number of databases. We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases. I provide our reasoning for each of these sources below.

Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy. Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchable-through-embase/

Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.

We do not feel that Google Scholar would be appropriate for this review. Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner. Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623

View more View less

Competing Interests

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Chalmers I, Glasziou P: Avoidable waste in the production and reporting of research evidence. Lancet. 2009; 374(9683): 86–9. PubMed Abstract | Publisher Full Text

[2] 2. Ioannidis JP, Greenland S, Hlatky MA, et al.: Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014; 383(9912): 166–75. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Glasziou P, Chalmers I: Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ. 2018; 363: k4645. Publisher Full Text

[4] 4. Van Calster B, Wynants L, Riley RD, et al.: Methodology over metrics: current scientific standards are a disservice to patients and society. J Clin Epidemiol. 2021; 138: 219–226. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Hardwicke TE, Serghiou S, Janiaud P, et al.: Calibrating the Scientific Ecosystem Through Meta-Research. Annu Rev Stat Appl. 2020; 7: 11–37. Publisher Full Text

[6] 6. Blanco D, Altman D, Moher D, et al.: Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019; 9(5): e026589. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Devriendt T, Shabani M, Borry P: Data Sharing in Biomedical Sciences: A Systematic Review of Incentives. Biopreserv Biobank. 2021; 19(3): 219–27. PubMed Abstract | Publisher Full Text

[8] 8. Thaler K, Kien C, Nussbaumer B, et al.: Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol. 2015; 68(7): 792–802. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Rowhani-Farid A, Aldcroft A, Barnett AG: Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial. R Soc Open Sci. 2020; 7(3): 191818. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. DeVito NJ, Bacon S, Goldacre B: Compliance with legal requirement to report clinical trial results on ClinicalTrials.gov: a cohort study. Lancet. 2020; 395(10221): 361–9. PubMed Abstract | Publisher Full Text

[11] 11. Hughes KL, Kirkham JJ, Clarke M, et al.: Assessing the impact of a research funder’s recommendation to consider core outcome sets. PLoS One. 2019; 14(9): e0222418. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Maruani A, Boutron I, Baron G, et al.: Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014; 349: g5579. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Weeks KL, Henstridge DC, Salim A, et al.: CORP: Practical tools for improving experimental design and reporting of laboratory studies of cardiovascular physiology and metabolism. Am J Physiol Heart Circ Physiol. 2019; 317(3): H627–39. PubMed Abstract | Publisher Full Text

[14] 14. Mott A: Interventions for improving the design and conduct of scientific research: A scoping review. 2022. Publisher Full Text

[15] 15. Michie S, van Stralen MM, West R: The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011; 6: 42. PubMed Abstract | Publisher Full Text | Free Full Text

[16] 16. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018; 169(7): 467–73. PubMed Abstract | Publisher Full Text

Interventions for improving the design and conduct of scientific research: A scoping review protocol

Abstract

Background

Objectives

Methods

Plain Language Summary

Keywords

Revised Amendments from Version 1

Introduction

Objectives

Protocol

Registration

Eligibility criteria

Information sources and searching strategy

Selection of sources of evidence

Data charting process

Analysis and synthesis of results

Discussion

Study status

Dissemination

Ethics

Data availability

Underlying data

Extended data

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Are you an NIHR-funded researcher?

Thank you!

Interventions for improving the design and conduct of scientific research: A scoping review protocol

Abstract

Background

Objectives

Methods

Plain Language Summary

Keywords

Revised Amendments from Version 1

Introduction

Objectives

Protocol

Registration

Eligibility criteria

Information sources and searching strategy

Selection of sources of evidence

Data charting process

Analysis and synthesis of results

Discussion

Study status

Dissemination

Ethics

Data availability

Underlying data

Extended data

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Competing Interests Policy

Stay Updated

Are you an NIHR-funded researcher?

Thank you!