Interventions for improving the design and conduct of scientific research: A scoping review protocol

Background Research waste is prevalent in many scientific fields despite a number of initiatives to improve research practices. Interventions to improve practice are often implemented without evaluating their effectiveness. It is therefore important to identify the interventions that have been evaluated, assess how they have been evaluated and to identify areas where further research is required. Objectives A scoping review will be undertaken to assess what interventions, aimed at researchers or research teams, to improve research design and conduct have been evaluated. This review will also consider when in the research pathway these interventions are implemented; what aspects of research design or conduct are being targeted; and who is implementing these interventions. Methods Interventions which aim to improve the design or conduct of research will be eligible for inclusion. The review will not include interventions aimed at hypothetical research projects or interventions implemented without evaluation. The following sources will be searched: MEDLINE, EMBASE, ERIC, HMIC, EconLit, Social Policy and Practice, ProQuest theses, and MetaArXiv. Hand searching of references and citations of included studies will also be undertaken. Searches will be limited to articles published in the last 10 years. Data extraction will be completed using a data extraction template developed for this review. Results will be tabulated by type of intervention, research stage, and outcome. A narrative review will also be provided addressing each of the objectives.

The following sources will be searched: MEDLINE, EMBASE, ERIC, HMIC, EconLit, Social Policy and Practice, ProQuest theses, and MetaArXiv.Hand searching of references and citations of included studies will also be undertaken.Searches will be limited to articles published in the last 10 years.Data extraction will be completed using a data extraction template developed for this review.
Results will be tabulated by type of intervention, research stage, and outcome.A narrative review will also be provided addressing each of the objectives.

Plain language summary
It has been shown that due to the poor way in which some research is done the results either cannot be trusted or are not useful.Examples of this include: not publishing research that is completed so the results cannot be used by others; using outdated or unreliable research methods; measuring outcomes that are not useful in the real world or not important to patients; having too small a sample of participants in a study to get a reliable answer to the research question.This means that the money that has been spent on the research and the time that any participants have given to that research has been wasted.
There have been a number of initiatives to try and stop this happening.It is important to evaluate these to test whether they actually work.Some initiatives that have been evaluated include: reminders to publish research once it is complete, providing researchers with tools for better research design and public recognition for good research practices.
We intend to search for and summarise all the initiatives that have already been tested.To do this we will do a comprehensive search for all studies that have tested these initiatives.By doing this we will be able to see what further initiatives could be developed and how we can better test them.

Plain language summary
It has been shown that due to the poor way in which some research is done the results either cannot be trusted or are not useful.Examples of this include: not publishing research that is completed so the results cannot be used by others; using outdated or unreliable research methods; measuring outcomes that are not useful in the real world or not important to patients; having too small a sample of participants in a study to get a reliable answer to the research question.This means that the money that has been spent on the research and the time that any participants have given to that research has been wasted.
There have been a number of initiatives to try and stop this happening.It is important to evaluate these to test whether they actually work.Some initiatives that have been evaluated include: reminders to publish research once it is complete, providing researchers with tools for better research design and public recognition for good research practices.

Introduction
Estimates have suggested that as much as 85% of research is waste 1 .With over $100 billion spent annually on medical research this amount of wastage is extremely costly.Glaziou and Chalmers 1 identified four overarching stages in the research process where waste can occur: question and priority setting; appropriate design and methods; availability of results; unbiased and usability of results.
Solutions to all these aspects of research waste have been suggested including the development of guidelines, policies, laws, incentives, and accepted standards 2,3 .However, research waste persists despite the development of better standards and practice, with the coronavirus disease 2019 (COVID-19) pandemic exacerbating poor practices due to the speed at which research was required 3,4 .
Hardwicke et al. 5 proposed a framework for improving research practice that has four stages: 1) identify problems; 2) investigate problems; 3) develop solutions; 4) evaluate solutions.They note that many solutions are developed without any consideration for evaluation and that unintended consequences or lack of anticipated benefits may go undetected.They suggest that as a result of this many existing evaluations of these interventions are retrospective observational studies.Previous reviews in this area have looked at individual interventions or individual problems, such as interventions to improve the reporting of research 6 , incentives for data sharing 7 , and interventions for increasing the publication of research 8 .
There has however been no overarching assessment of interventions to improve the design and conduct of research.Therefore, the aim of this scoping review is to map the evaluations of interventions which aim to improve the design or conduct of research.We will use this review to identify knowledge gaps and to assess how this type of research could be improved.Some examples of interventions that we were aware of prior to starting this research include: awarding badges for data sharing 9 ; legal requirements for publication 10 ; funder recommendations for design choices 11 ; email reminders for increasing publication 12 ; providing training and tools to researchers to allow them to design better studies 13 .

Objectives
The aims of this scoping review are to address the below questions.

Registration
The methodology of this scoping review has been registered with the Open Science Framework registry 14 .

Eligibility criteria
This review will include any evaluations of interventions that aim to improve the design or conduct of scientific research by targeting researchers or research teams.

Population.
Researchers or research teams undertaking or developing research projects.This will not include studies where the intervention is aimed at those participating in research (such as Studies Within A Trial).

Concepts.
For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be

Amendments from Version 1
Clarification has been made regarding the eligibility criteria in response to a reviewer.Specifically regarding the types of study design included and the definitions of interventions and evaluations.
This has been amended from a registered report protocol to a standard protocol to allow the review to be progressed in a timely manner, doing this has not changed the planned scoping review methods.
Any further responses from the reviewers can be found at the end of the article any coordinated sets of activities designed to change specified behaviour patterns 15 .This could include policies, laws, and guidelines provided there has been a reported evaluation of the effectiveness of the intervention.
Interventions aimed at any aspect of design and conduct will be included from initial design (e.g.question setting, protocol development, etc.), through undertaking the research (e.g.registration, changes to research design, etc.), and publishing (e.g.timely publication, reporting standards, data sharing, etc.).
Context.This review will include any interventions aimed at scientific research.This review will not include interventions designed to improve hypothetical research projects, such as those that a research student may develop for a research methods course.
There will be no other limitation on the context that the interventions are applied in.
Types of Studies.This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated.No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions.

Information sources and searching strategy
Literature searches of the following electronic databases will be made: MEDLINE, EMBASE, EconLit, ERIC, and Social Policy and Practice.The following grey literature sources will be searched: HMIC, ProQuest Dissertations and Theses Global and all articles submitted to the MetaArXiv preprint server.
Searches were developed with previously identified studies using the Yale Mesh Analyser and the PubReminer tools and guidance from an information specialist.The full search strategy is available as extended data 14 .Hand searching will be undertaken of references within and citations to included articles to identify any further relevant articles.
To aid the identification of relevant research we will contact authors, researchers and other stakeholders (funders, publishers, research organisations, etc.).
No limits will be applied relating to language or source of information.Where translation is required, Google Translate will be used.The searches will be limited to the last 10 years.

Selection of sources of evidence
All records identified will be de-duplicated using Endnote v20 then imported into Covidence Systematic Review software.Free alternatives that perform similar functions are also available.
A two-stage screening process will then be used, first screening the titles and abstracts of all identified studies and then screening the full-texts of any that appear eligible at the first stage.Both stages will be completed independently by two researchers with any disagreements resolved by discussion.This process will be piloted initially using a random selection of 25 items and will be continued until 75% or higher agreement is achieved.

Data charting process
Data extraction will be completed using a data extraction template.This process will be done by one researcher with a second independently checking with any disagreement being resolved through discussion.This process will be piloted on a small number of initial studies to ensure consistency and that all pertinent data is being captured in the data extraction template.
Data items.The below items of data are the initial items to be extracted, any further items identified during the extraction process will be reported.
-Author(s) or Organisation(s) As well as this a type of intervention will be assigned according to the intervention function categories in the behaviour change wheel framework as below 15 . -

Analysis and synthesis of results
Results of the searching process will be presented in a flowchart as per the PRISMA-SCR guidance 16 .
Extracted data will be presented in tables, the following items will be tabulated and this will be done separately for aspects of design and conduct.
-Type of Intervention as per behaviour change wheel

-Stage of research pathway -Outcome
Any further tables or visualisations of the data that will be informative for the objectives of this review will be considered at the time of analysis.A narrative review will be presented alongside this addressing each of the review objectives which will summarise the key findings and any gaps in the research.Whilst no formal assessment of research quality will be made in this review the research methodology used to evaluate these interventions will be presented and discussed.

Discussion
This scoping review will summarise the interventions targeted at researchers and research teams that have been evaluated for improving the design and conduct of scientific research.We will assess the characteristics and the expected outcomes of these interventions and look at when in the research pathway these are utilised.We then intend to use the results of this review to identify any gaps in this research and to inform the development and testing of an intervention.

Study status
Completion is anticipated to be by October 2022.

Dissemination
Following completion of the synthesis, the results of this project will be submitted for publication as a peer reviewed journal article.Data related to the results will be made available.

Ethics
As this research is a literature-based review no ethical approval is required.

Olalekan Lee Aiyegbusi
Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK This is a very relevant study which could potentially assist with the minimisation of research waste.Well written protocol.
Comments Page 2 of methods: "A narrative review will also be provided…" this is a bit confusing.Do you mean 'narrative synthesis'?The phrase 'narrative review' was used again in the final paragraph of the section on Analysis and synthesis of results (page 5).A narrative review is a different type of review -so please check and revise.
○ Page 4: I understand that the focus of the review is on evaluated interventions.However, it would be helpful if this review captures what is out there whether they are evaluated or not.Such information itself is important.These unevaluated interventions could just be mentioned in a Table without having to provide extra details.
○ Is the rationale for, and objectives of, the study clearly described?Yes

Are sufficient details of the methods provided to allow replication by others? Yes
Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: No competing interests were disclosed.
Is the rationale for, and objectives of, the study clearly described?Not applicable

Are sufficient details of the methods provided to allow replication by others? Not applicable
Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: My salary is supported by the Laura and John Arnold Foundation now called Arnold Ventures.
Reviewer Expertise: Meta-research (with a focus on incentives, reproducibility, reporting) I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Version 1
Reviewer Report 01 March 2022 https://doi.org/10.3310/nihropenres.14368.r28468 My only feedback and why I checked 'partly' for questions 1 and 4 above is that they have not defined what they mean by 'interventions' and by 'evaluated'.Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria).For instance, will they be including observational studies of interventions?Or will they limit their search to randomized controlled trials of interventions?Further, what does it mean for an intervention to be evaluated?Do they mean 'tested using the gold-standard'?There are plenty of interventions to improve research conduct but there are few randomized controlled trials.As such, more clarification/definitions will be very helpful.On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?
Have the authors pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks?Partly Is the rationale for, and objectives of, the study clearly described?Yes

Are sufficient details of the methods provided to allow replication by others? Partly
Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: The Laura and John Arnold Foundation funds the RIAT Support Center, which supports my salary.
Reviewer Expertise: Meta-research (with a focus on incentives, reproducibility, reporting) I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 08 Jun 2022

Andrew Mott
Thank you for reviewing our article, we hope that the our responses outlined below address your points.Further clarification has been provided regarding what is meant by an intervention in the "Concepts" section of the Eligibility Criteria and further on evaluated in the types of studies section.
"For this review, any intervention will be considered, for the purposes of this review an intervention will be defined as per the behaviour change wheel framework to be any coordinated sets of activities designed to change specified behaviour patterns 15.

Therefore, it would be good if they could clarify what study types will fit the inclusion criteria (and the exclusion criteria). For instance, will they be including observational studies of interventions?
Or will they limit their search to randomized controlled trials of interventions?Further, what does it mean for an intervention to be evaluated?Do they mean 'tested using the gold-standard'?There are plenty of interventions to improve research conduct but there are few randomized controlled trials.As such, more clarification/definitions will be very helpful.
We agree that there will be few trials and anticipate that there many of the studies will be retrospective observational designs.We therefore intend to include any quantitative or qualitative assessments of these interventions and to summarise the methods used to evaluate these interventions to date.To clarify this, additional text has been added to the "types of studies" section of the Eligibility Criteria as below.
"This review will include any quantitative or qualitative evaluation of interventions but will not include interventions that have not been evaluated.No limitations will be made by study type and will include observational, experimental/quasi-experimental, and qualitative designs as this review intends to summarise the methods used to evaluate these interventions." On a related note, is there a reason Scopus, Web of Science, Google Scholar and medRxiv were not included in the list of databases that will be searched?
We developed the searches with an information specialist and tested our key search terms on a number of databases.We therefore feel that our current strategy has the most suitable balance of coverage of the literature without the need to search too many databases.I provide our reasoning for each of these sources below.
Both MedRXIV and BioRXIV are now searchable through EMBASE which we intend to search so we should capture any appropriate references from these sources within our current strategy.Further details provided here: https://library.medschl.cam.ac.uk/blog/2021/11/medrxiv-and-biorxiv-now-searchablethrough-embase/ Whilst Web of Science is only a platform to search databases we did consider searching the Science Citation Index Expanded database within WOS and SCOPUS but found that many of the results these databases provided for key search terms did not provide a suitable balance of coverage to the number of search results, given the results from other databases chosen.
We do not feel that Google Scholar would be appropriate for this review.Google scholar is subject to personalisation bias and has no formal method for providing all results of a search or repeating a search in a reproducible manner.Some further details are provided here: https://dx.doi.org/10.5210/ojphi.v5i2.4623 Education -Knowledge of practices -Persuasion -Communication for improving actions -Incentivisation -Expectation of reward -Coercion -Expectation of punishment -Training -Imparting skills -Restriction -Rules or regulation -Environmental restructuring -Changing physical or social context -Modelling -Showing example of good practice -Enablement -Reducing barriers/increasing means to practice -Other -Anything not captured by framework