Search

Objectives: Evidence-based research (EBR) is the systematic and transparent use of prior research to inform a new study so that it answers questions that matter in a valid, efficient, and accessible manner. This study surveyed experts about existing (e.g., citation analysis) and new methods for monitoring EBR and collected ideas about implementing these methods., Study Design and Setting: We conducted a cross-sectional study via an online survey between November 2022 and March 2023. Participants were experts from the fields of evidence synthesis and research methodology in health research. Open-ended questions were coded by recurring themes; descriptive statistics were used for quantitative questions., Results: Twenty-eight expert participants suggested that citation analysis should be supplemented with content evaluation (not just what is cited but also in which context), content expert involvement, and assessment of the quality of cited systematic reviews. They also suggested that citation analysis could be facilitated with automation tools. They emphasized that EBR monitoring should be conducted by ethics committees and funding bodies before the research starts. Challenges identified for EBR implementation monitoring were resource constraints and clarity on responsibility for EBR monitoring., Conclusion: Ideas proposed in this study for monitoring the implementation of EBR can be used to refine methods and define responsibility but should be further explored in terms of feasibility and acceptability. Different methods may be needed to determine if the use of EBR is improving over time., Competing Interests: Declaration of competing interest The authors have no competing interests to declare., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: A measurement tool to assess systematic reviews 2 (AMSTAR 2) was originally developed for systematic reviews (SRs) of health-care interventions. The aim of this study was to assess the applicability of AMSTAR 2 to SRs of non-intervention studies., Study Design and Setting: This was a meta-research study. We used 20 SRs for each of the following four types of SRs: Diagnostic Test Accuracy reviews, Etiology and/or Risk reviews, Prevalence and/or Incidence reviews, and Prognostic reviews (80 in total). Three authors applied AMSTAR 2 independently to each included SRs. Then, the authors assessed the applicability of each item to that SR type and any SR type., Results: Researchers unanimously indicated that 7 of 16 AMSTAR 2 items were applicable for all four specific SR types and any SR type (items 2, 5, 6, 7, 10, 14 and 16), but 8 of 16 items for any SR type. These items could cover generic SR methods that do not depend on a specific SR type., Conclusion: AMSTAR 2 is only partially applicable for non-intervention SRs. There is a need to adapt/extend AMSTAR 2 for SRs of non-intervention studies. Our study can help to further define generic methodological aspects shared across SR types and methodological expectations for non-intervention SRs., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objective: To evaluate the reporting of abstracts of Cochrane Reviews according to PRISMA-A and to investigate a possible association with the abstract´s length., Methods: This is a retrospective, observational study based on all Cochrane Reviews indexed in Medline (via PubMed) until November 18, 2022. In the second part, a random sample of 440 abstracts was drawn, in which PRISMA-A adherence was assessed by two independent reviewers. Analyses were stratified by the year of publication and the number of words., Results: Overall, the median number of words of the 15,188 included abstracts was 469 (IQR 389-686 words), steadily increasing from 353 words in 2000 to 838 words in 2022, with less than one percent of the abstracts in 2022 having ≤ 300 words (in 2000: 30.7%). Analyses on PRISMA-A adherence in the random sample showed a mean score of 6.1 out of 12 fully reported items. Stratified by year, PRISMA-A adherence increased, with higher word counts in 2000-2010 and 2011-2015, while there was no difference in PRISMA-A adherence by abstract length in 2016-2022., Conclusion: Over the years, abstracts of Cochrane Reviews have become longer, running up to 1000 words. This conflicts with the Cochrane Handbook, which recommends a maximum length of 400 until it was aligned with MECIR in 2019, which has recommended a length of <700 words since 2012 but allows up to 1000 words. It is debatable whether such long abstracts meet the key goals of abstracts of being informative, accurate, appealing, and concise., (© 2023 The Authors. Journal of Evidence-Based Medicine published by Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.)

Introduction: One of the most conflicting methodological issues when conducting an overview is the overlap of primary studies across systematic reviews (SRs). Overlap in the pooled effect estimates across SRs may lead to overly precise effect estimates in the overview. SRs that focus on exercise-related interventions are often included in overviews aimed at grouping and determining the effectiveness of various interventions for managing specific health conditions. The aim of this systematic methodological review is to describe the strategies used by authors of overviews focusing on exercise-related interventions to manage the overlap of primary studies., Methods and Analysis: A comprehensive search strategy has been developed for different databases and their platforms. The databases to be consulted will be MEDLINE (Ovid), Embase (Ovid), The Cochrane Database of Systematic Reviews (Cochrane Library) and Epistemonikos. Two reviewers will independently screen the records identified through the search strategy and extract the information from the included overviews. The frequency and the type of overlap management strategies of the primary studies included in the SRs will be considered as the main outcome. In addition, the recognition of the lack of use of any overlap management strategy and the congruence between planning and conducting the overview focusing on overlap management strategies will be assessed. A subgroup analysis will be carried out according to the journal impact factor, year of publication and compliance with the Preferred Reporting Items for Overviews of Reviews statement., Ethics and Dissemination: This study will not involve human subjects and therefore does not require ethics committee approval. However, the conduct and reporting of the findings of this review will be conducted in a rigorous, systematic and transparent manner, which relates to research ethics.The findings of this review will be presented at scientific conferences and published as one or more studies in peer-review scientific journals related to rehabilitation or research methods., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2023. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Only rigorously prepared analyses can provide the highest level of evidence to inform decision-making. Several recent systematic reviews (SRs) examined the hypothesis that the early introduction of specific allergenic complementary foods (CFs) to infants may lead to a lower incidence of one or more allergic outcomes. However, the methodological rigour and quality of reporting of SRs in this area has not yet been systematically evaluated., Methods: We comprehensively searched PubMed, Medline (Ovid), and Web of Science Core Collection on 13th January 2022, using a pre-specified and tested search syntax for SRs with RCT evidence on the early introduction of allergenic CFs as a means for allergy prevention in infants and children. We examined the quality and risk of bias (RoB) using AMSTAR-2 and ROBIS tools, examined adherence to the Preferred Reporting Items for SRs and Meta-Analyses (PRISMA), and checked whether certainty of the evidence was assessed., Results: Twelve SRs were included. Application of both tools resulted in similar overall judgements in terms of direction and extent for nine of the 12 SRs. Nine SRs were found to be of critically low to low quality according to AMSTAR-2 and to be at high RoB according to ROBIS. One SR received a moderate quality rating (AMSTAR-2) and high RoB rating (ROBIS). However, for two SRs, judgements between AMSTAR-2 and ROBIS were at stark variance. Only two SRs fully adhered to the PRISMA checklist. Six SRs evaluated the certainty of the body of RCT evidence. Several SRs failed to consider unpublished studies either by an explicit a priori exclusion or by inadequate search strategies., Conclusions: Well-conducted SRs are important for decision-making and informing guideline development, the quality of their methodology should therefore be considered. The methodological rigour and the reporting quality of SRs on the timing of CF for allergy prevention must be improved., Registration: https://osf.io/7cs4b ., (© 2023. The Author(s).)

Introduction: Centralisation defined as the reorganisation of healthcare services into fewer specialised units serving a higher volume of patients is a potential measure for healthcare reforms aiming at reducing costs while improving quality. Research on centralisation of healthcare services is thus essential to inform decision-makers. However, so far studies on centralisation report a variability of outcomes, often neglecting outcomes at the health system level. Therefore, this study aims at developing a core outcome set (COS) for studies on centralisation of hospital procedures, which is intended for use in observational as well as in experimental studies., Methods and Analysis: We propose a five-stage study design: (1) systematic review, (2) focus group, (3) interview studies, (4) online survey, (5) Delphi survey. The study will be conducted from March 2022 to November 2023. First, an initial list of outcomes will be identified through a systematic review on reported outcomes in studies on minimum volume regulations. We will search MEDLINE, EMBASE, CENTRAL, CINHAL, EconLIT, PDQ-Evidence for Informed Health Policymaking, Health Systems Evidence, Open Grey and also trial registries. This will be supplemented with relevant outcomes from published studies on centralisation of hospital procedures. Second, we will conduct a focus group with representatives of patient advocacy groups for which minimum volume regulations are currently in effect in Germany or are likely to come into effect to identify outcomes important to patients. Furthermore, two interview studies, one with representatives of the German medical societies and one with representatives of statutory health insurance funds, as well as an online survey with health services researchers will be conducted. In our analyses of the suggested outcomes, we will largely follow the categorisation scheme developed by the Cochrane EPOC group. Finally, a two-round online Delphi survey with all stakeholder groups using predefined score criteria for consensus will be employed to first prioritise outcomes and then agree on the final COS., Ethics and Dissemination: This study has been approved by the Research Ethics Committee at the Brandenburg Medical School Theodor Fontane (MHB). The final COS will be disseminated to all stakeholders involved and through peer-reviewed publications and conferences., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2023. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Objective: To evaluate reporting of abstracts of systematic reviews according to the preferred reporting items for systematic reviews and meta-analyses for abstracts (PRISMA-A) 2013 checklist., Study Design and Setting: A random sample of 534 systematic reviews on effectiveness indexed in PubMed between 2000 and 2019 was assessed. Adherence of abstracts to PRISMA-A was analysed using descriptive statistics. Results were stratified by number of words, structure, and year of publication., Result: The mean score of fully reported PRISMA-A items was 5.4 of 12, with adherence varying widely between items (0% to 98.8%). Cochrane reviews received higher mean total scores than non-Cochrane reviews (6.3 vs. 5.2). Adherence to PRISMA-A increased linearly with increasing word count. In non-Cochrane reviews, authors of structured abstracts more often adhered to PRISMA-A than those of unstructured abstracts. No improvements in reporting of abstracts were found after the implementation of PRISMA-A in 2013., Conclusion: Adherence to PRISMA-A shows great potential for improvement. Therefore, authors, editors, and reviewers should be made aware of PRISMA-A by referring to it in the journal submission guidelines. As adherence to PRISMA-A increases with the number of words, journals should consider to increase the word limit to 250-300 words., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: Scoping reviews and evidence map methodologies are increasingly being used by researchers. The objective of this article is to examine how scoping reviews can reduce research waste., Study Design and Setting: This article summarizes the key issues facing the research community regarding research waste and how scoping reviews can make an important contribution to the reduction of research waste in both primary and secondary research., Results: The problem of research waste is an enduring challenge for global health, leading to a waste of human and financial resources and producing research outputs that do not provide answers to the most pressing research questions. Research waste occurs within primary research but also in secondary research such as evidence syntheses. The focus of scoping reviews on characterizing the nature of existing evidence on a topic and including all types of evidence, potentially reduces research waste in five ways: (1) identifying key research gaps on a topic, (2) determining appropriate outcome measures, (3) mapping existing methodological approaches, (4) developing a consistent understanding of terms and concepts used in existing evidence, and (5) ensuring scoping reviews do not exacerbate the issue of research waste., Conclusion: To ensure that scoping reviews do not themselves end up contributing to research waste, it is important to register the scoping review and to ensure that international reporting standards and methodological guidance are followed., (Crown Copyright © 2022. Published by Elsevier Inc. All rights reserved.)

Objective: Participant recruitment is one of the main challenges in research. It is suggested that including researcher photographs might increase participation rates, but empirical evidence is lacking. This study within a trial (SWAT) aims to assess whether invitation letters including researcher photographs increase the participation rate in the context of a survey on medical second opinions., Methods: Through 25 local register offices in Berlin and Brandenburg (Germany), we identified a random sample of 9990 persons. We randomly assigned our sample to the intervention group (IG) receiving an invitation letter with researcher photographs and control group (CG) receiving an invitation letter without photographs in a 1:1 ratio. Our primary outcome was the participation rate. Furthermore, we compared participants to non-participants' characteristics., Results: Of 9990 invitations, 9797 could be delivered (IG: 4890, CG: 4907). Of these, 1349 (13.8%) participated. There were 682/4890 (13.9%) participants in the IG and 662/4907 (13.5%) in the CG with an odds ratio of 1.030 (95% confidence interval: 0.918-1.156). Additional analyses on non-participant characteristics did not show any differences., Conclusion: We could not find any difference in the participation rates. Our study does not confirm the results of previous studies. The length of our questionnaire may have affected our results., Trial Registration: Queens University Belfast - SWAT Store, SWAT 104., (© 2022. The Author(s).)

Background: Restrictions in systematic reviews (SRs) can lead to bias and may affect conclusions. Therefore, it is important to report whether and which restrictions were used. This study aims to examine the use of restrictions regarding language, publication period, and study type, as well as the transparency of reporting in SRs of effectiveness., Methods: A retrospective observational study was conducted with a random sample of 535 SRs of effectiveness indexed in PubMed between 2000 and 2019. The use of restrictions and their reporting were analysed using descriptive statistics., Results: Of the total 535 SRs included, four out of every ten (41.3%) lacked information on at least one of the three restrictions considered (language, publication period, or study type). Overall, 14.6% of SRs did not provide information on restrictions regarding publication period, 19.1% regarding study type, and 18.3% regarding language. Of all included SRs, language was restricted in 46.4%, and in more than half of the SRs with restricted language (130/248), it was unclear whether the restriction was applied during either the search or the screening process, or both. The restrictions were justified for publication period in 22.2% of the respective SRs (33/149), study type in 6.5% (28/433), and language in 3.2% (8/248). Differences in reporting were found between countries as well as between Cochrane and non-Cochrane reviews., Conclusions: This study suggests that there is a lack of transparency in reporting on restrictions in SRs. Authors as well as editors and reviewers should be encouraged to improve the reporting and justification of restrictions to increase the transparency of SRs., (© 2022. The Author(s).)

Introduction: An increasing growth of systematic reviews (SRs) presents notable challenges for decision-makers seeking to answer clinical questions. In 1997, an algorithm was created by Jadad to assess discordance in results across SRs on the same question. Our study aims to (1) replicate assessments done in a sample of studies using the Jadad algorithm to determine if the same SR would have been chosen, (2) evaluate the Jadad algorithm in terms of utility, efficiency and comprehensiveness, and (3) describe how authors address discordance in results across multiple SRs., Methods and Analysis: We will use a database of 1218 overviews (2000-2020) created from a bibliometric study as the basis of our search for studies assessing discordance (called discordant reviews). This bibliometric study searched MEDLINE (Ovid), Epistemonikos and Cochrane Database of Systematic Reviews for overviews. We will include any study using Jadad (1997) or another method to assess discordance. The first 30 studies screened at the full-text stage by two independent reviewers will be included. We will replicate the authors' Jadad assessments. We will compare our outcomes qualitatively and evaluate the differences between our Jadad assessment of discordance and the authors' assessment., Ethics and Dissemination: No ethics approval was required as no human subjects were involved. In addition to publishing in an open-access journal, we will disseminate evidence summaries through formal and informal conferences, academic websites, and across social media platforms. This is the first study to comprehensively evaluate and replicate Jadad algorithm assessments of discordance across multiple SRs., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Systematic reviews of in-vitro studies, like any other study, can be of heterogeneous quality. The present study aimed to evaluate the methodological quality of systematic reviews of in-vitro dental studies., Methods: We searched for systematic reviews of in-vitro dental studies in PubMed, Web of Science, and Scopus databases published up to January 2022. We assessed the methodological quality of the systematic reviews using a modified "A MeaSurement Tool to Assess systematic Reviews" (AMSTAR-2) instrument. The 16 items, in the form of questions, were answered with yes, no, or py (partial yes). Univariable and multivariable linear regression models were used to examine the association between systematic review characteristics and AMSTAR-2 percent score. Overall confidence in the results of the systematic reviews was rated, based on weaknesses identified in critical and non-critical AMSTAR-2 items., Results: The search retrieved 908 potential documents, and after following the eligibility criteria, 185 systematic reviews were included. The most researched topics were ceramics and dental bonding. The overall rating for the confidence in the results was critically low in 126 (68%) systematic reviews. There was high variability in the response among the AMSTAR-2 items (0% to 75% positively answered). The univariable analyses indicated dental specialty (p = 0.03), number of authors (coef: 1.87, 95% CI: 0.26, 3.47, p = 0.02), and year of publication (coef: 2.64, 95% CI: 1.90, 3.38, p < 0.01) were significantly associated with the AMSTAR-2 percent score. Whereas, in the multivariable analysis only specialty (p = 0.01) and year of publication (coef: 2.60, 95% CI: 1.84, 3.35, p < 0.001) remained significant. Among specialties, endodontics achieved the highest AMSTAR-2 percent score., Conclusions: The methods of systematic reviews of in vitro dental studies were suboptimal. Year of publication and dental specialty were associated with AMSTAR-2 scores. The overall rating of the confidence in the results was low and critically low for most systematic reviews., (© 2022. The Author(s).)

Background: Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews., Methods: We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively., Results: Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%)., Conclusions: Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses., (© 2021. The Author(s).)

This randomized controlled trial (RCT) aimed to test the efficacy of enhanced access to Cochrane Handbook (Handbook) recommendations for judging the 2011 Cochrane risk of bias (RoB) domains for improving the adequacy of RoB judgments. Parallel-group RCT with a 1:1 allocation ratio (N = 2271 per group) was conducted. Eligible participants were corresponding authors of all published Cochrane reviews and protocols. After allocation by a random number generator, participants received 20 scenarios for assessing RoB. The intervention group was shown tables from the Handbook with instructions for assessing 2011 RoB tool together with scenarios they were supposed to assess-enhanced access to the Handbook. The control group was shown only a general link to the Handbook. The primary outcome was the proportion of participants that made an adequate judgment of RoB scenarios for analyzed domains. There were 240 responses out of 2020 delivered e-mail invitations in the intervention and 197/2254 in the control group. Only five participants from the intervention group judged RoB adequately in all the 20 scenarios and no one in the control group. The proportion of participants who adequately assessed all the scenarios within a domain was significantly higher in the intervention than in the control group. The frequency of adequate RoB judgments was 7.1% (95% CI: 5.0-9.3%, p < 0.001) higher in the intervention group (76.2%) than in the control group (69.0%). The enhanced access yields more adequate RoB assessments and could be incorporated in software supporting the RoB tool., (© 2021 John Wiley & Sons Ltd.)

Background: To develop and test an approach to test reproducibility of SRs., Methods: Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review., Results: We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory., Conclusion: Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy., (© 2021. The Author(s).)

Background: Overviews often identify and synthesise a large number of systematic reviews on the same topic, which is likely to lead to overlap (i.e. duplication) in primary studies across the reviews. Using a primary study result multiple times in the same analysis overstates its sample size and number of events, falsely leading to greater precision in the analysis. This paper aims to: (a) describe types of overlapping data that arise from the same primary studies reported across multiple reviews, (b) describe methods to identify and explain overlap of primary study data, and (c) present six case studies illustrating different approaches to manage overlap., Methods: We first updated the search in PubMed for methods from the MOoR framework relating to overlap of primary studies. One author screened the studies titles and abstracts, and any full-text articles retrieved, extracted methods data relating to overlap of primary studies and mapped it to the overlap methods from the MOoR framework. We also describe six case studies as examples of overviews that use specific overlap methods across the steps in the conduct of an overview. For each case study, we discuss potential methodological implications in terms of limitations, efficiency, usability, and resource use., Results: Nine methods studies were found and mapped to the methods identified by the MOoR framework to address overlap. Overlap methods were mapped across four steps in the conduct of an overview - the eligibility criteria step, the data extraction step, the assessment of risk of bias step, and the synthesis step. Our overview case studies used multiple methods to reduce overlap at different steps in the conduct of an overview., Conclusions: Our study underlines that there is currently no standard methodological approach to deal with overlap in primary studies across reviews. The level of complexity when dealing with overlap can vary depending on the yield, trends and patterns of the included literature and the scope of the overview question. Choosing a method might be dependent on the number of included reviews and their primary studies. Gaps in evaluation of methods to address overlap were found and further investigation in this area is needed.

Objectives: We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs)., Study Design and Setting: We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements., Results: We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement., Conclusion: Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Objectives: A measurement tool to assess systematic reviews (SRs) 2 (AMSTAR 2) allows for deriving the overall confidence in an SR. We investigated how authors derived the overall confidence rating and whether different schemes lead to different results., Study Design and Setting: We compared three different schemes (original 7-item scheme, a self-developed 5-item scheme, and the AMSTAR Web site) to derive the overall confidence in AMSTAR 2 using two distinct samples of SRs. Multiple bibliographic databases were searched for articles to analyze how AMSTAR 2 was applied by others., Results: In both samples (n = 60 and n = 58), the Friedman test revealed a significant difference between the schemes (P < 0.001). The Web site scheme was the least strict one, whereas between the 5-item and 7-item scheme, no differences were found in post hoc analyses. We included 53 publications applying AMSTAR 2 identified in our literature search. Only 37 of them (70%) used the original 7-item scheme. Less than half of them (18 of 37) reported how they derived the overall rating., Conclusion: Authors should clearly report how they have derived the overall rating when applying AMSTAR 2. Reporting should allow for reproducing the overall ratings for editors, peer reviewers, and readers., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Introduction: Methodological studies (ie, studies that evaluate the design, conduct, analysis or reporting of other studies in health research) address various facets of health research including, for instance, data collection techniques, differences in approaches to analyses, reporting quality, adherence to guidelines or publication bias. As a result, methodological studies can help to identify knowledge gaps in the methodology of health research and strategies for improvement in research practices. Differences in methodological study names and a lack of reporting guidance contribute to lack of comparability across studies and difficulties in identifying relevant previous methodological studies. This paper outlines the methods we will use to develop an evidence-based tool-the MethodologIcal STudy reportIng Checklist-to harmonise naming conventions and improve the reporting of methodological studies., Methods and Analysis: We will search for methodological studies in the Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, Embase, MEDLINE, Web of Science, check reference lists and contact experts in the field. We will extract and summarise data on the study names, design and reporting features of the included methodological studies. Consensus on study terms and recommended reporting items will be achieved via video conference meetings with a panel of experts including researchers who have published methodological studies., Ethics and Dissemination: The consensus study has been exempt from ethics review by the Hamilton Integrated Research Ethics Board. The results of the review and the reporting guideline will be disseminated in stakeholder meetings, conferences, peer-reviewed publications, in requests to journal editors (to endorse or make the guideline a requirement for authors), and on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network and reporting guideline websites., Registration: We have registered the development of the reporting guideline with the EQUATOR Network and publicly posted this project on the Open Science Framework (www.osf.io/9hgbq)., Competing Interests: Competing interests: ‘Yes, there are competing interests for one or more authors and I have provided a Competing Interests statement in my manuscript and in the box below’., (© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: It is challenging to keep systematic reviews (SR) current and updated. Cochrane designated some of its SRs as "stable," that is, not in need of updating. The issue of stabilizing an SR is an important in research synthesis, because it could help reduce research waste. The aim of this study was to analyze publicly available justifications for stabilizing a Cochrane review, with the ultimate goal of helping to make decisions about whether the update of any SR is warranted., Methods: We analyzed Cochrane reviews labeled as stable in Archie, Cochrane's system for managing the editorial/publishing process. From the "What's new" section of the reviews in the Cochrane Library, we extracted justification for stabilization., Results: We included 545 Cochrane reviews labeled in Archie as stable on October 28, 2019. The most common of the five reasons for stabilization was that "last search did not identify any potentially relevant studies likely to change conclusions" (N = 99; 18%), followed by "research area no longer active" (N = 86; 16%), "review is or will be superseded" (N = 41; 7.5%), "evidence is conclusive" (N=35; 6.4%), and "intervention no longer in general use" (N = 34; 6.2%). For the 269 (49%) Cochrane reviews, we considered that the justification for stabilization was not clearly described, that is, sufficiently informative., Conclusions: Cochrane reviews would benefit from more transparency and consistency in publicly available justifications for stabilizing reviews. Further work in this field will help make decisions about the futility of further research and deciding on enough evidence in the field of research synthesis., (© 2020 John Wiley & Sons Ltd.)

Background: Data extraction forms link systematic reviews with primary research and provide the foundation for appraising, analysing, summarising and interpreting a body of evidence. This makes their development, pilot testing and use a crucial part of the systematic reviews process. Several studies have shown that data extraction errors are frequent in systematic reviews, especially regarding outcome data., Methods: We reviewed guidance on the development and pilot testing of data extraction forms and the data extraction process. We reviewed four types of sources: 1) methodological handbooks of systematic review organisations (SRO); 2) textbooks on conducting systematic reviews; 3) method documents from health technology assessment (HTA) agencies and 4) journal articles. HTA documents were retrieved in February 2019 and database searches conducted in December 2019. One author extracted the recommendations and a second author checked them for accuracy. Results are presented descriptively., Results: Our analysis includes recommendations from 25 documents: 4 SRO handbooks, 11 textbooks, 5 HTA method documents and 5 journal articles. Across these sources the most common recommendations on form development are to use customized or adapted standardised extraction forms (14/25); provide detailed instructions on their use (10/25); ensure clear and consistent coding and response options (9/25); plan in advance which data are needed (9/25); obtain additional data if required (8/25); and link multiple reports of the same study (8/25). The most frequent recommendations on piloting extractions forms are that forms should be piloted on a sample of studies (18/25); and that data extractors should be trained in the use of the forms (7/25). The most frequent recommendations on data extraction are that extraction should be conducted by at least two people (17/25); that independent parallel extraction should be used (11/25); and that procedures to resolve disagreements between data extractors should be in place (14/25)., Conclusions: Overall, our results suggest a lack of comprehensiveness of recommendations. This may be particularly problematic for less experienced reviewers. Limitations of our method are the scoping nature of the review and that we did not analyse internal documents of health technology agencies.

Aim: To analyze descriptors/definitions of meta-epidemiological studies as well as study design of articles that were self-described as meta-epidemiological studies. Methods: We searched MEDLINE and Embase on 6 August 2019. We extracted definitions of meta-epidemiological studies, as well as study designs, statistics and units of analysis that were reported in studies self-labelled as meta-epidemiological studies. Results: We included 175 information sources in the analysis. Definitions of meta-epidemiological studies varied and some studies used the term meta-epidemiological study to describe methodological research-on-research studies. Less than a half of the studies (n = 54; 42.9%) used the two-step meta-epidemiological approach in data analysis. Among studies self-labelled as meta-epidemiological, 9.4% reported registration in PROSPERO and 11% indicated they reported the study in line with PRISMA. Conclusion: Research community would benefit from consensus about definition of meta-epidemiological study.

Objective: The objective of the study was to analyze methods of assessing "overall bias" in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews and sensitivity analyses related to overall risk of bias (RoB)., Study Design and Setting: From Cochrane reviews published within 3 years, from July 2015 to June 2018, we extracted data regarding methods of judging overall bias for a single trial, as well as details regarding methods used in frequency of RoB in sensitivity analyses., Results: Of the 1,452 analyzed Cochrane reviews, 409 mentioned assessment of overall RoB on a study level. In 107 reviews, authors clearly specified key domains that determined the overall RoB, whereas in the remaining reviews, assessment of overall bias was not in line with the Cochrane Handbook. Among 268 Cochrane reviews that had any RoB-related sensitivity analysis, in 56 (21%) reviews, the authors reported a significant change for at least one outcome compared with the initial analysis., Conclusion: Highly heterogeneous approaches to summarizing overall RoB on a study level and using RoB for sensitivity analyses may yield inconsistent and incomparable results across Cochrane reviews., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Background: Data extraction from reports about experimental or observational studies is a crucial methodological step informing evidence syntheses, such as systematic reviews (SRs) and overviews of SRs. Reporting discrepancies were defined as pairs of statements that could not both be true. Authors of SRs and overviews of SRs can encounter reporting discrepancies among multiple sources when extracting data-a manuscript and a conference abstract, and a manuscript and a clinical trial registry. However, these discrepancies can also be found within a single manuscript published in a scientific journal., Objectives: Hereby, we describe examples of internal reporting discrepancies that can be found in a single source, with the aim of raising awareness among authors of SRs and overviews of SRs about such potential methodological issues., Conclusions: Authors of SRs and overviews of SRs should check whether the same information is reported in multiple places within a study and compare that information. Independent data extraction by two reviewers increases the chance of finding discrepancies, if they exist. We provide advice on how to deal with different types of discordances and how to report such discordances when conducting SRs and overviews of SRs., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Results of medical tests are the main source to inform clinical decision making. The main information to assess the usefulness of medical tests for correct discrimination of patients are accuracy measures. For the estimation of test accuracy measures, many different study designs can be used. The study design is related to the clinical question to be answered (diagnosis, prognosis, prediction), determines the accuracy measures that can be calculated and it might have an influence on risk of bias. Therefore, a clear and consistent distinction of the different study designs in systematic reviews on test accuracy studies is very important. In this paper, we propose an algorithm for the classification of study designs of test accuracy, that compare the results of an index test (the test to be evaluated) with the results of a reference test (the test whose results are considered as correct/the gold standard) studies in systematic reviews.

Background: The assessment of multiple systematic reviews (AMSTAR) tool is widely used for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of randomized controlled trials (RCTs). Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what it has been validated., Methods: We searched MEDLINE (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR. Full-text studies were sought for all retrieved hits and screened by one reviewer. A second reviewer verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or by consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals., Results: We identified 247 studies. They included a median of 17 reviews (interquartile range (IQR): 8 to 47) per study. AMSTAR was modified in 23% (57/247) of studies. In most studies, an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes = 1) being the most frequent option (102/200; 51%). More than one third of the authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis, we compared overviews of reviews (n = 154) with the methodological publications (n = 93). The overviews of reviews were much less likely to mention both limitations with respect to study designs (if other studies other than RCTs were included in the reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35)., Conclusions: Authors, peer reviewers, and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ensure to have a methodological expert in their review team.

Background: Systematic reviews (SRs) of randomised controlled trials (RCTs) can provide the best evidence to inform decision-making, but their methodological and reporting quality varies. Tools exist to guide the critical appraisal of quality and risk of bias in SRs, but evaluations of their measurement properties are limited. We will investigate the interrater reliability (IRR), usability, and applicability of A MeaSurement Tool to Assess systematic Reviews (AMSTAR), AMSTAR 2, and Risk Of Bias In Systematic reviews (ROBIS) for SRs in the fields of biomedicine and public health., Methods: An international team of researchers at three collaborating centres will undertake the study. We will use a random sample of 30 SRs of RCTs investigating therapeutic interventions indexed in MEDLINE in February 2014. Two reviewers at each centre will appraise the quality and risk of bias in each SR using AMSTAR, AMSTAR 2, and ROBIS. We will record the time to complete each assessment and for the two reviewers to reach consensus for each SR. We will extract the descriptive characteristics of each SR, the included studies, participants, interventions, and comparators. We will also extract the direction and strength of the results and conclusions for the primary outcome. We will summarise the descriptive characteristics of the SRs using means and standard deviations, or frequencies and proportions. To test for interrater reliability between reviewers and between the consensus agreements of reviewer pairs, we will use Gwet's AC 1 statistic. For comparability to previous evaluations, we will also calculate weighted Cohen's kappa and Fleiss' kappa statistics. To estimate usability, we will calculate the mean time to complete the appraisal and to reach consensus for each tool. To inform applications of the tools, we will test for statistical associations between quality scores and risk of bias judgments, and the results and conclusions of the SRs., Discussion: Appraising the methodological and reporting quality of SRs is necessary to determine the trustworthiness of their conclusions. Which tool may be most reliably applied and how the appraisals should be used is uncertain; the usability of newly developed tools is unknown. This investigation of common (AMSTAR) and newly developed (AMSTAR 2, ROBIS) tools will provide empiric data to inform their application, interpretation, and refinement.

Background: Conducting a health technology assessment (HTA) of public health interventions (PHIs) poses some challenges. PHIs are often complex interventions, which affect the number and degree of interactions of the aspects to be assessed. Randomized controlled trials on PHIs are rare as they are difficult to conduct because of ethical or feasibility issues. The aim of this study is to provide an overview of the methodological characteristics and to compare the applied assessment methods in HTAs on PHIs., Methods: We will systematically search HTA agencies for HTAs on PHIs published between 2012 and 2016. We will identify the HTAs by screening the webpages of members of international HTA organizations. One reviewer will screen the list of HTAs on the webpages of members of international HTA organization, and a second review will double-check the excluded records. For this methodological review, we define a PHI as a population-based intervention on health promotion or for primary prevention of chronic or non-chronic diseases. Only full HTA reports will be included. At maximum, we will include a sample of 100 HTAs. In the case that we identify more than 100 relevant HTAs, we will perform a random selection. We will extract data on effectiveness, safety and economic as well as on social, cultural, ethical and legal aspects in a priori piloted standardized tables. We will not assess the risk of bias as we focus on exploring methodological features. Data extraction will be performed by one reviewer and verified by a second. We will synthesize data using tables and in a structured narrative way., Discussion: Our analysis will provide a comprehensive and current overview of methods applied in HTAs on PHIs. We will discuss approaches that may be promising to overcome the challenges of evaluating PHIs.

Background: Systematic reviews (SRs) in the field of neuropathic pain (NeuP) are increasingly important for decision-making. However, methodological flaws in SRs can reduce the validity of conclusions. Hence, it is important to assess the methodological quality of NeuP SRs critically. Additionally, it remains unclear which assessment tool should be used. We studied the methodological quality of SRs published in the field of NeuP and compared two assessment tools., Methods: We systematically searched 5 electronic databases to identify SRs of randomized controlled trials of interventions for NeuP available up to March 2015. Two independent reviewers assessed the methodological quality of the studies using the Assessment of Multiple Systematic Reviews (AMSTAR) and the revised AMSTAR (R-AMSTAR) tools. The scores were converted to percentiles and ranked into 4 grades to allow comparison between the two checklists. Gwet's AC1 coefficient was used for interrater reliability assessment., Results: The 97 included SRs had a wide range of methodological quality scores (AMSTAR median (IQR): 6 (5-8) vs. R-AMSTAR median (IQR): 30 (26-35)). The overall agreement score between the 2 raters was 0.62 (95% CI 0.39-0.86) for AMSTAR and 0.62 (95% CI 0.53-0.70) for R-AMSTAR. The 31 Cochrane systematic reviews (CSRs) were consistently ranked higher than the 66 non-Cochrane systematic reviews (NCSRs). The analysis of individual domains showed the best compliance in a comprehensive literature search (item 3) on both checklists. The results for the domain that was the least compliant differed: conflict of interest (item 11) was the item most poorly reported on AMSTAR vs. publication bias assessment (item 10) on R-AMSTAR. A high positive correlation between the total AMSTAR and R-AMSTAR scores for all SRs, as well as for CSRs and NCSRs, was observed., Conclusions: The methodological quality of analyzed SRs in the field of NeuP was not optimal, and CSRs had a higher quality than NCSRs. Both AMSTAR and R-AMSTAR tools produced comparable quality ratings. Our results point out to weaknesses in the methodology of existing SRs on interventions for the management NeuP and call for future improvement by better adherence to analyzed quality checklists, either AMSTAR or R-AMSTAR.

Background: Medical devices play an important role in the diagnosis, prevention, treatment and care of diseases. However, compared to pharmaceuticals, there is no rigorous formal regulation for demonstration of benefits and exclusion of harms to patients. The medical device industry argues that the classical evidence hierarchy cannot be applied for medical devices, as randomised clinical trials are impossible to perform. This article aims to identify the barriers for randomised clinical trials on medical devices., Methods: Systematic literature searches without meta-analysis and internal European Clinical Research Infrastructure Network (ECRIN) communications taking place during face-to-face meetings and telephone conferences from 2013 to 2017 within the context of the ECRIN Integrating Activity (ECRIN-IA) project., Results: In addition to the barriers that exist for all trials, we identified three major barriers for randomised clinical trials on medical devices, namely: (1) randomisation, including timing of assessment, acceptability, blinding, choice of the comparator group and considerations on the learning curve; (2) difficulties in determining appropriate outcomes; and (3) the lack of scientific advice, regulations and transparency., Conclusions: The present review offers potential solutions to break down the barriers identified, and argues for applying the randomised clinical trial design when assessing the benefits and harms of medical devices.

Distinguishing cohort studies from case series is difficult.We propose a conceptualization of cohort studies in systematic reviews of comparative studies. The main aim of this conceptualization is to clarify the distinction between cohort studies and case series. We discuss the potential impact of the proposed conceptualization on the body of evidence and workload.All studies with exposure-based sampling gather multiple exposures (with at least two different exposures or levels of exposure) and enable calculation of relative risks that should be considered cohort studies in systematic reviews, including non-randomized studies. The term "enables/can" means that a predefined analytic comparison is not a prerequisite (i.e., the absolute risks per group and/or a risk ratio are provided). Instead, all studies for which sufficient data are available for reanalysis to compare different exposures (e.g., sufficient data in the publication) are classified as cohort studies.There are possibly large numbers of studies without a comparison for the exposure of interest but that do provide the necessary data to calculate effect measures for a comparison. Consequently, more studies could be included in a systematic review. Therefore, on the one hand, the outlined approach can increase the confidence in effect estimates and the strengths of conclusions. On the other hand, the workload would increase (e.g., additional data extraction and risk of bias assessment, as well as reanalyses).

The term "innovation" refers to new products, but also to the process of developing and distributing new products and procedures. The operative disciplines are often associated with innovations because of their continuous, stepwise adaptation of daily practice to established procedures. Medical devices play a significant role in integrating surgical technology with surgical experience. The success of a surgical innovation and other invasive treatments does not only depend on the surgical procedure, but also on the context of the whole treatment process including the pre- and postoperative phase, the interaction of the surgical team and the setting. High standards have been set for the assessment of surgical innovations in terms of patient safety, efficacy and patient benefit, which will be discussed in the present paper. A stepwise approach to evaluation will be used, split into preclinical development, clinical development (feasibility and safety), evaluation phase (efficacy and patient benefit) and longtime surveillance. Our paper is based on the expert-based consented IDEAL approach as well as the consented recommendations of the European Association of Endoscopic Surgery (EAES). (As supplied by publisher)., (Copyright © 2015. Published by Elsevier GmbH.)

Background and Objective: Research collaborations in biomedical research have evolved over time. No studies have addressed research collaboration in network meta-analysis (NMA). In this study, we used social network analysis methods to characterize global collaboration patterns of published NMAs over the past decades. Methods: PubMed, EMBASE, Web of Science and the Cochrane Library were searched (at 9th July, 2015) to include systematic reviews incorporating NMA. Two reviewers independently selected studies and cross-checked the standardized data. Data was analyzed using Ucinet 6.0 and SPSS 17.0. NetDraw software was used to draw social networks. Results: 771 NMAs published in 336 journals from 3459 authors and 1258 institutions in 49 countries through the period 1997–2015 were included. More than three-quarters (n = 625; 81.06%) of the NMAs were published in the last 5-years. The BMJ (4.93%), Current Medical Research and Opinion (4.67%) and PLOS One (4.02%) were the journals that published the greatest number of NMAs. The UK and the USA (followed by Canada, China, the Netherlands, Italy and Germany) headed the absolute global productivity ranking in number of NMAs. The top 20 authors and institutions with the highest publication rates were identified. Overall, 43 clusters of authors (four major groups: one with 37 members, one with 12 members, one with 11 members and one with 10 members) and 21 clusters of institutions (two major groups: one with 62 members and one with 20 members) were identified. The most prolific authors were affiliated with academic institutions and private consulting firms. 181 consulting firms and pharmaceutical industries (14.39% of institutions) were involved in 199 NMAs (25.81% of total publications). Although there were increases in international and inter-institution collaborations, the research collaboration by authors, institutions and countries were still weak and most collaboration groups were small sizes. Conclusion: Scientific production on NMA is increasing worldwide with research leadership of Western countries (most notably, the UK, the USA and Canada). More authors, institutions and nations are becoming involved in research collaborations, but frequently with limited international collaborations. [ABSTRACT FROM AUTHOR]

Background and Objective: As systematic reviews may run out of date, it might be necessary to update them. Out-of-date reviews may jeopardize the comparability when used in the context of overviews (review of reviews). Methods: Seven electronic databases were searched for overviews up to November 2012. We first aimed to analyze whether the authors of overviews additionally searched for primary studies or alternatively explained why they did not. Second, we sought to analyze the actual publication lag (publication date of the overview--publication date of the review) in overviews and to develop recommendations for authors of overviews. Results: We included 147 overviews. The mean publication lag in overviews was more than 5 years. A median of 36% of the reviews were published more than 6 years ago. Only one in four reviews considered up-to-dateness. The methods for updating reviews were heterogeneous. We found no overview that systematically investigated whether an update was necessary. Conclusion: The issue of up-to-dateness when conducting overviews seems to be neglected by most authors of overviews. Authors should assess the quality of evidence, based on their included reviews first. [ABSTRACT FROM AUTHOR]

Objective The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews). Methods Based on a published overview investigating the hospital volume-outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five-point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated. Results The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter-rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers. Conclusion The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items. [ABSTRACT FROM AUTHOR]

Objectives: The objective of this study was to determine if the authors mention overlapping reviews in overviews (reviews of reviews). In addition, we aimed to calculate the actual overlap in published overviews using newly introduced, validated measures. Study Design and Settings: We systematically searched for overviews from 2009 to 2011. Reviews included in the overviews were obtained. Tables (review × primary publication) were generated for each overview. The first occurrence of a primary publication is defined as the index publication. We calculated the ''corrected covered area'' (CCA) as a measure of overlap by dividing the frequency of repeated occurrences of the index publication in other reviews by the product of index publications and reviews, reduced by the number of index publications. Subgroup analyses were performed to investigate further differences in the overviews. Results: Only 32 of 60 overviews mentioned overlaps. The median CCA was 4.0. Validation of the CCA and other overlap measures was in accordance with our predefined hypotheses. The degree of overlap tended to be higher in health technology assessment reports than in journal publications and was higher with increasing numbers of publications. Conclusions: Overlaps must be reported in well-conducted overviews, and this can comprehensively be accomplished using the CCA method. [ABSTRACT FROM AUTHOR]