Search

Comprehensive systematic reviews require many hours of manual review by researchers. This study retrospectively investigates the use of GPT-3.5 Turbo to screen titles and abstracts as a second reviewer or to prescreen citations for human review. The authors describe the process of prompt development as well as the performance of GPT-3.5 Turbo in terms of citations missed and the added number of citations for reconciliation for 5 different already completed reviews. Background: Systematic reviews are performed manually despite the exponential growth of scientific literature. Objective: To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews. Design: Diagnostic test accuracy study. Setting: Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations. Participants: None. Measurements: A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened. Results: Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level. Limitations: Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time. Conclusion: The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level. Primary Funding Source: None. [ABSTRACT FROM AUTHOR]

Background: Living practice guidelines are increasingly being used to ensure that recommendations are responsive to rapidly emerging evidence.Objective: To develop a framework that characterizes the processes of development of living practice guidelines in health care.Design: First, 3 background reviews were conducted: a scoping review of methods papers, a review of handbooks of guideline-producing organizations, and an analytic review of selected living practice guidelines. Second, the core team drafted the first version of the framework. Finally, the core team refined the framework through an online survey and online discussions with a multidisciplinary international group of stakeholders.Setting: International.Participants: Multidisciplinary group of 51 persons who have experience with guidelines.Measurements: Not applicable.Results: A major principle of the framework is that the unit of update in a living guideline is the individual recommendation. In addition to providing definitions, the framework addresses several processes. The planning process should address the organization's adoption of the living methodology as well as each specific guideline project. The production process consists of initiation, maintenance, and retirement phases. The reporting should cover the evidence surveillance time stamp, the outcome of reassessment of the body of evidence (when applicable), and the outcome of revisiting a recommendation (when applicable). The dissemination process may necessitate the use of different venues, including one for formal publication.Limitation: This study does not provide detailed or practical guidance for how the described concepts would be best implemented.Conclusion: The framework will help guideline developers in planning, producing, reporting, and disseminating living guideline projects. It will also help research methodologists study the processes of living guidelines.Primary Funding Source: None. [ABSTRACT FROM AUTHOR]

Incomplete and inadequate reporting is an avoidable waste that reduces the usefulness of research. The CONSORT (Consolidated Standards of Reporting Trials) Statement is an evidence-based reporting guideline that aims to improve research transparency and reduce waste. In 2008, the CONSORT Group developed an extension to the original statement that addressed methodological issues specific to trials of nonpharmacologic treatments (NPTs), such as surgery, rehabilitation, or psychotherapy. This article describes an update of that extension and presents an extension for reporting abstracts of NPT trials. To develop these materials, the authors reviewed pertinent literature published up to July 2016; surveyed authors of NPT trials; and conducted a consensus meeting with editors, trialists, and methodologists. Changes to the CONSORT Statement extension for NPT trials include wording modifications to improve readers' understanding and the addition of 3 new items. These items address whether and how adherence of participants to interventions is assessed or enhanced, description of attempts to limit bias if blinding is not possible, and specification of the delay between randomization and initiation of the intervention. The CONSORT extension for abstracts of NPT trials includes 2 new items that were not specified in the original CONSORT Statement for abstracts. The first addresses reporting of eligibility criteria for centers where the intervention is performed and for care providers. The second addresses reporting of important changes to the intervention versus what was planned. Both the updated CONSORT extension for NPT trials and the CONSORT extension for NPT trial abstracts should help authors, editors, and peer reviewers improve the transparency of NPT trial reports. [ABSTRACT FROM AUTHOR]

Background: The conduct of randomized, controlled trials of nonpharmacologic treatments presents specific challenges that are not adequately addressed in trial reports. Objective: To develop an extension of the CONSORT (Consolidated Standards of Reporting Trials) Statement for trials of nonpharmacologic treatments. Design: A consensus meeting was organized to develop an extension of the CONSORT Statement that addresses randomized trials of nonpharmacologic treatments. To prepare for the meeting, a survey was conducted to identify the specific issues for discussion. Setting: Consensus meeting in Paris, France. Participants: A total of 33 experts attended the meeting. The experts were methodologists (n = 17); surgeons (n = 6); editors (n = 5); and clinicians involved in rehabilitation (n = 1), psychotherapy (n = 2), education (n = 1), and implantable devices (n = 1). Measurements: Experts indicated which of the 22 items on the CONSORT checklist should be modified or which additional items should be added specifically for nonpharmacologic treatments. During a 3-day consensus meeting, all items were discussed and additional methodological issues related to nonpharmacologic research were identified. Results: The consensus was that 11 items on the CONSORT checklist needed some modifications for nonpharmacologic trials: item 1 (title and abstract), item 3 participants), item 4 (interventions), item 7 (sample size), item 8 (randomization), item 11 (blinding), item 12 (statistical methods), item 13 (participant flow), item 15 (baseline data), item 20 (discussion: interpretation), and item 21 (generalizability). In addition, the meeting participants added 1 item related to implementation of the intervention. Limitation: Evidence was not always available to support the inclusion of each checklist item. Conclusion: The methods and processes used to develop this extension could be used for other reporting guidelines. The use of this extension to the CONSORT Statement should improve the quality of reporting randomized, controlled trials assessing nonpharmacologic treatments. [ABSTRACT FROM AUTHOR]