Search

As generative artificial intelligence (AI), particularly Large Language Models (LLMs), continues to permeate healthcare, it remains crucial to supplement traditional automated evaluations with human expert evaluation. Understanding and evaluating the generated texts is vital for ensuring safety, reliability, and effectiveness. However, the cumbersome, time-consuming, and non-standardized nature of human evaluation presents significant obstacles to the widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs within healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, spans publications from January 2018 to February 2024. This review provides a comprehensive overview of the human evaluation approaches used in diverse healthcare applications.This analysis examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types, and sizes, the selection and recruitment of evaluators, frameworks and metrics, the evaluation process, and statistical analysis of the results. Drawing from diverse evaluation strategies highlighted in these studies, we propose a comprehensive and practical framework for human evaluation of generative LLMs, named QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of generative LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines.

Objectives: To identify correlates of survival and perioperative outcomes of upper tract urothelial carcinoma (UTUC) patients undergoing open (ORNU), laparoscopic (LRNU), and robotic (RRNU) radical nephroureterectomy (RNU). Methods: We conducted a retrospective, multicenter study that included non-metastatic UTUC patients who underwent RNU between 1990–2020. Multiple imputation by chained equations was used to impute missing data. Patients were divided into three groups based on their surgical treatment and were adjusted by 1:1:1 propensity score matching (PSM). Survival outcomes per group were estimated for recurrence-free survival (RFS), bladder recurrence-free survival (BRFS), cancer-specific survival (CSS), and overall survival (OS). Perioperative outcomes: Intraoperative blood loss, hospital length of stay (LOS), and overall (OPC) and major postoperative complications (MPCs; defined as Clavien–Dindo > 3) were assessed between groups. Results: Of the 2434 patients included, 756 remained after PSM with 252 in each group. The three groups had similar baseline clinicopathological characteristics. The median follow-up was 32 months. Kaplan–Meier and log-rank tests demonstrated similar RFS, CSS, and OS between groups. BRFS was found to be superior with ORNU. Using multivariable regression analyses, LRNU and RRNU were independently associated with worse BRFS (HR 1.66, 95% CI 1.22–2.28, p = 0.001 and HR 1.73, 95%CI 1.22–2.47, p = 0.002, respectively). LRNU and RRNU were associated with a significantly shorter LOS (beta −1.1, 95% CI −2.2–0.02, p = 0.047 and beta −6.1, 95% CI −7.2–5.0, p < 0.001, respectively) and fewer MPCs (OR 0.5, 95% CI 0.31–0.79, p = 0.003 and OR 0.27, 95% CI 0.16–0.46, p < 0.001, respectively). Conclusions: In this large international cohort, we demonstrated similar RFS, CSS, and OS among ORNU, LRNU, and RRNU. However, LRNU and RRNU were associated with significantly worse BRFS, but a shorter LOS and fewer MPCs.

[Background] Artificial intelligence (AI) is a promising tool in pathology, including cancer diagnosis, subtyping, grading, and prognostic prediction., [Methods] The aim of the study is to assess AI application in prostate cancer (PCa) histology. We carried out a systematic literature search in 3 databases. Primary outcome was AI accuracy in differentiating between PCa and benign hyperplasia. Secondary outcomes were AI accuracy in determining Gleason grade and agreement among AI and pathologists., [Results] Our final sample consists of 24 studies conducted from 2007 to 2021. They aggregate data from roughly 8000 cases of prostate biopsy and 458 cases of radical prostatectomy (RP). Sensitivity for PCa diagnostic exceeded 90% and ranged from 87% to 100%, and specificity varied from 68% to 99%. Overall accuracy ranged from 83.7% to 98.3% with AUC reaching 0.99. The meta-analysis using the Mantel-Haenszel method showed pooled sensitivity of 0.96 with I2 = 80.7% and pooled specificity of 0.95 with I2 = 86.1%. Pooled positive likehood ratio was 15.3 with I2 = 87.3% and negative – was 0.04 with I2 = 78.6%. SROC (symmetric receiver operating characteristics) curve represents AUC = 0.99. For grading the accuracy of AI was lower: sensitivity for Gleason grading ranged from 77% to 87%, and specificity from 82% to 90%., [Conclusions] The accuracy of AI for PCa identification and grading is comparable to expert pathologists. This is a promising approach which has several possible clinical applications resulting in expedite and optimize pathology reports. AI introduction into common practice may be limited by difficult and time-consuming convolutional neural network training and tuning.

OBJECTIVES To identify correlates of survival and perioperative outcomes of upper tract urothelial carcinoma (UTUC) patients undergoing open (ORNU), laparoscopic (LRNU), and robotic (RRNU) radical nephroureterectomy (RNU). METHODS We conducted a retrospective, multicenter study that included non-metastatic UTUC patients who underwent RNU between 1990-2020. Multiple imputation by chained equations was used to impute missing data. Patients were divided into three groups based on their surgical treatment and were adjusted by 1:1:1 propensity score matching (PSM). Survival outcomes per group were estimated for recurrence-free survival (RFS), bladder recurrence-free survival (BRFS), cancer-specific survival (CSS), and overall survival (OS). Perioperative outcomes: Intraoperative blood loss, hospital length of stay (LOS), and overall (OPC) and major postoperative complications (MPCs; defined as Clavien-Dindo > 3) were assessed between groups. RESULTS Of the 2434 patients included, 756 remained after PSM with 252 in each group. The three groups had similar baseline clinicopathological characteristics. The median follow-up was 32 months. Kaplan-Meier and log-rank tests demonstrated similar RFS, CSS, and OS between groups. BRFS was found to be superior with ORNU. Using multivariable regression analyses, LRNU and RRNU were independently associated with worse BRFS (HR 1.66, 95% CI 1.22-2.28, p = 0.001 and HR 1.73, 95%CI 1.22-2.47, p = 0.002, respectively). LRNU and RRNU were associated with a significantly shorter LOS (beta -1.1, 95% CI -2.2-0.02, p = 0.047 and beta -6.1, 95% CI -7.2-5.0, p < 0.001, respectively) and fewer MPCs (OR 0.5, 95% CI 0.31-0.79, p = 0.003 and OR 0.27, 95% CI 0.16-0.46, p < 0.001, respectively). CONCLUSIONS In this large international cohort, we demonstrated similar RFS, CSS, and OS among ORNU, LRNU, and RRNU. However, LRNU and RRNU were associated with significantly worse BRFS, but a shorter LOS and fewer MPCs.

We aim to determine the extent and content of guidance for authors regarding the use of generative-AI (GAI), Generative Pretrained models (GPTs) and Large Language Models (LLMs) powered tools among the top 100 academic publishers and journals in science. The websites of these publishers and journals were screened from between 19th and 20th May 2023. Among the largest 100 publishers, 17% provided guidance on the use of GAI, of which 12 (70.6%) were among the top 25 publishers. Among the top 100 journals, 70% have provided guidance on GAI. Of those with guidance, 94.1% of publishers and 95.7% of journals prohibited the inclusion of GAI as an author. Four journals (5.7%) explicitly prohibit the use of GAI in the generation of a manuscript, while 3 (17.6%) publishers and 15 (21.4%) journals indicated their guidance exclusively applies to the writing process. When disclosing the use of GAI, 42.8% of publishers and 44.3% of journals included specific disclosure criteria. There was variability in guidance of where to disclose the use of GAI, including in the methods, acknowledgments, cover letter, or a new section. There was also variability in how to access GAI guidance and the linking of journal and publisher instructions to authors. There is a lack of guidance by some top publishers and journals on the use of GAI by authors. Among those publishers and journals that provide guidance, there is substantial heterogeneity in the allowable uses of GAI and in how it should be disclosed, with this heterogeneity persisting among affiliated publishers and journals in some instances. The lack of standardization burdens authors and threatens to limit the effectiveness of these regulations. There is a need for standardized guidelines in order to protect the integrity of scientific output as GAI continues to grow in popularity., Comment: Pages 16, 1 figure, 2 tables

The swift progress and ubiquitous adoption of Generative AI (GAI), Generative Pre-trained Transformers (GPTs), and large language models (LLMs) like ChatGPT, have spurred queries about their ethical application, use, and disclosure in scholarly research and scientific productions. A few publishers and journals have recently created their own sets of rules; however, the absence of a unified approach may lead to a 'Babel Tower Effect,' potentially resulting in confusion rather than desired standardization. In response to this, we present the ChatGPT, Generative Artificial Intelligence, and Natural Large Language Models for Accountable Reporting and Use Guidelines (CANGARU) initiative, with the aim of fostering a cross-disciplinary global inclusive consensus on the ethical use, disclosure, and proper reporting of GAI/GPT/LLM technologies in academia. The present protocol consists of four distinct parts: a) an ongoing systematic review of GAI/GPT/LLM applications to understand the linked ideas, findings, and reporting standards in scholarly research, and to formulate guidelines for its use and disclosure, b) a bibliometric analysis of existing author guidelines in journals that mention GAI/GPT/LLM, with the goal of evaluating existing guidelines, analyzing the disparity in their recommendations, and identifying common rules that can be brought into the Delphi consensus process, c) a Delphi survey to establish agreement on the items for the guidelines, ensuring principled GAI/GPT/LLM use, disclosure, and reporting in academia, and d) the subsequent development and dissemination of the finalized guidelines and their supplementary explanation and elaboration documents., Comment: 20 pages, 1 figure, protocol

Measuring the quality of care is important in health care to improve the treatment of patients. In this investigation, we sought to identify five indicators ("pentafecta") that reflect the quality of care of patients who have cancer of the upper urinary tract (kidney and/or ureter) and are treated with surgical removal of the affected kidney and ureter. Furthermore, we searched for conditions that can predict a failure to achieve these criteria during treatment. The five indicators that define the pentafecta are the complete removal of the tumor without residuals, the complete removal of the ureter and its bladder part, the absence of complications related to the blood, the absence of severe complications related to the surgery, and the absence of tumor recurrence 12 months after the surgery. Of the 1718 patients included, 844 (49%) achieved all pentafecta criteria. These patients had higher chances at 5 years after the surgery to be alive and not to die from any cause (A) or from cancer (B) compared to those who did not achieve the pentafecta criteria (A: 68.7 vs. 50.1% and B: 79.8 vs. 62.7%, respectively). There were no conditions related to the patient that were found to predict a failure to achieve the pentafecta. Using quality indicators such as the proposed pentafecta for the assessment of the treatment of cancer patients may help define prognosis and improve patient care. Background: Measuring quality of care indicators is important for clinicians and decision making in health care to improve patient outcomes. Objective: The primary objective was to identify quality of care indicators for patients with upper tract urothelial carcinoma (UTUC) and to validate these in an international cohort treated with radical nephroureterectomy (RNU). The secondary objective was to assess the factors associated with failure to validate the pentafecta. Design: We performed a retrospective multicenter study of patients treated with RNU for EAU high-risk (HR) UTUC. Outcome meas

Simple Summary Measuring the quality of care is important in health care to improve the treatment of patients. In this investigation, we sought to identify five indicators (pentafecta) that reflect the quality of care of patients who have cancer of the upper urinary tract (kidney and/or ureter) and are treated with surgical removal of the affected kidney and ureter. Furthermore, we searched for conditions that can predict a failure to achieve these criteria during treatment. The five indicators that define the pentafecta are the complete removal of the tumor without residuals, the complete removal of the ureter and its bladder part, the absence of complications related to the blood, the absence of severe complications related to the surgery, and the absence of tumor recurrence 12 months after the surgery. Of the 1718 patients included, 844 (49%) achieved all pentafecta criteria. These patients had higher chances at 5 years after the surgery to be alive and not to die from any cause (A) or from cancer (B) compared to those who did not achieve the pentafecta criteria (A: 68.7 vs. 50.1% and B: 79.8 vs. 62.7%, respectively). There were no conditions related to the patient that were found to predict a failure to achieve the pentafecta. Using quality indicators such as the proposed pentafecta for the assessment of the treatment of cancer patients may help define prognosis and improve patient care. Background: Measuring quality of care indicators is important for clinicians and decision making in health care to improve patient outcomes. Objective: The primary objective was to identify quality of care indicators for patients with upper tract urothelial carcinoma (UTUC) and to validate these in an international cohort treated with radical nephroureterectomy (RNU). The secondary objective was to assess the factors associated with failure to validate the pentafecta. Design: We performed a retrospective multicenter study of patients treated with RNU for EAU high-risk (HR) UTUC.

Background: Measuring quality of care indicators is important for clinicians and decision making in health care to improve patient outcomes. Objective: The primary objective was to identify quality of care indicators for patients with upper tract urothelial carcinoma (UTUC) and to validate these in an international cohort treated with radical nephroureterectomy (RNU). The secondary objective was to assess the factors associated with failure to validate the pentafecta. Design: We performed a retrospective multicenter study of patients treated with RNU for EAU high-risk (HR) UTUC. Outcome measurements and statistical analysis: Five quality indicators were consensually approved, including a negative surgical margin, a complete bladder-cuff resection, the absence of hematological complications, the absence of major complications, and the absence of a 12-month postoperative recurrence. After multiple imputations and propensity-score matching, log-rank tests and a Cox regression were used to assess the survival outcomes. Logistic regression analyses assessed predictors for pentafecta failure. Results: Among the 1718 included patients, 844 (49%) achieved the pentafecta. The median follow-up was 31 months. Patients who achieved the pentafecta had superior 5-year overall-(OS) and cancer-specific survival (CSS) compared to those who did not (68.7 vs. 50.1% and 79.8 vs. 62.7%, respectively, all p < 0.001). On multivariable analyses, achieving the pentafecta was associated with improved recurrence-free survival (RFS), CSS, and OS. No preoperative clinical factors predicted a failure to validate the pentafecta. Conclusions: Establishing quality indicators for UTUC may help define prognosis and improve patient care. We propose a pentafecta quality criteria in RNU patients. Approximately half of the patients evaluated herein reached this endpoint, which in turn was independently associated with survival outcomes. Extended validation is needed.

Context: Pelvic lymph node dissection (PLND) yields the most accurate staging in patients undergoing radical prostatectomy (RP) for prostate cancer (PCa), although it can be associated with morbidity. Objective: To systematically evaluate the impact of PLND extent on perioperative morbidity in patients undergoing RP. A new PLND-related complication assessment tool is proposed. Evidence acquisition: A systematic review according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) was conducted. MEDLINE/ PubMed, Scopus, Embase and Web of Science databases were searched to yield studies discussing perioperative complications following RP and PLND. The extent of PLND was classified according to the European Association of Urology PCa guidelines. Studies were categorized according to the extent of PLND. Intra- and postoperative complications were classified as strongly, likely, or unlikely related to PLND. Anatomical site of perioperative complications was recorded. A cumulative meta-analysis of comparative studies was conducted using Review Manager 5.3 (Cochrane Collaboration, Oxford, UK). Evidence synthesis: Our search generated 3645 papers, with 176 studies meeting the inclusion criteria. Details of 77 303 patients were analyzed. Of these studies, 84 (47.7%), combining data on 28 428 patients, described intraoperative complications as an outcome of interest. Overall, 534 (1.8%) patients reported one or more intraoperative complications. Postoperative complications were reported in 151 (85.7%) studies, combining data on 73 629 patients. Overall, 10 401 (14.1%) patients reported one or more postoperative complication. The most reported postoperative complication strongly related to PLND was lymphocele (90.6%). The pooled metaanalysis revealed that RP + limited PLND/standard PLND had a significantly decreased risk of experiencing any intraoperative complication (risk ratio [RR]: 0.55; p = 0.01) and postoperative complication strongly relate

Context: Pelvic lymph node dissection (PLND) yields the most accurate staging in patients undergoing radical prostatectomy (RP) for prostate cancer (PCa), although it can be associated with morbidity. Objective: To systematically evaluate the impact of PLND extent on perioperative morbidity in patients undergoing RP. A new PLND-related complication assessment tool is proposed. Evidence acquisition: A systematic review according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) was conducted. MEDLINE/ PubMed, Scopus, Embase and Web of Science databases were searched to yield studies discussing perioperative complications following RP and PLND. The extent of PLND was classified according to the European Association of Urology PCa guidelines. Studies were categorized according to the extent of PLND. Intra- and postoperative complications were classified as strongly, likely, or unlikely related to PLND. Anatomical site of perioperative complications was recorded. A cumulative meta-analysis of comparative studies was conducted using Review Manager 5.3 (Cochrane Collaboration, Oxford, UK). Evidence synthesis: Our search generated 3645 papers, with 176 studies meeting the inclusion criteria. Details of 77 303 patients were analyzed. Of these studies, 84 (47.7%), combining data on 28 428 patients, described intraoperative complications as an outcome of interest. Overall, 534 (1.8%) patients reported one or more intraoperative complications. Postoperative complications were reported in 151 (85.7%) studies, combining data on 73 629 patients. Overall, 10 401 (14.1%) patients reported one or more postoperative complication. The most reported postoperative complication strongly related to PLND was lymphocele (90.6%). The pooled metaanalysis revealed that RP + limited PLND/standard PLND had a significantly decreased risk of experiencing any intraoperative complication (risk ratio [RR]: 0.55; p = 0.01) and postoperative complication strongly relate