Search

BackgroundNatural Language Understanding enables automatic extraction of relevant information from clinical text data, which are acquired every day in hospitals. In 2018, the language model Bidirectional Encoder Representations from Transformers (BERT) was introduced, generating new state-of-the-art results on several downstream tasks. The National NLP Clinical Challenges (n2c2) is an initiative that strives to tackle such downstream tasks on domain-specific clinical data. In this paper, we present the results of our participation in the 2019 n2c2 and related work completed thereafter. ObjectiveThe objective of this study was to optimally leverage BERT for the task of assessing the semantic textual similarity of clinical text data. MethodsWe used BERT as an initial baseline and analyzed the results, which we used as a starting point to develop 3 different approaches where we (1) added additional, handcrafted sentence similarity features to the classifier token of BERT and combined the results with more features in multiple regression estimators, (2) incorporated a built-in ensembling method, M-Heads, into BERT by duplicating the regression head and applying an adapted training strategy to facilitate the focus of the heads on different input patterns of the medical sentences, and (3) developed a graph-based similarity approach for medications, which allows extrapolating similarities across known entities from the training set. The approaches were evaluated with the Pearson correlation coefficient between the predicted scores and ground truth of the official training and test dataset. ResultsWe improved the performance of BERT on the test dataset from a Pearson correlation coefficient of 0.859 to 0.883 using a combination of the M-Heads method and the graph-based similarity approach. We also show differences between the test and training dataset and how the two datasets influenced the results. ConclusionsWe found that using a graph-based similarity approach has the potential to extrapolate domain specific knowledge to unseen sentences. We observed that it is easily possible to obtain deceptive results from the test dataset, especially when the distribution of the data samples is different between training and test datasets.

Background: FHIR (Fast Healthcare Interoperability Resources) has been proposed to enable health data interoperability. So far, its applicability has been demonstrated for selected research projects with limited data., Objective: This study aimed to design and implement a conceptual medical intelligence framework to leverage real-world care data for clinical decision-making., Methods: A Python package for the use of multimodal FHIR data (FHIRPACK [FHIR Python Analysis Conversion Kit]) was developed and pioneered in 5 real-world clinical use cases, that is, myocardial infarction, stroke, diabetes, sepsis, and prostate cancer. Patients were identified based on the ICD-10 (International Classification of Diseases, Tenth Revision) codes, and outcomes were derived from laboratory tests, prescriptions, procedures, and diagnostic reports. Results were provided as browser-based dashboards., Results: For 2022, a total of 1,302,988 patient encounters were analyzed. (1) Myocardial infarction: in 72.7% (261/359) of cases, medication regimens fulfilled guideline recommendations. (2) Stroke: out of 1277 patients, 165 received thrombolysis and 108 thrombectomy. (3) Diabetes: in 443,866 serum glucose and 16,180 glycated hemoglobin A 1c measurements from 35,494 unique patients, the prevalence of dysglycemic findings was 39% (13,887/35,494). Among those with dysglycemia, diagnosis was coded in 44.2% (6138/13,887) of the patients. (4) Sepsis: In 1803 patients, Staphylococcus epidermidis was the primarily isolated pathogen (773/2672, 28.9%) and piperacillin and tazobactam was the primarily prescribed antibiotic (593/1593, 37.2%). (5) PC: out of 54, three patients who received radical prostatectomy were identified as cases with prostate-specific antigen persistence or biochemical recurrence., Conclusions: Leveraging FHIR data through large-scale analytics can enhance health care quality and improve patient outcomes across 5 clinical specialties. We identified (1) patients with sepsis requiring less broad antibiotic therapy, (2) patients with myocardial infarction who could benefit from statin and antiplatelet therapy, (3) patients who had a stroke with longer than recommended times to intervention, (4) patients with hyperglycemia who could benefit from specialist referral, and (5) patients with PC with early increases in cancer markers., (©Alexander Brehmer, Christopher Martin Sauer, Jayson Salazar Rodríguez, Kelsey Herrmann, Moon Kim, Julius Keyl, Fin Hendrik Bahnsen, Benedikt Frank, Martin Köhrmann, Tienush Rassaf, Amir-Abbas Mahabadi, Boris Hadaschik, Christopher Darr, Ken Herrmann, Susanne Tan, Jan Buer, Thorsten Brenner, Hans Christian Reinhardt, Felix Nensa, Michael Gertz, Jan Egger, Jens Kleesiek. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 31.10.2024.)

Objective: Federated Learning (FL) enables collaborative model training while keeping data locally. Currently, most FL studies in radiology are conducted in simulated environments due to numerous hurdles impeding its translation into practice. The few existing real-world FL initiatives rarely communicate specific measures taken to overcome these hurdles. To bridge this significant knowledge gap, we propose a comprehensive guide for real-world FL in radiology. Minding efforts to implement real-world FL, there is a lack of comprehensive assessments comparing FL to less complex alternatives in challenging real-world settings, which we address through extensive benchmarking., Materials and Methods: We developed our own FL infrastructure within the German Radiological Cooperative Network (RACOON) and demonstrated its functionality by training FL models on lung pathology segmentation tasks across six university hospitals. Insights gained while establishing our FL initiative and running the extensive benchmark experiments were compiled and categorized into the guide., Results: The proposed guide outlines essential steps, identified hurdles, and implemented solutions for establishing successful FL initiatives conducting real-world experiments. Our experimental results prove the practical relevance of our guide and show that FL outperforms less complex alternatives in all evaluation scenarios., Discussion and Conclusion: Our findings justify the efforts required to translate FL into real-world applications by demonstrating advantageous performance over alternative approaches. Additionally, they emphasize the importance of strategic organization, robust management of distributed data and infrastructure in real-world settings. With the proposed guide, we are aiming to aid future FL researchers in circumventing pitfalls and accelerating translation of FL into radiological applications., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)

Large language models (LLMs) have broad medical knowledge and can reason about medical information across many domains, holding promising potential for diverse medical applications in the near future. In this study, we demonstrate a concerning vulnerability of LLMs in medicine. Through targeted manipulation of just 1.1% of the weights of the LLM, we can deliberately inject incorrect biomedical facts. The erroneous information is then propagated in the model's output while maintaining performance on other biomedical tasks. We validate our findings in a set of 1025 incorrect biomedical facts. This peculiar susceptibility raises serious security and trustworthiness concerns for the application of LLMs in healthcare settings. It accentuates the need for robust protective measures, thorough verification mechanisms, and stringent management of access to these models, ensuring their reliable and safe use in medical practice., (© 2024. The Author(s).)

Background: Immune checkpoint inhibition (ICI) currently is the most effective treatment to induce durable responses in metastatic melanoma. The aims of this study are the characterization of patients with early, late and non-response to ICI and analysis of survival outcomes in a real-world patient cohort., Methods: Patients who received PD-1-based immunotherapy for non-resectable stage-IV melanoma in any therapy line were selected from the prospective multicenter real-world DeCOG study ADOREG-TRIM (NCT05750511). Patients showing complete (CR) or partial (PR) response already during the first 3 months of treatment (Early Responders, EarlyR) were compared to patients showing CR/PR at a later time (Late Responders, LateR), a stable disease (SD) and to patients showing progressive disease (Non-Responders, NonR)., Results: Of 522 patients, 8.2 % were EarlyR (n = 43), 19.0 % were LateR (n = 99), 37.0 % had a SD (n = 193) and 35.8 % were NonR (n = 187). EarlyR, LateR and SD patients had comparable baseline characteristics. Multivariate logbinomial regression analyses adjusted for age and sex revealed positive tumor PD-L1 (RR=1.99, 95 %-CI=1.14-3.46, p = 0.015), and normal serum CRP (RR=1.59, 95 %-CI=0.93-2.70, p = 0.036) as independently associated with the achievement of an early response compared to NonR. The median progression-free and overall survival was 46.0 months (95 % CI 19.1; NR) and 47.8 months (95 %-CI 36.9; NR) for EarlyR, NR (95 %-CI NR; NR) for LateR, 8.1 months (7.0; 10.4) and 35.4 months (29.2; NR) for SD, and 2.0 months (95 %-CI 1.9; 2.1) and 6.1 months (95 %-CI 4.6; 8.8) for NonR patients., Conclusion: Less than 10 % of metastatic melanoma patients achieved an early response during the first 3 months of PD-1-based immunotherapy. Early responders were not superior to late responders in terms of response durability and survival., Competing Interests: Declaration of Competing Interest GCL received travel support from Sun Pharma, Pierre Fabre, research funding from Novartis. RH is employee of Helios Kliniken GmbH. PT has received honoraria from Bristol-Myers Squibb, Novartis, Merck Sharp & Dohme, Pierre Fabre, CureVac, Merck Serono, Sanofi, Roche, Kyowa Kirin and Biofrontera; and travel support from Bristol-Myers Squibb and Pierre Fabre. JU received honoraria (speaker honoraria or honoraria as a consultant) and travel support from Bristol-Myers Squibb, Kyowa Kirin, Merck Sharp & Dohme, Novartis, Pfizer, Pierre Fabre, Roche, Sanofi/Regeneron, Sunpharma outside the submitted work. CP received honoraria (speaker honoraria or honoraria as a consultant) and travel support from Novartis, BMS, MSD, Merck Serono, MSD, Celgene, AbbVie, Sunpharma, Pierre Fabre, UCB, Nutricia Milupa, Janssen and LEO, outside the submitted work. JU is on the advisory board or has received honoraria and travel support from Amgen, Bristol Myers Squibb, GSK, Immunocore, LeoPharma, Merck Sharp and Dohme, Novartis, Pierre Fabre, Roche, Sanofi outside the submitted work. AK served as a speaker and/or consultant and/or advisory board for MSD, AbbVie, Boehringer Ingelheim, Janssen, and Sanofi. PM declares research support from Bristol Myers Squibb, Merck Sharp & Dohme and Novartis; speakers and advisory board honoraria from Bristol Myers Squibb, Beiersdorf, Merck Sharp & Dohme, Pierre Fabre, Sun Pharma, Immunocore, Sanofi and Novartis, and travel support from Bristol Myers Squibb, Merck Sharp & Dohme, Sanofi, Sun Pharma, and Pierre Fabre, outside the submitted work. RG received honoraria as speaker from BMS, MSD, Novartis, Amgen, Merck Serono, Almirall Hermal, SUN, Sanofi/Regeneron, Pierre-Fabre, as advisory board member from BMS, Novartis, Almirall Hermal, MSD, Amgen, SUN, Sanofi/Regeneron, Pierre-Fabre, 4SC, MerckSerono, Pfizer, Immunocore, Delcath, for meeting support from SUN, Pierre-Fabre, Boehringer Ingelheim and for research projects (to institution) from Amgen, Merck-Serono, SUN Pharma, Sanofi/Regeneron, Kyowa-Kirin, Almirall-Hermal. FM has received travel support or/and speaker’s fees or/and advisor’s honoraria by Novartis, Roche, BMS, MSD, Pierre Fabre, Sanofi and Immunocore and research funding from Novartis and Roche. MW received grants from Bristol-Myers Squibb and Merck Sharp & Dohme, consulting fees from Merck Sharp & Dohme, Immunocore and Novartis, lecture honoraria from Bristol-Myers Squibb and Merck Sharp & Dohme and Pierre-Fabre, and advisory board honoraria from Merck Sharp & Dohme. AP reports grants from Bristol Myers Squibb (BMS) and Merck Sharp & Dohme (MSD) outside the submitted work. JCB is receiving speaker’s bureau honoraria from Amgen, Pfizer, Recordati and Sanofi, and is a paid consultant/advisory board member/DSMB member for Almirall, Boehringer Ingelheim, InProTher, ICON, MerckSerono, Pfizer, 4SC, and Sanofi/Regeneron. His group received research grants from Bristol-Myers Squibb, Merck Serono, HTG, IQVIA, and Alcedis. EL served as consultant and/or has received honoraria from Bristol-Myers Squibb, Merck Sharp & Dohme, Novartis, Pierre-Fabre, Sanofi, Sunpharma, Takeda and travel support from Bristol-Myers Squibb, Pierre- Fabre, Sunpharma and Novartis, outside the submitted work. LZ served as consultant and has received honoraria from BMS, MSD, Novartis, Pierre Fabre, Sanofi, and Sunpharma and travel support from MSD, BMS, Pierre Fabre, Sanofi, Sunpharma and Novartis, outside the submitted work. DS reports partial financial support from Bristol Myers Squibb for the conduct of this study and drug supply (nivolumab and ipilimumab) support; grants (or contracts) from Amgen, Array/Pfizer, Bristol-Myers Squibb, MSD, Novartis and Roche; consulting fees from 4SC, Amgen, Array Biopharma, AstraZeneca, Bristol-Myers Squibb, Daiichi Sankyo, Haystick, Immunocore, InFlarX, Innocent, LabCorp, Merck Serono, MSD, Nektar, NeraCare, Novartis, OncoSec, Pfizer, Philogen, Pierre Fabre, Replimune, Roche, Sandoz, Sanofi/Regeneron, Sun Pharma; honoraria from Bristol-Myers Squibb, MSD/Merck, Merck Serono, Novartis, Roche, Sanofi and Sun Pharma; support for attendings meetings or travel support from Bristol-Myers Squibb, MSD, Merck Serono, Novartis, Pierre Fabre and Sanofi; participation on drug safety monitoring or advisory boards for 4SC, Amgen, Array Biopharma, AstraZeneca, Bristol-Myers Squibb, Daiichi Sankyo, Immunocore, InFlarX, Merck Serono, MSD, Nektar, NeraCare, Novartis, OncoSec, Pfizer, Philogen, Pierre Fabre, Replimune, Roche, Sandoz, Sanofi/Regeneron and SunPharma; leadership roles for DeCOG, German Cancer Society, Hiege-Stiftung, Deutsche Hautkrebsstiftung, NVKH e.V. and EuMelaReg. SU declares research support from Bristol Myers Squibb and Merck Serono; speakers and advisory board honoraria from Bristol Myers Squibb, Merck Sharp & Dohme, Merck Serono, and Novartis; and meeting and travel support from Almirall, Bristol-Myers Squibb, IGEA Clinical Biophysics, Merck Sharp & Dohme, Novartis, Pierre Fabre, and Sun Pharma. All other authors declared no conflict of interest., (Copyright © 2024 The Authors. Published by Elsevier Ltd.. All rights reserved.)

Most clinical information is encoded as free text, not accessible for quantitative analysis. This study presents an open-source pipeline using the local large language model (LLM) "Llama 2" to extract quantitative information from clinical text and evaluates its performance in identifying features of decompensated liver cirrhosis. The LLM identified five key clinical features in a zero- and one-shot manner from 500 patient medical histories in the MIMIC IV dataset. We compared LLMs of three sizes and various prompt engineering approaches, with predictions compared against ground truth from three blinded medical experts. Our pipeline achieved high accuracy, detecting liver cirrhosis with 100% sensitivity and 96% specificity. High sensitivities and specificities were also yielded for detecting ascites (95%, 95%), confusion (76%, 94%), abdominal pain (84%, 97%), and shortness of breath (87%, 97%) using the 70 billion parameter model, which outperformed smaller versions. Our study successfully demonstrates the capability of locally deployed LLMs to extract clinical information from free text with low hardware requirements., (© 2024. The Author(s).)

At the Worldwide Developers Conference in June 2023, Apple introduced the Vision Pro. The Apple Vision Pro (AVP) is a mixed reality headset; more specifically, it is a virtual reality device with an additional video see-through capability. The video see-through capability turns the AVP into an augmented reality (AR) device. The AR feature is enabled by streaming the real world via cameras on the (virtual reality) screens in front of the user's eyes. This is, of course, not unique and is similar to other devices, such as the Varjo XR-3 (Varjo Technologies Oy). Nevertheless, the AVP has some interesting features, such as an inside-out screen that can show the headset wearer's eyes to "outsiders," and a button on the top, called the "digital crown," that allows a seamless blend of digital content with the user's physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to "The Ultimate Display," which Ivan Sutherland had already sketched in 1965. After a great response from the media and social networks to the release, we were able to test and review the new AVP ourselves in March 2024. Including an expert survey with 13 of our colleagues after testing the AVP in our institute, this Viewpoint explores whether the AVP can overcome clinical challenges that AR especially still faces in the medical domain; we also go beyond this and discuss whether the AVP could support clinicians in essential tasks to allow them to spend more time with their patients., (©Jan Egger, Christina Gsaxner, Gijs Luijten, Jianxu Chen, Xiaojun Chen, Jiang Bian, Jens Kleesiek, Behrus Puladi. Originally published in JMIR Serious Games (https://games.jmir.org), 18.09.2024.)

Background: Prostate-specific membrane antigen (PSMA)-PET was introduced into clinical practice in 2012 and has since transformed the staging of prostate cancer. Prostate Cancer Molecular Imaging Standardized Evaluation (PROMISE) criteria were proposed to standardise PSMA-PET reporting. We aimed to compare the prognostic value of PSMA-PET by PROMISE (PPP) stage with established clinical nomograms in a large prostate cancer dataset with follow-up data for overall survival., Methods: In this multicentre retrospective study, we used data from patients of any age with histologically proven prostate cancer who underwent PSMA-PET at the University Hospitals in Essen, Münster, Freiburg, and Dresden, Germany, between Oct 30, 2014, and Dec 27, 2021. We linked a subset of patient hospital records with patient data, including mortality data, from the Cancer Registry North-Rhine Westphalia, Germany. Patients from Essen University Hospital were randomly assigned to the development or internal validation cohorts (2:1). Patients from Münster, Freiburg, and Dresden University Hospitals were included in an external validation cohort. Using the development cohort, we created quantitative and visual PPP nomograms based on Cox regression models, assessing potential PPP predictors for overall survival, with least absolute shrinkage and selection operator penalty for overall survival as the primary endpoint. Performance was measured using Harrell's C-index in the internal and external validation cohorts and compared with established clinical risk scores (International Staging Collaboration for Cancer of the Prostate [STARCAP], European Association of Urology [EAU], and National Comprehensive Cancer Network [NCCN] risk scores) and a previous nomogram defined by Gafita et al (hereafter referred to as GAFITA) using receiver operating characteristic (ROC) curves and area under the ROC curve (AUC) estimates., Findings: We analysed 2414 male patients (1110 included in the development cohort, 502 in the internal cohort, and 802 in the external validation cohort), among whom 901 (37%) had died as of data cutoff (June 30, 2023; median follow-up of 52·9 months [IQR 33·9-79·0]). Predictors in the quantitative PPP nomogram were locoregional lymph node metastases (molecular imaging N2), distant metastases (extrapelvic nodal metastases, bone metastases [disseminated or diffuse marrow involvement], and organ metastases), tumour volume (in L), and tumour mean standardised uptake value. Predictors in the visual PPP nomogram were distant metastases (extrapelvic nodal metastases, bone metastases [disseminated or diffuse marrow involvement], and organ metastases) and total tumour lesion count. In the internal and external validation cohorts, C-indices were 0·80 (95% CI 0·77-0·84) and 0·77 (0·75-0·78) for the quantitative nomogram, respectively, and 0·78 (0·75-0·82) and 0·77 (0·75-0·78) for the visual nomogram, respectively. In the combined development and internal validation cohort, the quantitative PPP nomogram was superior to STARCAP risk score for patients at initial staging (n=139 with available staging data; AUC 0·73 vs 0·54; p=0·018), EAU risk score at biochemical recurrence (n=412; 0·69 vs 0·52; p<0·0001), and NCCN pan-stage risk score (n=1534; 0·81 vs 0·74; p<0·0001) for the prediction of overall survival, but was similar to GAFITA nomogram for metastatic hormone-sensitive prostate cancer (mHSPC; n=122; 0·76 vs 0·72; p=0·49) and metastatic castration-resistant prostate cancer (mCRPC; n=270; 0·67 vs 0·75; p=0·20). The visual PPP nomogram was superior to EAU at biochemical recurrence (n=414; 0·64 vs 0·52; p=0·0004) and NCCN across all stages (n=1544; 0·79 vs 0·73; p<0·0001), but similar to STARCAP for initial staging (n=140; 0·56 vs 0·53; p=0·74) and GAFITA for mHSPC (n=122; 0·74 vs 0·72; p=0·66) and mCRPC (n=270; 0·71 vs 0·75; p=0·23)., Interpretation: Our PPP nomograms accurately stratify high-risk and low-risk groups for overall survival in early and late stages of prostate cancer and yield equal or superior prediction accuracy compared with established clinical risk tools. Validation and improvement of the nomograms with long-term follow-up is ongoing (NCT06320223)., Funding: Cancer Registry North-Rhine Westphalia., Competing Interests: Declaration of interests MJK reports ownership of stocks in Bayer. JH reports participation on a data safety monitoring board for CARP-Study (carp-studie.de). VG reports grants or contracts from Pfizer, AstraZeneca, Bristol Myers Squibb (BMS), and MSD; consulting fees from BMS, Pfizer, Novartis, MSD Oncology, Ipsen, Janssen-Cilag, Eisai, Debiopharm, PCI Biotech, Gliead, Cureteq, Oncorena, and Synthekine; payment or honoraria from BMS, Pfizer, Ipsen, Eisai, MSD Oncology, Merck Serono, AstraZeneca, Janssen-Cilag, AAA/Novartis, Apogepha, Ono Pharmaceutical, Astellas Pharma, and Amgen; ownership of stocks in MSD, BMS, AstraZeneca, Genmab, and Bicycle Therapeutics; travel support from Pfizer, AstraZeneca, Janssen, Merck Serono, and Ipsen; participation on a data safety monitoring board or advisory board for the European Organisation for Research and Treatment of Cancer; and a leadership role for Arbeitsgemeinschaft Internistische Onkologie of the German Cancer Society. MS reports patents not related to the manuscript as planned, issued, or pending. ME reports grants or contracts from Blue Earth Diagnostics and Bayer; royalties from POSLUMA (an FDA-approved radiohybrid PSMA-targeted agent for PET imaging of prostate cancer in adults); consulting fees for Blue Earth Diagnostics, Novartis/AAA, Bayer, Telix, RayzeBio, Point Biopharma, and Janssen Pharmaceuticals; payment or honoraria from Novartis/AAA, Eckert-Ziegler, ABX, and Janssen Pharmaceuticals; patent application as inventor of radiohybrid PSMA; a leadership role in the European Association of Nuclear Medicine committee meeting; ownership of stocks in Novartis, Telix, Bayer, and Point Biopharma; support for travel from Blue Earth Diagnostics and Terumo; and payment for expert testimony from Paraxel and Bioclinica. TT reports grants from DAAD Researcher Grant and speaker fees from ABX. CK reports ownership of stocks in Telix and Actinium Pharmaceuticals; grants or contracts from Novartis, Mariana Oncology, and Amgen; payment or honoraria from Novartis and Pfizer; and travel support from Bayer, Janssen, and Amgen. KH reports grants or contracts from Novartis and Sofie Biosciences; consulting fees from Advanced Accelerator Applications, a Novartis company, Amgen, AstraZeneca, Bain Capital, Bayer, Boston Scientific, Convergent, Curium, Debiopharm, EcoR1, Fusion, GE Healthcare, Immedica, Isotopen Technologien München, Janssen, Merck, Molecular Partners, NVision, POINT Biopharma, Pfizer, Radiopharm Theranostics, Rhine Pharma, Siemens Healthineers, Sofie Biosciences, Telix, Theragnostics, and ymabs; payment or honoraria for educational events from PeerVoice; support for American Society of Clinical Oncology participation from Janssen; participation on a data safety monitoring board or advisory board for Fusion and GE Healthcare; and ownership of stocks in Sofie Biosciences, Pharma15, NVision, Convergent, Aktis Oncology, and AdvanCell. KR reports consulting fees from AAA/Novartis, Bayer, ABX, ABX-CRO, UroTrials, and Pharmtrace; honoraria or payment from AAA/Novartis, Bayer, and ABX; and travel support from Bayer. PTM reports payment to their institution from Siemens Healthineers. BAH reports consulting fees from Lightpoint medical, Janssen, Bayer, ABX, Astellas, Merck, Amgen, MSD/Pfizer, Novartis, BMS, Onkowissen, POINT Biopharma, Ipsen, AstraZeneca, Lightpoint medical, Telix, and Accord Healthcare; travel support from AstraZeneca, BMS, Janssen, Bayer, and Ipsen; grants or contracts from Janssen, Deutsche Forschungsgesellschaft, Novartis, and BMS; royalties or licenses from Uromed; payment or honoraria from Janssen, Amgen, Astellas, and Monrol; participation on a data safety monitoring board or advisory board for Janssen and ABX; and a leadership role for Deutsche Gesellschaft für Urologie. WPF reports grants from SOFIE Biosciences; consulting fees from Janssen, Calyx, Bayer, Novartis, and Urotrials; payment or Honoraria from Janssen, Calyx, Bayer, Novartis, Urotrials, Amgen, ABX, Eczacıbaşı Monrol, GE Healthcare, and Telix. All other authors declare no competing interests., (Copyright © 2024 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license. Published by Elsevier Ltd.. All rights reserved.)

Interactive segmentation is a crucial research area in medical image analysis aiming to boost the efficiency of costly annotations by incorporating human feedback. This feedback takes the form of clicks, scribbles, or masks and allows for iterative refinement of the model output so as to efficiently guide the system towards the desired behavior. In recent years, deep learning-based approaches have propelled results to a new level causing a rapid growth in the field with 121 methods proposed in the medical imaging domain alone. In this review, we provide a structured overview of this emerging field featuring a comprehensive taxonomy, a systematic review of existing methods, and an in-depth analysis of current practices. Based on these contributions, we discuss the challenges and opportunities in the field. For instance, we find that there is a severe lack of comparison across methods which needs to be tackled by standardized baselines and benchmarks.

Purpose: Idiopathic pulmonary fibrosis (IPF) is the most common interstitial lung disease, with a median survival time of 2 to 5 years. The focus of this study is to establish a novel imaging biomarker., Materials and Methods: In this study, 79 patients (19% female) with a median age of 70 years were studied retrospectively. Fully automated body composition analysis (BCA) features (bone, muscle, total adipose tissue, intermuscular, and intramuscular adipose tissue) were combined into Sarcopenia, Fat, and Myosteatosis indices and compared between patients with a survival of more or less than 2 years. In addition, we divided the cohort at the median (high=≥ median, low=

Background and Objective: Cell segmentation in bright-field histological slides is a crucial topic in medical image analysis. Having access to accurate segmentation allows researchers to examine the relationship between cellular morphology and clinical observations. Unfortunately, most segmentation methods known today are limited to nuclei and cannot segment the cytoplasm., Methods: We present a new network architecture Cyto R-CNN that is able to accurately segment whole cells (with both the nucleus and the cytoplasm) in bright-field images. We also present a new dataset CytoNuke, consisting of multiple thousand manual annotations of head and neck squamous cell carcinoma cells. Utilizing this dataset, we compared the performance of Cyto R-CNN to other popular cell segmentation algorithms, including QuPath's built-in algorithm, StarDist, Cellpose and a multi-scale Attention Deeplabv3+. To evaluate segmentation performance, we calculated AP50, AP75 and measured 17 morphological and staining-related features for all detected cells. We compared these measurements to the gold standard of manual segmentation using the Kolmogorov-Smirnov test., Results: Cyto R-CNN achieved an AP50 of 58.65% and an AP75 of 11.56% in whole-cell segmentation, outperforming all other methods (QuPath 19.46/0.91%; StarDist 45.33/2.32%; Cellpose 31.85/5.61%, Deeplabv3+ 3.97/1.01%). Cell features derived from Cyto R-CNN showed the best agreement to the gold standard (D¯=0.15) outperforming QuPath (D¯=0.22), StarDist (D¯=0.25), Cellpose (D¯=0.23) and Deeplabv3+ (D¯=0.33)., Conclusion: Our newly proposed Cyto R-CNN architecture outperforms current algorithms in whole-cell segmentation while providing more reliable cell measurements than any other model. This could improve digital pathology workflows, potentially leading to improved diagnosis. Moreover, our published dataset can be used to develop further models in the future., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Author(s). Published by Elsevier B.V. All rights reserved.)

Standardized reporting of multiparametric prostate MRI (mpMRI) is widespread and follows international standards (Pi-RADS). However, quantitative measurements from mpMRI are not widely comparable. Although T2 mapping sequences can provide repeatable quantitative image measurements and extract reliable imaging biomarkers from mpMRI, they are often time-consuming. We therefore investigated the value of quantitative measurements on a highly accelerated T2 mapping sequence, in order to establish a threshold to differentiate benign from malignant lesions. For this purpose, we evaluated a novel, highly accelerated T2 mapping research sequence that enables high-resolution image acquisition with short acquisition times in everyday clinical practice. In this retrospective single-center study, we included 54 patients with clinically indicated MRI of the prostate and biopsy-confirmed carcinoma (n = 37) or exclusion of carcinoma (n = 17). All patients had received a standard of care biopsy of the prostate, results of which were used to confirm or exclude presence of malignant lesions. We used the linear mixed-effects model-fit by REML to determine the difference between mean values of cancerous tissue and healthy tissue. We found good differentiation between malignant lesions and normal appearing tissue in the peripheral zone based on the mean T2 value. Specifically, the mean T2 value for tissue without malignant lesions was (151.7 ms [95% CI: 146.9-156.5 ms] compared to 80.9 ms for malignant lesions [95% CI: 67.9-79.1 ms]; p < 0.001). Based on this assessment, a limit of 109.2 ms is suggested. Aditionally, a significant correlation was observed between T2 values of the peripheral zone and PI-RADS scores (p = 0.0194). However, no correlation was found between the Gleason Score and the T2 relaxation time. Using REML, we found a difference of -82.7 ms in mean values between cancerous tissue and healthy tissue. We established a cut-off-value of 109.2 ms to accurately differentiate between malignant and non-malignant prostate regions. The addition of T2 mapping sequences to routine imaging could benefit automated lesion detection and facilitate contrast-free multiparametric MRI of the prostate., (© 2024. The Author(s).)

3D data from high-resolution volumetric imaging is a central resource for diagnosis and treatment in modern medicine. While the fast development of AI enhances imaging and analysis, commonly used visualization methods lag far behind. Recent research used extended reality (XR) for perceiving 3D images with visual depth perception and touch but used restrictive haptic devices. While unrestricted touch benefits volumetric data examination, implementing natural haptic interaction with XR is challenging. The research question is whether a multisensory XR application with intuitive haptic interaction adds value and should be pursued. In a study, 24 experts for biomedical images in research and medicine explored 3D medical shapes with 3 applications: a multisensory virtual reality (VR) prototype using haptic gloves, a simple VR prototype using controllers, and a standard PC application. Results of standardized questionnaires showed no significant differences between all application types regarding usability and no significant difference between both VR applications regarding presence. Participants agreed to statements that VR visualizations provide better depth information, using the hands instead of controllers simplifies data exploration, the multisensory VR prototype allows intuitive data exploration, and it is beneficial over traditional data examination methods. While most participants mentioned manual interaction as the best aspect, they also found it the most improvable. We conclude that a multisensory XR application with improved manual interaction adds value for volumetric biomedical data examination. We will proceed with our open-source research project ISH3DE (Intuitive Stereoptic Haptic 3D Data Exploration) to serve medical education, therapeutic decisions, surgery preparations, or research data analysis., (© 2024. The Author(s).)

Aortic dissections (ADs) are serious conditions of the main artery of the human body, where a tear in the inner layer of the aortic wall leads to the formation of a new blood flow channel, named false lumen. ADs affecting the aorta distally to the left subclavian artery are classified as a Stanford type B aortic dissection (type B AD). This is linked to substantial morbidity and mortality, however, the course of the disease for the individual case is often unpredictable. Computed tomography angiography (CTA) is the gold standard for the diagnosis of type B AD. To advance the tools available for the analysis of CTA scans, we provide a CTA collection of 40 type B AD cases from clinical routine with corresponding expert segmentations of the true and false lumina. Segmented CTA scans might aid clinicians in decision making, especially if it is possible to fully automate the process. Therefore, the data collection is meant to be used to develop, train and test algorithms., (© 2024. The Author(s).)

Background & Aims: Body composition assessment (BCA) parameters have recently been identified as relevant prognostic factors for patients with hepatocellular carcinoma (HCC). Herein, we aimed to investigate the role of BCA parameters for prognosis prediction in patients with HCC undergoing transarterial chemoembolization (TACE)., Methods: This retrospective multicenter study included a total of 754 treatment-naïve patients with HCC who underwent TACE at six tertiary care centers between 2010-2020. Fully automated artificial intelligence-based quantitative 3D volumetry of abdominal cavity tissue composition was performed to assess skeletal muscle volume (SM), total adipose tissue (TAT), intra- and intermuscular adipose tissue, visceral adipose tissue, and subcutaneous adipose tissue (SAT) on pre-intervention computed tomography scans. BCA parameters were normalized to the slice number of the abdominal cavity. We assessed the influence of BCA parameters on median overall survival and performed multivariate analysis including established estimates of survival., Results: Univariate survival analysis revealed that impaired median overall survival was predicted by low SM ( p < 0.001), high TAT volume ( p  = 0.013), and high SAT volume ( p  = 0.006). In multivariate survival analysis, SM remained an independent prognostic factor ( p  = 0.039), while TAT and SAT volumes no longer showed predictive ability. This predictive role of SM was confirmed in a subgroup analysis of patients with BCLC stage B., Conclusions: SM is an independent prognostic factor for survival prediction. Thus, the integration of SM into novel scoring systems could potentially improve survival prediction and clinical decision-making. Fully automated approaches are needed to foster the implementation of this imaging biomarker into daily routine., Impact and Implications: Body composition assessment parameters, especially skeletal muscle volume, have been identified as relevant prognostic factors for many diseases and treatments. In this study, skeletal muscle volume has been identified as an independent prognostic factor for patients with hepatocellular carcinoma undergoing transarterial chemoembolization. Therefore, skeletal muscle volume as a metaparameter could play a role as an opportunistic biomarker in holistic patient assessment and be integrated into decision support systems. Workflow integration with artificial intelligence is essential for automated, quantitative body composition assessment, enabling broad availability in multidisciplinary case discussions., (© 2024 The Authors.)

Designing implants for large and complex cranial defects is a challenging task, even for professional designers. Current efforts on automating the design process focused mainly on convolutional neural networks (CNN), which have produced state-of-the-art results on reconstructing synthetic defects. However, existing CNN-based methods have been difficult to translate to clinical practice in cranioplasty, as their performance on large and complex cranial defects remains unsatisfactory. In this paper, we present a statistical shape model (SSM) built directly on the segmentation masks of the skulls represented as binary voxel occupancy grids and evaluate it on several cranial implant design datasets. Results show that, while CNN-based approaches outperform the SSM on synthetic defects, they are inferior to SSM when it comes to large, complex and real-world defects. Experienced neurosurgeons evaluate the implants generated by the SSM to be feasible for clinical use after minor manual corrections. Datasets and the SSM model are publicly available at https://github.com/Jianningli/ssm ., (© 2024. The Author(s).)

The Sparsely Annotated Region and Organ Segmentation (SAROS) dataset was created using data from The Cancer Imaging Archive (TCIA) to provide a large open-access CT dataset with high-quality annotations of body landmarks. In-house segmentation models were employed to generate annotation proposals on randomly selected cases from TCIA. The dataset includes 13 semantic body region labels (abdominal/thoracic cavity, bones, brain, breast implant, mediastinum, muscle, parotid/submandibular/thyroid glands, pericardium, spinal cord, subcutaneous tissue) and six body part labels (left/right arm/leg, head, torso). Case selection was based on the DICOM series description, gender, and imaging protocol, resulting in 882 patients (438 female) for a total of 900 CTs. Manual review and correction of proposals were conducted in a continuous quality control cycle. Only every fifth axial slice was annotated, yielding 20150 annotated slices from 28 data collections. For the reproducibility on downstream tasks, five cross-validation folds and a test set were pre-defined. The SAROS dataset serves as an open-access resource for training and evaluating novel segmentation models, covering various scanner vendors and diseases., (© 2024. The Author(s).)

Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in combination with large scale pre-training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F 1 -detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Selma Ugurel reports a relationship with Bristol Myers Squibb that includes: consulting or advisory, funding grants, speaking and lecture fees, and travel reimbursement. Selma Ugurel reports a relationship with Merck Serono that includes: consulting or advisory, funding grants, and speaking and lecture fees. Selma Ugurel reports a relationship with Merck Sharp & Dohme that includes: consulting or advisory, speaking and lecture fees, and travel reimbursement. Selma Ugurel reports a relationship with Novartis that includes: consulting or advisory, speaking and lecture fees, and travel reimbursement. Selma Ugurel reports a relationship with Almirall that includes: travel reimbursement. Selma Ugurel reports a relationship with IGEA Clinical Biophysics that includes: travel reimbursement. Selma Ugurel reports a relationship with Pierre Fabre that includes: travel reimbursement. Selma Ugurel reports a relationship with Sun Pharma that includes: travel reimbursement. Jens T. Siveke reports a relationship with AstraZeneca that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with Bayer AG that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with Boehringer Ingelheim that includes: consulting or advisory, funding grants, and speaking and lecture fees. Jens T. Siveke reports a relationship with Bristol-Myers Squibb that includes: consulting or advisory, funding grants, and speaking and lecture fees. Jens T. Siveke reports a relationship with Immunocore that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with MSD Sharp Dohme that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with Novartis that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with Roche Genentech that includes: consulting or advisory, funding grants, and speaking and lecture fees. Jens T. Siveke reports a relationship with Servier that includes: consulting or advisory and speaking and lecture fees. Jens T. Siveke reports a relationship with Abalos Therapeutics that includes: funding grants. Jens T. Siveke reports a relationship with Celgene that includes: funding grants. Jens T. Siveke reports a relationship with Eisbach Bio that includes: funding grants. Jens T. Siveke reports a relationship with Pharma15 that includes: board membership and equity or stocks., (Copyright © 2024 The Author(s). Published by Elsevier B.V. All rights reserved.)

With the massive proliferation of data-driven algorithms, such as deep learning-based approaches, the availability of high-quality data is of great interest. Volumetric data is very important in medicine, as it ranges from disease diagnoses to therapy monitoring. When the dataset is sufficient, models can be trained to help doctors with these tasks. Unfortunately, there are scenarios where large amounts of data is unavailable. For example, rare diseases and privacy issues can lead to restricted data availability. In non-medical fields, the high cost of obtaining enough high-quality data can also be a concern. A solution to these problems can be the generation of realistic synthetic data using Generative Adversarial Networks (GANs). The existence of these mechanisms is a good asset, especially in healthcare, as the data must be of good quality, realistic, and without privacy issues. Therefore, most of the publications on volumetric GANs are within the medical domain. In this review, we provide a summary of works that generate realistic volumetric synthetic data using GANs. We therefore outline GAN-based methods in these areas with common architectures, loss functions and evaluation metrics, including their advantages and disadvantages. We present a novel taxonomy, evaluations, challenges, and research opportunities to provide a holistic overview of the current state of volumetric GANs., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Authors. Published by Elsevier B.V. All rights reserved.)

The recent release of ChatGPT, a chat bot research project/product of natural language processing (NLP) by OpenAI, stirs up a sensation among both the general public and medical professionals, amassing a phenomenally large user base in a short time. This is a typical example of the 'productization' of cutting-edge technologies, which allows the general public without a technical background to gain firsthand experience in artificial intelligence (AI), similar to the AI hype created by AlphaGo (DeepMind Technologies, UK) and self-driving cars (Google, Tesla, etc.). However, it is crucial, especially for healthcare researchers, to remain prudent amidst the hype. This work provides a systematic review of existing publications on the use of ChatGPT in healthcare, elucidating the 'status quo' of ChatGPT in medical applications, for general readers, healthcare professionals as well as NLP scientists. The large biomedical literature database PubMed is used to retrieve published works on this topic using the keyword 'ChatGPT'. An inclusion criterion and a taxonomy are further proposed to filter the search results and categorize the selected publications, respectively. It is found through the review that the current release of ChatGPT has achieved only moderate or 'passing' performance in a variety of tests, and is unreliable for actual clinical deployment, since it is not intended for clinical applications by design. We conclude that specialized NLP models trained on (bio)medical datasets still represent the right direction to pursue for critical clinical applications., Competing Interests: Declaration of Competing Interest The authors declare no conflict of interests., (Copyright © 2024 The Author(s). Published by Elsevier B.V. All rights reserved.)

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases., (© 2024. Springer Nature America, Inc.)