Search

In the 2023 phase of the international collaborative DECOVALEX modeling project, Task E focused on understanding thermal, hydrological, and mechanical (THM) processes related to predicting brine migration in the excavation damaged zone around a heated excavation in salt. Salt is attractive as a disposal medium for radioactive waste because it is self-healing and is essentially impermeable and non-porous in the far field. Investigation of the short-term, near-field behavior is important for radioactive waste disposal because this early period strongly controls the amount of inflowing brine. Brine leads to corrosion of waste forms and waste packages, and possible dissolution of radionuclides with brine transport being a potential transport vector to the accessible environment.The Task was divided into steps. Step 0 included matching unheated brine inflow data from boreholes at the Waste Isolation Pilot Plant (WIPP) and matching temperature observations during a Brine Availability Test in Salt (BATS) heater test. Step 1 included validation of models against a thermo-poroelastic analytical solution, and two-phase flow around an excavation. Finally, Step 2 required all the individual components covered in steps 0 and 1 to come together to match observed brine inflow behavior during the same BATS heater test.There were a range of approaches from the teams, from mechanistic to prescriptive. Given the uncertainties in the problem, some teams used one- or two-dimensional models of the processes, while other teams included more geometrical complexity in three-dimensional models. Task E was a learning experience for the teams involved, and feedback from the modeling teams has led to changes in follow-on BATS experiments at WIPP. The primary Task E lessons learned were the impact of hydrologic initialization methods (wetting up vs. drying down), the difference between confined and unconfined thermal expansion, and the large changes in permeability associated with heating and cooling.

Changes to components, known as engineering changes (ECs), rarely occur on their own. In fact, in complex assembly systems, most ECs are introduced in batches to ensure that changed components match. As a result, to implement ECs optimally, multiple component's stock must be considered until the change is executed on the EC effectivity date. This problem is known as the EC effectivity date optimisation problem, a variation of the general inventory control problem with deterministic and dynamic demand. As optimisation and monitoring of this problem is computationally expensive, research has suggested to investigate whether metaheuristics can provide adequate support. To fill this research gap, we present the results of a benchmark on basic metaheuristics for EC effectivity date optimisation. To do so, we have compared five common metaheuristics in their basic form (Ant Colony Optimisation, Genetic Algorithm, Particle Swarm Optimisation, Tabu Search, and Simulated Annealing) on a real-world test set. Of the tested algorithms the Genetic Algorithm identified most best solutions and returned good average results for the test cases. However, as its reliability was comparatively low, our research suggests a sequential application of the Genetic Algorithm and Tabu Search. [ABSTRACT FROM AUTHOR]

The success of startups is intricately linked to the ecosystem they operate within. It is thus critical to unpack the concept of the startup ecosystem, a task that demands scholarly and professional insight. There is a keen interest in dissecting the dimensions and components of these ecosystems, with a particular focus on those known to be startup-friendly. The motivation for this scrutiny comes from patterns showing that high-performing startups are predominantly clustered in specific global regions—a hardly coincidental phenomenon. This paper aims to explore these thriving ecosystems, drawing parallels to the Italian context, especially that of the south. R.Q.: Can comparing Italian startups with Silicon Valley reveal growth best practices? Utilizing a combination of primary and secondary interviews, this study benchmarks the southern Italian startup ecosystems against the renowned Silicon Valley. Despite Silicon Valley's occasional tumultuous phases, it maintains its stature as a benchmark of excellence. This comparative analysis seeks to extract best practices results from Silicon Valley, which, despite its challenges, continues to be a significant point of reference for emerging startup environments. [ABSTRACT FROM AUTHOR]

Molecular geometry and harmonic frequency calculations are essential in thermochemical computations, with density functional theory (DFT) being widely employed for vibrational frequency predictions due to its efficiency and accuracy. In this study, we assessed the precision of 28 Minnesota based functionals with three different basis sets using the VIBFREQ1295 dataset. Scaling factors are necessary for predicting fundamental frequencies, global scaling factors were fitted by using F38/10 and VIBFREQ1295 datasets. The superior performing functionals were then fitted based on vibrational frequency ranges to obtain frequency‐range‐specific scaling factors. We observed consistent outlier across various model chemistries in vibrational frequency predictions, with alternative scaling factors showing minimal impact on reducing outlier occurrences. Besides, large basis sets are not indispensably required for fundamental frequency predictions. M06‐L, revM06‐L, SOGGA11‐X, PW6B95‐D3(BJ), CF22D, and M06‐2X consistently exhibit excellent performance across the three basis sets. When using frequency‐range‐specific scaling factors, the root mean squard errors (RMSEs) and median absolute errors (MedAEs) of almost all the selected functionals were reduced. PW6B95‐D3(BJ), CF22D, and MN12‐SX exhibited the lowest RMSEs. Comparisons were also done for different data classifications; the dataset was classified by the elements of the molecules, vibrational frequency intervals, and vibrational modes. [ABSTRACT FROM AUTHOR]

The accurate identification of protein–ligand binding sites is of critical importance in understanding and modulating protein function. Accordingly, ligand binding site prediction has remained a research focus for over three decades with over 50 methods developed and a change of paradigm from geometry-based to machine learning. In this work, we collate 13 ligand binding site predictors, spanning 30 years, focusing on the latest machine learning-based methods such as VN-EGNN, IF-SitePred, GrASP, PUResNet, and DeepPocket and compare them to the established P2Rank, PRANK and fpocket and earlier methods like PocketFinder, Ligsite and Surfnet. We benchmark the methods against the human subset of our new curated reference dataset, LIGYSIS. LIGYSIS is a comprehensive protein–ligand complex dataset comprising 30,000 proteins with bound ligands which aggregates biologically relevant unique protein–ligand interfaces across biological units of multiple structures from the same protein. LIGYSIS is an improvement for testing methods over earlier datasets like sc-PDB, PDBbind, binding MOAD, COACH420 and HOLO4K which either include 1:1 protein–ligand complexes or consider asymmetric units. Re-scoring of fpocket predictions by PRANK and DeepPocket display the highest recall (60%) whilst IF-SitePred presents the lowest recall (39%). We demonstrate the detrimental effect that redundant prediction of binding sites has on performance as well as the beneficial impact of stronger pocket scoring schemes, with improvements up to 14% in recall (IF-SitePred) and 30% in precision (Surfnet). Finally, we propose top-N+2 recall as the universal benchmark metric for ligand binding site prediction and urge authors to share not only the source code of their methods, but also of their benchmark. Scientific contributions This study conducts the largest benchmark of ligand binding site prediction methods to date, comparing 13 original methods and 15 variants using 10 informative metrics. The LIGYSIS dataset is introduced, which aggregates biologically relevant protein–ligand interfaces across multiple structures of the same protein. The study highlights the detrimental effect of redundant binding site prediction and demonstrates significant improvement in recall and precision through stronger scoring schemes. Finally, top-N+2 recall is proposed as a universal benchmark metric for ligand binding site prediction, with a recommendation for open-source sharing of both methods and benchmarks. [ABSTRACT FROM AUTHOR]

For the welfare assessment on commercial broiler and turkey farms, not only the recording of animal-based indicators but also the evaluation of the resulting prevalence or rate is essential. Two evaluation methods were compared using data on welfare indicators collected over 1 year from 11 broiler and 11 turkey farms in Germany: the application of normative values from an evaluation framework and the calculation of a benchmark. The evaluation framework had recently been developed in a participatory process that provided an evaluation with target and alarm values. The target range was predominantly based on ethical considerations, while the alarm range was aligned with the current status quo from farm investigations. The 25th percentile and the 75th percentile of the benchmarking were similarly classified as target and alarm. When applying the evaluation framework across all indicators and flocks, 30.6% of broiler flocks were in the target range, while 41.4% were in the alarm range, mostly for indicators such as footpad dermatitis, weight uniformity, and mortality. For turkeys at week 5 or at the end of the fattening period, 51.6% and 32.9%, respectively, were in the target range and 12.3% and 14.4% were in the alarm range. Most alarm classifications were related to footpad dermatitis, low-weight uniformity, plumage damage, and skin injuries. The application of normative values led to a significantly worse average welfare rank over all indicators and flocks for broilers compared to the benchmark, while no difference was observed for turkeys. The farm selection process may have favored turkey farms with better management practices, resulting in a more rigorous benchmark than in broilers. In addition, the farm data used to set the normative values had indicated a poorer status quo in turkeys for certain indicators, resulting in less stringent limits for the alarm range. This highlights the challenges associated with both evaluation methods: normative values are affected by the process and criteria used to set them, while benchmarks are affected by the reference population, which calls for large databases with regular updates. Also, for normative values, developments in the sector and the latest scientific evidence should be used for recurrent validation. [ABSTRACT FROM AUTHOR]

ABSTRACT Anomaly detection, vital for identifying deviations from normative data patterns, is particularly crucial in sensor‐driven real‐world applications, which predominantly involve temporal data in the form of time series. Traditional evaluation of anomaly detection methods has relied on public benchmark datasets. Yet, recent revelations have uncovered inherent flaws and inadequacies in these datasets, casting doubt on the perceived progress in the field. To address this challenge, the UCR Time Series Anomaly Archive has been recently proposed—a meticulously curated database comprising 250 time series—designed to provide a robust and error‐free benchmark for anomaly detection research. This paper comprehensively evaluates state‐of‐the‐art anomaly detection techniques using the UCR Time Series Anomaly Archive. Our findings demonstrate the efficacy of current methods in accurately detecting anomalies across an important portion of datasets without additional optimization, underscoring the archive's utility as a foundational baseline for future research and development in anomaly detection methodologies. [ABSTRACT FROM AUTHOR]

Background: Shapley values have been used extensively in machine learning, not only to explain black box machine learning models, but among other tasks, also to conduct model debugging, sensitivity and fairness analyses and to select important features for robust modelling and for further follow-up analyses. Shapley values satisfy certain axioms that promote fairness in distributing contributions of features toward prediction or reducing error, after accounting for non-linear relationships and interactions when complex machine learning models are employed. Recently, feature selection methods using predictive Shapley values and p-values have been introduced, including powershap. Methods: We present a novel feature selection method, LLpowershap, that takes forward these recent advances by employing loss-based Shapley values to identify informative features with minimal noise among the selected sets of features. We also enhance the calculation of p-values and power to identify informative features and to estimate number of iterations of model development and testing. Results: Our simulation results show that LLpowershap not only identifies higher number of informative features but outputs fewer noise features compared to other state-of-the-art feature selection methods. Benchmarking results on four real-world datasets demonstrate higher or comparable predictive performance of LLpowershap compared to other Shapley based wrapper methods, or filter methods. LLpowershap is also ranked the best in mean ranking among the seven feature selection methods tested on the benchmark datasets. Conclusion: Our results demonstrate that LLpowershap is a viable wrapper feature selection method that can be used for feature selection in large biomedical datasets and other settings. [ABSTRACT FROM AUTHOR]

This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed n = 300 data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8)—a statistically significant difference (p = 2.482 × 10 - 10 ). Prompt engineering significantly improved scores for both GPT-4 (p = 1.661 × 10 - 4 ) and GPT-3.5 (p = 4.967 × 10 - 9 ). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from ‘Definitely AI’ to ‘Definitely Human’. They accurately identified the authorship, with 92.1% of the work categorized as ‘Definitely Human’ being human-authored. Simplifying this to a binary ‘AI’ or ‘Human’ categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students’ work, it often remains detectable by human evaluators. [ABSTRACT FROM AUTHOR]

Databases are an essential component of life in modern society. The database has become an online Transaction Processing (OLTP) operational support company. This study discusses the storage configuration in a virtual server database for OLTP Database with a benchmark against three storage configurations for four databases: SQL Server, MySQL, PostgreSQL, and MariaDB for Windows Server virtualization. Benchmarks using TPC-C to get the best performance from all three configurations have been tested. The results of this study show that the storage configuration of HDD as a virtual disk drive OS and NVME as a virtual disk drive database get better performance as online transaction processing (OLTP) compared to HDD as a virtual disk drive OS and HDD or SSD as a virtual disk drive database. Using the TPC-C benchmark, the study tested the performance of four databases (SQL Server, MySQL, PostgreSQL, and MariaDB) in various virtual storage configurations. The results of 36 experiments involving different numbers of virtual users (10, 50, and 100) showed that a hybrid configuration that used HDDs for the operating system and NVME for database storage provided the best performance. On SQL Server, SSD configurations show better results for smaller user loads. The results suggest that the combination of HDD and NVME is the optimal configuration for most databases in OLTP tasks, and it provides useful guidance in efficient storage investment decision-making. Further research is expected to be able to conduct tests. [ABSTRACT FROM AUTHOR]

Computational models in hydro-environmental engineering are diverse in their background formulation and span from two-dimensional depth-averaged shallow water models, to complex fully three-dimensional turbulence models resolving large-eddy simulation with surface capturing techniques, and to Lagrangian particle-based methods. This paper presents a first-of-its-kind comparison of six different computational hydraulics fluid dynamics models, namely Iber+, HO-SWM, GBVC, OpenFOAM (RANS), Hydro3D (LES) and DualSPHysics (SPH), in the prediction of mean velocities and free-surface dynamics in two benchmarks involving open-channel flows with symmetric lateral cavities. Results show that shallow-water models capture relatively well the main large-scale coherent structures of the in-cavity flow, with wider shear layers compared to three-dimensional models, and higher velocities in the main channel. Three-dimensional RANS, LES and SPH yield improved predictions of mean velocities compared with experimental data. Computational cost has been quantified for all models with a logarithmic growth when increasing model complexity. The transverse standing wave is captured by most models, with the shallow-water ones matching the theoretical value, while the three-dimensional models overestimate it slightly. [ABSTRACT FROM AUTHOR]

Hepatitis C virus (HCV) is an important cause of chronic liver disease. Among at-risk populations, access to care is challenging. The French Ministry of Health has supported a seek-and-treat pilot intervention aiming at micro-elimination in Perpignan, France, to inform scale-up of elimination efforts across the whole territory. University College Dublin (UCD) led a successful EU funded project, called HepCare, focusing on the micro-elimination of HCV. UCD was contracted to evaluate and benchmark the Perpignan results against results from HepCare. Using mixed-method approaches including qualitative interviews with patients, a focus group with healthcare professionals, and quantitative analyses of the cascade of care against results obtained at other European sites, we analyse the acceptability, reproducibility, replicability, and effectiveness of the Perpignan intervention. A total of 960 participants were recruited in the Perpignan area. HCV antibody test results were obtained for 928 (96.6%), of which 150 (15.6%) were antibody-positive. Of the antibody-positive participants, 68 (45.3%) tested positive for HCV-RNA, 141 (94%) were linked to care, and of the HCV-RNA-positive participants, 60 (88%) started treatment. Of those who underwent treatment, 34 (56.7%) completed treatment and achieved a sustained viral response (SVR) at dataset closure, 18 (30%) were still in treatment, 5 (8.3%) defaulted from treatment, and 3 (5%) had a virologic failure or died. The intervention in Perpignan was acceptable to patients, but had limitations in effectiveness, as shown in comparisons with HepCare results. To engage harder-to-reach cohorts in France, future models of care in the territory should incorporate peer support. [ABSTRACT FROM AUTHOR]

AI systems for decision‐making have become increasingly popular in several areas. However, it is possible to identify biased decisions in many applications, which have become a concern for the computer science, artificial intelligence, and law communities. Therefore, researchers are proposing solutions to mitigate bias and discrimination among decision‐makers. Some explored strategies are based on GANs to generate fair data. Others are based on adversarial learning to achieve fairness by encoding fairness constraints through an adversarial model. Moreover, it is usual for each proposal to assess its model with a specific metric, making comparing current approaches a complex task. Therefore, this work proposes a systematical benchmark procedure to assess the fair machine learning models. The proposed procedure comprises a fairness‐utility trade‐off metric (FU‐score$$ FU\hbox{-} score $$), the utility and fairness metrics to compose this assessment, the used datasets and preparation, and the statistical test. A previous work presents some of these definitions. The present work enriches the procedure by increasing the applied datasets and statistical guarantees when comparing the models' results. We performed this benchmark evaluation for the non‐generative adversarial models, analyzing the literature models from the same metric perspective. This assessment could not indicate a single model which better performs for all datasets. However, we built an understanding of how each model performs on each dataset with statistical confidence. [ABSTRACT FROM AUTHOR]

Large Language Models (LLMs) excel in fields such as natural language understanding, generation, complex reasoning, and biomedicine. With advancements in materials science, traditional manual annotation methods for phase diagrams have become inadequate due to their time-consuming nature and limitations in updating thermodynamic databases. To overcome these challenges, we propose a framework based on instruction tuning, utilizing LLMs for automated end-to-end annotation of phase diagrams. High-quality phase diagram images and expert descriptions are collected from handbooks and then preprocessed to correct errors, remove redundancies, and enhance information. These preprocessed data form a golden dataset, from which a subset are used to train LLMs through hierarchical sampling. The fine-tuned LLM is then tested for automated phase diagram annotation. Results show that the fine-tuned model achieves a cosine similarity of 0.8737, improving phase diagram comprehension accuracy by 7% compared to untuned LLMs. To the best of our knowledge, this is the first paper to propose using LLMs for the automated annotation of phase diagrams, replacing traditional manual annotation methods and significantly enhancing efficiency and accuracy. [ABSTRACT FROM AUTHOR]

This paper proposes a new Dynamic hAnd gesTurE (DATE) dataset for dynamic hand gestures. The DATE dataset contains 13,500 videos of 22 different subjects. The subjects wear different clothes, have different backgrounds, and are filmed from various camera angles. Two different benchmarks for our self-built DATE dataset are also proposed. The first one is the high accuracy approach, while the second benchmark is the lightweight approach. The operation of our benchmarks has two phases. In the first phase, videos are preprocessed with detection or segmentation tasks. Then, the processed data are classified by customized cutting-edge deep learning models in the second phase. Experimental results showed that our benchmarks obtained high accuracies in both the self-build dataset and a publicly recognized dataset. [ABSTRACT FROM AUTHOR]

This work proposes multiple variants of the Multiverse Optimizer Algorithm (MVO) with the use of Chaotic Maps and Fuzzy Logic. These variants come from the Fuzzy-Chaotic Multiverse Optimizer Algorithm (FCMVO), and the purpose is to analyze the behavior of the algorithm in some benchmark mathematical functions, to compare in which cases can perform best between the variants presented. The use of chaotic maps resides from some of the most used in the literature on optimization algorithms, and the main use is to adjust some of the parameters of behavior in the MVO algorithm; and other implementation is the use of Fuzzy Logic for dynamical adaptation of two parameters in the algorithm. In this study, we are presenting an extended comparison between the variants obtained from diverse chaotic maps, that in previous studies, we delimited to six variants which we called Elitist FCMVO. This comparison is done by evaluating the resulting variants from a set of three chaotic maps and comparing also with two different fuzzy inference systems (Mamdani and Sugeno types). The main objective is to present the obtained variants and compare which are the best chaotic maps in the FCMVO algorithm, to improve further in other study cases. [ABSTRACT FROM AUTHOR]

Benchmarking in healthcare, particularly in the context of complex surgical procedures like pancreatic surgery, plays a pivotal role in comparing and evaluating the quality of care provided to patients. There is a growing body of evidence validating existent metrics and introducing new ones in the pursuit of safety and excellence in pancreatic surgery. A systematic review adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines was conducted on metric development and validation across multiple databases, including PUBMED Medline, Scopus, and Web of Science, until February 2024. The extracted data were categorized into three domains according to the Donabedian model: structure, process, and outcomes. Thirty-four studies were deemed eligible for inclusion in this review. Among these articles, 20 contributed to metric development, while 14 studies validated them. A total of 234 metrics were identified across the 34 studies, of which 185 were included in the analysis. Thirty-three of these metrics were relative to structure, 79 to processes, and 73 to outcomes. The distribution of metric domains across the included studies revealed that structure, process, and outcome domains were reported in 12, 26, and 26 studies, respectively. In conclusion, this systematic review underscores the heterogeneity in metric development methodologies and the varying degrees of consensus among different quality indicators, despite the growing interest in benchmarking in pancreatic surgery. This review aims to inform future research efforts and contribute to the ongoing pursuit of excellence in pancreatic surgical care. [ABSTRACT FROM AUTHOR]

Progress in automatic sign language recognition and translation has been hindered by the scarcity of datasets available for the training of machine learning algorithms, a challenge that is even more acute for languages with smaller signing communities, such as Spanish. In this paper, we introduce a dataset of 300 isolated signs in Spanish Sign Language, collected online via a web application with contributions from 124 participants, resulting in a total of 8000 instances. This dataset, which is openly available, includes keypoints extracted using MediaPipe Holistic. The goal of this paper is to describe the construction and characteristics of the dataset and to provide a baseline classification method using a spatial–temporal graph convolutional network (ST-GCN) model, encouraging the scientific community to improve upon it. The experimental section offers a comparative analysis of the method's performance on the new dataset, as well as on two other well-known datasets. The dataset, code, and web app used for data collection are freely available, and the web app can also be used to test classifier performance on-line in real-time. [ABSTRACT FROM AUTHOR]

In the field of computer vision, the detection of road potholes at night represents a critical challenge in enhancing the safety of intelligent transportation systems. Ensuring road safety is of paramount importance, particularly in promptly repairing pothole issues. These abrupt road depressions can easily lead to vehicle skidding, loss of control, and even traffic accidents, especially when water has pooled in or submerged the potholes. Therefore, the detection and recognition of road potholes can significantly reduce vehicle damage and the incidence of safety incidents. However, research on road pothole detection lacks high-quality annotated datasets, particularly under low-light conditions at night. To address this issue, this study introduces a novel Nighttime Pothole Dataset (NPD), independently collected and comprising 3831 images that capture diverse scene variations. The construction of this dataset aims to counteract the insufficiency of existing data resources and strives to provide a richer and more realistic benchmark. Additionally, we develop a baseline detector, termed WT-YOLOv8, for the proposed dataset, based on YOLOv8. We also evaluate the performance of the improved WT-YOLOv8 method and eight state-of-the-art object detection methods on the NPD and the COCO dataset. The experimental results on the NPD demonstrate that WT-YOLOv8 achieves a 2.3% improvement in mean Average Precision (mAP) over YOLOv8. In terms of the key metrics—AP@0.5 and AP@0.75—it shows enhancements of 1.5% and 2.8%, respectively, compared to YOLOv8. The experimental results provide valuable insights into each method's strengths and weaknesses under low-light conditions. This analysis highlights the importance of a specialized dataset for nighttime pothole detection and shows variations in accuracy and robustness among methods, emphasizing the need for improved nighttime pothole detection techniques. The introduction of the NPD is expected to stimulate further research, encouraging the development of advanced algorithms for nighttime pothole detection, ultimately leading to more flexible and reliable road maintenance and road safety. [ABSTRACT FROM AUTHOR]

This study highlights the necessity for efficient database management in continuous authentication systems, which rely on large-scale behavioral biometric data such as keystroke patterns. A benchmarking framework was developed to evaluate the PostgreSQL and MySQL databases, minimizing repetitive coding through configurable functions and variables. The methodology involved experiments assessing select and insert queries under primary and complex conditions, simulating real-world scenarios. Our quantified results show PostgreSQL's superior performance in select operations. In primary tests, PostgreSQL's execution time for 1 million records ranged from 0.6 ms to 0.8 ms, while MySQL's ranged from 9 ms to 12 ms, indicating that PostgreSQL is about 13 times faster. For select queries with a where clause, PostgreSQL required 0.09 ms to 0.13 ms compared to MySQL's 0.9 ms to 1 ms, making it roughly 9 times more efficient. Insert operations were similar, with PostgreSQL at 0.0007 ms to 0.0014 ms and MySQL at 0.0010 ms to 0.0030 ms. In complex experiments with simultaneous operations, PostgreSQL maintained stable performance (0.7 ms to 0.9 ms for select queries during inserts), while MySQL's performance degraded significantly (7 ms to 13 ms). These findings underscore PostgreSQL's suitability for environments requiring low data latency and robust concurrent processing capabilities, making it ideal for continuous authentication systems. [ABSTRACT FROM AUTHOR]

Building on updated estimates of the social cost of carbon obtained from the most recent literature, this article proposes a social cost of carbon-based benchmark for carbon pricing to drive world countries' carbon pricing policies up to 2050, consistent with the Paris Agreement targets. By using a dataset on net effective carbon rates developed by the Organisation for Economic Co-operation and Development (OECD), we firstly compare both explicit and implicit carbon pricing in 2021 in 71 OECD and non-OECD countries with the social cost of carbon benchmark for 2021 and calculate the degree of internalization of the social cost of carbon averagely related to their carbon pricing instruments. We find that there is a serious gap in current climate policies, which are far from creating optimal pricing conditions to reduce global emissions to levels needed by the Paris Agreement. The economic and distributional feasibility of a full implementation of the carbon pricing benchmark is tested in the same set of countries using two indicators, which are calculated for 2025, 2030, 2040 and 2050. Since the test results are income-regressive among income country groups, benchmark implementation by countries within the cooperative approaches of Paris Agreement art. 6 should be accompanied by the creation of an international cooperative fund aimed to recycle at least part of the revenues collected by high-income countries to compensate affected population in lower-income countries. [ABSTRACT FROM AUTHOR]

Background It is unclear whether targeted monitoring of acute adrenal insufficiency (AI) related adverse events (AE) such as sick day episodes (SDEs) and hospitalization rate in congenital adrenal hyperplasia (CAH) is associated with a change in the occurrence of these events. Aim Study temporal trends of AI related AE in the I-CAH Registry. Methods In 2022, data on the occurrence of AI-related AE in children aged <18 years with 21-hydroxylase deficiency CAH were compared to data collected in 2019. Results In 2022, a total of 513 children from 38 centers in 21 countries with a median of 8 children (range 1-58) per center had 2470 visits evaluated over a 3-year period (2019-2022). The median SDE per patient year in 2022 was 0 (0-2.5) compared to 0.3 (0-6) in 2019 (P =.01). Despite adjustment for age, CAH phenotype and duration of study period, a difference in SDE rate was still apparent between the 2 cohorts. Of the 38 centers in the 2022 cohort, 21 had also participated in 2019 and a reduction in SDE rate was noted in 13 (62%), an increase was noted in 3 (14%), and in 5 (24%) the rate remained the same. Of the 474 SDEs reported in the 2022 cohort, 103 (22%) led to hospitalization compared to 299 of 1099 SDEs (27%) in the 2019 cohort (P =.02). Conclusion The I-CAH Registry can be used for targeted monitoring of important clinical benchmarks in CAH. However, changes in reported benchmarks need careful interpretation and longer-term monitoring. [ABSTRACT FROM AUTHOR]

Removing photobombing elements from images is a challenging task that requires sophisticated image inpainting techniques. Despite the availability of various methods, their effectiveness depends on the complexity of the image and the nature of the distracting element. To address this issue, we conducted a benchmark study to evaluate 10 state-of-the-art photobombing removal methods on a dataset of over 300 images. Our study focused on identifying the most effective image inpainting techniques for removing unwanted regions from images. We annotated the photobombed regions that require removal and evaluated the performance of each method using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Fréchet inception distance (FID). The results show that image inpainting techniques can effectively remove photobombing elements, but more robust and accurate methods are needed to handle various image complexities. Our benchmarking study provides a valuable resource for researchers and practitioners to select the most suitable method for their specific photobombing removal task. [ABSTRACT FROM AUTHOR]

Background: Predictive modeling based on multi-omics data, which incorporates several types of omics data for the same patients, has shown potential to outperform single-omics predictive modeling. Most research in this domain focuses on incorporating numerous data types, despite the complexity and cost of acquiring them. The prevailing assumption is that increasing the number of data types necessarily improves predictive performance. However, the integration of less informative or redundant data types could potentially hinder this performance. Therefore, identifying the most effective combinations of omics data types that enhance predictive performance is critical for cost-effective and accurate predictions. Methods: In this study, we systematically evaluated the predictive performance of all 31 possible combinations including at least one of five genomic data types (mRNA, miRNA, methylation, DNAseq, and copy number variation) using 14 cancer datasets with right-censored survival outcomes, publicly available from the TCGA database. We employed various prediction methods and up-weighted clinical data in every model to leverage their predictive importance. Harrell's C-index and the integrated Brier Score were used as performance measures. To assess the robustness of our findings, we performed a bootstrap analysis at the level of the included datasets. Statistical testing was conducted for key results, limiting the number of tests to ensure a low risk of false positives. Results: Contrary to expectations, we found that using only mRNA data or a combination of mRNA and miRNA data was sufficient for most cancer types. For some cancer types, the additional inclusion of methylation data led to improved prediction results. Far from enhancing performance, the introduction of more data types most often resulted in a decline in performance, which varied between the two performance measures. Conclusions: Our findings challenge the prevailing notion that combining multiple omics data types in multi-omics survival prediction improves predictive performance. Thus, the widespread approach in multi-omics prediction of incorporating as many data types as possible should be reconsidered to avoid suboptimal prediction results and unnecessary expenditure. [ABSTRACT FROM AUTHOR]

Die Leistungsfähigkeit der Aufbereitungseinheit für Medizinprodukte (AEMP) ist essenziell für die Gesundheitsversorgung. Herausforderungen wie Fachkräftemangel und zunehmend komplexere Medizinprodukte machen kontinuierliche Prozessverbesserungen in der AEMP erforderlich. Ein datenbasierter Vergleich zwischen AEMPs kann Schwachstellen aufdecken, durch Handlungsempfehlungen Zeit und Kosten bei der Prozessverbesserung sparen und somit die Patientenversorgung verbessern. Der Lehrstuhl für Medizintechnik der RWTH Aachen entwickelt mit der DGSV e.V. eine AEMP-Benchmark zur Erfassung und Analyse von AEMP-Prozessen in Deutschland. Aufbauend auf in Vorarbeiten erhobenen Anforderungen und potenziellen Vergleichsgrößen wurde 2023 mittels einer Online-Umfrage eine Pilot-Datenerhebung für das Benchmarking durchgeführt. Die 48 erhobenen Datensätze von unterschiedlichen AEMP wurden zunächst bereinigt. Die Vergleichsgrößen wurden in Schlüssel-Leistungsindikatoren (engl.: Key Performance Indicators, KPI) und Leistungsbestimmende Faktoren (engl.: Performance Shaping Factors, PSF) unterteilt. Anschließend wurde eine statistische Analyse des Einflusses der PSFs auf die KPIs mit einem Signifikanzniveau von 0,05 durchgeführt. Die KPIs zeigten eine weite Streuung bei Qualität und Sicherheit. Für die Qualität waren das Sterilisator-Nutzvolumen und die gemeinsame Auswertung von Reklamationen mit den Kunden besonders starke Einflüsse. Für die Sicherheit der Beschäftigten waren vorhandene Einarbeitungskonzepte und häufige externe Fortbildungen besonders positive Einflüsse. Weitere signifikante Faktoren wurden identifiziert und in den praktischen Kontext eingeordnet. Insgesamt wurde damit das Ziel des Benchmarkings erreicht, Unterschiede in der Leistungsfähigkeit aufzudecken und PSF mit einem besonders starken Einfluss zu identifizieren. Eine Weiterführung des Benchmarkings erscheint sinnvoll, da zeitliche Veränderungen sichtbar gemacht werden können und durch eine größere Datenbasis eine Verbesserung der statistischen Auswertung und die Identifikation weiterer signifikanter PSF möglich ist. [ABSTRACT FROM AUTHOR]

Simple Summary: Validation of plant pathogens' diagnostic assays requires positive and negative controls. High Throughput Sequencing (HTS) has shown capabilities for detecting multiple pathogens simultaneously. However, accurate pathogen identification from HTS data depends on the bioinformatic pipeline used. There is currently no consensus on the best pipeline, leading to inconsistent results. To address this, standardized artificial HTS datasets are proposed as benchmarks to evaluate the performance of bioinformatic pipelines used in detection scenarios using HTS. These datasets will aid in resolving challenges like unknown sensitivity and specificity of the bioinformatic tool, contributing to advancing plant pathogen detection using HTS. The validation of diagnostic assays in plant pathogen detection is a critical area of research. It requires the use of both negative and positive controls containing a known quantity of the target pathogen, which are crucial elements when calculating analytical sensitivity and specificity, among other diagnostic performance metrics. High Throughput Sequencing (HTS) is a method that allows the simultaneous detection of a theoretically unlimited number of plant pathogens. However, accurately identifying the pathogen from HTS data is directly related to the bioinformatic pipeline utilized and its effectiveness at correctly assigning reads to their associated taxa. To this day, there is no consensus about the pipeline that should be used to detect the pathogens in HTS data, and results often undergo review and scientific evaluation. It is, therefore, imperative to establish HTS resources tailored for evaluating the performance of bioinformatic pipelines utilized in plant pathogen detection. Standardized artificial HTS datasets can be used as a benchmark by allowing users to test their pipelines for various pathogen infection scenarios, some of the most prevalent being multiple infections, low titer pathogens, mutations, and new strains, among others. Having these artificial HTS datasets in the hands of HTS diagnostic assay validators can help resolve challenges encountered when implementing bioinformatics pipelines for routine pathogen detection. Offering these purely artificial HTS datasets as benchmarking tools will significantly advance research on plant pathogen detection using HTS and enable a more robust and standardized evaluation of the bioinformatic methods, thereby enhancing the field of plant pathogen detection. [ABSTRACT FROM AUTHOR]

Copy number variations (CNVs) play pivotal roles in disease susceptibility and have been intensively investigated in human disease studies. Long-read sequencing technologies offer opportunities for comprehensive structural variation (SV) detection, and numerous methodologies have been developed recently. Consequently, there is a pressing need to assess these methods and aid researchers in selecting appropriate techniques for CNV detection using long-read sequencing. Hence, we conducted an evaluation of eight CNV calling methods across 22 datasets from nine publicly available samples and 15 simulated datasets, covering multiple sequencing platforms. The overall performance of CNV callers varied substantially and was influenced by the input dataset type, sequencing depth, and CNV type, among others. Specifically, the PacBio CCS sequencing platform outperformed PacBio CLR and Nanopore platforms regarding CNV detection recall rates. A sequencing depth of 10x demonstrated the capability to identify 85% of the CNVs detected in a 50x dataset. Moreover, deletions were more generally detectable than duplications. Among the eight benchmarked methods, cuteSV, Delly, pbsv, and Sniffles2 demonstrated superior accuracy, while SVIM exhibited high recall rates. [ABSTRACT FROM AUTHOR]

Cell segmentation is a fundamental task in analyzing biomedical images. Many computational methods have been developed for cell segmentation and instance segmentation, but their performances are not well understood in various scenarios. We systematically evaluated the performance of 18 segmentation methods to perform cell nuclei and whole cell segmentation using light microscopy and fluorescence staining images. We found that general-purpose methods incorporating the attention mechanism exhibit the best overall performance. We identified various factors influencing segmentation performances, including image channels, choice of training data, and cell morphology, and evaluated the generalizability of methods across image modalities. We also provide guidelines for choosing the optimal segmentation methods in various real application scenarios. We developed Seggal, an online resource for downloading segmentation models already pre-trained with various tissue and cell types, substantially reducing the time and effort for training cell segmentation models. [ABSTRACT FROM AUTHOR]

Cell-type annotation is a critical step in single-cell data analysis. With the development of numerous cell annotation methods, it is necessary to evaluate these methods to help researchers use them effectively. Reference datasets are essential for evaluation, but currently, the cell labels of reference datasets mainly come from computational methods, which may have computational biases and may not reflect the actual cell-type outcomes. This study first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch and systematically evaluated 18 cell annotation methods. We assessed those methods under five scenarios, including intra-dataset validation, immune cell-subtype validation, unsupervised clustering, inter-dataset annotation, and unknown cell-type prediction. Accuracy and ARI were evaluation metrics. The results showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution. Our results indicated that experimentally labeled immune cell-subtype datasets revealed the deficiencies of unsupervised clustering methods and provided new dataset support for supervised methods. [ABSTRACT FROM AUTHOR]

The accurate reconstruction of indoor environments is crucial for applications in augmented reality, virtual reality, and robotics. However, existing indoor datasets are often limited in scale, lack ground truth point clouds, and provide insufficient viewpoints, which impedes the development of robust novel view synthesis (NVS) techniques. To address these limitations, we introduce a new large-scale indoor dataset that features diverse and challenging scenes, including basements and long corridors. This dataset offers panoramic image sequences for comprehensive coverage, high-resolution point clouds, meshes, and textures as ground truth, and a novel benchmark specifically designed to evaluate NVS algorithms in complex indoor environments. Our dataset and benchmark aim to advance indoor scene reconstruction and facilitate the creation of more effective NVS solutions for real-world applications. [ABSTRACT FROM AUTHOR]

Traditional methods for object detection typically necessitate a substantial amount of training data, and creating high-quality training data is time-consuming. We propose a novel Few-Shot Object Detection network (FSODv2) in this paper that aims to detect objects from previously unseen categories using only a few annotated examples. Attention RPN, Multi-Relation Detector, and Contrastive Training strategy are central to our method (Fan et al., in: CVPR, 2020), which exploit similarity between few shot support set and query set to detect novel objects while suppressing false detection in the background. We also contribute a new dataset, FSOD-1k, which contains 1000 categories of various objects with high-quality annotations to train our network. To the best of our knowledge, this is one of the first datasets designed for few-shot object detection. This paper improves our FSOD model through well-designed model calibration in three areas: (1) we propose an improved FPN with multi-scale support inputs to calibrate the multi-scale support-query feature matching by exploiting multi-scale features from the same support image with different input scales; (2) we introduce a support classification supervision branch to calibrate the support feature supervision, aligning to the query feature training supervision; (3) we propose backbone calibration to preserve prior knowledge while alleviating backbone bias toward base classes by employing classification dataset to help our model calibration procedure, where such dataset has previously only been used for pre-training in other related works. Besides, we propose a Fast Attention RPN to improve evaluation speed and save computational memory during inference. Once trained, our few-shot network can detect objects from previously unseen categories without further training or fine-tuning, resulting in new state-of-the-art performance on different datasets in the few-shot setting. Our method is general in scope and has numerous potential applications. The dataset link is https://github.com/fanq15/Few-Shot-Object-Detection-Dataset. [ABSTRACT FROM AUTHOR]

Quadratic Unconstrained Binary Optimization (QUBO) is a combinatorial optimization problem defined by an energy function that consists of a quadratic formula involving multiple binary variables. We propose the method of combing bit reduction and duplication technique that can generate hard instances of QUBO problems from any original QUBO problem without changing the size. The idea is to reduce the original QUBO problem for a specified number of bits and duplicate the same number of bits while ensuring all duplicated pair of bits take the same binary values. Experimental results show that generated QUBO instances are much hard for solving. [ABSTRACT FROM AUTHOR]

Most third-generation sequencing (TGS) processing tools rely on multiple sequence alignment (MSA) methods to manage sequencing errors. Despite the broad range of MSA approaches available, a limited selection of implementations are commonly used in practice for this type of application, and no comprehensive comparative assessment of existing tools has been undertaken to date. In this context, we have developed an automatic pipeline, named MSA Limit, designed to facilitate the execution and evaluation of diverse MSA methods across a spectrum of conditions representative of TGS reads. MSA Limit offers insights into alignment accuracy, time efficiency, and memory utilization. It serves as a valuable resource for both users and developers, aiding in the assessment of algorithmic performance and assisting users in selecting the most appropriate tool for their specific experimental settings. Through a series of experiments using real and simulated data, we demonstrate the value of such exploration. Our findings reveal that in certain scenarios, popular methods may not consistently exhibit optimal efficiency and that the choice of the most effective method varies depending on factors such as sequencing depth, genome characteristics, and read error patterns. MSA Limit is an open source and freely available tool. All code and data pertaining to it and this manuscript are available at . [ABSTRACT FROM AUTHOR]

Copyright of Procesamiento del Lenguaje Natural is the property of Sociedad Espanola para el Procesamiento del Lenguaje Natural and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Purpose: This work presents FASTRL, a benchmark set of instrument manipulation tasks adapted to the domain of reinforcement learning and used in simulated surgical training. This benchmark enables and supports the design and training of human-centric reinforcement learning agents which assist and evaluate human trainees in surgical practice. Methods: Simulation tasks from the Fundamentals of Arthroscopic Surgery Training (FAST) program are adapted to the reinforcement learning setting for the purpose of training virtual agents that are capable of providing assistance and scoring to the surgical trainees. A skill performance assessment protocol is presented based on the trained virtual agents. Results: The proposed benchmark suite presents an API for training reinforcement learning agents in the context of arthroscopic skill training. The evaluation scheme based on both heuristic and learned reward functions robustly recovers the ground truth ranking on a diverse test set of human trajectories. Conclusion: The presented benchmark enables the exploration of a novel reinforcement learning-based approach to skill performance assessment and in-procedure assistance for simulated surgical training scenarios. The evaluation protocol based on the learned reward model demonstrates potential for evaluating the performance of surgical trainees in simulation. [ABSTRACT FROM AUTHOR]

Understanding and modeling of agent's risk/reward preferences is a central problem in various applications of risk management including investment science and portfolio theory in particular. One of the approaches is to axiomatically define a set of performance measures and to use a benchmark to identify a particular measure from that set by either inverse optimization or functional dominance. For example, such a benchmark could be the rate of return of an existing attractive financial instrument. This work introduces deviation and drawdown measures that incorporate rates of return of indicated financial instruments (benchmarks). For discrete distributions and discrete sample paths, portfolio problems with such measures are reduced to linear programs and solved based on historical data in cases of a single benchmark and three benchmarks used simultaneously. The optimal portfolios and corresponding benchmarks have similar expected/cumulative rates of return in sample and out of sample, but the former are considerably less volatile. [ABSTRACT FROM AUTHOR]

The field of 'explainable' artificial intelligence (XAI) has produced highly acclaimed methods that seek to make the decisions of complex machine learning (ML) methods 'understandable' to humans, for example by attributing 'importance' scores to input features. Yet, a lack of formal underpinning leaves it unclear as to what conclusions can safely be drawn from the results of a given XAI method and has also so far hindered the theoretical verification and empirical validation of XAI methods. This means that challenging non-linear problems, typically solved by deep neural networks, presently lack appropriate remedies. Here, we craft benchmark datasets for one linear and three different non-linear classification scenarios, in which the important class-conditional features are known by design, serving as ground truth explanations. Using novel quantitative metrics, we benchmark the explanation performance of a wide set of XAI methods across three deep learning model architectures. We show that popular XAI methods are often unable to significantly outperform random performance baselines and edge detection methods, attributing false-positive importance to features with no statistical relationship to the prediction target rather than truly important features. Moreover, we demonstrate that explanations derived from different model architectures can be vastly different; thus, prone to misinterpretation even under controlled conditions. [ABSTRACT FROM AUTHOR]

Feature attribution maps are a popular approach to highlight the most important pixels in an image for a given prediction of a model. Despite a recent growth in popularity and available methods, the objective evaluation of such attribution maps remains an open problem. Building on previous work in this domain, we investigate existing quality metrics and propose new variants of metrics for the evaluation of attribution maps. We confirm a recent finding that different quality metrics seem to measure different underlying properties of attribution maps, and extend this finding to a larger selection of attribution methods, quality metrics, and datasets. We also find that metric results on one dataset do not necessarily generalize to other datasets, and methods with desirable theoretical properties do not necessarily outperform computationally cheaper alternatives in practice. Based on these findings, we propose a general benchmarking approach to help guide the selection of attribution methods for a given use case. Implementations of attribution metrics and our experiments are available online (https://github.com/arnegevaert/benchmark-general-imaging). [ABSTRACT FROM AUTHOR]

Purpose: The production and construction of buildings cause significant environmental impacts besides those arising from their operation. Recently, some European countries have started introducing life cycle assessment as a mandatory calculation method for new buildings, and it is foreseen that by 2030 this will be done in every member state, at first without any legal minimum values. Methods: Extensive databases on the embodied impacts of buildings, which would be needed to support setting the baseline impacts, are still missing. This paper proposes an approach for determining bottom-up reference values. A large building sample is generated describing "technically feasible" new buildings. Instead of analysing a few typical buildings, the main parameters describing a building are determined and the ranges are defined that these parameters typically take. With the variation of these parameters, a large building sample is generated, and the surfaces and built-in material quantities are determined for typical construction solutions to assess environmental performance. Results and discussion: The method is demonstrated by calculating the reference embodied benchmark values for new residential buildings in Hungary. The results show a baseline embodied Global Warming Potential of 9.5–15.5 kg CO2-eq/m2/yr for single-family houses and 9.1–14.3 kg CO2-eq/m2/yr for multi-family houses. Conclusions: This method is suitable for estimating the environmental impact of typical new buildings in countries where a large pool of real building data is not yet available. [ABSTRACT FROM AUTHOR]

Abstract The accurate identification of protein–ligand binding sites is of critical importance in understanding and modulating protein function. Accordingly, ligand binding site prediction has remained a research focus for over three decades with over 50 methods developed and a change of paradigm from geometry-based to machine learning. In this work, we collate 13 ligand binding site predictors, spanning 30 years, focusing on the latest machine learning-based methods such as VN-EGNN, IF-SitePred, GrASP, PUResNet, and DeepPocket and compare them to the established P2Rank, PRANK and fpocket and earlier methods like PocketFinder, Ligsite and Surfnet. We benchmark the methods against the human subset of our new curated reference dataset, LIGYSIS. LIGYSIS is a comprehensive protein–ligand complex dataset comprising 30,000 proteins with bound ligands which aggregates biologically relevant unique protein–ligand interfaces across biological units of multiple structures from the same protein. LIGYSIS is an improvement for testing methods over earlier datasets like sc-PDB, PDBbind, binding MOAD, COACH420 and HOLO4K which either include 1:1 protein–ligand complexes or consider asymmetric units. Re-scoring of fpocket predictions by PRANK and DeepPocket display the highest recall (60%) whilst IF-SitePred presents the lowest recall (39%). We demonstrate the detrimental effect that redundant prediction of binding sites has on performance as well as the beneficial impact of stronger pocket scoring schemes, with improvements up to 14% in recall (IF-SitePred) and 30% in precision (Surfnet). Finally, we propose top-N+2 recall as the universal benchmark metric for ligand binding site prediction and urge authors to share not only the source code of their methods, but also of their benchmark. Scientific contributions This study conducts the largest benchmark of ligand binding site prediction methods to date, comparing 13 original methods and 15 variants using 10 informative metrics. The LIGYSIS dataset is introduced, which aggregates biologically relevant protein–ligand interfaces across multiple structures of the same protein. The study highlights the detrimental effect of redundant binding site prediction and demonstrates significant improvement in recall and precision through stronger scoring schemes. Finally, top-N+2 recall is proposed as a universal benchmark metric for ligand binding site prediction, with a recommendation for open-source sharing of both methods and benchmarks.

Abstract Background The integration of 7 Tesla (7T) magnetic resonance imaging (MRI) with advanced multimodal artificial intelligence (AI) models represents a promising frontier in neuroimaging. The superior spatial resolution of 7TMRI provides detailed visualizations of brain structure, which are crucial forunderstanding complex central nervous system diseases and tumors. Concurrently, the application of multimodal AI to medical images enables interactive imaging‐based diagnostic conversation. Methods In this paper, we systematically investigate the capacity and feasibility of applying the existing advanced multimodal AI model ChatGPT‐4V to 7T MRI under the context of brain tumors. First, we test whether ChatGPT‐4V has knowledge about 7T MRI, and whether it can differentiate 7T MRI from 3T MRI. In addition, we explore whether ChatGPT‐4V can recognize different 7T MRI modalities and whether it can correctly offer diagnosis of tumors based on single or multiple modality 7T MRI. Results ChatGPT‐4V exhibited accuracy of 84.4% in 3T‐vs‐7T differentiation and accuracy of 78.9% in 7T modality recognition. Meanwhile, in a human evaluation with three clinical experts, ChatGPT obtained average scores of 9.27/20 in single modality‐based diagnosis and 21.25/25 in multiple modality‐based diagnosis. Our study indicates that single‐modality diagnosis and the interpretability of diagnostic decisions in clinical practice should be enhanced when ChatGPT‐4V is applied to 7T data. Conclusions In general, our analysis suggests that such integration has promise as a tool to improve the workflow of diagnostics in neurology, with a potentially transformative impact in the fields of medical image analysis and patient management.

Abstract Background Shapley values have been used extensively in machine learning, not only to explain black box machine learning models, but among other tasks, also to conduct model debugging, sensitivity and fairness analyses and to select important features for robust modelling and for further follow-up analyses. Shapley values satisfy certain axioms that promote fairness in distributing contributions of features toward prediction or reducing error, after accounting for non-linear relationships and interactions when complex machine learning models are employed. Recently, feature selection methods using predictive Shapley values and p-values have been introduced, including powershap. Methods We present a novel feature selection method, LLpowershap, that takes forward these recent advances by employing loss-based Shapley values to identify informative features with minimal noise among the selected sets of features. We also enhance the calculation of p-values and power to identify informative features and to estimate number of iterations of model development and testing. Results Our simulation results show that LLpowershap not only identifies higher number of informative features but outputs fewer noise features compared to other state-of-the-art feature selection methods. Benchmarking results on four real-world datasets demonstrate higher or comparable predictive performance of LLpowershap compared to other Shapley based wrapper methods, or filter methods. LLpowershap is also ranked the best in mean ranking among the seven feature selection methods tested on the benchmark datasets. Conclusion Our results demonstrate that LLpowershap is a viable wrapper feature selection method that can be used for feature selection in large biomedical datasets and other settings.

Abstract This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $$n = 300$$ n = 300 data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8)—a statistically significant difference (p = $$2.482 \times 10^{-10}$$ 2.482 × 10 - 10 ). Prompt engineering significantly improved scores for both GPT-4 (p = $$1.661 \times 10^{-4}$$ 1.661 × 10 - 4 ) and GPT-3.5 (p = $$4.967 \times 10^{-9}$$ 4.967 × 10 - 9 ). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from ‘Definitely AI’ to ‘Definitely Human’. They accurately identified the authorship, with 92.1% of the work categorized as ‘Definitely Human’ being human-authored. Simplifying this to a binary ‘AI’ or ‘Human’ categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students’ work, it often remains detectable by human evaluators.

Abstract Background In microbiome disease association studies, it is a fundamental task to test which microbes differ in their abundance between groups. Yet, consensus on suitable or optimal statistical methods for differential abundance testing is lacking, and it remains unexplored how these cope with confounding. Previous differential abundance benchmarks relying on simulated datasets did not quantitatively evaluate the similarity to real data, which undermines their recommendations. Results Our simulation framework implants calibrated signals into real taxonomic profiles, including signals mimicking confounders. Using several whole meta-genome and 16S rRNA gene amplicon datasets, we validate that our simulated data resembles real data from disease association studies much more than in previous benchmarks. With extensively parametrized simulations, we benchmark the performance of nineteen differential abundance methods and further evaluate the best ones on confounded simulations. Only classic statistical methods (linear models, the Wilcoxon test, t-test), limma, and fastANCOM properly control false discoveries at relatively high sensitivity. When additionally considering confounders, these issues are exacerbated, but we find that adjusted differential abundance testing can effectively mitigate them. In a large cardiometabolic disease dataset, we showcase that failure to account for covariates such as medication causes spurious association in real-world applications. Conclusions Tight error control is critical for microbiome association studies. The unsatisfactory performance of many differential abundance methods and the persistent danger of unchecked confounding suggest these contribute to a lack of reproducibility among such studies. We have open-sourced our simulation and benchmarking software to foster a much-needed consolidation of statistical methodology for microbiome research.

Abstract Background Predictive modeling based on multi-omics data, which incorporates several types of omics data for the same patients, has shown potential to outperform single-omics predictive modeling. Most research in this domain focuses on incorporating numerous data types, despite the complexity and cost of acquiring them. The prevailing assumption is that increasing the number of data types necessarily improves predictive performance. However, the integration of less informative or redundant data types could potentially hinder this performance. Therefore, identifying the most effective combinations of omics data types that enhance predictive performance is critical for cost-effective and accurate predictions. Methods In this study, we systematically evaluated the predictive performance of all 31 possible combinations including at least one of five genomic data types (mRNA, miRNA, methylation, DNAseq, and copy number variation) using 14 cancer datasets with right-censored survival outcomes, publicly available from the TCGA database. We employed various prediction methods and up-weighted clinical data in every model to leverage their predictive importance. Harrell’s C-index and the integrated Brier Score were used as performance measures. To assess the robustness of our findings, we performed a bootstrap analysis at the level of the included datasets. Statistical testing was conducted for key results, limiting the number of tests to ensure a low risk of false positives. Results Contrary to expectations, we found that using only mRNA data or a combination of mRNA and miRNA data was sufficient for most cancer types. For some cancer types, the additional inclusion of methylation data led to improved prediction results. Far from enhancing performance, the introduction of more data types most often resulted in a decline in performance, which varied between the two performance measures. Conclusions Our findings challenge the prevailing notion that combining multiple omics data types in multi-omics survival prediction improves predictive performance. Thus, the widespread approach in multi-omics prediction of incorporating as many data types as possible should be reconsidered to avoid suboptimal prediction results and unnecessary expenditure.

Abstract Background Single-cell chromatin accessibility assays, such as scATAC-seq, are increasingly employed in individual and joint multi-omic profiling of single cells. As the accumulation of scATAC-seq and multi-omics datasets continue, challenges in analyzing such sparse, noisy, and high-dimensional data become pressing. Specifically, one challenge relates to optimizing the processing of chromatin-level measurements and efficiently extracting information to discern cellular heterogeneity. This is of critical importance, since the identification of cell types is a fundamental step in current single-cell data analysis practices. Results We benchmark 8 feature engineering pipelines derived from 5 recent methods to assess their ability to discover and discriminate cell types. By using 10 metrics calculated at the cell embedding, shared nearest neighbor graph, or partition levels, we evaluate the performance of each method at different data processing stages. This comprehensive approach allows us to thoroughly understand the strengths and weaknesses of each method and the influence of parameter selection. Conclusions Our analysis provides guidelines for choosing analysis methods for different datasets. Overall, feature aggregation, SnapATAC, and SnapATAC2 outperform latent semantic indexing-based methods. For datasets with complex cell-type structures, SnapATAC and SnapATAC2 are preferred. With large datasets, SnapATAC2 and ArchR are most scalable.