Search

The field of healthcare has increasingly turned its focus towards Large Language Models (LLMs) due to their remarkable performance. However, their performance in actual clinical applications has been underexplored. Traditional evaluations based on question-answering tasks don't fully capture the nuanced contexts. This gap highlights the need for more in-depth and practical assessments of LLMs in real-world healthcare settings. Objective: We sought to evaluate the performance of LLMs in the complex clinical context of adult critical care medicine using systematic and comprehensible analytic methods, including clinician annotation and adjudication. Methods: We investigated the performance of three general LLMs in understanding and processing real-world clinical notes. Concepts from 150 clinical notes were identified by MetaMap and then labeled by 9 clinicians. Each LLM's proficiency was evaluated by identifying the temporality and negation of these concepts using different prompts for an in-depth analysis. Results: GPT-4 showed overall superior performance compared to other LLMs. In contrast, both GPT-3.5 and text-davinci-003 exhibit enhanced performance when the appropriate prompting strategies are employed. The GPT family models have demonstrated considerable efficiency, evidenced by their cost-effectiveness and time-saving capabilities. Conclusion: A comprehensive qualitative performance evaluation framework for LLMs is developed and operationalized. This framework goes beyond singular performance aspects. With expert annotations, this methodology not only validates LLMs' capabilities in processing complex medical data but also establishes a benchmark for future LLM evaluations across specialized domains.

Some property graph databases do not have a fixed schema, which can result in data type inconsistencies for properties on nodes and relationships, especially when importing data into a running database. Here we present a tool which can rapidly produce a detailed report on every property in the graph. When executed on a large knowledge graph, it allowed us to debug a complex ETL process and enforce 100% data type consistency., Comment: 5 pages, 3 figures, general approach applied to production databases

In this paper, we showcase the process of using an automated conjecturing program called \emph{TxGraffiti} written and maintained by the second author. We begin by proving a conjecture formulated by \emph{TxGraffiti} that for a claw-free graph $G$, the vertex cover number $\beta(G)$ is greater than or equal to the zero forcing number $Z(G)$. Our proof of this result is constructive, and yields a polynomial time algorithm to find a zero forcing set with cardinality $\beta(G)$. We also use the output of \emph{TxGraffiti} to construct several infinite families of claw-free graphs for which $Z(G)=\beta(G)$. Additionally, inspired by the aforementioned conjecture of \emph{TxGraffiti}, we also prove a more general relation between the zero forcing number and the vertex cover number for any connected graph with maximum degree $\Delta \ge 3$, namely that $Z(G)\leq (\Delta-2)\beta(G)$+1., Comment: Feedback welcome!

We present photometric metallicity measurements for a sample of 2.6 million bulge red clump stars extracted from the Blanco DECam Bulge Survey (BDBS). Similar to previous studies, we find that the bulge exhibits a strong vertical metallicity gradient, and that at least two peaks in the metallicity distribution functions appear at b < -5. We can discern a metal-poor ([Fe/H] ~ -0.3) and metal-rich ([Fe/H] ~ +0.2) abundance distribution that each show clear systematic trends with latitude, and may be best understood by changes in the bulge's star formation/enrichment processes. Both groups exhibit asymmetric tails, and as a result we argue that the proximity of a star to either peak in [Fe/H] space is not necessarily an affirmation of group membership. The metal-poor peak shifts to lower [Fe/H] values at larger distances from the plane while the metal-rich tail truncates. Close to the plane, the metal-rich tail appears broader along the minor axis than in off-axis fields. We also posit that the bulge has two metal-poor populations -- one that belongs to the metal-poor tail of the low latitude and predominantly metal-rich group, and another belonging to the metal-poor group that dominates in the outer bulge. We detect the X-shape structure in fields with |Z| > 0.7 kpc and for stars with [Fe/H] > -0.5. Stars with [Fe/H] < -0.5 may form a spheroidal or "thick bar" distribution while those with [Fe/H] > -0.1 are strongly concentrated near the plane., Comment: 26 pages, 22 figures, accepted for publication in MNRAS; the full data table is very long so only a stub table has been provided here; the full electronic table will be provided through MNRAS upon publication, but early access to the full table will be granted upon request to the authors

The Blanco DECam Bulge Survey (BDBS) has imaged more than 200 square degrees of the southern Galactic bulge, providing photometry in the ugrizy filters for $\sim 250$ million unique stars. The presence of a strong foreground disk population, along with complex reddening and extreme image crowding, has made it difficult to constrain the presence of young and intermediate age stars in the bulge population. We employed an accurate cross-match of BDBS with the latest data release (EDR3) from the Gaia mission, matching more than 140 million sources with BDBS photometry and Gaia EDR3 photometry and astrometry. We relied on Gaia EDR3 astrometry, without any photometric selection, to produce clean BDBS bulge colour-magnitude diagrams (CMDs). Gaia parallaxes were used to filter out bright foreground sources, and a Gaussian mixture model fit to Galactic proper motions could identify stars kinematically consistent with bulge membership. We applied this method to 127 different bulge fields of $1$ deg$^2$ each, with $|\ell| \leq 9.5^\circ$ and $-9.5^\circ \leq b \leq -2.5^\circ$. The astrometric cleaning procedure removes the majority of blue stars in each field, especially near the Galactic plane, where the ratio of blue to red stars is $\lesssim 10\%$, increasing to values $\sim 20\%$ at higher Galactic latitudes. We rule out the presence of a widespread population of stars younger than 2 Gyr. The vast majority of blue stars brighter than the turnoff belong to the foreground population, according to their measured astrometry. We introduce the distance between the observed red giant branch bump and the red clump as a simple age proxy for the dominant population in the field, and we confirm the picture of a predominantly old bulge. Further work is needed to apply the method to estimate ages to fields at higher latitudes, and to model the complex morphology of the Galactic bulge., Comment: 21 pages, 22 figures, accepted for publication in A&A

As an essential attribute of organic compounds, polarity has a profound influence on many molecular properties such as solubility and phase transition temperature. Thin layer chromatography (TLC) represents a commonly used technique for polarity measurement. However, current TLC analysis presents several problems, including the need for a large number of attempts to obtain suitable conditions, as well as irreproducibility due to non-standardization. Herein, we describe an automated experiment system for TLC analysis. This system is designed to conduct TLC analysis automatically, facilitating high-throughput experimentation by collecting large experimental data under standardized conditions. Using these datasets, machine learning (ML) methods are employed to construct surrogate models correlating organic compounds' structures and their polarity using retardation factor (Rf). The trained ML models are able to predict the Rf value curve of organic compounds with high accuracy. Furthermore, the constitutive relationship between the compound and its polarity can also be discovered through these modeling methods, and the underlying mechanism is rationalized through adsorption theories. The trained ML models not only reduce the need for empirical optimization currently required for TLC analysis, but also provide general guidelines for the selection of conditions, making TLC an easily accessible tool for the broad scientific community.