Search

Alignment approaches such as RLHF and DPO are actively investigated to align large language models (LLMs) with human preferences. Commercial large language models (LLMs) like GPT-4 have been recently employed to evaluate and compare different LLM alignment approaches. These models act as surrogates for human evaluators due to their promising abilities to approximate human preferences with remarkably faster feedback and lower costs. This methodology is referred to as LLM-as-a-judge. However, concerns regarding its reliability have emerged, attributed to LLM judges' biases and inconsistent decision-making. Previous research has sought to develop robust evaluation frameworks for assessing the reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address the internal inconsistency of LLMs. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-judge methods, which leads to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM judges on alignment tasks (e.g. summarization) by defining evaluation metrics with improved theoretical interpretability and disentangling reliability metrics with LLM internal inconsistency. We develop a framework to evaluate, compare, and visualize the reliability and alignment of LLM judges to provide informative observations that help choose LLM judges for alignment tasks. Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators., Comment: Preprint, under review. 17 pages, 7 figures, 16 tables

High resolution peripheral quantitative computed tomography (HR-pQCT) is an imaging technique capable of imaging trabecular bone in-vivo. HR-pQCT has a wide range of applications, primarily focused on bone to improve our understanding of musculoskeletal diseases, assess epidemiological associations, and evaluate the effects of pharmaceutical interventions. Processing HR-pQCT images has largely been supported using the scanner manufacturer scripting language (Image Processing Language, IPL, Scanco Medical). However, by expanding image processing workflows outside of the scanner manufacturer software environment, users have the flexibility to apply more advanced mathematical techniques and leverage modern software packages to improve image processing. The ORMIR_XCT Python package was developed to reimplement some existing IPL workflows and provide an open and reproducible package allowing for the development of advanced HR-pQCT data processing workflows.

Image dehazing is a meaningful low-level computer vision task and can be applied to a variety of contexts. In our industrial deployment scenario based on remote sensing (RS) images, the quality of image dehazing directly affects the grade of our crop identification and growth monitoring products. However, the widely used peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) provide ambiguous visual interpretation. In this paper, we design a new objective metric for RS image dehazing evaluation. Our proposed metric leverages a ground-based phenology observation resource to calculate the vegetation index error between RS and ground images at a hazy date. Extensive experiments validate that our metric appropriately evaluates different dehazing models and is in line with human visual perception., Comment: Accepted by ICME 2023 Industry Track

Line lists for the ground electronic ground state for six major isotopologues of nitric oxide are presented. The line lists are constructed using empirical energy levels (and line positions) and high-level {\it ab inito} intensities. The energy levels were obtained using a combination of two approaches, from an effective Hamiltonian and from solving the rovibronic Schr\"{o}dinger equation variationally. The effective hamiltonian model was obtained through a fit to the experimental line positions of NO available in the literature for all six isotopologues using the programs SPFIT and SPCAT. The variational model was built through a least squares fit of the \textit{ab inito} potential and spin-orbit curves to the experimentally derived energies and experimental line positions of the main isotopologue only using the Duo program. The \textit{ab inito} potential energy, spin-orbit and dipole moment curves (PEC, SOC and DMC) are computed using high-level {\it ab inito} methods and the MARVEL method is used to obtain energies of NO from experimental transition frequencies. Each line list covers a wavenumber range from 0 - 40,000 \cm with approximately 22,000 rovibronic states and 2.3-2.6 million transitions extending to $J_{max} = 184.5$ and $v_{max} = 51$. Partition functions are also calculated up to a temperature of 5000 K. The calculated absorption line intensities at 296 K using these line lists show excellent agreement with those included in the HITRAN and HITEMP databases. The computed NO line lists are the most comprehensive to date, covering a wider wavenumber and temperature range compared to both the HITRAN and HITEMP databases. These line lists are also more accurate than those used in HITEMP. The full line lists are available from the CDS and ExoMol databases; data will also be available from CDMS.