Search

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future., Comment: 25 pages, 9 figures, 7 tables

An uncertainty in studying twisted bilayer graphene (TBG) is the minimum energy geometry, which strongly affects the electronic structure. The minimum energy geometry is determined by the potential energy surface, which is dominated by van der Waals (vdW) interactions. In this work, large-scale diffusion quantum Monte Carlo (QMC) simulations are performed to evaluate the energy of bilayer graphene at various interlayer distances for four stacking registries. An accurate registry-dependent potential is fit to the QMC data and is used to describe interlayer interactions in the geometry of near-magic-angle TBG. The band structure for the optimized geometry is evaluated using the accurate local-environment tight-binding model. We find that compared to QMC, DFT-based vdW interactions can result in errors in the corrugation magnitude by a factor of 2 or more near the magic angle. The error in corrugation then propagates to the flat bands in twisted bilayer graphene, where the error in corrugation can affect the bandwidth by about 30\% and can change the nature and degeneracy of the flat bands.

We describe a new open-source Python-based package for high accuracy correlated electron calculations using quantum Monte Carlo (QMC) in real space: PyQMC. PyQMC implements modern versions of QMC algorithms in an accessible format, enabling algorithmic development and easy implementation of complex workflows. Tight integration with the PySCF environment allows for simple comparison between QMC calculations and other many-body wave function techniques, as well as access to high accuracy trial wave functions.

Fixed node diffusion Monte Carlo (DMC) has been performed on a test set of forward and reverse barrier heights for 19 non-hydrogen-transfer reactions, and the nodal error has been assessed. The DMC results are robust to changes in the nodal surface, as assessed by using different mean-field techniques to generate single determinant wave functions. Using these single determinant nodal surfaces, DMC results in errors of 1.5(5) kcal/mol on barrier heights. Using the large data set of DMC energies, we attempted to find good descriptors of the fixed node error. It does not correlate with a number of descriptors including change in density, but does correlate with the gap between the highest occupied and lowest unoccupied orbital energies in the mean-field calculation., Comment: 6 pages, 4 figures, 1 table