Search

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.

The spatial organization of chromatin within the nucleus plays a crucial role in gene expression and genome function. However, the quantitative relationship between this organization and nuclear biochemical processes remains under debate. In this study, we present a graph-based generative model, bioSBM, designed to capture long-range chromatin interaction patterns from Hi-C data and, importantly, simultaneously, link these patterns to biochemical features. Applying bioSBM to Hi-C maps of the GM12878 lymphoblastoid cell line, we identified a latent structure of chromatin interactions, revealing 12 distinct communities that strongly align with known biological annotations. Additionally, we infer a linear transformation that maps biochemical observables, such as histone marks, to the parameters of the generative graph model, enabling accurate genome-wide predictions of chromatin contact maps on out-of-sample data, both within the same cell line, and on the completely unseen HCT116 cell line under RAD21 depletion. These findings highlight bioSBM's potential as a powerful tool for elucidating the relationship between biochemistry and chromatin architecture and predicting long-range genome organization from independent biochemical data.

This paper introduces the concept of Language-Guided World Models (LWMs) -- probabilistic models that can simulate environments by reading texts. Agents equipped with these models provide humans with more extensive and efficient control, allowing them to simultaneously alter agent behaviors in multiple tasks via natural verbal communication. In this work, we take initial steps in developing robust LWMs that can generalize to compositionally novel language descriptions. We design a challenging world modeling benchmark based on the game of MESSENGER (Hanjie et al., 2021), featuring evaluation settings that require varying degrees of compositional generalization. Our experiments reveal the lack of generalizability of the state-of-the-art Transformer model, as it offers marginal improvements in simulation quality over a no-text baseline. We devise a more robust model by fusing the Transformer with the EMMA attention mechanism (Hanjie et al., 2021). Our model substantially outperforms the Transformer and approaches the performance of a model with an oracle semantic parsing and grounding capability. To demonstrate the practicality of this model in improving AI safety and transparency, we simulate a scenario in which the model enables an agent to present plans to a human before execution, and to revise plans based on their language feedback., Comment: SpLU-RoboNLP workshop at ACL 2024

The use of appearance codes in recent work on generative modeling has enabled novel view renders with variable appearance and illumination, such as day-time and night-time renders of a scene. A major limitation of this technique is the need to re-train new appearance codes for every scene on inference, so in this work we address this problem proposing a framework that learns a joint embedding space for the appearance and structure of the scene by enforcing a contrastive loss constraint between different modalities. We apply our framework to a simple Variational Auto-Encoder model on the RADIATE dataset \cite{sheeny2021radiate} and qualitatively demonstrate that we can generate new renders of night-time photos using day-time appearance codes without additional optimization iterations. Additionally, we compare our model to a baseline VAE that uses the standard per-image appearance code technique and show that our approach achieves generations of similar quality without learning appearance codes for any unseen images on inference.

Transaction fees represent a major incentive in many blockchain systems as a way to incentivize processing transactions. Unfortunately, they also introduce an enormous amount of incentive asymmetry compared to alternatives like fixed block rewards. We analyze some of the incentive compatibility issues that arise from transaction fees, which relate to the bids that users submit, the allocation rules that miners use to choose which transactions to include, and where they choose to mine in the context of longest-chain consensus. We start by surveying a variety of mining attacks including undercutting, fee sniping, and fee-optimized selfish mining. Then, we move to analyzing mechanistic notions of user incentive compatibility, myopic miner incentive compatibility, and off-chain-agreement-proofness, as well as why they are provably incompatible in their full form. Then, we discuss weaker notions of nearly and $\gamma$-weak incentive compatibility, and how all of these forms of incentive compatibility hold or fail in the trustless auctioneer setup of blockchains, examining classical mechanisms as well as more recent ones such as Ethereum's EIP-1559 mechanism and \cite{chung}'s burning second-price auction. Throughout, we generalize and interrelate existing notions, provide new unifying perspectives and intuitions on analysis, and discuss both specific and overarching open problems for future work.

Existing benchmarks for evaluating long video understanding falls short on two critical aspects, either lacking in scale or quality of annotations. These limitations arise from the difficulty in collecting dense annotations for long videos, which often require manually labeling each frame. In this work, we introduce an automated Annotation and Video Stream Alignment Pipeline (abbreviated ASAP). We demonstrate the generality of ASAP by aligning unlabeled videos of four different sports with corresponding freely available dense web annotations (i.e. commentary). We then leverage ASAP scalability to create LCric, a large-scale long video understanding benchmark, with over 1000 hours of densely annotated long Cricket videos (with an average sample length of ~50 mins) collected at virtually zero annotation cost. We benchmark and analyze state-of-the-art video understanding models on LCric through a large set of compositional multi-choice and regression queries. We establish a human baseline that indicates significant room for new research to explore. Our human studies indicate that ASAP can align videos and annotations with high fidelity, precision, and speed. The dataset along with the code for ASAP and baselines can be accessed here: https://asap-benchmark.github.io/.

Cellular functions crucially depend on the precise execution of complex biochemical reactions taking place on the chromatin fiber in the tightly packed environment of the cell nucleus. Despite the availability of large data sets probing this process from multiple angles, we still lack a bottom-up framework which can incorporate the sequence-specific nature of biochemistry in a unified model of 3D chromatin dynamics. Here we propose SEMPER (Sequence Enhanced Magnetic PolymER), a novel stochastic polymer model which naturally incorporates observational data about sequence-driven biochemical processes, such as binding of transcription factor proteins, in a 3D model of chromatin structure. By introducing a new algorithm for approximate Bayesian inference, we discuss how to estimate in a robust manner the relative importance of biochemical vs. polymer signals in the determination of the chromatin epigenetic states which is leading to a significant revision of the interpretation of previous models. Furthermore we show that, without additional input from the genome 3D structure, our model can predict with reasonable accuracy some notable and non trivial conformational features of chromatin folding within the nucleus. Our work highlights the importance of introducing physically realistic statistical models for predicting chromatin states from epigenetic data, and opens the way to a new class of more systematic approaches to interpret epigenomic data.

Non-invasive measurement of the arterial blood speed gives rise to important healthinformation such as cardio output and blood supplies to vital organs. The magnitude andchange in arterial blood speed are key indicators of the health conditions and development andprogression of diseases. We demonstrated a simple technique to directly measure the blood flowspeed in main arteries based on the diffused light model. The concept is demonstrated with aphantom that uses intralipid hydrogel to model the biological tissue and an embedded glass tubewith flowing human blood to model the blood vessel. The correlation function of the measuredphotocurrent was used to find the electrical field correlation function via the Siegert relation.We have shown that the characteristic decorrelation rate (i.e. the inverse of the decoherent time)is linearly proportional to the blood speed and independent of the tube diameter. This strikingproperty can be explained by an approximate analytic solution for the diffused light equation inthe regime where the convective flow is the dominating factor for decorrelation. As a result, wehave demonstrated a non-invasive method of measuring arterial blood speed without any priorknowledge or assumption about the geometric or mechanic properties of the blood vessels.

Click-through rate (CTR) prediction plays a critical role in recommender systems and online advertising. The data used in these applications are multi-field categorical data, where each feature belongs to one field. Field information is proved to be important and there are several works considering fields in their models. In this paper, we proposed a novel approach to model the field information effectively and efficiently. The proposed approach is a direct improvement of FwFM, and is named as Field-matrixed Factorization Machines (FmFM, or $FM^2$). We also proposed a new explanation of FM and FwFM within the FmFM framework, and compared it with the FFM. Besides pruning the cross terms, our model supports field-specific variable dimensions of embedding vectors, which acts as soft pruning. We also proposed an efficient way to minimize the dimension while keeping the model performance. The FmFM model can also be optimized further by caching the intermediate vectors, and it only takes thousands of floating-point operations (FLOPs) to make a prediction. Our experiment results show that it can out-perform the FFM, which is more complex. The FmFM model's performance is also comparable to DNN models which require much more FLOPs in runtime., Comment: In Proceedings of the Web Conference 2021 (WWW 2021), April 19-23, 2021, Ljubljana, Slovenia. 10 pages

Organic-inorganic metal halide perovskite devices have generated significant interest for LED, photodetector, and solar cell applications due to their attractive optoelectronic properties and substrate-choice flexibility1-4. These devices exhibit slow time-scale response, which have been explained by point defect migration5-6. In this work, we report the discovery of a room temperature intrinsic amplification process in methylammonium lead iodide perovskite (MAPbI3) that can be triggered by few photons, down to a single photon. The electrical properties of the material, by way of photoresponse, are modified by an input energy as small as 0.19 attojoules, the energy of a single photon. These observations cannot be explained by photo-excited electronic band-to-band transitions or prevailing model of photo-excited point defect migration since none of the above can explain the observed macroscopic property change by absorption of single or few photons. The results suggest the existence of an avalanche-like collective motion of iodides and their accumulation near the anode, which we will call ionic impact ionization (I3 mechanism). The proposed I3 process is the ionic analog of the electronic impact ionization, and has been considered impossible before because conventionally it takes far more energy to move ions out of their equilibrium position than electrons. We have performed first principle calculations to show that in MAPbI3 the activation energy for the I3 mechanism is appreciably lower than the literature value of the activation energy for the electronic impact ionization. The discovery of I3 process in perovskite material opens up possibilities for new classes of devices for photonic and electronic applications.

Increasing demand for understanding the vast heterogeneity of cellular phenotypes has driven the development of imaging flow cytometry (IFC), that combines features of flow cytometry with fluorescence and bright field microscopy. IFC combines the throughput and statistical advantage of flow cytometry with the ability to discretely measure events based on a real or computational image, as well as conventional flow cytometry metrics. A limitation of existing IFC systems is that, regardless of detection methodology, only two-dimensional (2D) cell images are obtained. Without tomographic three-dimensional (3D) resolution the projection problem remains: collapsing 3D information onto a 2D image, limiting the reliability of spot counting or co-localization crucial to cell phenotyping. Here we present a solution to the projection problem: three-dimensional imaging flow cytometry (3D-IFC), a high-throughput 3D cell imager based on optical sectioning microscopy. We combine orthogonal light-sheet scanning illumination with our previous spatiotemporal transformation detection to produce 3D cell image reconstruction from a cameraless single-pixel photodetector readout. We further demonstrate this capability by co-capturing 3D fluorescence and label-free side-scattering images of single cells in flow at a velocity of 0.2 m s-1, corresponding to a throughput of approximately 500 cells per second with 60,000 voxels (resized subsequently to 106 voxels) for each cell image at a resolution of less than 1 micron in X, Y, and Z dimensions. Improved high-throughput imaging tools are needed to phenotype-genotype recognized heterogeneity in the fields of immunology, oncology, cell- and gene- therapy, and drug discovery.

We study the conjecture that if $I,J$ are $\mu-$primary in a regular local ring $(R,\mu)$ with dim$(R)=n$, then $\frac{I \cap J}{IJ} \cong \Tor_1(R/I, R/J)$ needs at least $n$ generators, and a related conjecture about the number of generators of the first Koszul homology module of an Artinian local ring $(A,m)$. In this manuscript, we focus our attention on the complete intersection defect of the Artinian ring and its quotient by the Koszul elements. We prove that the number of generators of the first Koszul homology module of $x_1,...,x_n \in m$ on an Artinian local ring $(A,m)$ is at least $n+ \cid(A)- \cid(\frac{A}{(x_1,...,x_n)A})$, where $\cid A$ denotes the complete intersection defect of the Artinian local ring $A$.