Search

Autonomous agents that interact with graphical user interfaces (GUIs) hold significant potential for enhancing user experiences. To further improve these experiences, agents need to be personalized and proactive. By effectively comprehending user intentions through their actions and interactions with GUIs, agents will be better positioned to achieve these goals. This paper introduces the task of goal identification from observed UI trajectories, aiming to infer the user's intended task based on their GUI interactions. We propose a novel evaluation metric to assess whether two task descriptions are paraphrases within a specific UI environment. By Leveraging the inverse relation with the UI automation task, we utilized the Android-In-The-Wild and Mind2Web datasets for our experiments. Using our metric and these datasets, we conducted several experiments comparing the performance of humans and state-of-the-art models, specifically GPT-4 and Gemini-1.5 Pro. Our results show that Gemini performs better than GPT but still underperforms compared to humans, indicating significant room for improvement.

We study the primordial spectra and the gravitational-wave background (GWB) of three models of semi-classical, quantum or string gravity where the big bang is replaced by a bounce and the primordial tensor spectrum is blue: ekpyrotic universe with fast-rolling Galileons, string-gas cosmology with Atick-Witten conjecture and pre-big-bang cosmology. We find that the ekpyrotic scenario with Galileons does not produce a GWB amplitude detectable by present or third-generation interferometers, while the Atick-Witten-based string-gas model is ruled out in its present form for violating the big-bang-nucleosynthesis bound, contrary to the original string-gas scenario. In contrast, the GWB of the pre-big-bang scenario falls within the sensitivity window of both LISA and Einstein Telescope, where it takes the form of a single or a broken power law depending on the choice of parameters. The latter will be tightly constrained by both detectors., Comment: 1+36 pages, 4 figures. v2: Einstein Telescope sensitivity curve updated, several parts clarified, results unchanged, references added

Modern 3D printers allow for the accurate placing of microscopic material voxels, to form complex three-dimensional structures. The advancing capabilities in multi-material printing, coupled with the discovery of printable responsive materials, pave the way for the 3D printing of digital materials, and for the 3D printing of shape-transforming structures, or 4D printing. In this work, we leverage and combine these advancements to develop a versatile methodology for crafting digitized 4D-printed shape-transforming sheets. We 3D print digital responsive sheets, composed of active and passive voxels that are meticulously positioned to form thin structures that can be actuated on demand. This methodology enables us to encode the lateral geometry and the reference curvature, separately and simultaneously. Our approach grants us comprehensive control over the resulting shape, unlocking novel opportunities in synthetic shape-morphing materials. Furthermore, it has the potential to program anomalous mechanical properties and to be extended to multi-material and metamaterial systems., Comment: 12 pages, SI, 5 figures

We present a new style of fluxonium qubit, gatemonium, based on an all superconductorsemiconductor hybrid platform. The linear inductance is achieved using six hundred planar Al-InAs Josephson junctions (JJs) in series. By tuning the single junction with a gate voltage, we demonstrate electrostatic control of the effective Josephson energy, tuning the weight of the fictitious phase particle. One and two-tone spectroscopy of the gatemonium transitions further reveal details of the hybrid plasmon-fluxon spectrum. Accounting for the nonsinusoidal current-phase relation of the single junction, we fit the measured spectra to extract charging and inductive energies. We conduct time domain characterization of the plasmon modes in a second gatemonium device with different charging energy and JJ array inductance. We discuss future directions for this platform in gate voltage-tunable, high plasma frequency, enhanced impedance junction arrays, and enhanced coherence times for voltage tunable architectures.

Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection, followed by text generation. In this context, alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data for some of the component tasks. Yet, this enabling alignment step has usually been applied heuristically on the sentence level on a limited number of subtasks. In this paper, we propose extending the summary-source alignment framework by (1) applying it at the more fine-grained proposition span level, (2) annotating alignment manually in a multi-document setup, and (3) revealing the great potential of summary-source alignments to yield several datasets for at least six different tasks. Specifically, for each of the tasks, we release a manually annotated test set that was derived automatically from the alignment annotation. We also release development and train sets in the same way, but from automatically derived alignments. Using the datasets, each task is demonstrated with baseline models and corresponding evaluation metrics to spur future research on this broad challenge., Comment: Accepted to ACL-Findings 2024

Superlatives are used to single out elements with a maximal/minimal property. Semantically, superlatives perform a set comparison: something (or some things) has the min/max property out of a set. As such, superlatives provide an ideal phenomenon for studying implicit phenomena and discourse restrictions. While this comparison set is often not explicitly defined, its (implicit) restrictions can be inferred from the discourse context the expression appears in. In this work we provide an extensive computational study on the semantics of superlatives. We propose a unified account of superlative semantics which allows us to derive a broad-coverage annotation schema. Using this unified schema we annotated a multi-domain dataset of superlatives and their semantic interpretations. We specifically focus on interpreting implicit or ambiguous superlative expressions, by analyzing how the discourse context restricts the set of interpretations. In a set of experiments we then analyze how well models perform at variations of predicting superlative semantics, with and without context. We show that the fine-grained semantics of superlatives in context can be challenging for contemporary models, including GPT-4., Comment: 11 pages

Generating realistic images from arbitrary views based on a single source image remains a significant challenge in computer vision, with broad applications ranging from e-commerce to immersive virtual experiences. Recent advancements in diffusion models, particularly the Zero-1-to-3 model, have been widely adopted for generating plausible views, videos, and 3D models. However, these models still struggle with inconsistencies and implausibility in new views generation, especially for challenging changes in viewpoint. In this work, we propose Zero-to-Hero, a novel test-time approach that enhances view synthesis by manipulating attention maps during the denoising process of Zero-1-to-3. By drawing an analogy between the denoising process and stochastic gradient descent (SGD), we implement a filtering mechanism that aggregates attention maps, enhancing generation reliability and authenticity. This process improves geometric consistency without requiring retraining or significant computational resources. Additionally, we modify the self-attention mechanism to integrate information from the source view, reducing shape distortions. These processes are further supported by a specialized sampling schedule. Experimental results demonstrate substantial improvements in fidelity and consistency, validated on a diverse set of out-of-distribution objects. Additionally, we demonstrate the general applicability and effectiveness of Zero-to-Hero in multi-view, and image generation conditioned on semantic maps and pose., Comment: NeurIPS 2024. Project page: https://zero2hero-nvs.github.io

The discovery of a kilonova associated with the GW170817 binary neutron star merger had far-reaching implications for our understanding of several open questions in physics and astrophysics. Unfortunately, since then, only one robust binary neutron star merger was detected through gravitational waves, GW190425, and no electromagnetic counterpart was identified for it. We analyze all reported electromagnetic followup observations of GW190425 and find that while the gravitational-wave localization uncertainty was large, most of the 90% probability region could have been covered within hours had the search been coordinated. Instead, more than 5 days after the merger, the uncoordinated search covered only 50% of the probability, with some areas observed over 100 times, and some never observed. We further show that, according to some models, it would have been possible to detect the GW190425 kilonova, despite the larger distance and higher component masses compared to GW170817. These results emphasize the importance of coordinating followup of gravitational-wave events, not only to avoid missing future kilonovae, but also to discover them early. Such coordination, which is especially important given the rarity of these events, can be achieved with the Treasure Map, a tool developed specifically for this purpose., Comment: Submitted to ApJ

There are four non-isomorphic configurations of triples that can form a triangle in a $3$-uniform hypergraph. Forbidding different combinations of these four configurations, fifteen extremal problems can be defined, several of which already appeared in the literature in some different context. Here we systematically study all of these problems solving the new cases exactly or asymptotically. In many cases we also characterize the extremal constructions., Comment: 44 pages, 4 figures

Consumer demand forecasting is of high importance for many e-commerce applications, including supply chain optimization, advertisement placement, and delivery speed optimization. However, reliable time series sales forecasting for e-commerce is difficult, especially during periods with many anomalies, as can often happen during pandemics, abnormal weather, or sports events. Although many time series algorithms have been applied to the task, prediction during anomalies still remains a challenge. In this work, we hypothesize that leveraging external knowledge found in world events can help overcome the challenge of prediction under anomalies. We mine a large repository of 40 years of world events and their textual representations. Further, we present a novel methodology based on transformers to construct an embedding of a day based on the relations of the day's events. Those embeddings are then used to forecast future consumer behavior. We empirically evaluate the methods over a large e-commerce products sales dataset, extracted from eBay, one of the world's largest online marketplaces. We show over numerous categories that our method outperforms state-of-the-art baselines during anomalies., Comment: In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM 2022), 9 pages

To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data), while the rest of the data is indiscriminately assigned to model annotation (i.e., triage-to-model data). This may lead to inefficiencies in budget allocation for annotations, as easy data that the model could accurately annotate may be unnecessarily assigned to the expert, and hard data may be misclassified by the model. As a result, the overall annotation quality may be compromised. To address this issue, we propose a selective annotation framework called SANT. It effectively takes advantage of both the triage-to-human and triage-to-model data through the proposed error-aware triage and bi-weighting mechanisms. As such, informative or hard data is assigned to the expert for annotation, while easy data is handled by the model. Experimental results show that SANT consistently outperforms other baselines, leading to higher-quality annotation through its proper allocation of data to both expert and model workers. We provide pioneering work on data annotation within budget constraints, establishing a landmark for future triage-based annotation studies., Comment: Findings of EMNLP 2024

Deploying deep neural networks for risk-sensitive tasks necessitates an uncertainty estimation mechanism. This paper introduces hierarchical selective classification, extending selective classification to a hierarchical setting. Our approach leverages the inherent structure of class relationships, enabling models to reduce the specificity of their predictions when faced with uncertainty. In this paper, we first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves. Next, we develop algorithms for hierarchical selective classification (which we refer to as "inference rules"), and propose an efficient algorithm that guarantees a target accuracy constraint with high probability. Lastly, we conduct extensive empirical studies on over a thousand ImageNet classifiers, revealing that training regimes such as CLIP, pretraining on ImageNet21k and knowledge distillation boost hierarchical selective performance.

Conditional displacement with a qubit ancilla is a critical component in continuous-variable error correction protocols. We present the generalized conditional displacement operator, conditioned on a qudit ancilla, showing how it enhances error-correction with Gottesman-Kitaev-Preskill (GKP) codes and exploring potential implementations.

Automating data generation with Large Language Models (LLMs) has become increasingly popular. In this work, we investigate the feasibility and effectiveness of LLM-based data generation in the challenging setting of source-grounded information-seeking dialogs, with response attribution, over long documents. Our source texts consist of long and noisy meeting transcripts, adding to the task complexity. Since automating attribution remains difficult, we propose a semi-automatic approach: dialog queries and responses are generated with LLMs, followed by human verification and identification of attribution spans. Using this approach, we created MISeD -- Meeting Information Seeking Dialogs dataset -- a dataset of information-seeking dialogs focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance compared to off-the-shelf models, even those of larger size. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort.

The second "Mineral Detection of Neutrinos and Dark Matter" (MDvDM'24) meeting was held January 8-11, 2024 in Arlington, VA, USA, hosted by Virginia Tech's Center for Neutrino Physics. This document collects contributions from this workshop, providing an overview of activities in the field. MDvDM'24 was the second topical workshop dedicated to the emerging field of mineral detection of neutrinos and dark matter, following a meeting hosted by IFPU in Trieste, Italy in October 2022. Mineral detectors have been proposed for a wide variety of applications, including searching for dark matter, measuring various fluxes of astrophysical neutrinos over gigayear timescales, monitoring nuclear reactors, and nuclear disarmament protocols; both as paleo-detectors using natural minerals that could have recorded the traces of nuclear recoils for timescales as long as a billion years and as detectors recording nuclear recoil events on laboratory timescales using natural or artificial minerals. Contributions to this proceedings discuss the vast physics potential, the progress in experimental studies, and the numerous challenges lying ahead on the path towards mineral detection. These include a better understanding of the formation and annealing of recoil defects in crystals; identifying the best classes of minerals and, for paleo-detectors, understanding their geology; modeling and control of the relevant backgrounds; developing, combining, and scaling up imaging and data analysis techniques; and many others. During the last years, MDvDM has grown rapidly and gained attention. Small-scale experimental efforts focused on establishing various microscopic readout techniques are underway at institutions in North America, Europe and Asia. We are looking ahead to an exciting future full of challenges to overcome, surprises to be encountered, and discoveries lying ahead of us., Comment: Summary and proceedings of the MDvDM'24 conference, Jan 8-11 2024

The origin of the transition from asymptotically reversible to asymptotically irreversible response in amorphous solids subject to oscillatory shear is still unknown. It is known that the plastic events that result from shearing always involve localized particle rearrangements, but it is unclear why some are reversible while others are not. Here we show, using simulations and models, that overlaps between particle rearrangements caused by straining the solid in alternating directions can cause the response to become irreversible when they occur frequently. As the forcing amplitude increases, plastic events become more frequent, the number of such overlaps increases, and the probability of the system returning to previous states diminishes.

The emerging field of free-electron quantum optics enables electron-photon entanglement and holds the potential for generating nontrivial photon states for quantum information processing. Although recent experimental studies have entered the quantum regime, rapid theoretical developments predict that qualitatively unique phenomena only emerge beyond a certain interaction strength. It is thus pertinent to identify the maximal electron-photon interaction strength and the materials, geometries, and particle energies that enable one to approach it. We derive an upper limit to the quantum vacuum interaction strength between free electrons and single-mode photons, which illuminates the conditions for the strongest interaction. Crucially, we obtain an explicit energy selection recipe for electrons and photons to achieve maximal interaction at arbitrary separations and identify two optimal regimes favoring either fast or slow electrons over those with intermediate velocities. We validate the limit by analytical and numerical calculations on canonical geometries and provide near-optimal designs indicating the feasibility of strong quantum interactions. Our findings offer fundamental intuition for maximizing the quantum interaction between free electrons and photons and provide practical design rules for future experiments on electron-photon and electron-mediated photon-photon entanglement. They should also enable the evaluation of key metrics for applications such as the maximum power of free-electron radiation sources and the maximum acceleration gradient of dielectric laser accelerators.

Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named "Attribute First, then Generate", breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments ("select first") and then conditioning the generation process on them ("then generate"), we ensure these segments also act as the output's fine-grained attributions ("select" becomes "attribute"). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains - and in some cases enhances - both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors., Comment: ACL 2024

Active nematics exhibit spontaneous flows through a well-known linear instability of the uniformly-aligned quiescent state. Here we show that even a linearly stable uniform state can experience a nonlinear instability, resulting in a discontinuous transition to spontaneous flows. In this case, quiescent and flowing states may coexist. Through a weakly nonlinear analysis and a numerical study, we trace the bifurcation diagram of striped patterns and show that the underlying pitchfork bifurcation switches from supercritical (continuous) to subcritical (discontinuous) by varying the flow-alignment parameter. We predict that the discontinuous spontaneous flow transition occurs for a wide range of parameters, including systems of contractile flow-aligning rods. Our predictions are relevant to active nematic turbulence and can potentially be tested in experiments on either cell layers or active cytoskeletal suspensions.

Grounded text generation, encompassing tasks such as long-form question-answering and summarization, necessitates both content selection and content consolidation. Current end-to-end methods are difficult to control and interpret due to their opaqueness. Accordingly, recent works have proposed a modular approach, with separate components for each step. Specifically, we focus on the second subtask, of generating coherent text given pre-selected content in a multi-document setting. Concretely, we formalize Fusion-in-Context (FiC) as a standalone task, whose input consists of source texts with highlighted spans of targeted content. A model then needs to generate a coherent passage that includes all and only the target information. Our work includes the development of a curated dataset of 1000 instances in the reviews domain, alongside a novel evaluation framework for assessing the faithfulness and coverage of highlights, which strongly correlate to human judgment. Several baseline models exhibit promising outcomes and provide insightful analyses. This study lays the groundwork for further exploration of modular text generation in the multi-document setting, offering potential improvements in the quality and reliability of generated content. Our benchmark, FuseReviews, including the dataset, evaluation framework, and designated leaderboard, can be found at https://fusereviews.github.io/., Comment: NAACL 2024, findings

Restoration, generalization, and dimensionality reduction of a vector field from samples are the most common and crucial tasks in dynamical system analysis. An optimization-based algorithm to fulfill these tasks is suggested. Given noisy, sparse, or redundant sampled vector fields, the optimization process encapsulates the inherent geometry of the dynamical system derived from the Koopman eigenfunction space. The dynamic geometry is revealed via the exact penalty method, compromising accuracy and smoothness. This algorithm is backed up by promising results of denoising and generalization with a concise dynamics representation leading to dimensionality reduction.

The mod-2 arithmetic Milnor invariants, introduced by Morishita, provide a decomposition law for primes in canonical Galois extensions of $\mathbb{Q}$ with unitriangular Galois groups, and contain the Legendre and Redei symbols as special cases. Morishita further proposed a notion of mod-q arithmetic Milnor invariants, where q is a prime power, for number fields containing the q-th roots of unity and satisfying certain class field theory assumptions. We extend this theory from the number field context to general fields, by introducing a notion of a linking invariant for discrete valuations and orderings. We further express it as a Magnus homomorphism coefficient, and relate it to Massey product elements in Galois cohomology.

Background: Immune checkpoint inhibitors (ICIs) have revolutionized cancer treatment but can result in severe immune-related adverse events (IrAEs). Monitoring IrAEs on a large scale is essential for personalized risk profiling and assisting in treatment decisions. Methods: In this study, we conducted an analysis of clinical notes from patients who received ICIs at the Tel Aviv Sourasky Medical Center. By employing a Natural Language Processing algorithmic pipeline, we systematically identified seven common or severe IrAEs. We examined the utilization of corticosteroids, treatment discontinuation rates following IrAEs, and constructed survival curves to visualize the occurrence of adverse events during treatment. Results: Our analysis encompassed 108,280 clinical notes associated with 1,635 patients who had undergone ICI therapy. The detected incidence of IrAEs was consistent with previous reports, exhibiting substantial variation across different ICIs. Treatment with corticosteroids varied depending on the specific IrAE, ranging from 17.3% for thyroiditis to 57.4% for myocarditis. Our algorithm demonstrated high accuracy in identifying IrAEs, as indicated by an area under the curve (AUC) of 0.89 for each suspected note and F1 scores of 0.87 or higher for five out of the seven IrAEs examined at the patient level. Conclusions: This study presents a novel, large-scale monitoring approach utilizing deep neural networks for IrAEs. Our method provides accurate results, enhancing understanding of detrimental consequences experienced by ICI-treated patients. Moreover, it holds potential for monitoring other medications, enabling comprehensive post-marketing surveillance to identify susceptible populations and establish personalized drug safety profiles.

Cluster states are key resources for measurement-based quantum information processing. Photonic cluster and graph states, in particular, play indispensable roles in quantum network and quantum metrology. We demonstrate a semiconductor quantum dot based device in which the confined hole spin acts as a needle in a quantum knitting machine producing continuously and deterministically at sub-Gigahertz repetition rate single indistinguishable photons which are all polarization entangled to each other and to the spin in a one dimensional cluster state. By projecting two nonadjacent photons onto circular polarization bases we disentangle the spin from the photons emitted in between, thus continuously and deterministically preparing all-photonic cluster states for the first time. We use polarization tomography on four sequentially detected photons to demonstrate and to directly quantify the robustness of the cluster's entanglement and the determinism in its photon generation., Comment: 8 pages, 4 figures, 1 table

Magnetic confinement fusion reactors produce ash particles that must be removed for efficient operation. It is suggested to use autoresonance (a continuous phase-locking between anharmonic motion and a chirped drive) to remove the ash particles from a magnetic mirror, the simplest magnetic confinement configuration. An analogy to the driven pendulum is established via the guiding center approximation. The full 3D dynamics is simulated for $\alpha$ particles (the byproduct of DT fusion) in agreement with the approximated 1D model. Monte Carlo simulations sampling the phase space of initial conditions are used to quantify the efficiency of the method. The DT fuel particles are out of the bandwidth of the chirped drive and, therefore, stay in the mirror for ongoing fusion. The method is also applicable for advanced, aneutronic reactors, such as p-$^{11}$B., Comment: 5 pages, 4 figures

Naming is very important in software development, as names are often the only vehicle of meaning about what the code is intended to do. A recent study on how developers choose names collected the names given by different developers for the same objects. This enabled a study of these names' diversity and structure, and the construction of a model of how names are created. We reproduce different parts of this study in three independent experiments. Importantly, we employ methodological variations rather than striving of an exact replication. When the same results are obtained this then boosts our confidence in their validity by demonstrating that they do not depend on the methodology. Our results indeed corroborate those of the original study in terms of the diversity of names, the low probability of two developers choosing the same name, and the finding that experienced developers tend to use slightly longer names than inexperienced students. We explain name diversity by performing a new analysis of the names, classifying the concepts represented in them as universal (agreed upon), alternative (reflecting divergent views on a topic), or optional (reflecting divergent opinions on whether to include this concept at all). This classification enables new research directions concerning the considerations involved in naming decisions. We also show that explicitly using the model proposed in the original study to guide naming leads to the creation of better names, whereas the simpler approach of just asking participants to use longer and more detailed names does not., Comment: 35 pages with 10 figures and 6 tables