Search

Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents., Comment: preprint, 9 pages

Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute.

Large multimodal language models have shown remarkable proficiency in understanding and editing images. However, a majority of these visually-tuned models struggle to comprehend the textual content embedded in images, primarily due to the limitation of training data. In this work, we introduce TRINS: a Text-Rich image INStruction dataset, with the objective of enhancing the reading ability of the multimodal large language model. TRINS is built upon LAION using hybrid data annotation strategies that include machine-assisted and human-assisted annotation processes. It contains 39,153 text-rich images, captions, and 102,437 questions. Specifically, we show that the number of words per annotation in TRINS is significantly longer than that of related datasets, providing new challenges. Furthermore, we introduce a simple and effective architecture, called a Language-vision Reading Assistant (LaRA), which is good at understanding textual content within images. LaRA outperforms existing state-of-the-art multimodal large language models on the TRINS dataset, as well as other classical benchmarks. Lastly, we conducted a comprehensive evaluation with TRINS on various text-rich image understanding and generation tasks, demonstrating its effectiveness., Comment: CVPR 2024

The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models., Comment: In COLM 2024

Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning., Comment: Technical Report; The first two authors contributed equally

We study the top-Higgs coupling with a CP violating phase $\xi$ at a future multi-TeV muon collider. We focus on processes that are directly sensitive to the top quark Yukawa coupling: $t\bar{t}h$, $tbh\mu\nu$, and $t\bar{t}h\nu\bar{\nu}$ with $h\rightarrow b\bar{b}$ and semileptonic top decays. At different energies, different processes dominate the cross section, providing complementary information. At and above an energy of $\mathcal{O}(10)$ TeV, vector boson fusion processes dominate. As we show, in the Standard Model there is destructive interference in the vector boson fusion processes $t\bar{t}h\nu\bar{\nu}$ and $tbh\mu\nu$ between the top quark Yukawa and Higgs-gauge boson couplings. A CP-violating phase changes this interference, and the cross section measurement is very sensitive to the size of the CP-violating angle. Although we find that the cross sections are measured to $\mathcal{O}(50\%)$ statistical uncertainty at $1\sigma$, a 10 and 30 TeV muon collider can bound the CP-violating angle $|\xi|\lesssim9.0^\circ$ and $|\xi|\lesssim5.4^\circ$, respectively. However, cross section measurements are insensitive to the sign of the CP-violating angle. To determine that the coupling is truly CP violating, observables sensitive to CP-violation must be measured. We find in the $t\bar{t}h$ process the azimuthal angle between the $t+\bar{t}$ plane and the initial state muon+Higgs plane shows good discrimination for $\xi=\pm0.1\pi$. For the $tbh\mu\nu$ and $t\bar{t}h\nu\bar{\nu}$ processes, the operator proportional to $\left(\vec{p}_\mu\times\vec{p}_h\right)\cdot \vec{p}_t$ is sensitive to the sign of CP phase $\xi$. From these observables, we construct asymmetry parameters that show good distinction between different values and signs of the CP violating angle., Comment: v2: Matches published version, 33 pages, 11 figures, typos fixed, references added, discussion expanded, results unchanged; v1: 32 pages, 11 figures

Large language model (LLM) agents have been shown effective on a wide range of tasks, and by ensembling multiple LLM agents, their performances could be further improved. Existing approaches employ a fixed set of agents to interact with each other in a static architecture, which limits their generalizability to various tasks and requires strong human prior in designing these agents. In this work, we propose to construct a strategic team of agents communicating in a dynamic interaction architecture based on the task query. Specifically, we build a framework named Dynamic LLM-Agent Network ($\textbf{DyLAN}$) for LLM-agent collaboration on complicated tasks like reasoning and code generation. DyLAN enables agents to interact for multiple rounds in a dynamic architecture with inference-time agent selection and an early-stopping mechanism to improve performance and efficiency. We further design an automatic agent team optimization algorithm based on an unsupervised metric termed $\textit{Agent Importance Score}$, enabling the selection of best agents based on the contribution each agent makes. Empirically, we demonstrate that DyLAN performs well in both reasoning and code generation tasks with reasonable computational cost. DyLAN achieves 13.0% and 13.3% improvement on MATH and HumanEval, respectively, compared to a single execution on GPT-35-turbo. On specific subjects of MMLU, agent team optimization in DyLAN increases accuracy by up to 25.0%., Comment: Preprint, under review. 21 pages

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/., Comment: Preprint

The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model's generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research. We have publicly released our code at https://github.com/SALT-NLP/Bound-Cap-LLM., Comment: 27 pages, 13 figures, 11 tables, to be published in EACL 2023 Findings

Text-to-image models, which can generate high-quality images based on textual input, have recently enabled various content-creation tools. Despite significantly affecting a wide range of downstream applications, the distributions of these generated images are still not fully understood, especially when it comes to the potential stereotypical attributes of different genders. In this work, we propose a paradigm (Gender Presentation Differences) that utilizes fine-grained self-presentation attributes to study how gender is presented differently in text-to-image models. By probing gender indicators in the input text (e.g., "a woman" or "a man"), we quantify the frequency differences of presentation-centric attributes (e.g., "a shirt" and "a dress") through human annotation and introduce a novel metric: GEP. Furthermore, we propose an automatic method to estimate such differences. The automatic GEP metric based on our approach yields a higher correlation with human annotations than that based on existing CLIP scores, consistently across three state-of-the-art text-to-image models. Finally, we demonstrate the generalization ability of our metrics in the context of gender stereotypes related to occupations., Comment: Preprint, 23 pages, 14 figures. Project page at https://salt-nlp.github.io/GEP/

Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario. Simply augmenting the input with some demonstrations can significantly improve performance on few-shot NER. However, why such demonstrations are beneficial for the learning process remains unclear since there is no explicit alignment between the demonstrations and the predictions. In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling and show that (1) demonstrations composed of random tokens still make the model a better few-shot learner; (2) the length of random demonstrations and the relevance of random tokens are the main factors affecting the performance; (3) demonstrations increase the confidence of model predictions on captured superficial patterns. We have publicly released our code at https://github.com/SALT-NLP/RobustDemo., Comment: 14 pages, EMNLP 2022 Main Conference

The increasing numbers of rocky, terrestrial exoplanets known to orbit nearby stars (especially M dwarfs) has drawn increased attention to the possibility of studying these planets' surface properties, and atmospheric compositions & escape histories. Here we report the detection of the secondary eclipse of the terrestrial exoplanet GJ1252b using the Spitzer Space Telescope's IRAC2 4.5 micron channel. We measure an eclipse depth of 149(+25/-32) ppm, corresponding to a day-side brightness temperature of 1410(+91/-125) K and consistent with the prediction for no atmosphere. Comparing our measurement to atmospheric models indicates that GJ1252b has a surface pressure of <10 bar, substantially less than Venus. Assuming energy-limited escape, even a 100 bar atmosphere would be lost in <1 Myr, far shorter than estimated age of 3.9+/-0.4 Gyr. The expected mass loss could be overcome by mantle outgassing, but only if the mantle's carbon content were >7% by mass - over two orders of magnitude greater than that found in Earth. We therefore conclude that GJ1252b has no significant atmosphere. Model spectra with granitoid or feldspathic surface composition, but with no atmosphere, are disfavored at >2 sigma. The eclipse occurs just +1.4(+2.8/-1.0) min after orbital phase 0.5, indicating e cos omega=+0.0025(+0.0049/-0.0018), consistent with a circular orbit. Tidal heating is therefore likely to be negligible to GJ1252b's global energy budget. Finally, we also analyze additional, unpublished TESS transit photometry of GJ1252b which improves the precision of the transit ephemeris by a factor of ten, provides a more precise planetary radius of 1.180+/-0.078 R_E, and rules out any transit timing variations with amplitudes <1 min., Comment: ApJL in press. 16 pages, 12 figures, 10 eclipses, 1 bandpass. Models will be available at journal website

Named Entity Recognition (NER) systems often demonstrate great performance on in-distribution data, but perform poorly on examples drawn from a shifted distribution. One way to evaluate the generalization ability of NER models is to use adversarial examples, on which the specific variations associated with named entities are rarely considered. To this end, we propose leveraging expert-guided heuristics to change the entity tokens and their surrounding contexts thereby altering their entity types as adversarial attacks. Using expert-guided heuristics, we augmented the CoNLL 2003 test set and manually annotated it to construct a high-quality challenging set. We found that state-of-the-art NER systems trained on CoNLL 2003 training data drop performance dramatically on our challenging set. By training on adversarial augmented training examples and using mixup for regularization, we were able to significantly improve the performance on the challenging set as well as improve out-of-domain generalization which we evaluated by using OntoNotes data. We have publicly released our dataset and code at https://github.com/GT-SALT/Guided-Adversarial-Augmentation., Comment: ACL 2022 (Findings)

Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks. Existing work on continual sequence generation either always reuses existing parameters to learn new tasks, which is vulnerable to catastrophic forgetting on dissimilar tasks, or blindly adds new parameters for every new task, which could prevent knowledge sharing between similar tasks. To get the best of both worlds, in this work, we propose continual sequence generation with adaptive compositional modules to adaptively add modules in transformer architectures and compose both old and new modules for new tasks. We also incorporate pseudo experience replay to facilitate knowledge transfer in those shared modules. Experiment results on various sequences of generation tasks show that our framework can adaptively add modules or reuse modules based on task similarity, outperforming state-of-the-art baselines in terms of both performance and parameter efficiency. We make our code public at https://github.com/GT-SALT/Adaptive-Compositional-Modules., Comment: 15 pages, ACL 2022

Constraining the Higgs boson properties is a cornerstone of the LHC program and future colliders. In this Snowmass contribution, we study the potential to directly probe the Higgs-top CP-structure via the $t\bar{t}h$ production at the HL-LHC, 100 TeV FCC and muon colliders. We find the limits on the CP phase ($\alpha$) at 95% CL are $|\alpha| \lesssim 36^\circ$ with dileptonic $t\bar t (h\to b\bar b) $ and $|\alpha| \lesssim 25^\circ$ with combined $t\bar t (h\to \gamma\gamma) $ at the HL-LHC. The 100 TeV FCC brings a significant improvement in sensitivity with $|\alpha| \lesssim 3^\circ$ for the dileptonic $t\bar t (h\to b\bar b) $, due to the remarkable gain in the signal cross-section and the increased luminosity. At future muon colliders, we find that the bounds with semileptonic $t\bar t (h\to b\bar b) \nu\bar\nu$ are $|\alpha| \lesssim 9^\circ$ for 10 TeV and $|\alpha| \lesssim 3^\circ$ for 30 TeV, respectively., Comment: 12 pages, 8 figures, contribution to Snowmass 2021

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR., Comment: NAACL 2021