Search

In this paper, we report on communication experiments conducted in the summer of 2022 during a deep dive to the wreck of the Titanic. Radio transmission is not possible in deep sea water, and communication links rely on sonar signals. Due to the low bandwidth of sonar signals and the need to communicate readable data, text messaging is used in deep-sea missions. In this paper, we report results and experiences from a messaging system that converts speech to text in a submarine, sends text messages to the surface, and reconstructs those messages as synthetic lip-synchronous videos of the speakers. The resulting system was tested during an actual dive to Titanic in the summer of 2022. We achieved an acceptable latency for a system of such complexity as well as good quality. The system demonstration video can be found at the following link: https://youtu.be/C4lyM86-5Ig

LLMs are increasingly being used in workflows involving generating content to be consumed by humans (e.g., marketing) and also in directly interacting with humans (e.g., through chatbots). The development of such systems that are capable of generating verifiably persuasive messages presents both opportunities and challenges for society. On the one hand, such systems could positively impact domains like advertising and social good, such as addressing drug addiction, and on the other, they could be misused for spreading misinformation and shaping political opinions. To channel LLMs' impact on society, we need to develop systems to measure and benchmark their persuasiveness. With this motivation, we introduce PersuasionBench and PersuasionArena, the first large-scale benchmark and arena containing a battery of tasks to measure the persuasion ability of generative models automatically. We investigate to what extent LLMs know and leverage linguistic patterns that can help them generate more persuasive language. Our findings indicate that the persuasiveness of LLMs correlates positively with model size, but smaller models can also be made to have a higher persuasiveness than much larger models. Notably, targeted training using synthetic and natural datasets significantly enhances smaller models' persuasive capabilities, challenging scale-dependent assumptions. Our findings carry key implications for both model developers and policymakers. For instance, while the EU AI Act and California's SB-1047 aim to regulate AI models based on the number of floating point operations, we demonstrate that simple metrics like this alone fail to capture the full scope of AI's societal impact. We invite the community to explore and contribute to PersuasionArena and PersuasionBench, available at https://bit.ly/measure-persuasion, to advance our understanding of AI-driven persuasion and its societal implications.

In this paper we show how different sources of random numbers influence the outcomes of Monte Carlo simulations. We compare industry-standard pseudo-random number generators (PRNGs) to a quantum random number generator (QRNG) and show, using examples of Monte Carlo simulations with exact solutions, that the QRNG yields statistically significantly better approximations than the PRNGs. Our results demonstrate that higher accuracy can be achieved in the commonly known Monte Carlo method for approximating $\pi$. For Buffon's needle experiment, we further quantify a potential reduction in approximation errors by up to $1.89\times$ for optimal parameter choices when using a QRNG and a reduction of the sample size by $\sim 8\times$ for sub-optimal parameter choices. We attribute the observed higher accuracy to the underlying differences in the random sampling, where a uniformity analysis reveals a tendency of the QRNG to sample the solution space more homogeneously. Additionally, we compare the results obtained with the QRNG and PRNG in solving the non-linear stochastic Schr\"odinger equation, benchmarked against the analytical solution. We observe higher accuracy of the approximations of the QRNG and demonstrate that equivalent results can be achieved at 1/3 to 1/10-th of the costs.

A simple Hamiltonian modeling framework for general models in nonlinear optics is given. This framework is specialized to describe the Hamiltonian structure of electromagnetic phenomena in cubicly nonlinear optical media. The model has a simple Poisson bracket structure with the Hamiltonian encoding all of the nonlinear coupling of the fields. The field-independence of the Poisson bracket facilitates a straightforward Hamiltonian structure-preserving discretization using finite element exterior calculus. The generality and relative simplicity of this Hamiltonian framework makes it amenable for simulating a broad class of time-domain nonlinear optical problems. The main contribution of this work is a finite element discretization of Maxwell's equations in cubicly nonlinear media which is energy-stable and exactly conserves Gauss's laws. Moreover, this approach may be readily adapted to consider more general nonlinear media in subsequent work.

Hyperparameter optimization is a crucial problem in Evolutionary Computation. In fact, the values of the hyperparameters directly impact the trajectory taken by the optimization process, and their choice requires extensive reasoning by human operators. Although a variety of self-adaptive Evolutionary Algorithms have been proposed in the literature, no definitive solution has been found. In this work, we perform a preliminary investigation to automate the reasoning process that leads to the choice of hyperparameter values. We employ two open-source Large Language Models (LLMs), namely Llama2-70b and Mixtral, to analyze the optimization logs online and provide novel real-time hyperparameter recommendations. We study our approach in the context of step-size adaptation for (1+1)-ES. The results suggest that LLMs can be an effective method for optimizing hyperparameters in Evolution Strategies, encouraging further research in this direction., Comment: Proceedings of the GECCO'24 Companion: Genetic and Evolutionary Computation Conference Companion

Robustness and generalizability are the key properties of artificial neural network (ANN)-based controllers for maintaining a reliable performance in case of changes. It is demonstrated that exposing the ANNs to variations during training processes can improve their robustness and generalization capabilities. However, the way in which this variation is introduced can have a significant impact. In this paper, we define various training schedules to specify how these variations are introduced during an evolutionary learning process. In particular, we focus on morphological robustness and generalizability concerned with finding an ANN-based controller that can provide sufficient performance on a range of physical variations. Then, we perform an extensive analysis of the effect of these training schedules on morphological generalization. Furthermore, we formalize the process of training sample selection (i.e., morphological variations) to improve generalization as a reinforcement learning problem. Overall, our results provide deeper insights into the role of variability and the ways of enhancing the generalization property of evolved ANN-based controllers.

The widespread use of Generative Artificial Intelligence (GAI) among teenagers has led to significant misuse and safety concerns. To identify risks and understand parental controls challenges, we conducted a content analysis on Reddit and interviewed 20 participants (seven teenagers and 13 parents). Our study reveals a significant gap in parental awareness of the extensive ways children use GAI, such as interacting with character-based chatbots for emotional support or engaging in virtual relationships. Parents and children report differing perceptions of risks associated with GAI. Parents primarily express concerns about data collection, misinformation, and exposure to inappropriate content. In contrast, teenagers are more concerned about becoming addicted to virtual relationships with GAI, the potential misuse of GAI to spread harmful content in social groups, and the invasion of privacy due to unauthorized use of their personal data in GAI applications. The absence of parental control features on GAI platforms forces parents to rely on system-built controls, manually check histories, share accounts, and engage in active mediation. Despite these efforts, parents struggle to grasp the full spectrum of GAI-related risks and to perform effective real-time monitoring, mediation, and education. We provide design recommendations to improve parent-child communication and enhance the safety of GAI use., Comment: 16 pages

We explore exact generalized symmetries in the standard 2+1d lattice $\mathbb{Z}_2$ gauge theory coupled to the Ising model, and compare them with their continuum field theory counterparts. One model has a (non-anomalous) non-invertible symmetry, and we identify two distinct non-invertible symmetry protected topological phases. The non-invertible algebra involves a lattice condensation operator, which creates a toric code ground state from a product state. Another model has a mixed anomaly between a 1-form symmetry and an ordinary symmetry. This anomaly enforces a nontrivial transition in the phase diagram, consistent with the "Higgs=SPT" proposal. Finally, we discuss how the symmetries and anomalies in these two models are related by gauging, which is a 2+1d version of the Kennedy-Tasaki transformation., Comment: 33 pages plus appendices

In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics., Comment: CVPR2024 NTIRE Workshop

Communication is defined as "Who says what to whom with what effect". A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the message, the behavior data is often ignored while training large language models. We show that training LLMs on receiver behavior can actually help improve their content-understanding abilities. Specifically, we show that training LLMs to predict the receiver behavior of likes and comments improves the LLM's performance on a wide variety of downstream content understanding tasks. We show this performance increase over 46 video and image understanding tasks over 26 benchmark datasets across both 0-shot and fine-tuning settings, outperforming many supervised baselines. Moreover, since receiver behavior, such as likes and comments, is collected by default on the internet and does not need any human annotations to be useful, the performance improvement we get after training on this data is essentially free-lunch. We release the receiver behavior cleaned comments and likes of 750k images and videos collected from multiple platforms along with our instruction-tuning data.

Normative non-functional requirements specify constraints that a system must observe in order to avoid violations of social, legal, ethical, empathetic, and cultural norms. As these requirements are typically defined by non-technical system stakeholders with different expertise and priorities (ethicists, lawyers, social scientists, etc.), ensuring their well-formedness and consistency is very challenging. Recent research has tackled this challenge using a domain-specific language to specify normative requirements as rules whose consistency can then be analysed with formal methods. In this paper, we propose a complementary approach that uses Large Language Models to extract semantic relationships between abstract representations of system capabilities. These relations, which are often assumed implicitly by non-technical stakeholders (e.g., based on common sense or domain knowledge), are then used to enrich the automated reasoning techniques for eliciting and analyzing the consistency of normative requirements. We show the effectiveness of our approach to normative requirements elicitation and operationalization through a range of real-world case studies.

Generative Adversarial Networks (GANs) have shown great success in generating high quality images and are thus used as one of the main approaches to generate art images. However, usually the image generation process involves sampling from the latent space of the learned art representations, allowing little control over the output. In this work, we first employ GANs that are trained to produce creative images using an architecture known as Creative Adversarial Networks (CANs), then, we employ an evolutionary approach to navigate within the latent space of the models to discover images. We use automatic aesthetic and collaborative interactive human evaluation metrics to assess the generated images. In the human interactive evaluation case, we propose a collaborative evaluation based on the assessments of several participants. Furthermore, we also experiment with an intelligent mutation operator that aims to improve the quality of the images through local search based on an aesthetic measure. We evaluate the effectiveness of this approach by comparing the results produced by the automatic and collaborative interactive evolution. The results show that the proposed approach can generate highly attractive art images when the evolution is guided by collaborative human feedback., Comment: Preprint. The Version of Record of this contribution is to be published in the proceedings of the 13th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART) 2024

Reinforcement learning (RL) based autonomous driving has emerged as a promising alternative to data-driven imitation learning approaches. However, crafting effective reward functions for RL poses challenges due to the complexity of defining and quantifying good driving behaviors across diverse scenarios. Recently, large pretrained models have gained significant attention as zero-shot reward models for tasks specified with desired linguistic goals. However, the desired linguistic goals for autonomous driving such as "drive safely" are ambiguous and incomprehensible by pretrained models. On the other hand, undesired linguistic goals like "collision" are more concrete and tractable. In this work, we introduce LORD, a novel large models based opposite reward design through undesired linguistic goals to enable the efficient use of large pretrained models as zero-shot reward models. Through extensive experiments, our proposed framework shows its efficiency in leveraging the power of large pretrained models for achieving safe and enhanced autonomous driving. Moreover, the proposed approach shows improved generalization capabilities as it outperforms counterpart methods across diverse and challenging driving scenarios.

As deep neural networks evolve from convolutional neural networks (ConvNets) to advanced vision transformers (ViTs), there is an increased need to eliminate redundant data for faster processing without compromising accuracy. Previous methods are often architecture-specific or necessitate re-training, restricting their applicability with frequent model updates. To solve this, we first introduce a novel property of lightweight ConvNets: their ability to identify key discriminative patch regions in images, irrespective of model's final accuracy or size. We demonstrate that fully-connected layers are the primary bottleneck for ConvNets performance, and their suppression with simple weight recalibration markedly enhances discriminative patch localization performance. Using this insight, we introduce PaPr, a method for substantially pruning redundant patches with minimal accuracy loss using lightweight ConvNets across a variety of deep learning architectures, including ViTs, ConvNets, and hybrid transformers, without any re-training. Moreover, the simple early-stage one-step patch pruning with PaPr enhances existing patch reduction methods. Through extensive testing on diverse architectures, PaPr achieves significantly higher accuracy over state-of-the-art patch reduction methods with similar FLOP count reduction. More specifically, PaPr reduces about 70% of redundant patches in videos with less than 0.8% drop in accuracy, and up to 3.7x FLOPs reduction, which is a 15% more reduction with 2.5% higher accuracy. Code is released at https://github.com/tanvir-utexas/PaPr., Comment: Accepted in ECCV 2024. Code: https://github.com/tanvir-utexas/PaPr

Autonomous driving stands as a pivotal domain in computer vision, shaping the future of transportation. Within this paradigm, the backbone of the system plays a crucial role in interpreting the complex environment. However, a notable challenge has been the loss of clear supervision when it comes to Bird's Eye View elements. To address this limitation, we introduce CLIP-BEVFormer, a novel approach that leverages the power of contrastive learning techniques to enhance the multi-view image-derived BEV backbones with ground truth information flow. We conduct extensive experiments on the challenging nuScenes dataset and showcase significant and consistent improvements over the SOTA. Specifically, CLIP-BEVFormer achieves an impressive 8.5\% and 9.2\% enhancement in terms of NDS and mAP, respectively, over the previous best BEV model on the 3D object detection task., Comment: CVPR 2024

The prevalence of smart devices with the ability to capture moments in multiple modalities has enabled users to experience multimodal information online. However, large Language (LLMs) and Vision models (LVMs) are still limited in capturing holistic meaning with cross-modal semantic relationships. Without explicit, common sense knowledge (e.g., as a knowledge graph), Visual Language Models (VLMs) only learn implicit representations by capturing high-level patterns in vast corpora, missing essential contextual cross-modal cues. In this work, we design a framework to couple explicit commonsense knowledge in the form of knowledge graphs with large VLMs to improve the performance of a downstream task, predicting the effectiveness of multi-modal marketing campaigns. While the marketing application provides a compelling metric for assessing our methods, our approach enables the early detection of likely persuasive multi-modal campaigns and the assessment and augmentation of marketing theory.

Table understanding capability of Large Language Models (LLMs) has been extensively studied through the task of question-answering (QA) over tables. Typically, only a small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due to the vulnerability of LLMs to noise. To mitigate this, we propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) - a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question-answering LLM (QA LLM). To further aid the relevance scorer, CABINET employs a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets at https://github.com/Sohanpatnaik106/CABINET_QA., Comment: Accepted at ICLR 2024 (spotlight)

Nonreciprocal interactions break action-reaction symmetry in systems of interacting bodies. This process inevitably introduces non-Hermitian dynamics which with its hallmark signature called exceptional points (EPs) has been a subject of intense research across different disciplines ranging from photonics to metamaterials. Whether non-Hermiticity and EPs are a fundamental property of nature and if so, how nature utilizes them to gain competitive advantage have remained largely unanswered. Although biological systems feature many examples of non-reciprocal interactions with the potential to drive non-Hermitian dynamics, these are often theoretically overlooked and not experimentally investigated. Here, we demonstrate in an active matter composed of social animal Caenorhabditis elegans and bacteria, non-Hermitian dynamics, and the emergence of EPs owing to the nonreciprocal nature of oxygen sensing, nonequilibrium interfacial current, and bacterial consumption. We observed that when driven through the EP, the system collectively breaks parity-time (PT) symmetry leading to traveling waves and arrested phase separation. We further find that these features enable the collective ability to localize interfaces between broken and exact PT-phases. Remarkably, this ability provides a strong evolutionary advantage to animals living in soil. Altogether our results provide mechanistic insights into the detailed symmetries controlling the collective response of biological systems; answer a long-standing problem; and give an example of the EP-enabled dynamics in a biological system.

Quantization techniques commonly reduce the inference costs of neural networks by restricting the precision of weights and activations. Recent studies show that also reducing the precision of the accumulator can further improve hardware efficiency at the risk of numerical overflow, which introduces arithmetic errors that can degrade model accuracy. To avoid numerical overflow while maintaining accuracy, recent work proposed accumulator-aware quantization (A2Q), a quantization-aware training method that constrains model weights during training to safely use a target accumulator bit width during inference. Although this shows promise, we demonstrate that A2Q relies on an overly restrictive constraint and a sub-optimal weight initialization strategy that each introduce superfluous quantization error. To address these shortcomings, we introduce: (1) an improved bound that alleviates accumulator constraints without compromising overflow avoidance; and (2) a new strategy for initializing quantized weights from pre-trained floating-point checkpoints. We combine these contributions with weight normalization to introduce A2Q+. We support our analysis with experiments that show A2Q+ significantly improves the trade-off between accumulator bit width and model accuracy and characterize new trade-offs that arise as a consequence of accumulator constraints.

As software systems increasingly interact with humans in application domains such as transportation and healthcare, they raise concerns related to the social, legal, ethical, empathetic, and cultural (SLEEC) norms and values of their stakeholders. Normative non-functional requirements (N-NFRs) are used to capture these concerns by setting SLEEC-relevant boundaries for system behavior. Since N-NFRs need to be specified by multiple stakeholders with widely different, non-technical expertise (ethicists, lawyers, regulators, end users, etc.), N-NFR elicitation is very challenging. To address this challenge, we introduce N-Check, a novel tool-supported formal approach to N-NFR analysis and debugging. N-Check employs satisfiability checking to identify a broad spectrum of N-NFR well-formedness issues (WFI), such as conflicts, redundancy, restrictiveness, insufficiency, yielding diagnostics which pinpoint their causes in a user-friendly way that enables non-technical stakeholders to understand and fix them. We show the effectiveness and usability of our approach through nine case studies in which teams of ethicists, lawyers, philosophers, psychologists, safety analysts, and engineers used N-Check to analyse and debug 233 N-NFRs comprising 62 issues for the software underpinning the operation of systems ranging from assistive-care robots and tree-disease detection drones to manufacturing collaborative robots.

Autonomous driving is a complex and challenging task that aims at safe motion planning through scene understanding and reasoning. While vision-only autonomous driving methods have recently achieved notable performance, through enhanced scene understanding, several key issues, including lack of reasoning, low generalization performance and long-tail scenarios, still need to be addressed. In this paper, we present VLP, a novel Vision-Language-Planning framework that exploits language models to bridge the gap between linguistic understanding and autonomous driving. VLP enhances autonomous driving systems by strengthening both the source memory foundation and the self-driving car's contextual understanding. VLP achieves state-of-the-art end-to-end planning performance on the challenging NuScenes dataset by achieving 35.9\% and 60.5\% reduction in terms of average L2 error and collision rates, respectively, compared to the previous best method. Moreover, VLP shows improved performance in challenging long-tail scenarios and strong generalization capabilities when faced with new urban environments., Comment: CVPR2024

Let $k$ be a positive integer and let $G$ be a graph with $n$ vertices. A connected $k$-subpartition of $G$ is a collection of $k$ pairwise disjoint sets (a.k.a. classes) of vertices in $G$ such that each set induces a connected subgraph. The connected $k$-partition polytope of $G$, denoted by $P(G,k)$, is defined as the convex hull of the incidence vectors of all connected $k$-subpartitions of $G$. Many applications arising in off-shore oil-drilling, forest planning, image processing, cluster analysis, political districting, police patrolling, and biology are modeled in terms of finding connected (sub)partitions of a graph. This study focus on the facial structure of $P(G,k)$ and the computational complexity of the corresponding separation problems. We first propose a set of valid inequalities having non-null coefficients associated with a single class that extends and generalizes the ones in the literature of related problems, show sufficient conditions for these inequalities to be facet-defining, and design a polynomial-time separation algorithm for them. We also devise two sets of inequalities that consider multiple classes, prove when they define facets, and study the computational complexity of associated separation problems.

The problem of scheduling conflicting jobs on parallel machines consists in assigning a set of jobs to a set of machines so that no two conflicting jobs are allocated to the same machine, and the maximum processing time among all machines is minimized. We propose a new compact mixed integer linear formulation based on the representatives model for the vertex coloring problem, which overcomes a number of issues inherent in the natural assignment model. We present a polyhedral study of the associated polytope, and describe classes of valid inequalities inherited from the stable set polytope. We describe branch-and-cut algorithms for the problem, and report on computational experiments with benchmark instances. Our computational results on the hardest instances of the benchmark set show that the proposed algorithms are superior (either in running time or quality of the solutions) to the current state-of-the-art methods. We find that our new method performs better than the existing ones especially when the gap between the optimal value and the trivial lower bound (i.e., the sum of all processing times divided by the number of machines) increases.