Search

Multimodal foundation models (MFMs) such as OFASys show the potential to unlock analysis of complex data such as images, videos, and audio data via text prompts alone. However, their performance may suffer in the face of text input that differs even slightly from their training distribution, which is surprising considering the use of modality-specific data to "ground" the text input. This study demonstrates that prompt instability is a major concern for MFMs, leading to a consistent drop in performance across all modalities, but that instability can be mitigated with additional training with augmented data. We evaluate several methods for grounded prompt perturbation, where we generate perturbations and filter based on similarity to text and/or modality data. After re-training the models on the augmented data, we find improved accuracy and more stable performance on the perturbed test data regardless of perturbation condition, suggesting that the data augmentation strategy helps the models handle domain shifts more effectively. In error analysis, we find consistent patterns of performance improvement across domains, suggesting that retraining on prompt perturbations tends to help general reasoning capabilities in MFMs., Comment: in submission

We classify connected 3-node restricted excitatory-inhibitory networks, extending our previous paper (`Classification of 2-node Excitatory-Inhibitory Networks', Mathematical Biosciences 373 (2024) 109205). We assume that there are two node-types and two arrow-types, excitatory and inhibitory; all excitatory arrows are identical and all inhibitory arrows are identical; and excitatory (resp. inhibitory) nodes can only output excitatory (resp. inhibitory) arrows. The classification is performed under the following two network perspectives: ODE-equivalence and minimality; and valence less or equal to 2. The results of this and the previous work constitute a first step towards analysing dynamics and bifurcations of excitatory-inhibitory networks and have potential applications to biological network models., Comment: arXiv admin note: text overlap with arXiv:2403.02869

Multimodal models are expected to be a critical component to future advances in artificial intelligence. This field is starting to grow rapidly with a surge of new design elements motivated by the success of foundation models in natural language processing (NLP) and vision. It is widely hoped that further extending the foundation models to multiple modalities (e.g., text, image, video, sensor, time series, graph, etc.) will ultimately lead to generalist multimodal models, i.e. one model across different data modalities and tasks. However, there is little research that systematically analyzes recent multimodal models (particularly the ones that work beyond text and vision) with respect to the underling architecture proposed. Therefore, this work provides a fresh perspective on generalist multimodal models (GMMs) via a novel architecture and training configuration specific taxonomy. This includes factors such as Unifiability, Modularity, and Adaptability that are pertinent and essential to the wide adoption and application of GMMs. The review further highlights key challenges and prospects for the field and guide the researchers into the new advancements., Comment: 25 pages, 3 figures, 5 tables

Homeostasis is concerned with regulatory mechanisms, present in biological systems, where some specific variable is kept close to a set value as some external disturbance affects the system. Mathematically, the notion of homeostasis can be formalized in terms of an input-output function that maps the parameter representing the external disturbance to the output variable that must be kept within a fairly narrow range. This observation inspired the introduction of the notion of infinitesimal homeostasis, namely, the derivative of the input-output function is zero at an isolated point. This point of view allows for the application of methods from singularity theory to characterize infinitesimal homeostasis points (i.e. critical points of the input-output function). In this paper we review the infinitesimal approach to the study of homeostasis in input-output networks. An input-output network is a network with two distinguished nodes `input' and `output', and the dynamics of the network determines the corresponding input-output function of the system. This class of dynamical systems provides an appropriate framework to study homeostasis and several important biological systems can be formulated in this context. Moreover, this approach, coupled to graph-theoretic ideas from combinatorial matrix theory, provides a systematic way for classifying different types of homeostasis (homeostatic mechanisms) in input-output networks, in terms of the network topology. In turn, this leads to new mathematical concepts, such as, homeostasis subnetworks, homeostasis patterns, homeostasis mode interaction. We illustrate the usefulness of this theory with several biological examples: biochemical networks, chemical reaction networks (CRN), gene regulatory networks (GRN), Intracellular metal ion regulation and so on., Comment: 45 pages, 26 figures, submitted to the MBS special issue "Dynamical Systems in Life Sciences"

Hopf bifurcation in networks of coupled ODEs creates periodic states in which the relative phases of nodes are well defined near bifurcation. When the network is a fully inhomogeneous nearest-neighbour coupled unidirectional ring, and node spaces are 1-dimensional, we derive constraints on these phase shifts that apply to any ODE that respects the ring topology. We begin with a 3-node ring and generalise the results to any number of nodes. The main point is that such constraints exist even when the only structure present is the network topology. We also prove that the usual nondegeneracy conditions in the classical Hopf Bifurcation Theorem are valid generically for ring networks, by perturbing only coupling terms., Comment: 22 pages, 3 figures

Systems of ODEs coupled with the topology of a closed ring are common models in biology, robotics, electrical engineering, and many other areas of science. When the component systems and couplings are identical, the system has a cyclic symmetry group for unidirectional rings and a dihedral symmetry group for bidirectional rings. Hopf bifurcation in equivariant and network dynamics predicts the generic occurrence of periodic discrete rotating waves whose phase patterns are determined by the symmetry group. We review basic aspects of the theory in some detail and derive general properties of such rings. New results are obtained characterising the first bifurcation for long-range couplings and the direction in which discrete rotating wave states rotate., Comment: 41 pages, 7 figures

We classify connected 2-node excitatory-inhibitory networks under various conditions. We assume that, as well as for connections, there are two distinct node-types, excitatory and inhibitory. In our classification we consider four different types of excitatory-inhibitory networks: restricted, partially restricted, unrestricted and completely unrestricted. For each type we give two different classifications. Using results on ODE-equivalence and minimality, we classify the ODE-classes and present a minimal representative for each ODE-class. We also classify all the networks with valence $\le 2$. These classifications are up to renumbering of nodes and the interchange of `excitatory' and `inhibitory' on nodes and arrows.These classifications constitute a first step towards analysing dynamics and bifurcations of excitatory-inhibitory networks. The results have potential applications to biological network models, especially neuronal networks, gene regulatory networks, and synthetic gene networks.

Machine translation often suffers from biased data and algorithms that can lead to unacceptable errors in system output. While bias in gender norms has been investigated, less is known about whether MT systems encode bias about social relationships, e.g., "the lawyer kissed her wife." We investigate the degree of bias against same-gender relationships in MT systems, using generated template sentences drawn from several noun-gender languages (e.g., Spanish) and comprised of popular occupation nouns. We find that three popular MT services consistently fail to accurately translate sentences concerning relationships between entities of the same gender. The error rate varies considerably based on the context, and same-gender sentences referencing high female-representation occupations are translated with lower accuracy. We provide this work as a case study in the evaluation of intrinsic bias in NLP systems with respect to social relationships., Comment: 5th Workshop on Gender Bias in Natural Language Processing 2024

In this paper, we use the framework of infinitesimal homeostasis to study general design principles for the occurrence of homeostasis in gene regulatory networks. We assume that the dynamics of the genes explicitly includes both transcription and translation, keeping track of both mRNA and protein concentrations. Given a GRN we construct an associated Protein-mRNA Network (PRN), where each individual (mRNA and protein) concentration corresponds to a node and the edges are defined in such a way that the PRN becomes a bipartite directed graph. By simultaneously working with the GRN and the PRN we are able to apply our previous results about the classification of homeostasis types (i.e., topologically defined homeostasis generating mechanism) and their corresponding homeostasis patterns. Given an arbitrarily large and complex GRN $\mathcal{G}$ and its associated PRN $\mathcal{R}$, we obtain a correspondence between all the homeostasis types (and homeostasis patterns) of $\mathcal{G}$ and a subset the homeostasis types (and homeostasis patterns) of $\mathcal{R}$. Moreover, we completely characterize the homeostasis types of the PRN that do not have GRN counterparts., Comment: 46 pages, 3 figures, 6 tables

Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving the LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow scientific multimodal instructions. To test our methodology, we use a human-generated scientific instruction tuning dataset and train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. In comparison to the models that are finetuned with machine generated data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark., Comment: Preprint. Work in progress

Homeostasis is a regulatory mechanism that keeps a specific variable close to a set value as other variables fluctuate. The notion of homeostasis can be rigorously formulated when the model of interest is represented as an input-output network, with distinguished input and output nodes, and the dynamics of the network determines the corresponding input-output function of the system. In this context, homeostasis can be defined as an 'infinitesimal' notion, namely, the derivative of the input-output function is zero at an isolated point. Combining this approach with graph-theoretic ideas from combinatorial matrix theory provides a systematic framework for calculating homeostasis points in models and classifying the different homeostasis types in input-output networks. In this paper we extend this theory by introducing the notion of a homeostasis pattern, defined as a set of nodes, in addition to the output node, that are simultaneously infinitesimally homeostatic. We prove that each homeostasis type leads to a distinct homeostasis pattern. Moreover, we describe all homeostasis patterns supported by a given input-output network in terms of a combinatorial structure associated to the input-output network. We call this structure the homeostasis pattern network., Comment: 33 pages, 7 figures, 2 tables

We analyse the dynamics of networks in which a central pattern generator (CPG) transmits signals along one or more feedforward chains in a synchronous or phase-synchronous manner. Such propagating signals are common in biology, especially in locomotion and peristalsis, and are of interest for continuum robots. We construct such networks as feedforward lifts of the CPG. If the CPG dynamics is periodic, so is the lifted dynamics. Synchrony with the CPG manifests as a standing wave, and a regular phase pattern creates a travelling wave. We discuss Liapunov, asymptotic, and Floquet stability of the lifted periodic orbit and introduce transverse versions of these conditions that imply stability for signals propagating along arbitrarily long chains. We compare these notions to a simpler condition, transverse stability of the synchrony subspace, which is equivalent to Floquet stability when nodes are 1-dimensional., Comment: 38 pages, 9 figures

Many scientific fields -- including biology, health, education, and the social sciences -- use machine learning (ML) to help them analyze data at an unprecedented scale. However, ML researchers who develop advanced methods rarely provide detailed tutorials showing how to apply these methods. Existing tutorials are often costly to participants, presume extensive programming knowledge, and are not tailored to specific application fields. In an attempt to democratize ML methods, we organized a year-long, free, online tutorial series targeted at teaching advanced natural language processing (NLP) methods to computational social science (CSS) scholars. Two organizers worked with fifteen subject matter experts to develop one-hour presentations with hands-on Python code for a range of ML methods and use cases, from data pre-processing to analyzing temporal variation of language change. Although live participation was more limited than expected, a comparison of pre- and post-tutorial surveys showed an increase in participants' perceived knowledge of almost one point on a 7-point Likert scale. Furthermore, participants asked thoughtful questions during tutorials and engaged readily with tutorial content afterwards, as demonstrated by 10K~total views of posted tutorial recordings. In this report, we summarize our organizational efforts and distill five principles for democratizing ML+X tutorials. We hope future organizers improve upon these principles and continue to lower barriers to developing ML skills for researchers of all fields.

This report summarizes the work of the Energy Frontier Topical Group on EW Physics: Heavy flavor and top quark physics (EF03) of the 2021 Community Summer Study (Snowmass). It aims to highlight the physics potential of top-quark studies and heavy-flavor production processes (bottom and charm) at the HL-LHC and possible future hadron and lepton colliders and running scenarios.

How does a group of agents break indecision when deciding about options with qualities that are hard to distinguish? Biological and artificial multi-agent systems, from honeybees and bird flocks to bacteria, robots, and humans, often need to overcome indecision when choosing among options in situations in which the performance or even the survival of the group are at stake. Breaking indecision is also important because in a fully indecisive state agents are not biased toward any specific option and therefore the agent group is maximally sensitive and prone to adapt to inputs and changes in its environment. Here, we develop a mathematical theory to study how decisions arise from the breaking of indecision. Our approach is grounded in both equivariant and network bifurcation theory. We model decision from indecision as synchrony-breaking in influence networks in which each node is the value assigned by an agent to an option. First, we show that three universal decision behaviors, namely, deadlock, consensus, and dissensus, are the generic outcomes of synchrony-breaking bifurcations from a fully synchronous state of indecision in influence networks. Second, we show that all deadlock and consensus value patterns and some dissensus value patterns are predicted by the symmetry of the influence networks. Third, we show that there are also many `exotic' dissensus value patterns. These patterns are predicted by network architecture, but not by network symmetries, through a new synchrony-breaking branching lemma. This is the first example of exotic solutions in an application. Numerical simulations of a novel influence network model illustrate our theoretical results., Comment: 36 pages

We consider four long-standing Rigidity Conjectures about synchrony and phase patterns for hyperbolic periodic orbits of admissible ODEs for networks. Proofs of stronger local versions of these conjectures, published in 2010-12, are now known to have a gap, but remain valid for a broad class of networks. Using different methods we prove local versions of the conjectures under a stronger condition, `strong hyperbolicity', which is related to a network analogue of the Kupka-Smale Theorem. Under this condition we also deduce global versions of the conjectures and an analogue of the $H/K$ Theorem in equivariant dynamics. We prove the Rigidity Conjectures for all 1- and 2-colourings and all 2- and 3-node networks by proving that strong hyperbolicity is generic in these cases., Comment: 69 pages

When writing, a person may need to anticipate questions from their audience, but different social groups may ask very different types of questions. If someone is writing about a problem they want to resolve, what kind of follow-up question will a domain expert ask, and could the writer better address the expert's information needs by rewriting their original post? In this paper, we explore the task of socially-aware question generation. We collect a data set of questions and posts from social media, including background information about the question-askers' social groups. We find that different social groups, such as experts and novices, consistently ask different types of questions. We train several text-generation models that incorporate social information, and we find that a discrete social-representation model outperforms the text-only model when different social groups ask highly different questions from one another. Our work provides a framework for developing text generation models that can help writers anticipate the information expectations of highly different social groups., Comment: SIGDIAL 2022

Many people aim for change, but not everyone succeeds. While there are a number of social psychology theories that propose motivation-related characteristics of those who persist with change, few computational studies have explored the motivational stage of personal change. In this paper, we investigate a new dataset consisting of the writings of people who manifest intention to change, some of whom persist while others do not. Using a variety of linguistic analysis techniques, we first examine the writing patterns that distinguish the two groups of people. Persistent people tend to reference more topics related to long-term self-improvement and use a more complicated writing style. Drawing on these consistent differences, we build a classifier that can reliably identify the people more likely to persist, based on their language. Our experiments provide new insights into the motivation-related behavior of people who persist with their intention to change., Comment: 10 pages, Accepted to be published at SocialNLP at NAACL'21

We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER -- a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model's understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https://lit.eecs.umich.edu/fiber/., Comment: Accepted at ACL 2022 Main conference. Camera-ready version

Speakers of non-English languages often adopt loanwords from English to express new or unusual concepts. While these loanwords may be borrowed unchanged, speakers may also integrate the words to fit the constraints of their native language, e.g. creating Spanish "tuitear" from English "tweet." Linguists have often considered the process of loanword integration to be more dependent on language-internal constraints, but sociolinguistic constraints such as speaker background remain only qualitatively understood. We investigate the role of social context and speaker background in Spanish speakers' use of integrated loanwords on social media. We find first that newspaper authors use the integrated forms of loanwords and native words more often than social media authors, showing that integration is associated with formal domains. In social media, we find that speaker background and expectations of formality explain loanword and native word integration, such that authors who use more Spanish and who write to a wider audience tend to use integrated verb forms more often. This study shows that loanword integration reflects not only language-internal constraints but also social expectations that vary by conversation and speaker.