Search

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency., Comment: Accepted to KDD 2024 (ADS Track)

The high computational cost associated with solving for detailed chemistry poses a significant challenge for predictive computational fluid dynamics (CFD) simulations of turbulent reacting flows. These models often require solving a system of coupled stiff ordinary differential equations (ODEs). While deep learning techniques have been experimented with to develop faster surrogate models, they often fail to integrate reliably with CFD solvers. This instability arises because deep learning methods optimize for training error without ensuring compatibility with ODE solvers, leading to accumulation of errors over time. Recently, NeuralODE-based techniques have offered a promising solution by effectively modeling chemical kinetics. In this study, we extend the NeuralODE framework for stiff chemical kinetics by incorporating mass conservation constraints directly into the loss function during training. This ensures that the total mass and the elemental mass are conserved, a critical requirement for reliable downstream integration with CFD solvers. Proof-of-concept studies are performed with physics-constrained neuralODE (PC-NODE) approach for homogeneous autoignition of hydrogen-air mixture over a range of composition and thermodynamic conditions. Our results demonstrate that this enhancement not only improves the physical consistency with respect to mass conservation criteria but also ensures better robustness. Lastly, a posteriori studies are performed wherein the trained PC-NODE model is coupled with a 3D CFD solver for computing the chemical source terms. PC-NODE is shown to be more accurate relative to the purely data-driven neuralODE approach. Moreover, PC-NODE also exhibits robustness and generalizability to unseen initial conditions from within (interpolative capability) as well as outside (extrapolative capability) the training regime.

Determining the physicochemical properties of a protein can reveal important insights in their structure, biological functions, stability, and interactions with other molecules. Although tools for computing properties of proteins already existed, we could not find a comprehensive tool that enables the calculations of multiple properties for multiple input proteins on the proteome level at once. Facing this limitation, we have developed Multiple Protein Profiler (MPP) 1.0 as an integrated tool that allows the profiling of 12 individual properties of multiple proteins in a significant manner. MPP provides a tabular and graphic visualization of properties of multiple proteins. The tool is freely accessible at https://mproteinprofiler.microbiologyandimmunology.dal.ca/

We consider the construction of linear instability of parallel shear flows, which was developed by Zhiwu Lin (SIAM J. Math. Anal. 35(2), 2003). We give an alternative simple proof in Sobolev setting of the problem, which exposes the role of the Plemenj-Sokhotski formula in the emergence of the instability, as well as does not require the cone condition. Moreover, we localize this approach to obtain an approximation of the Kelvin-Helmholtz instability of a flat vortex sheet., Comment: 12 pages, 1 figure

We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including LLaMA-2 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module. To further strengthen the multimodal LLM's capabilities, we fine-tune the model with a multimodal instruction set manually collected to cover diverse topics and tasks beyond simple QAs. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks.

This paper considers a family of active scalar equations which modify the generalized surface quasi-geostrophic (gSQG) equations through its constitutive law or dissipative perturbations. These modifications are characteristically mild in the sense that they are logarithmic. The problem of well-posedness, in the sense of Hadamard, is then studied in a borderline setting of regularity in analog to the scaling-critical spaces of the gSQG equations. A novelty of the system considered is the nuanced form of smoothing provided by the proposed mild form of dissipation, which is able to support global well-posedness at the Euler endpoint, but in a setting where the inviscid counterpart is known to be ill-posed. A novelty of the analysis lies in the simultaneous treatment of modifications in the constitutive law, dissipative mechanism, and functional setting, which the existing literature has typically treated separately. A putatively sharp relation is identified between each of the distinct system-modifiers that is consistent with previous studies that considered these modifications in isolation. This unified perspective is afforded by the introduction of a linear model equation, referred to as the protean system, that successfully incorporates the more delicate commutator structure collectively possessed by the gSQG family and upon which each facet of well-posedness can be reduced to its study., Comment: 48 pages

A combustion chemistry acceleration scheme is developed based on deep operator networks (DeepONets). The scheme is based on the identification of combustion reaction dynamics through a modified DeepOnet architecture such that the solutions of thermochemical scalars are projected to new solutions in small and flexible time increments. The approach is designed to efficiently implement chemistry acceleration without the need for computationally expensive integration of stiff chemistry. An additional framework of latent-space dynamics identification with modified DeepOnet is also proposed which enhances the computational efficiency and widens the applicability of the proposed scheme. The scheme is demonstrated on simple chemical kinetics of hydrogen oxidation to more complex chemical kinetics of n-dodecane high- and low-temperature oxidations. The proposed framework accurately learns the chemical kinetics and efficiently reproduces species and temperature temporal profiles corresponding to each application. In addition, a very large speed-up with a great extrapolation capability is also observed with the proposed scheme., Comment: 10 pages, Conference paper (submitted to 13th National Meeting of the Combustion Institute, At College Station, Texas)