Search

Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench is released in https://wang-sj16.github.io/motif/., Comment: TI2V Bench is released in https://wang-sj16.github.io/motif/

The growing reliance on artificial intelligence (AI) in customer support has significantly improved operational efficiency and user experience. However, traditional machine learning (ML) approaches, which require extensive local training on sensitive datasets, pose substantial privacy risks and compliance challenges with regulations like the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA). Existing privacy-preserving techniques, such as anonymization, differential privacy, and federated learning, address some concerns but face limitations in utility, scalability, and complexity. This paper introduces the Privacy-Preserving Zero-Shot Learning (PP-ZSL) framework, a novel approach leveraging large language models (LLMs) in a zero-shot learning mode. Unlike conventional ML methods, PP-ZSL eliminates the need for local training on sensitive data by utilizing pre-trained LLMs to generate responses directly. The framework incorporates real-time data anonymization to redact or mask sensitive information, retrieval-augmented generation (RAG) for domain-specific query resolution, and robust post-processing to ensure compliance with regulatory standards. This combination reduces privacy risks, simplifies compliance, and enhances scalability and operational efficiency. Empirical analysis demonstrates that the PP-ZSL framework provides accurate, privacy-compliant responses while significantly lowering the costs and complexities of deploying AI-driven customer support systems. The study highlights potential applications across industries, including financial services, healthcare, e-commerce, legal support, telecommunications, and government services. By addressing the dual challenges of privacy and performance, this framework establishes a foundation for secure, efficient, and regulatory-compliant AI applications in customer interactions.

In recent years, critical infrastructure and power grids have increasingly been targets of cyber-attacks, causing widespread and extended blackouts. Digital substations are particularly vulnerable to such cyber incursions, jeopardizing grid stability. This paper addresses these risks by proposing a cybersecurity framework that leverages software-defined networking (SDN) to bolster the resilience of substations based on the IEC-61850 standard. The research introduces a strategy involving smart cyber switching (SCS) for mitigation and concurrent intelligent electronic device (CIED) for restoration, ensuring ongoing operational integrity and cybersecurity within a substation. The SCS framework improves the physical network's behavior (i.e., leveraging commercial SDN capabilities) by incorporating an adaptive port controller (APC) module for dynamic port management and an intrusion detection system (IDS) to detect and counteract malicious IEC-61850-based sampled value (SV) and generic object-oriented system event (GOOSE) messages within the substation's communication network. The framework's effectiveness is validated through comprehensive simulations and a hardware-in-the-loop (HIL) testbed, demonstrating its ability to sustain substation operations during cyber-attacks and significantly improve the overall resilience of the power grid., Comment: 5 Pages, 5 Figures

Substation Automation Systems (SAS) that adhere to the International Electrotechnical Commission (IEC) 61850 standard have already been widely implemented across various on-site local substations. However, the digitalization of substations, which involves the use of cyber system, inherently increases their vulnerability to cyberattacks. This paper proposes the detection of cyberattacks through an anomaly-based approach utilizing Machine Learning (ML) methods within central control systems of the power system network. Furthermore, when an anomaly is identified, mitigation and restoration strategies employing concurrent Intelligent Electronic Devices (CIEDs) are utilized to ensure robust substation automation system operations. The proposed ML model is trained using Sampled Value (SV) and Generic Object Oriented Substation Event (GOOSE) data from each substation within the entire transmission system. As a result, the trained ML models can classify cyberattacks and normal faults, while the use of CIEDs contributes to cyberattack mitigation, and substation restoration.

Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions. The literature on behavior prediction spans various tasks, including action anticipation, activity forecasting, intent prediction, goal prediction, and so on. Our survey aims to tie together this fragmented literature, covering recent technical innovations as well as the development of new large-scale datasets for model training and evaluation. We also summarize the widely-used metrics for different tasks and provide a comprehensive performance comparison of existing approaches on eleven action anticipation datasets. This survey serves as not only a reference for contemporary methodologies in action anticipation, but also a guideline for future research direction of this evolving landscape., Comment: 30 pages, 9 figures, 12 tables

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Controlling AUVs can be challenging because of the effect of complex non-linear hydrodynamic forces acting on the robot, which, unlike ground robots, are significant in water and cannot be ignored. The problem is especially challenging for small AUVs for which the dynamics can change significantly with payload changes and deployments under different water conditions. The common approach to AUV control is a combination of passive stabilization with added buoyancy on top and weights on the bottom, and a PID controller tuned for simple and smooth motion primitives. However, the approach comes at the cost of sluggish controls and often the need to re-tune controllers with configuration changes. We propose a fast (trainable in minutes), reinforcement learning based approach for full 6 degree of freedom (DOF) control of an AUV, enabled by a new, highly parallelized simulator for underwater vehicle dynamics. We demonstrate that the proposed simulator models approximate hydrodynamic forces with enough accuracy that a zero-shot transfer of the learned policy to a real robot produces performance comparable to a hand-tuned PID controller. Furthermore, we show that domain randomization on the simulator produces policies that are robust to small variations in vehicle's physical parameters.

We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Underwater scenes are challenging visual environments, as rendering through a medium such as water introduces both range and color dependent effects on image capture. We constrain 3D Gaussian Splatting (3DGS), a recent advance in radiance fields enabling rapid training and real-time rendering of full 3D scenes, with a physically grounded underwater image formation model. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands, and simulation-degraded real-world scenes, not only do we see increased quantitative performance on rendering novel viewpoints from the scene with the medium present, but are also able to recover the underlying true color of the scene and restore renders to be without the presence of the intervening medium. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation., Comment: Project page here: https://seasplat.github.io

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

There exists a capability gap in the design of currently available autonomous underwater vehicles (AUV). Most AUVs use a set of thrusters, and optionally control surfaces, to control their depth and pose. AUVs utilizing thrusters can be highly maneuverable, making them well-suited to operate in complex environments such as in close-proximity to coral reefs. However, they are inherently power-inefficient and produce significant noise and disturbance. Underwater gliders, on the other hand, use changes in buoyancy and center of mass, in combination with a control surface to move around. They are extremely power efficient but not very maneuverable. Gliders are designed for long-range missions that do not require precision maneuvering. Furthermore, since gliders only activate the buoyancy engine for small time intervals, they do not disturb the environment and can also be used for passive acoustic observations. In this paper we present ReefGlider, a novel AUV that uses only buoyancy for control but is still highly maneuverable from additional buoyancy control devices. ReefGlider bridges the gap between the capabilities of thruster-driven AUVs and gliders. These combined characteristics make ReefGlider ideal for tasks such as long-term visual and acoustic monitoring of coral reefs. We present the overall design and implementation of the system, as well as provide analysis of some of its capabilities., Comment: In IEEE International Conference on Robotics and Automation (ICRA), 2024

Using the standard parametrization of the dipole form, we have studied the vulnerability of $Q^2$ on the vector form factors ($f_i^{B_iB_f}(Q^2)$) and axial-vector form factors ($g_i^{B_iB_f}(Q^2)$), $i=1,2,3$ computed for the semileptonic $B_i \rightarrow B_f l \bar{\nu}$ decays for hyperons in the framework of chiral constituent quark model ($\chi$CQM). Both, strangeness changing as well as strangeness conserving decays have been examined. We also present the dependence of the ratio of hyperon semileptonic decay constants $g_1(Q^2)/f_1(Q^2)$ for these decays. Further, we calculate the CKM matrix elements $V_{ud}$ from strangeness conserving and $V_{us}$ from strangeness changing hyperon decays., Comment: 21 pages, 5 figures and 9 tables

We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos. Whereas existing methods rely on curated data with known audio-visual correspondence, our multimodal contrastive-consensus coding (MC3) embedding reinforces the associations between audio, language, and vision when all modality pairs agree, while diminishing those associations when any one pair does not. We show our approach can successfully discover how the long tail of human actions sound from egocentric video, outperforming an array of recent multimodal embedding techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal tasks., Comment: Accepted at CVPR 2024. Project page: https://vision.cs.utexas.edu/projects/soundingactions

An increasing number of observations and simulations suggests that low-power (<10$^{44}$ erg s$^{-1}$) jets may be a significant channel of feedback produced by active galactic nuclei (AGN), but little is known about their actual effect on their host galaxies from the observational point of view. We targeted four luminous type 2 AGN hosting moderately powerful radio emission ($\sim$10$^{44}$ erg s$^{-1}$), two of which and possibly a third are associated with jets, with optical integral field spectroscopy observations from the Multi Unit Spectroscopic Explorer (MUSE) at the Very Large Telescope (VLT) to analyze the properties of their ionized gas as well as the properties and effects of ionized outflows. We combined these observations with Very Large Array (VLA) and e-MERLIN data to investigate the relations and interactions between the radio jets and host galaxies. We detected ionized outflows as traced by the fast bulk motion of the gas. The outflows extended over kiloparsec scales in the direction of the jet, when present. In the two sources with resolved radio jets, we detected a strong enhancement in the emission-line velocity dispersion (up to 1000 km s$^{-1}$) perpendicular to the direction of the radio jets. We also found a correlation between the mass and the energetics of this high-velocity dispersion gas and the radio power, which supports the idea that the radio emission may cause the enhanced turbulence. This phenomenon, which is now being observed in an increasing number of objects, might represent an important channel for AGN feedback on galaxies., Comment: 18 pages, 17 figures

We present the Streaming Gaussian Dirichlet Random Field (S-GDRF) model, a novel approach for modeling a stream of spatiotemporally distributed, sparse, high-dimensional categorical observations. The proposed approach efficiently learns global and local patterns in spatiotemporal data, allowing for fast inference and querying with a bounded time complexity. Using a high-resolution data series of plankton images classified with a neural network, we demonstrate the ability of the approach to make more accurate predictions compared to a Variational Gaussian Process (VGP), and to learn a predictive distribution of observations from streaming categorical data. S-GDRFs open the door to enabling efficient informative path planning over high-dimensional categorical observations, which until now has not been feasible., Comment: 10 pages, 5 figures. Published in Springer Proceedings of Advanced Robotics, ISER 2023 Conference Proceedings

Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image. We introduce InstanceDiffusion that adds precise instance-level control to text-to-image diffusion models. InstanceDiffusion supports free-form language conditions per instance and allows flexible ways to specify instance locations such as simple single points, scribbles, bounding boxes or intricate instance segmentation masks, and combinations thereof. We propose three major changes to text-to-image models that enable precise instance-level control. Our UniFusion block enables instance-level conditions for text-to-image models, the ScaleU block improves image fidelity, and our Multi-instance Sampler improves generations for multiple instances. InstanceDiffusion significantly surpasses specialized state-of-the-art models for each location condition. Notably, on the COCO dataset, we outperform previous state-of-the-art by 20.4% AP$_{50}^\text{box}$ for box inputs, and 25.4% IoU for mask inputs., Comment: Preprint; Project page: https://people.eecs.berkeley.edu/~xdwang/projects/InstDiff/

Audio Universe: Tour of the Solar System is an audio-visual show for planetariums and flatscreen viewing. It is designed in collaboration with members of the blind and vision impaired (BVI) community, BVI specialist teachers and their pupils. It aims to be suitable for audiences with all sight levels by representing key concepts through sound and using a carefully constructed narration. We present results from 291 audience evaluations from online viewers and audience members of several planetarium showings in the UK and Italy. We find a strong appreciation from BVI and non-BVI audiences, with ~90% scoring 4 or 5 (out of 5) for both how useful and enjoyable the sounds are. We also present results from surveying planetariums and communication leaders known to have downloaded the show. We find international success for special events, for BVI audiences and for those with other special educational needs and disabilities (SEND; including sensory needs and learning difficulties). Feedback suggests this is due to its multi-sensory, clearly narrated, and low sensory load (calm) production. However, we also describe limitations identified during this evaluation exercise, including the show's limited incorporation into regular (non-special) planetarium programmes. This highlights an ongoing challenge of creating a fully inclusive planetarium experience., Comment: Published in Communicating Astronomy with the Public Journal: CAPj, 33, 6, (2023). Available here: https://www.capjournal.org/issues/33/index.php

We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions given text as input. The resulting model strongly outperforms baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases, users even prefer it to human-generated articles. Most notably, it enables various new and exciting applications far beyond what static articles on the web can provide, such as personalized instructions complete with intermediate steps and pictures in response to a user's individual situation., Comment: Accepted to CVPR 2024. Project website: http://facebookresearch.github.io/IllustratedInstructions. Code reproduction: https://github.com/sachit-menon/generating-illustrated-instructions-reproduction

In recent years, critical infrastructure and power grids have experienced a series of cyber-attacks, leading to temporary, widespread blackouts of considerable magnitude. Since most substations are unmanned and have limited physical security protection, cyber breaches into power grid substations present a risk. Nowadays, software-defined network (SDN), a popular virtual network technology based on the OpenFlow protocol is being widely used in the substation automation system. However, the susceptibility of SDN architecture to cyber-attacks has exhibited a notable increase in recent years, as indicated by research findings. This suggests a growing concern regarding the potential for cybersecurity breaches within the SDN framework. In this paper, we propose a hybrid intrusion detection system (IDS)-integrated SDN architecture for detecting and preventing the injection of malicious IEC 61850-based generic object-oriented substation event (GOOSE) messages in a digital substation. Additionally, this program locates the fault's location and, as a form of mitigation, disables a certain port. Furthermore, implementation examples are demonstrated and verified using a hardware-in-the-loop (HIL) testbed that mimics the functioning of a digital substation., Comment: 5 pages, 6 figures, 1 table, conference paper, supported by DOE (CESER) program

We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions--adjusted noise schedules for diffusion, and multi-stage training that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user's text prompt, where our generations are preferred 96% over prior work., Comment: ECCV 2024. Project page: https://emu-video.metademolab.com

We present a study of molecular gas, traced via CO (3-2) from ALMA data, of four z< 0.2, `radio quiet', type 2 quasars (log [L(bol)/(erg/s)] = 45.3 - 46.2; log [L(1.4 GHz)/(W/Hz)] = 23.7 - 24.3). Targets were selected to have extended radio lobes (>= 10 kpc), and compact, moderate-power jets (1 - 10 kpc; log [Pjet/(erg/s)]= 43.2 - 43.7). All targets show evidence of central molecular outflows, or injected turbulence, within the gas disks (traced via high-velocity wing components in CO emission-line profiles). The inferred velocities (Vout = 250 - 440 km/s) and spatial scales (0.6 - 1.6 kpc), are consistent with those of other samples of luminous low-redshift AGN. In two targets, we observe extended molecular gas structures beyond the central disks, containing 9 - 53 % of the total molecular gas mass. These structures tend to be elongated, extending from the core, and wrap-around (or along) the radio lobes. Their properties are similar to the molecular gas filaments observed around radio lobes of, mostly `radio loud', Brightest Cluster Galaxies. They have: projected distances of 5 - 13 kpc; bulk velocities of 100 - 340 km/s; velocity dispersion of 30 - 130 km/s; inferred mass outflow rates of 4 - 20 Msolar/yr; and estimated kinetic powers of log [Ekin/(erg/s)]= 40.3 - 41.7. Our observations are consistent with simulations that suggest moderate-power jets can have a direct (but modest) impact on molecular gas on small scales, through direct jet-cloud interactions. Then, on larger scales, jet-cocoons can push gas aside. Both processes could contribute to the long-term regulation of star formation., Comment: Accepted for publication in MNRAS; references updated; typos corrected

We present a comprehensive study of the molecular gas properties of 17 Type 2 quasars at $z <$ 0.2 from the Quasar Feedback Survey (L$_{[OIII]}$ > $10^{42.1}$ $\rm ergs^{-1}$), selected by their high [OIII] luminosities and displaying a large diversity of radio jet properties, but dominated by LIRG-like galaxies. With these data, we are able to investigate the impact of AGN and AGN feedback mechanisms on the global molecular interstellar medium. Using APEX and ALMA ACA observations, we measure the total molecular gas content using the CO(1-0) emission and homogeneously sample the CO spectral line energy distributions (SLEDs), observing CO transitions (J$_{up}$ = 1, 2, 3, 6, 7). We observe high $r_{21}$ ratios (r$_{21}$ = L'$_{CO(2-1)}$/L'$_{CO(1-0)}$) with a median $r_{21}$ = 1.06, similar to local (U)LIRGs (with $r_{21}$ $\sim$ 1) and higher than normal star-forming galaxies (with r$_{21}$ $\sim$ 0.65). Despite the high $r_{21}$ values, for the 7 targets with the required data we find low excitation in CO(6-5) & CO(7-6) ($r_{61}$ and $r_{62}$ < 0.6 in all but one target), unlike high redshift quasars in the literature, which are far more luminous and show higher line ratios. The ionised gas traced by [OIII] exhibit systematically higher velocities than the molecular gas traced by CO. We conclude that any effects of quasar feedback (e.g. via outflows and radio jets) do not have a significant instantaneous impact on the global molecular gas content and excitation and we suggest that it only occurs on more localised scales., Comment: 32 pages (20 in the main body of the paper and 12 in the appendix), 28 figures (10 in main body of paper and 18 in appendix) Accepted for publication in MNRAS. Data available at https://doi.org/10.25405/data.ncl.24312502

The $z$~0.1 type-2 QSO J1430+1339 (the 'Teacup') is a complex galaxy showing a loop of ionised gas ~10 kpc in diameter, co-spatial radio bubbles, a compact (~1 kpc) jet, and outflow activity. We used VLT/MUSE optical integral field spectroscopic observations to characterise the properties and effects of the galactic ionised outflow from kpc up to tens of kpc scales and compare them with those of the radio jet. We detect a velocity dispersion enhancement (>300 km/s) elongated over several kpc perpendicular to the radio jet, the AGN ionisation lobes, and the fast outflow, similar to what is found in other galaxies hosting compact, low-power jets, indicating that the jet strongly perturbs the host ISM. The mass outflow rate decreases with distance from the nucleus, from around 100 $M_\odot$/yr in the inner 1-2 kpc to <0.1 $M_\odot$/yr at 30 kpc. The ionised mass outflow rate is ~1-8 times higher than the molecular one, in contrast with what is often quoted in AGN. The driver of the multi-phase outflow is likely a combination of AGN radiation and the jet. The outflow mass-loading factor (~5-10) and the molecular gas depletion time (<10$^8$ yr) indicate that the outflow can significantly affect the star formation and the gas reservoir in the galaxy. However, the fraction of the ionised outflow that is able to escape the dark matter halo potential is likely negligible. We detect blue-coloured continuum emission co-spatial with the ionised gas loop. Here, stellar populations are younger (<100-150 Myr) than in the rest of the galaxy (~0.5-1 Gyr). This constitutes possible evidence for star formation triggered at the edge of the bubble due to the compressing action of the jet and outflow ('positive feedback'), as predicted by theory. All in all, the Teacup constitutes a rich system in which AGN feedback from outflows and jets, in both its negative and positive flavours, co-exist., Comment: 26 pages, 16 figures; Astronomy & Astrophysics, in press

Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions. We present VideoCutLER, a simple method for unsupervised multi-instance video segmentation without using motion-based learning signals like optical flow or training on natural videos. Our key insight is that using high-quality pseudo masks and a simple video synthesis method for model training is surprisingly sufficient to enable the resulting video model to effectively segment and track multiple instances across video frames. We show the first competitive unsupervised learning results on the challenging YouTubeVIS-2019 benchmark, achieving 50.7% APvideo^50 , surpassing the previous state-of-the-art by a large margin. VideoCutLER can also serve as a strong pretrained model for supervised video instance segmentation tasks, exceeding DINO by 15.9% on YouTubeVIS-2019 in terms of APvideo., Comment: Preprint. Code: https://github.com/facebookresearch/CutLER

Cosmological simulations fail to reproduce realistic galaxy populations without energy injection from active galactic nuclei (AGN) into the interstellar medium (ISM) and circumgalactic medium (CGM); a process called `AGN feedback'. Consequently, observational work searches for evidence that luminous AGN impact their host galaxies. Here, we review some of this work. Multi-phase AGN outflows are common, some with potential for significant impact. Additionally, multiple feedback channels can be observed simultaneously; e.g., radio jets from `radio quiet' quasars can inject turbulence on ISM scales, and displace CGM-scale molecular gas. However, caution must be taken comparing outflows to simulations (e.g., kinetic coupling efficiencies) to infer feedback potential, due to a lack of comparable predictions. Furthermore, some work claims limited evidence for feedback because AGN live in gas-rich, star-forming galaxies. However, simulations do not predict instantaneous, global impact on molecular gas or star formation. The impact is expected to be cumulative, over multiple episodes., Comment: Accepted for publication in IAU Symposium 378 Conference Proceedings "Black Hole Winds at all Scales"

We present a novel approach to inspecting galaxy spectra using sound, via their direct audio representation ('spectral audification'). We discuss the potential of this as a complement to (or stand-in for) visual approaches. We surveyed 58 respondents who use the audio representation alone to rate 30 optical galaxy spectra with strong emission lines. Across three tests, each focusing on different quantities measured from the spectra (signal-to-noise ratio, emission-line width, & flux ratios), we find that user ratings are well correlated with measured quantities. This demonstrates that physical information can be independently gleaned from listening to spectral audifications. We note the importance of context when rating these sonifications, where the order examples are heard can influence responses. Finally, we adapt the method used in this promising pilot study to spectral datacubes. We suggest that audification allows efficient exploration of complex, spatially-resolved spectral data., Comment: 6 pages, 3 figures, accepted for publication in RASTI. Supplementary data (including animated figure) available at https://doi.org/10.25405/data.ncl.22816442

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks., Comment: CVPR 2023 (Highlighted Paper). Website: https://imagebind.metademolab.com/ Code/Models: https://github.com/facebookresearch/ImageBind

Coral reefs are fast-changing and complex ecosystems that are crucial to monitor and study. Biological hotspot detection can help coral reef managers prioritize limited resources for monitoring and intervention tasks. Here, we explore the use of autonomous underwater vehicles (AUVs) with cameras, coupled with visual detectors and photogrammetry, to map and identify these hotspots. This approach can provide high spatial resolution information in fast feedback cycles. To the best of our knowledge, we present one of the first attempts at using an AUV to gather visually-observed, fine-grain biological hotspot maps in concert with topography of a coral reefs. Our hotspot maps correlate with rugosity, an established proxy metric for coral reef biodiversity and abundance, as well as with our visual inspections of the 3D reconstruction. We also investigate issues of scaling this approach when applied to new reefs by using these visual detectors pre-trained on large public datasets., Comment: CV4Animals Workshop at CVPR 2023