Search

Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions. The literature on behavior prediction spans various tasks, including action anticipation, activity forecasting, intent prediction, goal prediction, and so on. Our survey aims to tie together this fragmented literature, covering recent technical innovations as well as the development of new large-scale datasets for model training and evaluation. We also summarize the widely-used metrics for different tasks and provide a comprehensive performance comparison of existing approaches on eleven action anticipation datasets. This survey serves as not only a reference for contemporary methodologies in action anticipation, but also a guideline for future research direction of this evolving landscape., Comment: 30 pages, 9 figures, 12 tables

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Controlling AUVs can be challenging because of the effect of complex non-linear hydrodynamic forces acting on the robot, which, unlike ground robots, are significant in water and cannot be ignored. The problem is especially challenging for small AUVs for which the dynamics can change significantly with payload changes and deployments under different water conditions. The common approach to AUV control is a combination of passive stabilization with added buoyancy on top and weights on the bottom, and a PID controller tuned for simple and smooth motion primitives. However, the approach comes at the cost of sluggish controls and often the need to re-tune controllers with configuration changes. We propose a fast (trainable in minutes), reinforcement learning based approach for full 6 degree of freedom (DOF) control of an AUV, enabled by a new, highly parallelized simulator for underwater vehicle dynamics. We demonstrate that the proposed simulator models approximate hydrodynamic forces with enough accuracy that a zero-shot transfer of the learned policy to a real robot produces performance comparable to a hand-tuned PID controller. Furthermore, we show that domain randomization on the simulator produces policies that are robust to small variations in vehicle's physical parameters.

We introduce SeaSplat, a method to enable real-time rendering of underwater scenes leveraging recent advances in 3D radiance fields. Underwater scenes are challenging visual environments, as rendering through a medium such as water introduces both range and color dependent effects on image capture. We constrain 3D Gaussian Splatting (3DGS), a recent advance in radiance fields enabling rapid training and real-time rendering of full 3D scenes, with a physically grounded underwater image formation model. Applying SeaSplat to the real-world scenes from SeaThru-NeRF dataset, a scene collected by an underwater vehicle in the US Virgin Islands, and simulation-degraded real-world scenes, not only do we see increased quantitative performance on rendering novel viewpoints from the scene with the medium present, but are also able to recover the underlying true color of the scene and restore renders to be without the presence of the intervening medium. We show that the underwater image formation helps learn scene structure, with better depth maps, as well as show that our improvements maintain the significant computational improvements afforded by leveraging a 3D Gaussian representation., Comment: Project page here: https://seasplat.github.io

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

There exists a capability gap in the design of currently available autonomous underwater vehicles (AUV). Most AUVs use a set of thrusters, and optionally control surfaces, to control their depth and pose. AUVs utilizing thrusters can be highly maneuverable, making them well-suited to operate in complex environments such as in close-proximity to coral reefs. However, they are inherently power-inefficient and produce significant noise and disturbance. Underwater gliders, on the other hand, use changes in buoyancy and center of mass, in combination with a control surface to move around. They are extremely power efficient but not very maneuverable. Gliders are designed for long-range missions that do not require precision maneuvering. Furthermore, since gliders only activate the buoyancy engine for small time intervals, they do not disturb the environment and can also be used for passive acoustic observations. In this paper we present ReefGlider, a novel AUV that uses only buoyancy for control but is still highly maneuverable from additional buoyancy control devices. ReefGlider bridges the gap between the capabilities of thruster-driven AUVs and gliders. These combined characteristics make ReefGlider ideal for tasks such as long-term visual and acoustic monitoring of coral reefs. We present the overall design and implementation of the system, as well as provide analysis of some of its capabilities., Comment: In IEEE International Conference on Robotics and Automation (ICRA), 2024

Using the standard parametrization of the dipole form, we have studied the vulnerability of $Q^2$ on the vector form factors ($f_i^{B_iB_f}(Q^2)$) and axial-vector form factors ($g_i^{B_iB_f}(Q^2)$), $i=1,2,3$ computed for the semileptonic $B_i \rightarrow B_f l \bar{\nu}$ decays for hyperons in the framework of chiral constituent quark model ($\chi$CQM). Both, strangeness changing as well as strangeness conserving decays have been examined. We also present the dependence of the ratio of hyperon semileptonic decay constants $g_1(Q^2)/f_1(Q^2)$ for these decays. Further, we calculate the CKM matrix elements $V_{ud}$ from strangeness conserving and $V_{us}$ from strangeness changing hyperon decays., Comment: 21 pages, 5 figures and 9 tables

We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos. Whereas existing methods rely on curated data with known audio-visual correspondence, our multimodal contrastive-consensus coding (MC3) embedding reinforces the associations between audio, language, and vision when all modality pairs agree, while diminishing those associations when any one pair does not. We show our approach can successfully discover how the long tail of human actions sound from egocentric video, outperforming an array of recent multimodal embedding techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal tasks., Comment: Accepted at CVPR 2024. Project page: https://vision.cs.utexas.edu/projects/soundingactions

An increasing number of observations and simulations suggests that low-power (<10$^{44}$ erg s$^{-1}$) jets may be a significant channel of feedback produced by active galactic nuclei (AGN), but little is known about their actual effect on their host galaxies from the observational point of view. We targeted four luminous type 2 AGN hosting moderately powerful radio emission ($\sim$10$^{44}$ erg s$^{-1}$), two of which and possibly a third are associated with jets, with optical integral field spectroscopy observations from the Multi Unit Spectroscopic Explorer (MUSE) at the Very Large Telescope (VLT) to analyze the properties of their ionized gas as well as the properties and effects of ionized outflows. We combined these observations with Very Large Array (VLA) and e-MERLIN data to investigate the relations and interactions between the radio jets and host galaxies. We detected ionized outflows as traced by the fast bulk motion of the gas. The outflows extended over kiloparsec scales in the direction of the jet, when present. In the two sources with resolved radio jets, we detected a strong enhancement in the emission-line velocity dispersion (up to 1000 km s$^{-1}$) perpendicular to the direction of the radio jets. We also found a correlation between the mass and the energetics of this high-velocity dispersion gas and the radio power, which supports the idea that the radio emission may cause the enhanced turbulence. This phenomenon, which is now being observed in an increasing number of objects, might represent an important channel for AGN feedback on galaxies., Comment: 18 pages, 17 figures

We present the Streaming Gaussian Dirichlet Random Field (S-GDRF) model, a novel approach for modeling a stream of spatiotemporally distributed, sparse, high-dimensional categorical observations. The proposed approach efficiently learns global and local patterns in spatiotemporal data, allowing for fast inference and querying with a bounded time complexity. Using a high-resolution data series of plankton images classified with a neural network, we demonstrate the ability of the approach to make more accurate predictions compared to a Variational Gaussian Process (VGP), and to learn a predictive distribution of observations from streaming categorical data. S-GDRFs open the door to enabling efficient informative path planning over high-dimensional categorical observations, which until now has not been feasible., Comment: 10 pages, 5 figures. Published in Springer Proceedings of Advanced Robotics, ISER 2023 Conference Proceedings

Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image. We introduce InstanceDiffusion that adds precise instance-level control to text-to-image diffusion models. InstanceDiffusion supports free-form language conditions per instance and allows flexible ways to specify instance locations such as simple single points, scribbles, bounding boxes or intricate instance segmentation masks, and combinations thereof. We propose three major changes to text-to-image models that enable precise instance-level control. Our UniFusion block enables instance-level conditions for text-to-image models, the ScaleU block improves image fidelity, and our Multi-instance Sampler improves generations for multiple instances. InstanceDiffusion significantly surpasses specialized state-of-the-art models for each location condition. Notably, on the COCO dataset, we outperform previous state-of-the-art by 20.4% AP$_{50}^\text{box}$ for box inputs, and 25.4% IoU for mask inputs., Comment: Preprint; Project page: https://people.eecs.berkeley.edu/~xdwang/projects/InstDiff/

Audio Universe: Tour of the Solar System is an audio-visual show for planetariums and flatscreen viewing. It is designed in collaboration with members of the blind and vision impaired (BVI) community, BVI specialist teachers and their pupils. It aims to be suitable for audiences with all sight levels by representing key concepts through sound and using a carefully constructed narration. We present results from 291 audience evaluations from online viewers and audience members of several planetarium showings in the UK and Italy. We find a strong appreciation from BVI and non-BVI audiences, with ~90% scoring 4 or 5 (out of 5) for both how useful and enjoyable the sounds are. We also present results from surveying planetariums and communication leaders known to have downloaded the show. We find international success for special events, for BVI audiences and for those with other special educational needs and disabilities (SEND; including sensory needs and learning difficulties). Feedback suggests this is due to its multi-sensory, clearly narrated, and low sensory load (calm) production. However, we also describe limitations identified during this evaluation exercise, including the show's limited incorporation into regular (non-special) planetarium programmes. This highlights an ongoing challenge of creating a fully inclusive planetarium experience., Comment: Published in Communicating Astronomy with the Public Journal: CAPj, 33, 6, (2023). Available here: https://www.capjournal.org/issues/33/index.php

We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions given text as input. The resulting model strongly outperforms baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases, users even prefer it to human-generated articles. Most notably, it enables various new and exciting applications far beyond what static articles on the web can provide, such as personalized instructions complete with intermediate steps and pictures in response to a user's individual situation., Comment: Accepted to CVPR 2024. Project website: http://facebookresearch.github.io/IllustratedInstructions. Code reproduction: https://github.com/sachit-menon/generating-illustrated-instructions-reproduction

In recent years, critical infrastructure and power grids have experienced a series of cyber-attacks, leading to temporary, widespread blackouts of considerable magnitude. Since most substations are unmanned and have limited physical security protection, cyber breaches into power grid substations present a risk. Nowadays, software-defined network (SDN), a popular virtual network technology based on the OpenFlow protocol is being widely used in the substation automation system. However, the susceptibility of SDN architecture to cyber-attacks has exhibited a notable increase in recent years, as indicated by research findings. This suggests a growing concern regarding the potential for cybersecurity breaches within the SDN framework. In this paper, we propose a hybrid intrusion detection system (IDS)-integrated SDN architecture for detecting and preventing the injection of malicious IEC 61850-based generic object-oriented substation event (GOOSE) messages in a digital substation. Additionally, this program locates the fault's location and, as a form of mitigation, disables a certain port. Furthermore, implementation examples are demonstrated and verified using a hardware-in-the-loop (HIL) testbed that mimics the functioning of a digital substation., Comment: 5 pages, 6 figures, 1 table, conference paper, supported by DOE (CESER) program

We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions--adjusted noise schedules for diffusion, and multi-stage training that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user's text prompt, where our generations are preferred 96% over prior work., Comment: ECCV 2024. Project page: https://emu-video.metademolab.com

We present a study of molecular gas, traced via CO (3-2) from ALMA data, of four z< 0.2, `radio quiet', type 2 quasars (log [L(bol)/(erg/s)] = 45.3 - 46.2; log [L(1.4 GHz)/(W/Hz)] = 23.7 - 24.3). Targets were selected to have extended radio lobes (>= 10 kpc), and compact, moderate-power jets (1 - 10 kpc; log [Pjet/(erg/s)]= 43.2 - 43.7). All targets show evidence of central molecular outflows, or injected turbulence, within the gas disks (traced via high-velocity wing components in CO emission-line profiles). The inferred velocities (Vout = 250 - 440 km/s) and spatial scales (0.6 - 1.6 kpc), are consistent with those of other samples of luminous low-redshift AGN. In two targets, we observe extended molecular gas structures beyond the central disks, containing 9 - 53 % of the total molecular gas mass. These structures tend to be elongated, extending from the core, and wrap-around (or along) the radio lobes. Their properties are similar to the molecular gas filaments observed around radio lobes of, mostly `radio loud', Brightest Cluster Galaxies. They have: projected distances of 5 - 13 kpc; bulk velocities of 100 - 340 km/s; velocity dispersion of 30 - 130 km/s; inferred mass outflow rates of 4 - 20 Msolar/yr; and estimated kinetic powers of log [Ekin/(erg/s)]= 40.3 - 41.7. Our observations are consistent with simulations that suggest moderate-power jets can have a direct (but modest) impact on molecular gas on small scales, through direct jet-cloud interactions. Then, on larger scales, jet-cocoons can push gas aside. Both processes could contribute to the long-term regulation of star formation., Comment: Accepted for publication in MNRAS; references updated; typos corrected

We present a comprehensive study of the molecular gas properties of 17 Type 2 quasars at $z <$ 0.2 from the Quasar Feedback Survey (L$_{[OIII]}$ > $10^{42.1}$ $\rm ergs^{-1}$), selected by their high [OIII] luminosities and displaying a large diversity of radio jet properties, but dominated by LIRG-like galaxies. With these data, we are able to investigate the impact of AGN and AGN feedback mechanisms on the global molecular interstellar medium. Using APEX and ALMA ACA observations, we measure the total molecular gas content using the CO(1-0) emission and homogeneously sample the CO spectral line energy distributions (SLEDs), observing CO transitions (J$_{up}$ = 1, 2, 3, 6, 7). We observe high $r_{21}$ ratios (r$_{21}$ = L'$_{CO(2-1)}$/L'$_{CO(1-0)}$) with a median $r_{21}$ = 1.06, similar to local (U)LIRGs (with $r_{21}$ $\sim$ 1) and higher than normal star-forming galaxies (with r$_{21}$ $\sim$ 0.65). Despite the high $r_{21}$ values, for the 7 targets with the required data we find low excitation in CO(6-5) & CO(7-6) ($r_{61}$ and $r_{62}$ < 0.6 in all but one target), unlike high redshift quasars in the literature, which are far more luminous and show higher line ratios. The ionised gas traced by [OIII] exhibit systematically higher velocities than the molecular gas traced by CO. We conclude that any effects of quasar feedback (e.g. via outflows and radio jets) do not have a significant instantaneous impact on the global molecular gas content and excitation and we suggest that it only occurs on more localised scales., Comment: 32 pages (20 in the main body of the paper and 12 in the appendix), 28 figures (10 in main body of paper and 18 in appendix) Accepted for publication in MNRAS. Data available at https://doi.org/10.25405/data.ncl.24312502

The $z$~0.1 type-2 QSO J1430+1339 (the 'Teacup') is a complex galaxy showing a loop of ionised gas ~10 kpc in diameter, co-spatial radio bubbles, a compact (~1 kpc) jet, and outflow activity. We used VLT/MUSE optical integral field spectroscopic observations to characterise the properties and effects of the galactic ionised outflow from kpc up to tens of kpc scales and compare them with those of the radio jet. We detect a velocity dispersion enhancement (>300 km/s) elongated over several kpc perpendicular to the radio jet, the AGN ionisation lobes, and the fast outflow, similar to what is found in other galaxies hosting compact, low-power jets, indicating that the jet strongly perturbs the host ISM. The mass outflow rate decreases with distance from the nucleus, from around 100 $M_\odot$/yr in the inner 1-2 kpc to <0.1 $M_\odot$/yr at 30 kpc. The ionised mass outflow rate is ~1-8 times higher than the molecular one, in contrast with what is often quoted in AGN. The driver of the multi-phase outflow is likely a combination of AGN radiation and the jet. The outflow mass-loading factor (~5-10) and the molecular gas depletion time (<10$^8$ yr) indicate that the outflow can significantly affect the star formation and the gas reservoir in the galaxy. However, the fraction of the ionised outflow that is able to escape the dark matter halo potential is likely negligible. We detect blue-coloured continuum emission co-spatial with the ionised gas loop. Here, stellar populations are younger (<100-150 Myr) than in the rest of the galaxy (~0.5-1 Gyr). This constitutes possible evidence for star formation triggered at the edge of the bubble due to the compressing action of the jet and outflow ('positive feedback'), as predicted by theory. All in all, the Teacup constitutes a rich system in which AGN feedback from outflows and jets, in both its negative and positive flavours, co-exist., Comment: 26 pages, 16 figures; Astronomy & Astrophysics, in press

Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions. We present VideoCutLER, a simple method for unsupervised multi-instance video segmentation without using motion-based learning signals like optical flow or training on natural videos. Our key insight is that using high-quality pseudo masks and a simple video synthesis method for model training is surprisingly sufficient to enable the resulting video model to effectively segment and track multiple instances across video frames. We show the first competitive unsupervised learning results on the challenging YouTubeVIS-2019 benchmark, achieving 50.7% APvideo^50 , surpassing the previous state-of-the-art by a large margin. VideoCutLER can also serve as a strong pretrained model for supervised video instance segmentation tasks, exceeding DINO by 15.9% on YouTubeVIS-2019 in terms of APvideo., Comment: Preprint. Code: https://github.com/facebookresearch/CutLER

Cosmological simulations fail to reproduce realistic galaxy populations without energy injection from active galactic nuclei (AGN) into the interstellar medium (ISM) and circumgalactic medium (CGM); a process called `AGN feedback'. Consequently, observational work searches for evidence that luminous AGN impact their host galaxies. Here, we review some of this work. Multi-phase AGN outflows are common, some with potential for significant impact. Additionally, multiple feedback channels can be observed simultaneously; e.g., radio jets from `radio quiet' quasars can inject turbulence on ISM scales, and displace CGM-scale molecular gas. However, caution must be taken comparing outflows to simulations (e.g., kinetic coupling efficiencies) to infer feedback potential, due to a lack of comparable predictions. Furthermore, some work claims limited evidence for feedback because AGN live in gas-rich, star-forming galaxies. However, simulations do not predict instantaneous, global impact on molecular gas or star formation. The impact is expected to be cumulative, over multiple episodes., Comment: Accepted for publication in IAU Symposium 378 Conference Proceedings "Black Hole Winds at all Scales"

We present a novel approach to inspecting galaxy spectra using sound, via their direct audio representation ('spectral audification'). We discuss the potential of this as a complement to (or stand-in for) visual approaches. We surveyed 58 respondents who use the audio representation alone to rate 30 optical galaxy spectra with strong emission lines. Across three tests, each focusing on different quantities measured from the spectra (signal-to-noise ratio, emission-line width, & flux ratios), we find that user ratings are well correlated with measured quantities. This demonstrates that physical information can be independently gleaned from listening to spectral audifications. We note the importance of context when rating these sonifications, where the order examples are heard can influence responses. Finally, we adapt the method used in this promising pilot study to spectral datacubes. We suggest that audification allows efficient exploration of complex, spatially-resolved spectral data., Comment: 6 pages, 3 figures, accepted for publication in RASTI. Supplementary data (including animated figure) available at https://doi.org/10.25405/data.ncl.22816442

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks., Comment: CVPR 2023 (Highlighted Paper). Website: https://imagebind.metademolab.com/ Code/Models: https://github.com/facebookresearch/ImageBind

Coral reefs are fast-changing and complex ecosystems that are crucial to monitor and study. Biological hotspot detection can help coral reef managers prioritize limited resources for monitoring and intervention tasks. Here, we explore the use of autonomous underwater vehicles (AUVs) with cameras, coupled with visual detectors and photogrammetry, to map and identify these hotspots. This approach can provide high spatial resolution information in fast feedback cycles. To the best of our knowledge, we present one of the first attempts at using an AUV to gather visually-observed, fine-grain biological hotspot maps in concert with topography of a coral reefs. Our hotspot maps correlate with rugosity, an established proxy metric for coral reef biodiversity and abundance, as well as with our visual inspections of the 3D reconstruction. We also investigate issues of scaling this approach when applied to new reefs by using these visual detectors pre-trained on large public datasets., Comment: CV4Animals Workshop at CVPR 2023

Large language models (LLMs) have been leveraged for several years now, obtaining state-of-the-art performance in recognizing entities from modern documents. For the last few months, the conversational agent ChatGPT has "prompted" a lot of interest in the scientific community and public due to its capacity of generating plausible-sounding answers. In this paper, we explore this ability by probing it in the named entity recognition and classification (NERC) task in primary sources (e.g., historical newspapers and classical commentaries) in a zero-shot manner and by comparing it with state-of-the-art LM-based systems. Our findings indicate several shortcomings in identifying entities in historical text that range from the consistency of entity annotation guidelines, entity complexity, and code-switching, to the specificity of prompting. Moreover, as expected, the inaccessibility of historical archives to the public (and thus on the Internet) also impacts its performance., Comment: 5 pages, accepted to JCDL2023

This paper revisits the standard pretrain-then-finetune paradigm used in computer vision for visual recognition tasks. Typically, state-of-the-art foundation models are pretrained using large scale (weakly) supervised datasets with billions of images. We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well. Thus, our MAE-based pre-pretraining scales with both model and data size making it applicable for training foundation models. Pre-pretraining consistently improves both the model convergence and the downstream transfer performance across a range of model scales (millions to billions of parameters), and dataset sizes (millions to billions of images). We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition. Our largest model achieves new state-of-the-art results on iNaturalist-18 (91.7%), ImageNet-ReaL (91.1%), 1-shot ImageNet-1k (63.6%), and zero-shot transfer on Food-101 (96.2%). Our study reveals that model initialization plays a significant role, even for web-scale pretraining with billions of images, and our models are available publicly., Comment: ICCV 2023. Models available at https://github.com/facebookresearch/maws/

Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater. State-of-the-art methods for underwater color correction optimize the parameters of image formation models to restore the full spectrum of color to underwater imagery. However, these methods have high computational complexity that is unfavourable for realtime use by autonomous underwater vehicles (AUVs), as a result of having been primarily designed for offline color correction. Here, we present DeepSeeColor, a novel algorithm that combines a state-of-the-art underwater image formation model with the computational efficiency of deep learning frameworks. In our experiments, we show that DeepSeeColor offers comparable performance to the popular "Sea-Thru" algorithm (Akkaynak & Treibitz, 2019) while being able to rapidly process images at up to 60Hz, thus making it suitable for use onboard AUVs as a preprocessing step to enable more robust vision-based behaviours., Comment: 7 pages, 5 figures, 2 tables. Presented at the 2023 International Conference on Robotics and Automation (ICRA)

The current approach to exploring and monitoring complex underwater ecosystems, such as coral reefs, is to conduct surveys using diver-held or static cameras, or deploying sensor buoys. These approaches often fail to capture the full variation and complexity of interactions between different reef organisms and their habitat. The CUREE platform presented in this paper provides a unique set of capabilities in the form of robot behaviors and perception algorithms to enable scientists to explore different aspects of an ecosystem. Examples of these capabilities include low-altitude visual surveys, soundscape surveys, habitat characterization, and animal following. We demonstrate these capabilities by describing two field deployments on coral reefs in the US Virgin Islands. In the first deployment, we show that CUREE can identify the preferred habitat type of snapping shrimp in a reef through a combination of a visual survey, habitat characterization, and a soundscape survey. In the second deployment, we demonstrate CUREE's ability to follow arbitrary animals by separately following a barracuda and stingray for several minutes each in midwater and benthic environments, respectively., Comment: 7 pages

Recipe personalization through ingredient substitution has the potential to help people meet their dietary needs and preferences, avoid potential allergens, and ease culinary exploration in everyone's kitchen. To address ingredient substitution, we build a benchmark, composed of a dataset of substitution pairs with standardized splits, evaluation metrics, and baselines. We further introduce Graph-based Ingredient Substitution Module (GISMo), a novel model that leverages the context of a recipe as well as generic ingredient relational information encoded within a graph to rank plausible substitutions. We show through comprehensive experimental validation that GISMo surpasses the best performing baseline by a large margin in terms of mean reciprocal rank. Finally, we highlight the benefits of GISMo by integrating it in an improved image-to-recipe generation pipeline, enabling recipe personalization through user intervention. Quantitative and qualitative results show the efficacy of our proposed system, paving the road towards truly personalized cooking and tasting experiences.

In-situ visual observations of marine organisms is crucial to developing behavioural understandings and their relations to their surrounding ecosystem. Typically, these observations are collected via divers, tags, and remotely-operated or human-piloted vehicles. Recently, however, autonomous underwater vehicles equipped with cameras and embedded computers with GPU capabilities are being developed for a variety of applications, and in particular, can be used to supplement these existing data collection mechanisms where human operation or tags are more difficult. Existing approaches have focused on using fully-supervised tracking methods, but labelled data for many underwater species are severely lacking. Semi-supervised trackers may offer alternative tracking solutions because they require less data than fully-supervised counterparts. However, because there are not existing realistic underwater tracking datasets, the performance of semi-supervised tracking algorithms in the marine domain is not well understood. To better evaluate their performance and utility, in this paper we provide (1) a novel dataset specific to marine animals located at http://warp.whoi.edu/vmat/, (2) an evaluation of state-of-the-art semi-supervised algorithms in the context of underwater animal tracking, and (3) an evaluation of real-world performance through demonstrations using a semi-supervised algorithm on-board an autonomous underwater vehicle to track marine animals in the wild., Comment: To appear in IJCV SI: Animal Tracking