Search

We introduce Edify Image, a family of diffusion models capable of generating photorealistic image content with pixel-perfect accuracy. Edify Image utilizes cascaded pixel-space diffusion models trained using a novel Laplacian diffusion process, in which image signals at different frequency bands are attenuated at varying rates. Edify Image supports a wide range of applications, including text-to-image synthesis, 4K upsampling, ControlNets, 360 HDR panorama generation, and finetuning for image customization.

Physical insight into plasma evolution in the magnetohydrodynamic (MHD) limit can be revealed by decomposing the evolution according to the characteristic modes of the system. In this paper we explore aspects of the eigenenergy decomposition method (EEDM) introduced in an earlier study (Raboonik et al. 2024 , ApJ, 967:80). The EEDM provides an exact decomposition of nonlinear MHD disturbances into their component eigenenergies associated with the slow, Alfv\'en, and fast eigenmodes, together with two zero-frequency eigenmodes. Here we refine the EEDM by presenting globally analytical expressions for the eigenenergies. We also explore the nature of the zero-frequency ``pseudo-advective modes'' in detail. We show that in evolutions with pure advection of magnetic and thermal energy (without propagating waves) a part of the energy is carried by the pseudo advective modes. Exact expressions for the error terms associated with these modes--commonly encountered in numerical simulations--are also introduced. The new EEDM equations provide a robust tool for the exact and unique decomposition of nonlinear disturbances governed by homogeneous quasi-linear partial differential equations, even in the presence of local or global degeneracies.

Advances in large-scale artificial neural networks have facilitated novel insights into the functional topology of the brain. Here, we leverage this approach to study how semantic categories are organized in the human visual cortex. To overcome the challenge presented by the co-occurrence of multiple categories in natural images, we introduce BrainSAIL (Semantic Attribution and Image Localization), a method for isolating specific neurally-activating visual concepts in images. BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs a novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns in different areas of the higher visual cortex. We validate BrainSAIL on cortical regions with known category selectivity, demonstrating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL's ability to characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant advances in mapping and decomposing high-level visual representations in the human brain.

Science communication skills are considered essential learning objectives for undergraduate physics students. However, high enrollment and limited class resources present significant barriers to providing students ample opportunities to practice their formal presentation skills. We investigate the use of integrated critical reflection and peer evaluation activities in a physics senior seminar course both to improve student learning outcomes and to supplement highly restricted presentation time. Throughout the semester, each student delivers one 8-min multimedia presentation on either their research or an upper-division course topic. Following each presentation, audience members complete one of two randomly assigned peer evaluations: a treatment form that prompts critical reflection or a control form that does not. Each class period concludes with a short quiz on concepts presented in that day's presentations. We observe minimal differences in quiz scores between students in the control and treatment groups. Instead, we find that retention and transfer of presentation content correlate with certain metrics of presentation quality described in the Cognitive Theory of Multimedia Learning and with self-identified prior exposure to presentation topics., Comment: 6 pages, 3 figures, 2024 Physics Education Research Conference (PERC) Proceedings

The National Science Foundation's Daniel K. Inouye Solar Telescope (DKIST) will provide high-resolution, multi-line spectropolarimetric observations that are poised to revolutionize our understanding of the Sun. Given the massive data volume, novel inference techniques are required to unlock its full potential. Here, we provide an overview of our "SPIn4D" project, which aims to develop deep convolutional neural networks (CNNs) for estimating the physical properties of the solar photosphere from DKIST spectropolarimetric observations. We describe the magnetohydrodynamic (MHD) modeling and the Stokes profile synthesis pipeline that produce the simulated output and input data, respectively. These data will be used to train a set of CNNs that can rapidly infer the four-dimensional MHD state vectors by exploiting the spatiotemporally coherent patterns in the Stokes profile time series. Specifically, our radiative MHD model simulates the small-scale dynamo actions that are prevalent in quiet-Sun and plage regions. Six cases with different mean magnetic fields have been conducted; each case covers six solar-hours, totaling 109 TB in data volume. The simulation domain covers at least $25\times25\times8$ Mm with $16\times16\times12$ km spatial resolution, extending from the upper convection zone up to the temperature minimum region. The outputs are stored at a 40 s cadence. We forward model the Stokes profile of two sets of Fe I lines at 630 and 1565 nm, which will be simultaneously observed by DKIST and can better constrain the parameter variations along the line of sight. The MHD model output and the synthetic Stokes profiles are publicly available, with 13.7 TB in the initial release., Comment: Submitted to AAS Journals after revision. Comments welcome

Strip and pixels sensors, fabricated on high resistivity silicon substrate, normally of p-type, are used in detectors for High Energy Physics (HEP) typically in a hybrid detector assembly. Furthermore, and owing to their inherent advantages over hybrid sensors, Monolithic Active Pixel Sensors (MAPS) fabricated in CMOS technology have been increasingly implemented in HEP experiments. In all cases, their use in higher radiation areas (HL-LHC and beyond) will require options to improve their radiation hardness and time resolution. These aspects demand a deep understanding of their radiation damage and reliable models to predict their behaviours at high fluences. As a first step, we fabricated several Schottky and n-on-p diodes, to allow a comparison of results and provide a backup solution for test devices, on 6 or 4-inch p-type silicon wafers with 50 {\mu}m epitaxial thickness and of doping concentration as they are normally used in HEP detectors and CMOS MAPS devices. In this paper, details of the design and fabrication process, along with test results of the fabricated devices before irradiation, will be provided. Additional test results on irradiated devices will be provided in subsequent publications.

Large-scale generative language and vision-language models excel in in-context learning for decision making. However, they require high-quality exemplar demonstrations to be included in their context window. In this work, we ask: Can LLMs and VLMs generate their own examples from generic, sub-optimal demonstrations? We propose In-Context Abstraction Learning (ICAL), a method that builds a memory of multimodal experience from sub-optimal demonstrations and human feedback. Given a task demonstration that may contain inefficiencies or mistakes, a VLM abstracts the trajectory into a generalized program by correcting inefficient actions and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These abstractions are iteratively improved through human feedback while the agent attempts to execute the trajectory. The resulting examples, when used as exemplars in the prompt, significantly improve decision-making in retrieval-augmented LLM and VLM agents. Moreover, as the agent's library of examples grows, it becomes more efficient, relying less on human feedback and requiring fewer environment interactions per demonstration. Our ICAL agent surpasses the state-of-the-art in dialogue-based instruction following in TEACh, multimodal web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our task success rate improves over the SOTA from 14.3% to 22.7% using GPT4V. In Ego4D action forecasting, we improve over few-shot GPT-4V and remain competitive with supervised models. We show finetuning our retrieval-augmented in-context agent yields additional improvements. Our approach significantly reduces reliance on manual prompt engineering and consistently outperforms in-context learning from action plans that lack such abstractions., Comment: Project website: http://ical-learning.github.io/

Understanding the semantics of visual scenes is a fundamental challenge in Computer Vision. A key aspect of this challenge is that objects sharing similar semantic meanings or functions can exhibit striking visual differences, making accurate identification and categorization difficult. Recent advancements in text-to-image frameworks have led to models that implicitly capture natural scene statistics. These frameworks account for the visual variability of objects, as well as complex object co-occurrences and sources of noise such as diverse lighting conditions. By leveraging large-scale datasets and cross-attention conditioning, these models generate detailed and contextually rich scene representations. This capability opens new avenues for improving object recognition and scene understanding in varied and challenging environments. Our work presents StableSemantics, a dataset comprising 224 thousand human-curated prompts, processed natural language captions, over 2 million synthetic images, and 10 million attention maps corresponding to individual noun chunks. We explicitly leverage human-generated prompts that correspond to visually interesting stable diffusion generations, provide 10 generations per phrase, and extract cross-attention maps for each image. We explore the semantic distribution of generated images, examine the distribution of objects within images, and benchmark captioning and open vocabulary segmentation methods on our data. To the best of our knowledge, we are the first to release a diffusion dataset with semantic attributions. We expect our proposed dataset to catalyze advances in visual semantic understanding and provide a foundation for developing more sophisticated and effective visual models. Website: https://stablesemantics.github.io/StableSemantics, Comment: Dataset website: https://stablesemantics.github.io/StableSemantics

Humans experience the world through constantly changing visual stimuli, where scenes can shift and move, change in appearance, and vary in distance. The dynamic nature of visual perception is a fundamental aspect of our daily lives, yet the large majority of research on object and scene processing, particularly using fMRI, has focused on static stimuli. While studies of static image perception are attractive due to their computational simplicity, they impose a strong non-naturalistic constraint on our investigation of human vision. In contrast, dynamic visual stimuli offer a more ecologically-valid approach but present new challenges due to the interplay between spatial and temporal information, making it difficult to disentangle the representations of stable image features and motion. To overcome this limitation -- given dynamic inputs, we explicitly decouple the modeling of static image representations and motion representations in the human brain. Three results demonstrate the feasibility of this approach. First, we show that visual motion information as optical flow can be predicted (or decoded) from brain activity as measured by fMRI. Second, we show that this predicted motion can be used to realistically animate static images using a motion-conditioned video diffusion model (where the motion is driven by fMRI brain activity). Third, we show prediction in the reverse direction: existing video encoders can be fine-tuned to predict fMRI brain activity from video imagery, and can do so more effectively than image encoders. This foundational work offers a novel, extensible framework for interpreting how the human brain processes dynamic visual information.

Recent research on instructable agents has used memory-augmented Large Language Models (LLMs) as task planners, a technique that retrieves language-program examples relevant to the input instruction and uses them as in-context examples in the LLM prompt to improve the performance of the LLM in inferring the correct action and task plans. In this technical report, we extend the capabilities of HELPER, by expanding its memory with a wider array of examples and prompts, and by integrating additional APIs for asking questions. This simple expansion of HELPER into a shared memory enables the agent to work across the domains of executing plans from dialogue, natural language instruction following, active question asking, and commonsense room reorganization. We evaluate the agent on four diverse interactive visual-language embodied agent benchmarks: ALFRED, TEACh, DialFRED, and the Tidy Task. HELPER-X achieves few-shot, state-of-the-art performance across these benchmarks using a single agent, without requiring in-domain training, and remains competitive with agents that have undergone in-domain training., Comment: Videos and code https://helper-agent-llm.github.io/

We report on the discovery and validation of a transiting long-period mini-Neptune orbiting a bright (V = 9.0 mag) G dwarf (TOI 4633; R = 1.05 RSun, M = 1.10 MSun). The planet was identified in data from the Transiting Exoplanet Survey Satellite by citizen scientists taking part in the Planet Hunters TESS project. Modeling of the transit events yields an orbital period of 271.9445 +/- 0.0040 days and radius of 3.2 +/- 0.20 REarth. The Earth-like orbital period and an incident flux of 1.56 +/- 0.2 places it in the optimistic habitable zone around the star. Doppler spectroscopy of the system allowed us to place an upper mass limit on the transiting planet and revealed a non-transiting planet candidate in the system with a period of 34.15 +/- 0.15 days. Furthermore, the combination of archival data dating back to 1905 with new high angular resolution imaging revealed a stellar companion orbiting the primary star with an orbital period of around 230 years and an eccentricity of about 0.9. The long period of the transiting planet, combined with the high eccentricity and close approach of the companion star makes this a valuable system for testing the formation and stability of planets in binary systems., Comment: 24 pages, 16 figures, 4 tables

In this paper we introduce a new method for exact decomposition of propagating, nonlinear magnetohydrodynamic (MHD) disturbances into their component eigenenergies associated with the familiar slow, Alfv\'en, and fast wave eigenmodes, and the entropy and field-divergence pseudo-eigenmodes. First the mathematical formalism is introduced, where it is illustrated how the ideal-MHD eigensystem can be used to construct a decomposition of the time variation of the total energy density into contributions from the eigenmodes. The decomposition method is then demonstrated by applying it to the output of three separate nonlinear MHD simulations. The analysis of the simulations confirms that the component wave modes of a composite wavefield are uniquely identified by the method. The slow, Alfv\'en, and fast energy densities are shown to evolve in exactly the way expected from comparison with known linear solutions and nonlinear properties, including processes such as mode conversion. Along the way, some potential pitfalls for the numerical implementation of the decomposition method are identified and discussed. We conclude that the exact, nonlinear decomposition method introduced is a powerful and promising tool for understanding the nature of the decomposition of MHD waves as well as analysing and interpreting the output of dynamic MHD simulations.

Do machines and humans process language in similar ways? Recent research has hinted in the affirmative, finding that brain signals can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there are also clear differences in how LMs and humans represent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories. Using a data-driven approach, we identify two domains that are not captured well by LMs: social/emotional intelligence and physical commonsense. We then validate these domains with human behavioral experiments and show that fine-tuning LMs on these domains can improve their alignment with human brain responses.

We develop a general description of how information propagates through a magnetohydrodynamic (MHD) system based on the method of characteristics and use that to formulate numerical boundary conditions (BCs) that are intrinsically consistent with the MHD equations. Our formulation includes two major advances for simulations of the Sun. First, we derive data-driven BCs that optimally match the state of the plasma inferred from a time series of observations of a boundary (e.g., the solar photosphere). Second, our method directly handles random noise and systematic bias in the observations, and finds a solution for the boundary evolution that is strictly consistent with MHD and maximally consistent with the observations. We validate the method against a Ground Truth (GT) simulation of an expanding spheromak. The data-driven simulation can reproduce the GT simulation above the photosphere with high fidelity when driven at high cadence. Errors progressively increase for lower driving cadence until a threshold cadence is reached and the driven simulation can no longer accurately reproduce the GT simulation. However, our characteristic formulation of the BCs still requires adherence of the boundary evolution to the MHD equations even when the driven solution departs from the true solution in the driving layer. That increasing departure clearly indicates when additional information at the boundary is needed to fully specify the correct evolution of the system. The method functions even when no information about the evolution of some variables on the lower boundary is available, albeit with a further decrease in fidelity., Comment: Submitted for publication to the Astrophysical Journal; 77 pages, 12 figures

Pre-trained and frozen large language models (LLMs) can effectively map simple scene rearrangement instructions to programs over a robot's visuomotor functions through appropriate few-shot example prompting. To parse open-domain natural language and adapt to a user's idiosyncratic procedures, not known during prompt engineering time, fixed prompts fall short. In this paper, we introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting: relevant memories are retrieved based on the current dialogue, instruction, correction, or VLM description, and used as in-context prompt examples for LLM querying. The memory is expanded during deployment to include pairs of user's language and action plans, to assist future inferences and personalize them to the user's language and routines. HELPER sets a new state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TfD), with a 1.7x improvement over the previous state-of-the-art for TfD. Our models, code, and video results can be found in our project's website: https://helper-agent-llm.github.io., Comment: Project page with code & videos: https://helper-agent-llm.github.io

Understanding the functional organization of higher visual cortex is a central focus in neuroscience. Past studies have primarily mapped the visual and semantic selectivity of neural populations using hand-selected stimuli, which may potentially bias results towards pre-existing hypotheses of visual cortex functionality. Moving beyond conventional approaches, we introduce a data-driven method that generates natural language descriptions for images predicted to maximally activate individual voxels of interest. Our method -- Semantic Captioning Using Brain Alignments ("BrainSCUBA") -- builds upon the rich embedding space learned by a contrastive vision-language model and utilizes a pre-trained large language model to generate interpretable captions. We validate our method through fine-grained voxel-level captioning across higher-order visual regions. We further perform text-conditioned image synthesis with the captions, and show that our images are semantically coherent and yield high predicted activations. Finally, to demonstrate how our method enables scientific discovery, we perform exploratory investigations on the distribution of "person" representations in the brain, and discover fine-grained semantic selectivity in body-selective areas. Unlike earlier studies that decode text, our method derives voxel-wise captions of semantic selectivity. Our results show that BrainSCUBA is a promising means for understanding functional preferences in the brain, and provides motivation for further hypothesis-driven investigation of visual cortex., Comment: ICLR 2024. Project page: https://www.cs.cmu.edu/~afluo/BrainSCUBA

Deep neural network representations align well with brain activity in the ventral visual stream. However, the primate visual system has a distinct dorsal processing stream with different functional properties. To test if a model trained to perceive 3D scene geometry aligns better with neural responses in dorsal visual areas, we trained a self-supervised geometry-aware recurrent neural network (GRNN) to predict novel camera views using a 3D feature memory. We compared GRNN to self-supervised baseline models that have been shown to align well with ventral regions using the large-scale fMRI Natural Scenes Dataset (NSD). We found that while the baseline models accounted better for ventral brain regions, GRNN accounted for a greater proportion of variance in dorsal brain regions. Our findings demonstrate the potential for using task-relevant models to probe representational differences across visual streams., Comment: 2023 Conference on Cognitive Computational Neuroscience

Cellwise contamination remains a challenging problem for data scientists, particularly in research fields that require the selection of sparse features. Traditional robust methods may not be feasible nor efficient in dealing with such contaminated datasets. We propose CR-Lasso, a robust Lasso-type cellwise regularization procedure that performs feature selection in the presence of cellwise outliers by minimising a regression loss and cell deviation measure simultaneously. To evaluate the approach, we conduct empirical studies comparing its selection and prediction performance with several sparse regression methods. We show that CR-Lasso is competitive under the settings considered. We illustrate the effectiveness of the proposed method on real data through an analysis of a bone mineral density dataset.

Large-scale datasets are essential to modern day deep learning. Advocates argue that understanding these methods requires dataset transparency (e.g. "dataset curation, motivation, composition, collection process, etc..."). However, almost no one has suggested the release of the detailed definitions and visual category examples provided to annotators - information critical to understanding the structure of the annotations present in each dataset. These labels are at the heart of public datasets, yet few datasets include the instructions that were used to generate them. We introduce a new task, Labeling Instruction Generation, to address missing publicly available labeling instructions. In Labeling Instruction Generation, we take a reasonably annotated dataset and: 1) generate a set of examples that are visually representative of each category in the dataset; 2) provide a text label that corresponds to each of the examples. We introduce a framework that requires no model training to solve this task and includes a newly created rapid retrieval system that leverages a large, pre-trained vision and language model. This framework acts as a proxy to human annotators that can help to both generate a final labeling instruction set and evaluate its quality. Our framework generates multiple diverse visual and text representations of dataset categories. The optimized instruction set outperforms our strongest baseline across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.

A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach -- Brain Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.

We study the dynamics of an oscillating, free-floating robot that generates radially expanding gravity capillary waves at a fluid surface. In open water, the device does not self-propel; near a rigid boundary, it can be attracted or repelled. Visualization of the wave field dynamics reveals that when near a boundary, a complex interference of generated and reflected waves induces a wave amplitude fluctuation asymmetry. Attraction increases as wave frequency increases or robot-boundary separation decreases. Theory on confined gravity-capillary wave radiation dynamics developed by Hocking in the 1980s captures the observed parameter dependence due to these "Hocking fields." The flexibility of the robophysical system allows detailed characterization and analysis of locally generated nonequilibrium fluctuation-induced forces [M. Kardar and R. Golestanian, Rev. Mod. Phys. 71, 1233 (1999)]., Comment: 11 pages, 5 figures, 8 supplementary figures

Children typically learn the meanings of nouns earlier than the meanings of verbs. However, it is unclear whether this asymmetry is a result of complexity in the visual structure of categories in the world to which language refers, the structure of language itself, or the interplay between the two sources of information. We quantitatively test these three hypotheses regarding early verb learning by employing visual and linguistic representations of words sourced from large-scale pre-trained artificial neural networks. Examining the structure of both visual and linguistic embedding spaces, we find, first, that the representation of verbs is generally more variable and less discriminable within domain than the representation of nouns. Second, we find that if only one learning instance per category is available, visual and linguistic representations are less well aligned in the verb system than in the noun system. However, in parallel with the course of human language development, if multiple learning instances per category are available, visual and linguistic representations become almost as well aligned in the verb system as in the noun system. Third, we compare the relative contributions of factors that may predict learning difficulty for individual words. A regression analysis reveals that visual variability is the strongest factor that internally drives verb learning, followed by visual-linguistic alignment and linguistic variability. Based on these results, we conclude that verb acquisition is influenced by all three sources of complexity, but that the variability of visual structure poses the most significant challenge for verb learning.

The strongest known correspondence between ALMA 3mm emission and other solar observations is between the H-alpha line width and 3 mm brightness temperature, while the typical 3-5min p-mode oscillations found in many chromospheric diagnostics are often lacking from ALMA Band 3 and 6 observations. We study these issues using a publicly available data set of weak network flux near disk center at time SOL2017-03-17T15:42-16:45 that includes IBIS H-alpha and ALMA 3 mm data series. We confirm the correlation between the H-alpha line width and the 3 mm temperature, but find a different slope between the two diagnostics for hot versus cool regions, both of which are steeper than previous reports. The origin of the two slopes is unknown, but does hold for the duration of the observations. Spatially averaged power spectra of the IBIS data do show p-mode oscillations while the ALMA data do not. However, removing IBIS data at times corresponding to the ALMA calibration windows makes the averaged power spectra for the two data series nearly identical. Spatial maps of the power integrated over p-mode frequency bands agree well between the two data series and show the typical pattern of magnetic shadows and halos found in many chromospheric diagnostics. We therefore argue that the lack of observed p-modes in the ALMA data may be predominantly due to spectral windowing induced by the timing and duration of the calibration observations., Comment: 25 pages, 12 figures, Accepted for publication in Frontiers in Astronomy and Space Science special topic "The Sun Seen with the Atacama Large mm and sub-mm Array (ALMA) - First Results"

Study of the locomotion of a centipede (L. forficatus) at the air-water interface reveals that it does not predominantly use its 14 leg pairs to locomote; unlike most swimmers which propagate head-to-tail body bending waves, this species propels via tail-to-head waves. Its low mass and body-fluid contact yield locomotion dynamics in which fluid wave drag forces dominate inertia. Microorganism-inspired wave drag resistive force theory captures swimming performance of the macroscale animal, motivating a control hypothesis for the animals self-propulsion body and limb kinematics., Comment: Drag measurements presented in this paper are partially incorrect. New drag measurements that are accurate and correct will be included in an updated manuscript. Once the manuscript is complete, it will be submitted as a replacement

Domain adaptation for sensor-based activity learning is of utmost importance in remote health monitoring research. However, many domain adaptation algorithms suffer with failure to operate adaptation in presence of target domain heterogeneity (which is always present in reality) and presence of multiple inhabitants dramatically hinders their generalizability producing unsatisfactory results for semi-supervised and unseen activity learning tasks. We propose \emph{AEDA}, a novel deep auto-encoder-based model to enable semi-supervised domain adaptation in the existence of target domain heterogeneity and how to incorporate it to empower heterogeneity to any homogeneous deep domain adaptation architecture for cross-domain activity learning. Experimental evaluation on 18 different heterogeneous and multi-inhabitants use-cases of 8 different domains created from 2 publicly available human activity datasets (wearable and ambient smart homes) shows that \emph{AEDA} outperforms (max. 12.8\% and 8.9\% improvements for ambient smart home and wearables) over existing domain adaptation techniques for both seen and unseen activity learning in a heterogeneous setting.

We introduce TIDEE, an embodied agent that tidies up a disordered scene based on learned commonsense object placement and room arrangement priors. TIDEE explores a home environment, detects objects that are out of their natural place, infers plausible object contexts for them, localizes such contexts in the current scene, and repositions the objects. Commonsense priors are encoded in three modules: i) visuo-semantic detectors that detect out-of-place objects, ii) an associative neural graph memory of objects and spatial relations that proposes plausible semantic receptacles and surfaces for object repositions, and iii) a visual search network that guides the agent's exploration for efficiently localizing the receptacle-of-interest in the current scene to reposition the object. We test TIDEE on tidying up disorganized scenes in the AI2THOR simulation environment. TIDEE carries out the task directly from pixel and raw depth input without ever having observed the same room beforehand, relying only on priors learned from a separate set of training houses. Human evaluations on the resulting room reorganizations show TIDEE outperforms ablative versions of the model that do not use one or more of the commonsense priors. On a related room rearrangement benchmark that allows the agent to view the goal state prior to rearrangement, a simplified version of our model significantly outperforms a top-performing method by a large margin. Code and data are available at the project website: https://tidee-agent.github.io/.

Faced with changing markets and evolving consumer demands, beef industries are investing in grading systems to maximise value extraction throughout their entire supply chain. The Meat Standards Australia (MSA) system is a customer-oriented total quality management system that stands out internationally by predicting quality grades of specific muscles processed by a designated cooking method. The model currently underpinning the MSA system requires laborious effort to estimate and its prediction performance may be less accurate in the presence of unbalanced data sets where many "muscle x cook" combinations have few observations and/or few predictors of palatability are available. This paper proposes a novel predictive method for beef eating quality that bridges a spectrum of muscle x cook-specific models. At one extreme, each muscle x cook combination is modelled independently; at the other extreme a pooled predictive model is obtained across all muscle x cook combinations. Via a data-driven regularization method, we cover all muscle x cook-specific models along this spectrum. We demonstrate that the proposed predictive method attains considerable accuracy improvements relative to independent or pooled approaches on unique MSA data sets., Comment: 26 pages, 5 figures

New instruments sensitive to chromospheric radiation at X-ray, UV, Visible, IR, and sub-mm wavelengths have become available that significantly enhance our ability to understand the bi-directional flow of energy through the chromosphere. We describe the calibration, co-alignment, initial results, and public release of a new data set combining a large number of these instruments to obtain multi-wavelength photospheric, chromospheric, and coronal observations capable of improving our understanding of the connectivity between the photosphere and the corona via transient brightenings and wave signatures. The observations center on a bipolar region of enhanced network magnetic flux near disk center on SOL2017-03-17T14:00-17:00. The comprehensive data set provides one of the most complete views of chromospheric activity related to small scale brightenings in the corona and chromosphere to date. Our initial analysis shows strong spatial correspondence between the areas of broadest width of the Hydrogen-$\alpha$ spectral line and the hottest temperatures observed in ALMA Band 3 radio data, with a linear coefficient of $6.12\times 10^{-5}$\AA{}/K. The correspondence persists for the duration of co-temporal observations ($\approx 60$ minutes). Numerous transient brightenings were observed in multiple data series. We highlight a single, well observed transient brightening along a set of thin filamentary features with a duration of 20 minutes. The timing of the peak intensity transitions from the cooler (ALMA, 7000 K) to hotter (XRT, 3 MK) data series., Comment: 22 pages, 15 figures, accepted by ApJS

We present a new approach for correcting instrumental polarization by modeling the non-depolarizing effects of a complex series of optical elements to determine physically realizable Mueller matrices. Provided that the Mueller matrix of the optical system can be decomposed into a general elliptical diattenuator and general elliptical retarder, it is possible to model the cross-talk between both the polarized and unpolarized states of the Stokes vector and then use the acquired science observations to determine the best-fit free parameters. Here, we implement a minimization for solar spectropolarimetric measurements containing photospheric spectral lines sensitive to the Zeeman effect using physical constraints provided by polarized line and continuum formation. This model-based approach is able to provide an accurate correction even in the presence of large amounts of polarization cross-talk and conserves the physically meaningful magnitude of the Stokes vector, a significant improvement over previous ad hoc techniques., Comment: 16 pages, 4 figures, Accepted for publication in ApJ

Our environment is filled with rich and dynamic acoustic information. When we walk into a cathedral, the reverberations as much as appearance inform us of the sanctuary's wide open space. Similarly, as an object moves around us, we expect the sound emitted to also exhibit this movement. While recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning spatial auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene. By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds. We demonstrate that the continuous nature of NAFs enables us to render spatial acoustics for a listener at an arbitrary location, and can predict sound propagation at novel locations. We further show that the representation learned by NAFs can help improve visual learning with sparse views. Finally, we show that a representation informative of scene structure emerges during the learning of NAFs., Comment: NeurIPS 2022. Project page: https://www.andrew.cmu.edu/user/afluo/Neural_Acoustic_Fields/

Systems consisting of spheres rolling on elastic membranes have been used to introduce a core conceptual idea of General Relativity (GR): how curvature guides the movement of matter. However, such schemes cannot accurately represent relativistic dynamics in the laboratory because of the dominance of dissipation and external gravitational fields. Here we demonstrate that an ``active" object (a wheeled robot), which moves in a straight line on level ground and can alter its speed depending on the curvature of the deformable terrain it moves on, can exactly capture dynamics in curved relativistic spacetimes. Via the systematic study of the robot's dynamics in the radial and orbital directions, we develop a mapping of the emergent trajectories of a wheeled vehicle on a spandex membrane to the motion in a curved spacetime. Our mapping demonstrates how the driven robot's dynamics mix space and time in a metric, and shows how active particles do not necessarily follow geodesics in the real space but instead follow geodesics in a fiducial spacetime. The mapping further reveals how parameters such as the membrane elasticity and instantaneous speed allow the programming of a desired spacetime, such as the Schwarzschild metric near a non-rotating blackhole. Our mapping and framework facilitate creation of a robophysical analog to a general relativistic system in the laboratory at low cost that can provide insights into active matter in deformable environments and robot exploration in complex landscapes.