Search

Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable performance in vision and language reasoning capabilities for VIL tasks. Despite the progress, current VIL methods naively employ VLMs to learn high-level plans from human videos, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck. In this work, we present VLMimic, a novel paradigm that harnesses VLMs to directly learn even fine-grained action levels, only given a limited number of human videos. Specifically, VLMimic first grounds object-centric movements from human videos, and learns skills using hierarchical constraint representations, facilitating the derivation of skills with fine-grained action levels from limited human videos. These skills are refined and updated through an iterative comparison strategy, enabling efficient adaptation to unseen environments. Our extensive experiments exhibit that our VLMimic, using only 5 human videos, yields significant improvements of over 27% and 21% in RLBench and real-world manipulation tasks, and surpasses baselines by over 37% in long-horizon tasks., Comment: accepted for publication in the 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

Hydrodynamic second sound can be generated by heat pulses when the phonon-phonon interaction is dominantly momentum conserving, and the propagation of the temperature field becomes wavelike rather than diffusive. While the Boltzmann transport equation (BTE) has been widely applied to study phonon dynamics and thermal transport at the nanoscale, modeling the hydrodynamic transport regime remains challenging. The widely used relaxation time approximation (RTA) treats all phonon interactions as resistive without considering momentum conservation, resulting in the absence of phonon hydrodynamics. Rigorously solving BTE by inverting the full scattering matrix, however, is extremely computationally demanding and has been only applied to model one-dimensional temperature variations. Here, we present an analytical Green's function formalism for solving multidimensional Boltzmann transport equation (BTE) using phonon properties from first-principles calculations. This formalism involves Callaway's scattering approximation with separate relaxation times for momentum-conserving and momentum-destroying scattering events. The Green's function captures phonon dynamics in a wide range of temperature, spatial, and temporal scales, and successfully reproduces the transition from ballistic, hydrodynamic, to diffusive transport regimes. Our method avoids the computationally demanding inversion of large scattering matrices and shows good accuracies in predicting the temperature oscillation in ultrafast pump-probe characterizations with different geometries of thermal excitation.

As the signatories of this manifesto, we denounce the attention economy as inhumane and a threat to our sociopolitical and ecological well-being. We endorse policymakers' efforts to address the negative consequences of the attention economy's technology, but add that these approaches are often limited in their criticism of the systemic context of human attention. Starting from Buddhist philosophy, we advocate a broader approach: an ecology of attending, that centers on conceptualizing, designing, and using attention (1) in an embedded way and (2) focused on the alleviating of suffering. With 'embedded' we mean that attention is not a neutral, isolated mechanism but a meaning-engendering part of an 'ecology' of bodily, sociotechnical and moral frameworks. With 'focused on the alleviation of suffering' we explicitly move away from the (often implicit) conception of attention as a tool for gratifying desires., Comment: 21 pages, 1 figure

Seamlessly integrating stimulated Brillouin scattering (SBS) in a low-loss and mature photonic integration platform remains a complicated task. Virtually all current approaches fall short in simultaneously achieving strong SBS, low losses, and technological scalability. In this work we incorporate stong SBS into a standard silicon nitride platform by a simple deposition of a tellurium oxide layer, a commonly used material for acousto-optic modulators. In these heterogeneously integrated waveguides, we harness novel SBS interactions actuated by surface acoustic waves (SAWs) leading to more than two orders of magnitude gain enhancement. Three novel applications are demonstrated in this platform: (i) a silicon nitride Brillouin amplifier with 5 dB net optical gain, (ii) a compact intermodal stimulated Brillouin laser (SBL) capable of high purity radio frequency (RF) signal generation with 7 Hz intrinsic linewidth, and (iii) a widely tunable microwave photonic notch filter with ultra-narrow linewidth of 2.2 MHz enabled by Brillouin induced opacity. These advancements can unlock an array of new RF and optical technologies to be directly integrated in silicon nitride.

Against the backdrop of increasingly severe global environmental changes, accurately predicting and meeting renewable energy demands has become a key challenge for sustainable business development. Traditional energy demand forecasting methods often struggle with complex data processing and low prediction accuracy. To address these issues, this paper introduces a novel approach that combines deep learning techniques with environmental decision support systems. The model integrates advanced deep learning techniques, including LSTM and Transformer, and PSO algorithm for parameter optimization, significantly enhancing predictive performance and practical applicability. Results show that our model achieves substantial improvements across various metrics, including a 30% reduction in MAE, a 20% decrease in MAPE, a 25% drop in RMSE, and a 35% decline in MSE. These results validate the model's effectiveness and reliability in renewable energy demand forecasting. This research provides valuable insights for applying deep learning in environmental decision support systems., Comment: 25 pages

In this article, we consider the $d$-dimensional mollified stochastic heat equation (SHE) when the mollification parameter is turned off. Here, we concentrate on the high-dimensional case $d \geq 3$. Recently, the limiting higher moments of the two-dimensional mollified SHE have been established. However, this problem in high dimensions remains unexplored to date. The main theorems of this article aim to answer this question and prove some related properties: (1) Our first main result, based on the spectral theorem for the unbounded operator, proves the divergence of the higher moments of the high-dimensional mollified SHE even when the system is strictly inside the $L^{2}$-regime. This phenomenon is completely opposite to its two-dimensional counterpart; (2) To further differentiate the nature of the high-dimensional case from the case in two dimensions, our second main result proves the unboundedness of the sub-limiting higher moments of the three-dimensional mollified SHE at the $L^{2}$-criticality. Here, the sub-limiting higher moment is a natural limit of the higher moment of the three-dimensional mollified SHE at the $L^{2}$-criticality; (3) As an application, we provide partial results for the conjecture about the high-order critical regimes of the continuous directed polymer. The other byproduct of the above results gives a proper estimate for the critical exponent of the polymer in the $L^{2}$-regime.

Electric guitar tone modeling typically focuses on the non-linear transformation from clean to amplifier-rendered audio. Traditional methods rely on one-to-one mappings, incorporating device parameters into neural models to replicate specific amplifiers. However, these methods are limited by the need for specific training data. In this paper, we adapt a model based on the previous work, which leverages a tone embedding encoder and a feature wise linear modulation (FiLM) condition method. In this work, we altered conditioning method using a hypernetwork-based gated convolutional network (GCN) to generate audio that blends clean input with the tone characteristics of reference audio. By extending the training data to cover a wider variety of amplifier tones, our model is able to capture a broader range of tones. Additionally, we developed a real-time plugin to demonstrate the system's practical application, allowing users to experience its performance interactively. Our results indicate that the proposed system achieves superior tone modeling versatility compared to traditional methods., Comment: demo of the ISMIR paper

The total variation (TV) method is an image denoising technique that aims to reduce noise by minimizing the total variation of the image, which measures the variation in pixel intensities. The TV method has been widely applied in image processing and computer vision for its ability to preserve edges and enhance image quality. In this paper, we propose an improved TV model for image denoising and the associated numerical algorithm to carry out the procedure, which is particularly effective in removing several types of noises and their combinations. Our improved model admits a unique solution and the associated numerical algorithm guarantees the convergence. Numerical experiments are demonstrated to show improved effectiveness and denoising quality compared to other TV models. Such encouraging results further enhance the utility of the TV method in image processing.

Kepler-51 is a $\lesssim 1\,\mathrm{Gyr}$-old Sun-like star hosting three transiting planets with radii $\approx 6$-$9\,R_\oplus$ and orbital periods $\approx 45$-$130\,\mathrm{days}$. Transit timing variations (TTVs) measured with past Kepler and Hubble Space Telescope (HST) observations have been successfully modeled by considering gravitational interactions between the three transiting planets, yielding low masses and low mean densities ($\lesssim 0.1\,\mathrm{g/cm^3}$) for all three planets. However, the transit time of the outermost transiting planet Kepler-51d recently measured by the James Webb Space Telescope (JWST) 10 years after the Kepler observations is significantly discrepant from the prediction made by the three-planet TTV model, which we confirmed with ground-based and follow-up HST observations. We show that the departure from the three-planet model is explained by including a fourth outer planet, Kepler-51e, in the TTV model. A wide range of masses ($\lesssim M_\mathrm{Jup}$) and orbital periods ($\lesssim 10\,\mathrm{yr}$) are possible for Kepler-51e. Nevertheless, all the coplanar solutions found from our brute-force search imply masses $\lesssim 10\,M_\oplus$ for the inner transiting planets. Thus their densities remain low, though with larger uncertainties than previously estimated. Unlike other possible solutions, the one in which Kepler-51e is around the $2:1$ mean motion resonance with Kepler-51d implies low orbital eccentricities ($\lesssim 0.05$) and comparable masses ($\sim 5\,M_\oplus$) for all four planets, as is seen in other compact multi-planet systems. This work demonstrates the importance of long-term follow-up of TTV systems for probing longer period planets in a system., Comment: 48 pages, 26 figures, accepted for publication in AJ

This study presents the first global analysis of on-demand video streaming over Low Earth Orbit (LEO) satellite networks, using data from over one million households across 85 countries. We highlight Starlink's role as a major LEO provider, enhancing connectivity in underserved regions. Our findings reveal that while overall video quality on Starlink matches that of traditional networks, the inherent variability in LEO conditions -- such as throughput fluctuations and packet loss -- leads to an increase in bitrate switches and rebuffers. To further improve the quality of experience for the LEO community, we manipulate existing congestion control and adaptive bitrate streaming algorithms using simulation and real A/B tests deployed on over one million households. Our results underscore the need for video streaming and congestion control algorithms to adapt to rapidly evolving network landscapes, ensuring high-quality service across diverse and dynamic network types.

Eating disorders are complex mental health conditions that affect millions of people around the world. Effective interventions on social media platforms are crucial, yet testing strategies in situ can be risky. We present a novel LLM-driven experimental testbed for simulating and assessing intervention strategies in ED-related discussions. Our framework generates synthetic conversations across multiple platforms, models, and ED-related topics, allowing for controlled experimentation with diverse intervention approaches. We analyze the impact of various intervention strategies on conversation dynamics across four dimensions: intervention type, generative model, social media platform, and ED-related community/topic. We employ cognitive domain analysis metrics, including sentiment, emotions, etc., to evaluate the effectiveness of interventions. Our findings reveal that civility-focused interventions consistently improve positive sentiment and emotional tone across all dimensions, while insight-resetting approaches tend to increase negative emotions. We also uncover significant biases in LLM-generated conversations, with cognitive metrics varying notably between models (Claude-3 Haiku $>$ Mistral $>$ GPT-3.5-turbo $>$ LLaMA3) and even between versions of the same model. These variations highlight the importance of model selection in simulating realistic discussions related to ED. Our work provides valuable information on the complex dynamics of ED-related discussions and the effectiveness of various intervention strategies., Comment: 9 pages, 5 figures

We present the discovery of a low-density planet transiting TOI-5688 A b, a high-metallicity M2V star. This planet was discovered as part of the search for transiting giant planets ($R \gtrsim8$ M$_\oplus$) through the Searching for GEMS (Giant Exoplanets around M-dwarf Stars) survey. The planet TOI-5688 A b was discovered with the Transiting Exoplanet Survey Satellite (TESS), and characterized with ground-based transits from Red Buttes Observatory (RBO), the Table Mountain Observatory of Pomona College, and radial velocity (RV) measurements with the Habitable-Zone Planet Finder (HPF) on the 10 m Hobby Eberly Telescope (HET) and NEID on the WIYN 3.5 m telescope. From the joint fit of transit and RV data, the mass of the planet is $124\pm24$ M$_\oplus$ and the radius is $10.4\pm0.7$ R$_\oplus$. This planet has a density of $0.61^{+0.20}_{-0.15}$ g/cm${}^3$, and is on a $\sim2.95$ day orbit around its host star. The spectroscopic and photometric analysis of the host star TOI-5688 A shows that it is a high metallicity ([Fe/H] $ = 0.47\pm0.16$ dex) M2V star, favoring the core-accretion formation pathway as the likely formation scenario for this planet. In this paper, we analyze potential mechanisms of planet formation in the context of the formation of TOI-5688 A b. Additionally, observations with Gaia suggest the presence of a wide-separation binary companion, TOI-5688 B, which has a projected separation of $\sim5"$ (1110 AU) and is an M4V. This makes TOI-5688 A b part of a growing number of GEMS in wide-separation binary systems., Comment: 20 pages, 7 figures, Submitted to AJ, Comments are welcome

The Near Infrared Spectrograph (NIRSpec) on the James Webb Space Telescope affords the astronomical community an unprecedented space-based Multi-Object Spectroscopy (MOS) capability through the use of a programmable array of micro-electro-mechanical shutters. Launched in December 2021 and commissioned along with a suite of other observatory instruments throughout the first half of 2022, NIRSpec has been carrying out scientific observations since the completion of commissioning. These observations would not be possible without a rigorous program of engineering operations to actively monitor and maintain NIRSpec's hardware health and safety and enhance instrument efficiency and performance. Although MOS is only one of the observing modes available to users, the complexity and uniqueness of the Micro-Shutter Assembly (MSA) that enables it has presented a variety of engineering challenges, including the appearance of electrical shorts that produce contaminating glow in exposures. Despite these challenges, the NIRSpec Multi-Object Spectrograph continues to perform robustly with no discernible degradation or significant reduction in capability. This paper provides an overview of the NIRSpec micro-shutter subsystem's state of health and operability and presents some of the developments that have taken place in its operation since the completion of instrument commissioning., Comment: SPIE Astronomical Telescopes + Instrumentation: Ground-based and Airborne Instrumentation for Astronomy X (Yokohama 2024), paper number 13092-38

Transiting giant exoplanets around M-dwarf stars (GEMS) are rare, owing to the low-mass host stars. However, the all-sky coverage of TESS has enabled the detection of an increasingly large number of them to enable statistical surveys like the \textit{Searching for GEMS} survey. As part of this endeavour, we describe the observations of six transiting giant planets, which includes precise mass measurements for two GEMS (K2-419Ab, TOI-6034b) and statistical validation for four systems, which includes validation and mass upper limits for three of them (TOI-5218b, TOI-5616b, TOI-5634Ab), while the fourth one -- TOI-5414b is classified as a `likely planet'. Our observations include radial velocities from the Habitable-zone Planet Finder on the Hobby-Eberly Telescope, and MAROON-X on Gemini-North, along with photometry and high-contrast imaging from multiple ground-based facilities. In addition to TESS photometry, K2-419Ab was also observed and statistically validated as part of the K2 mission in Campaigns 5 and 18, which provides precise orbital and planetary constraints despite the faint host star and long orbital period of $\sim 20.4$ days. With an equilibrium temperature of only 380 K, K2-419Ab is one of the coolest known well-characterized transiting planets. TOI-6034 has a late F-type companion about 40\arcsec~away, making it the first GEMS host star to have an earlier main-sequence binary companion. These confirmations add to the existing small sample of confirmed transiting GEMS., Comment: Accepted in AJ

Indoor monocular depth estimation helps home automation, including robot navigation or AR/VR for surrounding perception. Most previous methods primarily experiment with the NYUv2 Dataset and concentrate on the overall performance in their evaluation. However, their robustness and generalization to diversely unseen types or categories for indoor spaces (spaces types) have yet to be discovered. Researchers may empirically find degraded performance in a released pretrained model on custom data or less-frequent types. This paper studies the common but easily overlooked factor-space type and realizes a model's performance variances across spaces. We present InSpaceType Dataset, a high-quality RGBD dataset for general indoor scenes, and benchmark 13 recent state-of-the-art methods on InSpaceType. Our examination shows that most of them suffer from performance imbalance between head and tailed types, and some top methods are even more severe. The work reveals and analyzes underlying bias in detail for transparency and robustness. We extend the analysis to a total of 4 datasets and discuss the best practice in synthetic data curation for training indoor monocular depth. Further, dataset ablation is conducted to find out the key factor in generalization. This work marks the first in-depth investigation of performance variances across space types and, more importantly, releases useful tools, including datasets and codes, to closely examine your pretrained depth models. Data and code: https://depthcomputation.github.io/DepthPublic/, Comment: BMVC 2024. This version supersedes 2309.13516

Neural network models for guitar amplifier emulation, while being effective, often demand high computational cost and lack interpretability. Drawing ideas from physical amplifier design, this paper aims to address these issues with a new differentiable digital signal processing (DDSP)-based model, called ``DDSP guitar amp,'' that models the four components of a guitar amp (i.e., preamp, tone stack, power amp, and output transformer) using specific DSP-inspired designs. With a set of time- and frequency-domain metrics, we demonstrate that DDSP guitar amp achieves performance comparable with that of black-box baselines while requiring less than 10\% of the computational operations per audio sample, thereby holding greater potential for usages in real-time applications., Comment: Preprint paper

In Multimodal Language Models (MLMs), the cost of manually annotating high-quality image-text pair data for fine-tuning and alignment is extremely high. While existing multimodal data augmentation frameworks propose ways to augment image-text pairs, they either suffer from semantic inconsistency between texts and images, or generate unrealistic images, causing knowledge gap with real world examples. To address these issues, we propose Attribute-based Multimodal Data Augmentation (ARMADA), a novel multimodal data augmentation method via knowledge-guided manipulation of visual attributes of the mentioned entities. Specifically, we extract entities and their visual attributes from the original text data, then search for alternative values for the visual attributes under the guidance of knowledge bases (KBs) and large language models (LLMs). We then utilize an image-editing model to edit the images with the extracted attributes. ARMADA is a novel multimodal data generation framework that: (i) extracts knowledge-grounded attributes from symbolic KBs for semantically consistent yet distinctive image-text pair generation, (ii) generates visually similar images of disparate categories using neighboring entities in the KB hierarchy, and (iii) uses the commonsense knowledge of LLMs to modulate auxiliary visual attributes such as backgrounds for more robust representation of original entities. Our empirical results over four downstream tasks demonstrate the efficacy of our framework to produce high-quality data and enhance the model performance. This also highlights the need to leverage external knowledge proxies for enhanced interpretability and real-world grounding.

Converting different modalities into generalized text, which then serves as input prompts for large language models (LLMs), is a common approach for aligning multimodal models, particularly when pairwise data is limited. Text-centric alignment method leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation, thereby enabling downstream models to effectively interpret various modal inputs. This study evaluates the quality and robustness of multimodal representations in the face of noise imperfections, dynamic input order permutations, and missing modalities, revealing that current text-centric alignment methods can compromise downstream robustness. To address this issue, we propose a new text-centric adversarial training approach that significantly enhances robustness compared to traditional robust training methods and pre-trained multimodal foundation models. Our findings underscore the potential of this approach to improve the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications., Comment: arXiv admin note: text overlap with arXiv:2407.05036

XR devices running chat-bots powered by Large Language Models (LLMs) have tremendous potential as always-on agents that can enable much better productivity scenarios. However, screen based chat-bots do not take advantage of the the full-suite of natural inputs available in XR, including inward facing sensor data, instead they over-rely on explicit voice or text prompts, sometimes paired with multi-modal data dropped as part of the query. We propose a solution that leverages an attention framework that derives context implicitly from user actions, eye-gaze, and contextual memory within the XR environment. This minimizes the need for engineered explicit prompts, fostering grounded and intuitive interactions that glean user insights for the chat-bot. Our user studies demonstrate the imminent feasibility and transformative potential of our approach to streamline user interaction in XR with chat-bots, while offering insights for the design of future XR-embodied LLM agents.

With the rapid development of large language models in recent years, there has been an increasing demand for domain-specific Agents that can cater to the unique needs of enterprises and organizations. Unlike general models, which strive for broad coverage, these specialized Agents rely on focused datasets tailored to their intended applications. This research proposes a pipeline that leverages the power of LLMs and the Retrieval-Augmented Generation related framework to construct high-quality instruction datasets for fine-tuning on specific domains using custom document collections. By ingesting domain-specific documents, the pipeline generates relevant and contextually appropriate instructions, thus effectively creating a comprehensive dataset for fine-tuning LLMs on the target domain. This approach overcomes the limitations of traditional dataset creation methods, which often rely on manual curation or web-scraping techniques that may introduce noise and irrelevant data. Notably, our pipeline offers a dynamic solution that can quickly adapt to updates or modifications in the domain-specific document collection, eliminating the need for complete retraining. Additionally, it addresses the challenge of data scarcity by enabling the generation of instruction datasets from a limited set of initial documents, rendering it suitable for unpopular or specialized domains where comprehensive datasets are scarce. As a case study, we apply this approach to the domain of psychiatry, a field requiring specialized knowledge and sensitive handling of patient information. The resulting fine-tuned LLM demonstrates showcases the viability of the proposed approach and underscores its potential for widespread adoption across various industries and domains where tailored, accurate, and contextually relevant language models are indispensable., Comment: 5 pages, SCA 2024: The 7th IEEE International Workshop on Smart Computing & Applications

We present a new analysis of Kepler-21, the brightest (V = 8.5) Kepler system with a known transiting exoplanet, Kepler-21 b. Kepler-21 b is a radius valley planet ($R = 1.6\pm 0.2 R_{\oplus}$) with an Earth-like composition (8.38$\pm$1.62 g/cc), though its mass and radius fall in the regime of possible "water worlds." We utilize new Keck/HIRES and WIYN/NEID radial velocity (RV) data in conjunction with Kepler and TESS photometry to perform a detailed study of activity mitigation between photometry and RVs. We additionally refine the system parameters, and we utilize Gaia astrometry to place constraints on a long-term RV trend. Our activity analysis affirms the quality of Kepler photometry for removing correlated noise from RVs, despite its temporal distance, though we reveal some cases where TESS may be superior. Using refined orbital parameters and updated composition curves, we rule out a ``water world" scenario for Kepler-21 b, and we identify a long period super-Jupiter planetary candidate, Kepler-21 (c).

TOI-6255~b (GJ 4256) is an Earth-sized planet (1.079$\pm0.065$ $R_\oplus$) with an orbital period of only 5.7 hours. With the newly commissioned Keck Planet Finder (KPF) and CARMENES spectrographs, we determined the planet's mass to be 1.44$\pm$0.14 $M_{\oplus}$. The planet is just outside the Roche limit, with $P_{\rm orb}/P_{\rm Roche}$ = 1.13 $\pm0.10$. The strong tidal force likely deforms the planet into a triaxial ellipsoid with a long axis that is $\sim$10\% longer than the short axis. Assuming a reduced stellar tidal quality factor $Q_\star^\prime \approx10^7$, we predict that tidal orbital decay will cause TOI-6255 to reach the Roche limit in roughly 400 Myr. Such tidal disruptions may produce the possible signatures of planet engulfment that have been on stars with anomalously high refractory elemental abundances compared to its conatal binary companion. TOI-6255 b is also a favorable target for searching for star-planet magnetic interactions, which might cause interior melting and hasten orbital decay. TOI-6255 b is a top target (Emission Spectroscopy Metric of about 24) for phase curve observations with the James Webb Space Telescope., Comment: 18 pages, 7 figures, 5 tables, accepted to AAS Journals. The first RV mass measurement from the Keck Planet Finder

The goal of future quantum networks is to enable new internet applications that are impossible to achieve using solely classical communication. Up to now, demonstrations of quantum network applications and functionalities on quantum processors have been performed in ad-hoc software that was specific to the experimental setup, programmed to perform one single task (the application experiment) directly into low-level control devices using expertise in experimental physics. Here, we report on the design and implementation of the first architecture capable of executing quantum network applications on quantum processors in platform-independent high-level software. We demonstrate the architecture's capability to execute applications in high-level software, by implementing it as a quantum network operating system -- QNodeOS -- and executing test programs including a delegated computation from a client to a server on two quantum network nodes based on nitrogen-vacancy (NV) centers in diamond. We show how our architecture allows us to maximize the use of quantum network hardware, by multitasking different applications on a quantum network for the first time. Our architecture can be used to execute programs on any quantum processor platform corresponding to our system model, which we illustrate by demonstrating an additional driver for QNodeOS for a trapped-ion quantum network node based on a single $^{40}\text{Ca}^+$ atom. Our architecture lays the groundwork for computer science research in the domain of quantum network programming, and paves the way for the development of software that can bring quantum network technology to society., Comment: 12 pages, 5 figures, supplementary materials (48 pages, 24 figures, 11 tables)

Removing audio effects from electric guitar recordings makes it easier for post-production and sound editing. An audio distortion recovery model not only improves the clarity of the guitar sounds but also opens up new opportunities for creative adjustments in mixing and mastering. While progress have been made in creating such models, previous efforts have largely focused on synthetic distortions that may be too simplistic to accurately capture the complexities seen in real-world recordings. In this paper, we tackle the task by using a dataset of guitar recordings rendered with commercial-grade audio effect VST plugins. Moreover, we introduce a novel two-stage methodology for audio distortion recovery. The idea is to firstly process the audio signal in the Mel-spectrogram domain in the first stage, and then use a neural vocoder to generate the pristine original guitar sound from the processed Mel-spectrogram in the second stage. We report a set of experiments demonstrating the effectiveness of our approach over existing methods, through both subjective and objective evaluation metrics., Comment: DAFx 2024

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results., Comment: SIGGRAPH 2024 (ACM Transactions on Graphics (TOG))

Recent advancements in Large Language Models (LLMs) and Prompt Engineering have made chatbot customization more accessible, significantly reducing barriers to tasks that previously required programming skills. However, prompt evaluation, especially at the dataset scale, remains complex due to the need to assess prompts across thousands of test instances within a dataset. Our study, based on a comprehensive literature review and pilot study, summarized five critical challenges in prompt evaluation. In response, we introduce a feature-oriented workflow for systematic prompt evaluation. In the context of text summarization, our workflow advocates evaluation with summary characteristics (feature metrics) such as complexity, formality, or naturalness, instead of using traditional quality metrics like ROUGE. This design choice enables a more user-friendly evaluation of prompts, as it guides users in sorting through the ambiguity inherent in natural language. To support this workflow, we introduce Awesum, a visual analytics system that facilitates identifying optimal prompt refinements for text summarization through interactive visualizations, featuring a novel Prompt Comparator design that employs a BubbleSet-inspired design enhanced by dimensionality reduction techniques. We evaluate the effectiveness and general applicability of the system with practitioners from various domains and found that (1) our design helps overcome the learning curve for non-technical people to conduct a systematic evaluation of summarization prompts, and (2) our feature-oriented workflow has the potential to generalize to other NLG and image-generation tasks. For future works, we advocate moving towards feature-oriented evaluation of LLM prompts and discuss unsolved challenges in terms of human-agent interaction.

A flexible multimode fiber is an exceptionally efficient tool for in vivo deep tissue imaging. Recent advances in compressive multimode fiber sensing allow for imaging with sub-diffraction spatial resolution and sub-Nyquist speed. At present, the technology is limited to imaging in a two-dimensional (2D) plane near the fiber distal facet while in real applications it is very important to visualize three-dimensional (3D) structures. Here we propose a new approach for fast sub-diffraction 3D imaging through a multimode fiber by using a single 2D scan, speckle illumination, and bucket detection. We experimentally demonstrate precise image plane location as well as 3D imaging of samples with various scattering coefficients. The full width at half maximum (FWHM) of the point spread function along the axial direction is 3 times smaller than the diffraction limit. Our study grants depth-resolving capacity to ultra-thin super-resolution fiber endoscopes for life science and medical applications., Comment: 8 pages,4 figures

Single-stranded RNA viruses co-assemble their capsid with the genome and variations in capsid structures can have significant functional relevance. In particular, viruses need to respond to a dehydrating environment to prevent genomic degradation and remain active upon rehydration. Theoretical work has predicted low-energy buckling transitions in icosahedral capsids which could protect the virus from further dehydration. However, there has been no direct experimental evidence, nor molecular mechanism, for such behaviour. Here we observe this transition using X-ray single particle imaging of MS2 bacteriophages after aerosolization. Using a combination of machine learning tools, we classify hundreds of thousands of single particle diffraction patterns to learn the structural landscape of the capsid morphology as a function of time spent in the aerosol phase. We found a previously unreported compact conformation as well as intermediate structures which suggest an incoherent buckling transition which does not preserve icosahedral symmetry. Finally, we propose a mechanism of this buckling, where a single 19-residue loop is destabilised, leading to the large observed morphology change. Our results provide experimental evidence for a mechanism by which viral capsids protect themselves from dehydration. In the process, these findings also demonstrate the power of single particle X-ray imaging and machine learning methods in studying biomolecular structural dynamics., Comment: 10 pages, 4 figures plus 9 pages supplementary information

Replicating analog device circuits through neural audio effect modeling has garnered increasing interest in recent years. Existing work has predominantly focused on a one-to-one emulation strategy, modeling specific devices individually. In this paper, we tackle the less-explored scenario of one-to-many emulation, utilizing conditioning mechanisms to emulate multiple guitar amplifiers through a single neural model. For condition representation, we use contrastive learning to build a tone embedding encoder that extracts style-related features of various amplifiers, leveraging a dataset of comprehensive amplifier settings. Targeting zero-shot application scenarios, we also examine various strategies for tone embedding representation, evaluating referenced tone embedding against two retrieval-based embedding methods for amplifiers unseen in the training time. Our findings showcase the efficacy and potential of the proposed methods in achieving versatile one-to-many amplifier modeling, contributing a foundational step towards zero-shot audio modeling applications., Comment: ISMIR 2024

Converting different modalities into general text, serving as input prompts for large language models (LLMs), is a common method to align multimodal models when there is limited pairwise data. This text-centric approach leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation. This enables downstream models to effectively interpret various modal inputs. This study assesses the quality and robustness of multimodal representations in the presence of missing entries, noise, or absent modalities, revealing that current text-centric alignment methods compromise downstream robustness. To address this issue, we propose a new text-centric approach that achieves superior robustness compared to previous methods across various modalities in different settings. Our findings highlight the potential of this approach to enhance the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.

We present a comprehensive analysis of young stellar object (YSO) variability within the M17 Southwest Extension (M17 SWex), using 3.5 years of monitoring data from the JCMT Transient Survey at sub-millimeter (sub-mm) and 9 years from the NEOWISE mission at mid-infrared (mid-IR). Our study encompasses observations of 147 bright sub-mm peaks identified within our deep JCMT co-added map as well as 156 YSOs in NEOWISE W1 and 179 in W2 that were previously identified in Spitzer surveys. We find three robust sub-mm variables: two are candidate YSOs and one is a likely extragalactic source. At mid-IR wavelengths, our analysis reveals secular and stochastic variability in 47 YSOs, with the highest fraction of secular variability occurring at the earliest evolutionary stage. This is similar to what has previously been observed for low-mass YSO variability within the Gould Belt. However, we observe less overall variability in M17SWex at both the sub-mm and mid-IR. We suspect that this lower fraction is due to the greater distance to M17 SWex. Our findings showcase the utility of multi-wavelength observations to better capture the complex variability phenomena inherent to star formation processes and demonstrate the importance of years-long monitoring of a diverse selection of star-forming environments., Comment: Accepted for Publication in The Astronomical Journal

Expressive music synthesis (EMS) for violin performance is a challenging task due to the disagreement among music performers in the interpretation of expressive musical terms (EMTs), scarcity of labeled recordings, and limited generalization ability of the synthesis model. These challenges create trade-offs between model effectiveness, diversity of generated results, and controllability of the synthesis system, making it essential to conduct a comparative study on EMS model design. This paper explores two violin EMS approaches. The end-to-end approach is a modification of a state-of-the-art text-to-speech generator. The parameter-controlled approach is based on a simple parameter sampling process that can render note lengths and other parameters compatible with MIDI-DDSP. We study these two approaches (in total, three model variants) through objective and subjective experiments and discuss several key issues of EMS based on the results., Comment: 15 pages, 2 figures, 3 tables

Visual programs are executable code generated by large language models to address visual reasoning problems. They decompose complex questions into multiple reasoning steps and invoke specialized models for each step to solve the problems. However, these programs are prone to logic errors, with our preliminary evaluation showing that 58% of the total errors are caused by program logic errors. Debugging complex visual programs remains a major bottleneck for visual reasoning. To address this, we introduce VDebugger, a novel critic-refiner framework trained to localize and debug visual programs by tracking execution step by step. VDebugger identifies and corrects program errors leveraging detailed execution feedback, improving interpretability and accuracy. The training data is generated through an automated pipeline that injects errors into correct visual programs using a novel mask-best decoding technique. Evaluations on six datasets demonstrate VDebugger's effectiveness, showing performance improvements of up to 3.2% in downstream task accuracy. Further studies show VDebugger's ability to generalize to unseen tasks, bringing a notable improvement of 2.3% on the unseen COVR task. Code, data and models are made publicly available at https://github.com/shirley-wu/vdebugger/, Comment: EMNLP 2024 Findings

The integration of pre-trained text-based large language models (LLM) with speech input has enabled instruction-following capabilities for diverse speech tasks. This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks. We propose the use of discrete speech units (DSU), rather than continuous-valued speech encoder outputs, that are converted to the LLM token embedding space using the speech adapter. We generate DSU using a self-supervised speech encoder followed by k-means clustering. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. We also explore various types of DSU extracted from different layers of the self-supervised speech encoder, as well as Mel frequency Cepstral Coefficients (MFCC). Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.

We develop a lattice model which exhibits topological transitions from $Z_2$ topological insulators to mirror symmetry-protected topological crystalline insulators by introducing additional spin-orbit coupling terms. The topological phase is characterized by the mirror winding number, defined within the mirror symmetry invariant subspace, which ensures the protection of gapless edge states and zero-energy corner states under specific boundary conditions. Additionally, we propose a feasible scheme using ultracold atoms confined in a stacked hexagonal optical lattice with Raman fields to realize the two-dimensional topological crystalline insulators. Detection of the mirror winding number in these systems can be achieved by implementing a simple quench sequence and observing the evolution of the time-of-flight patterns., Comment: 11 pages, 5 figures

Recent progresses in Josephson-junction-based superconducting circuits have propelled quantum information processing forward. However, the lack of a metastable state in most superconducting artificial atoms hinders the development of photonic quantum memory in this platform. Here, we use a single superconducting qubit-resonator system to realize a desired $\Lambda$-type artificial atom, and to demonstrate slow light with a group velocity of 3.6 km/s and the microwave storage with a memory time extending to several hundred nanoseconds via electromagnetically induced transparency. Our results highlight the potential of achieving microwave quantum memory, promising substantial advancements in quantum information processing within superconducting circuits., Comment: 10 pages, 6 figures

Soft grippers, with their inherent compliance and adaptability, show advantages for delicate and versatile manipulation tasks in robotics. This paper presents a novel approach to underactuated control of multiple soft actuators, explicitly focusing on the coordination of soft fingers within a soft gripper. Utilizing a single syringe pump as the actuation mechanism, we address the challenge of coordinating multiple degrees of freedom of a compliant system. The theoretical framework applies concepts from stable inversion theory, adapting them to the unique dynamics of the underactuated soft gripper. Through meticulous mechatronic system design and controller synthesis, we demonstrate the efficacy and applicability of our approach in achieving precise and coordinated manipulation tasks in simulation and experimentation. Our findings not only contribute to the advancement of soft robot control but also offer practical insights into the design and control of underactuated systems for real-world applications., Comment: 11 pages, 7 figures

Interlayer rotation in van der Waals (vdW) materials offers great potential for manipulating phonon dynamics and heat flow in advanced electronics with ever higher compactness and power density. However, despite extensive theoretical efforts in recent years, experimental measurements remain scarce especially due to the critical challenges of preparing single-crystalline twisted interfaces and probing interfacial thermal transport with sufficient resolution. Here, we exploited the intrinsic twisted interfaces in highly oriented pyrolytic graphite (HOPG). By developing novel experimental schemes based on microfabricated mesas, we managed to achieve simultaneous mechanical characterizations and thermal measurements. In particular, we pushed the HOPG mesas with a microprobe to identify and rotate single-crystalline intrinsic interfaces owing to their slippery nature as is well known in structural superlubricity. Remarkably, we observed over 30-fold suppression of thermal conductance for the slippery interfaces by using epitaxial graphite as a control. Nonetheless, the interfacial conductance remains around 600 $\mathrm{MWm^{-2}K^{-1}}$ which surpasses the highest values for artificially stacked vdW structures by more than five times. Further, atomic simulations revealed the predominant role of the transverse acoustic phonons. Together, our findings highlight a general physical picture that directly correlates interfacial thermal transport with sliding resistance, and lay the foundation for twist-enabled thermal management which are particularly beneficial to twistronics and slidetronics.

There exist three approaches for multilingual and crosslingual automatic speech recognition (MCL-ASR) - supervised pre-training with phonetic or graphemic transcription, and self-supervised pre-training. We find that pre-training with phonetic supervision has been underappreciated so far for MCL-ASR, while conceptually it is more advantageous for information sharing between different languages. This paper explores the approach of pre-training with weakly phonetic supervision towards data-efficient MCL-ASR, which is called Whistle. We relax the requirement of gold-standard human-validated phonetic transcripts, and obtain International Phonetic Alphabet (IPA) based transcription by leveraging the LanguageNet grapheme-to-phoneme (G2P) models. We construct a common experimental setup based on the CommonVoice dataset, called CV-Lang10, with 10 seen languages and 2 unseen languages. A set of experiments are conducted on CV-Lang10 to compare, as fair as possible, the three approaches under the common setup for MCL-ASR. Experiments demonstrate the advantages of phoneme-based models (Whistle) for MCL-ASR, in terms of speech recognition for seen languages, crosslingual performance for unseen languages with different amounts of few-shot data, overcoming catastrophic forgetting, and training efficiency.It is found that when training data is more limited, phoneme supervision can achieve better results compared to subword supervision and self-supervision, thereby providing higher data-efficiency. To support reproducibility and promote future research along this direction, we will release the code, models and data for the whole pipeline of Whistle at https://github.com/thu-spmi/CAT upon publication.