Search

Transforming two-dimensional (2D) images into three-dimensional (3D) volumes is a well-known yet challenging problem for the computer vision community. In the medical domain, a few previous studies attempted to convert two or more input radiographs into computed tomography (CT) volumes. Following their effort, we introduce a diffusion model-based technology that can rotate the anatomical content of any input radiograph in 3D space, potentially enabling the visualization of the entire anatomical content of the radiograph from any viewpoint in 3D. Similar to previous studies, we used CT volumes to create Digitally Reconstructed Radiographs (DRRs) as the training data for our model. However, we addressed two significant limitations encountered in previous studies: 1. We utilized conditional diffusion models with classifier-free guidance instead of Generative Adversarial Networks (GANs) to achieve higher mode coverage and improved output image quality, with the only trade-off being slower inference time, which is often less critical in medical applications; and 2. We demonstrated that the unreliable output of style transfer deep learning (DL) models, such as Cycle-GAN, to transfer the style of actual radiographs to DRRs could be replaced with a simple yet effective training transformation that randomly changes the pixel intensity histograms of the input and ground-truth imaging data during training. This transformation makes the diffusion model agnostic to any distribution variations of the input data pixel intensity, enabling the reliable training of a DL model on input DRRs and applying the exact same model to conventional radiographs (or DRRs) during inference., Comment: Website: https://pouriarouzrokh.github.io/RadRotator Online demo: https://huggingface.co/spaces/Pouriarouzrokh/RadRotator Article information: 16 pages, 11 figures

Physically assistive robots present an opportunity to significantly increase the well-being and independence of individuals with motor impairments or other forms of disability who are unable to complete activities of daily living. Speech interfaces, especially ones that utilize Large Language Models (LLMs), can enable individuals to effectively and naturally communicate high-level commands and nuanced preferences to robots. Frameworks for integrating LLMs as interfaces to robots for high level task planning and code generation have been proposed, but fail to incorporate human-centric considerations which are essential while developing assistive interfaces. In this work, we present a framework for incorporating LLMs as speech interfaces for physically assistive robots, constructed iteratively with 3 stages of testing involving a feeding robot, culminating in an evaluation with 11 older adults at an independent living facility. We use both quantitative and qualitative data from the final study to validate our framework and additionally provide design guidelines for using LLMs as speech interfaces for assistive robots. Videos and supporting files are located on our project website: https://sites.google.com/andrew.cmu.edu/voicepilot/

Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. This mitigates hallucinations and allows for the updating of knowledge without retraining the LLM. However, RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. Also, if there is contradictory content, the RAG response will likely reflect only one of the two possible responses. Therefore, quantifying uncertainty in the retrieval process is crucial for ensuring RAG trustworthiness. In this report, we introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks. First, a calibration set of questions answerable from the knowledge base is constructed. Each question's embedding is compared against document embeddings to identify the most relevant document chunks containing the answer and record their similarity scores. Given a user-specified error rate ({\alpha}), these similarity scores are then analyzed to determine a similarity score cutoff threshold. During inference, all chunks with similarity exceeding this threshold are retrieved to provide context to the LLM, ensuring the true answer is captured in the context with a (1-{\alpha}) confidence level. We provide a Python package that enables users to implement the entire workflow proposed in our work, only using LLMs and without human intervention., Comment: Github code: https://github.com/Mayo-Radiology-Informatics-Lab/conflare

Accurately predicting the 3D human posture and the pressure exerted on the body for people resting in bed, visualized as a body mesh (3D pose & shape) with a 3D pressure map, holds significant promise for healthcare applications, particularly, in the prevention of pressure ulcers. Current methods focus on singular facets of the problem -- predicting only 2D/3D poses, generating 2D pressure images, predicting pressure only for certain body regions instead of the full body, or forming indirect approximations to the 3D pressure map. In contrast, we introduce BodyMAP, which jointly predicts the human body mesh and 3D applied pressure map across the entire human body. Our network leverages multiple visual modalities, incorporating both a depth image of a person in bed and its corresponding 2D pressure image acquired from a pressure-sensing mattress. The 3D pressure map is represented as a pressure value at each mesh vertex and thus allows for precise localization of high-pressure regions on the body. Additionally, we present BodyMAP-WS, a new formulation of pressure prediction in which we implicitly learn pressure in 3D by aligning sensed 2D pressure images with a differentiable 2D projection of the predicted 3D pressure maps. In evaluations with real-world human data, our method outperforms the current state-of-the-art technique by 25% on both body mesh and 3D applied pressure map prediction tasks for people in bed., Comment: Accepted at CVPR 2024 Project Website: https://bodymap3d.github.io/ Code: https://github.com/RCHI-Lab/BodyMAP

We present results of a dual eclipse expedition to observe the solar corona from two sites during the annular solar eclipse of 2023 October 14, using a novel coronagraph designed to be accessible for amateurs and students to build and deploy. The coronagraph "CATEcor" builds on the standardized eclipse observing equipment developed for the Citizen CATE 2024 experiment. The observing sites were selected for likelihood of clear observations, for historic relevance (near the Climax site in the Colorado Rocky Mountains), and for centrality to the annular eclipse path (atop Sandia Peak above Albuquerque, New Mexico). The novel portion of CATEcor is an external occulter assembly that slips over the front of a conventional dioptric telescope, forming a "shaded-truss" externally occulted coronagraph. CATEcor is specifically designed to be easily constructed in a garage or "makerspace" environment. We successfully observed some bright features in the solar corona to an altitude of approximately 2.25 R$_\odot$ during the annular phases of the eclipse. Future improvements to the design, in progress now, will reduce both stray light and image artifacts; our objective is to develop a design that can be operated successfully by amateur astronomers at sufficient altitude even without the darkened skies of a partial or annular eclipse., Comment: Accepted by Solar Physics

We give a categorical formulation of the $p$-adic local Langlands correspondence for $\mathrm{GL}_2(\mathbb{Q}_p)$,as an embedding of the derived category of locally admissible representations into the category of Ind-coherent sheaves on the moduli stack of two-dimensional representations of $\mathrm{Gal}(\overline{\mathbb{Q}}_p/\mathbb{Q}_p)$. Moreover, we relate our version of the $p$-adic local Langlands correspondence for $\mathrm{GL}_2(\mathbb{Q}_p)$ to the cohomology of modular curves through a local-global compatibility formula., Comment: 66 pages

We present AdaFold, a model-based feedback-loop framework for optimizing folding trajectories. AdaFold extracts a particle-based representation of cloth from RGB-D images and feeds back the representation to a model predictive control to replan folding trajectory at every time step. A key component of AdaFold that enables feedback-loop manipulation is the use of semantic descriptors extracted from geometric features. These descriptors enhance the particle representation of the cloth to distinguish between ambiguous point clouds of differently folded cloths. Our experiments demonstrate AdaFold's ability to adapt folding trajectories of cloths with varying physical properties and generalize from simulated training to real-world execution., Comment: 8 pages, 6 figures, 5 tables

We analyze how the presence of closed timelike curves (CTCs) characterizing a time machine can be discerned by placing a local particle detector in a region of spacetime which is causally disconnected from the CTCs. Our study shows that not only can the detector tell if there are CTCs, but also that the detector can separate topological from geometrical information and distinguish periodic spacetimes without CTCs (like the Einstein cylinder), curvature, and spacetimes with topological identifications that enable time-machines., Comment: 20 pages, 3 figures; v2: fixed typos and updated to match published version

Although Digital Subtraction Angiography (DSA) is the most important imaging for visualizing cerebrovascular anatomy, its interpretation by clinicians remains difficult. This is particularly true when treating arteriovenous malformations (AVMs), where entangled vasculature connecting arteries and veins needs to be carefully identified.The presented method aims to enhance DSA image series by highlighting critical information via automatic classification of vessels using a combination of two learning models: An unsupervised machine learning method based on Independent Component Analysis that decomposes the phases of flow and a convolutional neural network that automatically delineates the vessels in image space. The proposed method was tested on clinical DSA images series and demonstrated efficient differentiation between arteries and veins that provides a viable solution to enhance visualizations for clinical use., Comment: Paper accepted for publication at SPIE Medical Imaging 2024

We investigate galaxy sizes at redshift $z\gtrsim 6$ with the cosmological radiation-magneto-hydrodynamic simulation suite THESAN(-HR). These simulations simultaneously capture the reionization of the large-scale intergalactic medium and resolved galaxy properties. The intrinsic size ($r^{\ast}_{1/2}$) of simulated galaxies increases moderately with stellar mass at $M_{\ast} \lesssim 10^{8}\,{\rm M}_{\odot}$ and decreases fast at larger masses, resulting in a hump feature at $M_{\ast}\sim 10^{8}\,{\rm M}_{\odot}$ that is insensitive to redshift. Low-mass galaxies are in the initial phase of size growth and are better described by a spherical shell model with feedback-driven gas outflows competing with the cold inflows. In contrast, massive galaxies fit better with the disk formation model. They generally experience a phase of rapid compaction and gas depletion, likely driven by internal disk instability rather than external processes. We identify four compact quenched galaxies in the $(95.5\,{\rm cMpc})^{3}$ volume of THESAN-1 at $z\simeq 6$, and their quenching follows reaching a characteristic stellar surface density akin to the massive compact galaxies at cosmic noon. Compared to observations, we find that the median UV effective radius ($R^{\rm UV}_{\rm eff}$) of simulated galaxies is at least three times larger than the observed ones at $M_{\ast}\lesssim 10^{9}\,{\rm M}_{\odot}$ or $M_{\rm UV}\gtrsim -20$ at $6 \lesssim z \lesssim 10$. This inconsistency, related to the hump feature of the intrinsic size--mass relation, persists across many other cosmological simulations with different galaxy formation models and demonstrates the potential of using galaxy morphology to constrain the physics of galaxy formation at high redshifts., Comment: 21 pages, 17 figures. Version accepted by MNRAS

In this work we make the connection between the Unruh-DeWitt particle detector model applied to quantum field theory in curved spacetimes and the rigorous construction of the spin-boson model. With some modifications, we show that existing results about the existence of a spin-boson ground state can be adapted to the Unruh-DeWitt model. In the most relevant scenario involving massless scalar fields in (3+1)-dimensional globally hyperbolic spacetimes, where the Unruh-DeWitt model describes a simplified model of light-matter interaction, we argue that common choices of the spacetime smearing functions regulate the ultraviolet behaviour of the model but can still exhibit infrared divergences. In particular, this implies the well-known expectation that the joint interacting Hilbert space of the model cannot be described by the tensor product of a two-dimensional complex Hilbert space and the Fock space of the vacuum representation. We discuss the conditions under which this problem does not arise and the relevance of the operator-algebraic approach for better understanding of particle detector models and their applications. Our work clarifies the connection between obstructions due to Haag's theorem and infrared bosons in the spin-boson models, and paves the way for more rigorous study of entanglement and communication in the UDW framework involving multiple detectors., Comment: 47+7 pages, single-column; 1 figure; v2: fixed typos and updated to match published version

This paper introduces DiffTORI, which utilizes Differentiable Trajectory Optimization as the policy representation to generate actions for deep Reinforcement and Imitation learning. Trajectory optimization is a powerful and widely used algorithm in control, parameterized by a cost and a dynamics function. The key to our approach is to leverage the recent progress in differentiable trajectory optimization, which enables computing the gradients of the loss with respect to the parameters of trajectory optimization. As a result, the cost and dynamics functions of trajectory optimization can be learned end-to-end. DiffTORI addresses the ``objective mismatch'' issue of prior model-based RL algorithms, as the dynamics model in DiffTORI is learned to directly maximize task performance by differentiating the policy gradient loss through the trajectory optimization process. We further benchmark DiffTORI for imitation learning on standard robotic manipulation task suites with high-dimensional sensory observations and compare our method to feed-forward policy classes as well as Energy-Based Models (EBM) and Diffusion. Across 15 model-based RL tasks and 35 imitation learning tasks with high-dimensional image and point cloud inputs, DiffTORI outperforms prior state-of-the-art methods in both domains., Comment: NeurIPS 2024 (Spotlight)

Reward engineering has long been a challenge in Reinforcement Learning (RL) research, as it often requires extensive human effort and iterative processes of trial-and-error to design effective reward functions. In this paper, we propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks, using only a text description of the task goal and the agent's visual observations, by leveraging feedbacks from vision language foundation models (VLMs). The key to our approach is to query these models to give preferences over pairs of the agent's image observations based on the text description of the task goal, and then learn a reward function from the preference labels, rather than directly prompting these models to output a raw reward score, which can be noisy and inconsistent. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains - including classic control, as well as manipulation of rigid, articulated, and deformable objects - without the need for human supervision, outperforming prior methods that use large pretrained models for reward generation under the same assumptions. Videos can be found on our project website: https://rlvlmf2024.github.io/, Comment: ICML 2024

We reconsider an old problem, namely the dimension of the $G$-invariant subspace in $V^{\otimes p} \otimes V^{*\otimes q}$, where $G$ is one of the classical groups ${\rm GL}(V)$, ${\rm SL}(V)$, ${\rm O}(V)$, ${\rm SO}(V)$, or ${\rm Sp}(V)$. Spanning sets for the invariant subspace have long been well known, but linear bases are more delicate. Beginning in the broader setting of polynomial invariants, we write down multigraded linear bases which we realize as certain arc diagrams (which may include hyperedges); then the bases for tensor invariants are obtained by restricting our attention to the diagrams that are 1-regular. We survey the many equivalent ways -- some old, some new -- to enumerate these diagrams., Comment: 16 pages + appendix