Search

Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.

Chalcogenide optical phase change materials (PCMs) have garnered significant interest for their growing applications in programmable photonics, optical analog computing, active metasurfaces, and beyond. Limited endurance or cycling lifetime is however increasingly becoming a bottleneck toward their practical deployment for these applications. To address this issue, we performed a systematic study elucidating the cycling failure mechanisms of Ge$_2$Sb$_2$Se$_4$Te (GSST), a common optical PCM tailored for infrared photonic applications, in an electrothermal switching configuration commensurate with their applications in on-chip photonic devices. We further propose a set of design rules building on insights into the failure mechanisms, and successfully implemented them to boost the endurance of the GSST device to over 67,000 cycles.

We investigate the potential of photonic lantern (PL) fiber fed spectrometers for two-dimensional spectroastrometry. Spectroastrometry, a technique for studying small angular scales by measuring centroid shifts as a function of wavelength, is typically conducted using long-slit spectrographs. However, slit-based spectroastrometry requires observations with multiple position angles to measure two-dimensional spectroastrometric signals. In a typical configuration of PL-fed spectrometers, light from the focal plane is coupled into the few-moded PL, which is then split into several single-mode outputs, with the relative intensities containing astrometric information. The single-moded beams can be fed into a high-resolution spectrometer to measure wavelength-dependent centroid shifts. We perform numerical simulations of a standard 6-port PL and demonstrate its capability of measuring spectroastrometric signals. The effects of photon noise, wavefront errors, and chromaticity are investigated. When the PL is designed to have large linear responses to tip-tilts at the wavelengths of interest, the centroid shifts can be efficiently measured. Furthermore, we provide mock observations of detecting accreting protoplanets. PL spectroastrometry is potentially a simple and efficient technique for detecting spectroastrometric signals., Comment: Accepted for publication in the Journal of Astronomical Telescopes, Instruments, and Systems (JATIS)

Photonic lanterns are waveguide devices enabling high throughput single mode spectroscopy and high angular resolution. We aim to present the first on-sky demonstration of a photonic lantern (PL) operating in visible light, to measure its throughput and assess its potential for high-resolution spectroscopy of compact objects. We used the SCExAO instrument (a double stage extreme AO system installed at the Subaru telescope) and FIRST mid-resolution spectrograph (R 3000) to test the visible capabilities of the PL on internal source and on-sky observations. The best averaged coupling efficiency over the PL field of view was measured at 51% +/- 10% with a peak at 80%. We also investigate the relationship between coupling efficiency and the Strehl ratio for a PL, comparing them with those of a single-mode fiber (SMF). Findings show that in the AO regime, a PL offers better coupling efficiency performance than a SMF, especially in the presence of low spatial frequency aberrations. We observed Ikiiki (alpha Leo - mR = 1.37) and `Aua (alpha Ori - mR = -1.17) at a frame rate of 200 Hz. Under median seeing conditions (about 1 arcsec measured in H band) and large tip/tilt residuals (over 20 mas), we estimated an average light coupling efficiency of 14.5% +/- 7.4%, with a maximum of 42.8% at 680 nm. We were able to reconstruct both star's spectra, containing various absorption lines. The successful demonstration of this device opens new possibilities in terms of high throughput single-mode fiber-fed spectroscopy in the Visible. The demonstrated on-sky coupling efficiency performance would not have been achievable with a single SMF injection setup under similar conditions, partly because the residual tip/tilt alone exceeded the field of view of a visible SMF (18 mas at 700 nm). Thus emphasizing the enhanced resilience of PL technology to such atmospheric disturbances. The additional, Comment: Accepted in Astronomy and Astrophysics journal on 9/11/2024

The Janis-Newman-Winicour (JNW) spacetime possesses a naked singularity, although it represents an exact particle-like solution to the Einstein-Klein-Gordon theory with a massless scalar field. Here, we investigate the possible formation of a naked singularity in the JNW spacetime, using the thin-shell approximation to describe the gravitational collapse. By introducing different matter contents to construct thin-shells, we demonstrate the impossibility of naked singularity formation during the gravitational collapse unless the causality or null energy condition of the thin-shell is violated. Therefore, the weak cosmic censorship is satisfied even with the naked singularity of the JNW spacetime., Comment: 13 pages, 6 figures

An electrocardiogram (ECG) captures the heart's electrical signal to assess various heart conditions. In practice, ECG data is stored as either digitized signals or printed images. Despite the emergence of numerous deep learning models for digitized signals, many hospitals prefer image storage due to cost considerations. Recognizing the unavailability of raw ECG signals in many clinical settings, we propose VizECGNet, which uses only printed ECG graphics to determine the prognosis of multiple cardiovascular diseases. During training, cross-modal attention modules (CMAM) are used to integrate information from two modalities - image and signal, while self-modality attention modules (SMAM) capture inherent long-range dependencies in ECG data of each modality. Additionally, we utilize knowledge distillation to improve the similarity between two distinct predictions from each modality stream. This innovative multi-modal deep learning architecture enables the utilization of only ECG images during inference. VizECGNet with image input achieves higher performance in precision, recall, and F1-Score compared to signal-based ECG classification models, with improvements of 3.50%, 8.21%, and 7.38%, respectively., Comment: Accepted in International Conference on Image Processing (ICIP) 2024

A Photonic Lantern (PL) is a novel device that efficiently converts a multi-mode fiber into several single-mode fibers. When coupled with an extreme adaptive optics (ExAO) system and a spectrograph, PLs enable high throughput spectroscopy at high angular resolution. The Subaru Coronagraphic Extreme Adaptive Optics (SCExAO) system of the Subaru Telescope recently acquired a PL that converts its multi-mode input into 19 single-mode outputs. The single mode outputs feed a R~4,000 spectrograph optimized for the 600 to 760 nm wavelength range. We present here the integration of the PL on SCExAO, and study the device performance in terms of throughput, field of view, and spectral reconstruction. We also present the first on-sky demonstration of a Visible PL coupled with an ExAO system, showing a significant improvement of x12 in throughput compared to the use of a sole single-mode fiber. This work paves the way towards future high throughput photonics instrumentation at small angular resolution., Comment: Proceeding published in SPIE Astronomical Telescopes + Instrumentation (2024) - Paper number 13096-25

Pose-driven human-image animation diffusion models have shown remarkable capabilities in realistic human video synthesis. Despite the promising results achieved by previous approaches, challenges persist in achieving temporally consistent animation and ensuring robustness with off-the-shelf pose detectors. In this paper, we present TCAN, a pose-driven human image animation method that is robust to erroneous poses and consistent over time. In contrast to previous methods, we utilize the pre-trained ControlNet without fine-tuning to leverage its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the network to align the latent space between the pose and appearance features. Additionally, by introducing an additional temporal layer to the ControlNet, we enhance robustness against outliers of the pose detector. Through the analysis of attention maps over the temporal axis, we also designed a novel temperature map leveraging pose information, allowing for a more static background. Extensive experiments demonstrate that the proposed method can achieve promising results in video synthesis tasks encompassing various poses, like chibi. Project Page: https://eccv2024tcan.github.io/, Comment: The first two authors contributed equally

Neural rendering-based urban scene reconstruction methods commonly rely on images collected from driving vehicles with cameras facing and moving forward. Although these methods can successfully synthesize from views similar to training camera trajectory, directing the novel view outside the training camera distribution does not guarantee on-par performance. In this paper, we tackle the Extrapolated View Synthesis (EVS) problem by evaluating the reconstructions on views such as looking left, right or downwards with respect to training camera distributions. To improve rendering quality for EVS, we initialize our model by constructing dense LiDAR map, and propose to leverage prior scene knowledge such as surface normal estimator and large-scale diffusion model. Qualitative and quantitative comparisons demonstrate the effectiveness of our methods on EVS. To the best of our knowledge, we are the first to address the EVS problem in urban scene reconstruction. Link to our project page: https://vegs3d.github.io/., Comment: The first two authors contributed equally. Project Page: https://vegs3d.github.io/

In the evolving landscape of computer vision (CV) technologies, the automatic detection and interpretation of gender and emotion in images is a critical area of study. This paper investigates social biases in CV models, emphasizing the limitations of traditional evaluation metrics such as precision, recall, and accuracy. These metrics often fall short in capturing the complexities of gender and emotion, which are fluid and culturally nuanced constructs. Our study proposes a sociotechnical framework for evaluating CV models, incorporating both technical performance measures and considerations of social fairness. Using a dataset of 5,570 images related to vaccination and climate change, we empirically compared the performance of various CV models, including traditional models like DeepFace and FER, and generative models like GPT-4 Vision. Our analysis involved manually validating the gender and emotional expressions in a subset of images to serve as benchmarks. Our findings reveal that while GPT-4 Vision outperforms other models in technical accuracy for gender classification, it exhibits discriminatory biases, particularly in response to transgender and non-binary personas. Furthermore, the model's emotion detection skew heavily towards positive emotions, with a notable bias towards associating female images with happiness, especially when prompted by male personas. These findings underscore the necessity of developing more comprehensive evaluation criteria that address both validity and discriminatory biases in CV models. Our proposed framework provides guidelines for researchers to critically assess CV tools, ensuring their application in communication research is both ethical and effective. The significant contribution of this study lies in its emphasis on a sociotechnical approach, advocating for CV technologies that support social good and mitigate biases rather than perpetuate them.

This paper presents a novel manipulation strategy that uses keypoint correspondences extracted from visuo-tactile sensor images to facilitate precise object manipulation. Our approach uses the visuo-tactile feedback to guide the robot's actions for accurate object grasping and placement, eliminating the need for post-grasp adjustments and extensive training. This method provides an improvement in deployment efficiency, addressing the challenges of manipulation tasks in environments where object locations are not predefined. We validate the effectiveness of our strategy through experiments demonstrating the extraction of keypoint correspondences and their application to real-world tasks such as block alignment and gear insertion, which require millimeter-level precision. The results show an average error margin significantly lower than that of traditional vision-based methods, which is sufficient to achieve the target tasks.

Generalizability in deep neural networks plays a pivotal role in medical image segmentation. However, deep learning-based medical image analyses tend to overlook the importance of frequency variance, which is critical element for achieving a model that is both modality-agnostic and domain-generalizable. Additionally, various models fail to account for the potential information loss that can arise from multi-task learning under deep supervision, a factor that can impair the model representation ability. To address these challenges, we propose a Modality-agnostic Domain Generalizable Network (MADGNet) for medical image segmentation, which comprises two key components: a Multi-Frequency in Multi-Scale Attention (MFMSA) block and Ensemble Sub-Decoding Module (E-SDM). The MFMSA block refines the process of spatial feature extraction, particularly in capturing boundary features, by incorporating multi-frequency and multi-scale features, thereby offering informative cues for tissue outline and anatomical structures. Moreover, we propose E-SDM to mitigate information loss in multi-task learning with deep supervision, especially during substantial upsampling from low resolution. We evaluate the segmentation performance of MADGNet across six modalities and fifteen datasets. Through extensive experiments, we demonstrate that MADGNet consistently outperforms state-of-the-art models across various modalities, showcasing superior segmentation performance. This affirms MADGNet as a robust solution for medical image segmentation that excels in diverse imaging scenarios. Our MADGNet code is available in GitHub Link., Comment: Accepted in Computer Vision and Pattern Recognition (CVPR) 2024

Electrically tunable optical devices present diverse functionalities for manipulating electromagnetic waves by leveraging elements capable of reversibly switching between different optical states. This adaptability in adjusting their responses to electromagnetic waves after fabrication is crucial for developing more efficient and compact optical systems for a broad range of applications including sensing, imaging, telecommunications, and data storage. Chalcogenide-based phase change materials (PCMs) have shown great promise due to their stable, non-volatile phase transition between amorphous and crystalline states. Nonetheless, optimizing the switching parameters of PCM devices and maintaining their stable operation over thousands of cycles with minimal variation can be challenging. In this paper, we report on the critical role of PCM pattern as well as electrical pulse form in achieving reliable and stable switching, extending the operational lifetime of the device beyond 13,000 switching events. To achieve this, we have developed a computer-aided algorithm that monitors optical changes in the device and adjusts the applied voltage in accordance with the phase transformation process, thereby significantly enhancing the lifetime of these reconfigurable devices. Our findings reveal that patterned PCM structures show significantly higher endurance compared to blanket PCM thin films.

Photonic lantern nulling (PLN) is a method for enabling the detection and characterization of close-in exoplanets by exploiting the symmetries of the ports of a mode-selective photonic lantern (MSPL) to cancel out starlight. A six-port MSPL provides four ports where on-axis starlight is suppressed, while off-axis planet light is coupled with efficiencies that vary as a function of the planet's spatial position. We characterize the properties of a six-port MSPL in the laboratory and perform the first testbed demonstration of the PLN in monochromatic light (1569 nm) and in broadband light (1450 nm to 1625 nm), each using two orthogonal polarizations. We compare the measured spatial throughput maps with those predicted by simulations using the lantern's modes. We find that the morphologies of the measured throughput maps are reproduced by the simulations, though the real lantern is lossy and has lower throughputs overall. The measured ratios of on-axis stellar leakage to peak off-axis throughput are around 10^(-2), likely limited by testbed wavefront errors. These null-depths are already sufficient for observing young gas giants at the diffraction limit using ground-based observatories. Future work includes using wavefront control to further improve the nulls, as well as testing and validating the PLN on-sky., Comment: 30 pages, 12 figures

This study presents a model of automatic speech recognition (ASR) designed to diagnose pronunciation issues in children with speech sound disorders (SSDs) to replace manual transcriptions in clinical procedures. Since ASR models trained for general purposes primarily predict input speech into real words, employing a well-known high-performance ASR model for evaluating pronunciation in children with SSDs is impractical. We fine-tuned the wav2vec 2.0 XLS-R model to recognize speech as pronounced rather than as existing words. The model was fine-tuned with a speech dataset from 137 children with inadequate speech production pronouncing 73 Korean words selected for actual clinical diagnosis. The model's predictions of the pronunciations of the words matched the human annotations with about 90% accuracy. While the model still requires improvement in recognizing unclear pronunciation, this study demonstrates that ASR models can streamline complex pronunciation error diagnostic procedures in clinical fields., Comment: 12 pages, 2 figures

We construct an emergent holographic dual description in the Luttinger-Ward functional approach, where the renormalization group (RG) flows of collective bi-local fields appear manifestly in the bulk effective action with an emergent extra dimension. This holographic dual effective field theory reproduces $1/N$ quantum corrections in a self-consistent manner when we take the UV limit in the bulk effective action. Going into the IR regime in the extra dimension, we observe that a partial class of the field theoretic $1/N$, $1/N^{2}$, ... quantum corrections are resummed in the all-loop order and reorganized to form a holographic dual effective field theory in a large $N$ fashion living on the one-higher dimensional spacetime. In this study, we apply this theoretical framework into an Sachdev-Ye-Kitaev (SYK) model. Taking the large $N$ limit in the holographic dual effective field theory, we obtain nonlinearly coupled second-order bulk differential equations of motion for the three bi-local order-parameter fields of fermion self-energy, Green's function, and polarization function. Here, both UV and IR boundary conditions are derived self-consistently from the boundary effective action. We solve these highly intertwined nonlinear differential equations based on the so called matching method. Our ansatz for the bi-local order-parameter fields coincide with the conformally invariant solution of the field theoretic large $N$ limit in the UV limit, but their overall coefficients $RG-flow$ along the extra dimensional space, respectively, reflecting effects of higher-order quantum corrections. As a result, we find an insulating behavior, where the self-energy diverges at IR. ..., Comment: Regularizations have been clarified and a general scope has been discussed

Photonic Lanterns (PLs) are tapered waveguides that gradually transition from a multi-mode fiber geometry to a bundle of single-mode fibers (SMFs). They can efficiently couple multi-mode telescope light into a multi-mode fiber entrance at the focal plane and convert it into multiple single-mode beams. Thus, each SMF samples its unique mode (lantern principal mode) of the telescope light in the pupil, analogous to subapertures in aperture masking interferometry (AMI). Coherent imaging with PLs can be enabled by interfering SMF outputs and applying phase modulation, which can be achieved using a photonic chip beam combiner at the backend (e.g., the ABCD beam combiner). In this study, we investigate the potential of coherent imaging by interfering SMF outputs of a PL with a single telescope. We demonstrate that the visibilities that can be measured from a PL are mutual intensities incident on the pupil weighted by the cross-correlation of a pair of lantern modes. From numerically simulated lantern principal modes of a 6-port PL, we find that interferometric observables using a PL behave similarly to separated-aperture visibilities for simple models on small angular scales ($<\lambda/D$) but with greater sensitivity to symmetries and capability to break phase angle degeneracies. Furthermore, we present simulated observations with wavefront errors and compare them to AMI. Despite the redundancy caused by extended lantern principal modes, spatial filtering offers stability to wavefront errors. Our simulated observations suggest that PLs may offer significant benefits in the photon noise-limited regime and in resolving small angular scales at low contrast regime., Comment: Accepted for publication in ApJ

The direct imaging of an Earth-like exoplanet will require sub-nanometric wavefront control across large light-collecting apertures, to reject host starlight and detect the faint planetary signal. Current adaptive optics (AO) systems, which use wavefront sensors that reimage the telescope pupil, face two challenges that prevent this level of control: non-common-path aberrations (NCPAs), caused by differences between the sensing and science arms of the instrument; and petaling modes: discontinuous phase aberrations caused by pupil fragmentation, especially relevant for the upcoming 30-m class telescopes. Such aberrations drastically impact the capabilities of high-contrast instruments. To address these issues, we can add a second-stage wavefront sensor to the science focal plane. One promising architecture uses the photonic lantern (PL): a waveguide that efficiently couples aberrated light into single-mode fibers (SMFs). In turn, SMF-confined light can be stably injected into high-resolution spectrographs, enabling direct exoplanet characterization and precision radial velocity measurements; simultaneously, the PL can be used for focal-plane wavefront sensing. We present a real-time experimental demonstration of the PL wavefront sensor on the Subaru/SCExAO testbed. Our system is stable out to around ~400 nm of low-order Zernike wavefront error, and can correct petaling modes. When injecting ~30 nm RMS of low order time-varying error, we achieve ~10x rejection at 1 s timescales; further refinements to the control law and lantern fabrication process should make sub-nanometric wavefront control possible. In the future, novel sensors like the PLWFS may prove to be critical in resolving the wavefront control challenges posed by exoplanet direct imaging., Comment: Accepted to ApJL

Programmable and reconfigurable optics hold significant potential for transforming a broad spectrum of applications, spanning space explorations to biomedical imaging, gas sensing, and optical cloaking. The ability to adjust the optical properties of components like filters, lenses, and beam steering devices could result in dramatic reductions in size, weight, and power consumption in future optoelectronic devices. Among the potential candidates for reconfigurable optics, chalcogenide-based phase change materials (PCMs) offer great promise due to their non-volatile and analogue switching characteristics. Although PCM have found widespread use in electronic data storage, these memory devices are deeply sub-micron-sized. To incorporate phase change materials into free-space optical components, it is essential to scale them up to beyond several hundreds of microns while maintaining reliable switching characteristics. This study demonstrated a non-mechanical, non-volatile transmissive filter based on low-loss PCMs with a 200 $\mu$m$ \times $200 $\mu$m switching area. The device/metafilter can be consistently switched between low- and high-transmission states using electrical pulses with a switching contrast ratio of 5.5 dB. The device was reversibly switched for 1250 cycles before accelerated degradation took place. The work represents an important step toward realizing free-space reconfigurable optics based on PCMs.