Search

We introduce PortraitGen, a powerful portrait video editing method that achieves consistent and expressive stylization with multimodal prompts. Traditional portrait video editing methods often struggle with 3D and temporal consistency, and typically lack in rendering quality and efficiency. To address these issues, we lift the portrait video frames to a unified dynamic 3D Gaussian field, which ensures structural and temporal coherence across frames. Furthermore, we design a novel Neural Gaussian Texture mechanism that not only enables sophisticated style editing but also achieves rendering speed over 100FPS. Our approach incorporates multimodal inputs through knowledge distilled from large-scale 2D generative models. Our system also incorporates expression similarity guidance and a face-aware portrait editing module, effectively mitigating degradation issues associated with iterative dataset updates. Extensive experiments demonstrate the temporal consistency, editing efficiency, and superior rendering quality of our method. The broad applicability of the proposed approach is demonstrated through various applications, including text-driven editing, image-driven editing, and relighting, highlighting its great potential to advance the field of video editing. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/PortraitGen/, Comment: Accepted by SIGGRAPH Asia 2024. Project Page: https://ustc3dv.github.io/PortraitGen/

Retrieval-augmented generation (RAG) techniques have proven to be effective in integrating up-to-date information, mitigating hallucinations, and enhancing response quality, particularly in specialized domains. While many RAG approaches have been proposed to enhance large language models through query-dependent retrievals, these approaches still suffer from their complex implementation and prolonged response times. Typically, a RAG workflow involves multiple processing steps, each of which can be executed in various ways. Here, we investigate existing RAG approaches and their potential combinations to identify optimal RAG practices. Through extensive experiments, we suggest several strategies for deploying RAG that balance both performance and efficiency. Moreover, we demonstrate that multimodal retrieval techniques can significantly enhance question-answering capabilities about visual inputs and accelerate the generation of multimodal content using a "retrieval as generation" strategy.

Deciphering natural language from brain activity through non-invasive devices remains a formidable challenge. Previous non-invasive decoders either require multiple experiments with identical stimuli to pinpoint cortical regions and enhance signal-to-noise ratios in brain activity, or they are limited to discerning basic linguistic elements such as letters and words. We propose a novel approach to decoding continuous language from single-trial non-invasive fMRI recordings, in which a three-dimensional convolutional network augmented with information bottleneck is developed to automatically identify responsive voxels to stimuli, and a character-based decoder is designed for the semantic reconstruction of continuous language characterized by inherent character structures. The resulting decoder can produce intelligible textual sequences that faithfully capture the meaning of perceived speech both within and across subjects, while existing decoders exhibit significantly inferior performance in cross-subject contexts. The ability to decode continuous language from single trials across subjects demonstrates the promising applications of non-invasive language brain-computer interfaces in both healthcare and neuroscience.

We propose FlashAvatar, a novel and lightweight 3D animatable avatar representation that could reconstruct a digital avatar from a short monocular video sequence in minutes and render high-fidelity photo-realistic images at 300FPS on a consumer-grade GPU. To achieve this, we maintain a uniform 3D Gaussian field embedded in the surface of a parametric face model and learn extra spatial offset to model non-surface regions and subtle facial details. While full use of geometric priors can capture high-frequency facial details and preserve exaggerated expressions, proper initialization can help reduce the number of Gaussians, thus enabling super-fast rendering speed. Extensive experimental results demonstrate that FlashAvatar outperforms existing works regarding visual quality and personalized details and is almost an order of magnitude faster in rendering speed. Project page: https://ustc3dv.github.io/FlashAvatar/, Comment: Project page: https://ustc3dv.github.io/FlashAvatar/

Recently, many works have been proposed to utilize the neural radiance field for novel view synthesis of human performers. However, most of these methods require hours of training, making them difficult for practical use. To address this challenging problem, we propose IntrinsicNGP, which can train from scratch and achieve high-fidelity results in few minutes with videos of a human performer. To achieve this target, we introduce a continuous and optimizable intrinsic coordinate rather than the original explicit Euclidean coordinate in the hash encoding module of instant-NGP. With this novel intrinsic coordinate, IntrinsicNGP can aggregate inter-frame information for dynamic objects with the help of proxy geometry shapes. Moreover, the results trained with the given rough geometry shapes can be further refined with an optimizable offset field based on the intrinsic coordinate.Extensive experimental results on several datasets demonstrate the effectiveness and efficiency of IntrinsicNGP. We also illustrate our approach's ability to edit the shape of reconstructed subjects., Comment: Project page:https://ustc3dv.github.io/IntrinsicNGP/. arXiv admin note: substantial text overlap with arXiv:2210.01651

We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. Thanks to the powerful representation ability of neural radiance field, the constructed model can represent complex facial attributes including hair, wearings, which can not be represented by traditional mesh blendshape. To construct the personalized semantic facial model, we propose to define the bases as several multi-level voxel fields. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes, and can render a photo-realistic human head image in tens of miliseconds with a given expression coefficient and view direction. With this novel representation, we apply it to many tasks like facial retargeting and expression editing. Experimental results demonstrate its strong representation ability and training/inference speed. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/NeRFBlendShape/, Comment: Accepted by SIGGRAPH Asia 2022 (Journal Track). Project page: https://ustc3dv.github.io/NeRFBlendShape/

In online shopping, ever-changing fashion trends make merchants need to prepare more differentiated products to meet the diversified demands, and e-commerce platforms need to capture the market trend with a prophetic vision. For the trend prediction, the attribute tags, as the essential description of items, can genuinely reflect the decision basis of consumers. However, few existing works explore the attribute trend in the specific community for e-commerce. In this paper, we focus on the community trend prediction on the item attribute and propose a unified framework that combines the dynamic evolution of two graph patterns to predict the attribute trend in a specific community. Specifically, we first design a communityattribute bipartite graph at each time step to learn the collaboration of different communities. Next, we transform the bipartite graph into a hypergraph to exploit the associations of different attribute tags in one community. Lastly, we introduce a dynamic evolution component based on the recurrent neural networks to capture the fashion trend of attribute tags. Extensive experiments on three real-world datasets in a large e-commerce platform show the superiority of the proposed approach over several strong alternatives and demonstrate the ability to discover the community trend in advance., Comment: Published as a full paper at WSDM 2022

Fermi liquid theory has been a foundation in understanding the electronic properties of materials. For weakly interacting two-dimensional (2D) electron or hole systems, electron-electron interactions are known to introduce quantum corrections to the Drude conductivity in the FL theory, giving rise to temperature dependent conductivity and magneto-resistance. Here we study the magneto-transport in a strongly interacting 2D hole system over a broad range of temperatures ($T$ = 0.09 to $>$1K) and densities $p=1.98-0.99\times10^{10}$ cm$^{-2}$ where the ratio between Coulomb energy and Fermi energy $r_s$ = 20 - 30. We show that while the system exhibits a negative parabolic magneto-resistance at low temperatures ($\lesssim$ 0.4K) characteristic of an interacting FL, the FL interaction corrections represent an insignificant fraction of the total conductivity. Surprisingly, a positive magneto-resistance emerges at high temperatures and grows with increasing temperature even in the regime $T \sim E_F$, close to the Fermi temperature. This unusual positive magneto-resistance at high temperatures is attributed to the collective viscous transport of 2D hole fluid in the hydrodynamic regime where holes scatter frequently with each other. These findings highlight the collective transport in a strongly interacting 2D system in the $r_s\gg 1$ regime and the hydrodynamic transport induced magneto-resistance opens up possibilities to new routes of magneto-resistance at high temperatures., Comment: 9 pages, 4 figures

In the field of atomically thin 2D materials, oxides are relatively unexplored in spite of the large number of layered oxide structures amenable to exfoliation. There is an increasing interest in ultra-thin film oxide nanostructures from applied points of view. In this perspective paper, recent progress in understanding the fundamental properties of 2D oxides is discussed. Two families of 2D oxides are considered: (1) van der Waals bonded layered materials in which the transition metal is in its highest valence state (represented by V$_2$O$_5$ and MoO$_3$) and (2) layered materials with ionic bonding between positive alkali cation layers and negatively charged transition metal oxide layers (LiCoO$_2$). The chemical exfoliation process and its combinaton with mechanical exfoliation are presented for the latter. Structural phase stability of the resulting nanoflakes, the role of cation size and the importance of defects in oxides are discussed. Effects of two-dimensionality on phonons, electronic band structures and electronic screening are placed in the context of what is known on other 2D materials, such as transition metal dichalcogenides. Electronic structure is discussed at the level of many-body-perturbation theory using the quasiparticle self-consistent $GW$ method, the accuracy of which is critically evaluated including effects of electron-hole interactions on screening and electron-phonon coupling. The predicted occurence of a two-dimensional electron gas on Li covered surfaces of LiCoO$_2$ and its relation to topological aspects of the band structure and bonding is presented as an example of the essential role of the surface in ultrathin materials. Finally, some case studies of the electronic transport and the use of these oxides in nanoscale field effect transistors are presented., Comment: 28 pages, 15 figures

A new reentrant insulating phase (RIP) in low magnetic fields has been reported in the literature in strongly interacting 2D carrier systems and was suggested to be related to the formation of a Wigner crystal [e.g. Qiu et al, PRL 108, 106404 (2012)]. We have studied the transformation between the metallic liquid phase and the low field RIP in a dilute 2D hole system with large interaction parameter $r_s$ (~20-30) in GaAs quantum wells. Instead of a sharp transition, increasing density (or lowering $r_s$) drives the RIP into a state where an incipient RIP coexists with the metallic 2D hole liquid. The non-trivial temperature dependent resistivity and the in-plane magnetic field induced enhancement of the RIP highlight the competition between two phases and the essential role of spin in this mixture phase, and are consistent with the Pomeranchuk effect in a mixture of Wigner crystal and Fermi liquid., Comment: arXiv admin note: text overlap with arXiv:1509.07463

Chemically exfoliated nanoscale few-layer thin Li$_x$CoO$_2$ samples are studied as function of annealing at various temperatures, using transmission electron microscopy (TEM) and Electron Energy Loss Spectroscopies (EELS), probing the O-K, Co-L$_{2,3}$ spectra along with low energy interband transitions. These spectra are compared with first-principles DFT calculations of -Im$[\varepsilon^{-1}(q,\omega)]$ and O-2p Partial Densities of States weighted by dipole matrix elements with the core wavefunction and including the O-1s core-hole and with known trends of the L$_2$/L$_3$ peak ratio to average Co valence. Trends in these spectra under the annealing procedures are established and correlated with the structural phase changes observed from diffraction TEM and High Resolution TEM images. The results are also correlated with conductivity measurements on samples subjected to the same annealing procedures. A gradual disordering of the Li and Co cations in the lattice is observed starting from a slight distortion of the pure LiCoO$_2$ $R\bar{3}m$ to $C2/m$ due to the lower Li content, followed by a $P2/m$ phase forming at 200$^o$C indicative of Li-vacancy ordering, formation of a spinel type $Fd\bar{3}m$ phase around 250$^o$C and ultimately a rocksalt type $Fm\bar{3}m$ phase above 350$^o$C. This disordering leads to a lowering of the band gap as established by low energy EELS. The O-K spectra of the rocksalt phase are only reproduced by a calculation for pure CoO and not for a model with random distribution of Li and Co. This indicates that there may be a loss of Li from the rocksalt regions of the sample at these higher temperatures. The conductivity measurements indicate a gradual drop in conductivity above 200$^o$C, which is clearly related to the more Li-Co interdiffused phases, in which a low-spin electronic structure is no longer valid and stronger correlation effects are expected.

The COVID-19 global pandemic and resulted lockdown policies have forced education in nearly every country to switch from a traditional co-located paradigm to a pure online 'distance learning from home' paradigm. Lying in the center of this learning paradigm shift is the emergence and wide adoption of distance communication tools and live streaming platforms for education. Here, we present a mixed-methods study on live streaming based education experience during the COVID-19 pandemic. We focus our analysis on Chinese higher education, carried out semi-structured interviews on 30 students, and 7 instructors from diverse colleges and disciplines, meanwhile launched a large-scale survey covering 6291 students and 1160 instructors in one leading Chinese university. Our study not only reveals important design guidelines and insights to better support current remote learning experience during the pandemic, but also provides valuable implications towards constructing future collaborative education supporting systems and experience after pandemic., Comment: Zhilong Chen and Hancheng Cao contribute equally to this work; Accepted to CHI 2021

Heterostructures of 2D van der Waals semiconductor materials offer a diverse playground for exploring fundamental physics and potential device applications. In InSe/GaSe heterostructures formed by sequential mechanical exfoliation and stacking of 2D monochalcogenides InSe and GaSe, we observe charge transfer between InSe and GaSe due to the 2D van der Waals interface formation and a strong hysteresis effect in the electron transport through the InSe layer when a gate voltage is applied through the GaSe layer. A gate voltage dependant conductance decay rate is also observed. We relate these observations to the gate voltage dependant dynamical charge transfer between InSe and GaSe layers.

Electron-electron interactions (EEIs) in 2D van der Waals structures is one of the topics with high current interest in physics. We report the observation of a negative parabolic magnetoresistance (MR) in multilayer 2D semiconductor InSe beyond the low-field weak localization/antilocalization regime, and provide evidence for the EEI origin of this MR behavior. Further, we analyze this negative parabolic MR and other observed quantum transport signatures of EEIs (temperature dependent conductance and Hall coefficient) within the framework of Fermi liquid theory and extract the gate voltage tunable Fermi liquid parameter $F_0^\sigma$ which quantifies the electron spin-exchange interaction strength.

The tunability of topological surface states and controllable opening of the Dirac gap are of great importance to the application of topological materials. In topological crystalline insulators (TCIs), crystal symmetry and topology of electronic bands intertwine to create topological surface states and thus the Dirac gap can be modulated by symmetry breaking structural changes of lattice. By transport measurement on heterostructures composed of p-type topological crystalline insulator SnTe and n-type conventional semiconductor PbTe, here we show a giant linear magnetoresistance (up to 2150% under 14 T at 2 K) induced by the Dirac Fermions at the PbTe/SnTe interface. In contrast, PbTe/SnTe samples grown at elevated temperature exhibit a cubic-to-rhombohedral structural phase transition of SnTe lattice below 100 K and weak antilocalization effect. Such distinctive magneto-resistance behavior is attributed to the broken mirror symmetry and gapping of topological surface states. Our work provides a promising application for future magneto-electronics and spintronics based on TCI heterostructures., Comment: 16 pages of text including four figures

Manipulating the electron spin with the aid of spin-orbit coupling (SOC) is an indispensable element of spintronics. Electrostatically gating a material with strong SOC results in an effective magnetic field which can in turn be used to govern the electron spin. In this work, we report the existence and electrostatic tunability of Rashba SOC in multilayer InSe. We observed a gate-voltage-tuned crossover from weak localization (WL) to weak antilocalization (WAL) effect in quantum transport studies of InSe, which suggests an increasing SOC strength. Quantitative analyses of magneto-transport studies and energy band diagram calculations provide strong evidence for the predominance of Rashba SOC in electrostatically gated InSe. Furthermore, we attribute the tendency of the SOC strength to saturate at high gate voltages to the increased electronic density of states-induced saturation of the electric field experienced by the electrons in the InSe layer. This explanation of nonlinear gate voltage control of Rashba SOC can be generalized to other electrostatically gated semiconductor nanomaterials in which a similar tendency of spin-orbit length saturation was observed (e.g. nanowire field effect transistors), and is thus of broad implications in spintronics. Identifying and controlling the Rashba SOC in InSe may serve pivotally in devising III-VI semiconductor-based spintronic devices in the future., Comment: 7 pages, 5 figures