Search

Relighting of human images enables post-photography editing of lighting effects in portraits. The current mainstream approach uses neural networks to approximate lighting effects without explicitly accounting for the principle of physical shading. As a result, it often has difficulty representing high-frequency shadows and shading. In this paper, we propose a two-stage relighting method that can reproduce physically-based shadows and shading from low to high frequencies. The key idea is to approximate an environment light source with a set of a fixed number of area light sources. The first stage employs supervised inverse rendering from a single image using neural networks and calculates physically-based shading. The second stage then calculates shadow for each area light and sums up to render the final image. We propose to make soft shadow mapping differentiable for the area-light approximation of environment lighting. We demonstrate that our method can plausibly reproduce all-frequency shadows and shading caused by environment illumination, which have been difficult to reproduce using existing methods., Comment: project page: [this URL](https://github.com/majita06/All-frequency_Full-body_Human_Image_Relighting)

Generalizable neural radiance field (NeRF) enables neural-based digital human rendering without per-scene retraining. When combined with human prior knowledge, high-quality human rendering can be achieved even with sparse input views. However, the inference of these methods is still slow, as a large number of neural network queries on each ray are required to ensure the rendering quality. Moreover, occluded regions often suffer from artifacts, especially when the input views are sparse. To address these issues, we propose a generalizable human NeRF framework that achieves high-quality and real-time rendering with sparse input views by extensively leveraging human prior knowledge. We accelerate the rendering with a two-stage sampling reduction strategy: first constructing boundary meshes around the human geometry to reduce the number of ray samples for sampling guidance regression, and then volume rendering using fewer guided samples. To improve rendering quality, especially in occluded regions, we propose an occlusion-aware attention mechanism to extract occlusion information from the human priors, followed by an image space refinement network to improve rendering quality. Furthermore, for volume rendering, we adopt a signed ray distance function (SRDF) formulation, which allows us to propose an SRDF loss at every sample position to improve the rendering quality further. Our experiments demonstrate that our method outperforms the state-of-the-art methods in rendering quality and has a competitive rendering speed compared with speed-prioritized novel view synthesis methods., Comment: project page: https://github.com/LarsPh/EG-HumanNeRF

Achieving aesthetically pleasing photography necessitates attention to multiple factors, including composition and capture conditions, which pose challenges to novices. Prior research has explored the enhancement of photo aesthetics post-capture through 2D manipulation techniques; however, these approaches offer limited search space for aesthetics. We introduce a pioneering method that employs 3D operations to simulate the conditions at the moment of capture retrospectively. Our approach extrapolates the input image and then reconstructs the 3D scene from the extrapolated image, followed by an optimization to identify camera parameters and image aspect ratios that yield the best 3D view with enhanced aesthetics. Comparative qualitative and quantitative assessments reveal that our method surpasses traditional 2D editing techniques with superior aesthetics., Comment: 10 pages

In this work, we introduce two types of makeup prior models to extend existing 3D face prior models: PCA-based and StyleGAN2-based priors. The PCA-based prior model is a linear model that is easy to construct and is computationally efficient. However, it retains only low-frequency information. Conversely, the StyleGAN2-based model can represent high-frequency information with relatively higher computational cost than the PCA-based model. Although there is a trade-off between the two models, both are applicable to 3D facial makeup estimation and related applications. By leveraging makeup prior models and designing a makeup consistency module, we effectively address the challenges that previous methods faced in robustly estimating makeup, particularly in the context of handling self-occluded faces. In experiments, we demonstrate that our approach reduces computational costs by several orders of magnitude, achieving speeds up to 180 times faster. In addition, by improving the accuracy of the estimated makeup, we confirm that our methods are highly advantageous for various 3D facial makeup applications such as 3D makeup face reconstruction, user-friendly makeup editing, makeup transfer, and interpolation., Comment: CVPR2024. Project: https://yangxingchao.github.io/makeup-priors-page

Pose and body shape editing in a human image has received increasing attention. However, current methods often struggle with dataset biases and deteriorate realism and the person's identity when users make large edits. We propose a one-shot approach that enables large edits with identity preservation. To enable large edits, we fit a 3D body model, project the input image onto the 3D model, and change the body's pose and shape. Because this initial textured body model has artifacts due to occlusion and the inaccurate body shape, the rendered image undergoes a diffusion-based refinement, in which strong noise destroys body structure and identity whereas insufficient noise does not help. We thus propose an iterative refinement with weak noise, applied first for the whole body and then for the face. We further enhance the realism by fine-tuning text embeddings via self-supervised learning. Our quantitative and qualitative evaluations demonstrate that our method outperforms other existing methods across various datasets., Comment: Accepted to WACV 2024, project page: https://www.cgg.cs.tsukuba.ac.jp/~okuyama/pub/diffbody/

This paper tackles text-guided control of StyleGAN for editing garments in full-body human images. Existing StyleGAN-based methods suffer from handling the rich diversity of garments and body shapes and poses. We propose a framework for text-guided full-body human image synthesis via an attention-based latent code mapper, which enables more disentangled control of StyleGAN than existing mappers. Our latent code mapper adopts an attention mechanism that adaptively manipulates individual latent codes on different StyleGAN layers under text guidance. In addition, we introduce feature-space masking at inference time to avoid unwanted changes caused by text inputs. Our quantitative and qualitative evaluations reveal that our method can control generated images more faithfully to given texts than existing methods., Comment: VISIAPP 2024, project page: https://www.cgg.cs.tsukuba.ac.jp/~yoshikawa/pub/style_human_clip/

Facial makeup enriches the beauty of not only real humans but also virtual characters; therefore, makeup for 3D facial models is highly in demand in productions. However, painting directly on 3D faces and capturing real-world makeup are costly, and extracting makeup from 2D images often struggles with shading effects and occlusions. This paper presents the first method for extracting makeup for 3D facial models from a single makeup portrait. Our method consists of the following three steps. First, we exploit the strong prior of 3D morphable models via regression-based inverse rendering to extract coarse materials such as geometry and diffuse/specular albedos that are represented in the UV space. Second, we refine the coarse materials, which may have missing pixels due to occlusions. We apply inpainting and optimization. Finally, we extract the bare skin, makeup, and an alpha matte from the diffuse albedo. Our method offers various applications for not only 3D facial models but also 2D portrait images. The extracted makeup is well-aligned in the UV space, from which we build a large-scale makeup dataset and a parametric makeup model for 3D faces. Our disentangled materials also yield robust makeup transfer and illumination-aware makeup interpolation/removal without a reference image., Comment: Eurographics 2023

We study the HII regions associated with the NGC 6334 molecular cloud observed in the sub-millimeter and taken as part of the B-fields In STar-forming Region Observations (BISTRO) Survey. In particular, we investigate the polarization patterns and magnetic field morphologies associated with these HII regions. Through polarization pattern and pressure calculation analyses, several of these bubbles indicate that the gas and magnetic field lines have been pushed away from the bubble, toward an almost tangential (to the bubble) magnetic field morphology. In the densest part of NGC 6334, where the magnetic field morphology is similar to an hourglass, the polarization observations do not exhibit observable impact from HII regions. We detect two nested radial polarization patterns in a bubble to the south of NGC 6334 that correspond to the previously observed bipolar structure in this bubble. Finally, using the results of this study, we present steps (incorporating computer vision; circular Hough Transform) that can be used in future studies to identify bubbles that have physically impacted magnetic field lines., Comment: Accepted for publication in Astrophysical Journal (ApJ)

The modern supervised approaches for human image relighting rely on training data generated from 3D human models. However, such datasets are often small (e.g., Light Stage data with a small number of individuals) or limited to diffuse materials (e.g., commercial 3D scanned human models). Thus, the human relighting techniques suffer from the poor generalization capability and synthetic-to-real domain gap. In this paper, we propose a two-stage method for single-image human relighting with domain adaptation. In the first stage, we train a neural network for diffuse-only relighting. In the second stage, we train another network for enhancing non-diffuse reflection by learning residuals between real photos and images reconstructed by the diffuse-only network. Thanks to the second stage, we can achieve higher generalization capability against various cloth textures, while reducing the domain gap. Furthermore, to handle input videos, we integrate illumination-aware deep video prior to greatly reduce flickering artifacts even with challenging settings under dynamic illuminations., Comment: Accepted to Pacific Graphics 2021, project page: http://www.cgg.cs.tsukuba.ac.jp/~tajima/pub/relighting_in_the_wild/

Semantic image synthesis is a process for generating photorealistic images from a single semantic mask. To enrich the diversity of multimodal image synthesis, previous methods have controlled the global appearance of an output image by learning a single latent space. However, a single latent code is often insufficient for capturing various object styles because object appearance depends on multiple factors. To handle individual factors that determine object styles, we propose a class- and layer-wise extension to the variational autoencoder (VAE) framework that allows flexible control over each object class at the local to global levels by learning multiple latent spaces. Furthermore, we demonstrate that our method generates images that are both plausible and more diverse compared to state-of-the-art methods via extensive experiments with real and synthetic datasets inthree different domains. We also show that our method enables a wide range of applications in image synthesis and editing tasks., Comment: Accepted to Pacific Graphics 2020, codes available at https://github.com/endo-yuki-t/DiversifyingSMIS

This paper tackles a challenging problem of generating photorealistic images from semantic layouts in few-shot scenarios where annotated training pairs are hardly available but pixel-wise annotation is quite costly. We present a training strategy that performs pseudo labeling of semantic masks using the StyleGAN prior. Our key idea is to construct a simple mapping between the StyleGAN feature and each semantic class from a few examples of semantic masks. With such mappings, we can generate an unlimited number of pseudo semantic masks from random noise to train an encoder for controlling a pre-trained StyleGAN generator. Although the pseudo semantic masks might be too coarse for previous approaches that require pixel-aligned masks, our framework can synthesize high-quality images from not only dense semantic masks but also sparse inputs such as landmarks and scribbles. Qualitative and quantitative results with various datasets demonstrate improvement over previous approaches with respect to layout fidelity and visual quality in as few as one- or five-shot settings., Comment: The source codes are available at https://github.com/endo-yuki-t/Fewshot-SMIS

We present new observations of the active star-formation region NGC 1333 in the Perseus molecular cloud complex from the James Clerk Maxwell Telescope B-Fields In Star-forming Region Observations (BISTRO) survey with the POL-2 instrument. The BISTRO data cover the entire NGC 1333 complex (~1.5 pc x 2 pc) at 0.02 pc resolution and spatially resolve the polarized emission from individual filamentary structures for the first time. The inferred magnetic field structure is complex as a whole, with each individual filament aligned at different position angles relative to the local field orientation. We combine the BISTRO data with low- and high- resolution data derived from Planck and interferometers to study the multiscale magnetic field structure in this region. The magnetic field morphology drastically changes below a scale of ~1 pc and remains continuous from the scales of filaments (~0.1 pc) to that of protostellar envelopes (~0.005 pc or ~1000 au). Finally, we construct simple models in which we assume that the magnetic field is always perpendicular to the long axis of the filaments. We demonstrate that the observed variation of the relative orientation between the filament axes and the magnetic field angles are well reproduced by this model, taking into account the projection effects of the magnetic field and filaments relative to the plane of the sky. These projection effects may explain the apparent complexity of the magnetic field structure observed at the resolution of BISTRO data toward the filament network., Comment: 33 pages, 21 figures, accepted for publication in ApJ

Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future uncertainty in both predictions by introducing latent codes. Unlike previous methods that infer output frames directly, our CNNs predict spatially-smooth intermediate data, i.e., for motion, flow fields for warping, and for appearance, color transfer maps, via self-supervised learning, i.e., without explicitly-provided ground truth. These intermediate data are applied not to each previous output frame, but to the input image only once for each output frame. This design is crucial to alleviate error accumulation in long-term predictions, which is the essential problem in previous recurrent approaches. The output frames can be looped like cinemagraph, and also be controlled directly by specifying latent codes or indirectly via visual annotations. We demonstrate the effectiveness of our method through comparisons with the state-of-the-arts on video prediction as well as appearance manipulation., Comment: Published at SIGGRAPH Asia 2019 (ACM Transactions on Graphics)

Relighting of human images has various applications in image synthesis. For relighting, we must infer albedo, shape, and illumination from a human portrait. Previous techniques rely on human faces for this inference, based on spherical harmonics (SH) lighting. However, because they often ignore light occlusion, inferred shapes are biased and relit images are unnaturally bright particularly at hollowed regions such as armpits, crotches, or garment wrinkles. This paper introduces the first attempt to infer light occlusion in the SH formulation directly. Based on supervised learning using convolutional neural networks (CNNs), we infer not only an albedo map, illumination but also a light transport map that encodes occlusion as nine SH coefficients per pixel. The main difficulty in this inference is the lack of training datasets compared to unlimited variations of human portraits. Surprisingly, geometric information including occlusion can be inferred plausibly even with a small dataset of synthesized human figures, by carefully preparing the dataset so that the CNNs can exploit the data coherency. Our method accomplishes more realistic relighting than the occlusion-ignored formulation., Comment: Published at SIGGRAPH Asia 2018 (ACM Transactions on Graphics). Project page with codes, pretrained models, and human model lists is at http://kanamori.cs.tsukuba.ac.jp/projects/relighting_human/

We present the POL-2 850 $\mu$m linear polarization map of the Barnard 1 clump in the Perseus molecular cloud complex from the B-fields In STar-forming Region Observations (BISTRO) survey at the James Clerk Maxwell Telescope. We find a trend of decreasing polarization fraction as a function of total intensity, which we link to depolarization effects towards higher density regions of the cloud. We then use the polarization data at 850 $\mu$m to infer the plane-of-sky orientation of the large-scale magnetic field in Barnard 1. This magnetic field runs North-South across most of the cloud, with the exception of B1-c where it turns more East-West. From the dispersion of polarization angles, we calculate a turbulence correlation length of $5.0 \pm 2.5$ arcsec ($1500$ au), and a turbulent-to-total magnetic energy ratio of $0.5 \pm 0.3$ inside the cloud. We combine this turbulent-to-total magnetic energy ratio with observations of NH$_3$ molecular lines from the Green Bank Ammonia Survey (GAS) to estimate the strength of the plane-of-sky component of the magnetic field through the Davis-Chandrasekhar-Fermi method. With a plane-of-sky amplitude of $120 \pm 60$ $\mu$G and a criticality criterion $\lambda_c = 3.0 \pm 1.5$, we find that Barnard 1 is a supercritical molecular cloud with a magnetic field nearly dominated by its turbulent component., Comment: 22 pages, 7 figures, accepted in the Astrophysical Journal

We report 850~$\mu$m dust polarization observations of a low-mass ($\sim$12 $M_{\odot}$) starless core in the $\rho$ Ophiuchus cloud, Ophiuchus C, made with the POL-2 instrument on the James Clerk Maxwell Telescope (JCMT) as part of the JCMT B-fields In STar-forming Region Observations (BISTRO) survey. We detect an ordered magnetic field projected on the plane of sky in the starless core. The magnetic field across the $\sim$0.1~pc core shows a predominant northeast-southwest orientation centering between $\sim$40$^\circ$ to $\sim$100$^\circ$, indicating that the field in the core is well aligned with the magnetic field in lower-density regions of the cloud probed by near-infrared observations and also the cloud-scale magnetic field traced by Planck observations. The polarization percentage ($P$) decreases with an increasing total intensity ($I$) with a power-law index of $-$1.03 $\pm$ 0.05. We estimate the plane-of-sky field strength ($B_{\mathrm{pos}}$) using modified Davis-Chandrasekhar-Fermi (DCF) methods based on structure function (SF), auto-correlation (ACF), and unsharp masking (UM) analyses. We find that the estimates from the SF, ACF, and UM methods yield strengths of 103 $\pm$ 46 $\mu$G, 136 $\pm$ 69 $\mu$G, and 213 $\pm$ 115 $\mu$G, respectively. Our calculations suggest that the Ophiuchus C core is near magnetically critical or slightly magnetically supercritical (i.e. unstable to collapse). The total magnetic energy calculated from the SF method is comparable to the turbulent energy in Ophiuchus C, while the ACF method and the UM method only set upper limits for the total magnetic energy because of large uncertainties., Comment: 22 pages, 12 figures

We present the 850 $\mu$m polarization observations toward the IC5146 filamentary cloud taken using the Submillimetre Common-User Bolometer Array 2 (SCUBA-2) and its associated polarimeter (POL-2), mounted on the James Clerk Maxwell Telescope (JCMT), as part of the B-fields In STar forming Regions Observations (BISTRO). This work is aimed at revealing the magnetic field morphology within a core-scale ($\lesssim 1.0$ pc) hub-filament structure (HFS) located at the end of a parsec-scale filament. To investigate whether or not the observed polarization traces the magnetic field in the HFS, we analyze the dependence between the observed polarization fraction and total intensity using a Bayesian approach with the polarization fraction described by the Rice likelihood function, which can correctly describe the probability density function (PDF) of the observed polarization fraction for low signal-to-noise ratio (SNR) data. We find a power-law dependence between the polarization fraction and total intensity with an index of 0.56 in $A_V\sim$ 20--300 mag regions, suggesting that the dust grains in these dense regions can still be aligned with magnetic fields in the IC5146 regions. Our polarization maps reveal a curved magnetic field, possibly dragged by the contraction along the parsec-scale filament. We further obtain a magnetic field strength of 0.5$\pm$0.2 mG toward the central hub using the Davis-Chandrasekhar-Fermi method, corresponding to a mass-to-flux criticality of $\sim$ $1.3\pm0.4$ and an Alfv\'{e}nic Mach number of $<$0.6. These results suggest that gravity and magnetic field is currently of comparable importance in the HFS, and turbulence is less important., Comment: 24 pages, 15 figures, accepted for publication in ApJ

We present the results of dust emission polarization measurements of Ophiuchus-B (Oph-B) carried out using the Submillimetre Common-User Bolometer Array 2 (SCUBA-2) camera with its associated polarimeter (POL-2) on the James Clerk Maxwell Telescope (JCMT) in Hawaii. This work is part of the B-fields In Star-forming Region Observations (BISTRO) survey initiated to understand the role of magnetic fields in star formation for nearby star-forming molecular clouds. We present a first look at the geometry and strength of magnetic fields in Oph-B. The field geometry is traced over $\sim$0.2 pc, with clear detection of both of the sub-clumps of Oph-B. The field pattern appears significantly disordered in sub-clump Oph-B1. The field geometry in Oph-B2 is more ordered, with a tendency to be along the major axis of the clump, parallel to the filamentary structure within which it lies. The degree of polarization decreases systematically towards the dense core material in the two sub-clumps. The field lines in the lower density material along the periphery are smoothly joined to the large scale magnetic fields probed by NIR polarization observations. We estimated a magnetic field strength of 630$\pm$410 $\mu$G in the Oph-B2 sub-clump using a Davis-Chandeasekhar-Fermi analysis. With this magnetic field strength, we find a mass-to-flux ratio $\lambda$= 1.6$\pm$1.1, which suggests that the Oph-B2 clump is slightly magnetically supercritical., Comment: 17 pages, 10 figures, Accepted for publication in ApJ

We present 850 $\mu$m imaging polarimetry data of the $\rho$ Oph-A core taken with the Submillimeter Common-User Bolometer Array-2 (SCUBA-2) and its polarimeter (POL-2), as part of our ongoing survey project, BISTRO (B-fields In STar forming RegiOns). The polarization vectors are used to identify the orientation of the magnetic field projected on the plane of the sky at a resolution of 0.01 pc. We identify 10 subregions with distinct polarization fractions and angles in the 0.2 pc $\rho$ Oph A core; some of them can be part of a coherent magnetic field structure in the $\rho$ Oph region. The results are consistent with previous observations of the brightest regions of $\rho$ Oph-A, where the degrees of polarization are at a level of a few percents, but our data reveal for the first time the magnetic field structures in the fainter regions surrounding the core where the degree of polarization is much higher ($> 5 \%$). A comparison with previous near-infrared polarimetric data shows that there are several magnetic field components which are consistent at near-infrared and submillimeter wavelengths. Using the Davis-Chandrasekhar-Fermi method, we also derive magnetic field strengths in several sub-core regions, which range from approximately 0.2 to 5 mG. We also find a correlation between the magnetic field orientations projected on the sky with the core centroid velocity components., Comment: 22 pages, 11 figures, accepted for publication in ApJ