Search

We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images, for novel view synthesis. Previous solutions, such as pixelNeRF [68], rely only on pixel-aligned features and suffer from feature ambiguity issues. As a result, they struggle with the disentanglement of geometry and appearance, leading to implausible geometries and blurry results. To address this challenge, we propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. Specifically, in addition to pixel-aligned features, we further constrain the radiance field learning to be conditioned on i) voxel-aligned features learned from a coarse volumetric grid and ii) fine surface-aligned features extracted from a regressed point cloud. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry, i.e. recovering more accurate geometries and synthesizing higher quality images of novel views. Extensive experiments against state-of-the-art methods on ShapeNet benchmarks demonstrate the superiority of our approach for single-image novel view synthesis.

Machine learning often requires a large amount of training data, and the training data obtained from various sources is often of poor quality, such as a large number of corrupted labels. Researchers using machine learning often apply some data cleaning techniques to clean up corrupted data. There are two popular methods to clean corrupted data: one is to set manual cleaning rules, and the other is to use positive samples for machine learning or statistical methods. Our work proposes a data cleaning method based on ACGAN since it is difficult to manually formulate cleaning rules, and there are often no positive samples of training data too. Our work does not need to artificially add cleaning rules or positive samples, and subtly uses mode dropping of GAN to eliminate the impact of noisy labels on corrupted data so which can be converted to relatively clean synthetic training data. Mode dropping of ACGAN will naturally happens, which is originally a disadvantage that usually needs to be eliminated in GAN, we tom the disadvantage into advantage, ACGAN will ignore some non-subject features when generating data, so as to eliminate the impact of noisy labels. And we also apply our method to correct noisy labels on corrupted training data.

Recent advances in unsupervised domain adaptation have achieved remarkable performance on semantic segmentation tasks. Despite such progress, existing works mainly focus on bridging the inter-domain gaps between the source and target domain, while only few of them noticed the intra-domain gaps within the target data. In this work, we propose a pixel-level intra-domain adaptation approach to reduce the intra-domain gaps within the target data. Compared with image-level methods, ours treats each pixel as an instance, which adapts the segmentation model at a more fine-grained level. Specifically, we first conduct the inter-domain adaptation between the source and target domain; Then, we separate the pixels in target images into the easy and hard subdomains; Finally, we propose a pixel-level adversarial training strategy to adapt a segmentation network from the easy to the hard subdomain. Moreover, we show that the segmentation accuracy can be further improved by incorporating a continuous indexing technique in the adversarial training. Experimental results show the effectiveness of our method against existing state-of-the-art approaches.

Despite the success of Lipschitz regularization in stabilizingGAN training, the exact reason of its effectiveness remains poorly un-derstood. The direct effect ofK-Lipschitz regularization is to restrict theL2-norm of the neural network gradient to be smaller than a thresholdK(e.g.,K= 1) such that‖∇f‖≤K. In this work, we uncover an evenmore important effect of Lipschitz regularization by examining its im-pact on the loss function:It degenerates GAN loss functions to almostlinear ones by restricting their domain and interval of attainable gradi-ent values. Our analysis shows that loss functions are only successful ifthey are degenerated to almost linear ones. We also show that loss func-tions perform poorly if they are not degenerated and that a wide rangeof functions can be used as loss function as long as they are sufficientlydegenerated by regularization. Basically, Lipschitz regularization ensuresthat all loss functionseffectively work in the same way.Empirically, weverify our proposition on the MNIST, CIFAR10 and CelebA datasets.

We propose Image2StyleGAN++, a flexible image editing framework with many applications. Our framework extends the recent Image2StyleGAN [1] in three ways. First, we introduce noise optimization as a complement to the W+ latent space embedding. Our noise optimization can restore high frequency features in images and thus significantly improves the quality of reconstructed images, e.g. a big increaseofPSNRfrom20dBto45dB.Second,we extend the global W+ latent space embedding to enable local embeddings. Third, we combine embedding with activation tensor manipulation to perform high quality local edits along with global semantic edits on images. Such edits motivate various high quality image editing applications, e.g. image reconstruction, image inpainting, image crossover, local style transfer, image editing using scribbles, and attribute level feature transfer. Examples of the edited images are shown across the paper for visual inspection.

We propose semantic region-adaptive normalization (SEAN),asimplebuteffectivebuildingblockforGenerative Adversarial Networks conditioned on segmentation masks that describe the semantic regions in the desired output image. Using SEAN normalization, we can build a network architecture that can control the style of each semantic region individually, e.g., we can specify one style reference image per region. SEAN is better suited to encode, transfer, and synthesize style than the best previous method in terms of reconstruction quality, variability, and visual quality. We evaluateSEANonmultipledatasetsandreportbetterquan titative metrics (e.g. FID, PSNR) than the current state of the art. SEAN also pushes the frontier of interactive image editing. We can interactively edit images by changing segmentation masks or the style for any given region. We can also interpolate styles from two reference images per region. Code: https://github.com/ZPdesu/SEAN.

We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing operations that can be applied to existing photographs. Taking the StyleGAN trained on the FFHD dataset as an example, we show results for image morphing, style transfer, and expression transfer. Studying the results of the embedding algorithm provides valuable insights into the structure of the StyleGAN latent space. We propose a set of experiments to test what class of images can be embedded, how they are embedded, what latent space is suitable for embedding, and if the embedding is semantically meaningful.

Computing discrete geodesic distance over triangle meshes is one of the fundamental problems in computational geometry and computer graphics. As the “Big Data Era” arrives, a fast and accurate solution to the geodesic computation problem on large scale models with constantly increasing resolutions is desired. However, it is still challenging to deal with the speed, memory cost and accuracy of the geodesic computation at the same time. This thesis addresses the aforementioned challenge by proposing the Edge- based Windows Grouping (EWG) technique. With the local geodesic information encoded in a “window”, EWG groups the windows based on the mesh edges and processes them together. Thus, the interrelationships among the grouped windows can be utilized to improve the performance of geodesic computation on triangle meshes. Based on EWG, a novel exact geodesic algorithm is proposed in this thesis, which is fast, accurate and memory-efficient. This algorithm computes the geodesic distances at mesh vertices by propagating the geodesic information from the source over the entire mesh. Its high performance comes from its low computational redundancy and management overhead, which are both introduced by EWG. First, the redundant windows on an edge can be removed by comparing its distance with those of the other windows on the same edge. Second, the windows grouped on an edge usually have similar geodesic distances and can be propagated in batches efficiently. To the best of my knowledge, the proposed exact geodesic algorithm is the fastest and most memory-efficient one among all existing methods. In addition, the proposed exact geodesic algorithm is revised and employed to construct the geodesic-metric-based Voronoi diagram on triangle meshes. In this application, the geodesic computation is the bottleneck in both the time and memory costs. The proposed method achieves low memory cost from the key observation that the Voronoi diagram boundaries usually only cross a minority of