Search

A Neural Radiance Field (NeRF) encodes the specific relation of 3D geometry and appearance of a scene. We here ask the question whether we can transfer the appearance from a source NeRF onto a target 3D geometry in a semantically meaningful way, such that the resulting new NeRF retains the target geometry but has an appearance that is an analogy to the source NeRF. To this end, we generalize classic image analogies from 2D images to NeRFs. We leverage correspondence transfer along semantic affinity that is driven by semantic features from large, pre-trained 2D image models to achieve multi-view consistent appearance transfer. Our method allows exploring the mix-and-match product space of 3D geometry and appearance. We show that our method outperforms traditional stylization-based methods and that a large majority of users prefer our method over several typical baselines., Comment: Project page: https://mfischer-ucl.github.io/nerf_analogies/

Bounding volumes are an established concept in computer graphics and vision tasks but have seen little change since their early inception. In this work, we study the use of neural networks as bounding volumes. Our key observation is that bounding, which so far has primarily been considered a problem of computational geometry, can be redefined as a problem of learning to classify space into free or occupied. This learning-based approach is particularly advantageous in high-dimensional spaces, such as animated scenes with complex queries, where neural networks are known to excel. However, unlocking neural bounding requires a twist: allowing -- but also limiting -- false positives, while ensuring that the number of false negatives is strictly zero. We enable such tight and conservative results using a dynamically-weighted asymmetric loss function. Our results show that our neural bounding produces up to an order of magnitude fewer false positives than traditional methods. In addition, we propose an extension of our bounding method using early exits that accelerates query speeds by 25%. We also demonstrate that our approach is applicable to non-deep learning models that train within seconds. Our project page is at: https://wenxin-liu.github.io/neural_bounding/., Comment: Published in Proc. of SIGGRAPH, 2024

Gradient-based optimization is now ubiquitous across graphics, but unfortunately can not be applied to problems with undefined or zero gradients. To circumvent this issue, the loss function can be manually replaced by a ``surrogate'' that has similar minima but is differentiable. Our proposed framework, ZeroGrads, automates this process by learning a neural approximation of the objective function, which in turn can be used to differentiate through arbitrary black-box graphics pipelines. We train the surrogate on an actively smoothed version of the objective and encourage locality, focusing the surrogate's capacity on what matters at the current training episode. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. As sampling the objective is expensive (it requires a full rendering or simulator run), we devise an efficient sampling scheme that allows for tractable run-times and competitive performance at little overhead. We demonstrate optimizing diverse non-convex, non-differentiable black-box problems in graphics, such as visibility in rendering, discrete parameter spaces in procedural modelling or optimal control in physics-driven animation. In contrast to other derivative-free algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k interlinked variables., Comment: Accepted at SIGGRAPH 2024. Project page: https://mfischer-ucl.github.io/zerograds

Repulsion between individuals within a finite radius is encountered in numerous applications, including cell exclusion, i.e. avoidance of overlapping cells, bird flocks, or microscopic pedestrian models. We define such individual based particle dynamics in one spatial dimension with minimal assumptions of the repulsion force f as well as their external velocity v and prove their characteristic properties. Moreover, we are able to perform a rigorous limit from the microscopic to the macroscopic scale, where we could recover the finite interaction radius as a density threshold. Specific choices for the repulsion force f lead to well known nonlinear diffusion equations on the macroscopic scale, as e.g. the porous medium equation. At both scaling levels numerical simulations are presented and compared to underline the analytical results.

Current differentiable renderers provide light transport gradients with respect to arbitrary scene parameters. However, the mere existence of these gradients does not guarantee useful update steps in an optimization. Instead, inverse rendering might not converge due to inherent plateaus, i.e., regions of zero gradient, in the objective function. We propose to alleviate this by convolving the high-dimensional rendering function that maps scene parameters to images with an additional kernel that blurs the parameter space. We describe two Monte Carlo estimators to compute plateau-free gradients efficiently, i.e., with low variance, and show that these translate into net-gains in optimization error and runtime performance. Our approach is a straightforward extension to both black-box and differentiable renderers and enables optimization of problems with intricate light transport, such as caustics or global illumination, that existing differentiable renderers do not converge on., Comment: Accepted to CVPR 2023. Project page and interactive demos at https://mfischer-ucl.github.io/prdpt/

The quadrupole coupling constant $C_Q$ and the asymmetry parameter $\eta$ have been determined for two complex aluminium hydrides from $^{27}$Al NMR spectra recorded for stationary samples by using the Solomon echo sequence. The thus obtained data for KAlH4 ($C_Q=(1.30\pm0.02)$ MHz, $\eta=(0.64\pm0.02)$) and NaAlH4 ($C_Q=(3.11\pm0.02)$ MHz, $\eta<0.01$) agree very well with data previously determined from MAS NMR spectra. The accuracy with which these parameters can be determined from static spectra turned out to be at least as good as via the MAS approach. The experimentally determined parameters ($\delta_{iso}$, $C_Q$ and $\eta$) are compared with those obtained from DFT-GIPAW (density functional theory - gauge-including projected augmented wave) calculations. Except for the quadrupole coupling constant for KAlH4, which is overestimated in the GIPAW calculations by about 30%, the agreement is excellent. Advantages of the application of the Solomon echo sequence for the measurement of less stable materials or for in-situ studies are discussed., Comment: 26 pages, 7 figures

We propose a method to accelerate the joint process of physically acquiring and learning neural Bi-directional Reflectance Distribution Function (BRDF) models. While BRDF learning alone can be accelerated by meta-learning, acquisition remains slow as it relies on a mechanical process. We show that meta-learning can be extended to optimize the physical sampling pattern, too. After our method has been meta-trained for a set of fully-sampled BRDFs, it is able to quickly train on new BRDFs with up to five orders of magnitude fewer physical acquisition samples at similar quality. Our approach also extends to other linear and non-linear BRDF models, which we show in an extensive evaluation., Comment: Accepted to Eurographics 2023; Project Page at https://ryushinn.github.io/metasampling

Understanding the non-deterministic behavior of deterministic nonlinear systems has been an implicit dream since Lorenz named it the "butterfly effect". A prominent example is the hysteresis and bistability of the Duffing oscillator, which in the classical description is attributed to the coexistence of two steady states in a double-well potential. However, this interpretation fails in the quantum-mechanical perspective, where a single unique steady state is allowed in the whole parameter space. Here, we measure the non-equilibrium dynamics of a superconducting Duffing oscillator and reconcile the classical and quantum descriptions in a unified picture of quantum metastability. We demonstrate that the two classically regarded steady states are in fact metastable states. They have a remarkably long lifetime in the classical hysteresis regime but must eventually relax into a single unique steady state allowed by quantum mechanics. By engineering the lifetime of the metastable states sufficiently large, we observe a first-order dissipative phase transition, which mimics a sudden change of the mean field in a 11-site Bose-Hubbard lattice. We also reveal the two distinct phases of the transition by quantum state tomography, namely a coherent-state phase and a squeezed-state phase separated by a critical point. Our results reveal a smooth quantum state evolution behind a sudden dissipative phase transition, and they form an essential step towards understanding hysteresis and instability in non-equilibrium systems.

There currently exist two main approaches to reproducing visual appearance using Machine Learning (ML): The first is training models that generalize over different instances of a problem, e.g., different images of a dataset. As one-shot approaches, these offer fast inference, but often fall short in quality. The second approach does not train models that generalize across tasks, but rather over-fit a single instance of a problem, e.g., a flash image of a material. These methods offer high quality, but take long to train. We suggest to combine both techniques end-to-end using meta-learning: We over-fit onto a single problem instance in an inner loop, while also learning how to do so efficiently in an outer-loop across many exemplars. To this end, we derive the required formalism that allows applying meta-learning to a wide range of visual appearance reproduction problems: textures, BRDFs, svBRDFs, illumination or the entire light transport of a scene. The effects of meta-learning parameters on several different aspects of visual appearance are analyzed in our framework, and specific guidance for different tasks is provided. Metappearance enables visual quality that is similar to over-fit approaches in only a fraction of their runtime while keeping the adaptivity of general models., Comment: Accepted as a journal paper to SIGGRAPH Asia 2022. Also see the project page at https://mfischer-ucl.github.io/metappearance

Fully- or semi-automatic image enhancement software helps users to increase the visual appeal of photos and does not require in-depth knowledge of manual image editing. However, fully-automatic approaches usually enhance the image in a black-box manner that does not give the user any control over the optimization process, possibly leading to edited images that do not subjectively appeal to the user. Semi-automatic methods mostly allow for controlling which pre-defined editing step is taken, which restricts the users in their creativity and ability to make detailed adjustments, such as brightness or contrast. We argue that incorporating user preferences by guiding an automated enhancement method simplifies image editing and increases the enhancement's focus on the user. This work thus proposes the Neural Image Correction & Enhancement Routine (NICER), a neural network based approach to no-reference image enhancement in a fully-, semi-automatic or fully manual process that is interactive and user-centered. NICER iteratively adjusts image editing parameters in order to maximize an aesthetic score based on image style and content. Users can modify these parameters at any time and guide the optimization process towards a desired direction. This interactive workflow is a novelty in the field of human-computer interaction for image enhancement tasks. In a user study, we show that NICER can improve image aesthetics without user interaction and that allowing user interaction leads to diverse enhancement outcomes that are strongly preferred over the unedited image. We make our code publicly available to facilitate further research in this direction., Comment: The code can be found at https://github.com/mr-Mojo/NICER

We have fabricated and studied a system of two tunable and coupled nonlinear superconducting resonators. The nonlinearity is introduced by galvanically coupled dc-SQUIDs. We simulate the system response by means of a circuit model, which includes an additional signal path introduced by the electromagnetic environment. Furthermore, we present two methods allowing us to experimentally determine the nonlinearity. First, we fit the measured frequency and flux dependence of the transmission data to simulations based on the equivalent circuit model. Second, we fit the power dependence of the transmission data to a model that is predicted by the nonlinear equation of motion describing the system. Our results show that we are able to tune the nonlinearity of the resonators by almost two orders of magnitude via an external coil and two on-chip antennas. The studied system represents the basic building block for larger systems, allowing for quantum simulations of bosonic many-body systems with a larger number of lattice sites.

While Alexa can perform over 100,000 skills on paper, its capability covers only a fraction of what is possible on the web. To reach the full potential of an assistant, it is desirable that individuals can create skills to automate their personal web browsing routines. Many seemingly simple routines, however, such as monitoring COVID-19 stats for their hometown, detecting changes in their child's grades online, or sending personally-addressed messages to a group, cannot be automated without conventional programming concepts such as conditional and iterative evaluation. This paper presents VASH (Voice Assistant Scripting Helper), a new system that empowers users to create useful web-based virtual assistant skills without learning a formal programming language. With VASH, the user demonstrates their task of interest in the browser and issues a few voice commands, such as naming the skills and adding conditions on the action. VASH turns these multi-modal specifications into skills that can be invoked invoice on a virtual assistant. These skills are represented in a formal programming language we designed called WebTalk, which supports parameterization, function invocation, conditionals, and iterative execution. VASH is a fully working prototype that works on the Chrome browser on real-world websites. Our user study shows that users have many web routines they wish to automate, 81% of which can be expressed using VASH. We found that VASH Is easy to learn, and that a majority of the users in our study want to use our system.

This paper presents ImagineNet, a tool that uses a novel neural style transfer model to enable end-users and app developers to restyle GUIs using an image of their choice. Former neural style transfer techniques are inadequate for this application because they produce GUIs that are illegible and hence nonfunctional. We propose a neural solution by adding a new loss term to the original formulation, which minimizes the squared error in the uncentered cross-covariance of features from different levels in a CNN between the style and output images. ImagineNet retains the details of GUIs, while transferring the colors and textures of the art. We presented GUIs restyled with ImagineNet as well as other style transfer techniques to 50 evaluators and all preferred those of ImagineNet. We show how ImagineNet can be used to restyle (1) the graphical assets of an app, (2) an app with user-supplied content, and (3) an app with dynamically generated GUIs.

Experiments with pedestrians revealed that the geometry of the domain, as well as the incentive of pedestrians to reach a target as fast as possible have a strong influence on the overall dynamics. In this paper, we propose and validate different mathematical models at the micro- and macroscopic levels to study the influence of both effects. We calibrate the models with experimental data and compare the results at the micro -- as well as macroscopic levels. Our numerical simulations reproduce qualitative experimental features on both levels, and indicate how geometry and motivation level influence the observed pedestrian density. Furthermore, we discuss the dynamics of solutions for different modeling approaches and comment on the analysis of the respective equations., Comment: 22 pages, 14 figures