Search

We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code and weights at https://github.com/apple/ml-depth-pro, Comment: Code and weights available at https://github.com/apple/ml-depth-pro

Reconstructing an object from photos and placing it virtually in a new environment goes beyond the standard novel view synthesis task as the appearance of the object has to not only adapt to the novel viewpoint but also to the new lighting conditions and yet evaluations of inverse rendering methods rely on novel view synthesis data or simplistic synthetic datasets for quantitative analysis. This work presents a real-world dataset for measuring the reconstruction and rendering of objects for relighting. To this end, we capture the environment lighting and ground truth images of the same objects in multiple environments allowing to reconstruct the objects from images taken in one environment and quantify the quality of the rendered views for the unseen lighting environments. Further, we introduce a simple baseline composed of off-the-shelf methods and test several state-of-the-art methods on the relighting task and show that novel view synthesis is not a reliable proxy to measure performance. Code and dataset are available at https://github.com/isl-org/objects-with-lighting ., Comment: Accepted at 3DV 2024, Oral presentation. For the project page see https://github.com/isl-org/objects-with-lighting

Semantic segmentation models struggle to generalize in the presence of domain shift. In this paper, we introduce contrastive learning for feature alignment in cross-domain adaptation. We assemble both in-domain contrastive pairs and cross-domain contrastive pairs to learn discriminative features that align across domains. Based on the resulting well-aligned feature representations we introduce a label expansion approach that is able to discover samples from hard classes during the adaptation process to further boost performance. The proposed approach consistently outperforms state-of-the-art methods for domain adaptation. It achieves 60.2% mIoU on the Cityscapes dataset when training on the synthetic GTA5 dataset together with unlabeled Cityscapes images.

Imaging in low light is extremely challenging due to low photon counts. Using sensitive CMOS cameras, it is currently possible to take videos at night under moonlight (0.05-0.3 lux illumination). In this paper, we demonstrate photorealistic video under starlight (no moon present, $<$0.001 lux) for the first time. To enable this, we develop a GAN-tuned physics-based noise model to more accurately represent camera noise at the lowest light levels. Using this noise model, we train a video denoiser using a combination of simulated noisy video clips and real noisy still images. We capture a 5-10 fps video dataset with significant motion at approximately 0.6-0.7 millilux with no active illumination. Comparing against alternative methods, we achieve improved video quality at the lowest light levels, demonstrating photorealistic video denoising in starlight for the first time., Comment: CVPR 2022. Project page: https://kristinamonakhova.com/starlight_denoising/

Geometric Deep Learning has recently made striking progress with the advent of continuous deep implicit fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is unlimited in resolution. Unfortunately, these methods are often unsuitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Implicit Fields. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define DeepMesh - an end-to-end differentiable mesh representation that can vary its topology. We validate our theoretical insight through several applications: Single view 3D Reconstruction via Differentiable Rendering, Physically-Driven Shape Optimization, Full Scene 3D Reconstruction from Scans and End-to-End Training. In all cases our end-to-end differentiable parameterization gives us an edge over state-of-the-art algorithms., Comment: arXiv admin note: substantial text overlap with arXiv:2006.03997

We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines., Comment: Code and data available at https://github.com/intel-isl/PhotorealismEnhancement Video available at https://youtu.be/P1IcaBn3ej0

Geometric Deep Learning has recently made striking progress with the advent of continuous Deep Implicit Fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is not limited in resolution. Unfortunately, these methods are often not suitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Signed Distance Functions. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define MeshSDF, an end-to-end differentiable mesh representation which can vary its topology. We use two different applications to validate our theoretical insight: Single-View Reconstruction via Differentiable Rendering and Physically-Driven Shape Optimization. In both cases our differentiable parameterization gives us an edge over state-of-the-art algorithms., Comment: 22 pages, 16 figures, Neural Information Processing Systems (NeurIPS 2020)

Convolutional networks for single-view object reconstruction have shown impressive performance and have become a popular subject of research. All existing techniques are united by the idea of having an encoder-decoder network that performs non-trivial reasoning about the 3D structure of the output space. In this work, we set up two alternative approaches that perform image classification and retrieval respectively. These simple baselines yield better results than state-of-the-art methods, both qualitatively and quantitatively. We show that encoder-decoder methods are statistically indistinguishable from these baselines, thus indicating that the current state of the art in single-view object reconstruction does not actually perform reconstruction but image classification. We identify aspects of popular experimental procedures that elicit this behavior and discuss ways to improve the current state of research.

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling., Comment: Published at the Conference on Computer Vision and Pattern Recognition (CVPR 2018)

We present a benchmark suite for visual perception. The benchmark is based on more than 250K high-resolution video frames, all annotated with ground-truth data for both low-level and high-level vision tasks, including optical flow, semantic instance segmentation, object detection and tracking, object-level 3D scene layout, and visual odometry. Ground-truth data for all tasks is available for every frame. The data was collected while driving, riding, and walking a total of 184 kilometers in diverse ambient conditions in a realistic virtual world. To create the benchmark, we have developed a new approach to collecting ground-truth data from simulated worlds without access to their source code or content. We conduct statistical analyses that show that the composition of the scenes in the benchmark closely matches the composition of corresponding physical environments. The realism of the collected data is further validated via perceptual experiments. We analyze the performance of state-of-the-art methods for multiple tasks, providing reference baselines and highlighting challenges for future research. The supplementary video can be viewed at https://youtu.be/T9OybWv923Y, Comment: Published at the International Conference on Computer Vision (ICCV 2017)

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set., Comment: Accepted to the 14th European Conference on Computer Vision (ECCV 2016)

In current models for jets of AGNs and their emission a shortcoming in the description and understanding of the connection between the largest and smallest scales exists. In this work we present a spatially resolved SSC model extended to parsec scales, which opens the possibility of probing the connections between the radio and high energy properties. We simulate an environment that leads to Fermi-I acceleration of leptonic particles and includes the full time dependence of this process. Omitting the restriction of a finite downstream region, we find that the spectral energy distribution (SED) produced by the accelerated particles strongly depends on their radial confinement behind the shock. The requirement, for both the restriction of high energy emission to a small region around the shock and the production of a flat radio spectrum, is an initial linear increase of the radius immediately behind the shock, which then slows down with increasing distance from the shock. A good representation of the data for the Blazar \textit{Mkn501} is achieved by a parameterized log-function. The prediction for the shape of the radio blob is given by the flux distribution with respect to shock distance., Comment: accepted for publication in ApJ, 10 pages, 5 figures

Molecular dynamics simulations yield large amounts of trajectory data. For their durable storage and accessibility an efficient compression algorithm is paramount. State of the art domain-specific algorithms combine quantization, Huffman encoding and occasionally domain knowledge. We propose the high resolution trajectory compression scheme (HRTC) that relies on piecewise linear functions to approximate quantized trajectories. By splitting the error budget between quantization and approximation, our approach beats the current state of the art by several orders of magnitude given the same error tolerance. It allows storing samples at far less than one bit per sample. It is simple and fast enough to be integrated into the inner simulation loop, store every time step, and become the primary representation of trajectory data.

Very fast variability on scales of minutes is regularly observed in Blazars. The assumption that these flares are emerging from the dominant emission zone of the very high energy (VHE) radiation within the jet challenges current acceleration and radiation models. In this work we use a spatially resolved and time dependent synchrotron-self-Compton (SSC) model that includes the full time dependence of Fermi-I acceleration. We use the (apparent) orphan $\gamma$-ray flare of \textit{Mrk501} during MJD 54952 and test various flare scenarios against the observed data. We find that a rapidly variable external radiation field can reproduce the high energy lightcurve best. However, the effect of the strong inverse Compton (IC) cooling on other bands and the X-ray observations are constraining the parameters to rather extreme ranges. Then again other scenarios would require parameters even more extreme or stronger physical constraints on the rise and decay of the source of the variability which might be in contradiction with constraints derived from the size of the black hole's ergosphere., Comment: accepted for publication in Astroparticle Physics

One of the main, unresolved questions about the nature of quasars is the position of the acceleration site responsible for the highest energies. The attempt to investigate this question in the energy regime with the highest resolution, the radio band, has the downside that no theoretical model exists that can connect these two regimes. The model in this work tries to shrink this gap by extending the general synchrotron self Compton (SSC) model up to length scales in the order of the resolution of radio observations. The resulting spectral energy distributions (SED) show a qualitative improvement in the representation of the radio spectrum. Furthermore the obtained emission morphology shows similar properties to the radio structures observed in jets of quasars. A complete and quantitative connection will however need either much higher numerical effort or an improved methodology.

In this paper a spatially resolved, fully self-consistent SSC model is presented. The observable spectral energy distribution (SED) evolves entirely from a low energetic delta distribution of injected electrons by means of the implemented microphysics of the jet. These are in particular the properties of the shock and the ambient plasma, which can be varied along the jet axis. Hence a large variety of scenarios can be computed, e.g. the acceleration of particles via multiple shocks. Two acceleration processes, shock acceleration and stochastic acceleration, are taken into account. From the resulting electron distribution the SED is calculated taking into account synchrotron radiation, inverse Compton scattering (full cross section) and synchrotron self absorption. The model can explain SEDs where cooling processes are crucial. It can verify high variability results from acausal simulations and produce variability not only via injection of particles, but due to the presence of multiple shocks. Furthermore a fit of the data, obtained in the 2010 multi-frequency campaign of Mrk501, is presented., Comment: 4 pages, 2 figures; appeared in the proceedings of the conference: "High Energy Phenomena in Relativistic Outflows (HEPRO) III", held in Barcelona, Spain, June 27th-July 1st 2011 (IJMPCS)