Search

Pattern images are everywhere in the digital and physical worlds, and tools to edit them are valuable. But editing pattern images is tricky: desired edits are often programmatic: structure-aware edits that alter the underlying program which generates the pattern. One could attempt to infer this underlying program, but current methods for doing so struggle with complex images and produce unorganized programs that make editing tedious. In this work, we introduce a novel approach to perform programmatic edits on pattern images. By using a pattern analogy -- a pair of simple patterns to demonstrate the intended edit -- and a learning-based generative model to execute these edits, our method allows users to intuitively edit patterns. To enable this paradigm, we introduce SplitWeave, a domain-specific language that, combined with a framework for sampling synthetic pattern analogies, enables the creation of a large, high-quality synthetic training dataset. We also present TriFuser, a Latent Diffusion Model (LDM) designed to overcome critical issues that arise when naively deploying LDMs to this task. Extensive experiments on real-world, artist-sourced patterns reveals that our method faithfully performs the demonstrated edit while also generalizing to related pattern styles beyond its training distribution., Comment: Website: https://bardofcodes.github.io/patterns/

The recent discovery of magic angle twisted bilayer graphene (MATBG), in which two sheets of monolayer graphene are precisely stacked to a specific angle, has opened up a plethora of new opportunities in the field of topology, superconductivity, and other strongly correlated effects. Most conventional ways of preparing twisted bilayer devices require the use of high process temperatures and solvents and are not well-suited for preparing samples which need to be flipped to be compatible with characterization techniques like STM, ARPES, PFM, SThM etc. Here, we demonstrate a very simple polymer-based method using Polyvinyl Chloride (PVC), which can be used for making flipped twisted bilayer graphene devices. This allowed us to produce flipped twisted samples without the need of any solvents and with high quality as confirmed by Piezoresponse Force Microscopy. We believe that this dry flip technique can be readily extended to twist 2D materials beyond graphene, especially air-sensitive materials which require operation under inert atmosphere, where often solvents cannot be used.

Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words, Comment: Project Page: https://ttchengab.github.io/continuous_3d_words

We present 3DMiner -- a pipeline for mining 3D shapes from challenging large-scale unannotated image datasets. Unlike other unsupervised 3D reconstruction methods, we assume that, within a large-enough dataset, there must exist images of objects with similar shapes but varying backgrounds, textures, and viewpoints. Our approach leverages the recent advances in learning self-supervised image representations to cluster images with geometrically similar shapes and find common image correspondences between them. We then exploit these correspondences to obtain rough camera estimates as initialization for bundle-adjustment. Finally, for every image cluster, we apply a progressive bundle-adjusting reconstruction method to learn a neural occupancy field representing the underlying shape. We show that this procedure is robust to several types of errors introduced in previous steps (e.g., wrong camera poses, images containing dissimilar shapes, etc.), allowing us to obtain shape and pose annotations for images in-the-wild. When using images from Pix3D chairs, our method is capable of producing significantly better results than state-of-the-art unsupervised 3D reconstruction techniques, both quantitatively and qualitatively. Furthermore, we show how 3DMiner can be applied to in-the-wild data by reconstructing shapes present in images from the LAION-5B dataset. Project Page: https://ttchengab.github.io/3dminerOfficial, Comment: In ICCV 2023

Detecting light at the single-photon level is one of the pillars of emergent photonic technologies. This is realized through state-of-the-art superconducting detectors that offer efficient, broadband and fast response. However, the use of superconducting thin films with low TC limits their operation temperature below 4K. In this work, we demonstrate proof-of-concept nanodetectors based on exfoliated, two-dimensional cuprate superconductor Bi2Sr2CaCu2O8-{\delta} (BSCCO) that exhibit single-photon sensitivity at telecom wavelength at a record temperature of T = 20K. These non-optimized devices exhibit a slow (ms) reset time and a low detection efficiency (10^(-4)). We realize the elusive prospect of single-photon sensitivity on a high-TC nanodetector thanks to a novel approach, combining van der Waals fabrication techniques and a non-invasive nanopatterning based on light ion irradiation. This result paves the way for broader application of single-photon technologies, relaxing the cryogenic constraints for single-photon detection at telecom wavelength.

Procedural material graphs are a compact, parameteric, and resolution-independent representation that are a popular choice for material authoring. However, designing procedural materials requires significant expertise and publicly accessible libraries contain only a few thousand such graphs. We present MatFormer, a generative model that can produce a diverse set of high-quality procedural materials with complex spatial patterns and appearance. While procedural materials can be modeled as directed (operation) graphs, they contain arbitrary numbers of heterogeneous nodes with unstructured, often long-range node connections, and functional constraints on node parameters and connections. MatFormer addresses these challenges with a multi-stage transformer-based model that sequentially generates nodes, node parameters, and edges, while ensuring the semantic validity of the graph. In addition to generation, MatFormer can be used for the auto-completion and exploration of partial material graphs. We qualitatively and quantitatively demonstrate that our method outperforms alternative approaches, in both generated graph and material quality.

We present a fine-tuning method to improve the appearance of 3D geometries reconstructed from single images. We leverage advances in monocular depth estimation to obtain disparity maps and present a novel approach to transforming 2D normalized disparity maps into 3D point clouds by using shape priors to solve an optimization on the relevant camera parameters. After creating a 3D point cloud from disparity, we introduce a method to combine the new point cloud with existing information to form a more faithful and detailed final geometry. We demonstrate the efficacy of our approach with multiple experiments on both synthetic and real images.

We present ANISE, a method that reconstructs a 3D~shape from partial observations (images or sparse point clouds) using a part-aware neural implicit shape representation. The shape is formulated as an assembly of neural implicit functions, each representing a different part instance. In contrast to previous approaches, the prediction of this representation proceeds in a coarse-to-fine manner. Our model first reconstructs a structural arrangement of the shape in the form of geometric transformations of its part instances. Conditioned on them, the model predicts part latent codes encoding their surface geometry. Reconstructions can be obtained in two ways: (i) by directly decoding the part latent codes to part implicit functions, then combining them into the final shape; or (ii) by using part latents to retrieve similar part instances in a part database and assembling them in a single shape. We demonstrate that, when performing reconstruction by decoding part representations into implicit functions, our method achieves state-of-the-art part-aware reconstruction results from both images and sparse point clouds.When reconstructing shapes by assembling parts retrieved from a dataset, our approach significantly outperforms traditional shape retrieval methods even when significantly restricting the database size. We present our results in well-known sparse point cloud reconstruction and single-view reconstruction benchmarks.

Mechanical properties of the tissue engineering scaffolds are known to play a crucial role in tissue regeneration. Here, we have utilized discrete network and finite element models to study fibrous scaffold mechanics and its dependence on structure. We have considered two loading conditions, first, uniaxial elongation (macroscopic), and second, localized cellular forces (microscopic). Using these two scenarios, we have tried to establish a link between scaffold micromechanics and its macroscopic mechanical properties. We have demonstrated that the macroscopic elastic modulus of fibrous scaffold is dependent on the sample shape, size and the degree of fiber fusion. Under microscopic loading conditions the deformation of the fibrous scaffolds is anisotropic with an orientation dependent decay with distance. Further, we explored the stiffness sensing under the conditions of fixed stress or fixed strain application by the cells. With fixed strain, we found that the stiffness sensed by a cell is proportional to scaffold's macroscopic Young's modulus. However, for fixed stress application, it also depends on cell's elasticity with stiffness experienced by stiff and soft cells differing by an order of magnitude. The analyses demonstrate that there exists a gap between mechanical properties of the fibrous scaffolds as measured from the macroscopic testings and those sensed by the cells. This warrants a need for extreme care during the designing and reporting of experiments involving cell-scaffold interactions. The insights from this work will help in the designing of tissue engineering scaffolds for applications where mechanical stimuli are a critical factor

Representing human-made objects as a collection of base primitives has a long history in computer vision and reverse engineering. In the case of high-resolution point cloud scans, the challenge is to be able to detect both large primitives as well as those explaining the detailed parts. While the classical RANSAC approach requires case-specific parameter tuning, state-of-the-art networks are limited by memory consumption of their backbone modules such as PointNet++, and hence fail to detect the fine-scale primitives. We present Cascaded Primitive Fitting Networks (CPFN) that relies on an adaptive patch sampling network to assemble detection results of global and local primitive detection networks. As a key enabler, we present a merging formulation that dynamically aggregates the primitives across global and local scales. Our evaluation demonstrates that CPFN improves the state-of-the-art SPFN performance by 13-14% on high-resolution point cloud datasets and specifically improves the detection of fine-scale primitives by 20-22%., Comment: ICCV 2021: 15 pages, 8 figures

We propose DeepMetaHandles, a 3D conditional generative model based on mesh deformation. Given a collection of 3D meshes of a category and their deformation handles (control points), our method learns a set of meta-handles for each shape, which are represented as combinations of the given handles. The disentangled meta-handles factorize all the plausible deformations of the shape, while each of them corresponds to an intuitive deformation. A new deformation can then be generated by sampling the coefficients of the meta-handles in a specific range. We employ biharmonic coordinates as the deformation function, which can smoothly propagate the control points' translations to the entire mesh. To avoid learning zero deformation as meta-handles, we incorporate a target-fitting module which deforms the input mesh to match a random target. To enhance deformations' plausibility, we employ a soft-rasterizer-based discriminator that projects the meshes to a 2D space. Our experiments demonstrate the superiority of the generated deformations as well as the interpretability and consistency of the learned meta-handles., Comment: CVPR2021 oral

Advances in imaging techniques enable high resolution 3D visualisation of vascular networks over time and reveal abnormal structural features such as twists and loops, and their quantification is an active area of research. Here we showcase how topological data analysis (TDA), the mathematical field that studies `shape' of data, can characterise the geometric, spatial and temporal organisation of vascular networks. We propose two topological lenses to study vasculature, which capture inherent multi-scale features and vessel connectivity, and surpass the single scale analysis of existing methods. We analyse images collected using intravital and ultramicroscopy modalities and quantify spatio-temporal variation of twists, loops, and avascular regions (voids) in 3D vascular networks. This topological approach validates and quantifies known qualitative trends such as dynamic changes in tortuosity and loops in response to antibodies that modulate vessel sprouting; furthermore, it quantifies the effect of radiotherapy on vessel architecture.

We introduce a modeling tool which can evolve a set of 3D objects in a functionality-aware manner. Our goal is for the evolution to generate large and diverse sets of plausible 3D objects for data augmentation, constrained modeling, as well as open-ended exploration to possibly inspire new designs. Starting with an initial population of 3D objects belonging to one or more functional categories, we evolve the shapes through part recombination to produce generations of hybrids or crossbreeds between parents from the heterogeneous shape collection. Evolutionary selection of offsprings is guided both by a functional plausibility score derived from functionality analysis of shapes in the initial population and user preference, as in a design gallery. Since cross-category hybridization may result in offsprings not belonging to any of the known functional categories, we develop a means for functionality partial matching to evaluate functional plausibility on partial shapes. We show a variety of plausible hybrid shapes generated by our functionality-aware model evolution, which can complement existing datasets as training data and boost the performance of contemporary data-driven segmentation schemes, especially in challenging cases. Our tool supports constrained modeling, allowing users to restrict or steer the model evolution with functionality labels. At the same time, unexpected yet functional object prototypes can emerge during open-ended exploration owing to structure breaking when evolving a heterogeneous collection.

We present a generative model to synthesize 3D shapes as sets of handles -- lightweight proxies that approximate the original 3D shape -- for applications in interactive editing, shape parsing, and building compact 3D representations. Our model can generate handle sets with varying cardinality and different types of handles (Figure 1). Key to our approach is a deep architecture that predicts both the parameters and existence of shape handles, and a novel similarity measure that can easily accommodate different types of handles, such as cuboids or sphere-meshes. We leverage the recent advances in semantic 3D annotation as well as automatic shape summarizing techniques to supervise our approach. We show that the resulting shape representations are intuitive and achieve superior quality than previous state-of-the-art. Finally, we demonstrate how our method can be used in applications such as interactive shape editing, completion, and interpolation, leveraging the latent space learned by our model to guide these tasks. Project page: http://mgadelha.me/shapehandles., Comment: 11 pages, 11 figures, accepted do CVPR 2020

We propose a novel, end-to-end trainable, deep network called ParSeNet that decomposes a 3D point cloud into parametric surface patches, including B-spline patches as well as basic geometric primitives. ParSeNet is trained on a large-scale dataset of man-made 3D shapes and captures high-level semantic priors for shape decomposition. It handles a much richer class of primitives than prior work, and allows us to represent surfaces with higher fidelity. It also produces repeatable and robust parametrizations of a surface compared to purely geometric approaches. We present extensive experiments to validate our approach against analytical and learning-based alternatives. Our source code is publicly available at: https://hippogriff.github.io/parsenet.

We present a novel reinforcement learning-based natural media painting algorithm. Our goal is to reproduce a reference image using brush strokes and we encode the objective through observations. Our formulation takes into account that the distribution of the reward in the action space is sparse and training a reinforcement learning algorithm from scratch can be difficult. We present an approach that combines self-supervised learning and reinforcement learning to effectively transfer negative samples into positive ones and change the reward distribution. We demonstrate the benefits of our painting agent to reproduce reference images with brush strokes. The training phase takes about one hour and the runtime algorithm takes about 30 seconds on a GTX1080 GPU reproducing a 1000x800 image with 20,000 strokes.

Reconstructing 3D shapes from single-view images has been a long-standing research problem. In this paper, we present DISN, a Deep Implicit Surface Network which can generate a high-quality detail-rich 3D mesh from an 2D image by predicting the underlying signed distance fields. In addition to utilizing global image features, DISN predicts the projected location for each 3D point on the 2D image, and extracts local features from the image feature maps. Combining global and local features significantly improves the accuracy of the signed distance field prediction, especially for the detail-rich areas. To the best of our knowledge, DISN is the first method that constantly captures details such as holes and thin structures present in 3D shapes from single-view images. DISN achieves the state-of-the-art single-view reconstruction performance on a variety of shape categories reconstructed from both synthetic and real images. Code is available at https://github.com/xharlie/DISN The supplementary can be found at https://xharlie.github.io/images/neurips_2019_supp.pdf, Comment: This project was in part supported by the gift funding to the University of Southern California from Adobe Research

Applications in virtual and augmented reality create a demand for rapid creation and easy access to large sets of 3D models. An effective way to address this demand is to edit or deform existing 3D models based on a reference, e.g., a 2D image which is very easy to acquire. Given such a source 3D model and a target which can be a 2D image, 3D model, or a point cloud acquired as a depth scan, we introduce 3DN, an end-to-end network that deforms the source model to resemble the target. Our method infers per-vertex offset displacements while keeping the mesh connectivity of the source model fixed. We present a training strategy which uses a novel differentiable operation, mesh sampling operator, to generalize our method across source and target models with varying mesh densities. Mesh sampling operator can be seamlessly integrated into the network to handle meshes with different topologies. Qualitative and quantitative results show that our method generates higher quality results compared to the state-of-the art learning-based methods for 3D shape generation. Code is available at github.com/laughtervv/3DN.

Edges, boundaries and contours are important subjects of study in both computer graphics and computer vision. On one hand, they are the 2D elements that convey 3D shapes, on the other hand, they are indicative of occlusion events and thus separation of objects or semantic concepts. In this paper, we aim to generate contour drawings, boundary-like drawings that capture the outline of the visual scene. Prior art often cast this problem as boundary detection. However, the set of visual cues presented in the boundary detection output are different from the ones in contour drawings, and also the artistic style is ignored. We address these issues by collecting a new dataset of contour drawings and proposing a learning-based method that resolves diversity in the annotation and, unlike boundary detectors, can work with imperfect alignment of the annotation and the actual ground truth. Our method surpasses previous methods quantitatively and qualitatively. Surprisingly, when our model fine-tunes on BSDS500, we achieve the state-of-the-art performance in salient boundary detection, suggesting contour drawing might be a scalable alternative to boundary annotation, which at the same time is easier and more interesting for annotators to draw., Comment: WACV 2019. For code and dataset, see http://www.cs.cmu.edu/~mengtial/proj/sketch/

An interesting interplay between two different modifiers and the surface of titanium dioxide leads to a significant change in photoelectrochemical properties of the designed hybrid materials. The semiconductor is photosensitized by one of the counterparts and exhibits the photoelectrochemical photocurrent switching effect thanks to interactions with graphene oxide - the second modifier mediates charge transfer processes in the system, allowing us to design the materials response at the molecular level. Based on the selection of molecular counterpart we may affect the behaviour of hybrids upon light irradiation in a different manner, which may be useful for the applications in photovoltaics, optoelectronics and photocatalysis. Here we focus particularly on the nanocomposites made of titanium dioxide with graphene oxide combined with either 2,3,5,6-tetrachlorobenzoquinone or 2,3-dichloro-5,6-dihydroxybenzoquinone - for these two materials we observed a major change in the charge transfer processes occurring in the system.

Automatic organization of personal photos is a problem with many real world ap- plications, and can be divided into two main tasks: recognizing the event type of the photo collection, and selecting interesting images from the collection. In this paper, we attempt to simultaneously solve both tasks: album-wise event recognition and image- wise importance prediction. We collected an album dataset with both event type labels and image importance labels, refined from an existing CUFED dataset. We propose a hybrid system consisting of three parts: A siamese network-based event-specific image importance prediction, a Convolutional Neural Network (CNN) that recognizes the event type, and a Long Short-Term Memory (LSTM)-based sequence level event recognizer. We propose an iterative updating procedure for event type and image importance score prediction. We experimentally verified that image importance score prediction and event type recognition can each help the performance of the other., Comment: Accepted as oral in BMVC 2017

In this paper, we introduce the problem of simultaneously detecting multiple photographic defects. We aim at detecting the existence, severity, and potential locations of common photographic defects related to color, noise, blur and composition. The automatic detection of such defects could be used to provide users with suggestions for how to improve photos without the need to laboriously try various correction methods. Defect detection could also help users select photos of higher quality while filtering out those with severe defects in photo curation and summarization. To investigate this problem, we collected a large-scale dataset of user annotations on seven common photographic defects, which allows us to evaluate algorithms by measuring their consistency with human judgments. Our new dataset enables us to formulate the problem as a multi-task learning problem and train a multi-column deep convolutional neural network (CNN) to simultaneously predict the severity of all the defects. Unlike some existing single-defect estimation methods that rely on low-level statistics and may fail in many cases on natural photographs, our model is able to understand image contents and quality at a higher level. As a result, in our experiments, we show that our model has predictions with much higher consistency with human judgments than low-level methods as well as several baseline CNN models. Our model also performs better than an average human from our user study., Comment: Accepted to WACV'18

We study the problem of Salient Object Subitizing, i.e. predicting the existence and the number of salient objects in an image using holistic cues. This task is inspired by the ability of people to quickly and accurately identify the number of items within the subitizing range (1-4). To this end, we present a salient object subitizing image dataset of about 14K everyday images which are annotated using an online crowdsourcing marketplace. We show that using an end-to-end trained Convolutional Neural Network (CNN) model, we achieve prediction accuracy comparable to human performance in identifying images with zero or one salient object. For images with multiple salient objects, our model also provides significantly better than chance performance without requiring any localization process. Moreover, we propose a method to improve the training of the CNN subitizing model by leveraging synthetic images. In experiments, we demonstrate the accuracy and generalizability of our CNN subitizing model and its applications in salient object detection and image retrieval.

Real-world applications could benefit from the ability to automatically generate a fine-grained ranking of photo aesthetics. However, previous methods for image aesthetics analysis have primarily focused on the coarse, binary categorization of images into high- or low-aesthetic categories. In this work, we propose to learn a deep convolutional neural network to rank photo aesthetics in which the relative ranking of photo aesthetics are directly modeled in the loss function. Our model incorporates joint learning of meaningful photographic attributes and image content information which can help regularize the complicated photo aesthetics rating problem. To train and analyze this model, we have assembled a new aesthetics and attributes database (AADB) which contains aesthetic scores and meaningful attributes assigned to each image by multiple human raters. Anonymized rater identities are recorded across images allowing us to exploit intra-rater consistency using a novel sampling strategy when computing the ranking loss of training image pairs. We show the proposed sampling strategy is very effective and robust in face of subjective judgement of image aesthetics by individuals with different aesthetic tastes. Experiments demonstrate that our unified model can generate aesthetic rankings that are more consistent with human ratings. To further validate our model, we show that by simply thresholding the estimated aesthetic scores, we are able to achieve state-or-the-art classification performance on the existing AVA dataset benchmark.

We present a novel algorithm for calculating the discrete fractional Hadamard transform for data vectors whose size N is a power of two. A direct method for calculation of the discrete fractional Hadamard transform requires $N^2$ multiplications, while in proposed algorithm the number of real multiplications is reduced to $N$log$_2N$., Comment: 22 pages, 4 figures

The violation of a Bell inequality is an experimental observation that forces one to abandon a local realistic worldview, namely, one in which physical properties are (probabilistically) defined prior to and independent of measurement and no physical influence can propagate faster than the speed of light. All such experimental violations require additional assumptions depending on their specific construction making them vulnerable to so-called "loopholes." Here, we use photons and high-efficiency superconducting detectors to violate a Bell inequality closing the fair-sampling loophole, i.e. without assuming that the sample of measured photons accurately represents the entire ensemble. Additionally, we demonstrate that our setup can realize one-sided device-independent quantum key distribution on both sides. This represents a significant advance relevant to both fundamental tests and promising quantum applications.

Single photons are an important prerequisite for a broad spectrum of quantum optical applications. We experimentally demonstrate a heralded single-photon source based on spontaneous parametric down-conversion in collinear bulk optics, and fiber-coupled bolometric transition-edge sensors. Without correcting for background, losses, or detection inefficiencies, we measure an overall heralding efficiency of 83 %. By violating a Bell inequality, we confirm the single-photon character and high-quality entanglement of our heralded single photons which, in combination with the high heralding efficiency, are a necessary ingredient for advanced quantum communication protocols such as one-sided device-independent quantum key distribution., Comment: 8 pages, 3 figures

Quantum teleportation [1] is a quintessential prerequisite of many quantum information processing protocols [2-4]. By using quantum teleportation, one can circumvent the no-cloning theorem [5] and faithfully transfer unknown quantum states to a party whose location is even unknown over arbitrary distances. Ever since the first experimental demonstrations of quantum teleportation of independent qubits [6] and of squeezed states [7], researchers have progressively extended the communication distance in teleportation, usually without active feed-forward of the classical Bell-state measurement result which is an essential ingredient in future applications such as communication between quantum computers. Here we report the first long-distance quantum teleportation experiment with active feed-forward in real time. The experiment employed two optical links, quantum and classical, over 143 km free space between the two Canary Islands of La Palma and Tenerife. To achieve this, the experiment had to employ novel techniques such as a frequency-uncorrelated polarization-entangled photon pair source, ultra-low-noise single-photon detectors, and entanglement-assisted clock synchronization. The average teleported state fidelity was well beyond the classical limit of 2/3. Furthermore, we confirmed the quality of the quantum teleportation procedure (without feed-forward) by complete quantum process tomography. Our experiment confirms the maturity and applicability of the involved technologies in real-world scenarios, and is a milestone towards future satellite-based quantum teleportation.