Search

We propose OmniNOCS, a large-scale monocular dataset with 3D Normalized Object Coordinate Space (NOCS) maps, object masks, and 3D bounding box annotations for indoor and outdoor scenes. OmniNOCS has 20 times more object classes and 200 times more instances than existing NOCS datasets (NOCS-Real275, Wild6D). We use OmniNOCS to train a novel, transformer-based monocular NOCS prediction model (NOCSformer) that can predict accurate NOCS, instance masks and poses from 2D object detections across diverse classes. It is the first NOCS model that can generalize to a broad range of classes when prompted with 2D boxes. We evaluate our model on the task of 3D oriented bounding box prediction, where it achieves comparable results to state-of-the-art 3D detection methods such as Cube R-CNN. Unlike other 3D detection methods, our model also provides detailed and accurate 3D object shape and segmentation. We propose a novel benchmark for the task of NOCS prediction based on OmniNOCS, which we hope will serve as a useful baseline for future work in this area. Our dataset and code will be at the project website: https://omninocs.github.io., Comment: Accepted to ECCV 2024, project website: https://omninocs.github.io

The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to WMKO's internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections., Comment: 10 pages, 5 figures, SPIE Astronomical Telescopes + Instrumentation 2022

We start with a discussion of the use of mathematics to model the real world then justify the role of Hilbert space formalism for such modelling in the general context of quantum logic. Following this, the incompleteness of the Schr\"odinger equation is discussed as well as the incompleteness of von Neumann's measurement approach \cite{vN}. Subsequently, it is shown that quantum mechanics is indeed completed by the addition of an observer, however the observer is not described in the Hamiltonian formalism but \emph{necessarily} by the quantum stochastic formalism discovered in \cite{HP}. Consequently, the complete theory of quantum mechanics appears to be the Quantum Filtering Theory \cite{ND,NLF}. Finally, it is shown how Schr\"odinger's cat may be understood as a quantum filter, providing an intuitively realistic model and an insight into how quantum filtering works., Comment: 30 pages

Extreme precision radial velocity (EPRV) measurements contend with internal noise (instrumental systematics) and external noise (intrinsic stellar variability) on the road to 10 cm/s "exo-Earth" sensitivity. Both of these noise sources are well-probed using "Sun-as-a-star" RVs and cross-instrument comparisons. We built the Solar Calibrator (SoCal), an autonomous system that feeds stable, disc-integrated sunlight to the recently commissioned Keck Planet Finder (KPF) at the W. M. Keck Observatory. With SoCal, KPF acquires signal-to-noise ~1200, R = ~98,000 optical (445--870 nm) spectra of the Sun in 5~sec exposures at unprecedented cadence for an EPRV facility using KPF's fast readout mode (<16 sec between exposures). Daily autonomous operation is achieved by defining an operations loop using state machine logic. Data affected by clouds are automatically flagged using a reliable quality control metric derived from simultaneous irradiance measurements. Comparing solar data across the growing global network of EPRV spectrographs with solar feeds will allow EPRV teams to disentangle internal and external noise sources and benchmark spectrograph performance. To facilitate this, all SoCal data products are immediately available to the public on the Keck Observatory Archive. We compared SoCal RVs to contemporaneous RVs from NEID, the only other immediately public EPRV solar dataset. We find agreement at the 30-40 cm/s level on timescales of several hours, which is comparable to the combined photon-limited precision. Data from SoCal were also used to assess a detector problem and wavelength calibration inaccuracies associated with KPF during early operations. Long-term SoCal operations will collect upwards of 1,000 solar spectra per six-hour day using KPF's fast readout mode, enabling stellar activity studies at high signal-to-noise on our nearest solar-type star., Comment: 22 pages, 11 figures, accepted to Publications of the Astronomical Society of the Pacific

Pre-trained multimodal foundation models have demonstrated remarkable generalizability but pose challenges for deployment due to their large sizes. One effective approach to reducing their sizes is layerwise distillation, wherein small student models are trained to match the hidden representations of large teacher models at each layer. Motivated by our observation that certain architecture components, referred to as modules, contribute more significantly to the student's performance than others, we propose to track the contributions of individual modules by recording the loss decrement after distillation each module and choose the module with a greater contribution to distill more frequently. Such an approach can be naturally formulated as a multi-armed bandit (MAB) problem, where modules and loss decrements are considered as arms and rewards, respectively. We then develop a modified-Thompson sampling algorithm named OPTIMA to address the nonstationarity of module contributions resulting from model updating. Specifically, we leverage the observed contributions in recent history to estimate the changing contribution of each module and select modules based on these estimations to maximize the cumulative contribution. We evaluate the effectiveness of OPTIMA through distillation experiments on various multimodal understanding and image captioning tasks, using the CoCa-Large model (Yu et al., 2022) as the teacher model.

Quantum entanglement-based imaging promises significantly increased resolution by extending the spatial separation of optical collection apertures used in very-long-baseline interferometry for astronomy and geodesy. We report a table-top entanglement-based interferometric imaging technique that utilizes two entangled field modes serving as a phase reference between two apertures. The spatial distribution of a simulated thermal light source is determined by interfering light collected at each aperture with one of the entangled fields and performing joint measurements. This experiment demonstrates the ability of entanglement to implement interferometric imaging.

Deep neural networks (DNNs) detect patterns in data and have shown versatility and strong performance in many computer vision applications. However, DNNs alone are susceptible to obvious mistakes that violate simple, common sense concepts and are limited in their ability to use explicit knowledge to guide their search and decision making. While overall DNN performance metrics may be good, these obvious errors, coupled with a lack of explainability, have prevented widespread adoption for crucial tasks such as medical image analysis. The purpose of this paper is to introduce SimpleMind, an open-source software framework for Cognitive AI focused on medical image understanding. It allows creation of a knowledge base that describes expected characteristics and relationships between image objects in an intuitive human-readable form. The SimpleMind framework brings thinking to DNNs by: (1) providing methods for reasoning with the knowledge base about image content, such as spatial inferencing and conditional reasoning to check DNN outputs; (2) applying process knowledge, in the form of general-purpose software agents, that are chained together to accomplish image preprocessing, DNN prediction, and result post-processing, and (3) performing automatic co-optimization of all knowledge base parameters to adapt agents to specific problems. SimpleMind enables reasoning on multiple detected objects to ensure consistency, providing cross checking between DNN outputs. This machine reasoning improves the reliability and trustworthiness of DNNs through an interpretable model and explainable decisions. Example applications are provided that demonstrate how SimpleMind supports and improves deep neural networks by embedding them within a Cognitive AI framework.

The James Webb Space Telescope (JWST) Early Release Observations (EROs) is a set of public outreach products created to mark the end of commissioning and the beginning of science operations for JWST. Colloquially known as the "Webb First Images and Spectra", these products were intended to demonstrate to the worldwide public that JWST is ready for science, and is capable of producing spectacular results. The package was released on July 12, 2022, and included images and spectra of the galaxy cluster SMACS~J0723.3-7327 and distant lensed galaxies, the interacting galaxy group Stephan's Quintet, NGC 3324 in the Carina star-forming complex, the Southern Ring planetary nebula NGC 3132, and the transiting hot Jupiter WASP 96b. This paper describes the ERO technical design, observations, and scientific processing of data underlying the colorful outreach products., Comment: 15 pages, accepted by ApJ Letters

This paper characterizes the actual science performance of the James Webb Space Telescope (JWST), as determined from the six month commissioning period. We summarize the performance of the spacecraft, telescope, science instruments, and ground system, with an emphasis on differences from pre-launch expectations. Commissioning has made clear that JWST is fully capable of achieving the discoveries for which it was built. Moreover, almost across the board, the science performance of JWST is better than expected; in most cases, JWST will go deeper faster than expected. The telescope and instrument suite have demonstrated the sensitivity, stability, image quality, and spectral range that are necessary to transform our understanding of the cosmos through observations spanning from near-earth asteroids to the most distant galaxies., Comment: 5th version as accepted to PASP; 31 pages, 18 figures; https://iopscience.iop.org/article/10.1088/1538-3873/acb293

How has recent AI Ethics literature addressed topics such as fairness and justice in the context of continued social and structural power asymmetries? We trace both the historical roots and current landmark work that have been shaping the field and categorize these works under three broad umbrellas: (i) those grounded in Western canonical philosophy, (ii) mathematical and statistical methods, and (iii) those emerging from critical data/algorithm/information studies. We also survey the field and explore emerging trends by examining the rapidly growing body of literature that falls under the broad umbrella of AI Ethics. To that end, we read and annotated peer-reviewed papers published over the past four years in two premier conferences: FAccT and AIES. We organize the literature based on an annotation scheme we developed according to three main dimensions: whether the paper deals with concrete applications, use-cases, and/or people's lived experience; to what extent it addresses harmed, threatened, or otherwise marginalized groups; and if so, whether it explicitly names such groups. We note that although the goals of the majority of FAccT and AIES papers were often commendable, their consideration of the negative impacts of AI on traditionally marginalized groups remained shallow. Taken together, our conceptual analysis and the data from annotated papers indicate that the field would benefit from an increased focus on ethical analysis grounded in concrete use-cases, people's experiences, and applications as well as from approaches that are sensitive to structural and historical power asymmetries., Comment: To appear in the FAccT 2022 proceedings

Small unmanned aircraft systems (sUAS) are becoming prominent components of many humanitarian assistance and disaster response (HADR) operations. Pairing sUAS with onboard artificial intelligence (AI) substantially extends their utility in covering larger areas with fewer support personnel. A variety of missions, such as search and rescue, assessing structural damage, and monitoring forest fires, floods, and chemical spills, can be supported simply by deploying the appropriate AI models. However, adoption by resource-constrained groups, such as local municipalities, regulatory agencies, and researchers, has been hampered by the lack of a cost-effective, readily-accessible baseline platform that can be adapted to their unique missions. To fill this gap, we have developed the free and open-source ADAPT multi-mission payload for deploying real-time AI and computer vision onboard a sUAS during local and beyond-line-of-site missions. We have emphasized a modular design with low-cost, readily-available components, open-source software, and thorough documentation (https://kitware.github.io/adapt/). The system integrates an inertial navigation system, high-resolution color camera, computer, and wireless downlink to process imagery and broadcast georegistered analytics back to a ground station. Our goal is to make it easy for the HADR community to build their own copies of the ADAPT payload and leverage the thousands of hours of engineering we have devoted to developing and testing. In this paper, we detail the development and testing of the ADAPT payload. We demonstrate the example mission of real-time, in-flight ice segmentation to monitor river ice state and provide timely predictions of catastrophic flooding events. We deploy a novel active learning workflow to annotate river ice imagery, train a real-time deep neural network for ice segmentation, and demonstrate operation in the field., Comment: To be published in Workshop on Practical Deep Learning in the Wild at AAAI Conference on Artificial Intelligence 2022, 9 pages, 5 figures

In this paper, we explore the possibility of building a unified foundation model that can be adapted to both vision-only and text-only tasks. Starting from BERT and ViT, we design a unified transformer consisting of modality-specific tokenizers, a shared transformer encoder, and task-specific output heads. To efficiently pre-train the proposed model jointly on unpaired images and text, we propose two novel techniques: (i) We employ the separately-trained BERT and ViT models as teachers and apply knowledge distillation to provide additional, accurate supervision signals for the joint training; (ii) We propose a novel gradient masking strategy to balance the parameter updates from the image and text pre-training losses. We evaluate the jointly pre-trained transformer by fine-tuning it on image classification tasks and natural language understanding tasks, respectively. The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models., Comment: preliminary work

This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long clip. We then extract their dense temporal embeddings. The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips. Our study reveals the impact of temporal granularity with three major findings. 1) Different video tasks may require features of different temporal granularities. 2) Intriguingly, some tasks that are widely considered to require temporal awareness can actually be well addressed by temporally persistent features. 3) The flexibility of TeG gives rise to state-of-the-art results on 8 video benchmarks, outperforming supervised pre-training in most cases.

Visual 2.5D perception involves understanding the semantics and geometry of a scene through reasoning about object relationships with respect to the viewer in an environment. However, existing works in visual recognition primarily focus on the semantics. To bridge this gap, we study 2.5D visual relationship detection (2.5VRD), in which the goal is to jointly detect objects and predict their relative depth and occlusion relationships. Unlike general VRD, 2.5VRD is egocentric, using the camera's viewpoint as a common reference for all 2.5D relationships. Unlike depth estimation, 2.5VRD is object-centric and not only focuses on depth. To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2.5D relationships among 512K objects from 11K images. We analyze this dataset and conduct extensive experiments including benchmarking multiple state-of-the-art VRD models on this task. Our results show that existing models largely rely on semantic cues and simple heuristics to solve 2.5VRD, motivating further research on models for 2.5D perception. The new dataset is available at https://github.com/google-research-datasets/2.5vrd.

We investigate the use of Neural Radiance Fields (NeRF) to learn high quality 3D object category models from collections of input images. In contrast to previous work, we are able to do this whilst simultaneously separating foreground objects from their varying backgrounds. We achieve this via a 2-component NeRF model, FiG-NeRF, that prefers explanation of the scene as a geometrically constant background and a deformable foreground that represents the object category. We show that this method can learn accurate 3D object category models using only photometric supervision and casually captured images of the objects. Additionally, our 2-part decomposition allows the model to perform accurate and crisp amodal segmentation. We quantitatively evaluate our method with view synthesis and image fidelity metrics, using synthetic, lab-captured, and in-the-wild data. Our results demonstrate convincing 3D object category modelling that exceed the performance of existing methods.

We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference. 3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets and do not support online inference, making them difficult to work on mobile devices. We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs. First, we design a video network search space and employ neural architecture search to generate efficient and diverse 3D CNN architectures. Second, we introduce the Stream Buffer technique that decouples memory from video clip duration, allowing 3D CNNs to embed arbitrary-length streaming video sequences for both training and inference with a small constant memory footprint. Third, we propose a simple ensembling technique to improve accuracy further without sacrificing efficiency. These three progressive techniques allow MoViNets to achieve state-of-the-art accuracy and efficiency on the Kinetics, Moments in Time, and Charades video action recognition datasets. For instance, MoViNet-A5-Stream achieves the same accuracy as X3D-XL on Kinetics 600 while requiring 80% fewer FLOPs and 65% less memory. Code will be made available at https://github.com/tensorflow/models/tree/master/official/vision., Comment: Accepted to CVPR 2021

The basic idea here is that observation (or one's experience) is fundamental and the `atomic world' is postulated as the source of such observation. Once this source has been inferred to exist one may attempt to explicitly derive its structure in such a way that the observation itself can be reproduced. And so here is a purely quantum mechanical model of observation coupled to its supposed source, and the observation itself is realised as a projection of this quantum system., Comment: 19 pages

Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to encode both medium and fine scale geometry as well as view-dependent appearance. To generate the proxies' textures, we learn a joint latent space allowing category-level appearance and geometry interpolation. The proxies are independently rasterized with their corresponding neural texture and composited using a U-Net, which generates an output photorealistic image including an alpha map. We demonstrate the effectiveness of our approach by reconstructing complex objects from a sparse set of views. We show results on a dataset of real images of eyeglasses frames, which are particularly challenging to reconstruct using classical methods. We also demonstrate that these coarse proxies can be handcrafted when the underlying object geometry is easy to model, like eyeglasses, or generated using a neural network for more complex categories, such as cars., Comment: ECCV 2020 Spotlight. Project website: https://gelato-paper.github.io

Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e.g., with different initializations) and aggregating their predictions. This approach is commonly reserved for the largest models, as it is commonly held that increasing the model size provides a more substantial reduction in error than ensembling smaller models. However, we show results from experiments on CIFAR-10 and ImageNet that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute, even when those individual models' weights and hyperparameters are highly optimized. Furthermore, this gap in improvement widens as models become large. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models, especially when the models approach the size of what their dataset can foster. Instead of using the common practice of tuning a single large model, one can use ensembles as a more flexible trade-off between a model's inference speed and accuracy. This also potentially eases hardware design, e.g., an easier way to parallelize the model across multiple workers for real-time or distributed inference.

Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions seen by a machine learning model and our expectation of the model to perform well on all classes. We analyze this mismatch from a domain adaptation point of view. First of all, we connect existing class-balanced methods for long-tailed classification to target shift, a well-studied scenario in domain adaptation. The connection reveals that these methods implicitly assume that the training data and test data share the same class-conditioned distribution, which does not hold in general and especially for the tail classes. While a head class could contain abundant and diverse training examples that well represent the expected data at inference time, the tail classes are often short of representative training data. To this end, we propose to augment the classic class-balanced learning by explicitly estimating the differences between the class-conditioned distributions with a meta-learning approach. We validate our approach with six benchmark datasets and three loss functions., Comment: Accepted for publication at CVPR2020

Federated Learning enables visual models to be trained on-device, bringing advantages for user privacy (data need never leave the device), but challenges in terms of data diversity and quality. Whilst typical models in the datacenter are trained using data that are independent and identically distributed (IID), data at source are typically far from IID. Furthermore, differing quantities of data are typically available at each device (imbalance). In this work, we characterize the effect these real-world data distributions have on distributed learning, using as a benchmark the standard Federated Averaging (FedAvg) algorithm. To do so, we introduce two new large-scale datasets for species and landmark classification, with realistic per-user data splits that simulate real-world edge learning scenarios. We also develop two new algorithms (FedVC, FedIR) that intelligently resample and reweight over the client pool, bringing large improvements in accuracy and stability in training. The datasets are made available online.

Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a simulation framework for enabling fast prototyping of algorithms for autonomy and enabling machine learning research in this domain, with the goal of reducing the time, money, and risks associated with field robotics. Our framework enables generation of racing tracks in multiple photo-realistic environments, orchestration of drone races, comes with a suite of gate assets, allows for multiple sensor modalities (monocular, depth, neuromorphic events, optical flow), different camera models, and benchmarking of planning, control, computer vision, and learning-based algorithms. We used our framework to host a simulation based drone racing competition at NeurIPS 2019. The competition binaries are available at our github repository., Comment: 14 pages, 6 figures

Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.

We present Contingency Model Predictive Control (CMPC), a novel and implementable control framework which tracks a desired path while simultaneously maintaining a contingency plan -- an alternate trajectory to avert an identified potential emergency. In this way, CMPC anticipates events that might take place, instead of reacting when emergencies occur. We accomplish this by adding an additional prediction horizon in parallel to the classical receding MPC horizon. The contingency horizon is constrained to maintain a feasible avoidance solution; as such, CMPC is selectively robust to this emergency while tracking the desired path as closely as possible. After defining the framework mathematically, we demonstrate its effectiveness experimentally by comparing its performance to a state-of-the-art deterministic MPC. The controllers drive an automated research platform through a left-hand turn which may be covered by ice. Contingency MPC prepares for the potential loss of friction by purposefully and intuitively deviating from the prescribed path to approach the turn more conservatively; this deviation significantly mitigates the consequence of encountering ice., Comment: American Control Conference, July 2019; 6 pages

Yes, it can. Data augmentation is perhaps the oldest preprocessing step in computer vision literature. Almost every computer vision model trained on imaging data uses some form of augmentation. In this paper, we use the inter-vertebral disk segmentation task alongside a deep residual U-Net as the learning model, to explore the effectiveness of augmentation. In the extreme, we observed that a model trained on patches extracted from just one scan, with each patch augmented 50 times; achieved a Dice score of 0.73 in a validation set of 40 cases. Qualitative evaluation indicated a clinically usable segmentation algorithm, which appropriately segments regions of interest, alongside limited false positive specks. When the initial patches are extracted from nine scans the average Dice coefficient jumps to 0.86 and most of the false positives disappear. While this still falls short of state-of-the-art deep learning based segmentation of discs reported in literature, qualitative examination reveals that it does yield segmentation, which can be amended by expert clinicians with minimal effort to generate additional data for training improved deep models. Extreme augmentation of training data, should thus be construed as a strategy for training deep learning based algorithms, when very little manually annotated data is available to work with. Models trained with extreme augmentation can then be used to accelerate the generation of manually labelled data. Hence, we show that extreme augmentation can be a valuable tool in addressing scaling up small imaging data sets to address medical image segmentation tasks.

This paper presents a weakly-supervised approach to object instance segmentation. Starting with known or predicted object bounding boxes, we learn object masks by playing a game of cut-and-paste in an adversarial learning setup. A mask generator takes a detection box and Faster R-CNN features, and constructs a segmentation mask that is used to cut-and-paste the object into a new image location. The discriminator tries to distinguish between real objects, and those cut and pasted via the generator, giving a learning signal that leads to improved object masks. We verify our method experimentally using Cityscapes, COCO, and aerial image datasets, learning to segment objects without ever having seen a mask in training. Our method exceeds the performance of existing weakly supervised methods, without requiring hand-tuned segment proposals, and reaches 90% of supervised performance.

Recent advances in video super-resolution have shown that convolutional neural networks combined with motion compensation are able to merge information from multiple low-resolution (LR) frames to generate high-quality images. Current state-of-the-art methods process a batch of LR frames to generate a single high-resolution (HR) frame and run this scheme in a sliding window fashion over the entire video, effectively treating the problem as a large number of separate multi-frame super-resolution tasks. This approach has two main weaknesses: 1) Each input frame is processed and warped multiple times, increasing the computational cost, and 2) each output frame is estimated independently conditioned on the input frames, limiting the system's ability to produce temporally consistent results. In this work, we propose an end-to-end trainable frame-recurrent video super-resolution framework that uses the previously inferred HR estimate to super-resolve the subsequent frame. This naturally encourages temporally consistent results and reduces the computational cost by warping only one image in each step. Furthermore, due to its recurrent nature, the proposed method has the ability to assimilate a large number of previous frames without increased computational demands. Extensive evaluations and comparisons with previous methods validate the strengths of our approach and demonstrate that the proposed framework is able to significantly outperform the current state of the art., Comment: Accepted at CVPR 2018

Human vision is able to immediately recognize novel visual categories after seeing just one or a few training examples. We describe how to add a similar capability to ConvNet classifiers by directly setting the final layer weights from novel training examples during low-shot learning. We call this process weight imprinting as it directly sets weights for a new category based on an appropriately scaled copy of the embedding layer activations for that training example. The imprinting process provides a valuable complement to training with stochastic gradient descent, as it provides immediate good classification performance and an initialization for any further fine-tuning in the future. We show how this imprinting process is related to proxy-based embeddings. However, it differs in that only a single imprinted weight vector is learned for each novel category, rather than relying on a nearest-neighbor distance to training instances as typically used with embedding methods. Our experiments show that using averaging of imprinted weights provides better generalization than using nearest-neighbor instance embeddings., Comment: CVPR 2018

Deep reinforcement learning yields great results for a large array of problems, but models are generally retrained anew for each new problem to be solved. Prior learning and knowledge are difficult to incorporate when training new models, requiring increasingly longer training as problems become more complex. This is especially problematic for problems with sparse rewards. We provide a solution to these problems by introducing Concept Network Reinforcement Learning (CNRL), a framework which allows us to decompose problems using a multi-level hierarchy. Concepts in a concept network are reusable, and flexible enough to encapsulate feature extractors, skills, or other concept networks. With this hierarchical learning approach, deep reinforcement learning can be used to solve complex tasks in a modular way, through problem decomposition. We demonstrate the strength of CNRL by training a model to grasp a rectangular prism and precisely stack it on top of a cube using a gripper on a Kinova JACO arm, simulated in MuJoCo. Our experiments show that our use of hierarchy results in a 45x reduction in environment interactions compared to the state-of-the-art on this task.

We consider efficient methods for computing solutions to and estimating uncertainties in dynamic inverse problems, where the parameters of interest may change during the measurement procedure. Compared to static inverse problems, incorporating prior information in both space and time in a Bayesian framework can become computationally intensive, in part, due to the large number of unknown parameters. In these problems, explicit computation of the square root and/or inverse of the prior covariance matrix is not possible. In this work, we develop efficient, iterative, matrix-free methods based on the generalized Golub-Kahan bidiagonalization that allow automatic regularization parameter and variance estimation. We demonstrate that these methods can be more flexible than standard methods and develop efficient implementations that can exploit structure in the prior, as well as possible structure in the forward model. Numerical examples from photoacoustic tomography, deblurring, and passive seismic tomography demonstrate the range of applicability and effectiveness of the described approaches. Specifically, in passive seismic tomography, we demonstrate our approach on both synthetic and real data. To demonstrate the scalability of our algorithm, we solve a dynamic inverse problem with approximately $43,000$ measurements and $7.8$ million unknowns in under $40$ seconds on a standard desktop., Comment: 27 pages, 12 figures

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. We achieve this by simultaneously training depth and camera pose estimation networks using the task of view synthesis as the supervisory signal. The networks are thus coupled via the view synthesis objective during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performing comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performing favorably with established SLAM systems under comparable input settings., Comment: Accepted to CVPR 2017. Project webpage: https://people.eecs.berkeley.edu/~tinghuiz/projects/SfMLearner/

Human keypoints are a well-studied representation of people.We explore how to use keypoint models to improve instance-level person segmentation. The main idea is to harness the notion of a distance transform of oracle provided keypoints or estimated keypoint heatmaps as a prior for person instance segmentation task within a deep neural network. For training and evaluation, we consider all those images from COCO where both instance segmentation and human keypoints annotations are available. We first show how oracle keypoints can boost the performance of existing human segmentation model during inference without any training. Next, we propose a framework to directly learn a deep instance segmentation model conditioned on human pose. Experimental results show that at various Intersection Over Union (IOU) thresholds, in a constrained environment with oracle keypoints, the instance segmentation accuracy achieves 10% to 12% relative improvements over a strong baseline of oracle bounding boxes. In a more realistic environment, without the oracle keypoints, the proposed deep person instance segmentation model conditioned on human pose achieves 3.8% to 10.5% relative improvements comparing with its strongest baseline of a deep network trained only for segmentation.

In this paper we present a dense ground truth dataset of nonrigidly deforming real-world scenes. Our dataset contains both long and short video sequences, and enables the quantitatively evaluation for RGB based tracking and registration methods. To construct ground truth for the RGB sequences, we simultaneously capture Near-Infrared (NIR) image sequences where dense markers - visible only in NIR - represent ground truth positions. This allows for comparison with automatically tracked RGB positions and the formation of error metrics. Most previous datasets containing nonrigidly deforming sequences are based on synthetic data. Our capture protocol enables us to acquire real-world deforming objects with realistic photometric effects - such as blur and illumination change - as well as occlusion and complex deformations. A public evaluation website is constructed to allow for ranking of RGB image based optical flow and other dense tracking algorithms, with various statistical measures. Furthermore, we present an RGB-NIR multispectral optical flow model allowing for energy optimization by adoptively combining featured information from both the RGB and the complementary NIR channels. In our experiments we evaluate eight existing RGB based optical flow methods on our new dataset. We also evaluate our hybrid optical flow algorithm by comparing to two existing multispectral approaches, as well as varying our input channels across RGB, NIR and RGB-NIR., Comment: preprint of our paper accepted by RA-L'16

It is hard to densely track a nonrigid object in long term, which is a fundamental research issue in the computer vision community. This task often relies on estimating pairwise correspondences between images over time where the error is accumulated and leads to a drift issue. In this paper, we introduce a novel optimization framework with an Anchor Patch constraint. It is supposed to significantly reduce overall errors given long sequences containing non-rigidly deformable objects. Our framework can be applied to any dense tracking algorithm, e.g. optical flow. We demonstrate the success of our approach by showing significant error reduction on 6 popular optical flow algorithms applied to a range of real-world nonrigid benchmarks. We also provide quantitative analysis of our approach given synthetic occlusions and image noise., Comment: Preprint version of our paper accepted by Journal of Intelligent and Fuzzy Systems

This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). CNNs achieve state of the art accuracy, thanks to their representation learning capabilities. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. efficiency. We call this new family of hybrid models conditional networks. Conditional networks can be thought of as: i) decision trees augmented with data transformation operators, or ii) CNNs, with block-diagonal sparse weight matrices, and explicit data routing functions. Experimental validation is performed on the common task of image classification on both the CIFAR and Imagenet datasets. Compared to state of the art CNNs, our hybrid models yield the same accuracy with a fraction of the compute cost and much smaller number of parameters., Comment: Microsoft Research Technical Report

The evolution of a pair of point vortices in whole space, subject to the inviscid Euler equations for incompressible fluid flow, is solved exactly for rotationally symmetric initial conditions. This exact solution shows that the vortex strength for a pair of point vortices can either remain stable or blow up in finite time, depending on the initial data., Comment: 2 pages

In this paper we shall re-visit the well-known Schr\"odinger and Lindblad dynamics of quantum mechanics. However, these equations may be realized as the consequence of a more general, underlying dynamical process. In both cases we shall see that the evolution of a quantum state $P_\psi=\varrho(0)$ has the not so well-known pseudo-quadratic form $\partial_t\varrho(t)=\mathbf{V}^\star\varrho(t)\mathbf{V}$ where $\mathbf{V}$ is a vector operator in a complex Minkowski space and the pseudo-adjoint $\mathbf{V}^\star$ is induced by the Minkowski metric $\boldsymbol{\eta}$. The interesting thing about this formalism is that its derivation has very deep roots in a new understanding of the differential calculus of time. This Minkowski-Hilbert representation of quantum dynamics is called the \emph{Belavkin Formalism}; a beautiful, but not well understood theory of mathematical physics that understands that both deterministic and stochastic dynamics may be `unraveled' in a second-quantized Minkowski space. Working in such a space provided the author with the means to construct a QS (quantum stochastic) Duhamel principle and known applications to a Schr\"odinger dynamics perturbed by a continual measurement process are considered. What is not known, but presented here, is the role of the Lorentz transform in quantum measurement, and the appearance of Riemannian geometry in quantum measurement is also discussed.

Here we shall consider the idea that the Hamiltonian evolution of a quantum system is generated by sequential observations of the system by a `pseudo-apparatus'. This representation of Hamiltonian dynamics, originally discovered by Belavkin, is a canonical dilation of the Schroedinger equation that reveals a Minkowski space structure outside of the context of Special Relativity. In particular, this formalism gives rise to the notion of a Boosted Schroedinger equation referred to a dilated time increment which is the manifestation of a Lorentz transform in the Minkowski space of the apparatus., Comment: 20 pages, canonical quantum measurement

This discovery of the Higgs boson last year has created new possibilities for testing candidate theories for explaining physics beyond the Standard Model. Here we explain the ways in which new physics can leave its marks in the experimental Higgs data, and how we can use the data to constrain and compare different models. In this proceedings paper we use two models, Minimal Universal Extra Dimensions and the 4D Composite Higgs model, as examples to demonstrate the technique., Comment: V2 corrected typo in author name. Submitted to the proceedings of the 41st ITEP Winter School, Moscow

We consider scenarios in which the 125 GeV resonance observed at the Large Hadron Collider is a Technicolor (TC) isosinglet scalar, the TC Higgs. By comparison with quantum chromodynamics, we argue that the couplings of the TC Higgs to the massive weak bosons are very close to the Standard Model (SM) values. The couplings to photons and gluons are model-dependent, but close to the SM values in several TC theories. The couplings of the TC Higgs to SM fermions are due to interactions beyond TC, such as Extended Technicolor: if such interactions successfully generate mass for the SM fermions, we argue that the couplings of the latter to the TC Higgs are also SM-like. We suggest a generic parameterization of the TC Higgs interactions with SM particles that accommodates a large class of TC models, and we perform a fit of these parameters to the Higgs LHC data. The fit reveals regions of parameter space where the form factors are of order unity and consistent with data at the 95% CL, in agreement with expectations in TC theories. This indicates that the discovered Higgs boson is consistent with the TC Higgs hypothesis for several TC theories., Comment: 26 pages, 8 figures

In this work we discuss our consistent implementation of the minimal model of Universal Extra Dimensions in CalcHEP. We pay special attention to the gauge invariance issues that arise due to the incorporation of 5D quantum corrections. After validating the implementation we perform a complete study of the tri-lepton signature, including a realistic estimate of the backgrounds, for the present LHC energy and luminosity. We also derive the expected LHC discovery reach for different luminosities, at centre-of-mass energies of both 7 TeV and 8 TeV., Comment: This version (and the previous versions) fixes some typos in the text. In the previous versions, some sentences were added or changed to make the explanation clearer. Also, some of the figures in previous versions were changed to correct minor plotting errors that do not affect the results

Large Hadron Collider (LHC) searches for the SM Higgs boson provide a powerful limit on models involving Universal Extra Dimensions (UED) where the Higgs production is enhanced. We have evaluated all one-loop diagrams for Higgs production from gluon fusion and decay to two photons within "minimal" UED (mUED), independently confirming previous results, and we have evaluated enhancement factors for Higgs boson production and decay over the mUED parameter space. Using these we have derived limits on the parameter space, combining data from both ATLAS and CMS collaborations for the most recent 7 TeV and 8 TeV LHC data. We have performed a rigorous statistical combination of several Higgs boson search channels which is important because mUED signatures from the Higgs boson are not universally enhanced. We have found that 1/R < 500 GeV is excluded at 95% CL, while for larger 1/R only a very narrow (\pm1-4 GeV) mass window around m_h = 125 GeV and another window (up to 2 GeV wide for 1/R > 1000 GeV) around m_h = 118 GeV are left. The latter is likely to be excluded as more data becomes available whereas the region around 125 GeV is where the recently discovered Higgs-like particle was observed and therefore where the exclusion limit is weaker. It is worth stressing that mUED predicts an enhancement for all channels for Higgs production by gluon fusion and decay while the vector boson fusion process WW/ZZ -> h -> AA is generically suppressed and WW/ZZ -> h -> WW*/ZZ* is standard. Therefore, as more 8 TeV LHC data becomes available, the information on individual Higgs boson production and decay processes provided by the CMS and ATLAS experiments can be effectively used to favour mUED or exclude it further., Comment: 39 pages, 6 figures

We study the continuity property of multiple Q-adapted quantum stochastic integrals with respect to noncommuting integrands given by the non-adapted multiple integral kernels in Fock scale. The noncommutative algebra of relatively (exponentially) bounded nonadapted quantum stochastic processes is studied in the kernel form as introduced by Belavkin in 1991. The differential Q-adapted formula generalizing Ito product formula for adapted integrals is presented in both strong and weak sense as a particular case of the quantum stochastic nonadapted Ito formula., Comment: Due to appear in communications on stochastic analysis journal (KRP volume). 21 pages

In this paper we first introduce the Fock-Guichardet formalism for the quantum stochastic integration, then the four fundamental processes of the dynamics are introduced in the canonical basis as the operator-valued measures of the QS integration over a space-time. Then rigorous analysis of the QS integrals is carried out, and continuity of the QS derivative is proved. Finally, Q-adapted dynamics is discussed, including Bosonic Q=1, Fermionic Q=-1, and monotone Q=0 quantum dynamics. These may be of particular interest to quantum field theory, quantum open systems, and quantum theory of stochastic processes., Comment: A similar version is due to appear in the proceedings of the 13th workshop: non-commutative harmonic analysis. 18 pages

Here it is shown that the unitary dynamics of a quantum object may be obtained as the conditional expectation of a counting process of object-clock interactions. Such a stochastic process arises from the quantization of the clock, and this is derived naturally from the matrix-algebra representation of the nilpotent Newton-Leibniz time differential [Belavkin]. It is observed that this condition expectation is a rigorous formulation of the Feynman Path Integral., Comment: 21 pages

All-electron variational and diffusion quantum Monte Carlo calculations of the ground state energies of the first row atoms (Li to Ne) are reported. We use trial wavefunctions of four types: single determinant Slater-Jastrow wavefunctions; multi-determinant Slater-Jastrow wavefunctions; single determinant Slater-Jastrow wavefunctions with backflow transformations; multi-determinant Slater-Jastrow wavefunctions with backflow transformations. At the diffusion quantum Monte Carlo level and using our best trial wavefunctions we recover 99% or more of the correlation energy for Li, Be, B, C, N, and Ne, 97% for O, and 98% for F., Comment: 6 pages, 4 figures

An oscillating universe cycles through a series of expansions and contractions. We propose a model in which ``phantom'' energy with a supernegative pressure ($p < - \rho$) grows rapidly and dominates the late-time expanding phase. The universe's energy density is then so large that the effects of quantum gravity are important at both the beginning and the end of each expansion (or contraction). The bounce can be caused by high energy modifications to the Friedmann equation governing the expansion of the universe, which make the cosmology nonsingular. The classic black hole overproduction of oscillating universes is resolved due to their destruction by the phantom energy., Comment: New York Academy of Sciences Proceedings, Origins of Time's Arrow Conference, October 2007

An oscillating universe cycles through a series of expansions and contractions. We propose a model in which ``phantom'' energy with $p < -\rho$ grows rapidly and dominates the late-time expanding phase. The universe's energy density is so large that the effects of quantum gravity are important at both the beginning and the end of each expansion (or contraction). The bounce can be caused by high energy modifications to the Friedmann equation, which make the cosmology nonsingular. The classic black hole overproduction of oscillating universes is resolved due to their destruction by the phantom energy., Comment: Four pages, one figure. V3: version to appear in JCAP