Search

Configuration evaluation is a key technology to be considered in the design of multiple aircrafts formation (MAF) configurations with high dynamic properties in engineering applications. This paper deduces the relationship between relative velocity, dynamic safety distance and dynamic adjacent distance of formation members, then divides the formation states into collision-state and matching-state. Meanwhile, probability models are constructed based on the binary normal distribution of relative distance and relative velocity. Moreover, configuration evaluation strategies are studied by quantitatively analyzing the denseness and the basic capabilities according to the MAF collision-state probability and the MAF matching-state probability, respectively. The scale of MAF is grouped into 5 levels, and previous lattice-type structures are extended into four degrees by taking the relative velocities into account to instruct the configuration design under complex task conditions. Finally, hardware-in-loop (HIL) simulation and outfield flight test results are presented to verify the feasibility of these evaluation strategies.

Objectives To assess the recall of a deep learning (DL) method to automatically detect kidney stones composition from digital photographs of stones. Materials and methods A total of 63 human kidney stones of varied compositions were obtained from a stone laboratory including calcium oxalate monohydrate (COM), uric acid (UA), magnesium ammonium phosphate hexahydrate (MAPH/struvite), calcium hydrogen phosphate dihydrate (CHPD/brushite), and cystine stones. At least two images of the stones, both surface and inner core, were captured on a digital camera for all stones. A deep convolutional neural network (CNN), ResNet-101 (ResNet, Microsoft), was applied as a multi-class classification model, to each image. This model was assessed using leave-one-out cross-validation with the primary outcome being network prediction recall. Results The composition prediction recall for each composition was as follows: UA 94% (n = 17), COM 90% (n = 21), MAPH/struvite 86% (n = 7), cystine 75% (n = 4), CHPD/brushite 71% (n = 14). The overall weighted recall of the CNNs composition analysis was 85% for the entire cohort. Specificity and precision for each stone type were as follows: UA (97.83%, 94.12%), COM (97.62%, 95%), struvite (91.84%, 71.43%), cystine (98.31%, 75%), and brushite (96.43%, 75%). Conclusion Deep CNNs can be used to identify kidney stone composition from digital photographs with good recall. Future work is needed to see if DL can be used for detecting stone composition during digital endoscopy. This technology may enable integrated endoscopic and laser systems that automatically provide laser settings based on stone composition recognition with the goal to improve surgical efficiency.

Atomic force microscope (AFM)-based nanomanufacturing, as a low-cost and easy-to-setup technique, enables high-resolution maskless nanofabrication processes for arbitrary nanopatterns, but there are strong needs to advance the nanopatterning processes. Thermal-mechanical AFM-based nanomachining process lower the normal forces to few hundreds of nanonewtons, but it requires special thermal probes. Vibration-assisted nanomachining process increases the tip lifetime but forces are still in the range of few hundreds of nanonewtons. We have innovated a system with both vibration and heated sample to achieve nanomachining process with lower normal forces. To provide cantilever selection guidance for this novel nanomachining system and to understand the effect of cantilever stiffness and sample temperature on feature dimensions, we used both soft and stiff probes for the nanomachining processes in this paper. We fabricated nanostructures with controllable shape dimensions were fabricated on 200 nm depth polymethyl methacrylate (PMMA) films by using both soft and stiff AFM probes with a few tens of nanonewtons normal forces in this novel AFM-based nanomachining system. Besides, we analyzed the effects of different machining parameters on the lithography performance, which helps to understand the machining parameters for fabricating nanopatterns of different dimensions. We also found that other than the setpoint force, the probe category and sample heating temperature significantly affect the dimensions of fabricated nanopatterns. And there is a statistically significant interaction effect between the spring constants of probes and the sample temperatures.

The recognition of human emotions from facial expression images is one of the most important topics in the machine vision and image processing fields. However, recognition becomes difficult when dealing with non-frontal faces. To alleviate the influence of poses, we propose an encoder-decoder generative adversarial network that can learn pose-invariant and expression-discriminative representations. Specifically, we assume that a facial image can be divided into an expressive component, an identity component, a head pose component and a remaining component. The encoder encodes each component into a feature representation space and the decoder recovers the original image from these encoded features. A classification loss on the components and an ℓ1 pixel-wise loss are applied to guarantee the rebuilt image quality and produce more constrained visual representations. Quantitative and qualitative evaluations on two multi-pose datasets demonstrate that the proposed algorithm performs favorably compared to state-of-the-art methods.

Bio-mimic wings are crucial constituents of Flapping-Wing Micro Air Vehicles, and the quality of such products plays a vital role in missions involving area exploration, aerial imaging and monitoring. Fused filament fabrication, as a versatile manufacturing approach, can be adopted to print single-layer artificial wings in an additive way. In such processes, quality of the printed wings is determined by the percentage of covered area, contingent on the vertical distance between the printing nozzle and the substrate. Even when properly calibrated, this distance is subject to variations due to the exceeding sensitivity to process drift, including vibration of the platform. In situ image analysis provides rapid guidance for process drift detection and assessment of the printed wing structures. While dynamic network analysis has been applied to quality assurance of steaming images, the computational overhead has made it unrealistic for monitoring high-definition images, particularly in the detection of images with incipient or tiny defects. In this study, we investigate image decomposition techniques, including wavelet transformation and bi-dimensional empirical mode decomposition (BEMD), to dissolve the original images into a series of basis images, which significantly reduces the number of communities in the network representation of images and diminishes the computational cost. Lastly, network generalized likelihood ratio control chart is developed to characterize anomalous variations in structures of the images. Our investigation shows that wavelet and EMD elevate the detection accuracy by 36.7% and 63.3%, while reducing the processing time by 43.5% and 73.9%, respectively, compared to the dynamic network approach without decomposition.

Nanoimprint lithography is a fabrication method by applying nanomolds on resists to form inversed patterns. It can be utilized to fabricate high-resolution nanopatterns in a low cost and rapid fashion on both flat and curved surfaces. The fidelity of fabricated master nanopatterns and elastomer nanomolds are essential to the quality of the nanoimprinted features. Although two-dimensional nanoimprint lithography has been widely investigated, there are limited researches about three-dimensional master nanopatterns and nanomolds. Even less of them discussed the fidelity of entire procedures throughout the initial master fabrication to the final nanoimprinting. In this paper, we demonstrated the effectiveness of our approach, atomic force microscope (AFM) based ultrasonic vibration assisted nanomachining, in fabricating three-dimensional master nanopatterns with complex contours and continuously height changes. Nanomolds for nanoimprint lithography were fabricated subsequently based on the master nanopatterns with a positive draft, which is naturally generated from the AFM tip. We investigated the fidelity of the nanomachining results and the residual effect of reusing mater patterns. Besides, we demonstrated the 3D nanoimprinting capability of the solvent-assisted microcontact molding process, discovering the possibility of tuning the layer-thickness of imprinted 3D nanopatterns.

We address the problem of performing backpropagation for computation graphs involving 3D transformation groups SO(3), SE(3), and Sim(3). 3D transformation groups are widely used in 3D vision and robotics, but they do not form vector spaces and instead lie on smooth manifolds. The standard backpropagation approach, which embeds 3D transformations in Euclidean spaces, suffers from numerical difficulties. We introduce a new library, which exploits the group structure of 3D transformations and performs backpropagation in the tangent spaces of manifolds. We show that our approach is numerically more stable, easier to implement, and beneficial to a diverse set of tasks. Our plug-and-play PyTorch library is available at https://github.com/princeton-vl/lietorch., Comment: fixed typos

We address the problem of scene flow: given a pair of stereo or RGB-D video frames, estimate pixelwise 3D motion. We introduce RAFT-3D, a new deep architecture for scene flow. RAFT-3D is based on the RAFT model developed for optical flow but iteratively updates a dense field of pixelwise SE3 motion instead of 2D motion. A key innovation of RAFT-3D is rigid-motion embeddings, which represent a soft grouping of pixels into rigid objects. Integral to rigid-motion embeddings is Dense-SE3, a differentiable layer that enforces geometric consistency of the embeddings. Experiments show that RAFT-3D achieves state-of-the-art performance. On FlyingThings3D, under the two-view evaluation, we improved the best published accuracy (d < 0.05) from 34.3% to 83.7%. On KITTI, we achieve an error of 5.77, outperforming the best published method (6.31), despite using no object instance supervision. Code is available at https://github.com/princeton-vl/RAFT-3D.

Synthetic images rendered by graphics engines are a promising source for training deep networks. However, it is challenging to ensure that they can help train a network to perform well on real images, because a graphics-based generation pipeline requires numerous design decisions such as the selection of 3D shapes and the placement of the camera. In this work, we propose a new method that optimizes the generation of 3D training data based on what we call "hybrid gradient". We parametrize the design decisions as a real vector, and combine the approximate gradient and the analytical gradient to obtain the hybrid gradient of the network performance with respect to this vector. We evaluate our approach on the task of estimating surface normal, depth or intrinsic decomposition from a single image. Experiments on standard benchmarks show that our approach can outperform the prior state of the art on optimizing the generation of 3D training data, particularly in terms of computational efficiency.

Single-view 3D is the task of recovering 3D properties such as depth and surface normals from a single image. We hypothesize that a major obstacle to single-image 3D is data. We address this issue by presenting Open Annotations of Single Image Surfaces (OASIS), a dataset for single-image 3D in the wild consisting of annotations of detailed 3D geometry for 140,000 images. We train and evaluate leading models on a variety of single-image 3D tasks. We expect OASIS to be a useful resource for 3D vision research. Project site: https://pvl.cs.princeton.edu/OASIS., Accepted to CVPR 2020

Recent advances have spurred incredible progress in self-supervised pretraining for vision. We investigate what factors may play a role in the utility of these pretraining methods for practitioners. To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows as well as how the utility changes as a function of the downstream task and the properties of the training data. We also find that linear evaluation does not correlate with finetuning performance. Code and data is available at \href{https://www.github.com/princeton-vl/selfstudy}{github.com/princeton-vl/selfstudy}., Comment: To appear in CVPR 2020

Surface enhanced Raman spectroscopy (SERS) is a surface-sensitive detection technique that dramatically increases the scattering signals of the analytes compared to traditional Raman spectroscopy. Rapid and low-cost fabrication of SERS substrates with easily tunable features remains a challenge, although many SERS substrates with high enhancement factors (EF) were investigated. Here, we report a novel and rapid approach of fabricating SERS substrates using ultrasonic vibration assisted nanomachining. Grids patterns with easily tuned dimensions were fabricated on PMMA surfaces. SERS substrates were fabricated after coating an 80 nm gold layer on the patterned silicon after reactive ion etching (RIE) using the patterned PMMA as the mask. Probing molecules of R6G with the area density of 2.8 × 10-12 M/mm2 can be detected, achieving an EF of 3.11 × 103. Results show a pattern with higher grids density tends to achieve a higher EF.

Surface electromyography signal plays an important role in hand function recovery training. In this paper, an IoT-enabled stroke rehabilitation system was introduced which was based on a smart wearable armband (SWA), machine learning (ML) algorithms, and a 3-D printed dexterous robot hand. User comfort is one of the key issues which should be addressed for wearable devices. The SWA was developed by integrating a low-power and tiny-sized IoT sensing device with textile electrodes, which can measure, pre-process, and wirelessly transmit bio-potential signals. By evenly distributing surface electrodes over user’s forearm, drawbacks of classification accuracy poor performance can be mitigated. A new method was put forward to find the optimal feature set. ML algorithms were leveraged to analyze and discriminate features of different hand movements, and their performances were appraised by classification complexity estimating algorithms and principal components analysis. According to the verification results, all nine gestures can be successfully identified with an average accuracy up to 96.20%. In addition, a 3-D printed five-finger robot hand was implemented for hand rehabilitation training purpose. Correspondingly, user’s hand movement intentions were extracted and converted into a series of commands which were used to drive motors assembled inside the dexterous robot hand. As a result, the dexterous robot hand can mimic the user’s gesture in a real-time manner, which shows the proposed system can be used as a training tool to facilitate rehabilitation process for the patients after stroke., An IoT-enabled stroke rehabilitation system was implemented which was based on a smart wearable armband, machine learning algorithms, and a 3D printed dexterous robot hand. The smart wearable armband was developed by integrating a low-power and tiny-sized IoT sensing device with textile electrodes, which can measure, pre-process, and wirelessly transmit bio-potential signals. Machine learning (ML) algorithms were leveraged to analyze and discriminate features of different hand movements. The patient's hand movement intentions were further extracted to control the dexterous robot hand and the robot hand can mimic the patient's gesture in a real-time manner which enables active response to patient's volition.

This paper explores object detection in the small data regime, where only a limited number of annotated bounding boxes are available due to data rarity and annotation expense. This is a common challenge today with machine learning being applied to many new tasks where obtaining training data is more challenging, e.g. in medical images with rare diseases that doctors sometimes only see once in their life-time. In this work we explore this problem from a generative modeling perspective by learning to generate new images with associated bounding boxes, and using these for training an object detector. We show that simply training previously proposed generative models does not yield satisfactory performance due to them optimizing for image realism rather than object detection accuracy. To this end we develop a new model with a novel unrolling mechanism that jointly optimizes the generative model and a detector such that the generated images improve the performance of the detector. We show this method outperforms the state of the art on two challenging datasets, disease detection and small data pedestrian detection, improving the average precision on NIH Chest X-ray by a relative 20% and localization accuracy by a relative 50%., Comment: Published in ICCV 2019

Understanding the spatial relations between objects in images is a surprisingly challenging task. A chair may be "behind" a person even if it appears to the left of the person in the image (depending on which way the person is facing). Two students that appear close to each other in the image may not in fact be "next to" each other if there is a third student between them. We introduce SpatialSense, a dataset specializing in spatial relation recognition which captures a broad spectrum of such challenges, allowing for proper benchmarking of computer vision techniques. SpatialSense is constructed through adversarial crowdsourcing, in which human annotators are tasked with finding spatial relations that are difficult to predict using simple cues such as 2D spatial configuration or language priors. Adversarial crowdsourcing significantly reduces dataset bias and samples more interesting relations in the long tail compared to existing datasets. On SpatialSense, state-of-the-art recognition models perform comparably to simple baselines, suggesting that they rely on straightforward cues instead of fully reasoning about this complex task. The SpatialSense benchmark provides a path forward to advancing the spatial reasoning capabilities of computer vision systems. The dataset and code are available at https://github.com/princeton-vl/SpatialSense., Accepted to ICCV 2019

Ensemble learning aims to train and combine multiple base learners in the hope of improving the overall performance. Existing ensemble algorithms rely on a centralized framework where each base learner has access to the entire training dataset. We combine the technology of blockchains which is mainly used for data validation with ensemble learning and propose a decentralized framework where data are distributed among multiple base learners, who exchange their respective data to improve the collective predictive abilities. We develop two realizations of this framework, based on static and dynamic decision trees, respectively. We evaluate our methods over 20 real-world datasets and compare them against other centralized ensemble methods. Experimental results show that the proposed method obtains improved accuracy scores through sample exchange and achieves competitive performance with state-of-the-art ensemble methods whereas the base learners store only a small fraction of the samples.

Highly expressive models such as deep neural networks (DNNs) have been widely applied to various applications. However, recent studies show that DNNs are vulnerable to adversarial examples, which are carefully crafted inputs aiming to mislead the predictions. Currently, the majority of these studies have focused on perturbation added to image pixels, while such manipulation is not physically realistic. Some works have tried to overcome this limitation by attaching printable 2D patches or painting patterns onto surfaces, but can be potentially defended because 3D shape features are intact. In this paper, we propose meshAdv to generate "adversarial 3D meshes" from objects that have rich shape features but minimal textural variation. To manipulate the shape or texture of the objects, we make use of a differentiable renderer to compute accurate shading on the shape and propagate the gradient. Extensive experiments show that the generated 3D meshes are effective in attacking both classifiers and object detectors. We evaluate the attack under different viewpoints. In addition, we design a pipeline to perform black-box attack on a photorealistic renderer with unknown rendering parameters., Comment: Published in IEEE CVPR2019

INTRODUCTION AND OBJECTIVES:Deep learning is a subset of machine learning and artificial intelligence in which algorithms are developed to train machines/computers to detect a specific feature or c...

We introduce a new embedding model to represent movie characters and their interactions in a dialogue by encoding in the same representation the language used by these characters as well as information about the other participants in the dialogue. We evaluate the performance of these new character embeddings on two tasks: (1) character relatedness, using a dataset we introduce consisting of a dense character interaction matrix for 4,378 unique character pairs over 22 hours of dialogue from eighteen movies; and (2) character relation classification, for fine- and coarse-grained relations, as well as sentiment relations. Our experiments show that our model significantly outperforms the traditional Word2Vec continuous bag-of-words and skip-gram models, demonstrating the effectiveness of the character embeddings we introduce. We further show how these embeddings can be used in conjunction with a visual question answering system to improve over previous results.

This paper presents a novel AFM-based 3D nanofabrication process using ultrasonic vibration assisted nanomachining. A set of three dimensional nanostructures on polymethyl methacrylate (PMMA) samples are fabricated with the assistance of high frequency in-plane circular xy-vibration and ultrasonic tip-sample z-vibration. Two methods for fabricating 3D nanostructures were investigated in this study, which are layer-by-layer nanomachining and one pass nanomachining with the depth controlled by setpoint force. Critical parameters in the process are identified, including setpoint force, overlap percentage, amplitude of z vibration and machining speed. By regulating these process parameters, multi-level 3D nanostructures were fabricated by multi-layer machining in vector mode and raster scan mode. Using different setpoint forces for regulating feature depths, other nanostructures, such as convex and concave circles, were fabricated in raster scan mode from gray-scale bitmap pattern images. Under each mode, 3D nanostructure over microscale area can be fabricated in just a few minutes with sub-10 nm resolution in z direction.

Fine-grained recognition concerns categorization at sub-ordinate levels, where the distinction between object classes is highly local. Compared to basic level recognition, fine-grained categorization can be more challenging as there are in general less data and fewer discriminative features. This necessitates the use of a stronger prior for feature selection. In this work, we include humans in the loop to help computers select discriminative features. We introduce a novel online game called "Bubbles" that reveals discriminative features humans use. The player's goal is to identify the category of a heavily blurred image. During the game, the player can choose to reveal full details of circular regions ("bubbles"), with a certain penalty. With proper setup the game generates discriminative bubbles with assured quality. We next propose the "BubbleBank" representation that uses the human selected bubbles to improve machine recognition performance. Finally, we demonstrate how to extend BubbleBank to a view-invariant 3D representation. Experiments demonstrate that our approach yields large improvements over the previous state of the art on challenging benchmarks.

We have seen remarkable recent progress in computational visual recognition, producing systems that can classify objects into thousands of different categories with increasing accuracy. However, one question that has received relatively less attention is "what labels should recognition systems output?" This paper looks at the problem of predicting category labels that mimic how human observers would name objects. This goal is related to the concept of entry-level categories first introduced by psychologists in the 1970s and 1980s. We extend these seminal ideas to study human naming at large scale and to learn computational models for predicting entry-level categories. Practical applications of this work include improving human-focused computer vision applications such as automatically generating a natural language description for an image or text-based image search.

This paper introduces a high precision 3D nanofabrication approach using ultrasonic vibration assisted nanomachining using an AFM operating in constant height control mode. Nanostructures with 3D features were successfully fabricated on PMMA film with the feature height manipulated through controlling the absolute heights of z-scanner in AFM. Two methods were used to move the AFM tip to create desire features, vector mode and raster scan mode. Relatively simple features, such as stair-like nanostructure with five steps was successfully fabricated in vector mode. Complex nanostructure with discrete height levels and continuous changes were successfully fabricated in raster scan mode. By carefully selecting the machining parameters, the feature dimension and height can be precisely controlled with only small variation from the designed value. Moreover, this paper explores the capability of transferring 3D nanostructures from PMMA film onto silicon substrate. After calibrating the recipe of Reactive Ion Etching (RIE) process, 3D nanostructures are successfully transferred to silicon wafer with controllable selectivity between PMMA and silicon. The results of fabricating 3D structures on silicon substrates show promising potential of many applications, such as mold preparation in nanoimprint lithography.

We propose CornerNet, a new approach to object detection where we detect an object bounding box as a pair of keypoints, the top-left corner and the bottom-right corner, using a single convolution neural network. By detecting objects as paired keypoints, we eliminate the need for designing a set of anchor boxes commonly used in prior single-stage detectors. In addition to our novel formulation, we introduce corner pooling, a new type of pooling layer that helps the network better localize corners. Experiments show that CornerNet achieves a 42.2% AP on MS COCO, outperforming all existing one-stage detectors., Extended version with additional results. Test AP on MS COOO improved from 42.1% to 42.2% after a bug fix

Depth estimation from a single image in the wild remains a challenging problem. One main obstacle is the lack of high-quality training data for images in the wild. In this paper we propose a method to automatically generate such data through Structure-from-Motion (SfM) on Internet videos. The core of this method is a Quality Assessment Network that identifies high-quality reconstructions obtained from SfM. Using this method, we collect single-view depth training data from a large number of YouTube videos and construct a new dataset called YouTube3D. Experiments show that YouTube3D is useful in training depth estimation networks and advances the state of the art of single-view depth estimation in the wild.

Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.

In this paper, we address the shape-from-shading problem by training deep networks with synthetic images. Unlike conventional approaches that combine deep learning and synthetic imagery, we propose an approach that does not need any external shape dataset to render synthetic images. Our approach consists of two synergistic processes: the evolution of complex shapes from simple primitives, and the training of a deep network for shape-from-shading. The evolution generates better shapes guided by the network training, while the training improves by using the evolved shapes. We show that our approach achieves state-of-the-art performance on a shape-from-shading benchmark.

The arising of domestic robots in smart infrastructure has raised demands for intuitive and natural interaction between humans and robots. To address this problem, a wearable wrist-worn camera (WwwCam) is proposed in this paper. With the capability of recognizing human hand gestures in real-time, it enables services such as controlling mopping robots, mobile manipulators, or appliances in smart-home scenarios. The recognition is based on finger segmentation and template matching. Distance transformation algorithm is adopted and adapted to robustly segment fingers from the hand. Based on fingers’ angles relative to the wrist, a finger angle prediction algorithm and a template matching metric are proposed. All possible gesture types of the captured image are first predicted, and then evaluated and compared to the template image to achieve the classification. Unlike other template matching methods relying highly on large training set, this scheme possesses high flexibility since it requires only one image as the template, and can classify gestures formed by different combinations of fingers. In the experiment, it successfully recognized ten finger gestures from number zero to nine defined by American Sign Language with an accuracy up to 99.38%. Its performance was further demonstrated by manipulating a robot arm using the implemented algorithms and WwwCam to transport and pile up wooden building blocks.

We study the problem of detecting human-object interactions (HOI) in static images, defined as predicting a human and an object bounding box with an interaction class label that connects them. HOI detection is a fundamental problem in computer vision as it provides semantic information about the interactions among the detected objects. We introduce HICO-DET, a new large benchmark for HOI detection, by augmenting the current HICO classification benchmark with instance annotations. To solve the task, we propose Human-Object Region-based Convolutional Neural Networks (HO-RCNN). At the core of our HO-RCNN is the Interaction Pattern, a novel DNN input that characterizes the spatial relations between two bounding boxes. Experiments on HICO-DET demonstrate that our HO-RCNN, by exploiting human-object spatial relations through Interaction Patterns, significantly improves the performance of HOI detection over baseline approaches., Comment: Accepted in WACV 2018

We propose TAL-Net, an improved approach to temporal action localization in video that is inspired by the Faster R-CNN object detection framework. TAL-Net addresses three key shortcomings of existing approaches: (1) we improve receptive field alignment using a multi-scale architecture that can accommodate extreme variation in action durations; (2) we better exploit the temporal context of actions for both proposal generation and action classification by appropriately extending receptive fields; and (3) we explicitly consider multi-stream feature fusion and demonstrate that fusing motion late is important. We achieve state-of-the-art performance for both action proposal and localization on THUMOS'14 detection benchmark and competitive performance on ActivityNet challenge., Comment: Accepted in CVPR 2018

Electromyographic (EMG) signal is playing an important role on hand function training as a neuromuscular rehabilitation tool. Various pattern recognition algorithms (PRAs) have been compared and evaluated in previous research, and Linear Discriminant Analysis (LDA) showed the higher offline accuracy for motion classification. However, it is rarely of comparison for different types of Discriminant Analysis (DA), and the surface electrodes are common methods for signal acquisition. This paper proposes to evaluate the offline performance of LDA and other types of DA, and using Myo armband for recording signals. The offline data was acquired by Myo armband, processing recognizing the data in BioPatRec, an open source platform for motion classification and hand prosthetics control. From the results of average offline accuracy, training time, and testing time of the five types, LDA and Quadratic Discriminant Analysis (QDA) have the better performance than others, and LDA is the fastest algorithm with simple computing.

State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by including an entirely separate temporal stream, indicating that the spatial stream is "missing" some of the signal captured by the temporal stream. In this work, we first investigate whether motion representations are indeed missing in the spatial stream of 3D CNNs. Second, we demonstrate that these motion representations can be improved by distillation, by tuning the spatial stream to predict the outputs of the temporal stream, effectively combining both models into a single stream. Finally, we show that our Distilled 3D Network (D3D) achieves performance on par with two-stream approaches, using only a single model and with no need to compute optical flow., Comment: Added link to code and models at https://www.jonathancstroud.com/d3d

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements., Comment: 43 pages, 16 figures. v3 includes additional comparisons with PASCAL VOC (per-category comparisons in Table 3, distribution of localization difficulty in Fig 16), a list of queries used for obtaining object detection images (Appendix C), and some additional references

Entry-level categories--the labels people use to name an object--were originally defined and studied by psychologists in the 1970s and 1980s. In this paper we extend these ideas to study entry-level categories at a larger scale and to learn models that can automatically predict entry-level categories for images. Our models combine visual recognition predictions with linguistic resources like WordNet and proxies for word "naturalness" mined from the enormous amount of text on the web. We demonstrate the usefulness of our models for predicting nouns (entry-level words) associated with images by people, and for learning mappings between concepts predicted by existing visual recognition systems and entry-level concepts. In this work we make use of recent successful efforts on convolutional network models for visual recognition by training classifiers for 7404 object categories on ConvNet activation features. Results for category mapping and entry-level category prediction for images show promise for producing more natural human-like labels. We also demonstrate the potential applicability of our results to the task of image description generation.

We study the problem of single-image depth estimation for images in the wild. We collect human annotated surface normals and use them to help train a neural network that directly predicts pixel-wise depth. We propose two novel loss functions for training with surface normal annotations. Experiments on NYU Depth, KITTI, and our own dataset demonstrate that our approach can significantly improve the quality of depth estimation in the wild.

We address the problem of temporal action localization in videos. We pose action localization as a structured prediction over arbitrary-length temporal windows, where each window is scored as the sum of frame-wise classification scores. Additionally, our model classifies the start, middle, and end of each action as separate components, allowing our system to explicitly model each actions temporal evolution and take advantage of informative temporal dependencies present in that structure. In this framework, we localize actions by searching for the structured maximal sum, a problem for which we develop a novel, provably-efficient algorithmic solution. The frame-wise classification scores are computed using features from a deep Convolutional Neural Network (CNN), which are trained end-to-end to directly optimize for a novel structured objective. We evaluate our system on the THUMOS 14 action detection benchmark and achieve competitive performance.

This paper presents the first study on forecasting human dynamics from static images. The problem is to input a single RGB image and generate a sequence of upcoming human body poses in 3D. To address the problem, we propose the 3D Pose Forecasting Network (3D-PFNet). Our 3D-PFNet integrates recent advances on single-image human pose estimation and sequence prediction, and converts the 2D predictions into 3D space. We train our 3D-PFNet using a three-step training strategy to leverage a diverse source of training data, including image and video based human pose datasets and 3D motion capture (MoCap) data. We demonstrate competitive performance of our 3D-PFNet on 2D pose forecasting and 3D pose recovery through quantitative and qualitative results., Comment: Accepted in CVPR 2017

The nonlinear motion state of object seriously affects the object tracking characteristics in complex motion scene. In this paper, we propose an interacting multiple model LK (IMM-LK) tracking algorithm to enhance the performance of tracking nonlinear moving object. LK tracking approach is based on the localized gradient obtaining stable optical-flow feature, based on LK, we build several motion models of the tracked object that interact with each other in the tracking process. The method extracts different model's object features, estimates the object state and calculates the matching rate of each model with the current motion model using theory of minimum variance. Combining with the optimal transfer matrix then we can track the nonlinear moving object. The proposed IMM-LK algorithm performs favorably against conventional LK tracking on the performance of tracking nonlinear moving object.

In this paper we introduce a new approach to phrase localization: grounding phrases in sentences to image regions. We propose a structured matching of phrases and regions that encourages the semantic relations between phrases to agree with the visual relations between regions. We formulate structured matching as a discrete optimization problem and relax it to a linear program. We use neural networks to embed regions and phrases into vectors, which then define the similarities (matching weights) between regions and phrases. We integrate structured matching with neural networks to enable end-to-end training. Experiments on Flickr30K Entities demonstrate the empirical effectiveness of our approach.

This work introduces a novel convolutional network architecture for the task of human pose estimation. Features are processed across all scales and consolidated to best capture the various spatial relationships associated with the body. We show how repeated bottom-up, top-down processing used in conjunction with intermediate supervision is critical to improving the performance of the network. We refer to the architecture as a “stacked hourglass” network based on the successive steps of pooling and upsampling that are done to produce a final set of predictions. State-of-the-art results are achieved on the FLIC and MPII benchmarks outcompeting all recent methods.

We introduce a new benchmark "Humans Interacting with Common Objects" (HICO) for recognizing human-object interactions (HOI). We demonstrate the key features of HICO: a diverse set of interactions with common object categories, a list of well-defined, sense-based HOI categories, and an exhaustive labeling of co-occurring interactions with an object category in each image. We perform an in-depth analysis of representative current approaches and show that DNNs enjoy a significant edge. In addition, we show that semantic knowledge can significantly improve HOI recognition, especially for uncommon categories.

Human actions capture a wide variety of interactions between people and objects. As a result, the set of possible actions is extremely large and it is difficult to obtain sufficient training examples for all actions. However, we could compensate for this sparsity in supervision by leveraging the rich semantic relationship between different actions. A single action is often composed of other smaller actions and is exclusive of certain others. We need a method which can reason about such relationships and extrapolate unobserved actions from known actions. Hence, we propose a novel neural network framework which jointly extracts the relationship between actions and uses them for training better action retrieval models. Our model incorporates linguistic, visual and logical consistency based cues to effectively identify these relationships. We train and test our model on a largescale image dataset of human actions. We show a significant improvement in mean AP compared to different baseline methods including the HEX-graph approach from Deng et al. [8].

Affordances are fundamental attributes of objects. Affordances reveal the functionalities of objects and the possible actions that can be performed on them. Understanding affordances is crucial for recognizing human activities in visual data and for robots to interact with the world. In this paper we introduce the new problem of mining the knowledge of semantic affordance: given an object, determining whether an action can be performed on it. This is equivalent to connecting verb nodes and noun nodes in WordNet, or filling an affordance matrix encoding the plausibility of each action-object pair. We introduce a new benchmark with crowdsourced ground truth affordances on 20 PASCAL VOC object classes and 957 action classes. We explore a number of approaches including text mining, visual mining, and collaborative filtering. Our analyses yield a number of significant insights that reveal the most effective ways of collecting knowledge of semantic affordances.

This paper proposes a novel geometric statistical measurement of long sequence moving objects, which can accurately measure the geometry of the moving objects in non-contact measurement environment. The proposed algorithm adopts detecting-learning method for tracking moving objects in a long-term, gets the moving sequence data, extracts the geometric contour and computes the geometric and motion parameters of the objects. Then we analyze the long sequence to train the parameters. Experimental data showed that the adoption of geometric measurement of moving objects based on detecting-learning mechanism performs favorably. The method can provide high-accuracy geometric and motion parameters of the objects.

We consider classification problems in which the label space has structure. A common example is hierarchical label spaces, corresponding to the case where one label subsumes another (e.g., animal subsumes dog). But labels can also be mutually exclusive (e.g., dog vs cat) or unrelated (e.g., furry, carnivore). To jointly model hierarchy and exclusion relations, the notion of a HEX (hierarchy and exclusion) graph was introduced in [7]. This combined a conditional random field (CRF) with a deep neural network (DNN), resulting in state of the art results when applied to visual object classification problems where the training labels were drawn from different levels of the ImageNet hierarchy (e.g., an image might be labeled with the basic level category "dog", rather than the more specific label "husky"). In this paper, we extend the HEX model to allow for soft or probabilistic relations between labels, which is useful when there is uncertainty about the relationship between two labels (e.g., an antelope is "sort of" furry, but not to the same degree as a grizzly bear). We call our new model pHEX, for probabilistic HEX. We show that the pHEX graph can be converted to an Ising model, which allows us to use existing off-the-shelf inference methods (in contrast to the HEX method, which needed specialized inference algorithms). Experimental results show significant improvements in a number of large-scale visual object classification tasks, outperforming the previous HEX model., Comment: International Conference on Computer Vision (2015)

A continuous-time hidden Markov model is considered where the dynamics of the hidden process change at a random 'change point' tau. Closed form recursive estimates for the conditional distribution of the hidden process and the change point tau are obtained, given the observations. (c) 2012 Elsevier B.V. All rights reserved. Refereed/Peer-reviewed

This paper addresses the problem of fine-grained recognition: recognizing subordinate categories such as bird species, car models, or dog breeds. We focus on two major challenges: learning expressive appearance descriptors and localizing discriminative parts. To this end, we propose an object representation that detects important parts and describes fine grained appearances. The part detectors are learned in a fully unsupervised manner, based on the insight that images with similar poses can be automatically discovered for fine-grained classes in the same domain. The appearance descriptors are learned using a convolutional neural network. Our approach requires only image level class labels, without any use of part annotations or segmentation masks, which may be costly to obtain. We show experimentally that combining these two insights is an effective strategy for fine-grained recognition.

Entry level categories - the labels people will use to name an object - were originally defined and studied by psychologists in the 1980s. In this paper we study entry-level categories at a large scale and learn the first models for predicting entry-level categories for images. Our models combine visual recognition predictions with proxies for word "naturalness" mined from the enormous amounts of text on the web. We demonstrate the usefulness of our models for predicting nouns (entry-level words) associated with images by people. We also learn mappings between concepts predicted by existing visual recognition systems and entry-level concepts that could be useful for improving human-focused applications such as natural language image description or retrieval.

While 3D object representations are being revived in the context of multi-view object class detection and scene understanding, they have not yet attained wide-spread use in fine-grained categorization. State-of-the-art approaches achieve remarkable performance when training data is plentiful, but they are typically tied to flat, 2D representations that model objects as a collection of unconnected views, limiting their ability to generalize across viewpoints. In this paper, we therefore lift two state-of-the-art 2D object representations to 3D, on the level of both local feature appearance and location. In extensive experiments on existing and newly proposed datasets, we show our 3D object representations outperform their state-of-the-art 2D counterparts for fine-grained categorization and demonstrate their efficacy for estimating 3D geometry from images via ultra-wide baseline matching and 3D reconstruction.

The growth of detection datasets and the multiple directions of object detection research provide both an unprecedented need and a great opportunity for a thorough evaluation of the current state of the field of categorical object detection. In this paper we strive to answer two key questions. First, where are we currently as a field: what have we done right, what still needs to be improved? Second, where should we be going in designing the next generation of object detectors? Inspired by the recent work of Hoiem et al. on the standard PASCAL VOC detection dataset, we perform a large-scale study on the Image Net Large Scale Visual Recognition Challenge (ILSVRC) data. First, we quantitatively demonstrate that this dataset provides many of the same detection challenges as the PASCAL VOC. Due to its scale of 1000 object categories, ILSVRC also provides an excellent test bed for understanding the performance of detectors as a function of several key properties of the object classes. We conduct a series of analyses looking at how different detection methods perform on a number of image-level and object-class-level properties such as texture, color, deformation, and clutter. We learn important lessons of the current object detection methods and propose a number of insights for designing the next generation object detectors.