Search

HighlightsWhile freshness is a critical value of food quality, its assessment requires complex methods, which are costly and time-consuming.In this work, it is demonstrated that spectral responses obtained from a portable VIS/NIR imaging spectrometer can be used to predict food freshness using a CNN-based machine learning algorithm.In the food industry, the method can assess real-time food freshness nondestructively and cost-effectively.Abstract. There has been strong demand for the development of accurate but simple methods to assess the freshness of foods. In this study, a system is proposed to determine the freshness of fish by analyzing the spectral response with a portable visible/near-infrared (VIS/NIR) imaging spectrometer and a convolution neural network (CNN) machine learning algorithm. Spectral response data from salmon and tuna, which were incubated at 25°C, were obtained every minute for 30 h and were categorized into three stages (fresh, likely spoiled, or spoiled) based on the time and pH. Using the obtained spectral data, a CNN-based machine learning algorithm was built to evaluate the freshness of the experimental samples. The accuracy of the spectral data in predicting the freshness was ~84% for salmon and ~88% for tuna. Keywords: CNN, Fish, Freshness, pH, Spectral data, VIS/NIR.

Highlights The hidden internal damage of falling impact on tomatoes will reduce the quality of products. Hyperspectral imaging and VIS/NIR spectrum analysis, including wavelength selection and classification model construction, have the possibility as a non-destructive and fast method to predict the effect of drop impact grades on tomato bruising damage. Abstract . Mechanical damage usually causes hidden internal damage to tomatoes (Solanum lycopersicum L.), which can reduce the product quality and can cause economic losses to farmers. The visible and near-infrared (VIS/NIR) spectra of tomato fruits were analyzed by using the wavelength selection algorithm (the combination of ant colony optimization and variable importance in projection), and the influence of impact grades of simulated transport on tomato fruit bruising was evaluated. A VIS/NIR hyperspectral imaging system was developed to capture hyperspectral images of tomatoes from 392–1034 nm spectral region and the part used in actual data analysis was 442-984 nm. Multivariate analysis classifier models (partial least squares discrimination analysis and ANN) were set up based on the original spectral dataset. On the basis of selected wavelength intervals, multivariate analysis classifier models were re-established. The overall classification accuracies of all models in the validation set are good, ranging from 64.29% to 100%. Especially in the two types of classification (bruising and normal), the range of correct accuracy is 89.29% to 100%, which shows very high predicted performance. The prediction performance of the model based on the selected wavelengths decreases slightly, but the prediction time is shortened by more than 70%. The results demonstrated that hyperspectral imaging and VIS/NIR spectrum analysis, including wavelength selection and classification model construction, have the possibility as a non-destructive and fast method to predict the effect of drop impact grades on tomato bruising damage. Keywords: ANN, Ant colony optimization (ACO), Partial least squares discrimination analysis (PLS-DA), Variable importance in projection (VIP), VIS/NIR hyperspectral imaging system, wavelength interval selection.

Conventional deep learning based methods for object detection require a large amount of bounding box annotations for training, which is expensive to obtain such high quality annotated data. Few-shot object detection, which learns to adapt to novel classes with only a few annotated examples, is very challenging since the fine-grained feature of novel object can be easily overlooked with only a few data available. In this work, aiming to fully exploit features of annotated novel object and capture fine-grained features of query object, we propose Dense Relation Distillation with Context-aware Aggregation (DCNet) to tackle the few-shot detection problem. Built on the meta-learning based framework, Dense Relation Distillation module targets at fully exploiting support features, where support features and query feature are densely matched, covering all spatial locations in a feed-forward fashion. The abundant usage of the guidance information endows model the capability to handle common challenges such as appearance changes and occlusions. Moreover, to better capture scale-aware features, Context-aware Aggregation module adaptively harnesses features from different scales for a more comprehensive feature representation. Extensive experiments illustrate that our proposed approach achieves state-of-the-art results on PASCAL VOC and MS COCO datasets. Code will be made available at https://github.com/hzhupku/DCNet., Accepted by CVPR2021

Recent works have made great progress in semantic segmentation by exploiting contextual information in a local or global manner with dilated convolutions, pyramid pooling or self-attention mechanism. However, few works have focused on harvesting boundary information to improve the segmentation performance. In order to enhance the feature similarity within the object and keep discrimination from other objects, we propose a boundary-aware graph convolution (BGC) module to propagate features within the object. The graph reasoning is performed among pixels of the same object apart from the boundary pixels. Based on the proposed BGC module, we further introduce the Boundary-aware Graph Convolution N et-work(BGCN et), which consists of two main components including a basic segmentation network and the BGC module, forming a coarse-to-fine paradigm. Specifically, the BGC module takes the coarse segmentation feature map as node features and boundary prediction to guide graph construction. After graph convolution, the reasoned feature and the input feature are fused together to get the refined feature, producing the refined segmentation result. We conduct extensive experiments on three popular semantic segmentation benchmarks including Cityscapes, PASCAL VOC 2012 and COCO Stuff, and achieve state-of-the-art performance on all three benchmarks.

Attachment is a deep and enduring emotional bond that connects one person to another across time and space. Our early attachment styles are established in childhood through the interaction between infants and caregivers. There are two attachment types, secure and insecure. The attachment experience affects personality development, particularly a sense of security, and research shows that it influences the ability to form stable relationships throughout life. It is also an important aspect of assessing the quality of parenting. Therefore, attachment has been widely studied in psychology research. It’s usually acquired by Ainsworth’s Strange Situation Assessment (SSA) through tedious observation. As far as we know, there is no computational method to predict infant attachment type. We try to use the Still-Face Paradigm (SFP) video and audio as input to predict attachment types through machine learning methods. In the present work, we recruited 64 infant-mother participants, collected videos of SFP when babies are 5–8 months of age and identified their attachment types including secure and insecure by SSA when those infants are almost 2 years old. For the visual part, we extract motion features and apply a RNN network with LSTM units model for classification. For the audio part, speech enhancement is conducted as data pre-processing, pitch frequency, short-time energy and Mel Frequency Cepstral Coefficient feature sequences are extracted. Then SVM is deployed to explore the patterns in it. The experiments show that our method is able to discriminate between the 2 classes of subjects with a good accuracy.

Sparse representation-based methods have been successfully applied to visual tracking. However, complex and inefficient optimization limits their deployment in practical tracking scenarios. In this paper, we propose a temporal consistency dictionary learning tracking algorithm to enable efficient dictionary learning and tracking executive. First, we present an objective function which introduces the fixed dictionary and variance dictionary to reconstruct the object’s appearance. In particular, the proposed method takes the temporal consistency into account by adding a regularization term into the objective function to constrain the difference of object appearance at adjacent frames. Then the optimization problem is solved in an iteration way. Moreover, the proposed method can encode the object’s local structural information, and the local patches from the same candidate altogether for a global appearance representation. Second, we develop an effective observation likelihood function based on the proposed model. It takes the influence of patches with large reconstruction errors into consideration, thereby, alleviating the drifting of the object. Finally, we present an appearance updating strategy to adapt to the object’s appearance variations by the online dictionary learning. Experimental evaluations on the TB50 and TB100 datasets show that the proposed tracking method outperforms sparse representation related visual tracking as well as other state-of-the-art tracking methods.

This paper proposes a system for recognition of infants' gaze behaviors and emotions from videos. In the current work, researchers believed that the information of eye region is crucial for gaze behavior recognition, and emotion recognition mostly depends on the appearance of face. However, because the differentiation of all parts of infant's body has not finished yet, we found that infants always express their intentions and emotions using their whole body, especially moving their heads. Therefore, we incorporate the head pose information as features into the gaze behavior recognition, and we extract the gaze features to improve the recognition of infants' emotions. In addition, we combine several Deep Neural Networks, which can not only capture the details of the images very well, but also make full use of the temporal features. In order to recognize infants' gaze behaviors, we design a feature-extraction convolutional neural network which can obtain the features of infants' gaze direction, then we feed these features with head pose into the next gaze behavior recurrent neural network. Moreover, we combine the features of facial express and gaze behavior to characterize infants' emotions, and expand this system with an emotion recurrent neural network. In the end, we achieve the recognition accuracy 98.31% and 94.71% respectively on our data set.

Children's social gaze behavior modeling and evaluation has obtained increasing attentions in various research areas. In psychology research, eye gaze behavior is very important to developmental disorders diagnosis and assessment. In robotics area, gaze interaction between children and robots also draws more and more attention. However, there exists no specific gaze estimator for children in social interaction context. Current approaches usually use models trained with adults' data to estimate children's gaze. Since gaze behaviors and eye appearances of children are different from those of adults, the current approaches, especially those with free-calibration assumptions which are utilized in usual human-robot interaction systems, will result in big errors. Note that children data is difficult to collect and label, so directly learning from children data is hard to achieve. We propose a new system to solve this problem, which combines a CNN feature extractor trained from adult data and a domain adaptation unit using geodesic flow kernel to adapt the source domain (adults) classifier to the target domain (children). Our system performs well in children's gaze estimation.

To predict and assess visual attention, saliencybased visual attention modeling is a popular approach. However, state-of-the-art models are developed for adults, in which children are not considered. Additionally, these models consider neither social cues like face, nor attention learning cues. The face is a vital part of visual attention. Psychological studies reveal that sub-facial areas are different in visual attention. Some models highlight faces in social scenes, but sub-facial areas are not taken into account. Attention learning reveals internal processing of visual attention. By learning how the cognitive system deals with visual stimuli, it is possible to predict visual attention behavior. In this paper, we propose a multilevel visual attention model to predict fixations of children when watching a talking face. Based on traditional saliency maps, the proposed model includes both separate facial areas and attention shift cue. An eye-tracking experiment is conducted to evaluate the model. Results show that the proposed model significantly outperforms conventional models in talking face scenes.

This study proposes a multi-level trajectory analysis method for modeling traffic behavior from an ego-centric view, where on-road vehicle trajectories are collected based on the authors' previous studies of an on-board system consisting of multiple 2D lidar sensors. From an input set of trajectories, a set of hot regions (topics) that trajectory points most frequently present are first discovered using a sticky HDP-HMM; then, the major paths of the trajectories' transitions across different hot regions are extracted by recursively mining frequent subsequences of topics; and finally, paths are modeled using a hierarchical hidden Markov model (HHMM), where the intra-path dynamics is represented using an HMM, in which each state corresponds to a hot region, while the inter-path transition is assumed to be Markovian. The model could be used for behavior prediction, i.e. whenever a vehicle is detected in a scene, predicting which route it will probably follow and how its trajectory will probably develop over time, which is essential to interpreting the potential risks for longer time horizons. Experiments are conducted using a large set of vehicle trajectories collected from motorways in Beijing, and promising results are presented.

Effective video-based detection methods are of great importance to intelligent transportation systems (ITS), and here we propose a method to localize and label objects. The method is able to detect pedestrians and bicycle riders in a complex scene. Our method is inspired by the common fate principle, which is a mechanism of visual perception in human beings, and which states tokens moving or functioning in a similar manner tend to be perceived as one unit. Our method embeds the principle in an Implicit Shape Model (ISM). In our method, keypoint-based object parts are firstly detected and then grouped by their motion patterns. Based on the grouping results, when the object parts vote for object centers and labels, each vote belonging to the same object part is assigned a weight according to its consistency with the votes of other object parts in the same motion group. Afterwards, the peaks, which correspond to detection hypotheses on the Hough image formed by summing up all weighted votes, become easier to find. Thus our method performs better in both position and label estimations. Experiments show the effectiveness of our method in terms of detection accuracy.

Successful multi-target tracking requires locating the targets and labeling their identities. For a laser-based tracking system, this mission becomes significantly challenging when the targets are in close proximity or frequently interact with one another. This paper presents a novel on-line learning-based method for laser-based multi-target tracking. When the targets do not interact with one another, multiple independent trackers are employed to train a classifier for each target. When the targets are in close proximity, the learned classifiers are used to assist in tracking. The tracking and learning supplement each other in the proposed method, which is helpful in dealing with difficult problems encountered in laser-based multi-target tracking; moreover, it ensures that the entire process can be completely automatic and available on-line. Various evaluations have demonstrated that this method performs better than previous methods when interactions occur, and it can maintain the correct tracking under various complex tracking situations.

Tracking generic human motion is highly challenging due to its high-dimensional state space and the various motion types involved. In order to deal with these challenges, a fusion formulation which integrates low- and high-dimensional tracking approaches into one framework is proposed. The low-dimensional approach successfully overcomes the high-dimensional problem of tracking the motions with available training data by learning motion models, but it only works with specific motion types. On the other hand, although the high-dimensional approach may recover the motions without learned models by sampling directly in the pose space, it lacks robustness and efficiency. Within the framework, the two parallel approaches, low- and high-dimensional, are fused via a probabilistic approach at each time step. This probabilistic fusion approach ensures that the overall performance of the system is improved by concentrating on the respective advantages of the two approaches and resolving their weak points. The experimental results, after qualitative and quantitative comparisons, demonstrate the effectiveness of the proposed approach in tracking generic human motion.

For reasons of public security, an intelligent surveillance system that can cover a large, crowded public area has become an urgent need. In this article, we propose a novel laser-based system that can simultaneously perform tracking, semantic scene learning, and abnormality detection in a fully online and unsupervised way. Furthermore, these three tasks cooperate with each other in one framework to improve their respective performances. The proposed system has the following key advantages over previous ones: (1) It can cover quite a large area (more than 60×35m), and simultaneously perform robust tracking, semantic scene learning, and abnormality detection in a high-density situation. (2) The overall system can vary with time, incrementally learn the structure of the scene, and perform fully online abnormal activity detection and tracking. This feature makes our system suitable for real-time applications. (3) The surveillance tasks are carried out in a fully unsupervised manner, so that there is no need for manual labeling and the construction of huge training datasets. We successfully apply the proposed system to the JR subway station in Tokyo, and demonstrate that it can cover an area of 60×35m, robustly track more than 150 targets at the same time, and simultaneously perform online semantic scene learning and abnormality detection with no human intervention.

Multitarget tracking becomes significantly more challenging when the targets are in close proximity or frequently interact with each other. This article presents a promising online system to deal with these problems. The novelty of this system is that laser and vision are integrated with tracking and online learning to complement each other in one framework: when the targets do not interact with each other, the laser-based independent trackers are employed and the visual information is extracted simultaneously to train some classifiers online for “possible interacting targets”. When the targets are in close proximity, the classifiers learned online are used alongside visual information to assist in tracking. Therefore, this mode of cooperation not only deals with various tough problems encountered in tracking, but also ensures that the entire process can be completely online and automatic. Experimental results demonstrate that laser and vision fully display their respective advantages in our system, and it is easy for us to obtain a good trade-off between tracking accuracy and the time-cost factor.

In our previous work, we reported a system that monitors an intersection using a network of horizontal laser scanners. This paper focuses on an algorithm for moving-object detection and tracking, given a sequence of distributed laser scan data of an intersection. The goal is to detect each moving object that enters the intersection; estimate state parameters such as size; and track its location, speed, and direction while it passes through the intersection. This work is unique, to the best of the authors' knowledge, in that the data is novel, which provides new possibilities but with great challenges; the algorithm is the first proposal that uses such data in detecting and tracking all moving objects that pass through a large crowded intersection with focus on achieving robustness to partial observations, some of which result from occlusions, and on performing correct data associations in crowded situations. Promising results are demonstrated using experimental data from real intersections, whereby, for 1063 objects moving through an intersection over 20 min, 988 are perfectly tracked from entrance to exit with an excellent tracking ratio of 92.9%. System advantages, limitations, and future work are discussed.

This work is motivated by the following two potential applications: 1) enhancing driving safety and 2) collecting traffic data in a large dynamic urban environment. A laser-scanner-based approach is proposed. The problem is formulated as a simultaneous localization and mapping (SLAM) with object tracking and classification, where the focus is on managing a mixture of data from both dynamic and static objects in a highly dynamic environment. A trajectory-oriented closure is also proposed using the sporadically available global positioning system (GPS) measurements in urban areas to assist for global accuracy, particularly when the vehicle makes a noncyclical measurement in a large outdoor environment. Experiments are conducted using the data that were collected along a course near 4.5 km in a highly dynamic environment. Possibilities of the approaches toward the two potential applications are demonstrated, and avenues for future works are discussed.

In this research, a novel system for monitoring an intersection using a network of single-row laser range scanners (subsequently abbreviated as "laser scanner") and video cameras is proposed. Laser scanners are set on the road side to profile an intersection horizontally from different viewpoints. The contour points of moving objects are captured at a certain horizontal plane with a high scanning rate (e.g., 37 Hz). A laser-based processing algorithm is developed, thus the moving objects entered the intersection are detected and tracked to estimate their state parameters, such as: location, speed, and direction at each time instance. In addition, laser data and processing results are forwarded to an associated video camera, so that a visualization as well as fusion-based processing can be achieved. An experiment in central Beijing is presented, demonstrating that a large quantity of physical dimension and detailed traffic data can be obtained through such a system.

Inspite extensive research on visual tracking of multiple people in computer vision area, the robustness and usability of visual trackers are still discouraging. Recently, a few laser-based detection and tracking methods have been developed in robotics area. However, poor features provided by laser data make the tracker fail in many situations. In this paper, we present a novel method that aims at reliably detecting and tracking multiple people in an open area. Multiple laser scanners and one camera are used as input sensors. In detection stage, laser-based detection algorithm captures newly appeared people and initializes the mean-shift-based visual tracker. In tracking stage, laser-based feet trajectory tracking result and visual body region tracking result are combined with a decision-level Bayesian fusion method. The experimental results demonstrate reliable and real-time performance of the method.

Laser-based people tracking systems have been developed for mobile robotic, and intelligent surveillance areas. Existing systems rely on laser point clustering method to extract object locations. However, for dense crowd tracking, laser points of different objects are often interlaced and undistinguishable due to measurement noise and they can not provide reliable features. It causes current systems quite fragile and unreliable. This paper presents a novel and robust laser-based dense crowd tracking method. Firstly, we introduce a stable feature extraction method based on accumulated distribution of successive laser frames. With this method, the noise that generates split and merged measurements is smoothed away and the pattern of rhythmic swing legs is utilized to extract each leg of persons. And then, a region coherency property is introduced to construct an efficient measurement likelihood model. The final tracker is based on the combination of independent Kalman filter and Rao-Blackwellized Monte Carlo data association filter (RBMC-DAF). In real experiments, we obtain raw data from multiple registered laser scanners, which measure two legs for each people on the height of 16cm from horizontal ground. Evaluation with real data shows that the proposed method is robust and effective. It achieves a significant improvement compared with existing laser-based trackers. In addition, the proposed method is much faster than previous works, and can overcome tracking errors resulted from mixed data of two closely situated persons.

Visual hand gesture based interfaces have been widely used for navigation of virtual environments and control of a robot arm in robotic systems. In fact, hand movement has abundant powers of expression with complex finger joint motions. In this paper, the full degree-of-freedom hand motion is tracked in a vision-based mode towards natural and intuitive control of a dexterous robotic hand. The main obstacle to handle human hand posture tracking is the high dimensionality. A novel three-dimensional model-based approach is proposed. It utilizes a genetic algorithm-based global optimization procedure after each particle-filtering step, which deals with the tracking in high-dimensional and multi-modal state space reasonably well. Experimental results show that present approach improves performance of hand posture tracking significantly and provides a feasible solution for guiding a dexterous hand.

International audience; Head pose is an important indicator of a person's visual focus of attention (VFoA). A traditional way to recognize VFoA is to consider accurate head pose or gaze estimations. However, these estimations usually degrade drastically in middle or low resolution video data. In this paper, a joint estimation of head pose and VFoA is proposed to address this issue; both head pose and VFoA are iteratively refined until convergence. This approach is evaluated in a specific scenario involving children around a table playing together with toys. Datasets are acquired and annotated by psychologists in Peking university. The experimental results demonstrate the usefulness of the join estimation process to recognize visual focus of attention in middle resolution video sequences.

International audience; Precise localization is an essential issue for autonomous driving applications, where GPS-based systems are challenged to meet requirements such as lane-level accuracy. This paper introduces a new visual-based localization approach in dynamic traffic environments, focusing on and exploiting properties of structured roads like straight roads or intersections. Such environments show several line segments on lane markings, curbs, poles, building edges, etc., which demonstrate the road's longitude, latitude and vertical directions. Based on this observation, we define a Road Structural Feature (RSF) as sets of segments along three perpendicular axes together with feature points. At each video frame, the proper road structure (or multiple road structures in case of an intersection) is predicted based on the geometric information given by a 2D map. The RSF is then detected from line segments and points extracted from the image, and used to estimate the pose of the vehicle. Experiments are conducted using video streams collected on major roads in downtown Beijing, which are structured and with intense dynamic traffic. GPS/IMU data have been collected and synchronized with the video streams as a reference in validation. The results show good performance compared with that of a more traditional visual odometry method. Future work will be addressed on using visual approach to improve GPS localization accuracy.

Labeling children's play behavior is an important process in children's social behavior analysis which is traditionally done by experienced coders. With the growing volume of data, automatic methods for labeling are increasingly required. This paper presents a novel method to label children's social behavior automatically in peer-play scenarios based on visual attention and mutual interaction computation. In this method, the discrete distribution of children's visual attention is computed based on face pose estimation. Then, the mutual interaction among children is calculated by "Attention Processes", which are continuous periods of time during which a certain child pays attention to the same target. After that, "Solitary Feature" and "Group Feature" are extracted by the mutual interaction of children and children's play behaviors will be classified into 3 types ("Solitary Play", "Parallel Play" and "Group Play") by these two kinds of features. Finally, this method is evaluated by a dataset of children's peer-play scenarios and the results show this method has a good performance in our dataset.

Gaze is a powerful cue for children's social behavior analysis. In this paper, a novel method is proposed to estimate children's gaze orientation in the experimental data of developmental psychology based on head pose estimation. In consideration of the possible errors of head pose estimation results, temporal information and potential targets are both introduced to improve the results of gaze estimation. At last, this method is evaluated by a dataset of children's peer-play scenarios and the results show that this method has a good performance. According to the experimental valuation and analysis, in a certain peer-play scenario, potential targets are powerful spatial cues for children's gaze estimation and temporal information also provides some cues to improve the estimation results.

It has become a well-known technology that 3D measurement of a large environment could be achieved by using a number of 2D LIDARs on a mobile platform. In such a system, calibration is essential for making collaborative use of different LIDAR data, while existing methods usually require modifications to the environments, such as putting calibration targets, or rely on special facilities, which is labor intensive and put many restrictions to potential applications. This research aims at developing a calibration method for multiple 2D LIDAR sensing systems, which could be conducted in a general outdoor environment using the features of a nature scene. Special focus is cast on solving the noisy sensing in a complex environment and the occlusions caused by largely different sensor viewpoints. A multi-type geometric feature based calibration algorithm is proposed, which extracts the features such as points, lines, planes and quadrics from the 3D points of each LIDAR sensing. Transformation parameters from each sensor to the frame on a moving platform is estimated by matching the multi-type features. Experiments are conducted using the data sets of an intelligent vehicle platform (POSS-V) through a driving in the campus of Peking University. Results of calibrating two LIDAR sensors with largely different viewpoints are presented, and the accuracy and robustness concerning noisy feature extractions are examined intensively.

Intention prediction is an active research topic, however, few work make use of the head pose. This research investigates the usefulness of head pose and pose change features into a Discrete Dynamic Bayesian Network to predict pedestrian's intention. In particular, we first focus on scenarios within a shopping center (using the standard CAVIAR database), to predict whether a pedestrian will enter a shop or not. We assume that different head pose patterns reflect different intentions. Thus, a hierarchical clustering method is used to process the histogram of head pose and the change of head pose. Then the clustering results are used as features feeding into a dynamic bayesian network. Experiments have been conducted to demonstrate the importance of using head pose in intention prediction and it shows a fairly good result which use only the information derived from pedestrian themselves regardless of the complex environment. The proposed approach is simple to use and makes it easy to integrate other features.

Tracking of pedestrians from a moving vehicle equipped with a monocular camera is still considered as a challenging problem in the fields of both computer vision and robotics. In this paper, we address this problem in a particle filter framework, which well incorporates different cues from detector, dynamic model and target-specific tracking. In order to eliminate the effect of ego-motion when predicting the movement of pedestrians, we train one dynamic model for each driving behavior (moving forward, turning left/right) given a set of training trajectories. The learnt dynamic model is then utilized to predict the future movement of the pedestrian in the tracking process. We demonstrate our system works robustly on challenging dataset with strong illumination changes.

This research aims at generating an omnidirectional perception at the host vehicle's surroundings, extracting accurate and continuous motion trajectories of the nearby vehicles using low cost laser scanners. A system of detecting and tracking on-road vehicles using multiple laser scanners is developed, where focuses are cast on solving data association of simultaneous measurements from multiple sensors at different viewpoints, and state estimation in case of partial observations in dense dynamic situations. Experimental results in freeways in Beijing are presented, system efficiency is demonstrated, where motion trajectories describing driving behaviors such as overtaking, lane changing and other interactions between driving objects are captured. In addition, the accuracy in vehicle detection and tracking is examined using a reference vehicle with a ground truth GPS.

Two challenging issues for object detection are how to separate near objects and how to separate similar different-class objects. Learned that during human's vision perception, tokens moving or functioning in a similar manner are perceived as one unit, stated by the common fate principle, we propose a method to detect objects of multiple classes. Our method extends the Implicit Shape Model (ISM) to incorporate motion grouping results of object parts, and meets the challenges. Keypoint-based object parts are firstly detected and then grouped by the similarities of their corresponding trajectories which are traced by keypoint tracking. The grouping results are combined into a Hough transform framework. In Hough transform based methods, each object part votes for object centers and labels according to a trained codebook. In our method, the votes are assigned different weights according to the motion grouping results. One vote is assigned larger weight if it has larger consistence with the votes of other object parts in the same motion group. In such a manner, peaks in the formed Hough images which correspond to object hypotheses become easier to find. And our method gains improvement in both object position and label estimation. Experiments are provided to show the merit in terms of detection accuracy.

Complex human activity detection is a challenging problem, especially when people interact with each other. Approaches utilizing local spatio-temporal features work well with background clutter, scale and illumination changing. However, most of them focus on classifying short video sequences. In real world applications such as surveillance, it's hard to get the well segmented video clip to classify. So how to detect and localize complex human activities in unsegmented videos is a problem need to be solved. In this paper, based on the local spatio-temporal feature, we propose a variation of Hough Voting method using the Implicit Shape Model which can localize and recognize complex human activity simultaneously. Our approach is tested on the UT-Interaction dataset, and demonstrates promising results in complex human activity detection and localization.

The study of traffic behavior at intersections is an important research direction in the field of intelligent transportation system (ITS). It is considered very practical to reduce accident, solve traffic jam and improve its accessibility. This paper proposed a trajectory analysis method for vehicles passing an intersection. The motion trajectories of vehicles were obtained by a traffic data collecting system using a network of single-row laser scanners. The intersection could be described by a group of route models which are built based on trajectory clustering, so large amount of trajectories could be classified to different routes and abnormal ones could also be detected. Experimental results based on real trajectories obtained from an intersection in Beijing show the validity of our proposed method.

Abnormal activity detection plays a crucial role in surveillance applications, and such system has become an urgent need for public security. In this paper, we propose a novel laser-based system, which can perform the online detection of abnormal activity with an unsupervised way. The proposed system has the following key features that make it advantageous over previous ones: (1) It can cover quite a large and crowded area, such as subway station, public square, intersection and etc. (2) The overall system can vary with time period, incrementally learn the behavior pattern of pedestrians and perform the fully online detection of abnormal activity. This feature makes our system be quite suitable for the real-time applications. (3) The abnormal activity detection is carried out with a fully unsupervised way, there is no need for manual labelling and constructing the huge training datasets. We successfully applied the proposed system into the JR subway station of Tokyo, which can cover a 60×35m area, track more 150 targets at the same time and simultaneously perform the robust detection of abnormal activity with no human intervention.

Working environments of modern robots have changed to unstructured, dynamic and outdoor scenes. There emerged several new challenges along with these changes, mainly in perception of both static and dynamic objects of the scenes. To tackle these new challenges, this research focused on study of advanced perception systems that can simultaneously model static scenes and track dynamic objects. Our research has three features. Multi-view and multi-type sensors, together with machine learning based algorithms, are utilized to obtain robust and reliable mapping/ tracking results. In addition, a car-based mobile perception system is developed for exploring large sites. Finally, to improve robustness of the multi-view and mobile perception system, some new camera calibration methods are proposed. This paper presents an overview of our recent study on above mentioned ideas and technologies. Specifically we will focus on multi-sensor based multiple target tracking, simultaneous 3D mapping and target tracking in a mobile platform, and camera calibration.

Kids activity means a lot to their parents, and in the analysis of the activities, video retrieval has played an important role. In this paper, we propose an effective approach for the retrieval of the kid's activities from home videos. The video sequences are taken from our test-bed environment that is designed in the form of a smart home, and feature various types of child-adult interactions. We present a novel retrieval method with two steps, first using spatio-temporal matching to obtain a coarse result, And then we propose a method to learn dominant child-adult interactive behaviors based on a sequence of home videos. Based on these dominant behaviors, we get rid of some false retrieval and obtain fine result. We implement and test our methodology on a newly-introduced dataset containing several types of kid's activities, and the retrieval result shows its potential application in the video analysis demain, it can find out most of the video clips relevant to the query one.

Learning the knowledge of scene structure and tracking a large number of targets are both active topics of computer vision in recent years, which plays a crucial role in surveillance, activity analysis, object classification and etc. In this paper, we propose a novel system which simultaneously performs the Learning-Semantic-Scene and Tracking, and makes them supplement each other in one framework. The trajectories obtained by the tracking are utilized to continually learn and update the scene knowledge via an online un-supervised learning. On the other hand, the learned knowledge of scene in turn is utilized to supervise and improve the tracking results. Therefore, this “adaptive learning-tracking loop” can not only perform the robust tracking in high density crowd scene, dynamically update the knowledge of scene structure and output semantic words, but also ensures that the entire process is completely automatic and online. We successfully applied the proposed system into the JR subway station of Tokyo, which can dynamically obtain the semantic scene structure and robustly track more than 150 targets at the same time.

Multi-target tracking becomes significantly more challenging when the targets are in close proximity or frequently interact with each other. This paper presents a promising tracking system to deal with these problems. The novelty of this system is that laser and vision, tracking and learning are integrated and can complement each other in one framework: when the targets do not interact with each other, the laser-based independent trackers are employed and the visual information is extracted simultaneously to train some classifiers for the “possible interacting targets”. When the targets are in close proximity, the learned classifiers and visual information are used to assist in tracking. Therefore, this mode of co-operation between them not only deals with various tough problems encountered in the tracking, but also ensures that the entire process can be completely on-line and automatic. Experimental results demonstrated that laser and vision fully display their respective advantages in our system, and it is easy for us to obtain a perfect trade-off between tracking accuracy and time-cost.

In the context of driving assistance, detection of moving targets is challenging when the automobile is witnessing very complex environment. The main challenges come from difficulties to tell foreground from moving background due to camera motion. We present an efficient method for target detection by hierarchically distinguishing the motion of targets from that of the background and using color information to help with the process. Firstly, point features are extracted and tracked to form feature trajectories. Secondly, distance measures are defined on the trajectories and hierarchically used to cluster the generated trajectories into regions. The last step is devoted to better detection of the targets by adding appearance information. Experiments show our method is able to give target detection results in real time even in very complex environments.

An intelligent vehicle system is deeloped, where five single-row laser scannners are mounted on the car to profile object geometry along the streets from different viewpoints and with different directions; a GPS (Global Positioning System)/IMU (Inertial Measurement Unit) based navigation unit is also applied to give outputs of vehicle pose. This paper focus on sensor alignment in order to achieve an omni-directional laser measurement of dynamic urban environment using an intelligent vehicle. An algorithm is proposed, where the laser scanners that facing towards different directions are aligned through horizontal and vertical matching of laser points. Experimental results are presented demonstrating that a comprehensive knowledge of the environment can be achieved, and shortages of each individual sensor can be compensated through such a sensor fusion.

Motivated by two potential applications, i.e. enhancing driving safety and traffic data collection, a system has been developed using a single-layer horizontal laser scanner as the major sensor for both localization and perception of the surroundings in a large dynamic urban environment. This research focuses on a classification method, that given a stream of laser measurements, classify the moving object into either a person, a group of people, a bicycle or a car. In this research, a number of features are defined after examining the property of data appearance. A classification method is proposed after examining the likelihood measures between each pair of feature and class. Experimental results are presented, demonstrating that the algorithm has efficiency with respect to both driving safety and traffic data collection in highly dynamic environment.

This chapter presents an approach to detect and track multiple people ro bustly in real time using laser scan frames. The detection and tracking of people in real time is a problem that arises in a variety of different contexts. Examples in clude intelligent surveillance for security purposes, scene analysis for service robot, and crowd behavior analysis for human behavior study. Over the last several years, an increasing number of laser-based people-tracking systems have been developed in both mobile robotics platforms and fixed platforms using one or multiple laser scanners. It has been proved that processing on laser scanner data makes the tracker much faster and more robust than a vision-only based one in complex situations. In this chapter, we present a novel robust tracker to detect and track multiple people in a crowded and open area in real time. First, raw data are obtained that measures two legs for each people at a height of 16 cm from horizontal ground with multiple registered laser scanners. A stable feature is extracted using accumulated distribu tion of successive laser frames. In this way, the noise that generates split and merged measurements is smoothed well, and the pattern of rhythmic swinging legs is uti lized to extract each leg. Second, a probabilistic tracking model is presented, and then a sequential inference process using a Bayesian rule is described. A sequential inference process is difficult to compute analytically, so two strategies are presented to simplify the computation. In the case of independent tracking, the Kalman fil ter is used with a more efficient measurement likelihood model based on a region coherency property. Finally, to deal with trajectory fragments we present a concise approach to fuse just a little visual information from synchronized video camera to laser data. Evaluation with real data shows that the proposed method is robust and effective. It achieves a significant improvement compared with existing laser-based trackers.

In this research, a novel system for monitoring an intersection using a network of single-row laser range scanners (subsequently abbreviated as "laser scanner") is proposed. Laser scanners are set on the road side to profile an intersection horizontally from different viewpoints. This is done so that cross sections of the intersection are captured at a high scanning rate (e.g., 37 Hz) and to contain the contour points of the moving objects entering the intersection. Different laser scanners data are integrated into a common spatial-temporal coordinate system and processed. Thus, the moving objects inside the intersection are detected and tracked to estimate their state parameters, such as: location, speed, and direction at each time instance. An experiment was conducted in central Beijing, where six laser scanners were used to cover a three-way intersection. A digital copy of the dynamic intersection was measured, and, through data processing, a large quantity of physical dimension traffic data was obtained.

We present a promising system to simultaneously detect and track multiple humans in the outside scene using laser and vision. The useful information of laser and vision is automatically extracted and combined in a Bayesian formulation. In order to compute MAP estimation, an effective probabilistic detection-based particle filter (PD-PF) has been proposed. Experiments and evaluations demonstrate that not only can our system perform robustly in real environments, but also obtain better approximation of MAP than previous methods in most complex situations.

This research is motivated by two potential applications - enhancing driving safety and collecting traffic data in a large dynamic urban environment. A laser scanner based approach is proposed, in which SLAM (simultaneous localization and mapping) is developed with moving object detection and tracking using a laser scanner for perception, using GPS to achieve global accuracy, and using yaw rate and wheel speed to diagnose pose errors. Experiments are conducted to collect data along a course (4.5 km) with a test-bed vehicle run in a highly dynamic environment. The algorithms are examined, possibilities with respect to the two potential applications are demonstrated, and future works are discussed.

In this research, we propose a method of SLAM in a dynamic large outdoor environment using a laser scanner. Focus are cast on solving two major problems: 1) achieving global accuracy especially in non-cyclical environment, 2) tackling a mixture of data from both dynamic and static objects. Algorithms are developed, where GPS data and control inputs are used to diagnose pose error and guide to achieve a global accuracy; Classification of laser points and objects are conducted not in an independent module but across the processing in a framework of SLAM with moving object detection and tracking. Experiments are conducted using the data from two test-bed vehicles, and performance of the algorithms are demonstrated.