Search

Tracking of pedestrians from a moving vehicle equipped with a monocular camera is still considered as a challenging problem in the fields of both computer vision and robotics. In this paper, we address this problem in a particle filter framework, which well incorporates different cues from detector, dynamic model and target-specific tracking. In order to eliminate the effect of ego-motion when predicting the movement of pedestrians, we train one dynamic model for each driving behavior (moving forward, turning left/right) given a set of training trajectories. The learnt dynamic model is then utilized to predict the future movement of the pedestrian in the tracking process. We demonstrate our system works robustly on challenging dataset with strong illumination changes.

This research aims at generating an omnidirectional perception at the host vehicle's surroundings, extracting accurate and continuous motion trajectories of the nearby vehicles using low cost laser scanners. A system of detecting and tracking on-road vehicles using multiple laser scanners is developed, where focuses are cast on solving data association of simultaneous measurements from multiple sensors at different viewpoints, and state estimation in case of partial observations in dense dynamic situations. Experimental results in freeways in Beijing are presented, system efficiency is demonstrated, where motion trajectories describing driving behaviors such as overtaking, lane changing and other interactions between driving objects are captured. In addition, the accuracy in vehicle detection and tracking is examined using a reference vehicle with a ground truth GPS.

Two challenging issues for object detection are how to separate near objects and how to separate similar different-class objects. Learned that during human's vision perception, tokens moving or functioning in a similar manner are perceived as one unit, stated by the common fate principle, we propose a method to detect objects of multiple classes. Our method extends the Implicit Shape Model (ISM) to incorporate motion grouping results of object parts, and meets the challenges. Keypoint-based object parts are firstly detected and then grouped by the similarities of their corresponding trajectories which are traced by keypoint tracking. The grouping results are combined into a Hough transform framework. In Hough transform based methods, each object part votes for object centers and labels according to a trained codebook. In our method, the votes are assigned different weights according to the motion grouping results. One vote is assigned larger weight if it has larger consistence with the votes of other object parts in the same motion group. In such a manner, peaks in the formed Hough images which correspond to object hypotheses become easier to find. And our method gains improvement in both object position and label estimation. Experiments are provided to show the merit in terms of detection accuracy.

Complex human activity detection is a challenging problem, especially when people interact with each other. Approaches utilizing local spatio-temporal features work well with background clutter, scale and illumination changing. However, most of them focus on classifying short video sequences. In real world applications such as surveillance, it's hard to get the well segmented video clip to classify. So how to detect and localize complex human activities in unsegmented videos is a problem need to be solved. In this paper, based on the local spatio-temporal feature, we propose a variation of Hough Voting method using the Implicit Shape Model which can localize and recognize complex human activity simultaneously. Our approach is tested on the UT-Interaction dataset, and demonstrates promising results in complex human activity detection and localization.

The study of traffic behavior at intersections is an important research direction in the field of intelligent transportation system (ITS). It is considered very practical to reduce accident, solve traffic jam and improve its accessibility. This paper proposed a trajectory analysis method for vehicles passing an intersection. The motion trajectories of vehicles were obtained by a traffic data collecting system using a network of single-row laser scanners. The intersection could be described by a group of route models which are built based on trajectory clustering, so large amount of trajectories could be classified to different routes and abnormal ones could also be detected. Experimental results based on real trajectories obtained from an intersection in Beijing show the validity of our proposed method.

Abnormal activity detection plays a crucial role in surveillance applications, and such system has become an urgent need for public security. In this paper, we propose a novel laser-based system, which can perform the online detection of abnormal activity with an unsupervised way. The proposed system has the following key features that make it advantageous over previous ones: (1) It can cover quite a large and crowded area, such as subway station, public square, intersection and etc. (2) The overall system can vary with time period, incrementally learn the behavior pattern of pedestrians and perform the fully online detection of abnormal activity. This feature makes our system be quite suitable for the real-time applications. (3) The abnormal activity detection is carried out with a fully unsupervised way, there is no need for manual labelling and constructing the huge training datasets. We successfully applied the proposed system into the JR subway station of Tokyo, which can cover a 60×35m area, track more 150 targets at the same time and simultaneously perform the robust detection of abnormal activity with no human intervention.

Working environments of modern robots have changed to unstructured, dynamic and outdoor scenes. There emerged several new challenges along with these changes, mainly in perception of both static and dynamic objects of the scenes. To tackle these new challenges, this research focused on study of advanced perception systems that can simultaneously model static scenes and track dynamic objects. Our research has three features. Multi-view and multi-type sensors, together with machine learning based algorithms, are utilized to obtain robust and reliable mapping/ tracking results. In addition, a car-based mobile perception system is developed for exploring large sites. Finally, to improve robustness of the multi-view and mobile perception system, some new camera calibration methods are proposed. This paper presents an overview of our recent study on above mentioned ideas and technologies. Specifically we will focus on multi-sensor based multiple target tracking, simultaneous 3D mapping and target tracking in a mobile platform, and camera calibration.

Kids activity means a lot to their parents, and in the analysis of the activities, video retrieval has played an important role. In this paper, we propose an effective approach for the retrieval of the kid's activities from home videos. The video sequences are taken from our test-bed environment that is designed in the form of a smart home, and feature various types of child-adult interactions. We present a novel retrieval method with two steps, first using spatio-temporal matching to obtain a coarse result, And then we propose a method to learn dominant child-adult interactive behaviors based on a sequence of home videos. Based on these dominant behaviors, we get rid of some false retrieval and obtain fine result. We implement and test our methodology on a newly-introduced dataset containing several types of kid's activities, and the retrieval result shows its potential application in the video analysis demain, it can find out most of the video clips relevant to the query one.

Learning the knowledge of scene structure and tracking a large number of targets are both active topics of computer vision in recent years, which plays a crucial role in surveillance, activity analysis, object classification and etc. In this paper, we propose a novel system which simultaneously performs the Learning-Semantic-Scene and Tracking, and makes them supplement each other in one framework. The trajectories obtained by the tracking are utilized to continually learn and update the scene knowledge via an online un-supervised learning. On the other hand, the learned knowledge of scene in turn is utilized to supervise and improve the tracking results. Therefore, this “adaptive learning-tracking loop” can not only perform the robust tracking in high density crowd scene, dynamically update the knowledge of scene structure and output semantic words, but also ensures that the entire process is completely automatic and online. We successfully applied the proposed system into the JR subway station of Tokyo, which can dynamically obtain the semantic scene structure and robustly track more than 150 targets at the same time.

Multi-target tracking becomes significantly more challenging when the targets are in close proximity or frequently interact with each other. This paper presents a promising tracking system to deal with these problems. The novelty of this system is that laser and vision, tracking and learning are integrated and can complement each other in one framework: when the targets do not interact with each other, the laser-based independent trackers are employed and the visual information is extracted simultaneously to train some classifiers for the “possible interacting targets”. When the targets are in close proximity, the learned classifiers and visual information are used to assist in tracking. Therefore, this mode of co-operation between them not only deals with various tough problems encountered in the tracking, but also ensures that the entire process can be completely on-line and automatic. Experimental results demonstrated that laser and vision fully display their respective advantages in our system, and it is easy for us to obtain a perfect trade-off between tracking accuracy and time-cost.

In the context of driving assistance, detection of moving targets is challenging when the automobile is witnessing very complex environment. The main challenges come from difficulties to tell foreground from moving background due to camera motion. We present an efficient method for target detection by hierarchically distinguishing the motion of targets from that of the background and using color information to help with the process. Firstly, point features are extracted and tracked to form feature trajectories. Secondly, distance measures are defined on the trajectories and hierarchically used to cluster the generated trajectories into regions. The last step is devoted to better detection of the targets by adding appearance information. Experiments show our method is able to give target detection results in real time even in very complex environments.

An intelligent vehicle system is deeloped, where five single-row laser scannners are mounted on the car to profile object geometry along the streets from different viewpoints and with different directions; a GPS (Global Positioning System)/IMU (Inertial Measurement Unit) based navigation unit is also applied to give outputs of vehicle pose. This paper focus on sensor alignment in order to achieve an omni-directional laser measurement of dynamic urban environment using an intelligent vehicle. An algorithm is proposed, where the laser scanners that facing towards different directions are aligned through horizontal and vertical matching of laser points. Experimental results are presented demonstrating that a comprehensive knowledge of the environment can be achieved, and shortages of each individual sensor can be compensated through such a sensor fusion.

Motivated by two potential applications, i.e. enhancing driving safety and traffic data collection, a system has been developed using a single-layer horizontal laser scanner as the major sensor for both localization and perception of the surroundings in a large dynamic urban environment. This research focuses on a classification method, that given a stream of laser measurements, classify the moving object into either a person, a group of people, a bicycle or a car. In this research, a number of features are defined after examining the property of data appearance. A classification method is proposed after examining the likelihood measures between each pair of feature and class. Experimental results are presented, demonstrating that the algorithm has efficiency with respect to both driving safety and traffic data collection in highly dynamic environment.

This chapter presents an approach to detect and track multiple people ro bustly in real time using laser scan frames. The detection and tracking of people in real time is a problem that arises in a variety of different contexts. Examples in clude intelligent surveillance for security purposes, scene analysis for service robot, and crowd behavior analysis for human behavior study. Over the last several years, an increasing number of laser-based people-tracking systems have been developed in both mobile robotics platforms and fixed platforms using one or multiple laser scanners. It has been proved that processing on laser scanner data makes the tracker much faster and more robust than a vision-only based one in complex situations. In this chapter, we present a novel robust tracker to detect and track multiple people in a crowded and open area in real time. First, raw data are obtained that measures two legs for each people at a height of 16 cm from horizontal ground with multiple registered laser scanners. A stable feature is extracted using accumulated distribu tion of successive laser frames. In this way, the noise that generates split and merged measurements is smoothed well, and the pattern of rhythmic swinging legs is uti lized to extract each leg. Second, a probabilistic tracking model is presented, and then a sequential inference process using a Bayesian rule is described. A sequential inference process is difficult to compute analytically, so two strategies are presented to simplify the computation. In the case of independent tracking, the Kalman fil ter is used with a more efficient measurement likelihood model based on a region coherency property. Finally, to deal with trajectory fragments we present a concise approach to fuse just a little visual information from synchronized video camera to laser data. Evaluation with real data shows that the proposed method is robust and effective. It achieves a significant improvement compared with existing laser-based trackers.

In this research, a novel system for monitoring an intersection using a network of single-row laser range scanners (subsequently abbreviated as "laser scanner") is proposed. Laser scanners are set on the road side to profile an intersection horizontally from different viewpoints. This is done so that cross sections of the intersection are captured at a high scanning rate (e.g., 37 Hz) and to contain the contour points of the moving objects entering the intersection. Different laser scanners data are integrated into a common spatial-temporal coordinate system and processed. Thus, the moving objects inside the intersection are detected and tracked to estimate their state parameters, such as: location, speed, and direction at each time instance. An experiment was conducted in central Beijing, where six laser scanners were used to cover a three-way intersection. A digital copy of the dynamic intersection was measured, and, through data processing, a large quantity of physical dimension traffic data was obtained.

We present a promising system to simultaneously detect and track multiple humans in the outside scene using laser and vision. The useful information of laser and vision is automatically extracted and combined in a Bayesian formulation. In order to compute MAP estimation, an effective probabilistic detection-based particle filter (PD-PF) has been proposed. Experiments and evaluations demonstrate that not only can our system perform robustly in real environments, but also obtain better approximation of MAP than previous methods in most complex situations.

This research is motivated by two potential applications - enhancing driving safety and collecting traffic data in a large dynamic urban environment. A laser scanner based approach is proposed, in which SLAM (simultaneous localization and mapping) is developed with moving object detection and tracking using a laser scanner for perception, using GPS to achieve global accuracy, and using yaw rate and wheel speed to diagnose pose errors. Experiments are conducted to collect data along a course (4.5 km) with a test-bed vehicle run in a highly dynamic environment. The algorithms are examined, possibilities with respect to the two potential applications are demonstrated, and future works are discussed.

In this research, we propose a method of SLAM in a dynamic large outdoor environment using a laser scanner. Focus are cast on solving two major problems: 1) achieving global accuracy especially in non-cyclical environment, 2) tackling a mixture of data from both dynamic and static objects. Algorithms are developed, where GPS data and control inputs are used to diagnose pose error and guide to achieve a global accuracy; Classification of laser points and objects are conducted not in an independent module but across the processing in a framework of SLAM with moving object detection and tracking. Experiments are conducted using the data from two test-bed vehicles, and performance of the algorithms are demonstrated.

Successful multi-target tracking requires locating the targets and labeling their identities. For the laser based tracking system, the latter becomes significantly more challenging when the targets frequently interact with each other. This paper presents a novel on-line supervised learning based method for tracking interacting targets with laser scanner. When the targets do not interact with each other, we collect samples and train a classifier for each target. When the targets are in close proximity, we use these classifiers to assist in tracking. Different evaluations demonstrate that this method has a better tracking performance than previous methods when interactions occur, and can maintain correct tracking under various complex tracking situations.

Successful multi-target tracking requires locating the targets and labeling their identities. This mission becomes significantly more challenging when many targets frequently interact with each other (present partial or complete occlusions). This paper presents an on-line supervised learning based method for tracking multiple interacting targets. When the targets do not interact with each other, multiple independent trackers are employed for training a classifier for each target. When the targets are in close proximity or present occlusions, the learned classifiers are used to assist in tracking. The tracking and learning supplement each other in the proposed method, which not only deals with tough problems encountered in multi-target tracking, but also ensures the entire process to be completely on-line. Various evaluations have demonstrated that this method performs better than previous methods when the interactions occur, and can maintain the correct tracking under various complex tracking situations, including crossovers, collisions and occlusions.

In this paper, we present a Probabilistic Detection-based Particle Filter (PD-PF) for tracking a variable number of interacting targets. When the objects do not interact with each other, our method performs like the deterministic detection-base methods. When the objects are in close proximity, the interactions and occlusions are modelled by a mixed proposal constructed by probabilistic detections and information from dynamic models. Specially, prior of detection-reliability minimizes the influence of non-detection or false alarm in the tracking. Moreover, we run independent PD-PF for each target, such that particles are sampled in a small state space, thus our method not only obtains a better approximation of posterior than joint particle filter or independent particle filter when interactions occur, but also has an acceptable computational complexity. Different evaluations demonstrate the validity and efficiency of the proposed method.

Laser based people tracking systems have been developed for mobile robotics and intelligent surveillance areas. Existing systems rely on simple laser point clustering methods to extract object locations. However, when dealing with multiple interacting people, laser points of different persons are often interlaced and undistinguishable due to measurement noise and they can not provide reliable features. It causes current systems quite fragile and unreliable. In this paper, we try to explore potentials from multi-level observations including weakly detected features, stably extracted features and foreground points. For inference, detection incorporated joint particle filter is used. And stably extracted features are utilized to properly estimate parameters of dynamic model for each target. In real experiments, we obtain raw data from multiple registered laser scanners, which measure two legs for each people. Evaluations with real data show that the proposed method is more robust and effective than existing approaches.

Laser based people tracking systems have been developed for mobile robotic or intelligent surveillance areas. Existing systems rely on laser point clustering to extract object locations. However, in a crowded environment, laser points of different objects are often interlaced and undistinguishable and can not provide reliable features. This paper presents a novel and robust laser-based tracking method for people in crowds. Firstly, we propose a stable feature extraction method based on accumulated distribution of successive laser frames. Then a robust tracking filter is proposed based on the combination of independent Bayesian filter and sampling based data association filter. Evaluations with real data show that the proposed method is robust and effective. It achieves a significant improvement compared with existing trackers.

In this paper, we present evaluation of the performance of a laser-based multi-people detection and tracking system. We propose metrics for frame-based as well as object- based evaluation paradigms considering complexity in multi- target tracking. In frame-based evaluation, accuracy of people localization is tested without considering people identity. In object-based evaluation, identity is included, so that the cases of broken/merged/crossed trackers are evaluated. Finally, we applied these metrics to the tracking results on real data.

We present a novel system that aims at reliably detecting and tracking multiple people in an open area. Multiple single-row laser scanners and one video camera are utilized. Feet trajectory tracking based on registration of distance information from multiple laser scanners and visual body region tracking based on color histogram are combined in a Bayesian formulation. Results from tests in a real environment are reported to demonstrate that the system can detect and track multiple people simultaneously with reliable and real-time performance.

3D hand motion capture plays an important role in multi-modal human-computer interfaces. Existing vision-based approaches mainly include two directions: model-based optimization framework and appearance-based learning approach. The main obstacle to handle with human hand motion capture is the high dimensionality associated with a full degree-of-freedom (DOF) articulated model. In this paper, a novel vision-based 3D hand motion capture algorithm is proposed. It views hand pose estimation and motion tracking as search problems and utilizes genetic algorithm (GA). Firstly, a learning integrating with optimization approach is introduced to estimate initial hand pose in 3D model based framework. And then a motion tracking method using GA-based particle filter (PF) is proposed to deal with the tracking problem in high-dimensional and multi-modal state space. Experimental results show that present approach significantly improves performance of motion tracking, especially in high-dimensional configuration space.

A command fusion based human-robot cooperation strategy in telerobotic systems is presented. The robotic task execution is segmented into four stages, namely free motion, contact motion, constraint motion and dexterous operation. At each stage, human and robot can send out respective commands based on their own evaluation of the task execution. A uniform formulation is proposed to describe the fusion process. These commands can be fused to produce a proper resultant output to the low level controller supported by the robot intelligence, which consists of model-based and sensor-based autonomous capabilities. The cooperation can ensure the collision and singularity free motion, the stable and safe contact with the environment, the shared control-based constraint motion and semi-autonomy based dexterous operation. The design of our telerobotic system is described and a telemanipulation experiment via Internet is introduced.