Search

Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose "informativeness" can be evaluated by the attention mechanism for inferring fine-grained gestures/actions. To this end, we propose a novel end-to-end \textbf{Regional Attention Network (RAN)}, which is a fully Convolutional Neural Network (CNN) to combine multiple contextual regions through attention mechanism, focusing on parts of the images that are most relevant to a given task. Our regions consist of one or more consecutive cells and are adapted from the strategies used in computing HOG (Histogram of Oriented Gradient) descriptor. The model is extensively evaluated on ten datasets belonging to 3 different scenarios: 1) head pose recognition, 2) drivers state recognition, and 3) human action and facial expression recognition. The proposed approach outperforms the state-of-the-art by a considerable margin in different metrics., Comment: This manuscript is the accepted version of the published paper in IEEE Transaction on Affective Computing

The development of medical imaging techniques has greatly supported clinical decision making. However, poor imaging quality, such as non-uniform illumination or imbalanced intensity, brings challenges for automated screening, analysis and diagnosis of diseases. Previously, bi-directional GANs (e.g., CycleGAN), have been proposed to improve the quality of input images without the requirement of paired images. However, these methods focus on global appearance, without imposing constraints on structure or illumination, which are essential features for medical image interpretation. In this paper, we propose a novel and versatile bi-directional GAN, named Structure and illumination constrained GAN (StillGAN), for medical image quality enhancement. Our StillGAN treats low- and high-quality images as two distinct domains, and introduces local structure and illumination constraints for learning both overall characteristics and local details. Extensive experiments on three medical image datasets (e.g., corneal confocal microscopy, retinal color fundus and endoscopy images) demonstrate that our method performs better than both conventional methods and other deep learning-based methods. In addition, we have investigated the impact of the proposed method on different medical image analysis and clinical tasks such as nerve segmentation, tortuosity grading, fovea localization and disease classification.

Automated detection of retinal structures, such as retinal vessels (RV), the foveal avascular zone (FAZ), and retinal vascular junctions (RVJ), are of great importance for understanding diseases of the eye and clinical decision-making. In this paper, we propose a novel Voting-based Adaptive Feature Fusion multi-task network (VAFF-Net) for joint segmentation, detection, and classification of RV, FAZ, and RVJ in optical coherence tomography angiography (OCTA). A task-specific voting gate module is proposed to adaptively extract and fuse different features for specific tasks at two levels: features at different spatial positions from a single encoder, and features from multiple encoders. In particular, since the complexity of the microvasculature in OCTA images makes simultaneous precise localization and classification of retinal vascular junctions into bifurcation/crossing a challenging task, we specifically design a task head by combining the heatmap regression and grid classification. We take advantage of three different en face angiograms from various retinal layers, rather than following existing methods that use only a single en face. We carry out extensive experiments on three OCTA datasets acquired using different imaging devices, and the results demonstrate that the proposed method performs on the whole better than either the state-of-the-art single-purpose methods or existing multi-task learning solutions. We also demonstrate that our multi-task learning method generalizes across other imaging modalities, such as color fundus photography, and may potentially be used as a general multi-task learning tool. We also construct three datasets for multiple structure detection, and part of these datasets with the source code and evaluation benchmark have been released for public access.

Recently, effort has been made to apply deep learning to the detection of mesh saliency. However, one major barrier is to collect a large amount of vertex-level annotation as saliency ground truth for training the neural networks. Quite a few pilot studies showed that this task is quite difficult. In this work, we solve this problem by developing a novel network trained in a weakly supervised manner. The training is end-to-end and does not require any saliency ground truth but only the class membership of meshes. Our Classification-for-Saliency CNN (CfS-CNN) employs a multi-view setup and contains a newly designed two-channel structure which integrates view-based features of both classification and saliency. It essentially transfers knowledge from 3D object classification to mesh saliency. Our approach significantly outperforms the existing state-of-the-art methods according to extensive experimental results. Also, the CfS-CNN can be directly used for scene saliency. We showcase two novel applications based on scene saliency to demonstrate its utility.

Accurate estimation and quantification of the corneal nerve fiber tortuosity in corneal confocal microscopy (CCM) is of great importance for disease understanding and clinical decision-making. However, the grading of corneal nerve tortuosity remains a great challenge due to the lack of agreements on the definition and quantification of tortuosity. In this paper, we propose a fully automated deep learning method that performs image-level tortuosity grading of corneal nerves, which is based on CCM images and segmented corneal nerves to further improve the grading accuracy with interpretability principles. The proposed method consists of two stages: 1) A pre-trained feature extraction backbone over ImageNet is fine-tuned with a proposed novel bilinear attention (BA) module for the prediction of the regions of interest (ROIs) and coarse grading of the image. The BA module enhances the ability of the network to model long-range dependencies and global contexts of nerve fibers by capturing second-order statistics of high-level features. 2) An auxiliary tortuosity grading network (AuxNet) is proposed to obtain an auxiliary grading over the identified ROIs, enabling the coarse and additional gradings to be finally fused together for more accurate final results. The experimental results show that our method surpasses existing methods in tortuosity grading, and achieves an overall accuracy of 85.64% in four-level classification. We also validate it over a clinical dataset, and the statistical analysis demonstrates a significant difference of tortuosity levels between healthy control and diabetes group. We have released a dataset with 1500 CCM images and their manual annotations of four tortuosity levels for public access. The code is available at: https://github.com/iMED-Lab/TortuosityGrading.

Over the past few years, a significant progress has been made in deep convolutional neural networks (CNNs)-based image recognition. This is mainly due to the strong ability of such networks in mining discriminative object pose and parts information from texture and shape. This is often inappropriate for fine-grained visual classification (FGVC) since it exhibits high intra-class and low inter-class variances due to occlusions, deformation, illuminations, etc. Thus, an expressive feature representation describing global structural information is a key to characterize an object/ scene. To this end, we propose a method that effectively captures subtle changes by aggregating context-aware features from most relevant image-regions and their importance in discriminating fine-grained categories avoiding the bounding-box and/or distinguishable part annotations. Our approach is inspired by the recent advancement in self-attention and graph neural networks (GNNs) approaches to include a simple yet effective relation-aware feature transformation and its refinement using a context-aware attention mechanism to boost the discriminability of the transformed feature in an end-to-end learning process. Our model is evaluated on eight benchmark datasets consisting of fine-grained objects and human-object interactions. It outperforms the state-of-the-art approaches by a significant margin in recognition accuracy., Comment: Accepted manuscript - IEEE Transaction on Image Processing

This paper integrates human driver factors with a model-based Collision Avoidance System (CAS) to enhance the safety of semi-autonomous vehicles. Driver Activity Recognition (DAR) through Driver Distraction States (DDS) has been used as the key component to trigger the CAS so that collisions can be averted. DDS has been generated using realistic normal driving scenarios and suitably integrated with a Full State Feedback (FSF) controller-based CAS. The integrated algorithm has been tested using a Hardware in Loop (HiL) setup, which is interfaced with the vehicle dynamics software IPG TruckMaker®. The performance of the algorithm has been evaluated for various on-road scenarios and found to be effective in avoiding rear-end collisions.

Video summarization aims to create a succinct representation of videos for efficient browsing and retrieval. We propose an innovative method for the task. It includes two main steps: (i) the first step proposes a Distinct Frame Patch (DFP) index for selecting a set of good candidate frames, and (ii) the second step proposes a novel Appearance based Linear Clustering (ALC) to refine them for distinct ones. While the first step measures the content of frames, the second step considers to what extent one frame is different from another in both the spatial and temporal spaces. The experiments are performed over two publicly accessible datasets. The results show the effectiveness and efficiency of the proposed method when compared with other state-of-the-art techniques.

This paper presents a novel keypoints-based attention mechanism for visual recognition in still images. Deep Convolutional Neural Networks (CNNs) for recognizing images with distinctive classes have shown great success, but their performance in discriminating fine-grained changes is not at the same level. We address this by proposing an end-to-end CNN model, which learns meaningful features linking fine-grained changes using our novel attention mechanism. It captures the spatial structures in images by identifying semantic regions (SRs) and their spatial distributions, and is proved to be the key to modelling subtle changes in images. We automatically identify these SRs by grouping the detected keypoints in a given image. The ``usefulness'' of these SRs for image recognition is measured using our innovative attentional mechanism focusing on parts of the image that are most relevant to a given task. This framework applies to traditional and fine-grained image recognition tasks and does not require manually annotated regions (e.g. bounding-box of body parts, objects, etc.) for learning and prediction. Moreover, the proposed keypoints-driven attention mechanism can be easily integrated into the existing CNN models. The framework is evaluated on six diverse benchmark datasets. The model outperforms the state-of-the-art approaches by a considerable margin using Distracted Driver V1 (Acc: 3.39%), Distracted Driver V2 (Acc: 6.58%), Stanford-40 Actions (mAP: 2.15%), People Playing Musical Instruments (mAP: 16.05%), Food-101 (Acc: 6.30%) and Caltech-256 (Acc: 2.59%) datasets., Comment: Published in IEEE Transaction on Image Processing 2021, Vol. 30, pp. 3691 - 3704

Scale-invariance, good localization and robustness to noise and distortions are the main properties that a local feature detector should possess. Most existing local feature detectors find excessive unstable feature points that increase the number of keypoints to be matched and the computational time of the matching step. In this paper, we show that robust and accurate keypoints exist in the specific scale-space domain. To this end, we first formulate the superimposition problem into a mathematical model and then derive a closed-form solution for multiscale analysis. The model is formulated via difference-of-Gaussian (DoG) kernels in the continuous scale-space domain, and it is proved that setting the scale-space pyramid's blurring ratio and smoothness to 2 and 0.627, respectively, facilitates the detection of reliable keypoints. For the applicability of the proposed model to discrete images, we discretize it using the undecimated wavelet transform and the cubic spline function. Theoretically, the complexity of our method is less than 5\% of that of the popular baseline Scale Invariant Feature Transform (SIFT). Extensive experimental results show the superiority of the proposed feature detector over the existing representative hand-crafted and learning-based techniques in accuracy and computational time. The code and supplementary materials can be found at~{\url{https://github.com/mogvision/FFD}}.

An importance measure of 3D objects inspired by human perception has a range of applications since people want computers to behave like humans in many tasks. This paper revisits a well-defined measure, distinction of 3D surface mesh, which indicates how important a region of a mesh is with respect to classification. We develop a method to compute it based on a classification network and a Markov Random Field (MRF). The classification network learns view-based distinction by handling multiple views of a 3D object. Using a classification network has an advantage of avoiding the training data problem which has become a major obstacle of applying deep learning to 3D object understanding tasks. The MRF estimates the parameters of a linear model for combining the view-based distinction maps. The experiments using several publicly accessible datasets show that the distinctive regions detected by our method are not just significantly different from those detected by methods based on handcrafted features, but more consistent with human perception. We also compare it with other perceptual measures and quantitatively evaluate its performance in the context of two applications. Furthermore, due to the view-based nature of our method, we are able to easily extend mesh distinction to 3D scenes containing multiple objects.

Precise characterization and analysis of corneal nerve fiber tortuosity are of great importance in facilitating examination and diagnosis of many eye-related diseases. In this paper we propose a fully automated method for image-level tortuosity estimation, comprising image enhancement, exponential curvature estimation, and tortuosity level classification. The image enhancement component is based on an extended Retinex model, which not only corrects imbalanced illumination and improves image contrast in an image, but also models noise explicitly to aid removal of imaging noise. Afterwards, we take advantage of exponential curvature estimation in the 3D space of positions and orientations to directly measure curvature based on the enhanced images, rather than relying on the explicit segmentation and skeletonization steps in a conventional pipeline usually with accumulated pre-processing errors. The proposed method has been applied over two corneal nerve microscopy datasets for the estimation of a tortuosity level for each image. The experimental results show that it performs better than several selected state-of-the-art methods. Furthermore, we have performed manual gradings at tortuosity level of four hundred and three corneal nerve microscopic images, and this dataset has been released for public access to facilitate other researchers in the community in carrying out further research on the same and related topics.

Automatic video summarization aims to provide brief representation of videos. Its evaluation is quite challenging, usually relying on comparison with user summaries. This study views it in a different perspective in terms of verifying the consistency of user summaries, as the outcome of video summarization is usually judged based on them. We focus on human consistency evaluation of static video summaries in which the user summaries are evaluated among themselves using the consistency modelling method we proposed recently. The purpose of such consistency evaluation is to check whether the users agree among themselves. The evaluation is performed on different publicly available datasets. Another contribution lies in the creation of static video summaries from the available video skims of the SumMe datatset. The results show that the level of agreement varies significantly between the users for the selection of key frames, which denotes the hidden challenge in automatic video summary evaluation. Moreover, the maximum agreement level of the users for a certain dataset, may indicate the best performance that the automatic video summarization techniques can achieve using that dataset.

Non-predictive or inaccurate weather forecasting can severely impact the community of users such as farmers. Numerical weather prediction models run in major weather forecasting centers with several supercomputers to solve simultaneous complex nonlinear mathematical equations. Such models provide the medium-range weather forecasts, i.e., every 6 h up to 18 h with grid length of 10–20 km. However, farmers often depend on more detailed short-to medium-range forecasts with higher-resolution regional forecasting models. Therefore, this research aims to address this by developing and evaluating a lightweight and novel weather forecasting system, which consists of one or more local weather stations and state-of-the-art machine learning techniques for weather forecasting using time-series data from these weather stations. To this end, the system explores the state-of-the-art temporal convolutional network (TCN) and long short-term memory (LSTM) networks. Our experimental results show that the proposed model using TCN produces better forecasting compared to the LSTM and other classic machine learning approaches. The proposed model can be used as an efficient localized weather forecasting tool for the community of users, and it could be run on a stand-alone personal computer.

The estimation of vascular network topology in complex networks is important in understanding the relationship between vascular changes and a wide spectrum of diseases. Automatic classification of the retinal vascular trees into arteries and veins is of direct assistance to the ophthalmologist in terms of diagnosis and treatment of eye disease. However, it is challenging due to their projective ambiguity and subtle changes in appearance, contrast, and geometry in the imaging process. In this paper, we propose a novel method that is capable of making the artery/vein (A/V) distinction in retinal color fundus images based on vascular network topological properties. To this end, we adapt the concept of dominant set clustering and formalize the retinal blood vessel topology estimation and the A/V classification as a pairwise clustering problem. The graph is constructed through image segmentation, skeletonization, and identification of significant nodes. The edge weight is defined as the inverse Euclidean distance between its two end points in the feature space of intensity, orientation, curvature, diameter, and entropy. The reconstructed vascular network is classified into arteries and veins based on their intensity and morphology. The proposed approach has been applied to five public databases, namely INSPIRE, IOSTAR, VICAVR, DRIVE, and WIDE, and achieved high accuracies of 95.1%, 94.2%, 93.8%, 91.1%, and 91.0%, respectively. Furthermore, we have made manual annotations of the blood vessel topologies for INSPIRE, IOSTAR, VICAVR, and DRIVE datasets, and these annotations are released for public access so as to facilitate researchers in the community.

Local feature description is to assign a unique signature to a key-point such that it becomes distinctive from the others regardless of changes in viewpoint, illumination, rotation, scale as well as distortions and noise. This paper proposes a novel approach to construct such a descriptor. For preserving both homogeneous and heterogeneous features of a given support region, we interweave the texture information so that the key-point is more likely to be assigned a distinctive signature and neighboring key-points will be less likely to share the same texture information. The main idea behind our descriptor is to increase the areas of our observations in the given scene while the length of the local support region is fixed. Gradient magnitude and divergence, as measurement parameters of texture information, are applied to a group of pixels instead of employing a pixel-wise strategy that make the descriptor more resistant to noise, distortions and illumination variation. The required storage of the proposed descriptor is just 72 floats and its computational complexity is much lower than those of existing ones. A comparative study between the proposed method and the selected state-of-the-art ones over multiple publicly accessible datasets with different characteristics shows its superiority, robustness and computational efficiency under various geometric changes, illumination variation, distortions and noise. The code and supplementary materials can be found at https://github.com/mogvision/InterTex-Feature-Descriptor .

Feature extraction and matching provide the basis of many methods for object registration, modeling, retrieval, and recognition. However, this approach typically introduces false matches, due to lack of features, noise, occlusion, and cluttered backgrounds. In registration, these false matches lead to inaccurate estimation of the underlying transformation that brings the overlapping shapes into best possible alignment. In this paper, we propose a novel boosting-inspired method to tackle this challenging task. It includes three key steps: (i) underlying transformation estimation in the weighted least squares sense, (ii) boosting parameter estimation and regularization via Tsallis entropy, and (iii) weight re-estimation and regularization via Shannon entropy and update with a maximum fusion rule. The process is iterated. The final optimal underlying transformation is estimated as a weighted average of the transformations estimated from the latest iterations, with weights given by the boosting parameters. A comparative study based on real shape data shows that the proposed method outperforms four other state-of-the-art methods for evaluating the established point matches, enabling more accurate and stable estimation of the underlying transformation. HighlightsWe provide a novel method for evaluating point matches established by typical methods.Real data based experimental results show that the method outperforms the state-of-the-art.Under certain conditions, the finally estimated underlying transformation is optimal.The transformation can be recovered with an error as small as 5% from typical point matches.

As a measure of regional importance in agreement with human perception of 3D shape, mesh saliency should be based on local geometric information within a mesh but more than that. Recent research has shown that global consideration has a significant role in mesh saliency. This paper proposes a local-to-global framework for computing mesh saliency where we offer novel solutions to solve three inherent problems: (1) an algorithm based on statistic Laplacian which does not only compute local saliency, but also facilitates the later computation of global saliency; (2) a local-to-global method based on pooling and global distinctness to compute global saliency; (3) a framework to integrate local and global saliency. Experiments demonstrate that our approach can effectively detect salient features consistent with human perceptual interest. We also provide comparisons to existing state-of-the-art methods for mesh saliency and show improved results produced by our method.

In the above article [1] , there were two errors in the printed article that the authors want to correct.

Text information in natural images is very important to cross-media retrieval, index and understanding. However, its detection is challenging due to varying backgrounds, low contrast between text and non-text regions, perspective distortion and other disturbing factors. In this paper, we propose a novel text line detection method which can detect text line aligned with a straight line in any direction. It is mainly composed of three steps. In the first step, we use the maximal stable extremal region detector with dam line constraint to detect candidate text regions, we then define a similarity measurement between two regions which combines sizes, absolute distance, relative distance, contextual information and color histograms. In the second step, we propose a text line identification algorithm based on the defined similarity measurement. The algorithm firstly searches three regions as the seeds of a line, and then expands to obtain all regions in the line. In the last step, we develop a filter to remove non-text lines. The filter uses a sparse classifier based on two dictionaries which are learned from feature vectors extracted from morphological skeletons of those candidate text lines. A comparative study using two datasets shows the excellent performance of the proposed method for accurate text line detection with horizontal or arbitrary consistent orientation.

We propose a self-organizing Radial Basis Function (RBF) neural network method for parameterization of freeform surfaces from larger, noisy and unoriented point clouds. In particular, an adaptive sequential learning algorithm is presented for network construction from a single instance of point set. The adaptive learning allows neurons to be dynamically inserted and fully adjusted (e.g. their locations, widths and weights), according to mapping residuals and data point novelty associated to underlying geometry. Pseudo-neurons, exhibiting very limited contributions, can be removed through a pruning procedure. Additionally, a neighborhood extended Kalman filter (NEKF) was developed to significantly accelerate parameterization. Experimental results show that this adaptive learning enables effective capture of global low-frequency variations while preserving sharp local details, ultimately leading to accurate and compact parameterization, as characterized by a small number of neurons. Parameterization using the proposed RBF network provides simple, low cost and low storage solutions to many problems such as surface construction, re-sampling, hole filling, multiple level-of-detail meshing and data compression from unstructured and incomplete range data. Performance results are also presented for comparison.

Non-rigid registration finds many applications such as photogrammetry, motion tracking, model retrieval, and object recognition. In this paper we propose a novel convex hull aided registration method (CHARM) to match two point sets subject to a non-rigid transformation. First, two convex hulls are extracted from the source and target respectively. Then, all points of the point sets are projected onto the reference plane through each triangular facet of the hulls. From these projections, invariant features are extracted and matched optimally. The matched feature point pairs are mapped back onto the triangular facets of the convex hulls to remove outliers that are outside any relevant triangular facet. The rigid transformation from the source to the target is robustly estimated by the random sample consensus (RANSAC) scheme through minimizing the distance between the matched feature point pairs. Finally, these feature points are utilized as the control points to achieve non-rigid deformation in the form of thin-plate spline of the entire source point set towards the target one. The experimental results based on both synthetic and real data show that the proposed algorithm outperforms several state-of-the-art ones with respect to sampling, rotational angle, and data noise. In addition, the proposed CHARM algorithm also shows higher computational efficiency compared to these methods.

Leakage in retinal angiography currently is a key feature for confirming the activities of lesions in the management of a wide range of retinal diseases, such as diabetic maculopathy and paediatric malarial retinopathy. This paper proposes a new saliency-based method for the detection of leakage in fluorescein angiography. A superpixel approach is firstly employed to divide the image into meaningful patches (or superpixels) at different levels. Two saliency cues, intensity and compactness, are then proposed for the estimation of the saliency map of each individual superpixel at each level. The saliency maps at different levels over the same cues are fused using an averaging operator. The two saliency maps over different cues are fused using a pixel-wise multiplication operator. Leaking regions are finally detected by thresholding the saliency map followed by a graph-cut segmentation. The proposed method has been validated using the only two publicly available datasets: one for malarial retinopathy and the other for diabetic retinopathy. The experimental results show that it outperforms one of the latest competitors and performs as well as a human expert for leakage detection and outperforms several state-of-the-art methods for saliency detection.

This paper formalizes overlapping free form shape registration as a minimization problem, which minimizes a weighted sum of registration errors of tentative correspondences with the weights subject to boundary conditions. The tentative correspondences are established using the traditional closest point criterion. Then the powerful barrier method is employed to transform the constrained minimization problem to an unconstrained one. Setting the first order derivative of the unconstrained objective function to zero results in the weights being solved with a closed form solution. The weights are finally globally optimized using the deterministic annealing scheme. Outliers due to occlusion, appearance and disappearance of points in either free form shape are explicitly modelled using a constant. The camera motion parameters are updated in the weighted least squares sense. A comparative study based on both synthetic data and real images shows that the proposed algorithm is promising for the accurate and robust automatic registration of overlapping free form shapes.

In this paper, a novel entropy that can describe both long and short-tailed probability distributions of constituents of a thermodynamic system out of its thermodynamic limit is first derived from the Lyapunov function for a Markov chain. We then maximize this entropy for the estimation of the probabilities of possible correspondences established using the traditional closest point criterion between two overlapping range images. When we change our viewpoint to look carefully at the minimum solution to the probability estimate of the correspondences, the iterative range image registration process can also be modeled as a Markov chain in which lessons from past experience in estimating those probabilities are learned. To impose the two-way constraint, outliers are explicitly modeled due to the almost ubiquitous occurrence of occlusion, appearance, and disappearance of points in either image. The estimated probabilities of the correspondences are finally embedded into the powerful mean field annealing scheme for global optimization, leading the camera motion parameters to be estimated in the weighted least-squares sense. A comparative study using real images shows that the proposed algorithm usually outperforms the state-of-the-art ICP variants and the latest genetic algorithm for automatic overlapping range image registration.

Iterative algorithms are often used for range image matching. In this paper, we treat the iterative process of range image matching as a live biological system: evolving from one generation to another. Whilst different generations of the population are regarded as range images captured at different viewpoints, the iterative process is simulated using time. The well-known replicator equations in theoretical biology are then adapted to estimate the probabilities of possible correspondences established using the traditional closest point criterion. To reduce the effect of image resolutions on the final results for efficient and robust overlapping range image matching, the relative fitness difference (rather than the absolute fitness difference) is employed in the replicator equations in order to model the probability change of possible correspondences being real over successive iterations. The fitness of a possible correspondence is defined as the negative of a power of its squared Euclidean distance. While the replicator dynamics penalize those individuals with low fitness, they are further penalised with a parameter, since distant points are often unlikely to represent their real replicators. While the replicator equations assume that all individuals are equally likely to meet each other and thus treat them equally, we penalise those individuals competing for the same points as their possible replicators. The estimated probabilities of possible correspondences being real are finally embedded into the powerful deterministic annealing scheme for global optimization, resulting in the camera motion parameters being estimated in the weighted least squares sense. A comparative study based on real range images with partial overlap has shown that the proposed algorithm is promising for automatic matching of overlapping range images.

The traditional closest point criterion has been widely used for 3D free form shape matching, object recognition, internet search, computer graphics and medical imaging. However, the rationale of this criterion has not yet been well understood and exploited. In this paper, we apply vector operations and the triangle inequality to carefully analyse this criterion and reveal that this criterion can guarantee that the found point matches satisfy the orientation, rigidity and matching error constraints and thus, are of high relative quality. Such properties not only shed light on and deepen our understanding of this criterion about its generality and practicality and improve our awareness about whether the established point matches are consistent with each other, but also provide us with a possibility to develop novel algorithms for the reliability evaluation of existing point matches and an efficient establishment of more accurate point matches. The experimental results based on real images show that the possible point matches established through extracting and matching spin images often violate these constraints and these constraints can often be successfully applied to reject (probably part of) the unlikely point matches for more accurate free form shape matching results.

3D modelling finds a wide range of applications in industry. However, due to the presence of surface scanning noise, accumulative registration errors, and improper data fusion, reconstructed object surfaces using range images captured from multiple viewpoints are often distorted with thick patches, false connections, blurred features and artefacts. Moreover, the existing integration methods are often expensive in the sense of both computational time and data storage. These shortcomings limit the wide applications of 3D modelling using the latest laser scanning systems. In this paper, the k-means clustering approach (from the pattern recognition and machine learning literatures) is employed to minimize the integration error and to optimize the fused point locations. To initialize the clustering approach, an automatic method is developed, shifting points in the overlapping areas between neighbouring views towards each other, so that the initialized cluster centroids are in between the two overlapping surfaces. This results in more efficient and effective integration of data. While the overlapping areas were initially detected using a single distance threshold, they are then refined using the k-means clustering method. For more accurate integration results, a weighting scheme reflecting the imaging principle is developed to integrate the corresponding points in the overlapping areas. The fused point set is finally triangulated using an improved Delaunay method, guaranteeing a watertight surface. A comparative study based on real images shows that the proposed algorithm is efficient in the sense of either running time or memory usage and reduces significantly the integration error, while desirably retaining geometric details of 3D object surfaces of interest.

The SoftAssign algorithm is an elegant free form shape matching algorithm. While its objective function can be interpreted as consisting of three desired terms: minimising a weighted sum of matching errors of combinations of all the points in the two free form shapes to be matched, equalising their weights (probabilities) of being real ones and also maximising the overlapping area between the free form shapes to be matched, the last term has no effect on the optimisation of the parameters of interest due to normalisation. In this paper, we reformulate the last two terms using the inequality about the geometric and algebraic averages and the sum of the powers of these probabilities. For the sake of computational efficiency, instead of considering combinations of all the points in the overlapping free form shapes to be matched, we employ the traditional closest point criterion to establish possible correspondences between the two overlapping free form shapes to be matched. The saddle point solution of the resulting objective function no longer yields a closed form solution to the parameters of interest. For easy computation, we then adopt a pseudo-linearisation method to linearise the first order derivative of the objective function, leading the parameters of interest to be tracked and estimated with a closed form solution. The parameters of interest are finally optimised using the efficient deterministic annealing scheme with the camera motion parameters estimated using the quaternion method in the weighted least squares sense. A comparative study based on both synthetic data and real images with partial overlap has shown that the proposed algorithm is promising for the automatic matching of overlapping 3D free form shapes subject to a large range of motions.

Feature extraction and matching (FEM) for 3D shapes finds numerous applications in computer graphics and vision for object modeling, retrieval, morphing, and recognition. However, unavoidable incorrect matches lead to inaccurate estimation of the transformation relating different datasets. Inspired by AdaBoost, this paper proposes a novel iterative re-weighting method to tackle the challenging problem of evaluating point matches established by typical FEM methods. Weights are used to indicate the degree of belief that each point match is correct. Our method has three key steps: (i) estimation of the underlying transformation using weighted least squares, (ii) penalty parameter estimation via minimization of the weighted variance of the matching errors, and (iii) weight re-estimation taking into account both matching errors and information learnt in previous iterations. A comparative study, based on real shapes captured by two laser scanners, shows that the proposed method outperforms four other state-of-the-art methods in terms of evaluating point matches between overlapping shapes established by two typical FEM methods, resulting in more accurate estimates of the underlying transformation. This improved transformation can be used to better initialize the iterative closest point algorithm and its variants, making 3D shape registration more likely to succeed. © 1995-2012 IEEE.

In this paper, we propose a novel algorithm for the automatic registration of two overlapping range images. Since it is relatively difficult to compare the registration errors of different point matches, we project them onto a virtual image plane for more accurate comparison using the classical pin-hole perspective projection camera model. While the traditional ICP algorithm is more interested in the points in the second image close to the sphere centred at the transformed point, the novel algorithm is more interested in the points in the second image as collinear as possible to the transformed point. The novel algorithm then extracts useful information from both the registration error and projected error histograms for the elimination of false matches without any feature extraction, image segmentation or the requirement of motion estimation from outliers corrupted data and, thus, has an advantage of easy implementation. A comparative study based on real images captured under typical imaging conditions has shown that the novel algorithm produces good registration results.

Three-dimensional free form shape matching is a fundamental problem in both the machine vision and pattern recognition literatures. However, the automatic approach to 3D free form shape matching still remains open. In this paper, we propose using k closest points in the second view for the automatic 3D free form shape matching. For the sake of computational efficiency, the optimised k-D tree is employed for the search of the k closest points. Since occlusion and appearance and disappearance of points almost always occur, slack variables have to be employed, explicitly modelling outliers in the process of matching. Then the relative quality of each possible point match is estimated using the graduated assignment algorithm, leading the camera motion parameters to be estimated by the quaternion method in the weighted least-squares sense. The experimental results based on both synthetic data and real images without any pre-processing show the effectiveness and efficiency of the proposed algorithm for the automatic matching of overlapping 3D free form shapes with either sparse or dense points.

The goal of this paper is twofold: firstly, we propose a novel interpretation for collinearity in the process of camera pose estimation from given correspondences between a 3D model and its 2D projective image. In contrast with the existing interpretations for collinearity, the focus of expansion (FOE) theory is a special case of our novel interpretation for collinearity and besides the projection of camera position on the image plane, every image point can become a FOE. Secondly, we propose a novel method, based on the collinearity equation, for camera pose estimation from given point correspondences between a 3D model and its projective image. A comparative study based on both synthetic data and real images has shown that the novel algorithm is promising.

Automatic image registration is an attractive and unresolved problem in the machine vision literature. This paper presents two novel structural constraints, namely proximity and closeness constraints, that improve both the accuracy and robustness of an existing motion consistency based algorithm (GICP) for automatic image registration. It also defines the conditions when such constraints are fired at specific points of the registration of two overlapping range images. The proximity constraint says that neighbouring points should be neighbouring before and after a rigid motion. The closeness constraint implements a local closest point search from the second image to the first image. While the GICP algorithm uses an exhaustive search for the closest points, in this paper, we employ the optimised k–d tree data structure to accelerate the closest point search. A large number of experiments based on real range images demonstrate that the combination of rigid motion constraints with these novel proximity and closeness constraints leads to a more accurate and robust evaluation of possible correspondences leading, in turn, to more accurate, robust and efficient automatic image registration results. © 2004 Elsevier B.V. All rights reserved.

Automatic range image registration and matching is an attractive but unresolved problem in both the machine vision and pattern recognition literature. Since automatic range image registration and matching is inherently a very difficult problem, the algorithms developed are becoming more and more complicated. In this paper, we propose a novel practical algorithm for automatic free-form surface matching. This method directly manipulates the possible point matches established by the traditional ICP criterion based on both the collinearity and closeness constraints without any feature extraction, image pre-processing, or motion estimation from outliers corrupted data. A comparative study based on a large number of real range images has shown the accuracy and robustness of the novel algorithm.

In this paper we consider representation issues of rigid body transformations based on geometric properties of reflected correspondence vectors. A sufficient and necessary representation of rigid body transformations is put forward followed by an accurate and robust algorithm for the registration of free-form surfaces. The algorithm makes full use of global rigid motion constraints derived from the representation of rigid body transformations and can effectively deal with occlusion, noise, and appearance and disappearance of points. A comparative study based on both synthetic data and real images show that the registration method is accurate and robust.

Integration is a crucial step in the reconstruction of complete 3D surface model from multiple scans.\ud Ever-present registration errors and scanning noise make integration a nontrivial problem. In this paper,\ud we propose a novel method for multi-view scan integration where we solve it as a labelling problem.\ud Unlike previous methods, which have been based on various merging schemes, our labelling-based\ud method is essentially a selection strategy. The overall surface model is composed of surface patches from\ud selected input scans. We formulate the labelling via a higher-order Markov Random Field (MRF) which\ud assigns a label representing an index of some input scan to every point in a base surface. Using a higherorder\ud MRF allows us to more effectively capture spatial relations between 3D points. We employ belief\ud propagation to infer this labelling and experimentally demonstrate that this integration approach\ud provides significantly improved integration via both qualitative and quantitative comparisons.

Inspired by retinex theory, we propose a novel method for selecting key points from a depth map of a 3D freeform shape; we also use these key points as a basis for shape registration. To find key points, first, depths are transformed using the Hotelling method and normalized to reduce their dependence on a particular viewpoint. Adaptive smoothing is then applied using weights which decrease with spatial gradient and local inhomogeneity; this preserves local features such as edges and corners while ensuring smoothed depths are not reduced. Key points are those with locally maximal depths, faithfully capturing shape. We show how such key points can be used in an efficient registration process, using two state-of-the-art iterative closest point variants. A comparative study with leading alternatives, using real range images, shows that our approach provides informative, expressive, and repeatable points leading to the most accurate registration results. HighlightsA novel method of key point detection from given 3D freeform shapes is proposed.The detected points are useful for efficient registration of 3D freeform shapes.Evaluation based on real range images shows that better results are obtained.

We propose a novel algorithm for correspondenceless pose estimation based on a realistic camera setup for a class of industrial problems. The algorithm provides unique closed form solutions to all parameters of interest and its superior performance is demonstrated by extensive comparisons with an epipolar geometry-based algorithm.

Accurate motion estimation algorithms are based on a number of invariant properties that can be inferred from the motion. A large number of calibration algorithms have been proposed over the last two decades mainly based on analytic, perspective, or epipolar geometries. Extending Chasles' screw motion concept to the estimation of motion parameters in computer vision, we have presented an analysis of geometric properties of image correspondence vectors synthesized into a single coordinate frame and developed calibration algorithms using both simulated and real range image data.15,16 In this paper, we extend that work by defining the relevant geometric properties of image correspondence vectors from the point of view of invariants and by developing two calibration algorithms using the Monte Carlo method and median filtering. The algorithms are applied to real and synthetic image data corrupted by noise and outliers. Experimental results demonstrate that the median filter based algorithm is generally more robust and accurate than the Monte Carlo based algorithm and that the geometric analysis of invariant properties of correspondence vectors is a useful framework to motion parameter estimation.

This paper presents a method for detecting points of interest on 3D meshes. It comprises two major stages. In the first, we capture saliency in the spectral domain by detecting spectral irregularities of a mesh. Such saliency corresponds to the interesting portions of surface in the spatial domain. In the second stage, to transfer saliency information from the spectral domain to the spatial domain, we rely on spectral irregularity diffusion (SID) based on heat diffusion. SID captures not only the information about neighbourhoods of a given point in a multiscale manner, but also cues related to the global structure of a shape. It thus preserves information about both local and global saliency. We finally extract points of interest by looking for global and local maxima of the saliency map. We demonstrate the advantages of our proposed method using both visual and quantitative comparisons based on a publicly available benchmark.

Three-dimensional surface registration transforms multiple three-dimensional data sets into the same coordinate system so as to align overlapping components of these sets. Recent surveys have covered different aspects of either rigid or nonrigid registration, but seldom discuss them as a whole. Our study serves two purposes: 1) To give a comprehensive survey of both types of registration, focusing on three-dimensional point clouds and meshes and 2) to provide a better understanding of registration from the perspective of data fitting. Registration is closely related to data fitting in which it comprises three core interwoven components: model selection, correspondences and constraints, and optimization. Study of these components 1) provides a basis for comparison of the novelties of different techniques, 2) reveals the similarity of rigid and nonrigid registration in terms of problem representations, and 3) shows how overfitting arises in nonrigid registration and the reasons for increasing interest in intrinsic techniques. We further summarize some practical issues of registration which include initializations and evaluations, and discuss some of our own observations, insights and foreseeable research trends.

For accurate registration of overlapping free form shapes, different points in one shape must select different points in another as their most sensible correspondents. To reach this ideal state, in this paper we develop a novel algorithm to penalize those points in one shape that select the same closest point in another as their tentative correspondents. The novel algorithm then models the relative weight change over time of a tentative correspondence as the difference between the negative functions of the numbers of points in one shape that actually and ideally select the same closest point in another. Such modeling results in an optimal estimation of the weights of different tentative correspondences, in the sense of deterministic annealing, that lead the camera motion parameters to be estimated in the weighted least squares sense. The proposed algorithm is initialized using the pure translational motion derived from the centroids difference of the overlapping free form shapes being registered. Experimental results show that it outperforms three selected state-of-the-art algorithms on the whole for the accurate and robust registration of real overlapping free form shapes captured using two different laser scanners under typical imaging conditions.

In this paper, we make a detailed study of two rigid-motion constraints. The importance of these two constraints is twofold: first, they reveal the inherent relationship between the three-dimensional-two-dimensional (3-D-2-D) point correspondences and the motion parameters of interest; second, they can be used to measure the traditional ICP criterion established point match qualities based on which different point matches can be compared and relatively good point matches can be selected for motion-parameter update in the projective registration of free-form surfaces subject to small translational motions. The experimental results based on both synthetic data and real images have shown that the rigid motion constraints are powerful in evaluating the possible 3-D-2-D point matches established by the traditional ICP criterion, thus achieving encouraging projective registration results.

An important aspect of current computer vision research is the analysis of scenes, and particularly the extraction of their 3D structure and their segmentation into objects. Linked to that is the interpretation of the images and/or recognition of types of images, either from the output of the 3D reconstruction and segmentation or directly from the images. Solving such problems is often accomplished by combining a variety of methods and/or attributes of the images, as the papers in this special issue show. Ladicky et al. provide us with a framework to unify dense stereo reconstruction and object segmentation, where both are formulated as Random Field labelling which are jointly optimized. Evaluation is done on an enhanced Leuven data set, which is publicly available. The paper by Hwang and Grauman provides an enhanced image search methodology by incorporating high-level human scene perception aspects, such as the order of associated key-words, into the visual representation used. Sun et al. present a system that iteratively combines object detection, 3D scene layout estimation, and segmentation of the objects’ support region. As knowledge of the scene becomes available from the layout estimation, it can be used to improve the confidence in the object detection and so remove false detections. The results are demonstrated on the authors’ own dataset and two publicly available datasets.