Search

Image stitching of similar scenes is a challenging task when scenes are captured under varying illuminations between the scenes, varying camera positions, varying orientations either in axial or azimuth. In this paper, we explore a seamless image stitching algorithm to address the above-said issues by applying techniques of dehazing on the acquired scenes and before identifying the image features and holoentropy aided feature matching on the Scale Invariant Feature Transform (SIFT) based features for the image. Experimentation of the proposed system is compared with the existing image stitching methods using squared distance, Minkowski and pairwise Euclidean distance for feature matching. The proposed seamless stitching method is evaluated based on the metrics, horizontal square gradient value (HSGV) and vertical square gradient value (VSGV). The obtained results are shown to be feasible for stitching the nonuniform or illumination variation multiple images. The exploration of above said stitching algorithm is intended to reduce the number of computations and inconsistencies in the stitched results.

Due to overexposure, jitter, motion, and other spatiotemporal-varying perturbations, the collected images always undergo various visual distortions (e.g., deformation, partially occluded signs, fisheye respective, affine or 3D projections, in-plane and out-of-plane rotation) during acquisition or transmission procedure. Deep neural networks (DNNs) perform poorly on such pristine images in terms of high-level abstract operations, e.g., object categorization and semantic segmentation. To conquer this legacy, a distortion-tolerant model denoted as CapsNetSIFT is proposed to enhance representability and detectability of target in distorted imagery. We modify and integrate capsule network (CapsNet) with scale invariant feature transform (SIFT) together, both of which boast innate invariance to spacial-scale transformations. Two key insights, the customized multi-dimensional CapsNet (MD-CapsNet) and vector matching SIFT (VM-SIFT), can cooperate together and reinforce each other: the former encodes and provides representative feature vectors for the later, whilst the later localizes space-scale invariant interval dimensions (instead of pixels) and establish correspondence between source standard images (high-quality training images) and distorted ones (testing images). Thus, the category of one source standard image owning the most associations is the ground-truth category. Evaluation results reveal that employing CapsNetSIFT for distorted target recognition (CUB-200–2011, Stanford Dogs, Stanford Cars, and our hand-crafted dataset), significantly improves the resistance against various simulated distortions, and outperforms state-of-the-arts with relatively higher training and testing accuracy (93.97% and 91.03%).

This paper aims to develop deep-learning-based algorithms to automatically diagnose the nonerosive reflux disease (NERD) using the near focus narrow band imaging (NF-NBI) images, which are collected by clinicians of King’s College Hospital. To diagnose this disease, we propose a deep learning classification system to distinguish the NF-NBI images captured in the esophagus of healthy people and the NERD patients, which is a binary classification of two classes: non-NERD and NERD. To achieve an effective and accurate classification, we first propose an algorithm to automatically extract the region of interest (ROI) from the NF-NBI images and then generate image patches through a patch-generating algorithm. After that, we train six representative state-of-the-art deep convolutional neural network (CNN) models (ResNet18, ResNet50, ResNet101, DenseNet201, InceptionV3, and Inception-ResNetV2) to extract robust hierarchical features from these patches and classify them based on the hierarchical features. Finally, to determine the classification results of each subject, majority voting is employed to the corresponding generated NF-NBI image patches. We verify our classification system by ten-fold cross-validation using the clinical dataset. We perform subject-dependent and subject-independent experiments. In both experiments, we compare the classification performance of the ROI-based CNN models (the CNN models with our proposed ROI-based algorithms) with the CNN models. Meanwhile, we compare the classification performance of the ROI-based CNN models with the local binary pattern (LBP)-based support vector machine (SVM) classifier, the histograms of oriented gradients (HOG)-based SVM classifier, and the scale-invariant feature transform (SIFT)-based SVM classifier. The results show that the ROI-based CNN models are able to obtain higher average mean of ten-fold test accuracy on image level than the CNN models in the subject-dependent experiment (29.0% improvement) and the subject-independent experiment (10.5% improvement), which demonstrate the effectiveness of our proposed ROI-based algorithms. Meanwhile, the ROI-based CNN models are able to obtain higher average mean of ten-fold test accuracy on image level than the SVM classifiers in the subject-dependent experiment (20.5% improvement) and the subject-independent experiment (14.0% improvement), which demonstrate the ROI-based CNN models have better classification performance than the SVM classifiers. Among the ROI-based CNN models, the ROI-based InceptionV3 model achieves the best classification performance in the subject-dependent experiment, while the ROI-based Inception-ResNetV2 model achieves the best classification performance in the subject-independent experiment, which suggests the ROI-based Inception-ResNetV2 model has better generalization ability than the ROI-based InceptionV3 model. Moreover, the highest mean of ten-fold test accuracy (77.8%) on subject level obtained by using the ROI-based InceptionV3 model or the ROI-based Inception-ResNetV2 model demonstrates the practicality of our proposed classification system for assisting clinical diagnosis of the NERD.

Person re-identification plays an important role in searching for a specific person in a camera network with non-overlapping cameras. The most critical problem for re-identification is feature representation. In this paper, a multi-level cross-view consistent feature learning framework is proposed for person re-identification. First, local deep, LOMO and SIFT features are extracted to form multi-level features. Specifically, local features from the lower and higher layers of a convolutional neural network (CNN) are extracted, these features complement each other as they extract apparent and semantic properties. Second, an ID-based cross-view multi-level dictionary learning (IDB-CMDL) is carried out to obtain sparse and discriminant feature representation. Third, a cross-view consistent word learning is performed to get the cross-view consistent BoVW histograms from sparse feature representation. Finally, a multi-level metric learning fuses multiple BoVW histograms, and learns the sample distance in the subspace for ranking. Experiments on the public CUHK03, Market1501, and DukeMTMC-ReID datasets show results that are superior to many state-of-the-art methods for person re-identification.

Convolutional neural networks (CNN) have achieved tremendous success in tackling various tasks of computer vision problems such as classification of images and recognition of objects. The general trend has been to make networks larger and more complex in order to achieve higher precision. In any case, these advances to improve precision are not really making the systems increasingly effective for size and speed. In this report, an attempt to overcome the aforementioned issues is described. Some well-known scale and rotation invariant local features such as SIFT, SURF, ORB, BRISK, and KAZE were first observed to check which of these having the strong potentiality to replace inclusively the convolutional layers of CNN models. To do this, while retaining the fully connected layers of the corresponding CNN model as the classifier, the convolutional parts of the model were inclusively taken out and assigned the high-level feature maps at the flatten layer with scale and rotation invariant local features. The computational experiments were carried out using the challenging mammographic images provided by MIAS (mammographic image analysis society). It contains 322 grayscale images from 2 different severity, each of which has 1024x1024 pixels. Employing only the fully connected layers of the VGG16 model as the classifier, it is found that SIFT and SURF based features offer much better training and testing accuracies compared to ORB, BRISK, and KAZE in classifying MIAS image datasets. Using SIFT and SURF local features and employing only VGG16 or MobileNetV2 fully connected layers as a classifier, it can be shown that SIFT and SURF based deep learning exhibit state-of-the-art performance on MIAS datasets. SIFT and SURF based deep learning are capable of outperforming the performance of the original VGG16 and MobileNetV2 configuration in terms of accuracy and training time needed on the same datasets.

Computer-aided breast mass classification is an effective and widely used technology to assist pathologists in formulating clinical diagnoses and improving working efficiencies. Existing studies usually use a single image feature to perform breast mass classification. Herein, we propose a simple, yet effective, model called the DE-Ada*, which is an organic integration of multi-feature fusions, for breast mass classification. Firstly, we extract a set of complementary features, namely, scale-invariant feature transform (SIFT), GIST, histogram of oriented gradient (HOG), local binary pattern (LBP), residual network (ResNet), densely connected convolutional networks (DenseNet), and visual geometry group (VGG), to characterize mammograms from diverse perspectives. We attempt to mine the cross-modal pathological semantics among these features and complete their early fusion. The dynamic weight of any feature or cross-modal pathological semantics is computed and utilized to complete mid-level feature fusion. Finally, we design two voting-based ensemble learning strategies to implement late feature fusion. Our experiments demonstrate that the DE-Ada* model outperforms baselines on two well-known mammographic datasets. Our model encourages the use of cross-modal pathological semantics to deal with the overfitting problem.

This paper presents an approach for Structure from Motion (SfM) for unorganized complex image sets. To achieve high accuracy and robustness, image triplets are employed and an (approximate) internal camera calibration is assumed to be known. The complexity of an image set is determined by the camera configurations which may include wide as well as weak baselines. Wide baselines occur for instance when terrestrial images and images from small Unmanned Aerial Systems (UAS) are combined. The resulting large (geometric/radiometric) distortions between images make image matching difficult possibly leading to an incomplete result. Weak baselines mean an insufficient distance between cameras compared to the distance of the observed scene and give rise to critical camera configurations. Inappropriate handling of such configurations may lead to various problems in triangulation-based SfM up to total failure. The focus of our approach lies on a complete linking of images even in case of wide or weak baselines. We do not rely on any additional information such as camera configurations, Global Positioning System (GPS) or an Inertial Navigation System (INS). As basis for generating suitable triplets to link the images, an iterative graph-based method is employed formulating image linking as the search for a terminal Steiner minimum tree in the line graph. SIFT (Lowe, 2004) descriptors are embedded into Hamming space for fast image similarity ranking. This is employed to limit the number of pairs to be geometrically verified by a computationally and more complex wide baseline matching method (Mayer et al., 2012). Critical camera configurations which are not suitable for geometric verification are detected by means of classification (Michelini and Mayer, 2019). Additionally, we propose a graph-based approach for the optimization of the hierarchical merging of triplets to efficiently generate larger image subsets. By this means, a complete, 3D reconstruction of the scene is obtained. Experiments demonstrate that the approach is able to produce reliable orientation for large image sets comprising wide as well as weak baseline configurations.

Efficient matching of local image features is a fundamental task in many computer vision applications. However, the real-time performance of top matching algorithms is compromised in computationally limited devices, such as mobile phones or drones, due to the simplicity of their hardware and their finite energy supply. In this paper we introduce BEBLID, an efficient learned binary image descriptor. It improves our previous real-valued descriptor, BELID, making it both more efficient for matching and more accurate. To this end we use AdaBoost with an improved weak-learner training scheme that produces better local descriptions. Further, we binarize our descriptor by forcing all weak-learners to have the same weight in the strong learner combination and train it in an unbalanced data set to address the asymmetries arising in matching and retrieval tasks. In our experiments BEBLID achieves an accuracy close to SIFT and better computational efficiency than ORB, the fastest algorithm in the literature.

Panoramic video is a sort of video recorded at the same point of view to record the full scene. With the development of video surveillance and the requirement for 3D converged video surveillance in smart cities, CPU and GPU are required to possess strong processing abilities to make panoramic video. The traditional panoramic products depend on post processing, which results in high power consumption, low stability and unsatisfying performance in real time. In order to solve these problems,we propose a real-time panoramic video stitching framework.The framework we propose mainly consists of three algorithms, LORB image feature extraction algorithm, feature point matching algorithm based on LSH and GPU parallel video stitching algorithm based on CUDA.The experiment results show that the algorithm mentioned can improve the performance in the stages of feature extraction of images stitching and matching, the running speed of which is 11 times than that of the traditional ORB algorithm and 639 times than that of the traditional SIFT algorithm. Based on analyzing the GPU resources occupancy rate of each resolution image stitching, we further propose a stream parallel strategy to maximize the utilization of GPU resources. Compared with the L-ORB algorithm, the efficiency of this strategy is improved by 1.6-2.5 times, and it can make full use of GPU resources. The performance of the system accomplished in the paper is 29.2 times than that of the former embedded one, while the power dissipation is reduced to 10W., Comment: under review for Pattern Recognition Letters

Acquiring the relative pose between an uncooperative target and a chaser satellite poses a unique problem for any active debris removal mission. A lack of prior knowledge regarding the target's motion, mass distribution and shape limits the possibilities for accurately tracking the target. In this paper, a stereo-camera pair, mounted on a chaser satellite, is used to extract unique features on the surface of an unknown, uncooperative target using the scale invariant feature transform (SIFT). The features are used as measurement input to an extended Kalman Filter (EKF) that makes use of the simultaneous localisation and mapping (SLAM) approach. The orientation and position of the target relative to the chaser is estimated, along with the target's linear and angular velocities. This motion is estimated relative to the camera reference frame (CRF) while the system simultaneously calculates the shape and size of the target. A simulation environment is created to test and verify the estimation algorithm. The integration of the feature extractor with the EKF is tested using real camera data. A laboratory experiment was conducted to capture image sequences of a moving target. A number of practical considerations is highlighted when such a system is to be applied to a mission. Results show that stereo cameras, along with the EKF-SLAM approach is a viable method for fully autonomous space debris mitigation systems.

Image-based classification of histology sections plays an important role in predicting clinical outcomes. In this paper, we propose a Locality-Constrained Group Lasso Coding (LCGLC) method for microvessel image classification, which realizes the automatic “hot spot” detection of angiogenesis for human liver carcinoma. First, we extract Scale-Invariant Feature Transform (SIFT) descriptors on the Single-Opponent (SO) feature map, which simulates the biological functionality of human visual systems. Then, we present the feature-biased dictionary learning to effectively generate the dictionary of SIFT descriptors. With the learned dictionary, our LCGLC method introduces the locality constraint in classical group lasso problem to encode SIFT descriptors. Furthermore, we apply the Spatial Pyramid Matching (SPM) for the code pooling of microvessel images. Finally, we use Support Vector Machine (SVM) to classify a tissue image as having angiogenesis or not. Comprehensive experiments on the microvessel dataset show that the proposed LCGLC method achieves better performance compared with other representative approaches.

This paper presents a novel Deep Neural Network aimed at fast and robust visual loop detection targeted to underwater images. In order to help the proposed network to learn the features that define loop closings, a global image descriptor built upon clusters of local SIFT descriptors is proposed. Also, a method allowing unsupervised training is presented, eliminating the need for a hand-labelled ground truth. Once trained, the Neural Network builds two descriptors of an image that can be easily compared to other image descriptors to ascertain if they close a loop or not. The experimental results, performed using real data gathered in coastal areas of Mallorca (Spain), show the validity of our proposal and favourably compares it to previously existing methods.

Drowsiness during driving is a very serious concern in the society. Thousands of human lives are affected due to drowsy driving. So a system that can detect drowsy drivers is necessary in this situation. This paper does a comparative analysis of the approaches called Scale Invariant Feature Transform(SIFT) and RootSIFT for drowsy features extraction. RootSIFT is an enhanced SIFT descriptor. SIFT has been the widely used technique for feature extraction due to its invariance to scale, rotation, illumination, viewpoint and translations. So the enhancement to SIFT to detect drowsy features has made an outcome more likely. The enhanced SIFT called RootSIFT achieves 93.55% accuracy, which is better than normal SIFT in extracting the drowsy features.

Inspired by generative adversarial network (GAN), we propose a novel unsupervised approach for loop closure detection in autonomous unmanned systems. A binary GAN model dedicated to mobile application scenarios is designed to obtain binary feature descriptors, which are further incorporated into the most commonly used Bag of Visual Words (BoVW) model for loop closure detection. Compared with those hand-crafted features like SIFT and ORB, the performance of loop closure detection in complex environments with strong viewpoint and condition changes can be greatly improved. Compared with existing supervised approach based on convolutional neural network like AlexNet and AMOSNet, the cost-expensive task of supervised data annotation is totally avoided, which make the proposed approach more practical.

The detection of fatigue cracks has gained widespread attention as it affects the safety of daily lives. Many crack detection methods have been developed up to now and perform well. However, there are still some problems left. It is hard to check the huge objects (such as bridge, building) by using vibration or ultrasonic sensor. Especially the inspection of bridges needs human inspectors. It is high cost, difficult access and dangerous. Vision based fatigue crack detection is considered as a good solution as the system can be carried by a drone and overcomes the above problem. In this study, FAST, ORB, SIFT and SURF features are employed to detect the fatigue cracks by feature matching and this study compares their performance. The experimental results show that ORB based feature matching gives the best result and it is an effective method to detect the fatigue crack.

Multi-module images registration (IR) is a challenging task in the area of hyperspectral images collected from satellite and airborne sensors. This is because of substantial illumination variation and geometric distortions (GD), high accuracies and robust matching of multi-module hyperspectral images is a difficult task. Along with, the ground surface variation challenges feature point set identification in quantity and quality, which is a general problem posed by feature set based IR methods. For addressing the problem existing IR model are designed using SIFT and CNN. However, under extreme appearance difference, identified feature point sets extracted by existing IR method might contain higher ratio of outliers, whereas inlier might be insufficient and unequally dispersed. For overcoming research problem, this paper presented a novel and efficient automatic IR method for hyperspectral image. Feature extraction is done using SURF and SIFT descriptors initially then, using K-Nearest neighbour (KNN) graph the mismatch point is eliminated. Further, the positional accuracy is improved using least square sum method. Multi-quadric transformation method is used for attaining better IR quality at the end. The proposed IR method attain significant performance when compared with existing IR method.

A combination of multivariate empirical mode decomposition (MEMD) and the delay vector variance (DVV) method is explored for the diagnosis of plant-wide oscillations. The proposed algorithm improves upon the already presented method based solely on the DVV approach. The proposed modification makes it possible for the DVV method to work not only for the non-stationary data but also the case where multiple oscillations sources are present in the control loop. The focus of this work has been to distinguish between linear and non-linear causes of oscillations and isolating the source of non-linearity. A noise assisted multivariate empirical mode decomposition is used to sift out different oscillating components in the plant wide data and the nature of the oscillation is then ascertained using the already presented DVV approach. In the case of oscillations due to non-linear causes the method can isolate the source using the extent of non-linearity. The method is tested on simulated and industrial case studies and the results are compared with existing methods based on higher order statistics [8] and surrogate based methods [27] .

Recent image retrieval techniques are focusing on multiple image features for the efficient image retrieval. It has been an inevitable requirement to fetch the images from a variety of semantic groups and datasets. It is vital to retrieve the images based on their primitive features shape, texture, color and spatial information to cater the versatile image datasets. State-of-the-art detectors and descriptors are capable of finding the interest points based on their specialty. To encompass the strength of the image features for the information fusion purpose this contribution presents a novel technique to fuse the spatial color information with shaped extracted features and object recognition. For RGB channels L2 spatial color arrangements are applied and features are extracted, thereby fused with intensity ranged shapes formed by connecting the discovered edges and corners for the grey level image. Perifoveal receptive field estimation with 128-bit cascade matching with symmetric sampling on the detected interest points that discovers the potential information for the complex, overlay, foreground and background objects. Firstly the process is accomplished by reducing the massive features vectors, selecting high variance coefficient and secondly obtaining the indexing and retrieval by employing a Bag-of-Words approach. Extensive experiments are conducted on ten highly recognized image dataset benchmarks, specialized for texture, shapes, colors and objects including ImageNet, Caltech-256, Caltech-101, 102-Flower, Corel-10,000, 17-Flower, Corel-1000, COIL, ALOT and FTVL tropical fruits. To check the affectivity and robustness of the proposed method, it is compared with state-of-the-art detectors and descriptors SIFT, SURF, HOG, LBP, DoG, MSER and RGBLBP. Encouraging results reported that the proposed method has a remarkable performance in most of the image categories of versatile image datasets and can gain better precision to those of the state-of-the-art detectors and descriptors.

A method for face recognition system for both challenging frontal and profile faces is proposed in this paper. The proposed system consists of face pre-processing, feature extraction and classification components. During pre-processing, a region-of-interest for face region is extracted based on facial landmark points, obtained by a Tree Structured Part Model. During feature extraction, Scale Invariant Feature Transform descriptors are computed from patches over detected face region. These descriptors undergo to different feature learning techniques to obtain different feature representations for the input image. The performance of these feature representations are obtained using multi-class linear Support Vector Machine classifier during classification. Finally, the scores from different feature learning techniques are fused to take the decision to recognize the subjects. Extensive experimental results have been demonstrated to show the effectiveness of the proposed face recognition system. The comparison with the exiting state-of-art methods for ORL, IITK, CVL, AR, CASIA-Face-V5, FERET and CAS-PEAL face databases, show the superiority of the proposed system.

Because the fault features of planetary gear are very weak, the feature extraction process of planetary gear fault under traditional thinking is becoming more and more complex, and the amount of extracted features is very small. To solve these problems, a new fault diagnosis method for planetary gear based on image feature extraction and bag-of-words (BoW) model is proposed, which does not need complex noises reduction and redundant features elimination. In order to extract a large number of features quickly, the FAST-Unoriented-SIFT feature extraction algorithm is proposed, which combines the advantages of FAST and SIFT algorithms, and ignores the orientations of features. After converting the original vibration signals to gray-scale images, the features are extracted directly by FAST-Unoriented-SIFT, and finally the BoW model is built on the extracted features to realize fault classification. The overall recognition rate reaches 98.67%. The experimental results certify the effectiveness of the proposed method.

To hallucinate super-resolution (super-res) face from a real low-quality face, a super-resolution technique based on definition-scalable inference (SRDSI) is proposed in this paper. In the proposed strategy, all high-res labeled faces are first decomposed into basic faces and enhanced faces to train a basic face and an enhanced face inferring model, and then two inferring models are used to hallucinate super-res basic face with low-definition and enhanced faces with high-frequency information from a single low-res face. Finally, the basic face is merged with its enhanced face into a super-res face with high-definition. In addition, this paper employs SIFT key-points to evaluate the similarity between the super-res face and its high-res labeled face. Experimental results show that SRDSI can effectively recover more structural information as well as SIFT key-points from real low-res faces and achieves better performance than state-of-the-art super-resolution techniques in terms of both visual and objective quality.

Low-resolution images present challenges to a variety of object recognition problems in a variety of surveillance and navigation applications. In recent years, deep learning has advanced the state of the art in image super-resolution (SR) in terms of pixel domain peak signal to noise ratio (PSNR)/ mean square error (MSE). Inspired by the recent advances of deep convolutional neural networks in general image SR tasks, we develop a computer vision task-driven image SR solution by learning super-resolved gradient images using multiple convolutional neural networks for different scales. Recovering super-resolved gradient images at multiple scales, enables the system to recover more information useful for high level vision tasks than simply SR in the pixel domain. In particular, we propose a residual learning framework to perform image SR in the Difference of Gaussian (DOG) domain. The trained residual network models are then adapted to drive a widely adopted key point algorithm for image recognition, i.e. the SIFT detection and matching. Experimental results show that the proposed approach can significantly improve the SIFT keypoints repeatability compared to the state of the art in pixel domain image SR solutions.

One of the tasks Law Enforcement Agencies are responsible for is to find evidence of criminal activities in the Darknet. However, visiting thousands of domains to locate visual information containing illicit acts manually requires a considerable amount of time and human resources. To support this task, in this paper, we explore the automatic classification of images uploaded to Tor darknet. Unfortunately, the foreground objects on such images are not always presented standalone, without background, due to the environmental conditions. To address this challenge on the digital investigation of Tor darknet visual content, we propose to classify automatically only relevant parts of the image combining saliency maps, i.e. to select the regions with the most salient information, with Bag of Visual Words (BoVW). We introduce Semantic Attention Keypoint Filtering (SAKF), a filtering strategy that removes non-significant features at a pixel level that mainly do not belong to the object of interest or foreground. We assessed SAKF on seven publicly available datasets, obtaining from 1.64 to 15.73 points higher accuracies than the method set as the baseline, i.e. BoVW using dense SIFT (Scale-Invariant Feature Transform) descriptors. We also compared SAKF filtering performance against the deep features extracted from two well-known Convolutional Neural Network (CNN) architectures, namely MobileNet and ResNet50. Experimental results reveal the effectiveness of the proposed approach and highlight that the use of automatic image classification could be advantageous to support daily Law Enforcement Agencies investigations on Tor darknet.

Extracting an effective representation to analyze affection in videos is an inherent challenge. To address this problem, we present a novel emotion recognition system that can intelligently analyze and automatically recognize video emotions. We observe that most of the emotions in a video are closely related to the roles, especially those of the protagonist, which motivates us to explore how to utilize the protagonist’s information to help recognize video emotions. By analyzing the traits of the protagonists, we suggest a new solution to detect the protagonists to adapt not only to the whole video but also to a video clip. Moreover, a new keyframe selection strategy based on the protagonist is designed to select a set of representative frames from video clips. Furthermore, the scale invariant feature transform (SIFT) features matrix, built from each keyframe, is fed into a convolutional neural network (CNN) to learn the discriminative representations, which makes the CNN and local features complement each other. Considering that emotions are usually continuous, we introduce temporal information into the CNN by using optical flow images. Additionally, we extract some handcrafted visual and audio features as a supplement. Finally, all the features, including the features learned from the CNN and the handcrafted features, are fused and input into a support vector machine (SVM) and a support vector regression (SVR) for video emotion recognition. The proposed system is validated on a public dataset (LIRIS-ACCEDE) and a new dataset (PMSZU). The experimental results demonstrate the promising performance of the proposed system, which achieves better performance than the compared methods. Our designed system for recognizing video emotions could also facilitate the development of similar video expert systems, for applications such as video recommendations, video classification and video retrieval.

The deep learning models especially the CNN have achieved amazing performance on natural image retrieval. However, remote sensing images captured with anamorphic lens are still retrieved via manual selection or traditional SIFT-based methods. How to leverage the advanced CNN models for remote sensing image retrieval is a new task of significance. This paper focuses on the aurora images captured with all-sky-imagers (ASI). By analyzing the imaging principle of ASI and characteristics of aurora, a salient region determination (SRD) scheme is proposed and embedded into the Mask R-CNN framework. Thus, we can regard an image as a "bag" of salient regions (BoSR). In practice, each salient region is represented with a CNN feature extracted from the SRD embedded Mask R-CNN. After clustered to generate a visual vocabulary, each CNN feature is quantized to its nearest center for indexing. In the stage of online retrieval, by computing the similarity scores between query image and all images in the dataset, ranking results can be obtained and image with the highest value is exported as the top rank. Extensive experiments are conducted on the big aurora data, and the results demonstrate that the proposed method improves the retrieval accuracy and efficiency.

A ghost-free multi-exposure image fusion technique using the dense SIFT descriptor and the guided filter is proposed in this paper. The results suggest that the presented scheme produces high-quality images using ordinary cameras and that too without the ghosting artifact. To do so, the dense SIFT descriptor is used to extract the local contrast information from source images. Whereas, for the dynamic scenes, the histogram equalization and median filtering are used to calculate the color dissimilarity feature. Three weighting terms: local contrast, brightness, and color dissimilarity feature are used to estimate the initial weights. The estimated initial weights contain discontinuities. Therefore, the guided filter is used to remove the noise and discontinuity in initial weights. Finally, the fusion is performed using a pyramid decomposition method. Experimental results prove the superiority of the proposed technique over existing state-of-the-art methods in terms of both subjective and objective evaluation.

Facial expression recognition (FER) is the interesting research area that enables us to recognize the expression of the human face in the day-to-day life. Most of the traditional methods fail to recognize the expressions accurately as the expressions are based on the movements of the parts in the human face. The paper proposes the effective method of FER using the proposed Whale- Grasshopper Optimization algorithm based Multi-Support Vector Neural Network (W-GOA-based MultiSVNN). The features from the facial image is extracted using the Scale-Invariant Feature Transform (SIFT) and the proposed Scatter Local Directional Pattern (SLDP). The extracted features are classified using the proposed classifier to recognize the expression of the face. The proposed method of facial recognition enhances the recognition accuracy. The experimentation of the proposed algorithm is performed using the databases, such as Cohn-Kanade AU-Coded Expression Database and The Japanese Female Facial Expression (JAFFE) Database. The proposed algorithm outperforms the existing methods in terms of the accuracy, TPR, and FPR and the values are found to be 0.96, 0.96, and 0.009, respectively.

As an essential steam-water separation device in the secondary loop of nuclear power plant, the steam-water separation efficiency of corrugated plate dryer plays a decisive role in the economic indicators and safety of a nuclear power plant. Thus the research on the internal steam-water separation mechanism is of extremely significant value and the flow characteristics of the free falling water film on the corrugated plate wall are the basis of the research on water film rupture. The phenomenon of water film rupture on the wall surface will have a great impact on the secondary carrying, gas-water separation efficiency and even the economic indicators of nuclear power plants. However, it is meaningful to define whether the water film has broken and the fracture criterion because the method of defining the water film breakdown by the naked eye is extremely unreliable and inaccurate. Therefore, this paper proposes a method for defining the critical point of water film rupture on corrugated plate wall by the application of the SIFT (Scale-invariant feature transform) feature selection algorithm and SVM (Support Vector Machine) classification method. The SIFT (Scale-invariant feature transform) algorithm is applied to obtaining the characteristic information of the wall water film rupture image when the airflow velocity in the flow channel of the experimental section of the corrugated plate captured by the high-speed camera is gradually increased. Meanwhile, the two-class model of SVM (Support Vector Machine) is used to classify water film images and define the critical point of water film rupture on the corrugated plate wall by training, detection and inversion. The results show that the algorithm based on SIFT feature selection and SVM classification method can well define the critical point of water film rupture, keep the certainty and consistency of the water film rupture determination benchmark, and avoid the introduction of human identification error. It greatly improved the stability of the measurement and provides a reference for the determination of the baseline of water film rupture.

Low-light is a challenging environment for image processing and computer vision tasks, either in contrast enhancement for better visibility and quality, or application oriented tasks such as detection. We found that the current trend of low-light enhancement research is heavily on quality improvement. In this work, we aim to shift the focus towards a more functional direction, that is enhancement that prioritizes feature retrieval. For this reason, we first propose to model low-light enhancement as a set of localized functions using Gaussian Process ( GP ) that is trained at runtime using data from a simple Convolutional Neural Network (CNN) to provide the necessary feature information as reference. The CNN is in turn trained using large amount of synthetic data, based upon the luminance distribution of real world low-light images to learn the relationship between features and pixels. Secondly, we also proposed two new evaluation metrics to better assess enhancement algorithms to support high level computer vision tasks, namely, local features matching and intensity histogram similarity. In our experiments, our proposed low-light enhancement framework outperforms the state-of-the-arts with significant improvement in Recall, F 1 , and F 2 -score of SIFT features matching, and achieve comparable results for l 1 -norm distance of histograms as well as the PSNR. Moreover, our analysis of the performance showed that the PSNR quality metric is not only unable to assess the practicality of the results, but also inappropriately gives high assessment to low visual quality images.

Bag-of-features (BoF) model based melanoma classification methods can effectively assist dermatologists to diagnose skin diseases. Codebook learning is a key step in the BoF model and the k-means clustering algorithm is often used to learn a codebook. However, the cluster centers generated by k-means algorithm are irresistibly attracted to the denser regions. This produces a suboptimal codebook in which most of the clusters are located in dense regions and a few are in sparse regions. Therefore, this can easily affect the classification accuracy. In this paper, we develop a novel methodology for classifying skin lesions. Firstly, we propose a new codebook learning algorithm based on feature similarity measurement (FSM) to effectively quantify the original features of melanomas. We utilize the combination of the linearly independent and linear prediction (LP) algorithms to measure feature similarity. Especially, the codewords learned by the proposed FSM algorithm are not affected by the density of samples. Therefore, a more discriminating BoF histogram for the melanoma classification is achieved. Secondly, we propose a melanoma classification method based on the FSM codebook learning algorithm. In particular, we adopt the BoF histogram fusion strategy of different feature descriptors, i.e., RGB color histogram and scale-invariant feature transform (SIFT). Finally, the experimental results show that the proposed melanoma classification method outperforms some state-of-the-art methods in terms of classification accuracy and efficiency. The results also show the performance of the proposed method is greatly improved by the use of the proposed codebook learning algorithm.

Image processing technology has been used in precision dairy farming to support management decisions. Vision-based animal identification systems can become a potential alternative to RFID. In this paper, a vision system is proposed to extract body images and identify Holstein cows. Side view videos of dairy cattle walking in a straight line were collected. Cow mask was detected using Adaptive SOM method. The largest inscribed rectangle was extracted to locate the cow's body area. A total of 528 videos were collected from 66 cows, and 3 videos were randomly selected for each cow to build template datasets, while the rest of the videos were used as test data. Feature points of the body image were extracted and matched with the template dataset to identify the unknown cow. Four feature extraction methods and two matching methods were investigated and evaluated. The results showed that the highest identification accuracy was 96.72% when the FAST, SIFT and FLANN methods were used for feature extraction, descriptor, and matching, respectively. However, the combination of ORB and BruteForce had better computational efficiency on the basis of high accuracy. Software was implemented and can realise accurate identification of dairy cattle in real-time.

Ancient villages are the carrier of a nation's profound history and culture. They are the integration of history, culture, architecture and sculpture, and have many value attributes. With the development of society, the protection of ancient villages is very important. The establishment of digital archives is an important means to protect ancient villages. Because of the large number and wide distribution of ancient villages, crowd sourcing can quickly and widely access the digital resources of ancient villages. Because of the uneven quality and repetition of the images collected from ancient villages, it is necessary to screen the images of ancient villages. Therefore, this paper proposes a screening model of ancient villages based on SIFT and convolution neural network. Firstly, this paper chooses edge gray change rate and NIQE quality score to evaluate the quality of ancient village image; secondly, extracts SIFT features of image for matching, calculates matching similarity to determine whether the matched image is myopic repetition. Finally, the image is filtered or preserved by using convolution neural network with the edge gray change rate, NIQE quality score and some image attributes as features. Experiments show that the ancient village image screening model designed in this paper has higher accuracy and recall rate than other methods, and has better screening effect.

Descriptor and metric learning using deep convolutional neural networks (CNNs) have drawn attention of researchers in the domain of computer vision due to their remarkable performance over traditional methods. Different networks like two-channel, Siamese and triplet, etc., have been proposed recently with the aim to learn a metric or a low dimensional embedding from image patches and have outperformed traditional local descriptors like scale-invariant feature transform (SIFT). Plain CNNs resulting from stacking of several convolutional layers are employed in recent works. In this article, we have followed a recent approach called Deep Compare for metric and descriptor learning and have proposed improved architecture for two-channel and Siamese networks. Our proposed modification is inspired from the novel dense convolutional neural network known as DenseNet architecture. The proposed two-channel and Siamese networks employ dense convolutional layers which reuse feature maps from preceding layers. Our networks, trained with pairs of patches, outperform Deep Compare networks by a significant margin justifying the proposed architecture and obtain results comparable to triplet networks on UBC benchmark dataset. Moreover, we have obtained promising results on patch verification, image matching and patch retrieval tasks on large scale HPatches benchmark dataset using the descriptors from our Siamese network.

Nowadays, the traditional measurement methods applied in ship sections/blocks are heavily relied on years of experiences and could hardly meet the shipbuilding requirements due to their complicated and time-consuming processes. Furthermore, along with the development of photogrammetry, the image-based three-dimensional (3D) reconstruction technology has emerged and become flourishing. In this paper, we propose a novel measurement method based on image-based 3D reconstruction technology to measure the construction deformation of ship blocks with high accuracy in a simple way. The scheme only needs photos to rebuild the 3D model of ship block. It includes the camera calibration, the image preprocessing, the feature points matching followed by the steps of dense point cloud generation, meshes partition and 3D reconstruction with texture mapping. In the step of feature points matching, the scale-invariant feature transform (SIFT) algorithm is used and optimized with the addition of the global vector to reduce the mismatch and enhance the reconstruction accuracy. The shipyard experiments with shooting photos of arbitrary steel parts proved that the proposed image-based 3D reconstruction technology is able to be applied to measure the ship blocks with satisfied accuracy and less time in shipbuilding industry or other industries with large-scale one-of-a-kind product as well.

An organization uses a symbol as its representation in the market for ease of identification and uniqueness. Logos are used to identify and retrieve the materials, even in a complex environment for further analysis. Algorithms based on support vector machine and neural networks provide better results in retrieval of the document from small dataset. But inlarge data sets the existing models lags in their classification performance. This proposed model uses ant colony optimization (ACO) along with the local descriptor scale-invariant feature transform (SIFT), as a hybrid model for retrieving document from dataset. This hybrid model enhances the performance of the retrieval model in terms of increased efficiency, leading to an accuracy of 95.86% with a high output precision of 97.67%.

Evaluating the performance of photovoltaic panels inevitably involves having the exact model of solar cells. Different approaches to model solar cells have been proposed in literature which can generally be classified as either traditional or intelligent methods. To obtain the accurate model of such highly nonlinear systems, however, is still a challenging task, defying researchers. This study proposes an Improved Lozi Map based Chaotic Optimization Algorithm (ILCOA) algorithm to estimate the unknown parameters of the solar cells. Remarkable local and global searching abilities of the proposed algorithm give it a distinct edge over other optimization methods, enabling it to sift the whole search space for the global optimum. The efficacy of the proposed approach is finally highlighted by comparing its performance with those of three other algorithms including BMO, CWOA, and LCOA in terms of RMSE, Relative Error, Mean Absolute Error (MAE), Normalized MAE, and Mean Bias Error.

In this paper, the task of unsupervised visual object categorization (UVOC) is addressed. We utilize a variant of Self-organizing Map (SOM) to cluster images in two different scenarios: disjoint (images from Caltech256) and non-disjoint (images from MSRC2) sets. First, we ran several tests to evaluate different image representation techniques: features obtained by a deep convolutional network were compared with those obtained by handcrafted methods, such as SIFT combined with a set of interest point detectors. As expected, we found that deep convolutional network features significantly outperformed its handcrafted counterparts. After choosing the best image representation technique, we compared the state-of-the-art image clustering algorithms with a SOM-based subspace clustering method that identifies automatically the relevant features in the high-dimensional image representations. The results have shown that our method achieves substantially lower clustering error than all competitors in several challenging testing settings.

Facial expression recognition (FER) plays an important role in the applications of human computer interaction. Given the wide use of convolutional neural networks (CNNs) in automatic video and image classification systems, higher-level features can be automatically learned from hierarchical neural networks with big data. However, learning CNNs require large amount of training data for adequate generalization, while the Scale-invariant feature transform (SIFT) does not need large training samples to generate useful feature. In this paper, we propose a new hybrid feature representation for the recognition of facial expressions from a single image frame that uses a combination of SIFT and deep-learning feature of different level extracted from the CNN model, then adopt the combined features and classify the expression by support vector machines (SVM). The performance of the proposed method has been validated on public CK+ databases. To evaluate the generalization ability of our method, we also performed an experiment on a cross-database environment. Experimental results show that the proposed approach can achieve better classification rates compared with state-of-art CNN methods, which indicate the considerable potential of combining shallow feature with deep-learning feature.

Scene classification has become an effective way to interpret the High Spatial Resolution (HSR) remote sensing images. Recently, Convolutional Neural Networks (CNN) have been found to be excellent for scene classification. However, only using the deep models as feature extractor on the aerial image directly is not proper, because the extracted deep features can not capture spatial scale variability and rotation variability in HSR remote sensing images. To relieve this limitation, a bidirectional adaptive feature fusion strategy is investigated to deal with the remote sensing scene classification. The deep learning feature and the SIFT feature are fused together to get a discriminative image presentation. The fused feature can not only describe the scenes effectively by employing deep learning feature but also overcome the scale and rotation variability with the usage of the SIFT feature. By fusing both SIFT feature and global CNN feature, our method achieves state-of-the-art scene classification performances on the UCMerced, the Sydney and the AID datasets.

The development of a virtual scenario is desirable in many hostile environments such as in mining or sewer pipelines to inspect its current condition and to identify possible failures. Obtaining a virtual image through the reconstruction of the mineshaft or the inner pipe is a valuable tool for assessment and fault detection. The approach presented in this paper uses a fuzzy controlled robot so that the video information can reliably be taken. The robot fuzzy control guarantees its motion in a straight line while taking the images from the pipe. The control algorithm therefore prevents the robot from overturning while moving inside the pipe. Results from our 3D reconstruction approach using the autonomous mobile robot is presented. The proposed method overcomes the limitations of different detectors by using a novel combination of DoG and Harris Laplace. Once the frame selection is made, then the regions with affine transformations are used to find image features as correspondences using image descriptors such as SIFT (Scale-Invariant Feature Transform). The correspondences are required to obtain the orientation and position of the camera and to obtain the scene structure leading to the creation of the 3D virtual environment. The 3D reconstruction method outperformed similar commercial software which highlights its potential.

Taking into consideration that the prior CMF detection methods rely on several fixed threshold values in the filtering process, we propose an efficient CMF detection method with an automatic threshold selection, named as CMF-iteMS. The CMF-iteMS recommends a PatchMatch-based CMF detection method that adapts Fourier-Mellin Transform (FMT) as the feature extraction technique while a new automatic threshold selection based on iterative means of regions size (iteMS) procedure is introduced to have flexibility in changing the threshold value for various characteristics (quality, sizes, and attacks) in each input image. To ensure the reliability of the proposed CMF-iteMS, the method is compared with four state-of-the-art CMF detection methods based on Scale Invariant Feature Transform (SIFT), patch matching, multi-scale analysis and symmetry techniques using three available datasets that cover the variety of characteristics in CMF images. The results show that the F-score of the CMF-iteMS outperformed existing CMF detection methods by exceeding an average of 90% F-score values for image-level evaluation and 82% of F-score value for pixel-level evaluation for all datasets in original size. As special attention is given to the image resizing attack, the method is able to maintain the highest performance even if the images in the datasets are resized to 0.25 parameter.

A vision inspection system is developed for detecting surface damages on cables in long-span cable-stayed bridges. The system consists of a climbing robot, an image processing platform, and 4 fixed cameras. While the climbing robot loads the whole system and ascends along the cables, 4 cameras can continuously acquire cable surface images. Then, image processing techniques can be employed to achieve defect identification. In our optical system, a single image may only collect a part of the whole defect because of the field of view (FOV) of each camera. In order to obtain the morphology of the whole defect, we present an efficient scale-invariant feature transform (SIFT) algorithm to achieve the multi-image mosaic with partially overlapped regions in different defect images. Experimental results demonstrate that the proposed vision inspection technologies are suitable for application to the bridge cables for defect detection and condition assessment.

Stable local image feature detection is a fundamental problem in computer vision and is critical for obtaining the corresponding interest points among images. As a popular and robust feature extraction algorithm, the scale invariant feature transform (SIFT) is widely used in various domains, such as image stitching and remote sensing image registration. However, the computational complexity of SIFT is extremely high, which limits its application in real-time systems and large-scale data processing tasks. Thus, we propose several efficient optimizations to realize a high-performance SIFT (HartSift) by exploiting the computing resources of CPUs and GPUs in a heterogeneous machine. Our experimental results show that HartSift processes an image within 3.07 ∼ 7.71 ms, which is 55.88 ∼ 121.99 times, 5.17 ∼ 6.88 times, and 1.25 ∼ 1.79 times faster than OpenCV SIFT, SiftGPU, and CudaSift, respectively.

This paper presents an approach to estimating the plant or fruit size from objects on the digital images of natural scenes. Two images are taken from camera positions with a known distance between them. For a selected object and a pair of its boundary pixels in the first image, a corresponding pixel pair is found in the second image. The Euclidean distances within the two pairs compare in similar triangles and combine with the known distance between the camera positions to estimate the size of the object in metric units. A dense correspondence among images is first required. It can be determined by joint information of multiple 2D views of the same scene utilizing image registration techniques. These are used to align corresponding image points, without having to rely on distinct features, such as geometric properties. This is most advantageous for weak textured areas, e.g. uniform fruit shades, where many methods for corresponding point detection fail. The pixel neighbourhoods of candidate corresponding pairs are compared by template matching to verify their similarity and to select the most reliable correspondences. A stratified rigid-to-elastic registration approach generates a deformation matrix whose elements define translational vectors on a pixel basis. The accuracy of correspondence, checked by colour template matching, is assessed by mean absolute error, D , between corresponding pixel-pair intensities within the templates. By matching 26 random fruit image pairs, the proposed approach detected 1390.5 ± 1129.8 reliable corresponding points on the surface of the fruit with the accuracy of D ≤ 5 , on average. This means approximately 18% of all objects’ pixels and considerably exceeds the results of comparable methods for high-density correspondence detection, such as correlation-based correspondence, non-rigid dense correspondence (NRDC), and scale invariant feature transform (SIFT). The number and quality of corresponding points obtained by the proposed algorithm turned out to be reliable and sufficiently robust for an accurate estimation of distances between objects and camera positions (with an overall accuracy of 0.11 m ± 0.06 m) and height of plants (with an accuracy of 0.14 m ± 0.1 m) when taking two similar outdoor photographs of a scene.

The object of interest of this paper is automatic iris classification when dealing with missing information. Our approach uses and extends a method for face recognition, based on Scale Invariant Feature Transform (SIFT). We adapted this method for iris classification and tested it on occluded iris images. We add to the keypoint matching procedure new conditions that improve the classification rate. We tested different parameters involved in the SIFT extraction process and the keypoint matching scheme on eleven image datasets with different levels of occlusion. For testing, a standardized segmented UPOL iris database was employed. We experimentally prove that the proposed approach has better results when compared with both the original method and the Daugman procedure on all datasets.

In today’s diversified world, with every person having their own idiosyncratic identity, the field of face detection and recognition has received significant scope. Since birth, humans develop and harvest similar facial features, making their recognition a challenging problem. Thus, the development of faster, more accurate and robust algorithms is of prime importance. The most critical requirements in developing a reliable face recognition algorithm is a large database of facial images and a well-developed systematic procedure to evaluate the system. In this paper, we bring out an approach to implement quick facial recognition using the concept of vocabulary trees in accordance with ORB, SIFT and SURF detectors and compare their performance. The model is trained using the Grimace, Face95 and ORL databases. The experiments are conducted by employing various state-of-the-art clustering algorithms and the results are compared for accuracy. The results and comparisons obtained on the databases show that the ORB detector clustered using Spectral clustering algorithm to construct the Vocabulary Tree outperforms the other techniques.

Ego motion determination is an important application to recognize the motion of an observer in real time. Here we have implemented feature point detector using FFME for ego-motion determination. The main objective of this work is to detect the motions of any objects such as humans, animals, vehicles, missiles and so on. The camera plays a vital role in surveillance of isolated and or restricted area. To improve the intelligence of camera some efficient video processing algorithms are needed. In this process, the detection of unwanted features were derived and developed for some video processing applications. There are various types of feature detectors like edge, corner and points which can be detected from a video frame. To detect those features, feature point detectors like SIFT, ORB, FAST, BRIEF and FFME are more reliable with good efficiency, less memory size and high processing speed. These feature point detectors are compared and evaluated under parameters of speed and number of feature point matches. Although, FFME algorithm has high speed when compared to other feature point detection algorithms, it is prone to more errors caused by illumination differences, as it detects the feature points using Gradient magnitude of the pixel. To overcome this, we have introduced a novel work that uses FFME with gradient phase technique. The resultant output exhibits high processing speed with less error when compared to other feature point detectors and even FFME with gradient magnitude.

Places classification is one of the points of discussion in the computer vision and robotics community. Some renowned techniques such as local-invariant feature extractors (e.g. Scale-invariant feature transform SIFT, Speeded Up Robust Features SURF), as well as Visual BoW approach were used in place classification problems. Nowadays, deep learning methods such as Convolutional Neural Networks (CNNs) have the advantages towards computer vision problems including place classification problem. Albeit, there are several renowned datasets existed to help the community to learn the models, there is no publicly exists in places dataset for specifically places in Indonesia. This paper presents methodology to collect data of visual places in Indonesia, learn deep features from the data, and classify visual places in Indonesia. We aims to contribute a large dataset as well as deep learning models of places in Indonesian. There are more than 16K images collected and augmented to build the places (specifically places in Indonesia) dataset. The highest accuracy score achieved by the models is 92%.

Now a days, the biometric system is playing a significant role, especially in the banking sector, law enforcement region. In previous times the unimodal system was used for biometrics but now the trend is for multimodal biometrics which is an important part of pattern recognition and grabs a lot of attention of the people in the research area. A lot of work is done on unimodal systems but very less work is done on multimodal characteristics. This paper deals with the secure authentication process using fingerprint and face recognition multimodal systems for high classification accuracy and low error rates. The feature extraction and optimization are done using SIFT & PSO for the facial category and the ridges and minutiae extractions are done for the fingerprint. The proposed solution is evaluated on the Ollivati dataset for faces and the FVC200 dataset for fingerprints. From the result and discussions, it can be seen that the proposed approach can attain an accuracy of approximately 99.2% which is our desired objective, and low false acceptance and rejection rates which increases the authenticity of our multimodal system.

Crack detection using image processing has recently become a major research topic in nondestructive inspection (NDT) and structural health monitoring (SHM). However, crack detection methods are not robust to variations such as illumination, weather, and noise, due to the speed and accuracy of stitching cannot meet the requirements of practical applications. Therefore, an automated crack detection system built basing on deep learning to replace manual visual inspection. This article presents a SIFT matching method based on an alternate-selection strategy to eliminate residual reprojection and geometric distortion errors. A variety of network models based on the fully convolutional network (FCN) architecture are proposed to address the problems of local information losses and partial refinement capacity reductions for improving detection accuracy and reduce the error mark under a complex background, which are frequently encountered in the crack detection algorithms of deep learning methods. Combining pretraining with fine-tuning to verify the detection performance of FCN-based networks and use the feature descriptor to determine the location and size of each crack. Verifing the robustness of the proposed approach for the steel-beam crack detection task on the crack image dataset by an extensive experimental evaluation.