Search

HighlightsAn ensemble method using color segmentation, deep learning, and image transformation was developed.Experiments were conducted to compare the method with other state-of-the-art tracking algorithms.The optimized ensemble method to track bolls achieved 94.4% accuracy using weakly trained tiny YOLOv2 models.The method achieved 7.6 frames per second and outperformed five other tracking methods.Abstract. In robotic applications, good perception can be computationally costly and create undesirable latency before a control decision is initiated. Most of the methods available for object detection deep learning are either fast with low accuracy or slow with high accuracy. Fast and accurate methods are necessary to track and localize objects such as cotton bolls that may be visible or occluded by each other or not well illuminated. In this study, an ensemble of a deep learning method and other image processing techniques was used to detect cotton bolls in-field on defoliated plants. In each image, a trained deep learning method, the YOLOv2 model, was used to detect open cotton bolls, and color segmentation was applied to confirm if the bolls detected by the YOLOv2 model were actually white to avoid false positives. Boll tracking was performed by following the spatial movement of good features on the edges of the bolls using the Lucas-Kanade algorithm. An image transformation algorithm was applied to the next image in case the previously detected boll was lost to retrieve the information of the missing boll. Each tracked and localized boll was stored and counted to give the total number of bolls detected. In this study, detection accuracy was sacrificed for image processing speed by using the YOLOv2 model. Detection accuracy was improved by using an ensemble method that combined image color segmentation, optical flow, and image transformation. This method was compared to eight other open-source methods implemented in OpenCV. The ensemble method detected and counted bolls at a speed of 7.6 fps with an accuracy of 94.4% using the Jetson TX2 embedded system to process 1K resolution images, outperforming the other OpenCV methods in various measurements. Keywords: Boll counting, Cotton, Cotton harvesting, DarkFlow, Darknet, Deep learning, Machine vision, YOLOv2.

In this paper, we propose SelfSphNet, that is, a self-supervised learning network to estimate the motion of an arbitrarily moving spherical camera without the need for any labeled training data. Recently, numerous learning-based methods for camera motion estimation have been proposed. However, most of these methods require an enormous amount of labeled training data, which is difficult to acquire experimentally. To solve this problem, our SelfSphNet employs two loss functions to estimate the frame-to-frame camera motion, thus giving two supervision signals to the network with the usage of unlabeled training data. First, a 5 DoF epipolar angular loss, which is composed of a dense optical flow of spherical images, estimates the 5 DoF motion between two image frames. This loss function utilizes a unique property of the spherical optical flow, which allows the rotational and translational components to be decoupled by using a derotation operation. This operation is derived from the fact that spherical images can be rotated to any orientation without any loss of information, hence making it possible to “decouple” the dense optical flow between pairs of spherical images to a pure translational state. Next, a photometric reprojection loss estimates the full 6 DoF motion using a depth map generated from the decoupled optical flow. This minimization strategy enables our network to be optimized without using any labeled training data. To confirm the effectiveness of our proposed approach (SelfSphNet), several experiments to estimate the camera trajectory, as well as the camera motion, were conducted in comparison to a previous self-supervised learning approach, SfMLearner, and a fully supervised learning approach whose baseline network is the same as SelfSphNet. Moreover, transfer learning in a new scene was also conducted to verify that our proposed method can optimize the network with newly collected unlabeled data.

Satellite images are primary data in weather prediction modeling. Deep learning-based approach, a viable candidate for automatic image processing, requires large sets of annotated data with diverse characteristics for training purposes. Accuracy of weather prediction improves with data having a relatively dense temporal resolution. We have employed interpolation and data augmentation techniques for enhancement of the temporal resolution and diversifications of characters in a given dataset. Algorithm requires classical approaches during preprocessing steps. Three optical flow methods using 14 different constraint optimization techniques and five error estimates are tested here. The artificially enriched data (optimal combination from the previous exercise) are used as a training set for a convolutional neural network to classify images in terms of storm or nonstorm. Several cyclone data (eight cyclone datasets of a different class) were used for training. A deep learning model is trained and tested with artificially densified and classified storm data for cyclone classification and locating the cyclone vortex giving minimum 90% and 84% accuracy, respectively. In the final step, we show that the linear regression method can be used for predicting the path.

Non-destructive ultrasonic analysis of materials is a method for assessing the integrity of the inspected components. It is commonly used in monitoring critical parts of the power plants, in aeronautics, oil and gas, and the automotive industry. Since most ultrasonic inspections rely on expert's previous experience they must constantly practice on new, unseen data. Acquiring enough data for training human experts on non- destructive ultrasonic scan analysis can be an expensive and time-consuming task. The only possibility to get new data for practicing is to implant synthetic defects in real metal blocks. Artificial defects are made by temperature strain, electrical discharge, and physical damage. All of those methods are very complicated and expensive to perform. Also metal blocks have to be taken from the components of the power plants to have the same structure and be realistic. In this work, some attempts have been made to generate 3D ultrasonic scans using computer vision and deep learning methods.

Simple Summary Animal behaviors are critical for survival, which is expressed over a long period of time. The emergence of computer vision and deep learning technologies creates new possibilities for understanding the biological basis of these behaviors and accurately quantifying behaviors, which contributes to attaining high production efficiency and precise management in precision farming. Here, we demonstrate that a dual-stream 3D convolutional neural network with RGB and optical flow video clips as input can be used to classify behavior states of fish schools. The FlowNet2 based on deep learning, combined with a 3D convolutional neural network, was first applied to identify fish behavior. Additionally, the results indicate that the proposed non-invasive recognition method can quickly, accurately, and automatically identify fish behaviors across hundreds of hours of video. Abstract The rapid and precise recognition of fish behavior is critical in perceiving health and welfare by allowing farmers to make informed management decisions on recirculating aquaculture systems while reducing labor. The conventional recognition methods are to obtain movement information by implanting sensors on the skin or in the body of the fish, which can affect the normal behavior and welfare of the fish. We present a novel nondestructive method with spatiotemporal and motion information based on deep learning for real-time recognition of fish schools’ behavior. In this work, a dual-stream 3D convolutional neural network (DSC3D) was proposed for the recognition of five behavior states of fish schools, including feeding, hypoxia, hypothermia, frightening and normal behavior. This DSC3D combines spatiotemporal features and motion features by using FlowNet2 and 3D convolutional neural networks and shows significant results suitable for industrial applications in automatic monitoring of fish behavior, with an average accuracy rate of 95.79%. The model evaluation results on the test dataset further demonstrated that our proposed method could be used as an effective tool for the intelligent perception of fish health status.

In the resilient and reliable electrical power system, the condition of high voltage insulation plays a crucial role. In the field of high voltage insulation integrity, the partial discharge (PD) inception and development trends are essential for assessment criteria in diagnostics systems. The observed trend to employ more and more sophisticated algorithms with machine learning features and artificial intelligence (AI) elements is observed everywhere. The classification and identification of features in PD images is perceived as a critical requirement for an effective high voltage insulation diagnosis. In this context, techniques allowing for anomaly detection, trends observation, and feature extraction in partial discharge patterns are important. In this paper, the application of few algorithms belonging to image processing, machine learning and optical flow is presented. The feature extraction refers to image segmentation and detection of coherent forms in the images. The anomaly detection algorithms can trigger early detection of the trend changes or the appearance of a new discharge form, and hence are suitable for PD monitoring applications. Anomaly detection can also handle transients and disturbances that appear in the PD image as an indication of an abnormal state. The future monitoring systems should be equipped with trend evolution algorithms. In this context, two examples of insulation aging and application of PD-based monitoring are shown. The first one refers to deep convolutional neural networks used for classification of deterioration stages in high voltage insulation. The latter one demonstrates application of optical flow approach for motion detection in partial discharge images. The motivation for the research was the strive to machine-controlled pattern analysis, leading towards intelligent PD-based diagnostics.

In this paper, a deep learning-based traffic state discrimination method is proposed to detect traffic congestion at urban intersections. The detection algorithm includes two parts, global speed detection and a traffic state discrimination algorithm. Firstly, the region of interest (ROI) is selected as the road intersection from the input image of the You Only Look Once (YOLO) v3 object detection algorithm for vehicle target detection. The Lucas-Kanade (LK) optical flow method is employed to calculate the vehicle speed. Then, the corresponding intersection state can be obtained based on the vehicle speed and the discrimination algorithm. The detection of the vehicle takes the position information obtained by YOLOv3 as the input of the LK optical flow algorithm and forms an optical flow vector to complete the vehicle speed detection. Experimental results show that the detection algorithm can detect the vehicle speed and traffic state discrimination method can judge the traffic state accurately, which has a strong anti-interference ability and meets the practical application requirements.

Optical Flow Estimation is an essential component for many image processing techniques. This field of research in computer vision has seen an amazing development in recent years. In particular, the introduction of Convolutional Neural Networks for optical flow estimation has shifted the paradigm of research from the classical traditional approach to deep learning side. At present, state of the art techniques for optical flow are based on convolutional neural networks and almost all top performing methods incorporate deep learning architectures in their schemes. This paper presents a brief analysis of optical flow estimation techniques and highlights most recent developments in this field. A comparison of the majority of pertinent traditional and deep learning methodologies has been undertaken resulting the detailed establishment of the respective advantages and disadvantages of the traditional and deep learning categories. An insight is provided into the significant factors that affect the success or failure of the two classes of optical flow estimation. In establishing the foremost existing and inherent challenges with traditional and deep learning schemes, probable solutions have been proposed indeed.

Objects counting in video frames is an important field of research that provides appropriate information about objects and their groups in video streams. An adequate technique to achieve this goal is using digital image/video processing methods with a detection method called CenterN et that is based on a deep learning architecture (i.e., a ConvNet architecture). This paper presents an objects counting method based on a combination of a detection model and optical flow generated between video frames.

Image/video processing has been one of the major developments in the recent history with its applications in areas of Road safety, military, medical and agriculture fields. Due to its complexity a generic solution for multiple object detection in extremely crowded scenes remains to be found. Traditional methods of optical flow, connected component analysis and image segmentation have been extensively studied in image processing and video processing material. With recent developments of machine learning and numerical optimization techniques the use of deep neural networks are getting frequent in image processing applications. Among such deep learningbased methods commonly used in this context are RCNN variants, Mask RCNN and YOLOv3. An exhaustive comparison of the traditional methods and deep learning-based methods and also deep learning methods are discussed in this paper. This study will be of use in selection of a method for any extremely crowded scene object detection problem.

An early fire detection in indoor environment is essential for people’s safety. During the past few years, many approaches using image processing and computer vision techniques were proposed. However, it is still a challenging task for application of video smoke detection in indoor environment, because the limitations of data for training and lack of efficient algorithms. The purpose of this paper is to present a new smoke detection method by using surveillance cameras. The proposed method is composed of two stages. In the first stage, motion regions between consecutive frames are located by using optical flow. In the second stage, a deep convolutional neural network is used to detect smoke in motion regions. Besides, to overcome the problem of lacking data, simulated smoke images are used to enrich the dataset. The proposed method is tested on our data set and real video sequences. Experiments show that the new method is successfully applied to various indoor smoke videos and significant for improving the accuracy of fire smoke detection. Source code and the dataset have been made available online.

Traditional image global registration algorithms are limited in principle and cannot accurately register large depth of field scenes or moving objects. The local registration method based on dense optical flow has the advantage of not being limited by a single transformation matrix, so that a better registration result can be obtained. However, traditional dense optical flow algorithms are limited by large computational complexity and are difficult to achieve real-time estimation, thus limiting their application. In recent years, many dense optical flow algorithms based on deep learning (such as PWC-Net) have emerged, which have achieved the effect of surpassing traditional optical flow algorithms on public datasets and can be estimated in real time. Based on this, this paper proposes an algorithm flow based on deep learning to predict dense optical flow and use it for registration. And a self-built optical flow data set for supervised learning of the network has also been proposed. Using the same network, the registration results of our datasets are better than those of existing datasets.

At present, the GF-4 satellite is the world’s highest resolution geostationary orbit optical imaging satellite. The GF-4 satellite has the advantages of wide-swath and high-frequency imaging, so it can provide quasi-real-time images for ship surveillance. In fact, using the GF-4 satellite image to detect moving ships is to detect the gray value of the wake formed by the ship moving at sea. However, because of the geometric features of the ship’s wake vary with the speed, the ship detection method based on geometric features is not applicable. Since, not only the ship is moving in the image but also the cloud is moving all the time, the optical flow method is not applicable. Ship detection methods based on machine learning and deep learning are also not applicable due to too few texture features of the wake. In this paper, we present a novel method for ship detection in thin-cloud environments. The real image processing of GF-4 satellite shows that the proposed method can detect multiple moving ships accurately and quickly in a thin-cloud environment.

Non-contact vital sign monitoring enables the estimation of vital signs, such as heart rate, respiratory rate and oxygen saturation (SpO2), by measuring subtle color changes on the skin surface using a video camera. For patients in a hospital ward, the main challenges in the development of continuous and robust non-contact monitoring techniques are the identification of time periods and the segmentation of skin regions of interest (ROIs) from which vital signs can be estimated. We propose a deep learning framework to tackle these challenges. Approach: This paper presents two convolutional neural network (CNN) models. The first network was designed for detecting the presence of a patient and segmenting the patient’s skin area. The second network combined the output from the first network with optical flow for identifying time periods of clinical intervention so that these periods can be excluded from the estimation of vital signs. Both networks were trained using video recordings from a clinical study involving 15 pre-term infants conducted in the high dependency area of the neonatal intensive care unit (NICU) of the John Radcliffe Hospital in Oxford, UK. Main results: Our proposed methods achieved an accuracy of 98.8% for patient detection, a mean intersection-over-union (IOU) score of 88.6% for skin segmentation and an accuracy of 94.5% for clinical intervention detection using two-fold cross validation. Our deep learning models produced accurate results and were robust to different skin tones, changes in light conditions, pose variations and different clinical interventions by medical staff and family visitors. Significance: Our approach allows cardio-respiratory signals to be continuously derived from the patient’s skin during which the patient is present and no clinical intervention is undertaken.

This paper proposes a non-contact vibration measurement method based on deep learning and image processing. The deep learning method is used to realize the automatic and efficient selection of effective pixels and the optical flow method is used to extract vibration signals to realize non-contact and targetless visual vibration measurement. In this study, a carbon plate board and aluminum C-beam structure were measured and verified under artificial and non-human excitation in a laboratory environment. Additionally, bridge and cable structures in an outdoor environment were selected as measurement targets to verify the reliability of the proposed method. This paper compares the experimental results of Canny and Sobel edge detection algorithms and deep learning methods to verify the efficiency of deep learning. The results demonstrate that our method is robust, even under real-world unfavorable conditions, meaning it can serve as a novel measurement method in the field of vibration measurement.

Super-Resolution (SR) of low-quality images is an important topic of research in image processing and computer vision field. Using multi-frame, super-resolution algorithm can reconstruct high-resolution images by incorporating the information of the subsequent images. Most of the super-resolution techniques for multi-frames either use a more traditional or mathematical approach or deep learning based approach with optical flow in consideration. In this paper, we develop a way to combine the optical flow enabled sub-pixel registration method for mapping into the high-resolution grid and a deep residual learning approach for restoring features with noise removal. The results exhibit a significant gain over the state of art methods and the bi-cubic interpolation method.

Video monitoring-based exception behavior detection of crowds and vehicles has become a hot research hotspot in image processing, machine vision and other related fields. In view of the difficulty of detecting abnormal targets in complex structured outdoor scenes, an anomaly detection method combining optical flow method and convolutional neural network (CNN) is proposed in this paper, the method can be used to detect and warn abnormal targets in complex structured scenes. Extract video motion characteristics by Lucas-Kanade (LK) optical flow, normalize the extracted optical flow through a simple scaling method, detect and alert the anomalies of video crowds and vehicles adopting CNN, evaluate the abnormal behavior detection method using the accuracy and time. The experiment results show the method can detect the abnormal behaviors of crowds and vehicles in complex scenes in time and effectively.

This paper presents a novel approach for segmenting moving objects in unconstrained environments using guided convolutional neural networks. This guiding process relies on foreground masks from independent algorithms (i.e. state-of-the-art algorithms) to implement an attention mechanism that incorporates the spatial location of foreground and background to compute their separated representations. Our approach initially extracts two kinds of features for each frame using colour and optical flow information. Such features are combined following a multiplicative scheme to benefit from their complementarity. These unified colour and motion features are later processed to obtain the separated foreground and background representations. Then, both independent representations are concatenated and decoded to perform foreground segmentation. Experiments conducted on the challenging DAVIS 2016 dataset demonstrate that our guided representations not only outperform non-guided, but also recent and top-performing video object segmentation algorithms.

To enable magnetic resonance imaging (MRI)-guided radiotherapy with real-time adaptation, motion must be quickly estimated with low latency. The motion estimate is used to adapt the radiation beam to the current anatomy, yielding a more conformal dose distribution. As the MR acquisition is the largest component of latency, deep learning (DL) may reduce the total latency by enabling much higher undersampling factors compared to conventional reconstruction and motion estimation methods. The benefit of DL on image reconstruction and motion estimation was investigated for obtaining accurate deformation vector fields (DVFs) with high temporal resolution and minimal latency. 2D cine MRI acquired at 1.5 T from 135 abdominal cancer patients were retrospectively included in this study. Undersampled radial golden angle acquisitions were retrospectively simulated. DVFs were computed using different combinations of conventional- and DL-based methods for image reconstruction and motion estimation, allowing a comparison of four approaches to achieve real-time motion estimation. The four approaches were evaluated based on the end-point-error and root-mean-square error compared to a ground-truth optical flow estimate on fully-sampled images, the structural similarity (SSIM) after registration and time necessary to acquire k-space, reconstruct an image and estimate motion. The lowest DVF error and highest SSIM were obtained using conventional methods up to [Formula: see text]. For undersampling factors [Formula: see text], the lowest DVF error and highest SSIM were obtained using conventional image reconstruction and DL-based motion estimation. We have found that, with this combination, accurate DVFs can be obtained up to [Formula: see text] with an average root-mean-square error up to 1 millimeter and an SSIM greater than 0.8 after registration, taking 60 milliseconds. High-quality 2D DVFs from highly undersampled k-space can be obtained with a high temporal resolution with conventional image reconstruction and a deep learning-based motion estimation approach for real-time adaptive MRI-guided radiotherapy.

The measurement of optical flow is an important problem in image processing. There are a number of methods available for optical flow estimation, including traditional variational methods, deep learning based supervised/unsupervised methods. In this work, we propose a deep convolutional network (CNN) based supervised coarse-to-fine approach, which is trained in end-to-end fashion. The proposed method is tested on standard optical flow benchmark datasets including Flying Chairs, MPI Sintel Clean and Final, KITTI. Experimental results show that the proposed framework is able to achieve comparable results to previous approaches with much smaller network architecture.