Search

In this paper, we propose a large-scale video based animal counting dataset collected by drones (AnimalDrone) for agriculture and wildlife protection. The dataset consists of two subsets, i.e. , PartA captured on site by drones and PartB collected from the Internet, with rich annotations of more than 4 million objects in 53, 644 frames and corresponding attributes in terms of density, altitude and view. Moreover, we develop a new graph regularized flow attention network (GFAN) to perform density map estimation in dense crowds of video clips with arbitrary crowd density, perspective, and flight altitude. Specifically, our GFAN method leverages optical flow to warp the multi-scale feature maps in sequential frames to exploit the temporal relations, and then combines the enhanced features to predict the density maps. Moreover, we introduce the multi-granularity loss function including pixel-wise density loss and region-wise count loss to enforce the network to concentrate on discriminative features for different scales of objects. Meanwhile, the graph regularizer is imposed on the density maps of multiple consecutive frames to maintain temporal coherency. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method, compared with several state-of-the-art counting algorithms. The AnimalDrone dataset is available at https://github.com/VisDrone/AnimalDrone .

Visual odometry (VO) is one of the essential techniques in mobile robots field; an accurate VO system is of great significance for mobile robot simultaneous localization and mapping. As for traditional monocular VO systems, they work by presuming the monocular scale is 1 (scale = 1), or relying on ground truth (GT) to estimate scale. As a result, the traditional monocular VO systems estimate the pose state with big drift or cannot work on the image sequence without GT. Although some classical monocular VO systems have been proposed, they still have imperfect performance or even unable to work in some extreme scene conditions, such as scene is monotony without obvious texture information or camera large-scale displacement motion. As for learning-based VO system, it is realized by training deep neural networks in supervised or self-supervised manner to end-to-end estimate the pose state; however, the accuracy of pose estimation entirely depends on the ability of networks. Although the ability of networks can be improved by increasing the number of training data sets and optimizing the network structure, it is inevitable to encounter problems such as insufficient generalization ability and insufficient accuracy on rotational pose estimation. In this article, a monocular VO system named DL_Hybrid is proposed, which takes full advantage of DL networks used in image processing and geometric localization theory based on hybrid pose estimation methods. The DL_Hybrid VO system can estimate a six-DoF pose one-frame-by-one-frame and recover camera trajectory, and it can extract accurate key points from per-frame even in extreme scene condition, and it has good performance even in the extreme moving condition, such as camera rotation-only action or static action, also it can work well in the condition of camera large-scale displacement motion. The real scale is also accurately estimated without depending on GT, and the pose estimation method is designed based on hybrid 2d–2d and 3d–2d localization theory to make the DL_Hyrid VO system to estimate translational and rotational information with accuracy and robustness. Experimental results show that the proposed DL_Hybrid VO system has a better performance than traditional and learning-based VO systems.

HighlightsAn ensemble method using color segmentation, deep learning, and image transformation was developed.Experiments were conducted to compare the method with other state-of-the-art tracking algorithms.The optimized ensemble method to track bolls achieved 94.4% accuracy using weakly trained tiny YOLOv2 models.The method achieved 7.6 frames per second and outperformed five other tracking methods.Abstract. In robotic applications, good perception can be computationally costly and create undesirable latency before a control decision is initiated. Most of the methods available for object detection deep learning are either fast with low accuracy or slow with high accuracy. Fast and accurate methods are necessary to track and localize objects such as cotton bolls that may be visible or occluded by each other or not well illuminated. In this study, an ensemble of a deep learning method and other image processing techniques was used to detect cotton bolls in-field on defoliated plants. In each image, a trained deep learning method, the YOLOv2 model, was used to detect open cotton bolls, and color segmentation was applied to confirm if the bolls detected by the YOLOv2 model were actually white to avoid false positives. Boll tracking was performed by following the spatial movement of good features on the edges of the bolls using the Lucas-Kanade algorithm. An image transformation algorithm was applied to the next image in case the previously detected boll was lost to retrieve the information of the missing boll. Each tracked and localized boll was stored and counted to give the total number of bolls detected. In this study, detection accuracy was sacrificed for image processing speed by using the YOLOv2 model. Detection accuracy was improved by using an ensemble method that combined image color segmentation, optical flow, and image transformation. This method was compared to eight other open-source methods implemented in OpenCV. The ensemble method detected and counted bolls at a speed of 7.6 fps with an accuracy of 94.4% using the Jetson TX2 embedded system to process 1K resolution images, outperforming the other OpenCV methods in various measurements. Keywords: Boll counting, Cotton, Cotton harvesting, DarkFlow, Darknet, Deep learning, Machine vision, YOLOv2.

One of the challenges of PV power generation is solar intermittency from weather conditions. Solar irradiance prediction is therefore required to deal with this issue. Several prediction methods have been proposed based on whole sky image processing techniques. This paper presents a combination technique of image processing with a convolution neural network (CNN) based on GoogLeNet for raising trigger events before the sun cover happens 1 to 2 min in advance. The captured sky images are preprocessed and in the next step, we use Hough transform to find the sun position and use optical flow to track cloud movement. Finally, we use a CNN to generate trigger events in advance before the sun occlusion happens. The results of prediction stage show error percentage as low as 5.26% in a clear sky day. Keywords: Ground base sky image, Photovoltaic power forecasting, Solar irradiance, GoogLeNet

In this paper, we propose SelfSphNet, that is, a self-supervised learning network to estimate the motion of an arbitrarily moving spherical camera without the need for any labeled training data. Recently, numerous learning-based methods for camera motion estimation have been proposed. However, most of these methods require an enormous amount of labeled training data, which is difficult to acquire experimentally. To solve this problem, our SelfSphNet employs two loss functions to estimate the frame-to-frame camera motion, thus giving two supervision signals to the network with the usage of unlabeled training data. First, a 5 DoF epipolar angular loss, which is composed of a dense optical flow of spherical images, estimates the 5 DoF motion between two image frames. This loss function utilizes a unique property of the spherical optical flow, which allows the rotational and translational components to be decoupled by using a derotation operation. This operation is derived from the fact that spherical images can be rotated to any orientation without any loss of information, hence making it possible to “decouple” the dense optical flow between pairs of spherical images to a pure translational state. Next, a photometric reprojection loss estimates the full 6 DoF motion using a depth map generated from the decoupled optical flow. This minimization strategy enables our network to be optimized without using any labeled training data. To confirm the effectiveness of our proposed approach (SelfSphNet), several experiments to estimate the camera trajectory, as well as the camera motion, were conducted in comparison to a previous self-supervised learning approach, SfMLearner, and a fully supervised learning approach whose baseline network is the same as SelfSphNet. Moreover, transfer learning in a new scene was also conducted to verify that our proposed method can optimize the network with newly collected unlabeled data.

Satellite images are primary data in weather prediction modeling. Deep learning-based approach, a viable candidate for automatic image processing, requires large sets of annotated data with diverse characteristics for training purposes. Accuracy of weather prediction improves with data having a relatively dense temporal resolution. We have employed interpolation and data augmentation techniques for enhancement of the temporal resolution and diversifications of characters in a given dataset. Algorithm requires classical approaches during preprocessing steps. Three optical flow methods using 14 different constraint optimization techniques and five error estimates are tested here. The artificially enriched data (optimal combination from the previous exercise) are used as a training set for a convolutional neural network to classify images in terms of storm or nonstorm. Several cyclone data (eight cyclone datasets of a different class) were used for training. A deep learning model is trained and tested with artificially densified and classified storm data for cyclone classification and locating the cyclone vortex giving minimum 90% and 84% accuracy, respectively. In the final step, we show that the linear regression method can be used for predicting the path.

This work presents two methods that facilitate a 3D reconstruction of microscopic blood vessels in the volume slightly larger than 1 mm3. The source of the data are histological serial sections, i.e., microscopic images of probes, stained with immunohistochemistry. Odd and even sections have different stainings in our primary data set. Thus, firstly, an approach to register an alternately-stained series is presented. With image filtering and a feature-detection-based registration we obtain a registered stack of 148 serial sections. The series has missing sections, locally damaged sections, artifacts from acquisition. All these hinder correct connectivity of blood vessels. With our second approach we interpolate the missing information while maintaining the connectivity. We achieve this with deformations based on dense optical flow. The presented methodology is applicable to further histological series. A combination of both approaches allows us to reconstruct more than 76% larger volumes. An important detail was the composition mode of images. Summarizing, we use methods from image processing and computer vision to create large-scale 3D models from immunostained histological serial sections.

This study presents an automated registration method based on optical flow for an adaptive optics scanning laser ophthalmoscope. The method was designed to align and average images to obtain a higher signal-to-noise ratio image. A correlation based optical flow image registration method, which has large registration degrees of freedom, is adopted as a local registration method. By comparing the images before and after image registration, we show the effectiveness of our method. Furthermore, the advantage of our method, which is the containment of large registration degrees of freedom, is confirmed.

Non-destructive ultrasonic analysis of materials is a method for assessing the integrity of the inspected components. It is commonly used in monitoring critical parts of the power plants, in aeronautics, oil and gas, and the automotive industry. Since most ultrasonic inspections rely on expert's previous experience they must constantly practice on new, unseen data. Acquiring enough data for training human experts on non- destructive ultrasonic scan analysis can be an expensive and time-consuming task. The only possibility to get new data for practicing is to implant synthetic defects in real metal blocks. Artificial defects are made by temperature strain, electrical discharge, and physical damage. All of those methods are very complicated and expensive to perform. Also metal blocks have to be taken from the components of the power plants to have the same structure and be realistic. In this work, some attempts have been made to generate 3D ultrasonic scans using computer vision and deep learning methods.

Simple Summary Animal behaviors are critical for survival, which is expressed over a long period of time. The emergence of computer vision and deep learning technologies creates new possibilities for understanding the biological basis of these behaviors and accurately quantifying behaviors, which contributes to attaining high production efficiency and precise management in precision farming. Here, we demonstrate that a dual-stream 3D convolutional neural network with RGB and optical flow video clips as input can be used to classify behavior states of fish schools. The FlowNet2 based on deep learning, combined with a 3D convolutional neural network, was first applied to identify fish behavior. Additionally, the results indicate that the proposed non-invasive recognition method can quickly, accurately, and automatically identify fish behaviors across hundreds of hours of video. Abstract The rapid and precise recognition of fish behavior is critical in perceiving health and welfare by allowing farmers to make informed management decisions on recirculating aquaculture systems while reducing labor. The conventional recognition methods are to obtain movement information by implanting sensors on the skin or in the body of the fish, which can affect the normal behavior and welfare of the fish. We present a novel nondestructive method with spatiotemporal and motion information based on deep learning for real-time recognition of fish schools’ behavior. In this work, a dual-stream 3D convolutional neural network (DSC3D) was proposed for the recognition of five behavior states of fish schools, including feeding, hypoxia, hypothermia, frightening and normal behavior. This DSC3D combines spatiotemporal features and motion features by using FlowNet2 and 3D convolutional neural networks and shows significant results suitable for industrial applications in automatic monitoring of fish behavior, with an average accuracy rate of 95.79%. The model evaluation results on the test dataset further demonstrated that our proposed method could be used as an effective tool for the intelligent perception of fish health status.

The active movement of children poses a safety risk in the absence of adult supervision. To reduce the risk of accidents in children, an automatic detection system for the direction of children's movements is crucially needed. In this study, detection of the direction of human movement based on image processing was carried out with the input of videos produce from 4 CCTV installed in each corner of the room. The system will detect the direction of object movement with classification of orientation, namely front, back, right and left. The detection method used in this research is Optical Flow. Optical Flow will calculate the value of the direction or orientation of the movement of an object. The orientation obtained is then accumulated with HOOF (Histogram Orientation of Optical Flow), where HOOF will collect the orientation of objects on the whole frame according to a 8-part Cartesian angle. The results of the orientation with Optical Flow will be compared with the direction of detection measured manually to determine whether the detection of movement direction using Optical Flow is running well. According to the results, it is known that the Optical Flow method has succeeded in detecting the direction of movement accurately based on different camera angles. Keywords : Image Processing, CCTV, Optical Flow, HOOF

In the resilient and reliable electrical power system, the condition of high voltage insulation plays a crucial role. In the field of high voltage insulation integrity, the partial discharge (PD) inception and development trends are essential for assessment criteria in diagnostics systems. The observed trend to employ more and more sophisticated algorithms with machine learning features and artificial intelligence (AI) elements is observed everywhere. The classification and identification of features in PD images is perceived as a critical requirement for an effective high voltage insulation diagnosis. In this context, techniques allowing for anomaly detection, trends observation, and feature extraction in partial discharge patterns are important. In this paper, the application of few algorithms belonging to image processing, machine learning and optical flow is presented. The feature extraction refers to image segmentation and detection of coherent forms in the images. The anomaly detection algorithms can trigger early detection of the trend changes or the appearance of a new discharge form, and hence are suitable for PD monitoring applications. Anomaly detection can also handle transients and disturbances that appear in the PD image as an indication of an abnormal state. The future monitoring systems should be equipped with trend evolution algorithms. In this context, two examples of insulation aging and application of PD-based monitoring are shown. The first one refers to deep convolutional neural networks used for classification of deterioration stages in high voltage insulation. The latter one demonstrates application of optical flow approach for motion detection in partial discharge images. The motivation for the research was the strive to machine-controlled pattern analysis, leading towards intelligent PD-based diagnostics.

Besides its importance for greenhouse emission reduction, the remote detection, localization and quantification of gas leaks in industrial facilities remains a challenging problem in industry and research. In that sense, the development of new data processing techniques that allow deriving new and/or more accurate information about the gas leaks from made measurements has gained more attention in the recent years. This becomes apparent from the increased use of optical gas imaging (OGI) cameras (specialised mid-wave infrared cameras e.g. for methane and carbon dioxide) along with image processing and computer vision techniques, to tackle these challenges. In this work, deep-learning-based optical flow methods are evaluated for determining gas velocities from gas images of an OGI camera. For this, a dataset of simulated and real gas images under controlled and real conditions is used for supervised training and validation of two different state of the art CNNs for optical flow computation: FlowNetC, FlowNet2 and PWC-Net. Classical optical flow methods based on variational methods are also considered and the differences in performance and accuracy between classical and deep-learning-based methods are shown. In addition, FlowNet2 is further improved for working with gas images by fine tuning the network weights. This approach has demonstrated to make FlowNet2 more reliable and less sensitive to image noise and jitter in the experiments. For further validation, a set of real gas images acquired in a wind channel and one from a biogas plant with reference mean gas velocities from a 3D anemometer are being used. The results show that the fine-tuned version of FlowNet2 (FNet2-G) allow computing larger optical flow magnitudes than classical optical flow methods while being less sensitive to image noise under field conditions. The obtained results also show the potential of deep-learning-based approaches for image processing tasks such as gas segmentation, disparity computation and scene flow in stereo gas images.

In this paper, we concentrate on dense estimation of disparities between fish-eye images without corrections. Because of the distortions, fish-eye images cannot be processed directly utilizing the classical adaptive support weight (ASW) method for perspective images. To address this problem, we propose a modified hemispherical ASW method in a hemispherical framework. First, 3D epipolar curves are calculated directly on a hemispherical model to deal with the problem that 2D epipolar curves cannot cover the whole image disc. Then, a modified ASW method with hemispherical support window and hemispherical geodesic distance is presented. Moreover, a three-dimensional epipolar distance transform (3DEDT) is proposed and fused into the matching cost to cope with the textureless region problem. The benefit of this approach is demonstrated by realizing the dense stereo matching for fish-eye images using a public fish-eye data set, for which both objectively evaluated as well as visually convincing results are provided.

The interaction technology in general is inseparable from the rapid development of Human-Computer Interaction technology or Human and Computer Interaction. Human and computer mouse interactions are called Virtual Mouse. Virtual mouse is designed for users to be able to interact directly with computers without using input devices such as conventional mice but using their hands as driving objects or using other media such as colors. In this research, testing is done on how to track an object that moves in order to do the mouse by using an intermediary in the form of an image processing-based webcam video that is taken in real-time using object tracking in the form of three color variables arranged based on RGB composition using the optical flow method in detecting its movement. Based on research that has been carried out as a whole the color object used as a pointer controller or mouse with the optical flow method can be detected properly in every condition.

Recent advances in computation power have allowed computed tomography (CT) to utilise iterative reconstruction (IR) algorithms. The IR technique can handle noisy data and reconstructs optimal CT images from limited projected images. As of cyclic image processing, IR improves the quality of CT images. This approach requires a minimum number of projections to reconstruct an image; however, decreasing the number of projections to 90 can create artefacts and degrade reconstruction quality. To overcome this limitation, the optical flow technique can compute flow vectors between two consecutive projections to generate projected images between frames. Here, optical flow-based frame interpolation combined with the ordered subset-modified iterative technique is proposed to reduce computation time, lower the number of projections, and increase reconstruction quality of CT images. The proposed technique can be used to reconstruct a CT image from 90 projections at 4 degree intervals between projection sequences. This approach produces a much better quality reconstruction compared to that produced by an analytical algorithm, which uses 360 projections. The inclusion of an ordered subset reconstructs CT images quickly by accelerating streaming architecture.

Automatic surveillance of abnormal events is a major unsolved problem in city management. By successful implementation of automatic surveillance of abnormal events, a significant amount of human resources in video monitoring can be economized. One solution to this application is computer vision technology. This approach utilizes an image processing algorithm to extract specific features and then uses discriminator algorithms to give an alert. In this paper, we propose to apply a particle filter-based algorithm to feature series extracted from videos in order to give alerts when abnormal events occur. The whole process consists of feature series generation and particle filter tracking. To represent the features of a video, an L2-norm extractor is designed based on the optical flow. Then, the particle filter keeps track of these feature series. The occurrence of abnormal events will cause the shift of feature series and a large error in PF tracking. This, in turn, will allow computers to understand and define the occurrences of anomalies. Experiments on UMN dataset show that our algorithm reaches 90% accuracy in frame-level detection.

International audience; Image sequences allow visualizing dynamic systems and understanding their intrinsic characteristics. One first component of this dynamics is retrieved from the estimation of the velocity displayed on the sequence. Motion estimation has been extensively studied in the literature of image processing and computer vision. In this paper, we step beyond the traditional optical flow methods and address the problem of recovering the acceleration from the whole temporal sequence. This issue has been poorly investigated, even if this is of major importance for major data types, such as fluid flow images. Acceleration is defined as the space-time function resulting from the forces applied to the studied system. To estimate its value, we propose a variational approach where an energy function is designed to model both the motion and the acceleration fields. The contributions of the paper are twofold: first, we introduce a unified variational formulation of motion and acceleration under space-time constraints; second, we describe the minimization scheme, which allows retrieving the estimations, and provide the full information on the discretization schemes. Last, experiments illustrate the potentiality of the method on synthetic and real image sequences, visualizing fluid-like flows, where direct and precise calculation of acceleration is of primary importance.

In this paper, a deep learning-based traffic state discrimination method is proposed to detect traffic congestion at urban intersections. The detection algorithm includes two parts, global speed detection and a traffic state discrimination algorithm. Firstly, the region of interest (ROI) is selected as the road intersection from the input image of the You Only Look Once (YOLO) v3 object detection algorithm for vehicle target detection. The Lucas-Kanade (LK) optical flow method is employed to calculate the vehicle speed. Then, the corresponding intersection state can be obtained based on the vehicle speed and the discrimination algorithm. The detection of the vehicle takes the position information obtained by YOLOv3 as the input of the LK optical flow algorithm and forms an optical flow vector to complete the vehicle speed detection. Experimental results show that the detection algorithm can detect the vehicle speed and traffic state discrimination method can judge the traffic state accurately, which has a strong anti-interference ability and meets the practical application requirements.

Optical Flow Estimation is an essential component for many image processing techniques. This field of research in computer vision has seen an amazing development in recent years. In particular, the introduction of Convolutional Neural Networks for optical flow estimation has shifted the paradigm of research from the classical traditional approach to deep learning side. At present, state of the art techniques for optical flow are based on convolutional neural networks and almost all top performing methods incorporate deep learning architectures in their schemes. This paper presents a brief analysis of optical flow estimation techniques and highlights most recent developments in this field. A comparison of the majority of pertinent traditional and deep learning methodologies has been undertaken resulting the detailed establishment of the respective advantages and disadvantages of the traditional and deep learning categories. An insight is provided into the significant factors that affect the success or failure of the two classes of optical flow estimation. In establishing the foremost existing and inherent challenges with traditional and deep learning schemes, probable solutions have been proposed indeed.

International audience; Event-Based cameras (also known as Dynamic Vision Sensors "DVS") have been used extensively in robotics during the last ten years and have proved the ability to solve many problems encountered in this domain. Their technology is very different from conventional cameras which requires rethinking the existing paradigms and reviewing all the classical image processing and computer vision algorithms. We show in this paper how Event-Based cameras are naturally adapted to estimate on the fly scene gradients and hence the visual flow. Our work starts with a complete study of existing event-based optical flow algorithms that are suitable to be integrated into real-time robotics applications. Then, we provide a data-set that includes different scenarios along with a set of visual flow ground-truth. Finally, we propose an evaluation of existing event-based visual flow algorithms using the proposed ground truth data-set.

The objective is to detect landslide and report it as soon as it is detected so that appropriate measures can be taken in time in order to reduce the loss of life and infrastructure and to issue advisories to the public. A camera surveillance system with an image processing algorithm for 24/7 monitoring of flow is proposed to detect landslides. The warning system (up to the issuance of a Common Alerting Protocol alert) is also developed. We develop an algorithm that processes the camera feed and accounts for the factors like frames per second (FPS), structural similarity, resolution of the camera and optical flow in order to detect the occurrence of a landslide. Using a network of such cameras and communicating over the network results in a distributed intelligent system. We also estimate the deterioration caused by the disaster from the output image to estimate the extent of the damage incurred.

In this chapter, we propose an optimization of the a-contrario clustering method using the probabilistic Guillaume Khenchaff Measure (MGK) quality technique. A-contrario is used for tracking salient objects in the scene in real time. This method analyzes the data contained in a motion vector, which contains the scattered optical flow accumulated points of interest. The aim of our study is to improve the first results obtained from the Number of False Alarm (NFA) criterion by using MGK to bring together the group of points endowed with a coherent movement of the binary tree. The idea is to isolate dynamic points so that we can use static points in the future.

Objects counting in video frames is an important field of research that provides appropriate information about objects and their groups in video streams. An adequate technique to achieve this goal is using digital image/video processing methods with a detection method called CenterN et that is based on a deep learning architecture (i.e., a ConvNet architecture). This paper presents an objects counting method based on a combination of a detection model and optical flow generated between video frames.

Image/video processing has been one of the major developments in the recent history with its applications in areas of Road safety, military, medical and agriculture fields. Due to its complexity a generic solution for multiple object detection in extremely crowded scenes remains to be found. Traditional methods of optical flow, connected component analysis and image segmentation have been extensively studied in image processing and video processing material. With recent developments of machine learning and numerical optimization techniques the use of deep neural networks are getting frequent in image processing applications. Among such deep learningbased methods commonly used in this context are RCNN variants, Mask RCNN and YOLOv3. An exhaustive comparison of the traditional methods and deep learning-based methods and also deep learning methods are discussed in this paper. This study will be of use in selection of a method for any extremely crowded scene object detection problem.

In recent years, with the development of relevant technologies in the field of machine vision, the processing of visual image information has become the focus of research. Among them, the detection of moving targets is a very important research direction in the field of machine vision, which lays a foundation for the recognition of moving targets and tracking of moving targets. The task of moving target detection is to identify the physical movement of the target in a specific area. In this paper, the relevant image processing techniques used in the process of moving target detection are briefly described, including image preprocessing, image segmentation, feature extraction and so on. Then it describes the algorithms commonly used for moving target detection in recent years, including background difference method, inter-frame method, optical flow method, and compares the advantages and limitations of these methods. In view of the shortcomings of these methods, it summarizes the previous solutions. Finally, the improvement of these algorithms in recent years is pointed out.

Detection of moving objects in sequences of images is an important research field, with applications for surveillance, tracking and object recognition among others. An algorithm to estimate motion in video image sequences, with moving object distinction and differentiation, is proposed. The motion estimation is based in three consecutive RGB image frames, which are converted to gray scale and filtered, before being used to calculate optical flow, applying Gunnar Farneback’s method. The areas of higher optical flow are maintained and the areas of lower optical flow are discarded using Otsu’s adaptive threshold method. To distinguish between different moving objects, a border following method was applied to calculate each object’s contour. The method was successful detecting and distinguishing moving objects in different types of image datasets, including datasets obtained from moving cameras. This extended version contemplates more results obtained, using the demonstrated methodology, with other datasets.

Nowadays the use of visual navigation in orchard is rapidly growing, while current research mainly focuses on orchard row detection or obstacle detection separately. The economic, fast and reliable navigation vision system that completing orchard row following and obstacle avoidance simultaneously and the obstacle avoidance algorithm for vision system without complex multiple navigation systems combining are needed further research. In this paper, a trinocular vision system for orchard vehicle is set up and the obstacle avoidance algorithm based on coupling of geometric constraint and virtual force field is designed for its vision system. First of all, based on analyzing the different vision system characteristics and the detection demand of orchard obstacle avoidance the trinocular vision system for orchard vehicle is set up based on a wide-angle camera and binocular stereo vision system. Then, the image processing algorithm of orchard row and obstacle detection is designed. For orchard row detection, the trunk regions enhancement algorithm is designed based on grayscale morphology filtering the trunk regions prediction algorithm is designed based on optical flow method to improve the trunk regions detection speed and accuracy. For obstacle detection, the background equalization algorithm based on H-channel image characteristics and the interference weaken algorithm with G-channel image characteristics are designed. Based on the row and obstacle detection results the algorithm for obstacle avoidance in orchard based on coupling of geometric constraint and virtual force field is designed. In the obstacle avoidance process, the virtual force field method is used as the mainly control of the vehicle obstacle avoidance process, and the geometric constraint between visual model of trinocular vision system and obstacle position is coupling with it to fulfill the avoidance demand of the obstacle with special shape and other condition that influence the virtual force field calculation. The experimental results show that the trinocular vision system for orchard vehicle built in this paper has strong adaptability to the actual environment which is accurate and stable for orchard row detection and obstacle detection simultaneously. The average deviation is 4.76 cm and 7.05 cm at 0.5 m/s and 1.0 m/s respectively, the average deviation of obstacle distance detection in the Z-axis direction is 3.18 cm, and in the X-axis direction is 0.45 cm. And the obstacle avoidance algorithm designed in this paper is effective in the orchard which can meet the actual production requirements. The research results will lay a foundation for the development of intelligent equipment and unmanned management in orchard.

An early fire detection in indoor environment is essential for people’s safety. During the past few years, many approaches using image processing and computer vision techniques were proposed. However, it is still a challenging task for application of video smoke detection in indoor environment, because the limitations of data for training and lack of efficient algorithms. The purpose of this paper is to present a new smoke detection method by using surveillance cameras. The proposed method is composed of two stages. In the first stage, motion regions between consecutive frames are located by using optical flow. In the second stage, a deep convolutional neural network is used to detect smoke in motion regions. Besides, to overcome the problem of lacking data, simulated smoke images are used to enrich the dataset. The proposed method is tested on our data set and real video sequences. Experiments show that the new method is successfully applied to various indoor smoke videos and significant for improving the accuracy of fire smoke detection. Source code and the dataset have been made available online.

This paper proposes a novel image processing mechanism for augmented reality based autonomous navigation system. Autonomous vehicles navigate independently without any sort of human assistance. In order to navigate from one location to another, these systems require assistance in the form of markers i.e., bookmarks or reference points which are pre-determined and already stored in a database. This approach makes it difficult for the vehicle to navigate in an unknown and uncertain environment. Therefore, the proposed system will navigate on its own without any pre-trained dataset. Three types of markers are detected and stored in the intelligent repository by eliminating step by step triviality. Optical flow and image processing techniques have been proposed to detect markers from the real time video frames. This approach is divided into three parts: (i) Marker detection and categorization; (ii) Marker Identification and (iii) Obstacle detection. Obstacles are also detected during run time without any training of dataset. This will result in minimizing the gap between human and machine thinking skills.

Traditional image global registration algorithms are limited in principle and cannot accurately register large depth of field scenes or moving objects. The local registration method based on dense optical flow has the advantage of not being limited by a single transformation matrix, so that a better registration result can be obtained. However, traditional dense optical flow algorithms are limited by large computational complexity and are difficult to achieve real-time estimation, thus limiting their application. In recent years, many dense optical flow algorithms based on deep learning (such as PWC-Net) have emerged, which have achieved the effect of surpassing traditional optical flow algorithms on public datasets and can be estimated in real time. Based on this, this paper proposes an algorithm flow based on deep learning to predict dense optical flow and use it for registration. And a self-built optical flow data set for supervised learning of the network has also been proposed. Using the same network, the registration results of our datasets are better than those of existing datasets.

Appearance defect inspection is crucial for quality control in the context of Industry 4.0. This research introduces a joint surface defect inspection and classification framework for polyvinyl chloride (PVC) pipe based on the low-cost visual sensors and high-efficiency computer vision algorithms. First, we build a robust imaging system to acquire the surface of PVC (S-PVC) by considering its characteristics and the illumination condition into the modeling process. Second, we adopt the region of interest method to eliminate the background interference captured in the S-PVC imaging and design an efficient S-PVC defect inspection and classification method. Third, we build an automatic machine prototype to evaluate the efficiency of the proposed method. Experimental results demonstrate that our framework has the advantages of low latency, high precision, and robustness.

Internalization of nanoparticles into intracellular area includes key information in biomedical field, such as cell signal pathway and drug delivery. Although the tracking of the individual nanoparticles in the cytoplasmic area has revealed the movement of the target in terms of single-particle level, the whole cell-level study is fundamental in order to efficiently acquire a large dataset of intracellular transport. In the present study, visualization and data analysis methods for understanding the entire cell-level intracellular transport in a living cell is suggested, by applying computer vision techniques to the cell images collected on the camera image sensor. Using the changes in the optical flow of the quantum dot-labeled vesicles for the entire intracellular area, our method showed the possibility of the time series analysis of vesicle movement related to the transport by two different types of molecular motors, dynein and kinesin.

At present, the GF-4 satellite is the world’s highest resolution geostationary orbit optical imaging satellite. The GF-4 satellite has the advantages of wide-swath and high-frequency imaging, so it can provide quasi-real-time images for ship surveillance. In fact, using the GF-4 satellite image to detect moving ships is to detect the gray value of the wake formed by the ship moving at sea. However, because of the geometric features of the ship’s wake vary with the speed, the ship detection method based on geometric features is not applicable. Since, not only the ship is moving in the image but also the cloud is moving all the time, the optical flow method is not applicable. Ship detection methods based on machine learning and deep learning are also not applicable due to too few texture features of the wake. In this paper, we present a novel method for ship detection in thin-cloud environments. The real image processing of GF-4 satellite shows that the proposed method can detect multiple moving ships accurately and quickly in a thin-cloud environment.

Accurate and fast visual localization is required in many applications of mini-robotics. Obtaining the best possible result for a given platform requires a balanced combination of camera settings and efficient image processing of the acquired image. In this paper, we study the trade-off between a high-resolution and a high-speed acquisition mode of a conventional camera for an experimental platform of magnetically-propelled mini-robots. Specifically, we propose a two stage localization algorithm based on fast pre-location using block matching followed by optical flow correction for subpixel accuracy of localization. In the experimental evaluation, we show that the difference in localization using two images of the same scene in resolutions 1000 × 1000px and 200 × 200px is only 0.2px of the higher resolution. The computational cost of the lower resolution is 16times lower than that of the higher resolution. This allows greater accuracy localization at a higher frame rate, which significantly improves the dynamics of control. Experimental results demonstrate the precision and speed of the proposed algorithm in the task of tracking a magnetically propelled robot on the platform.

International audience; The fall is one of the major problems that threaten the health of the elderly. According to world statistics, between 28 and 35% of seniors aged over 65 suffer from at least one fall per year. Continuous monitoring and rapid detection of critical events such as falls allows for rapid response and minimizes impacts. For this, several fall detection devices have been designed by the researchers. This paper proposes a fall detection device for elderly people at home using a portable camera worn on the hips to preserve privacy. The method of fall detection that we propose uses two image processing tools: The oriented gradient histogram (HOG) and the optical flow. The results of tests being carried on 14 subjects show that falls can be detected from standing, sitting or lying with a general sensitivity of 95%, from a set of data resulting from 20 tests, performed by each of the volunteers, for each of these three scenarios as well as for activities of daily living; the HOG-based method allowed the detection of falls but introduced many false detections which led to a specificity of 46.66%. The introduction of the optical flow has improved the specificity by reducing it to 68.33%. The system has been shown to be effective for rotations with a specificity increased by 50% over the use of HOG only. Also, the specificity has slightly increased for the events: sit down and lie down. However, this increase is accompanied by a decrease in sensitivity, since some falls are not detected by the optical flow, as the case of falls from an ‘elongated’ position.

Gait recognition is gaining popularity as it can recognize people in a non-intrusive and a non-contact manner. However, gait recognition is known for its susceptibility to clothing conditions. In this paper, we propose a solution specific to clothing conditions in the Gulf region where Abaya and Kandura are considered traditional clothing. The paper proposes a solution capable of training users based on traditional clothing and recognizing them in Western style clothing and vice-a-versa. The solution uses depth imaging, optical flow, accumulated motion and Discrete Cosine Transformation (DCT). Motion is calculated from consecutive images where the magnitudes and phases of motion vectors are accumulated into separate matrices. DCT and zonal coding is then applied to these matrices to form one concise feature vector that represents a walk. Experimental results, with 38 participants, showed that the proposed method is suitable for gait recognizing with such clothing constraints. The average classification accuracy is 88%. In comparison to an existing method, it is shown that the proposed method results in much more accurate recognition results yet at a higher computational cost.

Non-contact vital sign monitoring enables the estimation of vital signs, such as heart rate, respiratory rate and oxygen saturation (SpO2), by measuring subtle color changes on the skin surface using a video camera. For patients in a hospital ward, the main challenges in the development of continuous and robust non-contact monitoring techniques are the identification of time periods and the segmentation of skin regions of interest (ROIs) from which vital signs can be estimated. We propose a deep learning framework to tackle these challenges. Approach: This paper presents two convolutional neural network (CNN) models. The first network was designed for detecting the presence of a patient and segmenting the patient’s skin area. The second network combined the output from the first network with optical flow for identifying time periods of clinical intervention so that these periods can be excluded from the estimation of vital signs. Both networks were trained using video recordings from a clinical study involving 15 pre-term infants conducted in the high dependency area of the neonatal intensive care unit (NICU) of the John Radcliffe Hospital in Oxford, UK. Main results: Our proposed methods achieved an accuracy of 98.8% for patient detection, a mean intersection-over-union (IOU) score of 88.6% for skin segmentation and an accuracy of 94.5% for clinical intervention detection using two-fold cross validation. Our deep learning models produced accurate results and were robust to different skin tones, changes in light conditions, pose variations and different clinical interventions by medical staff and family visitors. Significance: Our approach allows cardio-respiratory signals to be continuously derived from the patient’s skin during which the patient is present and no clinical intervention is undertaken.

This paper proposes a method for detecting possible collisions using a monocular camera. Robust collision detection from a monocular video stream is a challenging problem with applications ranging from robotics, human navigation and automation. Our target application is an assistive system for aiding visually impaired people to avoid collision to obstacles while walking. The proposed method is based on a new algorithm to estimate the time to collision (TTC) (from an obstacle to the camera plane) using: i) optical flow to estimate motion, ii) the pinhole camera model to get a TTC analytical equation for each pixel and iii) image processing techniques to reduce possible errors. We evaluate the method in three different experiments reflecting different scenarios, allowing a analysis of its performance.

It is difficult to apply most video temporal super-resolution studies for real-world scenes because they are optimized for a specific range of characteristics. In this paper, we propose a video temporal super-resolution method that is tolerant to motion diversity and noise. Our proposed method improves its robustness by fine-tuning the pre-trained SPyNet that is trained for videos with simple motions and moderate conditions. Moreover, our proposed network learns to accurately synthesize two frames generated by a backward warping function without requiring any additional information using the architecture of a modified DHDN. This enables our proposed method to efficiently synthesize two warped frames by saving the computational complexity for pre-training and extracting the additional information. Finally, we apply the self-ensemble method, which is commonly used in studies on image processing but not on video processing. The application of the self-ensemble method enables our network to generate stable output frames with improved quality without any additional training. Our proposed network proved its performance by ranking 5th in the AIM 2019 video temporal super-resolution challenge; the performance gap between our proposed network and the 3rd-and 4th-ranked solutions was very small. The source code and pre-trained models are available at https://github.com/BumjunPark/DVTSR.

This paper presents an approach for real-time detection of three types of eye blinks: eye blink (blinking both eyes simultaneously), left and right winks. The process of blink detection has been divided into four parts viz. face localization in facial images acquired through a video camera, eye pair localization, pixels’ motion analysis using optical flow technique, and classification of eye blinks. Blink detection has been performed using a video camera and MATLAB software with image processing and computer vision toolbox. The algorithm takes about 60 ms time for processing a frame and 250 ms time for confirmation and classification of the detected blink. An experiment was conducted to evaluate the performance of the proposed approach in which 10 users voluntarily participated. The performance of the proposed method has been tested under two lighting conditions: natural lighting conditions and controlled lighting conditions. Also, the performance has been tested by varying the distance of the user from the camera. Here, it is observed that the system gives best performance when used under controlled lighting conditions and the user sitting at a distance of about 0.5 m. Accuracy of the proposed approach has been found to be 96, 92 and 88% for detection of eye blink, left wink and right wink, respectively. The proposed method has also been tested on ZJU dataset where it has given precision, detection accuracy and false alarm rate of values 94.11, 91.2 and 1.54%, respectively. The proposed system has been used and evaluated for performing various mouse analogous functions using eye blinks and winks. It has given an accuracy of 90, 80 and 90% in performing left click, double click, and right click operations, respectively.

Space objects and stars appear similar in images acquired by the wide field of view (FOV) survey telescope. This work investigates a unique property of the telescope observing a space object in satellite tracking mode, namely that the azimuth and altitude angles of the object and those of the optical axis of the telescope vary, in theory, in the same way. Based on this property we derive that the movement distance of the object between the two adjacent frames is minimal compared to the distance of the star. With this conclusion, it is possible to detect the object from a large number of background stars. To improve the robustness of the detection, the set of candidate objects is created. Finally, a clustering algorithm is employed to successfully extract the motion trajectory of the object. Unlike traditional detection methods or techniques based on image processing and analysis, our proposed detection is closely related to the parameters of the trajectory-following performance, which provides a more reliable basis for improving the detection rate. The feasibility and accuracy of the algorithm was verified by the 1.2-meter wide FOV survey telescope at the Jilin base of the Changchun observatory, with a detection rate of over 98%. The test results indicate that the method can satisfy the demand for detecting the object in an open-loop tracking. If the detection method is implemented in hardware, it can detect the object in a closed-loop tracking. As a result, it will have a wider scope for applications.

This paper proposes a non-contact vibration measurement method based on deep learning and image processing. The deep learning method is used to realize the automatic and efficient selection of effective pixels and the optical flow method is used to extract vibration signals to realize non-contact and targetless visual vibration measurement. In this study, a carbon plate board and aluminum C-beam structure were measured and verified under artificial and non-human excitation in a laboratory environment. Additionally, bridge and cable structures in an outdoor environment were selected as measurement targets to verify the reliability of the proposed method. This paper compares the experimental results of Canny and Sobel edge detection algorithms and deep learning methods to verify the efficiency of deep learning. The results demonstrate that our method is robust, even under real-world unfavorable conditions, meaning it can serve as a novel measurement method in the field of vibration measurement.

A fully automated 2D+time myocardial segmentation framework is proposed for Cardiac Magnetic Resonance (CMR) Blood-Oxygen-Level-Dependent (BOLD) datasets. Ischemia detection with CINE BOLD CMR relies on spatio-temporal patterns in myocardial intensity but these patterns also trouble supervised segmentation methods, the de-facto standard for myocardial segmentation in cine MRI. Segmentation errors severely undermine the accurate extraction of these patterns. In this paper we build a joint motion and appearance method that relies on dictionary learning to find a suitable subspace. Our method is based on variational pre-processing and spatial regularization using Markov Random Fields (MRF), to further improve performance. The superiority of the proposed segmentation technique is demonstrated on a dataset containing cardiac phase-resolved BOLD (CP-BOLD) MR and standard CINE MR image sequences acquired in baseline and ischemic condition across 10 canine subjects. Our unsupervised approach outperforms even supervised state-of-the-art segmentation techniques by at least 10% when using Dice to measure accuracy on BOLD data and performs at-par for standard CINE MR. Furthermore, a novel segmental analysis method attuned for BOLD time-series is utilized to demonstrate the effectiveness of the proposed method in preserving key BOLD patterns.

One of the most important features of the proper operation of technical objects is monitoring the vibrations of their mechanical components. The currently significant proportion of the research methods in this regard includes a group of research methods based on the conversion of vibrations using sensors providing data from individual locations. In parallel with the continuous improvement of these tools, new methods for acquiring information on the condition of the object have emerged due to the rapid development of visual systems. Their actual effectiveness determined the switch from research laboratories to actual industrial installations. In many cases, the application of the visualization methods can supplement the conventional methods applied and, under particular conditions, can effectively replace them. The decisive factor is their non-contact nature and the possibility for simultaneous observation of multiple points of the selected area. Visual motion magnification (MM) is an image processing method that involves the conscious and deliberate deformation of input images to the form that enables the visual observation of vibration processes which are not visible in their natural form. The first part of the article refers to the basic terms in the field of expressing motion in an image (based on the Lagrangian and Eulerian approaches), the formulation of the term of optical flow (OF), and the interpretation of an image in time and space. The following part of the article reviews the main processing algorithms in the aspect of computational complexity and visual quality and their modification for applications under specific conditions. The comparison of the MM methods presented in the paper and recommendations for their applications across a wide variety of fields were supported with examples originating from recent publications. The effectiveness of visual methods based on motion magnification in machine diagnosis and the identification of malfunctions are illustrated with selected examples of the implementation derived from authors’ workshop practice under industrial conditions.