Search

This research proposes a sensor for tracking the motion of a human head via optical tactile sensing. It implements the use of a fibrescope a non-metal alternative to a webcam. Previous works have included robotics grippers to mimic the sensory features of human skin, that used monochrome cameras and depth cameras. Tactile sensing has shown advantages in feedback-based interactions between robots and their environment. The methodology in this paper is utilised to track motion of objects in physical contact with these sensors to replace external camera based motion capture systems. Our immediate application is related to detection of human head motion during radiotherapy procedures. The motion was analysed in two degrees of freedom, respective to the tactile sensor (translational in z-axis, and rotational around y-axis), to produce repeatable and accurate results. The movements were stimulated by a robot arm, which also provided ground truth values from its end- effector. The fibrescope was implemented to ensure the device's compatibility with electromagnetic waves. The cameras and the ground truth values were time synchronised using robotics operating systems tools. Image processing methods were compared between grayscale and binary image sequences, followed by motion tracking estimation using deterministic approaches. These included Lukas-Kanade Optical Flow and Simple Blob Detection, by OpenCV. The results showed that the grayscale image processing along with the Lukas- Kanade algorithm for motion tracking can produce better tracking abilities, although further exploration to improve the accuracy is still required. [ABSTRACT FROM AUTHOR]

The degraded resolution and sensitivity characteristics of background-oriented schlieren (BOS) can be recovered by utilizing an optical flow (OF)-based image processing scheme. However, the background patterns conventionally employed in BOS setups suit the needs of the cross-correlation approach, whereas OF is based on a completely different mathematical background. Thus, in order to characterize the resolution and sensitivity response of OF-based BOS to the background generation configurations, a parametric study is performed. First, a synthetic assessment based on an analytical solution of a one-dimensional shock tube problem is conducted. Then, a numerical assessment utilizing direct numerical simulation data of density-driven turbulence is performed. Finally, the applicability of the documented conclusions in realistic scenarios is tested through an experimental assessment over a plume of a swirling heated jet. [ABSTRACT FROM AUTHOR]

The interest of science community towards planet Mars has progressively increased in past decades. Rovers are already doing an excellent job in exploring the Martian surface environment and showed that the Red Planet is not suited for life. However, there is a possibility of enabling human exploration by having astronauts take shelter in habitats located within lava tubes, which can provide protection against radiation and dust storms. These tubes are extremely hard to explore with rovers, thus a flying vehicle will be more suitable for such tasks. INGENUITY (Martian helicopter) had already demonstrated that it is possible to fly on Mars by doing small and simple manoeuvres. Based on NASA's design, this article will introduce the development of guidance, navigation, and control operations of a Martian Inspection Drone (MID) that will be capable of finding the lava tube and scan it in real time using a RealSense camera. Linux based system was chosen in the development process, which allow ROS catkin workspace to connect ArduPilot and Gazebo. The navigation is accomplished using a combination of an inertial measurement unit, a camera, and a laser rangefinder. The flight computer runs an optical flow algorithm for control and navigation purpose. Also, a convolutional neural network is used to perform lava tube entrance detection and inspection. • Martian flying drone simulated with ArduPilot, Gazebo and ROS. • Martian environment simulation. • Martian lava tube detection using convolutional neural networks. • Martian lava tube inspection using a realsense camera. [ABSTRACT FROM AUTHOR]

With the goal of solving the problem of feature extractors lacking strong supervision training and insufficient time information concerning single‐sequence model learning, a hierarchical sequence memory network with a multi‐level iterative optimisation strategy is proposed for continuous sign language recognition. This method uses the spatial‐temporal fusion convolution network (STFC‐Net) to extract the spatial‐temporal information of RGB and Optical flow video frames to obtain the multi‐modal visual features of a sign language video. Then, in order to enhance the temporal relationships of visual feature maps, the hierarchical memory sequence network is used to capture local utterance features and global context dependencies across time dimensions to obtain sequence features. Finally, the decoder decodes the final sentence sequence. In order to enhance the feature extractor, the authors adopted a multi‐level iterative optimisation strategy to fine‐tune STFC‐Net and the utterance feature extractor. The experimental results on the RWTH‐Phoenix‐Weather multi‐signer 2014 dataset and the Chinese sign language dataset show the effectiveness and superiority of this method. [ABSTRACT FROM AUTHOR]

Automated assessment of broiler chicken welfare poses particular problems due to the large numbers of birds involved and the variety of different welfare measures that have been proposed. Active (sustained, defect-free) walking is both a universally agreed measure of bird health and a behavior that can be recognized by existing technology. This makes active walking an ideal starting point for automated assessment of chicken welfare at both individual and flock level. [ABSTRACT FROM AUTHOR]

Classifying sports videos is complex due to their dynamic nature. Traditional methods, like optical flow and the Histogram of Oriented Gradient (HOG), are limited by their need for expertise and lack of universality. Deep learning, particularly Convolutional Neural Networks (CNNs), offers more effective feature recognition in sports videos, but standard CNNs struggle with fast-paced or low-resolution sports videos. Our novel neural network model addresses these challenges. It begins by selecting important frames from sports footage and applying a fuzzy noise reduction algorithm to enhance video quality. The model then uses a bifurcated neural network to extract detailed features, leading to a densely connected neural network with a specific activation function for categorizing videos. We tested our model on a High-Definition Sports Video Dataset covering over 20 sports and a low-resolution dataset. Our model outperformed established classifiers like DenseNet, VggNet, Inception v3, and ResNet-50. It achieved high precision (0.9718), accuracy (0.9804), F-score (0.9761), and recall (0.9723) on the high-resolution dataset, and significantly better precision (0.8725) on the low-resolution dataset. Correspondingly, the highest values on the matrix of four traditional models are: precision (0.9690), accuracy (0.9781), F-score (0.9670), recall (0.9681) on the high-resolution dataset, and precision (0.8627) on the low-resolution dataset. This demonstrates our model's superior performance in sports video classification under various conditions, including rapid motion and low resolution. It marks a significant step forward in sports data analytics and content categorization. [ABSTRACT FROM AUTHOR]

To properly maintain and manage track surfaces, the amount of collapsed settlement of a ballast bed must be determined. However, high maintenance costs make it difficult for railway operators to fully maintain and manage track surfaces. Therefore, we propose a method of estimating the shape of the ballast bed using Dense Inverse Search (DIS) optical flow in images captured using a camcorder, which is inexpensive. This paper describes the outline of the method, an evaluation of its accuracy through simulation and model experiments, and the results of applying the method to actual images captured. [ABSTRACT FROM AUTHOR]

Optical and digital images are one of the most important channels for transmitting information. At the Institute for Information Transmission Problems of the Russian Academy of Sciences (IITP RAS), this topic has always, since the founding of the institute, been given the closest attention. The institute's employees have made significant contributions to both the domestic and global science of image processing. In this work, the authors touched only on the main stages of the development of the theory, methods, and algorithms for image processing and analysis in the Laboratory of Digital Optics of the Institute for Information Transmission Problems of the Russian Academy of Sciences. [ABSTRACT FROM AUTHOR]

Physics-based optical flow models have been successful in capturing the deformities in fluid motion arising from digital imagery. However, a common theoretical framework analyzing several physics-based models is missing. In this regard, we formulate a general framework for fluid motion estimation using a constraint-based refinement approach. We demonstrate that for a particular choice of constraint, our results closely approximate the classical continuity equation-based method for fluid flow. This closeness is theoretically justified by augmented Lagrangian method in a novel way. The convergence of Uzawa iterates is shown using a modified bounded constraint algorithm. The mathematical well-posedness is studied in a Hilbert space setting. Further, we observe a surprising connection to the Cauchy-Riemann operator that diagonalizes the system leading to a diffusive phenomenon involving the divergence and the curl of the flow. Several numerical experiments are performed and the results are shown on different datasets. Additionally, we demonstrate that a flow-driven refinement process involving the curl of the flow outperforms the classical physics-based optical flow method without any additional assumptions on the image data. [ABSTRACT FROM AUTHOR]

Automated assessment of broiler chicken welfare poses particular problems due to the large numbers of birds involved and the variety of different welfare measures that have been proposed. Active (sustained, defect-free) walking is both a universally agreed measure of bird health and a behavior that can be recognized by existing technology. This makes active walking an ideal starting point for automated assessment of chicken welfare at both individual and flock level.

Overlapping occlusion has seriously limited the yield estimation of bagged grapes in recent years, due to the ever-increasing grape volume after bagging and the large surface area of grape leaves. The unstable speed of manual video shooting can also lead to the loss of bagging grape targets. In this study, a yield estimation was proposed for the bagged grape using self-correcting Non-Maximum Suppression (NMS)-ByteTrack. Firstly, the bagged grapes were detected in the video using object detection (YOLOv5s). The NMS operation was also post-positioned to the tracking stage to retain the fruit detection boxes that filtered under the occlusion. Specifically, the detection boxes of bagged grapes were detected by object detection (YOLOv5s), whereas, the prediction boxes of bagged grapes were predicted to calculate the intersection over union using the Kalman filter. If the intersection over the union was less than the given threshold, the detection box was filtered to reserve the detection boxes closer to the predicted position. Then, the final detection box was obtained using the NMS operation. Secondly, the camera motion compensation and improved Kalman filter algorithm were added to automatically correct the position of the fruit prediction boxes and track them using ByteTrack. Specifically, the camera motion compensation was used to first extract the background key points in the bagged grape pictures of the previous frame and the current frame except for the tracking target. The sparse optical flow was used to match the extracted background in the key points of the bagged grapes. Then the affine transformation matrix of background motion was calculated by the RANSAC algorithm, and the Kalman filter was utilized to predict the bagged grape prediction boxes of the current frame. Finally, the affine transformation matrix was obtained to convert the prediction boxes in the coordinate system from the previous to the current frame, in order to realize the position information correction for the prediction boxes of the current frame. In addition, the improved Kalman filter algorithm can be used to change the aspect ratio in the state vector, and then to express the position of the bagged grape tracking boxes in the Kalman filter directly as the width, in order to more accurately estimate the position of the bagged grape tracking boxes. As such, a line-counting strategy was proposed to automatically count the bagged grapes. The strategy was also to set the counting line in the middle of the video, and also retain the idle frames without the bagged grapes in the first few seconds of the video. Furthermore, the bagged grapes with the same ID were only counted once to avoid repeated collision line counting. The successful collision between the center point of the tracking box of bagged grapes and the counting line was achieved in the automatic counting of bagged grapes. The dataset was collected from the Paidengte Agricultural Science and Technology Demonstration Park, Bishan District, Chongqing of China. The mobile phone cameras of Redmi K40 and OPPO Reno6pro+ were selected to capture the pictures of the same bagged grape at 8:00, 12:00, and 18:00 from different angles, such as frontal, sideways, and overhead shots. The total shooting time was about 6 h, the shooting height was about 1.5 m from the ground, and the shooting route was line by line. A total of 500 images and six videos were obtained for the bagged grapes. Among them, 500 images were expanded to 2000 images after saturation, brightness enhancement, brightness reduction, and mirror operations. Then, 2000 images were randomly divided into the training and validation sets in the ratio of 8:2, where six valid videos were used as the test set. Relevant experiments were conducted using this dataset. The experimental results showed that the yield estimation achieved a significant improvement in the tracking performance of bagged grapes using self-correcting NMS-ByteTrack. The multi-object tracking accuracy and precision, as well as the identification F1-score, were 64.6%, 82.4% and 80.8%, respectively, which increased by 1.7, 1.0 and 4.1 percentage points, respectively, compared with the ByteTrack. The number of ID switch was reduced by 3 times. Then, the average counting accuracy reached 82.8% in terms of counting performance, compared with manual counting. In addition, a comparison was also made with the five tracking methods. A better tracking performance was achieved, compared with the rest. The applicability of this estimation was also verified in tracking and counting bagged grapes. Therefore, the yield estimation can be expected to effectively promote the tracking and counting of bagged grapes in real scenarios using self-correcting NMS-ByteTrack. More accurate yield estimation of bagged grapes was also achieved in this field. [ABSTRACT FROM AUTHOR]

In recent years, the analysis of macro- and micro-expression has drawn the attention of researchers. These expressions provide visual cues to an individual's emotions, which can be used in a broad range of potential applications such as lie detection and policing. In this paper, we address the challenge of spotting facial macro- and micro-expression from videos and present compelling results by using a deep learning approach to analyze the optical flow features. Unlike other deep learning approaches that are mainly based on Convolutional Neural Networks (CNNs), we propose a Transformer-based deep learning approach that predicts a score indicating the probability of a frame being within an expression interval. In contrast to other Transformer-based models that achieve high performance by being pre-trained on large datasets, our deep learning model, called SL-Swin, which incorporates Shifted Patch Tokenization and Locality Self-Attention into the backbone Swin Transformer network, effectively spots macro- and micro-expressions by being trained from scratch on small-size expression datasets. Our evaluation outcomes surpass the MEGC 2022 spotting baseline result, obtaining an overall F1-score of 0.1366. Additionally, our approach performs well on the MEGC 2021 spotting task, with an overall F1-score of 0.1824 and 0.1357 on the CAS(ME) 2 and SAMM Long Videos, respectively. The code is publicly available on GitHub. [ABSTRACT FROM AUTHOR]

The detection of moving objects in images is a crucial research objective; however, several challenges, such as low accuracy, background fixing or moving, 'ghost' issues, and warping, exist in its execution. The majority of approaches operate with a fixed camera. This study proposes a robust feature threshold moving object identification and segmentation method with enhanced optical flow estimation to overcome these challenges. Unlike most optical flow Otsu segmentation for fixed cameras, a background feature threshold segmentation technique based on a combination of the Horn–Schunck (HS) and Lucas–Kanade (LK) optical flow methods is presented in this paper. This approach aims to obtain the segmentation of moving objects. First, the HS and LK optical flows with the image pyramid are integrated to establish the high-precision and anti-interference optical flow estimation equation. Next, the Delaunay triangulation is used to solve the motion occlusion problem. Finally, the proposed robust feature threshold segmentation method is applied to the optical flow field to attract the moving object, which is the. extracted from the Harris feature and the image background affine transformation model. The technique uses morphological image processing to create the final moving target foreground area. Experimental results verified that this method successfully detected and segmented objects with high accuracy when the camera was either fixed or moving. [ABSTRACT FROM AUTHOR]

In recent years, autonomous robots have been practically used outdoors for transportation, delivery, and other applications. To ensure safety and reliability, it is necessary to measure and evaluate the accuracy of a robot's movement from the outside. In this study, we develop a device to externally measure the coordinates and azimuth angle of a robot by combining image processing and distance measurement. In this study, an azimuth angle detection method that combines coarse-angle detection using AKAZE feature detection and small-angle detection using optical flow is proposed. The experimental results show that the azimuth angle can be detected with a standard deviation of σ=0.70° indoors and σ=0.99° outdoors. [ABSTRACT FROM AUTHOR]

This study presents the algorithm ACTION, defined as automatic center detection of tropical cyclone (TC) using image processing based on the operational radar network. Based on the high visibility of weather radar imagery, a TC's motion vector is calculated from the continuous image change using optical flow, producing the TC's rotation center. The algorithm performance was verified by analyzing the typhoons (TCs in the northwestern Pacific) that affected the Korean Peninsula from 2018–2021, demonstrating a high detection rate of 80.8% within an error distance of 40 km against the best track of the Korea Meteorological Administration (KMA). The detection rate was 100% for typhoons with temporally consistent morphological characteristics. ACTION automatically generates TC center information upon the TC's initial entry inside the observation radius even in the absence of uniform radar data. As ACTION is capable of performing real-time calculations that are directly applied to rapidly generated weather radar images, it is currently being utilized by the KMA. The high temporal resolution TC center information calculated through ACTION is expected to improve the efficiency of TC forecasting. [ABSTRACT FROM AUTHOR]

Change detection is an important task in remote sensing image processing and analysis. However, due to position errors and wind interference, bi-temporal low-altitude remote sensing images collected by SUAVs often suffer from different viewing angles. The existing methods need to use an independent registration network for registration before change detection, which greatly reduces the integrity and speed of the task. In this work, we propose an end-to-end network architecture RegCD-Net to address change detection problems in the bi-temporal SUAVs' low-altitude remote sensing images. We utilize global and local correlations to generate an optical flow pyramid and realize image registration through layer-by-layer optical flow fields. Then we use a nested connection to combine the rich semantic information in deep layers of the network and the precise location information in the shallow layers and perform deep supervision through the combined attention module to finally achieve change detection in bi-temporal images. We apply this network to the task of change detection in the garbage-scattered areas of nature reserves and establish a related dataset. Experimental results show that our RegCD-Net outperforms several state-of-the-art CD methods with more precise change edge representation, relatively few parameters, fast speed, and better integration without additional registration networks. [ABSTRACT FROM AUTHOR]

— In this study, an image processing-based deep learning approach for short-term forecast of solar radiation has been developed. For this purpose, firstly, cloud movements occurred during the day are tracked and future cloud movements are forecasted, accordingly. Subsequently, using the cloud motion estimation and extraterrestrial solar radiation data, 1-min averaged solar radiation values are estimated for 5-min time horizon. Shi-Tomasi method is employed to determine the feature points to be tracked on the sky images whereas, Lucas-Kanade optical flow method is employed to track the determined feature points on the sequential images. Average cloud velocity and directions are calculated by the help of linear regression method from tracked cloud movements. A hybrid approach including K-means and red/blue ratio is built to classify the pixels of the image whether they are clouds or sky. Finally, short-term solar radiations are estimated using the Long-Short Term Memory (LSTM) deep learning method. The performance of the proposed approach is compared with other methods in the literature. As a result it is concluded that, developed approach outperforms most methods in the literature with RMSE values of 47.576, 53.830, 68.103, and 92.386 for four different days and can be used as an alternative approach. [ABSTRACT FROM AUTHOR]

Sports video analysis has received much attention as it turned to be a hot research area in the field of image processing. This motivation offers opportunities that develop fascinating applications supported by analysis of different sports, especially soccer. Ball identification, in soccer images, is an essential task not only for goal‐scoring but also for performance evaluation. However, ball detection suffers from several hurdles such as occlusions, fast‐moving objects, shadows, poor lighting, color contrast, and other static background objects. Although several ball detection techniques have been introduced such as Frame Difference, Mixture of Gaussian (MoG), Optical Flow, and so forth; ball detection in soccer games is still an open research area. In this paper, a new Fuzzy Based Ball Detection (FB2D) strategy is proposed for identifying the ball through a set of image sequences extracted from a soccer match video. FB2D can accurately identify the ball even if it is attached to the white lines drawn on the playground or partially occluded behind players. FB2D is compared to recent ball detection techniques. Experimental results show that FB2D outperforms recent detection techniques as it introduces both the highest level of detection accuracy in the testing stage and the lowest possible error. [ABSTRACT FROM AUTHOR]

As image acquisition devices have outstanding potential for gathering vibration information, computer vision has received a lot of interest in structural health monitoring (SHM). In this work, a fully automated peak picking methodology based on computer vision in tandem with deep learning is proposed to realize vibration measurements and identify natural frequencies from the plot of the power spectral density transmissibility (PSDT). A deep-learning-enhanced image processing technology was used to extract the vibration signals with automatic active pixel selection, while a convolutional neural network was used to further process the vibration measurements so that the frequencies could be identified from PSDT-based functions. The proposed method was verified by three case studies, including the dynamic testing of two laboratory models and the field testing of the stay cable. The findings showed that the proposed deep-learning-enhanced approach has a high potential for use in SHM by automatically performing vibration measurement and frequency extraction. [ABSTRACT FROM AUTHOR]

Copyright of Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji is the property of Gazi University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Quantification of abnormal contractile motions of cardiac tissue has been a noteworthy challenge and significant limitation in assessing and classifying the drug-induced arrhythmias (i.e., Torsades de pointes). To overcome these challenges, researchers have taken advantage of computational image processing tools to measure contractile motion from cardiomyocytes derived from human induced pluripotent stem cells (hiPSC-CMs). However, the amplitude and frequency analysis of contractile motion waveforms does not produce sufficient information to objectively classify the degree of variations between two or more sets of cardiac contractile motions. In this paper, we generated contractile motion data from beating hiPSC-CMs using motion tracking software based on optical flow analysis, and then implemented a computational algorithm, phase space reconstruction (PSR), to derive parameters (embedding, regularity, and fractal dimensions) to further characterize the dynamic nature of the cardiac contractile motions. Application of drugs known to cause cardiac arrhythmia induced significant changes to these resultant dimensional parameters calculated from PSR analysis. Integrating this new computational algorithm with the existing analytical toolbox of cardiac contractile motions will allow us to expand current assessments of cardiac tissue physiology into an automated, high-throughput, and quantifiable manner which will allow more objective assessments of drug-induced proarrhythmias.

Recently, deep neural networks have shown surprising results in solving most of the traditional image processing problems. However, the video frame interpolation field does not show relatively good performance because the receptive field requires a vast spatio-temporal range. To reduce the computational complexity, in most frame interpolation studies, motion is first calculated with the optical flow, then interpolated frames are generated through backward warping. However, while the backward warping process is simple to implement, the interpolated image contains mixed motion and ghosting defects. Therefore, we propose a new network that does not use the backward warping method through the proposed max-min warping. Since max-min warping generates a clear warping image in advance according to the size of the motion and the network is configured to select the warping result according to the warped layer, using the proposed method, it is possible to optimize the computational complexity while selecting a contextually appropriate image. The video interpolation method using the proposed method showed 34.847 PSNR in the Vimeo90k dataset and 0.13 PSNR improvement compared to the Quadratic Video Interpolation method, showing that it is an efficient frame interpolation self-supervised learning. [ABSTRACT FROM AUTHOR]

This study sought an effective detection method not only for flame but also for the smoke generated in the event of a fire. To this end, the flame region was pre-processed using the color conversion and corner detection method, and the smoke region could be detected using the dark channel prior and optical flow. This eliminates unnecessary background regions and allows selection of fire-related regions. Where there was a pre-processed region of interest, inference was conducted using a deep-learning-based convolutional neural network (CNN) to accurately determine whether it was a flame or smoke. Through this approach, the detection accuracy is improved by 5.5% for flame and 6% for smoke compared to when a fire is detected through the object detection model without separate pre-processing. [ABSTRACT FROM AUTHOR]

Optical flow estimation is a fundamental task in computer vision and image processing. Due to the difficulty in obtaining the ground truth of flow field, unsupervised learning approaches attract more and more research interests in recent years. However, despite of their good generalization capability, unsupervised optical flow methods suffer in the scenarios with large displacement, small objects, and occlusions. In this work, we propose a novel optical flow network based on decoder with multi-scale kernels. Different from previous U-Net like or pyramidal methods, we design our network based on RAFT architecture that with a 4D correlation layer and recurrent decoder. More importantly, we incorporate three novel ideas with regard to the input, information processing and output of the update units improve the performance. Firstly, we utilize various motion-related information as input to the update units. Secondly, we propose a module of multi-scale update unit. Thirdly, for the final flow up-sampling procedure, we propose an image-guided up-sampling loss to guide the learning of up-sampling masks. Our model is trained by the occlusion-aware photometric loss, edge-aware smoothness loss, self-supervised loss, and image-guided up-sampling loss. Experimental results demonstrate that our model achieves the state-of-the-art performance on both Sintel and KITTI and outperforms other unsupervised optical flow methods remarkably. [ABSTRACT FROM AUTHOR]

Fractional calculus is an abstract idea exploring interpretations of differentiation having non-integer order. For a very long time, it was considered as a topic of mere theoretical interest. However, the introduction of several useful definitions of fractional derivatives has extended its domain to applications. Supported by computational power and algorithmic representations, fractional calculus has emerged as a multifarious domain. It has been found that the fractional derivatives are capable of incorporating memory into the system and thus suitable to improve the performance of locality-aware tasks such as image processing and computer vision in general. This article presents an extensive survey of fractional-order derivative-based techniques that are used in computer vision. It briefly introduces the basics and presents applications of the fractional calculus in six different domains viz. edge detection, optical flow, image segmentation, image de-noising, image recognition, and object detection. The fractional derivatives ensure noise resilience and can preserve both high and low-frequency components of an image. The relative similarity of neighboring pixels can get affected by an error, noise, or non–homogeneous illumination in an image. In that case, the fractional differentiation can model special similarities and help compensate for the issue suitably. The fractional derivatives can be evaluated for discontinuous functions, which help estimate discontinuous optical flow. The order of the differentiation also provides an additional degree of freedom in the optimization process. This study shows the successful implementations of fractional calculus in computer vision and contributes to bringing out challenges and future scopes. [ABSTRACT FROM AUTHOR]

Determining the optical flow of a video is a compute-intensive task essential for computer vision. For achieving this processing in real time, the whole algorithm deployment chain must be thought of for efficiency first. The development is usually divided into two parts: first, designing an algorithm that meets precision constraints, then, implementing and optimizing its execution on the targeted platform. We argue that unifying those operations enhances performance on the embedded processor. This paper is based on an industrial use case of computer vision. The objective is to determine dense optical flow in real time on an embedded GPU platform: the Nvidia AGX Xavier. The CLG (combined local–global) optical flow method, initially chosen, is analyzed to understand the convergence speed of its underlying optimization problem. The Jacobi solver is selected for implementation because of its parallel nature. The whole multi-level processing is then ported to the GPU, using several specific optimization strategies. In particular, we analyze the impact of fusing the solver's iterations with the roofline model. As a result, with a 30 W power budget, our implementation runs at 60FPS, on 640 × 512 images, with a four-level processing. Hopefully, this example should provide feedback on the issues that arise when trying to port a method to a parallel platform and serve for further implementations of computer vision algorithms on specialized hardware. [ABSTRACT FROM AUTHOR]

This study presents the algorithm ACTION, defined as automatic center detection of tropical cyclone (TC) using image processing based on the operational radar network. Based on the high visibility of weather radar imagery, a TC’s motion vector is calculated from the continuous image change using optical flow, producing the TC’s rotation center. The algorithm performance was verified by analyzing the typhoons (TCs in the northwestern Pacific) that affected the Korean Peninsula from 2018–2021, demonstrating a high detection rate of 80.8% within an error distance of 40 km against the best track of the Korea Meteorological Administration (KMA). The detection rate was 100% for typhoons with temporally consistent morphological characteristics. ACTION automatically generates TC center information upon the TC’s initial entry inside the observation radius even in the absence of uniform radar data. As ACTION is capable of performing real-time calculations that are directly applied to rapidly generated weather radar images, it is currently being utilized by the KMA. The high temporal resolution TC center information calculated through ACTION is expected to improve the efficiency of TC forecasting.

In this article, we propose a novel model for facial micro-expression (FME) recognition. The proposed model basically comprises a transformer, which is recently used for computer vision and has never been used for FME recognition. A transformer requires a huge amount of data compared to a convolution neural network. Then, we use motion features, such as optical flow and late fusion to complement the lack of FME dataset. The proposed method was verified and evaluated using the SMIC and CASME II datasets. Our approach achieved state-of-the-art (SOTA) performance of 0.7447 and 73.17% in SMIC in terms of unweighted F1 score (UF1) and accuracy (Acc.), respectively, which are 0.31 and 1.8% higher than previous SOTA. Furthermore, UF1 of 0.7106 and Acc. of 70.68% were shown in the CASME II experiment, which are comparable with SOTA. [ABSTRACT FROM AUTHOR]

In the field of computer vision, high-dynamic dance motion recognition is a difficult problem to solve. Its goal is to recognize human motion by analyzing video data using image processing and classification recognition technology. Video multifeature fusion has sparked a surge in research in a variety of fields. Several pixel points that can be distinguished and displayed in several adjacent images that can reflect their characteristics are referred to as multifeature fusion. It is responsible for a significant portion of the similarity results between the two video segments. Motion recognition relies heavily on video multifeature fusion, which has a direct impact on the robustness and accuracy of recognition results. The directional gradient histogram features, optical flow direction histogram features, and audio features extracted from dance video are used to characterize dance movements after all of the characteristics of dance movements have been considered. This paper focuses on the high-dynamic dance action recognition method based on video multifeature fusion, which aims to combine high-dynamic dance action recognition and video multifeature fusion. [ABSTRACT FROM AUTHOR]

Recent advances in neural signal processing led to significant improvements in the performance of learned image/video restoration and super-resolution (SR). An important benefit of data-driven deep learning approaches to image processing is that neural models can be optimized for any differentiable loss function, including perceptual loss functions, leading to perceptual image/video restoration and SR, which cannot be easily handled by traditional model-based methods. We start with a brief problem statement and a short discussion on traditional vs. data-driven solutions. We next review recent advances in neural architectures, such as residual blocks, dense connections, residual-in-residual dense blocks, residual blocks with generative neurons, self-attention and visual transformers. We then discuss loss functions and evaluation (assessment) criteria for image/video restoration and SR, including fidelity (distortion) and perceptual criteria, and the relation between them, where we briefly review the perception vs. distortion trade-off. We can consider learned image/video restoration and SR as learning either a nonlinear regressive mapping from degraded to ideal images based on the universal approximation theorem, or a generative model that captures the probability distribution of ideal images. We first review regressive inference via residual and/or dense convolutional networks (ConvNet). We also show that using a new architecture with residual blocks based on a generative neuron model can outperform classical residual ConvNets in peak-signal-to-noise ratio (PSNR). We next discuss generative inference based on adversarial training, such as SRGAN and ESRGAN, which can reproduce realistic textures, or based on normalizing flow such as SRFlow by optimizing log-likelihood. We then discuss problems in applying supervised training to real-life restoration and SR, including overfitting image priors and overfitting the degradation model seen in the training set. We introduce multiple-model SR and real-world SR (from unpaired training data) formulations to overcome these problems. Integration of traditional model-based methods and deep learning for non-blind restoration/SR is introduced as another solution to model overfitting in supervised learning. In learned video restoration and SR (VSR), we first discuss how to best exploit temporal correlations in video, including sliding temporal window vs. recurrent architectures for propagation, and aligning frames in the pixel domain using optical flow vs. in the feature space using deformable convolutions. We next introduce early fusion with feature-space alignment, employed by the EDVR network, which obtains excellent PSNR performance. However, it is well-known that videos with the highest PSNR may not be the most appealing to humans, since minimizing the mean-square error may result in blurring of details. We then address perceptual optimization of VSR models to obtain natural texture and motion. Although perception-distortion tradeoff has been well studied for images, few works address perceptual VSR. In addition to using perceptual losses, such as MS-SSIM, LPIPS, and/or adversarial training, we also discuss explicit loss functions/criteria to enforce and evaluate temporal consistency. We conclude with a discussion of open problems. [ABSTRACT FROM AUTHOR]

This study presents an automated registration method based on optical flow for an adaptive optics scanning laser ophthalmoscope. The method was designed to align and average images to obtain a higher signal-to-noise ratio image. A correlation based optical flow image registration method, which has large registration degrees of freedom, is adopted as a local registration method. By comparing the images before and after image registration, we show the effectiveness of our method. Furthermore, the advantage of our method, which is the containment of large registration degrees of freedom, is confirmed.

This work presents two methods that facilitate a 3D reconstruction of microscopic blood vessels in the volume slightly larger than 1 mm3. The source of the data are histological serial sections, i.e., microscopic images of probes, stained with immunohistochemistry. Odd and even sections have different stainings in our primary data set. Thus, firstly, an approach to register an alternately-stained series is presented. With image filtering and a feature-detection-based registration we obtain a registered stack of 148 serial sections. The series has missing sections, locally damaged sections, artifacts from acquisition. All these hinder correct connectivity of blood vessels. With our second approach we interpolate the missing information while maintaining the connectivity. We achieve this with deformations based on dense optical flow. The presented methodology is applicable to further histological series. A combination of both approaches allows us to reconstruct more than 76% larger volumes. An important detail was the composition mode of images. Summarizing, we use methods from image processing and computer vision to create large-scale 3D models from immunostained histological serial sections.

High‐dynamic‐range image (HDRI) displays scenes as vivid as the real scenes. HDRI can be reconstructed by fusing a set of bracketed‐exposure low‐dynamic‐range images (LDRI). For the reconstruction, many works succeed in removing the ghost artefacts caused by moving objects. The critical issue is reconstructing the areas which are saturated due to bad exposure and occluded due to motion with no ghost artefacts. To overcome this issue, this paper proposes texture and exposure awareness based refill. The proposed work first locates the saturated and occluded areas existing in input image set, then refills background textures or patches containing rough exposure and colour information into located areas. Proposed work can be integrated with multiple existing ghost removal works to improve the reconstruction result. Experimental results show that proposed work removes the ghost artefacts caused by saturated and occluded areas in subjective evaluation. For the objective evaluation, the proposed work improves the HDR‐VDP‐2 evaluation result for multiple conventional works by 1.33% on average. [ABSTRACT FROM AUTHOR]

Cryo-electron microscopy using single particle analysis requires the computational averaging of thousands of projection images captured from identical macromolecules. However, macromolecules usually present some degree of flexibility showing different conformations. Computational approaches are then required to classify heterogeneous single particle images into homogeneous sets corresponding to different structural states. Nonetheless, sometimes the attainable resolution of reconstructions obtained from these smaller homogeneous sets is compromised because of reduced number of particles or lack of images at certain macromolecular orientations. In these situations, the current solution to improve map resolution is returning to the electron microscope and collect more data. In this work, we present a fast approach to partially overcome this limitation for heterogeneous data sets. Our method is based on deforming and then moving particles between different conformations using an optical flow approach. Particles are then merged into a unique conformation obtaining reconstructions with improved resolution, contrast and signal-to-noise ratio. We present experimental results that show clear improvements in the quality of obtained 3D maps, however, there are also limits to this approach, i.e., the method is restricted to small deformations and cannot determine local patterns of flexibility of small elements, such as secondary structures, which we discuss in the manuscript. [Display omitted] • Cryo-EM using SPA requires the computational averaging of thousands of projections captured from identical macromolecules. • Computational approaches are required to classify heterogeneous particles into homogeneous sets at different conformations. • We propose a fast and efficient approach that is aimed at "making the best" from relatively limited data sets. • Our method focuses on artificially increasing the number of particles in a class. • In this regard, we use a regularized optical flow approach to move particles between different classes/conformations. [ABSTRACT FROM AUTHOR]

Summary: Cell membrane deformation is an important feature that occurs during many physiological processes, and its study has been put to good use to investigate cardiomyocyte function. Several methods have been developed to extract information on cardiomyocyte contractility. However, no existing computational framework has provided, in a single platform, a straightforward approach to acquire, process, and quantify this type of cellular dynamics. For this reason, we develop CONTRACTIONWAVE, high-performance software written in Python programming language that allows the user to process large data image files and obtain contractility parameters by analyzing optical flow from images obtained with videomicroscopy. The software was validated by using neonatal, adult-, and human-induced pluripotent stem-cell-derived cardiomyocytes, treated or not with drugs known to affect contractility. Results presented indicate that CONTRACTIONWAVE is an excellent tool for examining changes to cardiac cellular contractility in animal models of disease and for pharmacological and toxicology screening during drug discovery. Motivation: Quantification of cardiomyocyte contractility is an important step in understanding the cellular mechanisms involved in the pathogenesis of cardiac disease. However, accurate and reproducible measurements are hampered by several factors inherent to the software options currently available. These include limited ability to process large datasets, the inability to provide calibrated measurements of contractility speed, the absence of a unified and accessible computational package, the high cost of some programs, and often the need for advanced programming skills. Aiming to overcome these limitations, we developed CONTRACTIONWAVE, open-source Python software that provides high-performance algorithms for large-scale analysis of cardiac contraction.

OpenMV Board is designed for purpose of non-complex image processing applications. It is an image processing sensor that has been a MicroPython embedded operatingsystem( OS). In the study, it is aimed to develop gesture control applications for electrical household appliances and small budget devices. Therefore, the hardware to be used should be cheap and the algorithm should be simple. Thus three gesture control applications have been developed by using OpenMv board for use in different electrical appliances. These are 1-level control, 2-multi-component simple system control and 3-page flip. The algorithms used in the study are independent of the user because they are optical flow-based. Thus, the use of low-cost simple gesture control applications for industrial purposes (electrical appliances) can be realized. Algorithms developed for applications were written on the OpenMV IDE. These application results were monitored in real-time through the IDE. In addition, the algorithm developed for level control has been embedded and tested on an SD card on OpenMv independent of OpenMV IDE. During the test, output information was generated using OpenMV pins and the level indicator created using yellow, green and red LEDs connected to the pins was checked real-time. Thus, the algorithm was tested on a computer-independent embedded system. [ABSTRACT FROM AUTHOR]

• We would like to submit the manuscript entitled "High-order relational generative adversarial network for video super-resolution", which we wish to be considered as a research article for publication in "Pattern Recognition". Because our contributed research covers the area" Image processing and representation", this manuscript belongs in this journal. No conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication. • Both the instability of motion compensation and feature aggregation often undermine the SR results of videos. Only the global spatio-temporal dependencies of same scale features are considered as one-order relations, with difficulty in handling the complex motions and context variations. To deal with these issues, we argue that high-order relations should be modelled and applied in the generative strategies to exploit feature alignments and long-range information fusion. In fact, the high-order relations mean that the dependencies of features among pixel positions are mined not only from global feature representations but also through local patch interactions. Moreover, the stronger relations can be further revealed across different scales, by which the aggregation weight matrices with high-order statistics are adaptively determined. The videos are regarded as the complementary fusion of motion and context. Thus, simultaneously capturing the underlying relations of the motion and context is the key to video SR. In this paper, we propose a high-order relational generative adversarial network (HOR-GAN) for accurate video SR, which has strong capacity of alleviating the erroneous motion compensation and merging the useful information among the consecutive frames. • In summary, we highlight the main contributions of this paper as follows: We propose a HOR-GAN framework for highly accurate and realistic video SR. By exploiting high-order relations of feature patches under the construction of the pyramid graphs, the proposed HOR-GAN works well to produce a better feature alignment and fuse more spatio-temporal information. We adopt the dual discriminators to provide spatially coherent feedback to the generator, thus making the generator focus more on fine-grained features. The effectiveness of the proposed method has been justified through the extensive experiments. • We design a motion-aware relation module to accurately align neighboring frames with the reference ones. In this module, a patch-wise matching strategy is first used to build cross-scale correspondences of similar patches. The graph attention layers then adaptively aggregate the local patch features to further decrease the alignment error. We develop a context-aware relation module to make full use of high-order dependencies among all warped feature patches. We introduce the multi-scale graph convolution layers to mine contextual interaction relations for better fusing the spatio-temporal features, in which the position information is encoded to augment the detail recovery. Finally, each pixel is enhanced via a global self-attention. • We deeply appreciate your consideration of our manuscript, and we look forward to receiving comments from the reviewers. If you have any queries, please don't hesitate to contact me. Yours sincerely, Rui Chen Video super-resolution can reconstruct a sequence of high-resolution frames with temporally consistent contents from their corresponding low-resolution sequences. The key challenge for this task is how to effectively utilize both inter-frame temporal relations and intra-frame spatial relations. The existing methods for super-resolving the videos commonly estimate optical flows to align the features of multiple frames based on temporal correlations. However, motion estimation is often error-prone and hence largely hinders the recovery of plausible details. Moreover, high-order contextual dependencies in the feature space are rarely exploited for further enhancing the spatio-temporal information fusion. To this end, we propose a novel generative adversarial network to super-resolve low-resolution videos, which makes full use of patch embeddings and is effective in exploring high-order spatio-temporal relations of the feature patches. Specifically, a motion-aware relation module is designed to handle the alignment between neighboring frames and reference ones. Depending on a patch-matching strategy for adaptive selection of multiple most similar patches, the cross-scale graph is constructed to reliably aggregate these patches using a feature pyramid. Based on the structure of multi-scale graph, a context-aware relation module is developed to capture high-order dependencies among resulting warped patches for better leveraging long-range complementary contexts. To further enhance reconstruction ability, the temporal position information of video sequences is also encoded into this module. Dual discriminators with cycle consistent constraints are adopted to provide more informative feedback to the generator while maintaining the global coherence. Extensive experiments have demonstrated the effectiveness of the proposed method in terms of quantitative and qualitative evaluation metrics. [ABSTRACT FROM AUTHOR]

Visual odometry (VO) is one of the essential techniques in mobile robots field; an accurate VO system is of great significance for mobile robot simultaneous localization and mapping. As for traditional monocular VO systems, they work by presuming the monocular scale is 1 (scale = 1), or relying on ground truth (GT) to estimate scale. As a result, the traditional monocular VO systems estimate the pose state with big drift or cannot work on the image sequence without GT. Although some classical monocular VO systems have been proposed, they still have imperfect performance or even unable to work in some extreme scene conditions, such as scene is monotony without obvious texture information or camera large-scale displacement motion. As for learning-based VO system, it is realized by training deep neural networks in supervised or self-supervised manner to end-to-end estimate the pose state; however, the accuracy of pose estimation entirely depends on the ability of networks. Although the ability of networks can be improved by increasing the number of training data sets and optimizing the network structure, it is inevitable to encounter problems such as insufficient generalization ability and insufficient accuracy on rotational pose estimation. In this article, a monocular VO system named DL_Hybrid is proposed, which takes full advantage of DL networks used in image processing and geometric localization theory based on hybrid pose estimation methods. The DL_Hybrid VO system can estimate a six-DoF pose one-frame-by-one-frame and recover camera trajectory, and it can extract accurate key points from per-frame even in extreme scene condition, and it has good performance even in the extreme moving condition, such as camera rotation-only action or static action, also it can work well in the condition of camera large-scale displacement motion. The real scale is also accurately estimated without depending on GT, and the pose estimation method is designed based on hybrid 2d–2d and 3d–2d localization theory to make the DL_Hyrid VO system to estimate translational and rotational information with accuracy and robustness. Experimental results show that the proposed DL_Hybrid VO system has a better performance than traditional and learning-based VO systems. [ABSTRACT FROM AUTHOR]

[Display omitted] • Vision-based cutting tool motion registration methods are presented. • Tool motion is recorded with low- and high-speed cameras with high resolutions. • Motion is estimated using edge detection, optical flow, and DIC schemes. • Tool motion from vision-based measurements agree with integrated accelerations. • Modal parameters from vision-based response compare well with standard procedures. This paper presents the use of vision-based methods for cutting tool motion registration and modal analysis. Motion of three illustrative tools were recorded using low- and high-speed cameras with sufficiently high resolutions. The tool's own features are used to register motion. Pixels within images from recordings of the vibrating tools are treated as non-contact motion sensors. Comparative analysis of three different methods of motion registration are presented to evaluate their suitability for the application of interest. These include variants of expanded edge detection and tracking schemes, expanded optical flow-based schemes, and established digital image correlation methods. Performance of different methods was observed to be governed by the tool's own features, illumination conditions, noise, and the image acquisition parameters. Extracted motion was benchmarked against twice integrated measured tool point accelerations, and motion was generally observed to compare well. Modal parameters extracted from vision-based measurements were also observed to agree with those extracted using more traditional experimental modal analysis procedures using a contact type accelerometer as the transducer. Since methods presented are generalized, they can suitably be adapted for other applications of interest. [ABSTRACT FROM AUTHOR]