Search

Abstract Geodetic monitoring measurements (e.g., of terrain surfaces) are used to detect deformations. Terrestrial laser scanning (TLS) or unmanned aircraft systems (UAS) equipped with lightweight cameras are often utilized for land surveying, resulting in point clouds that represent the surface of the captured object. For image-based acquisition of the area of interest, point clouds must first be generated from overlapping images, for which the Structure-from-Motion (SfM) method is commonly used. To perform deformation analyses and derive changes from them, at least two temporally different measurement epochs of the same area are required. In this article, we present both point cloud- and feature-based models from TLS and SfM-based UAS point clouds. In addition, an image-based 2D approach using optical flow is applied as an example for landslide simulation to detect changes on object surfaces. To eliminate erroneous results in the analyses due to vegetation areas, the 3D data is filtered using the CANUPO algorithm. The results of this research study show, that the task of deformation detection has some challenges, depending on the use case and the methodology. The point cloud-based methods are suitable to detect pure changes between two point clouds. Also, the direction of these changes can be determined to distinguish between material uplift and downlift. In contrast, the feature-based descriptor (Fast Point Feature Histogram, FPFH) assigns pairs of points between two epochs based on similar geometry in both point clouds therewith individual movements can be detected. However, areas that have changed significantly cannot be assigned. Optical flow shows point changes in similar dimensions to the target deformations and allows deformation analysis with much less computational effort than with 3D point clouds. Considering these findings, point cloud-based method are suitable for determining surface-based information, while the feature-based and image-based methods are capable of extracting local changes.

It is a challenging and meaningful task to carry out UAV-based livestock monitoring in high-altitude (more than 4500 m on average) and cold regions (annual average – 4 °C) on the Qinghai Tibet Plateau. The purpose of artificial intelligence (AI) is to execute automated tasks and to solve practical problems in actual applications by combining the software technology with the hardware carrier to create integrated advanced devices. Only in this way, the maximum value of AI could be realized. In this paper, a real-time tracking system with dynamic target tracking ability is proposed. It is developed based on the tracking-by-detection architecture using YOLOv7 and Deep SORT algorithms for target detection and tracking, respectively. In response to the problems encountered in the tracking process of complex and dense scenes, our work (1) Uses optical flow to compensate the Kalman filter, to solve the problem of mismatch between the target bounding box predicted by the Kalman filter (KF) and the input when the target detection in the current frame is complex, thereby improving the prediction accuracy; (2) Using a low confidence trajectory filtering method to reduce false positive trajectories generated by Deep SORT, thereby mitigating the impact of unreliable detection on target tracking. (3) A visual servo controller has been designed for the Unmanned Aerial Vehicle (UAV) to reduce the impact of rapid movement on tracking and ensure that the target is always within the field of view of the UAV camera, thereby achieving automatic tracking tasks. Finally, the system was tested using Tibetan yaks on the Qinghai Tibet Plateau as tracking targets, and the results showed that the system has real-time multi tracking ability and ideal visual servo effect in complex and dense scenes. [ABSTRACT FROM AUTHOR]

Motion object detection (MOD) with freely moving cameras is a challenging task in computer vision. To extract moving objects, most studies have focused on the difference in motion features between foreground and background, which works well for dynamic scenes with relatively regular movements and variations. However, abrupt illumination changes and occlusions often occur in real-world scenes, and the camera may also pan, tilt, rotate, and jitter, etc., resulting in local irregular variations and global discontinuities in motion features. Such complex and changing scenes bring great difficulty in detecting moving objects. To solve this problem, this paper proposes a new MOD method that effectively leverages local and global visual information for foreground/background segmentation. Specifically, on the global side, to support a wider range of camera motion, the relative inter-frame transformations are optimized to absolute transformations referenced to intermediate frames in a global form after enriching the inter-frame matching pairs. The global transformation is fine-tuned using the spatial transformer network (STN). On the local side, to address the problem of dynamic background scenes, foreground object detection is optimized by utilizing the pixel differences between the current frame and the local background model, as well as the consistency of local spatial variations. Then, the spatial information is combined using optical flow segmentation methods, enhancing the precision of the object information. The experimental results show that our method achieves a detection accuracy improvement of over 1.5% compared with the state-of-the-art methods on the datasets of CDNET2014, FBMS-59, and CBD. It demonstrates significant effectiveness in challenging scenarios such as shadows, abrupt changes in illumination, camera jitter, occlusion, and moving backgrounds. [ABSTRACT FROM AUTHOR]

Hierarchical concepts have proven useful in many classical and learning-based optical flow methods regarding both accuracy and robustness. In this paper we show that such concepts are still useful in the context of recent neural networks that follow RAFT's paradigm refraining from hierarchical strategies by relying on recurrent updates based on a single-scale all-pairs transform. To this end, we introduce MS-RAFT+: a novel recurrent multi-scale architecture based on RAFT that unifies several successful hierarchical concepts. It employs a coarse-to-fine estimation to enable the use of finer resolutions by useful initializations from coarser scales. Moreover, it relies on RAFT's correlation pyramid that allows to consider non-local cost information during the matching process. Furthermore, it makes use of advanced multi-scale features that incorporate high-level information from coarser scales. And finally, our method is trained subject to a sample-wise robust multi-scale multi-iteration loss that closely supervises each iteration on each scale, while allowing to discard particularly difficult samples. In combination with an appropriate mixed-dataset training strategy, our method performs favorably. It not only yields highly accurate results on the four major benchmarks (KITTI 2015, MPI Sintel, Middlebury and VIPER), it also allows to achieve these results with a single model and a single parameter setting. Our trained model and code are available at https://github.com/cv-stuttgart/MS_RAFT_plus. [ABSTRACT FROM AUTHOR]

Facial action unit (AU) recognition and optical flow estimation are two highly correlated tasks, since optical flow can provide motion information of facial muscles to facilitate AU recognition. However, most existing AU recognition methods handle the two tasks independently by offline extracting optical flow as auxiliary information or directly ignoring the use of optical flow. In this paper, we propose a novel end-to-end joint framework of AU recognition and optical flow estimation, in which the two tasks contribute to each other. Moreover, due to the lack of optical flow annotations in AU datasets, we propose to estimate optical flow in a self-supervised manner. To regularize the self-supervised estimation of optical flow, we propose an identical mapping constraint for the optical flow guided image warping process, in which the estimated optical flow between two same images is required to not change the image during warping. Experiments demonstrate that our framework (i) outperforms most of the state-of-the-art AU recognition methods on the challenging BP4D and GFT benchmarks, and (ii) also achieves competitive self-supervised optical flow estimation performance. • An end-to-end joint framework of AU recognition and self-supervised optical flow estimation. • An identity mapping constraint to ensure the reliability of self-supervised optical flow estimation. • An AU occurrence probability map regression loss for exploiting AU location information and facial spatial information. • Our approach outperforms most of the state-of-the-art AU recognition methods. • Our approach achieves comparable self-supervised optical flow estimation performance. [ABSTRACT FROM AUTHOR]

In current times, reconstruction of remote sensing images using super-resolution is a prominent topic of study. Remote sensing data have a complex spatial distribution. Compared with natural pictures, remote sensing pictures often contain subtler and more complicated information. Most super-resolution reconstruction algorithms cannot restore all the information contained in remote sensing images when reconstructing them. The content of some areas in the reconstructed images may be too smooth, and some areas may even have color changes, resulting in lower quality reconstructed images. In response to the problems presenting in current reconstruction algorithms about super-resolution, this article proposes the SRGAN-MSAM-DRC model (SRGAN model with multi-scale attention mechanism and dense residual connection). This model roots in generative adversarial networks and incorporates multi-scale attention mechanisms and dense residual connections into the generator. Furthermore, residual blocks are incorporated into the discriminator. We use some remote sensing image datasets of real-world data to evaluate this model, and the results indicate the SRGAN-MSAM-DRC model has shown enhancements in three evaluation metrics for reconstructed images about super-resolution. Compared to the basic SRGAN model, the SSIM (structural similarity), PSNR (peak signal-to-noise ratio), and IE (image entropy) increase by 5.0%, 4.0%, and 4.1%, respectively. From the results, we know the quality of the reconstructed images of remote sensing using the SRGAN-MSAM-DRC model is better than basic SRGAN model, and verifies that the model has good applicability and performance in reconstruction of remote sensing images using super-resolution. [ABSTRACT FROM AUTHOR]

Flight accidents caused by spatial disorientation (SD) greatly affect flight safety. Few studies have been devoted to the evaluation of SD. 10 pilots and 10 non-pilots were recruited for the experimental induction of SD. Videos for giving optical flow stimuli were played at two different flow speeds to induce SD. Subjective judgment and center of foot pressure (CoP) data were collected from the tests. The data were combined to determine the occurrence of SD and analyze the SD types. The number of self-reported SD events was slightly smaller in the pilots than in the non-pilots. The average upper bound of the confidence interval for the standard deviation of CoP was 0.32 ± 0.09 cm and 0.38 ± 0.12 cm in the pilots and non-pilots, respectively. This indicator was significantly lower in the pilots than in the non-pilots (P= 0.03). The success rate of the experimental induction of unrecognized SD was 26.7% and 45.0% in the pilots and non-pilots, respectively. The method offered a new to analyze unrecognized SD. We could determine the occurrence unrecognized SD. This is an essential means of reducing flight accidents caused by unrecognized SD. [ABSTRACT FROM AUTHOR]

n smart cities, sanitation workers are the key to urban construction. Accurate targeting of sanitation workers can help managers better monitor and manage them. Pedestrian detection is the core and key component of object detection technology. The difficulty of pedestrian detection in the actual feature recognition is still how to quickly and accurately identify the identity in the complex video scene. To realize the effective detection of sanitation workers, the study designs an identity identification scheme conducive to the friendly management of smart urban management. Since the optical flow feature extraction method and HSV color space extraction method can not meet the actual detection efficiency, this study innovatively integrates the two methods to improve the detection accuracy of the mode. Meanwhile, the study also introduces PCA algorithm to identify the specific identity of sanitation workers. In the actual detection of sanitation workers, the identification rate of two sanitation workers is high, and the similarity is 98.61%. This technology greatly reduces the false detection rate of actual detection and improves the detection accuracy. [ABSTRACT FROM AUTHOR]

We propose an adaptive fractional multi-scale optimization optical flow algorithm, which for the first time improves the over-smoothing of optical flow estimation under the total variation model from the perspective of global feature and local texture balance, and solves the problem that the convergence of fractional optical flow algorithms depends on the order parameter. Specifically, a fractional-order discrete L1-regularization Total Variational Optical Flow model is constructed. On this basis, the Ant Lion algorithm is innovatively used to realize the iterative calculation of the optical flow equation, and the fractional order is dynamically adjusted to obtain an adaptive optimization algorithm with strong search accuracy and high efficiency. In this paper, the flexibility of optical flow estimation in weak gradient texture scenes is increased, and the optical flow extraction rate of target features at multiple scales is greatly improved. We show excellent recognition performance and stability under the MPI_Sintel and Middlebury benchmarks. [ABSTRACT FROM AUTHOR]

Video super-resolution (VSR) remains challenging for real-world applications due to complex and unknown degradations. Existing methods lack the flexibility to handle video sequences with different degradation levels, thus failing to reflect real-world scenarios. To address this problem, we propose a degradation-adaptive video super-resolution network (DAVSR) based on a bidirectional propagation network. Specifically, we adaptively employ three distinct degradation levels to process input video sequences, aiming to obtain training pairs that reflect a variety of real-world corrupted images. We also equip the network with a pre-cleaning module to reduce noise and artifacts in the low-quality video sequences prior to information propagation. Additionally, compared to previous flow-based methods, we employ an unsupervised optical flow estimator to acquire a more precise optical flow to guide inter-frame alignment. Meanwhile, while maintaining network performance, we streamline the propagation network branches and the structure of the reconstruction module of the baseline network. Experiments are conducted on datasets with diverse degradation types to validate the effectiveness of DAVSR. Our method exhibits an average improvement of 0.18 dB over a recent SOTA approach (DBVSR) in terms of the PSNR metric. Extensive experiments demonstrate the effectiveness of our network in handling real-world video sequences with different degradation levels. [ABSTRACT FROM AUTHOR]

This paper proposes a new sensor using optical flow to stabilize a quadrotor when a GPS signal is not available. Normally, optical flow varies with the attitude of the aerial vehicle. This produces positive feedback on the attitude control that destabilizes the orientation of the vehicle. To avoid this, we propose a novel sensor using an optical flow camera with a 6DoF IMU (Inertial Measurement Unit) mounted on a two-axis anti-shake stabilizer mobile aerial gimbal. We also propose a robust algorithm based on Sliding Mode Control for stabilizing the optical flow sensor downwards independently of the aerial vehicle attitude. This method improves the estimation of the position and velocity of the quadrotor. We present experimental results to show the performance of the proposed sensor and algorithms. [ABSTRACT FROM AUTHOR]

Current marker-based facial motion capture methods might lose the target markers in some cases, such as those with considerable occlusion and blur. Manually revising these statuses requires extensive labor-intensive work. Thus, a robust marker tracking method that provides long-term stability must be developed, thereby simplifying manual operations. In this paper, we present a new facial marker tracking system that focuses on the accuracy and stability of performance capture. The tracking system includes a synthetic analysis step with the robust optical flow tracking method and the proposed Marker-YOLO detector. To illustrate the strength of our system, a real dataset of the performance of voluntary actors was obtained, and ground truth labels were given by artists for subsequent experiments. The results showed that our approach outperforms state-of-the-art trackers such as SiamDW and ECO in specific tasks while running at a real-time speed of 38 fps. The root-mean-squared error and area under the curve results verified the improvements in the accuracy and stability of our approach. [ABSTRACT FROM AUTHOR]

Purpose: We employed automated analysis of scanning laser ophthalmoscopy (SLO) to determine if mechanical strains imposed on disc, and retinal and choroidal vessels during horizontal duction in children differ from those of adults. Methods: Thirty-one children aged 11.3 ± 2.7 (standard deviation) years underwent SLO in central gaze, and 35° ab- and adduction. Automated registration with deep learning-based optical flow analysis quantified vessel deformations as horizontal, vertical, shear, and equivalent strains. Choroidal vessel displacements in lightly pigmented fundi, and central disc vessel displacements, were also observed. Results: As in adults, strain in vessels during horizontal duction was greatest at the disc and decreased with distance from it. Strain in the pediatric disc was similar to published values in young adults,1 but in the peripapillary region was greater and propagated significantly more peripherally to at least three disc radii from it. During adduction in children, the nasal disc was compressed and disc vessels distorted, but the temporal half experienced tensile strain, while peripapillary tissues were compressed. The pattern was similar but strains were less in abduction (p <.001). Choroidal vessels were visualized in 24 of the 62 eyes and shifted directionally opposite overlying retinal vessels. Conclusions: Horizontal duction deforms the normal pediatric optic disc, central retinal vessels, peripapillary retina, and choroid, shearing the inner retina over the choroid. These mechanical effects occur at the sites of remodeling of the disc, sclera, and choroid associated with typical adult features that later emerge later, including optic cup enlargement, temporal disc tilting, and peripapillary atrophy. [ABSTRACT FROM AUTHOR]

Optical flow estimation is an important hot research in computer vision. Although existing methods had got a considerable progress in improving their performance, they still have drawbacks, such as heavily computational burden, inaccurate pixel-level offset estimation, and poor interpretability. To address these issues, this letter proposes a pyramidal deep Taylor expansion (PDTE) framework, including: First, we seriously interpreted the relationship between optical flow computation and Taylor expansion. Then, the proposed PDTE is constructed by employing global motion aggregation (GMA) to calculate each derivative part which contributes to the final estimated optical flow. Quantitative and qualitative results on the Sintel and KITTI datasets validate that the proposed PDTE scheme is effective and outperforms the state-of-the-art optical flow estimation methods. The results of extensive experiments in the ablation study demonstrate that PDTE performs well on shape preservation and the accuracy improvement of optical flow estimation, even pixel-level offset calculation. • A novel pyramidal deep Taylor expansion (PDTE) framework for optical flow estimation. • Each derivative part calculated by a global motion aggregation (GMA) method. • Both quantitative and qualitative results verify the effectiveness of our PDTE. [ABSTRACT FROM AUTHOR]

• Design and train 55-layer CNN model on the CIFAR-100 dataset. • Build two-stream network to extract spatial and temporal features from the images. • Feature optimization is performed using a Genetic algorithm. Human Gait Recognition (HGR) is referred to as a biometric tactic that is broadly used for the recognition of an individual by using the pattern of walking. There are some key factors such as angle variation, clothing variation, foot shadows, and carrying conditions that affect the human gait. In this work, a new approach is proposed for the HGR that contains five major steps. In the first step, the video data is converted into image frames. In the second step, RGB to GRAY conversion is carried out. After that, a two-stream network is designed by using a 55-layer CNN model called CNN-55 trained on CIFAR-100. The CNN-55 is designed from scratch and trained on the CIFAR-100 dataset by selecting hyperparameters. This pre-trained CNN-55 is used to build a two-stream network. In Stream-1 the optical flow frames are obtained by Horn and Schunk algorithm. These frames are fed into a CNN-55 to extract temporal features. In Stream-2 the GRAY frames are fed to the CNN-55 model for extraction of spatial features. After that, both vectors are serially fused. In the fourth step, the fused feature vector is fed into the Genetic Algorithm for optimization. Finally, the feature vector is fed into the One-Versus-All SVM classifier for recognition. The system is tested on all CASIA-B angles such as 000, 180, 360, 540, 720, 900, 1080, 1260, 1440, 1620, and 1800 which provides accuracy of 97.10%, 96.80%, 94.60%, 98.0%, 98.30%, 96.80%, 97.60, 96.90%, 99.60%, 96.80%, and 97.60%, respectively. The proposed method produces better outcomes compared to recent techniques. [ABSTRACT FROM AUTHOR]

Recent optical flow estimators usually employ deep models designed for image classification as the encoders for feature extraction and matching. However, those encoders developed for image classification may be sub-optimal for flow estimation. In contrast, the decoder design of optical flow estimators often requires meticulous design for flow estimation. The disconnect between the encoder and decoder could negatively affect optical flow estimation. To address this issue, we propose a neural architecture search method, FlowNAS, to automatically find the more suitable and stronger encoder architecture for existing flow decoders. We first design a suitable search space, including various convolutional operators, and construct a weight-sharing super-network for efficiently evaluating the candidate architectures. To better train the super-network, we present a Feature Alignment Distillation module that utilizes a well-trained flow estimator to guide the training of the super-network. Finally, a resource-constrained evolutionary algorithm is exploited to determine an optimal architecture (i.e., sub-network). Experimental results show that FlowNAS can be easily incorporated into existing flow estimators and achieves state-of-the-art performance with the trade-off between accuracy and efficiency. Furthermore, the encoder architecture discovered by FlowNAS with the weights inherited from the super-network achieves 4.67% F1-all error on KITTI, an 8.4% reduction of RAFT baseline, surpassing state-of-the-art handcrafted GMA and AGFlow models, while reducing the model complexity and latency. The source code and trained models will be released at https://github.com/VDIGPKU/FlowNAS. [ABSTRACT FROM AUTHOR]

Background: X-ray imaging addresses many challenges with visible light imaging in extreme environments, where optical diagnostics such as digital image correlation (DIC) and particle image velocimetry (PIV) suffer biases from index of refraction changes and/or cannot penetrate visibly occluded objects. However, conservation of intensity—the fundamental principle of optical image correlation algorithms—may be violated if ancillary components in the X-ray path besides the surface or fluid of interest move during the test. Objective: The test series treated in this work sought to characterize the safe use of fiber-epoxy composites in aerospace and aviation industries during fire accident scenarios. Stereo X-ray DIC was employed to measure test unit deformation in an extreme thermal environment involving a visibly occluded test unit, incident surface heating to temperatures above 600oC, and flames and soot from combusting epoxy decomposition gasses. The objective of the current work is to evaluate two solutions to resolve the violation of conservation of intensity that resulted from both the test unit and the thermal chamber deforming during the test. Methods: The first solution recovered conservation of intensity by pre-processing the path-integrated X-ray images to isolate the DIC pattern of the test unit from the thermal chamber components. These images were then correlated with standard, optical DIC software. The second solution, called Path-Integrated Digital Image Correlation (PI-DIC), modified the fundamental matching criterion of DIC to account for multiple, independently-moving components contributing to the final image intensity. The PI-DIC algorithm was extended from a 2D framework to a stereo framework and implemented in a custom DIC software. Results: Both solutions accurately measured the cylindrical shape of the undeformed test unit, recovering radii values of R = 76.20 ± 0.12 mm compared to the theoretical radius of R theor = 76.20 mm. During the test, the test unit bulged asymmetrically as decomposition gasses pressurized the interior and eventually burned in a localized jet. Both solutions measured the heterogeneous radius of this bulge, which reached a maximum of approximately R = 91 mm, with a discrepancy of 2–3% between the two solutions. Conclusions: Two solutions that resolve the violation of conservation of intensity for path-integrated X-ray images were developed. Both were successfully employed in a stereo X-ray DIC configuration to measure deformation of an aluminum-skinned, fiber-epoxy composite test unit in a fire accident scenario. [ABSTRACT FROM AUTHOR]

To resolve high-precision deformation from low-quality speckle images, a high-order time-domain digital image correlation algorithm is proposed. This algorithm is based on the nonlinear optical flow equation for low-quality speckle images collected in extreme testing environments. A subpixel displacement image containing Gaussian white noise is established by using the simulated speckle method. The displacement of the high-order time-domain digital image correlation algorithm is solved by the least squares iteration method, and the test accuracy of the proposed algorithm is analyzed. The results show that the nonlinear optical flow equation can describe the time continuity of pixel gray level changes before and after deformation. The proposed algorithm can also analyze the velocity and acceleration term of deformation. The high-order time-domain digital image correlation algorithm is insensitive to the change in image grayscale and has high measurement accuracy. Therefore, this method can solve the problem of resolving high-precision deformation from low-quality speckle images. [ABSTRACT FROM AUTHOR]

This paper proposes a new enterprise intelligentization framework, by making the transition from process transformation to artificial intelligence (AI) transformation. The novel transformation framework can be decomposed into the conceptual model of AI strategic planning, the procedural model, the operational model, and the analytics model. For leading-edge microchip production, a new AI transformation project regarding the reticle SMIF pod (RSP) transport system designed by a medium-sized semiconductor tool vendor in Taiwan is presented. The technical advantages, gained from the implementation of the presented AI transformation project, over the existing RSP systems are manifold. The throughput and yield rate significantly increase on a semiconductor-fabrication-plant basis. The clean room construction costs less by approximately 3 million dollars per FAB, mainly attributed to the redesigned automatic optical inspection flow. The proposed model-based framework proves to be a viable tool from the process transformation to the AI transformation in the semiconductor manufacturing. [ABSTRACT FROM AUTHOR]

Particle image velocimetry (PIV) and optical flow velocimetry (OFV) are important velocity measurement methods in the field of fluid dynamics. Nevertheless, the conventional cross correlation-based PIV method is beset by diminished resolution, while the OFV method exhibits computational sluggishness and susceptibility to noise. These constraints have somewhat delimited the applicability of PIV and OFV techniques. Recent attempts have introduced deep learning-based methods for analyzing PIV images, offering high-resolution velocity fields with computational efficiency, but their accuracy needs improvement. This study proposes four neural networks based on the well-established FlowNetS. They incorporate two distinct velocity constraints, namely, first-order velocity smoothing regularization and second-order grad (curl)–grad (div) regularization. In the networks, these constraints are used either independently or in combination with optical flow conservation (OFC). The performances of the networks have been assessed on six different flow configurations, and the results show that the network with the second-order regularization markedly outperforms the original network across all flows, demonstrating an enhanced capacity to capture larger-scale vortices. The network with the first-order regularization also exhibits superior performance compared to the original network except in the case of cylinder flow. Unexpectedly, the introduction of the OFC constraints results in a decline in network performance. This anomaly may stem from the network's inherent capability to capture optical flow features, rendering the OFC constraint less effective in providing guidance. In summary, this study underscores the substantial potential of neural networks incorporated with judicious physical constraints in PIV applications, enabling the determination of high-resolution, high-accuracy flow fields. [ABSTRACT FROM AUTHOR]

With the goal of solving the problem of feature extractors lacking strong supervision training and insufficient time information concerning single‐sequence model learning, a hierarchical sequence memory network with a multi‐level iterative optimisation strategy is proposed for continuous sign language recognition. This method uses the spatial‐temporal fusion convolution network (STFC‐Net) to extract the spatial‐temporal information of RGB and Optical flow video frames to obtain the multi‐modal visual features of a sign language video. Then, in order to enhance the temporal relationships of visual feature maps, the hierarchical memory sequence network is used to capture local utterance features and global context dependencies across time dimensions to obtain sequence features. Finally, the decoder decodes the final sentence sequence. In order to enhance the feature extractor, the authors adopted a multi‐level iterative optimisation strategy to fine‐tune STFC‐Net and the utterance feature extractor. The experimental results on the RWTH‐Phoenix‐Weather multi‐signer 2014 dataset and the Chinese sign language dataset show the effectiveness and superiority of this method. [ABSTRACT FROM AUTHOR]

Due to the complexity of real optical flow capture, the existing research still has not performed real optical flow capture of infrared (IR) images with the production of an optical flow based on IR images, which makes the research and application of deep learning-based optical flow computation limited to the field of RGB images only. Therefore, in this paper, we propose a method to produce an optical flow dataset of IR images. We utilize the RGB-IR cross-modal image transformation network to rationally transform existing RGB image optical flow datasets. The RGB-IR cross-modal image transformation is based on the improved Pix2Pix implementation, and in the experiments, the network is validated and evaluated using the RGB-IR aligned bimodal dataset M3FD. Then, RGB-IR cross-modal transformation is performed on the existing RGB optical flow dataset KITTI, and the optical flow computation network is trained using the IR images generated by the transformation. Finally, the computational results of the optical flow computation network before and after training are analyzed based on the RGB-IR aligned bimodal data. [ABSTRACT FROM AUTHOR]

Multi-View Stereo (MVS) algorithms remain a significant challenge in reconstructing a 3D model with high completeness due to the difficulty in recovering weakly textured regions and detailed parts of large-scale urban scenes. Although the Image Pyramid Structure is a popular approach for dealing with weakly textured regions, it also leads to the loss of detailed information. The proposed method solves these problems with three new strategies: (1) We propose the optical flow consistency for recovering details. The optical flow consistency improved the sensitivity of the image pyramid structure to details by estimating the motion vector of each pixel point. We proposed a novel detail restorer based on optical flow consistency which improves the link between adjacent scales in the image pyramid structure. (2) Geometric consistency based on epipolar line constraints is proposed to recover weakly textured regions. The proposed epipolar line constraints improve the robustness of traditional geometric consistency, which avoids the problem of mismatching in weakly textured regions. (3) A depth-filling strategy is utilized to fill the loss of depth information of weakly textured regions. Image gradient is utilized to fill the gap of depth information. The filled result is utilized as the priori information to smooth the depth of weakly textured regions. Experimental results on the ETH3D , UDD5 and SenseFly benchmark datasets demonstrate that the proposed method outperforms three state-of-the-art methods (ACMMP, EPNet, DeepC-MVS), significantly improving the completeness of the 3D models. The source code of the develop method is available at https://github.com/Liaoyongjian1/HC-MVS.git. [ABSTRACT FROM AUTHOR]

The interest of science community towards planet Mars has progressively increased in past decades. Rovers are already doing an excellent job in exploring the Martian surface environment and showed that the Red Planet is not suited for life. However, there is a possibility of enabling human exploration by having astronauts take shelter in habitats located within lava tubes, which can provide protection against radiation and dust storms. These tubes are extremely hard to explore with rovers, thus a flying vehicle will be more suitable for such tasks. INGENUITY (Martian helicopter) had already demonstrated that it is possible to fly on Mars by doing small and simple manoeuvres. Based on NASA's design, this article will introduce the development of guidance, navigation, and control operations of a Martian Inspection Drone (MID) that will be capable of finding the lava tube and scan it in real time using a RealSense camera. Linux based system was chosen in the development process, which allow ROS catkin workspace to connect ArduPilot and Gazebo. The navigation is accomplished using a combination of an inertial measurement unit, a camera, and a laser rangefinder. The flight computer runs an optical flow algorithm for control and navigation purpose. Also, a convolutional neural network is used to perform lava tube entrance detection and inspection. • Martian flying drone simulated with ArduPilot, Gazebo and ROS. • Martian environment simulation. • Martian lava tube detection using convolutional neural networks. • Martian lava tube inspection using a realsense camera. [ABSTRACT FROM AUTHOR]

This paper solves the task of unsupervised video object segmentation (UVOS) that segments the objects of interest through the entire videos without any annotation. In recent years, many unsupervised video object segmentation (UVOS) methods have been proposed. Although these methods perform well, they rely on networks with heavy weights, often leading to large model size. In order to reduce the model size while keeping a competitive performance, we propose a saliency-based dual-attention (SDA) method for UVOS in this paper. In our method, we take optical flow and video frames as inputs and extract the appearance information and motion information from optical flow and video frames. We design a two-branch network with appearance information and motion information. The information from these two branches is fused via a saliency-based dual-attention module to segment the primary object in one path. The saliency-based dual-attention module is composed of saliency attention and saliency-based reverse attention. To demonstrate the effectiveness of our network, we tested it on the DAVIS-2016 and SegtrackV2 datasets. Experimental results demonstrate that our method can achieve competitive results in terms of accuracy and model size. [ABSTRACT FROM AUTHOR]

Motion information from image sequences (videos) plays a key role in solving a number of real-world problems such as surveillance, traffic management, fire identification, weather prediction, and COVID-19 detection, etc. Generally, this motion is estimated in terms of optical flow between two consecutive image frames. Optical flow is a 2D vector field that illustrates object motion behavior. This paper presents a nonlinear fractional order variational model for estimating optical flow. The objective of this work is to provide dense and discontinuity preserving optical flow for different spectra and make the model robust against outliers. In particular, the presented model generalizes the integer order variational models for fractional order (0, 1). For this purpose, the proposed model is formulated with the help of Charbonnier norm and Caputo fractional derivative. Charbonnier norm is nonlinear in nature, which makes the model robust against noise and outliers, whereas Caputo fractional derivatives are well-capable to deal with discontinuous functions such as images, and therefore, preserve motion discontinuities. The Caputo derivative also manifests long-term memory effect and allows to choose the optimal value of the fractional order that corresponds to a stable solution. The numerical implementation of the formulated variational functional is performed by discretizing the fractional derivative using the Grünwald–Letnikov derivative. The resulting system of equations is solved using multi-variable fixed point iteration scheme. The entire framework is embedded in coarse-to-fine warping strategy, which helps in finding the global extremal. The experimental results are carried out on 20 different application oriented datasets such as fire and smoke, fluid, CXR, etc. The performance of the model is tested using different error measures and demonstrated against several outliers. Error concentration is shown through 3D histograms. The edge preserving nature is also realized through intensity distribution in RGB color channels. A detailed convergence analysis is also provided for the presented model. The validity of the proposed model is also verified through comparisons with other existing models. [ABSTRACT FROM AUTHOR]

Estimating depth, ego-motion, and optical flow from consecutive frames is a critical task in robot navigation and has received significant attention in recent years. In this study, we propose PDF-Former, an unsupervised joint estimation network comprising a full transformer-based framework, as well as a competition and cooperation mechanism. The transformer framework captures global feature dependencies and is customized for different task types, thereby improving the performance of sequential tasks. The competition and cooperation mechanisms enable the network to obtain additional supervisory information at different training stages. Specifically, the competition mechanism is implemented early in training to achieve iterative optimization of 6 DOF poses (rotation and translation information from the target image to the two reference images), the depth of target image, and optical flow (from the target image to the two reference images) estimation in a competitive manner. In contrast, the cooperation mechanism is implemented later in training to facilitate the transmission of results among the three networks and mutually optimize the estimation results. We conducted experiments on the KITTI dataset, and the results indicate that PDF-Former has significant potential to enhance the accuracy and robustness of sequential tasks in robot navigation. • A transformer-based framework for joint pose, depth, and optical flow estimation is proposed. • The competition–cooperation mechanisms are proposed to promote performance improvement. • Achieved notable accuracy in public datasets. [ABSTRACT FROM AUTHOR]

Video anomaly detection (VAD) task can be addressed as a semi-supervised learning problem as datasets are highly biased towards normal samples. The popular reconstruction methods train the network based on only normal images. These methods detect anomaly events by comparing the input with the reconstructed image, assuming that the network can not accurately reconstruct anomalous regions. However, these methods suffer from the serious flaw that the anomaly regions are generalized sufficiently well. This problem reduces the ability of anomaly detection by narrowing the gap between reconstructed and anomaly input images. By converting the reconstruction process into an inpainting process, we introduce Inpainting-GAN, a distinctive anomaly detection model that utilizes generative adversarial networks (GANs) as the backbone network. Each input image is partially occluded in random way by a series of disjoint masks before being fed into our model, any abnormal areas in the frame will definitely be covered. Our network is trained to restore those masked regions to normal parts, which overcomes the drawback of reconstruction models. To address the temporal continuity in videos, we have incorporated optical flow as a motion constraint during the training of generator. Both qualitative and quantitative experimental results show that our proposed method shows competitive performance in VAD compared to some reconstruction methods. The experimental results on the UCSD Ped1, Ped2 and CUHK Avenue datasets have obtained AUC values of 78.6%, 94.7% and 81.4%. [ABSTRACT FROM AUTHOR]

When considering the temporal relationships, most previous video super-resolution (VSR) methods follow the iterative or recurrent framework. The iterative framework adopts neighboring low-resolution (LR) frames from a sliding window, while the recurrent framework utilizes the output generated in the previous SR procedure. The hybrid framework combines them but still cannot fully leverage the temporal relationships. Meanwhile, the existing methods are limited in the receptive field of the optical flow or lack semantic constrains on motion information. In this work, we propose an omniscient framework to fully explore the temporal relationships in the video, which encompasses both LR frames and SR outputs from the past, present, and future. The omniscient framework is more generic because the iterative, recurrent, and hybrid frameworks can be regarded as its special cases. Besides, when addressing the motion information, most previous VSR methods adopt the explicit motion estimation and compensation, while many recent methods turn to implicit alignment. In implicit alignment methods, because basic non-local means suffers from heavy computational costs, we improve it by capturing the non-local correlations in a relatively local manner to reduce the complexity. Moreover, we integrate the explicit and implicit methods into an explicit-implicit alignment module to better utilize motion information. We have conducted extensive experiments on public datasets, which show that our method is superior over the state-of-the-art methods in objective metrics, subjective visual quality, and complexity. In particular, on datasets of Vid4 and UDM10, our method improves PSNR by 0.19 dB, 0.49 dB against the most advanced method BasicVSR++, respectively. [ABSTRACT FROM AUTHOR]

Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication. [ABSTRACT FROM AUTHOR]

Owing to the physical and economic impacts of urban flooding, effective flood risk management is of crucial importance. Thus, it is essential to employ reliable techniques for monitoring water levels in urban creeks and detecting abrupt fluctuations in weather patterns. Ground-based cameras alongside a creek offer a cost-effective solution, since they can be deployed for determining water levels through image-based analysis. Previous research has examined the benefits of image processing and artificial intelligence techniques to achieve this goal. However, the current methods only analyze static image features and ignore the valuable motion information that may exist in adjacent frames that are captured minutes apart. In addressing this limitation, our approach involves computing dense optical flow fields from consecutive images taken by a stationary camera and integrating these representations into a deep-learning workflow. We evaluated the capacity of both our method and alternative approaches to measure not only the absolute water level (i.e., whether the water height is low, medium, high, or flooding) but also the relative water level (i.e., whether the water level is rising or falling). The results showed that optical flow-based representations significantly improved the ability to measure the relative water level, while pairs of successive grayscale images effectively determined the absolute water level. [ABSTRACT FROM AUTHOR]