Search

Facial expressions (FEs) spotting aims to split long videos into intervals of neutral expression, macro-expression, or micro-expression. Recent works mainly focus on feature descriptor or optical flow methods, suffering from difficulty capturing subtle facial motion and efficient temporal aggregation. This paper proposes a novel end-to-end network, named FESNet (Facial Expression Spotting Network), to solve the above challenges. The main idea is to model the subtle facial motion as local spatial discrepancy and incorporate temporal correlation by multi-scale temporal convolution. The FESNet comprises a local spatial discrepancy module (LSDM) and a multi-scale temporal aggregation module (MTAM). The LSDM first extracts the static spatial features from each frame by residual convolution and learns the inner spatial correlation by multi-head attention. Moreover, the subtle facial motion of facial expression is modeled as the discrepancy between the first frame and the current frame of the input interval, making frame-wise spatial proposals. Using the local spatial discrepancy features and proposals as input, the MTAM incorporates the temporal correlation by multi-scale temporal convolution and performs cascade refinement to make the final prediction. Furthermore, this paper proposes a smooth loss to ensure the temporal consistency of the cascade refined proposals from MTAM. Comprehensive experiments show that FESNet achieves competitive performance compared to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Surveillance video analytics encounters unprecedented challenges in 5G and IoT environments, including complex intra-class variations, short-term and long-term temporal dynamics, and variable video quality. This study introduces Edge-Enhanced TempoFuseNet, a cutting-edge framework that strategically reduces spatial resolution to allow the processing of low-resolution images. A dual upscaling methodology based on bicubic interpolation and an encoder–bank–decoder configuration is used for anomaly classification. The two-stream architecture combines the power of a pre-trained Convolutional Neural Network (CNN) for spatial feature extraction from RGB imagery in the spatial stream, while the temporal stream focuses on learning short-term temporal characteristics, reducing the computational burden of optical flow. To analyze long-term temporal patterns, the extracted features from both streams are combined and routed through a Gated Recurrent Unit (GRU) layer. The proposed framework (TempoFuseNet) outperforms the encoder–bank–decoder model in terms of performance metrics, achieving a multiclass macro average accuracy of 92.28%, an F1-score of 69.29%, and a false positive rate of 4.41%. This study presents a significant advancement in the field of video anomaly recognition and provides a comprehensive solution to the complex challenges posed by real-world surveillance scenarios in the context of 5G and IoT. [ABSTRACT FROM AUTHOR]

Virtual reality (VR) locomotion should allow users to move freely in the virtual space while staying within the tracking area in the real space. The redirected walking (RDW) technique enables users to walk naturally in an unlimited virtual space within a limited tracking area by rotating the virtual scene view. However, conflicting visual and vestibular signals during RDW can lead to user discomfort and decreased immersion. To avoid user discomfort, an RDW gain should be within the detection threshold (DT) range. However, a large angle of walking redirection is required when physically avoiding obstacles or escaping from a narrow space, so DT expansion is necessary. In this study, to change the curvature DT range and enhance RDW performance, we proposed an optical flow (OF)-generating vection in a virtual environment. Further, we investigate methods to reduce user discomfort and increase RDW efficiency considering familiar and unfamiliar VR users. The findings showed that the introduction of OF led to a reduction in the DT range for all users, irrespective of the OF’s direction. However, conditions with OF resulted in an extended DT range for users familiar with VR while concurrently diminishing the DT range for those who were VR unfamiliar. To delve further, our analysis indicated that when both the OF and redirecting directions were identical, the RDW performance was robust to VR familiarity, whereas in opposing directions, the DT range increased for VR-familiar users. Our study findings suggested using OF for the RDW technique and extending its applicability in virtual environments. [ABSTRACT FROM AUTHOR]

Optical flow is the apparent motion of the brightness patterns in an image. The pyramidal form of the Lucas-Kanade (LK) method is frequently used for its computation but experiments have shown that the method has deficiencies. Problems arise because of numerical issues in the least squares (LS) problem min A x − b 2 2 , A ∈ R m × 2 and m ≫ 2 , which must be solved many times. Numerical properties of the solution x 0 = A † b = (A T A) − 1 A T b of the LS problem are considered and it is shown that the property m ≫ 2 has implications for the error and stability of x 0 . In particular, it can be assumed that b has components that lie in the column space (range) R (A) of A, and the space that is orthogonal to R (A) , from which it follows that the upper bound of the condition number of x 0 is inversely proportional to cos θ , where θ is the angle between b and its component that lies in R (A) . It is shown that the maximum values of this condition number, other condition numbers and the errors in the solutions of the LS problems increase as the pyramid is descended from the top level (coarsest image) to the base (finest image), such that the optical flow computed at the base of the pyramid may be computationally unreliable. The extension of these results to the problem of total least squares is addressed by considering the stability of the optical flow vectors when there are errors in A and b. Examples of the computation of the optical flow demonstrate the theoretical results, and the implications of these results for extended forms of the LK method are discussed. [ABSTRACT FROM AUTHOR]

Optical identification is often done with spatial or temporal visual pattern recognition and localization. Temporal pattern recognition, depending on the technology, involves a trade-off between communication frequency, range, and accurate tracking. We propose a solution with light-emitting beacons that improves this trade-off by exploiting fast event-based cameras and, for tracking, sparse neuromorphic optical flow computed with spiking neurons. The system is embedded in a simulated drone and evaluated in an asset monitoring use case. It is robust to relative movements and enables simultaneous communication with, and tracking of, multiple moving beacons. Finally, in a hardware lab prototype, we demonstrate for the first time beacon tracking performed simultaneously with state-of-the-art frequency communication in the kHz range. [ABSTRACT FROM AUTHOR]

In this paper, we address the challenge of detecting small moving targets in dynamic environments characterized by the concurrent movement of both platform and sensor. In such cases, simple image-based frame registration and optical flow analysis cannot be used to detect moving targets. To tackle this, it is necessary to use sensor and platform meta-data in addition to image analysis for temporal and spatial anomaly detection. To this end, we investigate techniques that utilize inertial data to enhance frame-to-frame registration, consistently yielding improved detection outcomes when compared against purely feature-based techniques. For cases where image registration is not possible even with metadata, we propose single-frame spatial anomaly detection and then estimate the range to the target using the platform velocity. The behavior of the estimated range over time helps us to discern targets from clutter. Finally, we show that a KNN classifier can be used to further reduce the false alarm rate without a significant reduction in detection performance. The proposed strategies offer a robust solution for the detection of moving targets in dynamically challenging settings. [ABSTRACT FROM AUTHOR]

The spatter generated by the interaction between laser and powder during Powder Bed Fusion-Laser Melting (PBF-LM) can significantly affect the quality of printed parts. A high-speed camera is used to observe the dynamic process of spatter's behavior under different layer thickness and laser powers during the printing process, and to analyze the printed samples' surface roughness, microstructure, and mechanical properties. In terms of spatter image processing, employing an optical flow approach to track and quantify the number of spatters efficiently eliminates statistical redundancy and improves statistical correctness. It is found that under the same laser power, the number of spatters produced by the laser scan direction with the gas flow (LSD-W) is more than that by the laser scan direction against the gas flow (LSD-A), and the number of spatters produced increases with the increase of laser power. Analyzing the mechanical properties and surface roughness of the printed samples under different process parameters quantitatively reveals that differences in the spatter amount generated under different process parameters in the PBF-LM process is not the determining factor affecting the difference in tensile strength of printed parts. During LSD-W, the number of spatters generated at laser power of 170 W and layer thickness of 0.03 mm is 87, and the tensile strength of the printed sample is 618 MPa. During LSD-W, the number of spatters generated at laser power of 320 W and layer thickness of 0.05 mm is 211, and the tensile strength of the printed sample is 680 MPa. Instead, spatter generation has a more direct impact on the surface roughness of printed parts. The layer thickness is 0.03 mm, the laser power is 170 W, and (Ra = 2.372 μm) is the surface roughness of the sample. The layer thickness is 0.05 mm, the laser power is 320 W, and (Ra = 8.163 μm) is the surface roughness of the sample. [ABSTRACT FROM AUTHOR]

The critical goal of gait recognition is to acquire the inter-frame walking habit representation from the gait sequences. The relations between frames, however, have not received adequate attention in comparison to the intra-frame features. In this paper, motivated by optical flow, the bilateral motion-oriented block is proposed to explore motion description at the feature level. It can allow the classic 2D convolutional structure to have the capability to directly portray gait movement patterns while preventing costly computations on the estimation of optical flow. Based on such features, we develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution. Furthermore, the dynamic information is sensitive to inaccurate segmentation on the edge, so a correction block is devised to eliminate the segmentation noise of silhouettes for getting more precise gait modality. Subsequently, the temporal feature set and the spatial features are combined to comprehensively characterize gait processes. Extensive experiments are conducted on CASIA-B and OU-MVLP datasets, and the results achieve an outstanding identification performance, which has demonstrated the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

To accurately describe the thermal comfort of indoor occupant used to regulate heating, ventilation and air conditioning (HVAC) system is a key goal to reduce energy consumption in intelligent buildings. In this paper, we propose a noncontact measurement of occupant thermal discomfort behavior as an index of thermal comfort. The method takes the existing monitoring image data in the building as the input to infer the occupant thermal discomfort directly, which saves the cost because there is no need to install new sensors in the building and the occupant does not need to wear additional equipment. The framework combined three channels of body posture, motion and performance information to infer occupant thermal discomfort behavior, it consists of a human detection and crop module, a posture analysis module, an optical flow extraction module and a 3D convolutional neural network module. The three channels describe the actions from different perspectives, and the proposed method makes full use of the complementarity of the three modalities to identify the occupant's thermal discomfort behaviors. Sixteen postures related to thermal discomfort were identified through a questionnaire, and 14,800 video clips containing these postures were collected for experimental evaluation. The results demonstrate the superior performances of our approach to the state-of-the-art techniques. The framework achieves noninvasive, cost-effective thermal comfort evaluation, and has potential value in improving energy efficiency of HVAC systems. [ABSTRACT FROM AUTHOR]

North East Monsoon (NEM) is the major source of rainfall for the south-eastern parts of peninsular India. Short time rainfall prediction data (i.e., nowcasting) are based on the observations from Doppler weather radars which has a high spatial and temporal resolution. This study focuses on the short-term ensemble prediction system using weather radar data to predict precipitation during the NEM and is the first of its kind in the Indian region to make an assessment of the operational performance of the prediction system. Six rainfall events have been studied for the assessment of short-term prediction system where the precipitation systems are different and include a tropical storm observed over different days during the 2022 NEM season. To assess the performance of the system, Fractional Skill Scores (FSS) at a 1 km window have been computed for a lead time of 0–2 h for all the rainfall events with more than 750 samples using different optical flow methods and ensemble sizes. The best average skill score and maximum skill score obtained at a 2 h lead time is 0.65 and 0.78 for tropical storms, 0.5 and 0.78 for stratiform and 0.15 and 0.38 for convective precipitation. It has found that the performance of the model is best for precipitation systems that are widespread and have a longer life period. [ABSTRACT FROM AUTHOR]

In this paper, we focus on exploring effective methods for faster and accurate semantic segmentation. A common practice to improve the performance is to attain high-resolution feature maps with strong semantic representation. Two strategies are widely used: atrous convolutions and feature pyramid fusion, while both are either computationally intensive or ineffective. Inspired by the Optical Flow for motion alignment between adjacent video frames, we propose a Flow Alignment Module (FAM) to learn Semantic Flow between feature maps of adjacent levels and broadcast high-level features to high-resolution features effectively and efficiently. Furthermore, integrating our FAM to a standard feature pyramid structure exhibits superior performance over other real-time methods, even on lightweight backbone networks, such as ResNet-18 and DFNet. Then to further speed up the inference procedure, we also present a novel Gated Dual Flow Alignment Module to directly align high-resolution feature maps and low-resolution feature maps where we term the improved version network as SFNet-Lite. Extensive experiments are conducted on several challenging datasets, where results show the effectiveness of both SFNet and SFNet-Lite. In particular, when using Cityscapes test set, the SFNet-Lite series achieve 80.1 mIoU while running at 60 FPS using ResNet-18 backbone and 78.8 mIoU while running at 120 FPS using STDC backbone on RTX-3090. Moreover, we unify four challenging driving datasets (i.e., Cityscapes, Mapillary, IDD, and BDD) into one large dataset, which we named Unified Driving Segmentation (UDS) dataset. It contains diverse domain and style information. We benchmark several representative works on UDS. Both SFNet and SFNet-Lite still achieve the best speed and accuracy trade-off on UDS, which serves as a strong baseline in such a challenging setting. The code and models are publicly available at https://github.com/lxtGH/SFSegNets. [ABSTRACT FROM AUTHOR]

In this research, we explore the latest advancements in the domains of vehicle detection, recognition, and tracking through vision-based methodologies. Spanning from traditional techniques to cutting-edge deep learning models, the paper provides a thorough analysis of their strengths, limitations, and real-world applications. We examine the evolution of detection methods, including edge-based approaches and motion segmentation techniques like frame differencing, background subtraction, and optical flow, showcasing their efficacy in identifying moving vehicles. Convolutional neural networks (CNNs), in particular, are highlighted for the integration of appearance-based approaches due to their transformative impact on accuracy, particularly in difficult settings. In addition, we explore vehicle recognition and classification methods, highlighting their applications in traffic analysis, security, and urban planning. These methods include color recognition, license plate recognition, logo recognition, and vehicle type classification. In addition to comparing tracking algorithms based on accuracy, computing economy, and adaptability, the evaluation includes methodologies for tracking vehicles that include model-based, shape-based, and feature-based approaches. In conclusion, this research provides a thorough examination of the development of vision-based vehicle analysis, making it a useful tool for academics, professionals, and decision-makers working to improve traffic surveillance systems and plan for the future of urban transportation. [ABSTRACT FROM AUTHOR]

Although supervised approaches for traffic sign classification have demonstrated excellent performance, they are limited to classifying several traffic signs defined in the training dataset. This prevents them from being applied to different domains, i.e., different countries. Herein, we propose a self-supervised approach for few-shot learning-based traffic sign classification. A center-awareness similarity network is designed for the traffic sign problem and trained using an optical flow dataset. Unlike existing supervised traffic sign classification methods, the proposed method does not depend on traffic sign categories defined by the training dataset. It applies to any traffic signs from different countries. We construct a Korean traffic sign classification (KTSC) dataset, including 6000 traffic sign samples and 59 categories. We evaluate the proposed method with baseline methods using the KTSC, German traffic sign, and Belgian traffic sign classification datasets. Experimental results show that the proposed method extends the ability of existing supervised methods and can classify any traffic sign, regardless of region/country dependence. Furthermore, the proposed approach significantly outperforms baseline methods for patch similarity. This approach provides a flexible and robust solution for classifying traffic signs, allowing for accurate categorization of every traffic sign, regardless of regional or national differences. [ABSTRACT FROM AUTHOR]

With the rapid development in technology in recent years, the use of cameras and the production of video and image data have similarly increased. Therefore, there is a great need to develop and improve video surveillance techniques to their maximum extent, particularly in terms of their speed, performance, and resource utilization. It is challenging to accurately detect anomalies and increase the performance by minimizing false positives, especially in crowded and dynamic areas. Therefore, this study proposes a hybrid video anomaly detection model combining multiple machine learning algorithms with pixel-based video anomaly detection (PBVAD) and frame-based video anomaly detection (FBVAD) models. In the PBVAD model, the motion influence map (MIM) algorithm based on spatio–temporal (ST) factors is used, while in the FBVAD model, the k-nearest neighbors (kNN) and support vector machine (SVM) machine learning algorithms are used in a hybrid manner. An important result of our study is the high-performance anomaly detection achieved using the proposed hybrid algorithms on the UCF-Crime data set, which contains 128 h of original real-world video data and has not been extensively studied before. The AUC performance metrics obtained using our FBVAD-kNN algorithm in experiments were averaged to 98.0%. Meanwhile, the success rates obtained using our PBVAD-MIM algorithm in the experiments were averaged to 80.7%. Our study contributes significantly to the prevention of possible harm by detecting anomalies in video data in a near real-time manner. [ABSTRACT FROM AUTHOR]

The Maritime Continent (MC) regularly experiences powerful convective storms that produce intense rainfall, flooding and landslides, which numerical weather prediction models struggle to forecast. Nowcasting uses observations to make more accurate predictions of convective activity over short timescales (∼ 0–6 h). Optical flow algorithms are effective nowcasting methods as they are able to accurately track clouds across observed image series and predict forward trajectories. Optical flow is generally applied to weather radar observations; however, the radar coverage network over the MC is not complete and the signal cannot penetrate the high mountainous regions. In this research, we apply optical flow algorithms from the pySTEPS nowcasting library to satellite imagery to generate both deterministic and probabilistic nowcasts over the MC. The deterministic algorithm shows skill up to 4 h on spatial scales of 10 km and coarser and outperforms a persistence nowcast for all lead times. Lowest skill is observed over the mountainous regions during the early afternoon, and highest skill is seen during the night over the sea. A key feature of the probabilistic algorithm is its attempt to reduce uncertainty in the lifetime of small-scale convection. Composite analysis of 3 h lead time nowcasts, initialised in the morning and afternoon, produces reliable ensembles but with an under-dispersive distribution and produces area under the curve scores (i.e. ratio of hit rate to false alarm rate across all probability thresholds) of 0.80 and 0.71 over the sea and land, respectively. When directly comparing the two approaches, the probabilistic nowcast shows greater skill at ≤ 60 km spatial scales, whereas the deterministic nowcast shows greater skill at larger spatial scales ∼ 200 km. Overall, the results show promise for the use of pySTEPS and satellite retrievals as an operational nowcasting tool over the MC. [ABSTRACT FROM AUTHOR]

Although the excitation-time imaging condition offers a lower memory consumption and higher computational efficiency compared to cross-correlation imaging condition, it has not been widely used in industrial applications because of the accuracy problem of traveltime calculation and the influence of low-wave-number noise. In this paper, we introduce the physics-informed neural network (PINN) algorithm to achieve a high-precision traveltime calculation of the source forward wavefield. Subsequently, we introduce a technique for high-precision wavefield decomposition of the reverse-time wavefield via the optical flow vector, enabling us to realize a correlation-weighted stacking imaging of each wavefield. Model experiments and real data processing show that the proposed traveltime calculation algorithm based on PINN offers high accuracy and good applicability in the excitation-time reverse-time migration imaging of complex models, and correlation-weighted stacking imaging based on optical flow vector-based wavefield separation can significantly suppress the noise with low wave-number and achieve high-precision imaging of complex models. [ABSTRACT FROM AUTHOR]

In this article, we address the problem of estimating fluid flows between two adjacent images containing fluid and non-fluid objects. Typically, traditional optical flow estimation methods lack accuracy, because of the highly deformable nature of fluid, the lack of definitive features, and the motion differences between fluid and non-fluid objects. Our approach captures fluid motions using an affine motion model for each small patch of an image. To obtain robust patch matches, we propose a best-buddies similarity-based method to address the lack of definitive features but many similar features in fluid phenomena. A dense set of affine motion models was then obtained by performing nearest-neighbor interpolation. Finally, dense fluid flow was recovered by applying the affine transformation to each patch and was improved by minimizing a variational energy function. Our method was validated using different types of fluid images. Experimental results show that the proposed method achieves the best performance. [ABSTRACT FROM AUTHOR]

We propose a deep multi-stream model for left ventricular ejection fraction (LVEF) prediction in 2D echocardiographic (2DE) examinations. We use four standard 2DE views as model input, which are automatically selected from the full 2DE examination. The LVEF prediction model processes eight streams of data (images + optical flow) and consists of convolutional neural networks terminated with transformer layers. The model is made robust to missing, misclassified and duplicate views via pre-training, sampling strategies and parameter sharing. The model is trained and evaluated on an existing clinical dataset (12,648 unique examinations) with varying properties in terms of quality, examining physician, and ultrasound system. We report R 2 = 0.84 and mean absolute error = 4.0% points for the test set. When evaluated on two public benchmarks, the model performs on par or better than all previous attempts on fully automatic LVEF prediction. Code and trained models are available on a public project repository. [ABSTRACT FROM AUTHOR]

Automated assessment of broiler chicken welfare poses particular problems due to the large numbers of birds involved and the variety of different welfare measures that have been proposed. Active (sustained, defect-free) walking is both a universally agreed measure of bird health and a behavior that can be recognized by existing technology. This makes active walking an ideal starting point for automated assessment of chicken welfare at both individual and flock level. [ABSTRACT FROM AUTHOR]

Detecting violent behavior in videos to ensure public safety and security poses a significant challenge. Precisely identifying and categorizing instances of violence in real-life closed-circuit television, which vary across specifications and locations, requires comprehensive understanding and processing of the sequential information embedded in these videos. This study aims to introduce a model that adeptly grasps the spatiotemporal context of videos within diverse settings and specifications of violent scenarios. We propose a method to accurately capture spatiotemporal features linked to violent behaviors using optical flow and RGB data. The approach leverages a Conv3D-based ResNet-3D model as the foundational network, capable of handling high-dimensional video data. The efficiency and accuracy of violence detection are enhanced by integrating an attention mechanism, which assigns greater weight to the most crucial frames within the RGB and optical-flow sequences during instances of violence. Our model was evaluated on the UBI-Fight, Hockey, Crowd, and Movie-Fights datasets; the proposed method outperformed existing state-of-the-art techniques, achieving area under the curve scores of 95.4, 98.1, 94.5, and 100.0 on the respective datasets. Moreover, this research not only has the potential to be applied in real-time surveillance systems but also promises to contribute to a broader spectrum of research in video analysis and understanding. [ABSTRACT FROM AUTHOR]

Satellite time-series data contain information in three dimensions—spatial, spectral, and temporal—and are widely used for monitoring, simulating, and evaluating Earth activities. However, some time-phase images in the satellite time series data are missing due to satellite sensor malfunction or adverse atmospheric conditions, which prevents the effective use of the data. Therefore, we need to complement the satellite time series data with sequence image interpolation. Linear interpolation methods and deep learning methods that have been applied to sequence image interpolation lead to large errors between the interpolation results and the real images due to the lack of accurate estimation of pixel positions and the capture of changes in objects. Inspired by video frame interpolation, we combine optical flow estimation and deep learning and propose a method named Multi-Scale Optical Flow-Intermediate Feature Joint Network. This method learns pixel occlusion and detailed compensation information for each channel and jointly refines optical flow and intermediate features at different scales through an end-to-end network together. In addition, we set a spectral loss function to optimize the network's learning of the spectral features of satellite images. We have created a time-series dataset using Landsat-8 satellite data and Sentinel-2 satellite data and then conducted experiments on this dataset. Through visual and quantitative evaluation of the experimental results, we discovered that the interpolation results of our method retain better spectral and spatial consistency with the real images, and that the results of our method on the test dataset have a 7.54% lower Root Mean Square Error than other approaches. [ABSTRACT FROM AUTHOR]

This work presents a novel RGB-D dynamic Simultaneous Localisation and Mapping (SLAM) method that improves the precision, stability, and efficiency of localisation while relying on lightweight deep learning in a dynamic environment compared to the traditional static feature-based visual SLAM algorithm. Based on ORB-SLAM3, the GCNv2-tiny network instead of the ORB method, improves the reliability of feature extraction and matching and the accuracy of position estimation; then, the semantic segmentation thread employs the lightweight YOLOv5s object detection algorithm based on the GSConv network combined with a depth image to determine potentially dynamic regions of the image. Finally, to guarantee that the static feature points are used for position estimation, dynamic probability is employed to determine the true dynamic feature points based on the optical flow, semantic labels, and the state in last frame. We have performed experiments on the TUM datasets to verify the feasibility of the algorithm. Compared with the classical dynamic visual SLAM algorithm, the experimental results demonstrate that the absolute trajectory error is greatly reduced in dynamic environments, and that the computing efficiency is improved by 31.54% compared with the real-time dynamic visual SLAM algorithm with close accuracy, demonstrating the superiority of DLD-SLAM in accuracy, stability, and efficiency. [ABSTRACT FROM AUTHOR]

Classifying sports videos is complex due to their dynamic nature. Traditional methods, like optical flow and the Histogram of Oriented Gradient (HOG), are limited by their need for expertise and lack of universality. Deep learning, particularly Convolutional Neural Networks (CNNs), offers more effective feature recognition in sports videos, but standard CNNs struggle with fast-paced or low-resolution sports videos. Our novel neural network model addresses these challenges. It begins by selecting important frames from sports footage and applying a fuzzy noise reduction algorithm to enhance video quality. The model then uses a bifurcated neural network to extract detailed features, leading to a densely connected neural network with a specific activation function for categorizing videos. We tested our model on a High-Definition Sports Video Dataset covering over 20 sports and a low-resolution dataset. Our model outperformed established classifiers like DenseNet, VggNet, Inception v3, and ResNet-50. It achieved high precision (0.9718), accuracy (0.9804), F-score (0.9761), and recall (0.9723) on the high-resolution dataset, and significantly better precision (0.8725) on the low-resolution dataset. Correspondingly, the highest values on the matrix of four traditional models are: precision (0.9690), accuracy (0.9781), F-score (0.9670), recall (0.9681) on the high-resolution dataset, and precision (0.8627) on the low-resolution dataset. This demonstrates our model's superior performance in sports video classification under various conditions, including rapid motion and low resolution. It marks a significant step forward in sports data analytics and content categorization. [ABSTRACT FROM AUTHOR]

We investigated the assessment of blood flow within polypoidal lesions using optical coherence tomography angiography (OCTA) to determine intravitreal brolucizumab (IVBr) efficacy for treating polypoidal choroidal vasculopathy (PCV). We retrospectively studied 46 eyes with PCV that completed 1-year IVBr treatment. Blood flow signals within polypoidal lesions were evaluated using OCTA after loading-phase treatment, and 1-year outcomes were compared between eyes in which blood flow signals disappeared versus persisting. After loading-phase treatment, blood flow signals within polypoidal lesions disappeared in 31 eyes and persisted in 15. In the former group, visual acuity improved significantly throughout the year (P < 0.01), while in the latter there was no significant difference between baseline and after 1 year. The total number of injections was significantly lower with than without disappearance of blood flow signals (6.0 vs. 6.9, P < 0.01). The intended injection interval at the last visit was significantly longer in the former than in the latter group (15.7 weeks vs. 12.5 weeks, P < 0.01). These results indicate that PCV cases showing disappearance of blood flow signals within polypoidal lesions by OCTA after loading-phase treatment had favorable 1-year outcomes of IVBr. Therefore, evaluating blood flow within polypoidal lesions by OCTA may allow noninvasive prediction of PCV treatment outcomes. [ABSTRACT FROM AUTHOR]

The ability to predict the future based on past experience lies at the core of the brain's ability to adapt behavior. However, the neural mechanisms that participate in generating and updating predictions are not clearly understood. Further, the evolutionary antecedents and the prevalence of predictive processing among vertebrates are even less explored. Here, we show evidence of predictive processing via the involvement of cerebellar circuits in larval zebrafish. We presented stereotyped optic flow stimuli to larval zebrafish to evoke swims and discovered that lesioning the cerebellum abolished prediction-dependent modulation of swim latency. When expectations of optic flow direction did not match with reality, error signals arrive at Purkinje cells via the olivary climbing fibers, whereas granule cells and Purkinje cells encode signals of expectation. Strong neural representations of expectation correlate with faster swim responses and vice versa. In sum, our results show evidence for predictive processing in nonmammalian vertebrates with the involvement of cerebellum, an evolutionarily conserved brain structure. [ABSTRACT FROM AUTHOR]

A space-time variational method is developed for retrieving upper-level vortex winds from geostationary satellite rapid infrared scans over hurricanes. In this method, new vortex-flow-dependent correlation functions are formulated for the radial and tangential components of the vortex wind. These correlation functions are used to construct the background error covariance matrix and its square root matrix. The resulting square root matrix is then employed to precondition the cost function, constrained by an advection equation formulated for rapidly scanned infrared image movements. This newly formulated and preconditioned cost function is more suitable for deriving upper-level vortex winds from GOES-16 rapid infrared scans over hurricanes than the cost function in the recently adopted optical flow technique. The new method was applied to band-13 (10.3 µm) brightness temperature images scanned every min from GOES-16 over Hurricanes Laura on 27 August 2020 and Hurricanes Ida on 29 August 2021. The retrieved vortex winds were shown to not only be much denser than operationally produced atmospheric motion vectors (AMVs) but also more rotational and better organized around the eyewall than the super-high-resolution AMVs derived from optical-flow technique. By comparing their component velocities (projected along radar beams) with limited radar velocity observations available near the cloud top, the vortex winds retrieved using the new method were also shown to be more accurate than the super-high-resolution AMVs derived from the optical-flow technique. The new method is computationally efficient for real-time applications and potentially useful for hurricane wind nowcasts. Furthermore, the combined use of VF-dependent covariance functions and imagery advection equation is not only novel but was also found to be critically important for the improved performance of the method. This finding implies that similar combined approaches can be developed with improved performance for retrieving vortex flows rapidly scanned using other types of remote sensing on different scales, such as tornadic mesocyclones rapidly scanned by phased-array radars. [ABSTRACT FROM AUTHOR]

The ability to perceive biological motion is crucial for human survival, social interactions, and communication. Over the years, researchers have studied the mechanisms and neurobiological substrates that enable this ability. In a previous study, we proposed a descriptive Bayesian simulation model to represent the dorsal pathway of the visual system, which processes motion information. The model was inspired by recent studies that questioned the impact of dynamic form cues in biological motion perception and was trained to distinguish the direction of a soccer ball from a set of complex biological motion soccer-kick stimuli. However, the model was unable to simulate the reaction times of the athletes in a credible manner, and a few subjects could not be simulated. In this current work, we implemented a novel disremembering strategy to incorporate neural adaptation at the decision-making level, which improved the model's ability to simulate the athletes' reaction times. We also introduced receptive fields to detect rotational optic flow patterns not considered in the previous model to simulate a new subject and improve the correlation between the simulation and experimental data. The findings suggest that rotational optic flow plays a critical role in the decision-making process and sheds light on how different individuals perform at different levels. The correlation analysis of human versus simulation data shows a significant, almost perfect correlation between experimental and simulated angular thresholds and slopes, respectively. The analysis also reveals a strong relation between the average reaction times of the athletes and the simulations. [ABSTRACT FROM AUTHOR]

The ability to detect and assess world‐relative object‐motion is a critical computation performed by the visual system. This computation, however, is greatly complicated by the observer's movements, which generate a global pattern of motion on the observer's retina. How the visual system implements this computation is poorly understood. Since we are potentially able to detect a moving object if its motion differs in velocity (or direction) from the expected optic flow generated by our own motion, here we manipulated the relative motion velocity between the observer and the object within a stationary scene as a strategy to test how the brain accomplishes object‐motion detection. Specifically, we tested the neural sensitivity of brain regions that are known to respond to egomotion‐compatible visual motion (i.e., egomotion areas: cingulate sulcus visual area, posterior cingulate sulcus area, posterior insular cortex [PIC], V6+, V3A, IPSmot/VIP, and MT+) to a combination of different velocities of visually induced translational self‐ and object‐motion within a virtual scene while participants were instructed to detect object‐motion. To this aim, we combined individual surface‐based brain mapping, task‐evoked activity by functional magnetic resonance imaging, and parametric and representational similarity analyses. We found that all the egomotion regions (except area PIC) responded to all the possible combinations of self‐ and object‐motion and were modulated by the self‐motion velocity. Interestingly, we found that, among all the egomotion areas, only MT+, V6+, and V3A were further modulated by object‐motion velocities, hence reflecting their possible role in discriminating between distinct velocities of self‐ and object‐motion. We suggest that these egomotion regions may be involved in the complex computation required for detecting scene‐relative object‐motion during self‐motion. [ABSTRACT FROM AUTHOR]

Industrial activities such as metallurgy, coal and oil combustion, cement production, and petrochemistry release aerosol particles into the atmosphere. We propose analyzing the aerosol composition of plumes emitted by different industrial stacks using PRISMA (PRecursore IperSpettrale della Missione Applicativa) satellite hyperspectral observations. Three industrial sites have been observed: a coal-fired power plant in Matla, South Africa (imaged on 25 September 2021); a steel plant in Wuhan, China (24 March 2021); and gas flaring at an oil extraction site in Hassi Messaoud, Algeria (9 July 2021). Below-plume surface reflectances are constrained using a combination of PRISMA and Sentinel-2/MSI images. Radiative transfer simulations are performed for each scene including the surface, background atmosphere, and plume optical properties. The plume aerosol optical thickness (AOT), particle radius, volume of coarse-mode aerosol, and soot are then retrieved within the plumes following an optimal estimation framework. The mean plume retrieved AOT at 500 nm ranges between 0.27 and 1.27 and the median radius between 0.10 and 0.12 µ m. We found a volume fraction of soot of 3.6 % and 10.4 % in the sinter plant and coal-fired plant plumes, respectively. The mass flow rate of particulate matter at a point source estimated by an integrated mass enhancement method varies from 840 ± 155 g s -1 for the flaring emission to 1348 ± 570 g s -1 at the coal-fired plant. [ABSTRACT FROM AUTHOR]

Video anomaly detection by reconstruction is a challenging task. One of its challenges is related to the volume of input data frames needed to be processed to detect anomalies. The challenge usually manifests itself as increased training and especially testing time. The proposed architecture boosts performance while maintaining the same test time as our previously introduced AnoDetNet architecture. The proposed architecture is a cascaded framework that is a succession of reconstruction and an auxiliary network. Upon training, the auxiliary network acts as guidance through the use of combined loss. The combined training of the networks results in a performance increase compared with the reconstruction case alone. Considering that the auxiliary network's results are not used in the test phase, the overall anomaly detection test time does not change compared with the non-cascaded architecture. Two possible auxiliary networks, namely edge detection and optical flow estimation are studied. The proposed architecture results in state-of-the-art results on the Ped2 and Avenue datasets. [ABSTRACT FROM AUTHOR]

A good understanding of geometrical concepts as well as a broad familiarity with objects lead to excellent human perception of moving objects. The human ability to detect and segment moving objects works in the presence of multiple objects, complex background geometry, motion of the observer and even camouflage. How we perceive moving objects so reliably is a longstanding research question in computer vision and borrows findings from related areas such as psychology, cognitive science and physics. One approach to the problem is to teach a deep network to model all of these effects. This is in contrast with the strategy used by human vision, where cognitive processes and body design are tightly coupled and each is responsible for certain aspects of correctly identifying moving objects. Similarly, from the computer vision perspective there is evidence that classical, geometry-based techniques are better suited to the "motion-based" parts of the problem, while deep networks are more suitable for modeling appearance. In this work, we argue that the coupling of camera rotation and camera translation can create complex motion fields that are difficult for a deep network to untangle directly. We present a novel probabilistic model to estimate the camera's rotation given the motion field. We then rectify the flow field to obtain a rotation-compensated motion field for subsequent segmentation. This strategy of first estimating camera motion, and then allowing a network to learn the remaining parts of the problem, yields improved results on the widely used DAVIS benchmark as well as the more recent motion segmentation data set MoCA (Moving Camouflaged Animals). [ABSTRACT FROM AUTHOR]

Forecasting heavy precipitation accurately is a challenging task for most deep learning (DL)-based models. To address this, we present a novel DL architecture called "multi-scale feature fusion" (MFF) that can forecast precipitation with a lead time of up to 3 h. The MFF model uses convolution kernels with varying sizes to create multi-scale receptive fields. This helps to capture the movement features of precipitation systems, such as their shape, movement direction, and speed. Additionally, the architecture utilizes the mechanism of discrete probability to reduce uncertainties and forecast errors, enabling it to predict heavy precipitation even at longer lead times. For model training, we use 4 years of radar echo data from 2018 to 2021 and 1 year of data from 2022 for model testing. We compare the MFF model with three existing extrapolative models: time series residual convolution (TSRC), optical flow (OF), and UNet. The results show that MFF achieves superior forecast skills with high probability of detection (POD), low false alarm rate (FAR), small mean absolute error (MAE), and high structural similarity index (SSIM). Notably, MFF can predict high-intensity precipitation fields at 3 h lead time, while the other three models cannot. Furthermore, MFF shows improvement in the smoothing effect of the forecast field, as observed from the results of radially averaged power spectral (RAPS). Our future work will focus on incorporating multi-source meteorological variables, making structural adjustments to the network, and combining them with numerical models to further improve the forecast skills of heavy precipitations at longer lead times. [ABSTRACT FROM AUTHOR]

To solve the tracking accuracy degradation problem in scenarios with large displacements or nonrigid motion during target tracking, this paper proposes an optical flow estimation method based on multiscale anisotropic convolution. The network structure is improved in a step-by-step manner by extracting the data flow from the network according to the observed features. For the low-level neural network, a layered multiscale structure is used to build a cascade network by using hybrid dilated convolution to obtain feature information at different scales while ensuring the tracking accuracy. For the upper-layer neural network, hybrid inflated deformable convolution is used by learning the contextual long-range correlations and multidirectional adaptive offsets of features. Experiments are conducted on the Flying Chairs, KITTI, and MPI datasets. The results show that compared with various popular algorithm methods, the model in this paper reduces endpoint errors while retaining edge information in regions with large displacements or nonrigid motion. Code is available at https://github.com/yifanna/MACFlow-pytorch. [ABSTRACT FROM AUTHOR]

Detecting respiration rate (RR) is a promising and practical heat stress diagnostic method for cows, with significant potential benefits for dairy operations in monitoring thermal conditions and managing cooling treatments. Currently, the optical flow method is widely employed for automatic video-based RR estimation. However, the optical flow-based approach for RR estimation can be time-consuming and susceptible to interference from various unrelated cow movements, such as rising, lying down, and body shaking. The aim of this study was to propose a novel optical flow-based algorithm for remotely and rapidly detecting the respiration rate of cows in free stalls. To accomplish this, we initially collected 250 sixty-second video episodes from a commercial dairy farm, which included some episodes with interfering motions. We manually observed the respiration rate for each episode, considering it as the ground truth RR. The analysis revealed that certain cow movements, including posture changes and body shaking, introduced noise that compromises the precision of RR detection. To address this issue, we implemented noise filters, with the Butterworth filter proving highly effective in mitigating noise resulting from cow movements. The selection of the region of interest was found to have a substantial impact on the accuracy of RR detection. Opting for the central region was recommended for optimal results. The comparison between the RR estimated by the modified cow respiration rate (MCRR) algorithm and the ground truth RR showed a good agreement with a mean absolute relative error of 7.6 ± 8.9% and a Pearson correlation coefficient of 0.86. Additionally, the results also indicated that reducing the original frame rate from 25 to 5 frames per second and adjusting the image pixel size from 630 × 450 to 79 × 57 pixels notably reduced computational time from 39.8 to 2.8 s, albeit with a slight increase in mean absolute relative error to 8.0 ± 9.0%. [ABSTRACT FROM AUTHOR]

Block matching methods are one of the most widely used methods in motion estimation and compensation. In this work, we propose a new hybrid block matching motion estimation algorithm based on the Lucas and Kanade method as a distortion criterion to improve the accuracy of estimated motion. The proposed algorithm proceeds in three steps. In the first step, a small hexagonal pattern is used, in order to find the smaller motion vectors and thus fewer searching points. In the second step, the modified large hexagonal pattern is used to identify the direction of motion vectors. In the third step, the small hexagonal search pattern is used to refine the solution search. The proposed algorithm is tested on several both synthetic and real images sequences. The experimental results show that our proposal could achieve good performances in terms of amplitude and angular errors, prediction quality, and computational complexity, compared to some related works. [ABSTRACT FROM AUTHOR]

Violence in recent years poses the biggest threat to society, which needs to be addressed by all means. Video-based Violence detection is very tough to discern when the person or things that are recipients of a violent act are in motion. Detection of violence in video content is a critical task with applications spanning security surveillance, content moderation, and public safety. Leveraging the power of deep learning, the Violence Guard Freeze-In Freeze-Out Inception V3(VGFrIFrOI3) deep learning model in conjunction with optical flow-based characteristics proposes an effective solution for automated violence detection in videos. This architecture is known for its efficiency and accuracy in image classification tasks and in extracting meaningful features from video frames. By fine-tuning Inception V3 on video datasets annotated for violent and non-violent actions, the network can be permitted to learn discriminative features that simplify the detection of any violent behavior. Furthermore, the aforementioned model incorporates temporal information by processing video frames sequentially and aggregating features across multiple frames using techniques, such as temporal convolutional networks or recurrent neural networks. To assess the performance of this approach, a performance comparison of the proposed model against already existing methods was conducted, demonstrating the model’s superior accuracy and robustness in detecting violent actions. The recommended approach not only offers a highly accurate solution for violence detection in video content but also provides insights into the potential of deep learning architectures like Inception V3 in addressing real-world challenges in video analysis and surveillance. The Mosaicking processing, additionally carried out in the pre-processing step, improves the algorithm performance by deploying space search minimization and optical flow-based feature extraction, aiming to extemporize accuracy.

This paper introduces an open-source pixel-wise Digital Image Correlation tool written in Python and targeting graphics processing units (GPUs) with the help of Cupy and Rapids-cuCim libraries. It is capable of computing the kinematic fields that transform an image into another in an efficient and quick way and it allows to treat large images in the GPU. Even if GCPU_OpticalFlow can be easily used by communities concerned by the estimation of displacement, it is particularly tuned to estimate consistent strain (gradient) field. The detection of a crack in a material is presented in this work as a demonstration.

Optical flow estimation captures the motion information of objects in a scene through analyzing the displacement of pixels in an image over time. This technology provides a powerful tool for vision systems, allowing them to understand and perceive changes in dynamic environments. Optical flow estimation has a wide range of applications in fields such as military, medicine, traffic regulation, and intelligent robotics. This study systematically explores two key directions in the field of optical flow estimation—traditional methods and emerging strategies based on deep learning—aiming to provide a comprehensive and in-depth perspective to help scholars gain a deeper understanding of the development of the optical flow estimation field. First, the core principles and constraints of conventional optical flow estimation are briefly analyzed, focusing on reviewing the faced challenges and associated solutions based on differential, variational, and matching optical flow estimation principles. Then, we discuss the backbone networks and training strategies used in deep learning approaches in depth, with a particular focus on the current challenges faced under supervised and unsupervised conditions, as well as existing solutions. In addition, to evaluate the performance of these methods, existing datasets and evaluation indicators are analyzed and comprehensive comparisons on several publicly available datasets are conducted. Finally, we discuss prospects related to various application fields and future research directions in the field of optical flow estimation.

The increasing interest in developing robots capable of navigating autonomously has led to the necessity of developing robust methods that enable these robots to operate in challenging and dynamic environments. Visual odometry (VO) has emerged in this context as a key technique, offering the possibility of estimating the position of a robot using sequences of onboard cameras. In this paper, a VO algorithm is proposed that achieves sub-pixel precision by combining optical flow and direct methods. This approach uses only a downward-facing, monocular camera, eliminating the need for additional sensors. The experimental results demonstrate the robustness of the developed method across various surfaces, achieving minimal drift errors in calculation.

Data and reports indicate an increasing frequency and intensity of natural disasters worldwide. Buildings play a crucial role in disaster responses and damage assessments, aiding in planning rescue efforts and evaluating losses. Despite advances in applying deep learning to building extraction, challenges remain in handling complex natural disaster scenes and reducing reliance on labeled datasets. Recent advances in satellite video are opening a new avenue for efficient and accurate building extraction research. By thoroughly mining the characteristics of disaster video data, this work provides a new semantic segmentation model for accurate and efficient building extraction based on a limited number of training data, which consists of two parts: the prediction module and the automatic correction module. The prediction module, based on a base encoder–decoder structure, initially extracts buildings using a limited amount of training data that are obtained instantly. Then, the automatic correction module takes the output of the prediction module as input, constructs a criterion for identifying pixels with erroneous semantic information, and uses optical flow values to extract the accurate corresponding semantic information on the corrected frame. The experimental results demonstrate that the proposed method outperforms other methods in accuracy and computational complexity in complicated natural disaster scenes.

When analyzing microscopic time-lapse observations, frame alignment is an essential task to visually understand the morphological and translation dynamics of cells and tissue. While in traditional single-sample microscopy, the region of interest (RoI) is fixed, multi-sample microscopy often uses a single microscope that scans multiple samples over a long period of time by laterally relocating the sample stage. Hence, the relocation of the optics induces a statistical RoI offset and can introduce jitter as well as drift, which results in a misaligned RoI for each sample’s time-lapse observation (stage drift). We introduce a robust approach to automatically align all frames within a time-lapse observation and compensate for frame drift. In this study, we present a sub-pixel precise alignment approach based on recurrent all-pairs field transforms (RAFT); a deep network architecture for optical flow. We show that the RAFT model pre-trained on the Sintel dataset performed with near perfect precision for registration tasks on a set of ten contextually unrelated time-lapse observations containing 250 frames each. Our approach is robust for elastically undistorted and translation displaced (x,y) microscopic time-lapse observations and was tested on multiple samples with varying cell density, obtained using different devices. The approach only performed well for registration and not for tracking of the individual image components like cells and contaminants. We provide an open-source command-line application that corrects for stage drift and jitter.

Infrared small target detection plays a crucial role in maritime security. However, detecting small targets within heavy sea clutter environments remains challenging. Existing methods often fail to deliver satisfactory performance in the presence of substantial clutter interference. This paper analyzes the spatial–temporal appearance characteristics of small targets and sea clutter. Based on this analysis, we propose a novel detection method based on the appearance stable isotropy measure (ASIM). First, the original images are processed using the Top-Hat transformation to obtain the salient regions. Next, a preliminary threshold operation is employed to extract the candidate targets from these salient regions, forming a candidate target array image. Third, to distinguish between small targets and sea clutter, we introduce two characteristics: the gradient histogram equalization measure (GHEM) and the local optical flow consistency measure (LOFCM). GHEM evaluates the isotropy of the candidate targets by examining their gradient histogram equalization, while LOFCM assesses their appearance stability based on local optical flow consistency. To effectively combine the complementary information provided by GHEM and LOFCM, we propose ASIM as a fusion characteristic, which can effectively enhance the real target. Finally, a threshold operation is applied to determine the final targets. Experimental results demonstrate that our proposed method exhibits superior comprehensive performance compared to baseline methods. [ABSTRACT FROM AUTHOR]

SLAM (simultaneous localization and mapping) plays a crucial role in autonomous robot navigation. A challenging aspect of visual SLAM systems is determining the 3D camera orientation of the motion trajectory. In this paper, we introduce an end-to-end network structure, InertialNet, which establishes the correlation between the image sequence and the IMU signals. Our network model is built upon inertial measurement learning and is employed to predict the camera's general motion pose. By incorporating an optical flow substructure, InertialNet is independent of the appearance of training sets and can be adapted to new environments. It maintains stable predictions even in the presence of image blur, changes in illumination, and low-texture scenes. In our experiments, we evaluated InertialNet on the public EuRoC dataset and our dataset, demonstrating its feasibility with faster training convergence and fewer model parameters for inertial measurement prediction. [ABSTRACT FROM AUTHOR]

In this paper, a real-time optical flow estimation method based on a cross-stage network is proposed. The proposed model is designed with a network structure with encoders and decoders. The proposed method combines cross-stage network technology with the network structure of FlowNet2 and RAFT to achieve improved parameter number and estimation performance. For real-time optical flow estimation, it is important to maintain performance while reducing the number of parameters in the network. In the proposed method, structural convergence is performed to increase performance while reducing the number of parameters by applying the cross-stage network structure. The proposed model is designed to solve the bottlenecks in model accuracy and complexity by separating feature extraction and flow estimation processes. Flying Chairs, Flying Things 3D, and KITTI datasets were used to evaluate the performance of the proposed model, and the experimental results show superior performance compared to previous traditional methods. [ABSTRACT FROM AUTHOR]

Results of studying stabilization of a homogeneous hydrogen–air flame on an optical discharge plasma in a high-velocity flow are reported. The main aspect of experiments is providing stable combustion behind the region of laser beam focusing without any mechanical flame holders. The laser radiation parameters are sufficient for creating a quasi-steady plasma in the flow. It is shown that the optical discharge stabilizes the flame front in a wide range of equivalence ratios for flow velocities up to m/s. The laser radiation parameters within the range of their variation from one experiment to another exert a minor effect on the turbulent flame velocity. Flame stabilization behind the optical discharge region has some specific features. An important parameter is heat release due to hydrogen combustion. A dimensionless criterion is derived: the turbulent flame velocity is a linear function of this criterion. [ABSTRACT FROM AUTHOR]

Pig husbandry constitutes a significant segment within the broader framework of livestock farming, with porcine well-being emerging as a paramount concern due to its direct implications on pig breeding and production. An easily observable proxy for assessing the health of pigs lies in their daily patterns of movement. The daily movement patterns of pigs can be used as an indicator of their health, in which more active pigs are usually healthier than those who are not active, providing farmers with knowledge of identifying pigs' health state before they become sick or their condition becomes life-threatening. However, the conventional means of estimating pig mobility largely rely on manual observations by farmers, which is impractical in the context of contemporary centralized and extensive pig farming operations. In response to these challenges, multi-object tracking and pig behavior methods are adopted to monitor pig health and welfare closely. Regrettably, these existing methods frequently fall short of providing precise and quantified measurements of movement distance, thereby yielding a rudimentary metric for assessing pig health. This paper proposes a novel approach that integrates optical flow and a multi-object tracking algorithm to more accurately gauge pig movement based on both qualitative and quantitative analyses of the shortcomings of solely relying on tracking algorithms. The optical flow records accurate movement between two consecutive frames and the multi-object tracking algorithm offers individual tracks for each pig. By combining optical flow and the tracking algorithm, our approach can accurately estimate each pig's movement. Moreover, the incorporation of optical flow affords the capacity to discern partial movements, such as instances where only the pig's head is in motion while the remainder of its body remains stationary. The experimental results show that the proposed method has superiority over the method of solely using tracking results, i.e., bounding boxes. The reason is that the movement calculated based on bounding boxes is easily affected by the size fluctuation while the optical flow data can avoid these drawbacks and even provide more fine-grained motion information. The virtues inherent in the proposed method culminate in the provision of more accurate and comprehensive information, thus enhancing the efficacy of decision-making and management processes within the realm of pig farming. [ABSTRACT FROM AUTHOR]

Accurate building extraction holds paramount importance in various applications such as urbanization rate calculations, urban planning, and resource allocation. In response to the escalating demand for precise low-altitude unmanned aerial vehicle (UAV) building segmentation in intricate scenarios, this study introduces a semi-supervised methodology to alleviate the labor-intensive process of procuring pixel-level annotations. Within the framework of adversarial networks, we employ a dual-channel parallel generator strategy that amalgamates the morphology-driven optical flow estimation channel with an enhanced multilayer sensing Deeplabv3+ module. This approach aims to comprehensively capture both the morphological attributes and textural intricacies of buildings while mitigating the dependency on annotated data. To further enhance the network's capability to discern building features, we introduce an adaptive attention mechanism via a feature fusion module. Additionally, we implement a composite loss function to augment the model's sensitivity to building structures. Across two distinct low-altitude UAV datasets within the domain of UAV-based building segmentation, our proposed method achieves average mean pixel intersection-over-union (mIoU) ratios of 82.69% and 79.37%, respectively, with unlabeled data constituting 70% of the overall dataset. These outcomes signify noteworthy advancements compared with contemporaneous networks, underscoring the robustness of our approach in tackling intricate building segmentation challenges in the domain of UAV-based architectural analysis. [ABSTRACT FROM AUTHOR]

Over the past four decades, Arctic sea ice coverage has steadily declined. This loss of sea ice has amplified solar radiation and heat absorption from the ocean, exacerbating both polar ice loss and global warming. It has also accelerated changes in sea ice movement, posing safety risks for ship navigation. In recent years, numerical prediction models have dominated the field of sea ice movement prediction. However, these models often rely on extensive data sources, which can be limited in specific time periods or regions, reducing their applicability. This study introduces a novel approach for predicting Arctic sea ice motion within a 10-day window. We employ a Self-Attention ConvLSTM deep learning network based on single-source data, specifically optical flow derived from the Advanced Microwave Scanning Radiometer Earth Observing System 36.5 GHz data, covering the entire Arctic region. Upon verification, our method shows a reduction of 0.80 to 1.18 km in average mean absolute error over a 10-day period when compared to ConvLSTM, demonstrating its improved ability to capture the spatiotemporal correlation of sea ice motion vector fields and provide accurate predictions. [ABSTRACT FROM AUTHOR]

Inspired by biological vision mechanism, event‐based cameras have been developed to capture continuous object motion and detect brightness changes independently and asynchronously, which overcome the limitations of traditional frame‐based cameras. Complementarily, spiking neural networks (SNNs) offer asynchronous computations and exploit the inherent sparseness of spatio‐temporal events. Notably, event‐based pixel‐wise optical flow estimations calculate the positions and relationships of objects in adjacent frames; however, as event camera outputs are sparse and uneven, dense scene information is difficult to generate and the local receptive fields of the neural network also lead to poor moving objects tracking. To address these issues, an improved event‐based self‐attention optical flow estimation network (SA‐FlowNet) that independently uses criss‐cross and temporal self‐attention mechanisms, directly capturing long‐range dependencies and efficiently extracting the temporal and spatial features from the event streams is proposed. In the former mechanism, a cross‐domain attention scheme dynamically fusing the temporal‐spatial features is introduced. The proposed network adopts a spiking‐analogue neural network architecture using an end‐to‐end learning method and gains significant computational energy benefits especially for SNNs. The state‐of‐the‐art results of the error rate for optical flow prediction on the Multi‐Vehicle Stereo Event Camera (MVSEC) dataset compared with the current SNN‐based approaches is demonstrated. [ABSTRACT FROM AUTHOR]