Search

The overrepresentation of centrifugal motion in the middle temporal visual area (area MT) has long been thought to provide an efficient coding strategy for optic flow processing. However, this overrepresentation compromises the detection of approaching objects, which is essential for survival. In the present study, we revisited this long-held notion by reanalyzing motion selectivity in area MT of three macaque monkeys (two males, one female) using random-dot stimuli instead of spot stimuli. We found no differences in the number of neurons tuned to centrifugal versus centripetal motion; however, centrifugally tuned neurons showed stronger tuning than centripetally tuned neurons. This was attributed to the heightened suppression of responses in centrifugal neurons to centripetal motion compared with that of centripetal neurons to centrifugal motion. Our modeling implies that this intensified suppression accounts for superior detection performance for weak centripetal motion stimuli. Moreover, through Fisher information analysis, we establish that the population sensitivity to motion direction in peripheral vision corresponds well with retinal motion statistics during forward locomotion. While these results challenge established concepts, considering the interplay of logarithmic Gaussian receptive fields and spot stimuli can shed light on the previously documented overrepresentation of centrifugal motion. Significantly, our findings reconcile a previously found discrepancy between MT activity and human behavior, highlighting the proficiency of peripheral MT neurons in encoding motion direction efficiently. [ABSTRACT FROM AUTHOR]

Vision plays a crucial role in the ability of compound-eyed insects to perceive the characteristics of their surroundings. Compound-eyed insects (such as the honeybee) can change the optical flow input of the visual system by autonomously controlling their behavior, and this is referred to as visual–motor coordination (VMC). To analyze an insect's VMC mechanism in dynamic scenes, we developed a platform for studying insects that actively shape the optic flow of visual stimuli by adapting their flight behavior. Image-processing technology was applied to detect the posture and direction of insects' movement, and automatic control technology provided dynamic scene stimulation and automatic acquisition of perceptual insect behavior. In addition, a virtual mapping technique was used to reconstruct the visual cues of insects for VMC analysis in a dynamic obstacle scene. A simulation experiment at different target speeds of 1–12 m/s was performed to verify the applicability and accuracy of the platform. Our findings showed that the maximum detection speed was 8 m/s, and triggers were 95% accurate. The outdoor experiments showed that flight speed in the longitudinal axis of honeybees was more stable when facing dynamic barriers than static barriers after analyzing the change in geometric optic flow. Finally, several experiments showed that the platform can automatically and efficiently monitor honeybees' perception behavior, and can be applied to study most insects and their VMC. [ABSTRACT FROM AUTHOR]

GLFNet is proposed to be utilized for the detection and matching of local features among remote-sensing images, with existing sparse feature points being leveraged as guided points. Local feature matching is a crucial step in remote-sensing applications and 3D reconstruction. However, existing methods that detect feature points in image pairs and match them separately may fail to establish correct matches among images with significant differences in lighting or perspectives. To address this issue, the problem is reformulated as the extraction of corresponding features in the target image, given guided points from the source image as explicit guidance. The approach is designed to encourage the sharing of landmarks by searching for regions in the target image with features similar to the guided points in the source image. For this purpose, GLFNet is developed as a feature extraction and search network. The main challenge lies in efficiently searching for accurate matches, considering the massive number of guided points. To tackle this problem, the search network is divided into a coarse-level match network-based guided point transformer that narrows the search space and a fine-level regression network that produces accurate matches. The experimental results on challenging datasets demonstrate that the proposed method provides robust matching and benefits various applications, including remote-sensing image registration, optical flow estimation, visual localization, and reconstruction registration. Overall, a promising solution is offered by this approach to the problem of local feature matching in remote-sensing applications. [ABSTRACT FROM AUTHOR]

Introduction: Flexible ureteroscopy (fURS) is the most common procedure for treatment of urolithiasis. We previously utilized kinematic evaluations of simulated fURS to demonstrate that certain body movements are associated with efficient ureteroscopic manipulation for complex tasks. In this study, we incorporated computer vision to create an efficiency score using the ureteroscope travel distance (DIST), task time (TIME), spectral arc length (SPARC), and percentage of purposeful wall collisions (COLL). The goal is a simulation-based system that can abstract these automated performance metrics (APMs) to differentiate between novice and expert ureteroscope handling. Methods: A ureteroscopic simulation box was used. Body kinematics, task time, and ureteroscopic movements were analyzed using a motion capture system and video camera. Optical flow computer vision was used to track the ureteroscope. DIST, TIME, and SPARC were automatically calculated. Wall collisions were automatically captured and independently judged by two authors; an algorithm was developed to automatically determine the COLL variable. A mixed-effects model was used to aggregate these variables and distinguish between surgeons' first and final task attempts. Normalized values of these metrics were added to create a composite ureteroscopic efficiency score (CUES). Results: Twelve urologists completed the simulated tasks. The COLL assessment algorithm determined beneficial wall collisions with an accuracy of 77%. Normalized values of TIME, DIST, SPARC, and COLL were combined to create a composite ureteroscopic efficiency score (CUES). Compared with the first attempt, both the second and third attempts showed statistically significant improvements in CUES. The ROC-AUC score reached 0.86, suggesting excellent discrimination between attempts. There was also a statistically significant difference in CUES when comparing resident and attending performance. Conclusions: APMs can be abstracted using computer vision and artificial intelligence; an aggregate composite score (CUES) may be a promising method for evaluation of ureteroscopic efficiency. [ABSTRACT FROM AUTHOR]

Dynamic objects bring about a large number of error accumulations in pose estimation of mobile robots in dynamic scenes, and result in the failure to build a map that is consistent with the surrounding environment. Along these lines, this paper presents a robust semi-direct 3D simultaneous localization and mapping (SLAM) algorithm for mobile robots based on dense optical flow. First, a preliminary estimation of the robot's pose is conducted using the sparse direct method and the homography matrix is utilized to compensate for the current frame image to reduce the image deformation caused by rotation during the robot's motion. Then, by calculating the dense optical flow field of two adjacent frames and segmenting the dynamic region in the scene based on the dynamic threshold, the local map points projected within the dynamic regions are eliminated. On this basis, the robot's pose is optimized by minimizing the reprojection error. Moreover, a high-performance keyframe selection strategy is developed, and keyframes are inserted when the robot's pose is successfully tracked. Meanwhile, feature points are extracted and matched to the keyframes for subsequent optimization and mapping. Considering that the direct method is subject to tracking failure in practical application scenarios, the feature points and map points of keyframes are employed in robot relocation. Finally, all keyframes and map points are used as optimization variables for global bundle adjustment (BA) optimization, so as to construct a globally consistent 3D dense octree map. A series of simulations and experiments demonstrate the superior performance of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Vidos from a first-person or egocentric perspective offer a promising tool for recognizing various activities related to daily living. In the egocentric perspective, the video is obtained from a wearable camera, and this enables the capture of the person's activities in a consistent viewpoint. Recognition of activity using a wearable sensor is challenging due to various reasons, such as motion blur and large variations. The existing methods are based on extracting handcrafted features from video frames to represent the contents. These features are domain-dependent, where features that are suitable for a specific dataset may not be suitable for others. In this paper, we propose a novel solution to recognize daily living activities from a pre-segmented video clip. The pre-trained convolutional neural network (CNN) model VGG16 is used to extract visual features from sampled video frames and then aggregated by the proposed pooling scheme. The proposed solution combines appearance and motion features extracted from video frames and optical flow images, respectively. The methods of mean and max spatial pooling (MMSP) and max mean temporal pyramid (TPMM) pooling are proposed to compose the final video descriptor. The feature is applied to a linear support vector machine (SVM) to recognize the type of activities observed in the video clip. The evaluation of the proposed solution was performed on three public benchmark datasets. We performed studies to show the advantage of aggregating appearance and motion features for daily activity recognition. The results show that the proposed solution is promising for recognizing activities of daily living. Compared to several methods on three public datasets, the proposed MMSP–TPMM method produces higher classification performance in terms of accuracy (90.38% with LENA dataset, 75.37% with ADL dataset, 96.08% with FPPA dataset) and average per-class precision (AP) (58.42% with ADL dataset and 96.11% with FPPA dataset). [ABSTRACT FROM AUTHOR]

The ocean is filled with mesoscale eddies that account for most of the oceanic kinetic energy. The importance of eddies in transporting properties and energy across the ocean basins has led to numerous efforts to track their motion. Here, we implement a computer vision technique—the optical flow—to map the pathways of mesoscale eddies in the South Atlantic Ocean. The optical flow is applied to the pairs of consecutive sea surface height maps produced from a nearly 30-year-long satellite altimetry record. In contrast to other methods to estimate the eddy propagation velocity, the optical flow can reveal the temporal evolution of eddy motion, which is particularly useful in the regions of strong currents. We present the time-dependent estimates of the speed and direction of eddy propagation in the Eulerian frame of reference. In an excellent agreement with earlier studies, the obtained pattern of eddy propagation reveals the interaction of eddies with the background flow and the bottom topography. We show that in the Antarctic Circumpolar Current, the variability of the eddy propagation velocity is correlated with the variability of the surface geostrophic velocity, demonstrating the robustness of the optical flow to detect the time-variable part of eddy motion. [ABSTRACT FROM AUTHOR]

The quantitative measurement of the microvascular blood-flow velocity is critical to the early diagnosis of microvascular dysfunction, yet there are several challenges with the current quantitative flow velocity imaging techniques for the microvasculature. Optical flow analysis allows for the quantitative imaging of the blood-flow velocity with a high spatial resolution, using the variation in pixel brightness between consecutive frames to trace the motion of red blood cells. However, the traditional optical flow algorithm usually suffers from strong noise from the background tissue, and a significant underestimation of the blood-flow speed in blood vessels, due to the errors in detecting the feature points in optical images. Here, we propose a temporal direction filtering and peak interpolation optical flow method (TPIOF) to suppress the background noise, and improve the accuracy of the blood-flow velocity estimation. In vitro phantom experiments and in vivo animal experiments were performed to validate the improvements in our new method. [ABSTRACT FROM AUTHOR]

Video-based group emotion recognition is an important research area in computer vision and is of great significance for the intelligent understanding of videos and human–computer interactions. Previous studies have adopted the traditional two-stage shallow pipeline to extract visual or audio features and train classifiers. A single feature or two are insufficient to comprehensively represent video information. In addition, sparse expression of emotions has not been addressed effectively. Therefore, in this study, we propose a novel deep convolutional neural networks (CNNs) architecture for video-based group emotion recognition that fuses multimodal feature information such as vision, audio, optical flow, and face. To address the problem of sparse emotional expressions in videos, we constructed an improved keyframe extraction algorithm for a visual stream to extract keyframes with more emotional features. A subnetwork incorporating spatial and channel attention was designed to automatically concentrate on the regions and channels carrying distinctive information in each keyframe to more accurately represent the emotional features of the visual stream. The proposed model was used to conduct extensive experiments on a video group affect dataset. It outperformed other video-based group emotion recognition methods. [ABSTRACT FROM AUTHOR]

Motivation The registration of serial section electron microscope images is a critical step in reconstructing biological tissue volumes, and it aims to eliminate complex nonlinear deformations from sectioning and replicate the correct neurite structure. However, due to the inherent properties of biological structures and the challenges posed by section preparation of biological tissues, achieving an accurate registration of serial sections remains a significant challenge. Conventional nonlinear registration techniques, which are effective in eliminating nonlinear deformation, can also eliminate the natural morphological variation of neurites across sections. Additionally, accumulation of registration errors alters the neurite structure. Results This article proposes a novel method for serial section registration that utilizes an unsupervised optical flow network to measure feature similarity rather than pixel similarity to eliminate nonlinear deformation and achieve pairwise registration between sections. The optical flow network is then employed to estimate and compensate for cumulative registration error, thereby allowing for the reconstruction of the structure of biological tissues. Based on the novel serial section registration method, a serial split technique is proposed for long-serial sections. Experimental results demonstrate that the state-of-the-art method proposed here effectively improves the spatial continuity of serial sections, leading to more accurate registration and improved reconstruction of the structure of biological tissues. Availability and implementation The source code and data are available at https://github.com/TongXin-CASIA/EFSR. [ABSTRACT FROM AUTHOR]

Glaucoma is a leading cause of irreversible blindness worldwide. To date, intraocular pressure (IOP) is the only modifiable risk factor in glaucoma treatment, but even in treated patients, the disease can progress. Cannabinoids, which have been known to lower IOP since the 1970s, have been shown to have beneficial effects in glaucoma patients beyond their IOP-lowering properties. In addition to the classical cannabinoid receptors CB1 and CB2, knowledge of non-classical cannabinoid receptors and the endocannabinoid system has increased in recent years. In particular, the CB2 receptor has been shown to mediate anti-inflammatory, anti-apoptotic, and neuroprotective properties, which may represent a promising therapeutic target for neuroprotection in glaucoma patients. Due to their vasodilatory effects, cannabinoids improve blood flow to the optic nerve head, which may suggest a vasoprotective potential and counteract the altered blood flow observed in glaucoma patients. The aim of this review was to assess the available evidence on the effects and therapeutic potential of cannabinoids in glaucoma patients. The pharmacological mechanisms underlying the effects of cannabinoids on IOP, neuroprotection, and ocular hemodynamics have been discussed. [ABSTRACT FROM AUTHOR]

The development of high-performance, low-cost unmanned aerial vehicles paired with rapid progress in vision-based perception systems herald a new era of autonomous flight systems with mission-ready capabilities. One of the key features of an autonomous UAV is a robust mid-air collision avoidance strategy. This paper proposes a vision-based in-flight collision avoidance system based on background subtraction using an embedded computing system for unmanned aerial vehicles (UAVs). The pipeline of proposed in-flight collision avoidance system is as follows: (i) subtract dynamic background subtraction to remove it and to detect moving objects, (ii) denoise using morphology and binarization methods, (iii) cluster the moving objects and remove noise blobs, using Euclidean clustering, (iv) distinguish independent objects and track the movement using the Kalman filter, and (v) avoid collision, using the proposed decision-making techniques. This work focuses on the design and the demonstration of a vision-based fast-moving object detection and tracking system with decision-making capabilities to perform evasive maneuvers to replace a high-vision system such as event camera. The novelty of our method lies in the motion-compensating moving object detection framework, which accomplishes the task with background subtraction via a two-dimensional transformation approximation. Clustering and tracking algorithms process detection data to track independent objects, and stereo-camera-based distance estimation is conducted to estimate the three-dimensional trajectory, which is then used during decision-making procedures. The examination of the system is conducted with a test quadrotor UAV, and appropriate algorithm parameters for various requirements are deduced. [ABSTRACT FROM AUTHOR]

Research on video anomaly detection has mainly been based on video data. However, many real-world cases involve users who can conceive potential normal and abnormal situations within the anomaly detection domain. This domain knowledge can be conveniently expressed as text descriptions, such as "walking" or "people fighting", which can be easily obtained, customized for specific applications, and applied to unseen abnormal videos not included in the training dataset. We explore the potential of using these text descriptions with unlabeled video datasets. We use large language models to obtain text descriptions and leverage them to detect abnormal frames by calculating the cosine similarity between the input frame and text descriptions using the CLIP visual language model. To enhance the performance, we refined the CLIP-derived cosine similarity using an unlabeled dataset and the proposed text-conditional similarity, which is a similarity measure between two vectors based on additional learnable parameters and a triplet loss. The proposed method has a simple training and inference process that avoids the computationally intensive analyses of optical flow or multiple frames. The experimental results demonstrate that the proposed method outperforms unsupervised methods by showing 8% and 13% better AUC scores for the ShanghaiTech and UCFcrime datasets, respectively. Although the proposed method shows −6% and −5% than weakly supervised methods for those datasets, in abnormal videos, the proposed method shows 17% and 5% better AUC scores, which means that the proposed method shows comparable results with weakly supervised methods that require resource-intensive dataset labeling. These outcomes validate the potential of using text descriptions in unsupervised video anomaly detection. [ABSTRACT FROM AUTHOR]

Accurate forecast of heavy precipitation remains a challenging task in most deep learning (DL)-based models. This study proposes a novel DL architecture named 'Multi-scale Feature Fusion' (MFF) for precipitation nowcasting for a lead time of up to 3 h. The basic idea is to apply convolution kernels with various sizes to achieve multi-scale receptive fields and then capture the movement features of the precipitation system (e.g. shape, movement direction, and moving speed). Meanwhile, the architecture implants the mechanism of discrete probability to reduce uncertainties and forecast errors, so that heavy precipitations can be produced even at longer lead time. The model uses four year's radar echo data from 2018 to 2021 for model training and one year's data of 2022 for model testing. The model is compared with three existing extrapolative models: time series residual convolution (TSRC), optical flow (OF), and UNet. Results show that MFF obtains relatively superior forecast skills with a high probability of detection (POD), low false alarm rate (FAR), small mean absolute error (MAE), and high structural similarity index (SSIM). The most commendable result is that MFF can predict high-intensity precipitation fields at 3 h lead time while the other three models can not. Additionally, it can be found from the results of radially averaged power spectral (RAPS) that MFF shows improvement in the smoothing effect of the forecast field. Future works will pay more attention to multi-source meteorological variables, the structural adjustments of the network, and the combinations with numerical models to further improve the forecast skills of heavy precipitations at longer lead times. [ABSTRACT FROM AUTHOR]

3D velocity field estimation is the key to high spatial resolution and high-accuracy measurements in tomographic particle image velocimetry (Tomo-PIV), especially when characterizing flow fields with delicate vortex structures. However, the cross-correlation velocity estimation method has limited spatial resolution due to the limitation of the interrogation sub-volume size. The 3D optical flow method can improve the spatial resolution of the velocity field, but its accuracy needs to be improved because it does not take into account the physical properties of the fluids. In this study, we propose a novel velocity decomposition-based 3D variational optical flow (VD-3DVOF) method to achieve high spatial resolution and high-accuracy measurement of 3D fluids. First, we present a novel regularization term based on the velocity decomposition theorem to constrain the different physical quantities, which can prevent the physical quantities from being over-smoothed. Second, we present a novel data term based on particle volume reconstruction feature weighting to reduce the influence of reconstruction errors on the velocity field estimation accuracy. Third, we present a multiscale technique and a volume warping operation to prevent the solution from falling into local optimal solutions. The newly proposed method considers both the physical properties of the fluid and the errors of reconstructed particle volumes. Velocity fields are estimated over synthetic and experimental particle volumes, and the results and comparisons show that the newly proposed VD-3DVOF method successfully achieves better performance and greater measurement accuracy than existing 3D motion estimation methods. [ABSTRACT FROM AUTHOR]

Adaptive human performance relies on the central nervous system to regulate the engagement of cognitive–motor resources as task demands vary. Despite numerous studies which employed a split-belt induced perturbation to examine biomechanical outcomes during locomotor adaptation, none concurrently examined the cerebral cortical dynamics to assess changes in mental workload. Additionally, while prior work suggests that optic flow provides critical information for walking regulation, a few studies have manipulated visual inputs during adaption to split-belt walking. This study aimed to examine the concurrent modulation of gait and Electroencephalography (EEG) cortical dynamics underlying mental workload during split-belt locomotor adaptation, with and without optic flow. Thirteen uninjured participants with minimal inherent walking asymmetries at baseline underwent adaptation, while temporal–spatial gait and EEG spectral metrics were recorded. The results revealed a reduction in step length and time asymmetry from early to late adaptation, accompanied by an elevated frontal and temporal theta power; the former being well corelated to biomechanical changes. While the absence of optic flow during adaptation did not affect temporal–spatial gait metrics, it led to an increase of theta and low-alpha power. Thus, as individuals adapt their locomotor patterns, the cognitive–motor resources underlying the encoding and consolidation processes of the procedural memory were recruited to acquire a new internal model of the perturbation. Also, when adaption occurs without optic flow, a further reduction of arousal is accompanied with an elevation of attentional engagement due to enhanced neurocognitive resources likely to maintain adaptive walking patterns. [ABSTRACT FROM AUTHOR]

Supervised deep learning methods reported recently have shown promising capability and efficiency in particle image velocimetry (PIV) processes compared to the traditional cross correlation and optical flow methods. However, the deep learning-based methods in previous reports require synthesized particle images and simulated flows for training prior to applications, conflicting with experimental scenarios. To address this crucial limitation, unsupervised deep learning methods have also been proposed for flow velocity reconstruction, but they are generally limited to rough flow reconstructions with low accuracy in velocity due to, for example, particle occlusion and out-of-boundary motions. This paper proposes a new unsupervised deep learning model named UnPWCNet-PIV (an unsupervised optical flow network using Pyramid, Warping, and Cost Volume). Such a pyramidical network with specific enhancements on flow reconstructions holds capabilities to manage particle occlusion and boundary motions. The new model showed comparable accuracy and robustness with the advanced supervised deep learning methods, which are based on synthesized images, together with superior performance on experimental images. This paper presents the details of the UnPWCNet-PIV architecture and the assessments of its accuracy and robustness on both synthesized and experimental images. [ABSTRACT FROM AUTHOR]

Particle image velocimetry (PIV), as a key technique in experimental fluid mechanics, is able to estimate complex velocity field through consecutive input particle images. In this study, an attention-mechanism incorporated deep recurrent network called ARaft-FlowNet has been proposed, on the basis of a previously established Recurrent All-Pairs Field Transforms optical flow model. The attention module is added to improve the network's capability of recognizing tracer particles' motion. Moreover, a parameterized dataset, ParaPIV-Dataset, is generated to explore the influence of particle parameters on deep learning networks, including particle diameter, image particle density, Gaussian noise, and peak intensity. The accuracy and generalizability of the newly proposed model has been evaluated and analyzed comprehensively. The results indicate that ARaft-FlowNet achieves state-of-the-art performance. Compared to previous methods, ARaft-FlowNet shows an accuracy improvement of 62.9%, 10.9%, and 9.4% in cylindrical flow, surface quasi-geostrophic flow, and DNS-turbulence flow. Meanwhile, the proposed model shows the strongest generalization and best capability to deal with complex flow fields with small-scale vortices. Additionally, tests on experimental turbulent jet data reveal that ARaft-FlowNet is able to deal with real PIV images with brightness variations and noise. [ABSTRACT FROM AUTHOR]

The growing intelligence and prevalence of drones have led to an increase in their disorderly and illicit usage, posing substantial risks to aviation and public safety. This paper focuses on addressing the issue of drone detection through surveillance cameras. Drone targets in images possess distinctive characteristics, including small size, weak energy, low contrast, and limited and varying features, rendering precise detection a challenging task. To overcome these challenges, we propose a novel detection method that extends the input of YOLOv5s to a continuous sequence of images and inter-frame optical flow, emulating the visual mechanisms employed by humans. By incorporating the image sequence as input, our model can leverage both temporal and spatial information, extracting more features of small and weak targets through the integration of spatiotemporal data. This integration augments the accuracy and robustness of drone detection. Furthermore, the inclusion of optical flow enables the model to directly perceive the motion information of drone targets across consecutive frames, enhancing its ability to extract and utilize features from dynamic objects. Comparative experiments demonstrate that our proposed method of extended input significantly enhances the network's capability to detect small moving targets, showcasing competitive performance in terms of accuracy and speed. Specifically, our method achieves a final average precision of 86.87%, representing a noteworthy 11.49% improvement over the baseline, and the speed remains above 30 frames per second. Additionally, our approach is adaptable to other detection models with different backbones, providing valuable insights for domains such as Urban Air Mobility and autonomous driving. [ABSTRACT FROM AUTHOR]

Pyroelectric infrared sensors (PIR) are widely used as infrared (IR) detectors due to their basic implementation, low cost, low power, and performance. Combined with a Fresnel lens, they can be used as a binary detector in applications of presence and motion control. Furthermore, due to their features, they can be used in autonomous intelligent devices or included in robotics applications or sensor networks. In this work, two neural processing architectures are presented: (1) an analog processing approach to achieve the behavior of a presynaptic neuron from a PIR sensor. An analog circuit similar to the leaky integrate and fire model is implemented to be able to generate spiking rates proportional to the IR stimuli received at a PIR sensor. (2) An embedded postsynaptic neuron where a spiking neural network matrix together with an algorithm based on digital processing techniques is introduced. This structure allows connecting a set of sensors to the post-synaptic circuit emulating an optic nerve. As a case study, the entire neural processing approach presented in this paper is applied to optical flow detection considering a four-PIR array as input. The results validate both the spiking approach for an analog sensor presented and the ability to retrieve the analog information sent as spike trains in a simulated optic nerve. [ABSTRACT FROM AUTHOR]

Lubricant oil is used in the rotary compressor to lubricate the bearings and to seal the clearance of sliding parts. However, a portion of lubricant oil in the upper cavity will ultimately exhaust from the compressor through the discharge tube, carried by the refrigerant. The accumulated lubricant in the tubes will deteriorate the heat transfer of the condenser and the evaporator, resulting in decreased efficiency. This work focuses on characterizing the pathway of lubricant droplets in a rotary compressor. In-situ measurements include lubricant spray penetration velocity at different phases in a rotating cycle and droplet velocities at different locations, i.e., lower cavity, above the rotor/stator, and at the discharge tube level, different frequencies, and different pressures via time-resolved shadowgraph. Particle Image Velocimetry (PIV) and Optical Flow Velocimetry (OFV) are adopted to quantify the velocities of spray and droplets. The penetration velocity of the lubricant spray in the lower cavity, horizontal swirling velocity above the rotor/stator and at the discharge tube level, and the vertical recirculation velocity above the rotor/stator are characterized. The results support a droplet pathway model in the rotary compressor that can guide the optimization of the next-generation rotary compressors. • OFV and PIV measure the spray and droplet velocity in a rotary compressor. • The spray shifts outward in the lower cavity with phase variations in one cycle. • A swirling jet and a vertical re-circulation in the upper cavity. • A lubricant oil pathway model is proposed. [ABSTRACT FROM AUTHOR]

Overlapping occlusion has seriously limited the yield estimation of bagged grapes in recent years, due to the ever-increasing grape volume after bagging and the large surface area of grape leaves. The unstable speed of manual video shooting can also lead to the loss of bagging grape targets. In this study, a yield estimation was proposed for the bagged grape using self-correcting Non-Maximum Suppression (NMS)-ByteTrack. Firstly, the bagged grapes were detected in the video using object detection (YOLOv5s). The NMS operation was also post-positioned to the tracking stage to retain the fruit detection boxes that filtered under the occlusion. Specifically, the detection boxes of bagged grapes were detected by object detection (YOLOv5s), whereas, the prediction boxes of bagged grapes were predicted to calculate the intersection over union using the Kalman filter. If the intersection over the union was less than the given threshold, the detection box was filtered to reserve the detection boxes closer to the predicted position. Then, the final detection box was obtained using the NMS operation. Secondly, the camera motion compensation and improved Kalman filter algorithm were added to automatically correct the position of the fruit prediction boxes and track them using ByteTrack. Specifically, the camera motion compensation was used to first extract the background key points in the bagged grape pictures of the previous frame and the current frame except for the tracking target. The sparse optical flow was used to match the extracted background in the key points of the bagged grapes. Then the affine transformation matrix of background motion was calculated by the RANSAC algorithm, and the Kalman filter was utilized to predict the bagged grape prediction boxes of the current frame. Finally, the affine transformation matrix was obtained to convert the prediction boxes in the coordinate system from the previous to the current frame, in order to realize the position information correction for the prediction boxes of the current frame. In addition, the improved Kalman filter algorithm can be used to change the aspect ratio in the state vector, and then to express the position of the bagged grape tracking boxes in the Kalman filter directly as the width, in order to more accurately estimate the position of the bagged grape tracking boxes. As such, a line-counting strategy was proposed to automatically count the bagged grapes. The strategy was also to set the counting line in the middle of the video, and also retain the idle frames without the bagged grapes in the first few seconds of the video. Furthermore, the bagged grapes with the same ID were only counted once to avoid repeated collision line counting. The successful collision between the center point of the tracking box of bagged grapes and the counting line was achieved in the automatic counting of bagged grapes. The dataset was collected from the Paidengte Agricultural Science and Technology Demonstration Park, Bishan District, Chongqing of China. The mobile phone cameras of Redmi K40 and OPPO Reno6pro+ were selected to capture the pictures of the same bagged grape at 8:00, 12:00, and 18:00 from different angles, such as frontal, sideways, and overhead shots. The total shooting time was about 6 h, the shooting height was about 1.5 m from the ground, and the shooting route was line by line. A total of 500 images and six videos were obtained for the bagged grapes. Among them, 500 images were expanded to 2000 images after saturation, brightness enhancement, brightness reduction, and mirror operations. Then, 2000 images were randomly divided into the training and validation sets in the ratio of 8:2, where six valid videos were used as the test set. Relevant experiments were conducted using this dataset. The experimental results showed that the yield estimation achieved a significant improvement in the tracking performance of bagged grapes using self-correcting NMS-ByteTrack. The multi-object tracking accuracy and precision, as well as the identification F1-score, were 64.6%, 82.4% and 80.8%, respectively, which increased by 1.7, 1.0 and 4.1 percentage points, respectively, compared with the ByteTrack. The number of ID switch was reduced by 3 times. Then, the average counting accuracy reached 82.8% in terms of counting performance, compared with manual counting. In addition, a comparison was also made with the five tracking methods. A better tracking performance was achieved, compared with the rest. The applicability of this estimation was also verified in tracking and counting bagged grapes. Therefore, the yield estimation can be expected to effectively promote the tracking and counting of bagged grapes in real scenarios using self-correcting NMS-ByteTrack. More accurate yield estimation of bagged grapes was also achieved in this field. [ABSTRACT FROM AUTHOR]

In the study of action recognition based on optical flow, improving the recognition speed of two-stream neural networks is challenging. In this paper, a new network structure Teacher Guided Student Network (TGSNet) which is based on two-stream and teacher–student architecture is proposed to judge the category of action rapidly in the application. There are two sub-networks with optical flow and RGB frame stream in the network, the optical flow sub-network is assigned as the teacher and the RGB frame stream sub-network as the student. In the training stage, the optical flow sub-network computes the optical flow of the video frame and trains the sub-network then transmits the feature to the RGB frame stream sub-network. The RGB frame stream sub-network uses the RGB frame to mimic the optical flow to train the sub-network. In the test stage, there is only RGB frame stream sub-network existing for action recognition rapidly without computing optical flow. The experimental results show that the TGSNet feeds only by RGB frame stream get a competitive accuracy of 56.7% and a better run-time on HMDB51. [ABSTRACT FROM AUTHOR]

Shale reservoirs are characterized by the presence of nano-micron pores in abundance, easy breakage due to bedding development, and difficulty in preparing true core samples. Therefore, 3D reconstruction of digital cores has become an important means of studying the microstructure features of shale. Currently, for shale matrix of dense reservoirs, existing deep learning-based reconstruction methods suffer from high costs and low accuracy, mainly due to the insufficient extraction capability of the networks for nano-micron pore features and low core resolution during reconstruction. To accurately predict the nano-micron scale pore structure of shale, we propose a shale 3D reconstruction method based on context-aware generative adversarial networks and high-resolution optical flow estimation (COFRnet-3DWGAN). This method optimizes the 3DWGAN(Wasserstein GAN) and high-resolution optical flow estimation networks by incorporating context-awareness, enhances the feature extraction capability of the networks to improve the learning degree of core nano-micron pores, and optimizes the core resolution by increasing the resolution of optical flow between core sequence images, thereby improving the accuracy of core reconstruction. The results show that, compared with WGAN, the proposed method is closer to the true core in terms of porous media morphological functions, porosity and permeability distribution, and pore structure parameters, indicating that this method has certain advantages in improving the accuracy of shale reconstruction. [ABSTRACT FROM AUTHOR]

Self-motion perception relies primarily on the integration of the visual, vestibular, proprioceptive, and somatosensory systems. There is a gap in understanding how a temporal lag between visual and vestibular motion cues affects visual–vestibular weighting during self-motion perception. The beta band is an index of visual–vestibular weighting, in that robust beta event-related synchronization (ERS) is associated with visual weighting bias, and robust beta event-related desynchronization is associated with vestibular weighting bias. The present study examined modulation of event-related spectral power during a heading judgment task in which participants attended to either visual (optic flow) or physical (inertial cues stimulating the vestibular, proprioceptive and somatosensory systems) motion cues from a motion simulator mounted on a MOOG Stewart Platform. The temporal lag between the onset of visual and physical motion cues was manipulated to produce three lag conditions: simultaneous onset, visual before physical motion onset, and physical before visual motion onset. There were two main findings. First, we demonstrated that when the attended motion cue was presented before an ignored cue, the power of beta associated with the attended modality was greater than when visual–vestibular cues were presented simultaneously or when the ignored cue was presented first. This was the case for beta ERS when the visual-motion cue was attended to, and beta event-related desynchronization when the physical-motion cue was attended to. Second, we tested whether the power of feature-binding gamma ERS (demonstrated in audiovisual and visual–tactile integration studies) increased when the visual–vestibular cues were presented simultaneously versus with temporal asynchrony. We did not observe an increase in gamma ERS when cues were presented simultaneously, suggesting that electrophysiological markers of visual–vestibular binding differ from markers of audiovisual and visual–tactile integration. All event-related spectral power reported in this study were generated from dipoles projecting from the left and right motor areas, based on the results of Measure Projection Analysis. [ABSTRACT FROM AUTHOR]

High Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processing techniques. One of the most challenging task in multiframe Low Dynamic Range (LDR) images fusion for HDR is to eliminate ghosting artifacts caused by motion. In traditional algorithms, optical flow is generally used to align dynamic scenes before image fusion, which can achieve good results in cases of small-scale motion scenes but causes obvious ghosting artifacts when motion magnitude is large. Recently, attention mechanisms have been introduced during the alignment stage to enhance the network's ability to remove ghosts. However, significant ghosting artifacts still occur in some scenarios with large-scale motion or oversaturated areas. We proposea novel Distilled Feature TransformerBlock (DFTB) structure to distill and re-extract information from deep image features obtained after U-Net downsampling, achieving ghost removal at the semantic level for HDR fusion. We introduce a Feature Distillation Transformer Block (FDTB), based on the Swin-Transformer and RFDB structure. FDTB uses multiple distillation connections to learn more discriminative feature representations. For the multiexposure moving scene image fusion HDR ghost removal task, in the previous method, the use of deep learning to remove the ghost effect in the composite image has been perfect, and it is almost difficult to observe the ghost residue of moving objects in the composite HDR image. The method in this paper focuses more on how to save the details of LDR image more completely after removing the ghost to synthesize high-quality HDR image. After using the proposed FDTB, the edge texture details of the synthesized HDR image are saved more perfectly, which shows that FDTB has a better effect in saving the details of image fusion. Futhermore, we propose a new depth framework based on DFTB for fusing and removing ghosts from deep image features, called TransU-Fusion. First of all, we use the encoder in U-Net to extract image features of different exposures and map them to different dimensional feature spaces. By utilizing the symmetry of the U-Net structure, we can ultimately output these feature images as original size HDR images. Then, we further fuse high-dimensional space features using Dilated Residual Dense Block (DRDB) to expand the receptive field, which is beneficial for repairing over-saturated regions. We use the transformer in DFTB to perform low-pass filtering on low-dimensional space features and interact with global information to remove ghosts. Finally, the processed features are merged and output as an HDR image without ghosting artifacts through the decoder. After testing on datasets and comparing with benchmark and state-of-the-art models, the results demonstrate our model's excellent information fusion ability and stronger ghost removal capability. [ABSTRACT FROM AUTHOR]

Micro-expressions are the small, brief facial expression changes that humans momentarily show during emotional experiences, and their data annotation is complicated, which leads to the scarcity of micro-expression data. To extract salient and distinguishing features from a limited dataset, we propose an attention-based multi-scale, multi-modal, multi-branch flow network to thoroughly learn the motion information of micro-expressions by exploiting the attention mechanism and the complementary properties between different optical flow information. First, we extract optical flow information (horizontal optical flow, vertical optical flow, and optical strain) based on the onset and apex frames of micro-expression videos, and each branch learns one kind of optical flow information separately. Second, we propose a multi-scale fusion module to extract more prosperous and more stable feature expressions using spatial attention to focus on locally important information at each scale. Then, we design a multi-optical flow feature reweighting module to adaptively select features for each optical flow separately by channel attention. Finally, to better integrate the information of the three branches and to alleviate the problem of uneven distribution of micro-expression samples, we introduce a logarithmically adjusted prior knowledge weighting loss. This loss function weights the prediction scores of samples from different categories to mitigate the negative impact of category imbalance during the classification process. The effectiveness of the proposed model is demonstrated through extensive experiments and feature visualization on three benchmark datasets (CASMEII, SAMM, and SMIC), and its performance is comparable to that of state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Spatial orientation is a prerequisite for most behaviors. In insects, the underlying neural computations take place in the central complex (CX), the brain's navigational center. In this region different streams of sensory information converge to enable context-dependent navigational decisions. Accordingly, a variety of CX input neurons deliver information about different navigation-relevant cues. In bees, direction encoding polarized light signals converge with translational optic flow signals that are suited to encode the flight speed of the animals. The continuous integration of speed and directions in the CX can be used to generate a vector memory of the bee's current position in space in relation to its nest, i.e., perform path integration. This process depends on specific, complex features of the optic flow encoding CX input neurons, but it is unknown how this information is derived from the visual periphery. Here, we thus aimed at gaining insight into how simple motion signals are reshaped upstream of the speed encoding CX input neurons to generate their complex features. Using electrophysiology and anatomical analyses of the halictic bees Megalopta genalis and Megalopta centralis, we identified a wide range of motion-sensitive neurons connecting the optic lobes with the central brain. While most neurons formed pathways with characteristics incompatible with CX speed neurons, we showed that one group of lobula projection neurons possess some physiological and anatomical features required to generate the visual responses of CX optic-flow encoding neurons. However, as these neurons cannot explain all features of CX speed cells, local interneurons of the central brain or alternative input cells from the optic lobe are additionally required to construct inputs with sufficient complexity to deliver speed signals suited for path integration in bees. [ABSTRACT FROM AUTHOR]

The optic flow, i.e., the displacement of retinal images of objects in the environment induced by self-motion, is an important source of spatial information, especially for fast-flying insects. Spatial information over a wide range of distances, from the animal's immediate surroundings over several hundred metres to kilometres, is necessary for mediating behaviours, such as landing manoeuvres, collision avoidance in spatially complex environments, learning environmental object constellations and path integration in spatial navigation. To facilitate the processing of spatial information, the complexity of the optic flow is often reduced by active vision strategies. These result in translations and rotations being largely separated by a saccadic flight and gaze mode. Only the translational components of the optic flow contain spatial information. In the first step of optic flow processing, an array of local motion detectors provides a retinotopic spatial proximity map of the environment. This local motion information is then processed in parallel neural pathways in a task-specific manner and used to control the different components of spatial behaviour. A particular challenge here is that the distance information extracted from the optic flow does not represent the distances unambiguously, but these are scaled by the animal's speed of locomotion. Possible ways of coping with this ambiguity are discussed. [ABSTRACT FROM AUTHOR]

We propose a framework to analyze the relationship between the movement features of a wheel gymnast around the mounting phase of Unit 2 of the vault event and execution (E-score) deductions from a machine-learning perspective. We first developed an automation system from a video of a wheel gymnast performing a tuck-front somersault to extract the four frames highlighting its Unit 2 performance of the vault event, such as take-off, pike-mount, the starting point of time on the wheel, and final position before the thrust. We implemented this automation using recurrent all-pairs field transforms (RAFT) and XMem, i.e., deep network architectures respectively for optical flow estimation and video object segmentation. We then used a markerless pose-estimation system called OpenPose to acquire the coordinates of the gymnast's body joints, such as shoulders, hips, and knees then calculate the joint angles at the extracted video frames. Finally, we constructed a regression model to estimate the E-score deductions during Unit 2 on the basis of the joint angles using an ensemble learning algorithm called Random Forests, with which we could automatically select a small number of features with the nonzero values of feature importances. By applying our framework of markerless motion analysis to videos of male wheel gymnasts performing the vault, we achieved precise estimation of the E-score deductions during Unit 2 with a determination coefficient of 0.79. We found the two movement features of particular importance for them to avoid significant deductions: time on the wheel and angles of knees at the pike-mount position. The selected features well reflected the maturity of the gymnast's skills related to the motions of riding the wheel, easily noticeable to the judges, and their branching conditions were almost consistent with the general vault regulations. [ABSTRACT FROM AUTHOR]

We propose an algorithm for generating a panoramic image of a pipe's inner surface based on inverse perspective mapping (IPM). The objective of this study is to generate a panoramic image of the entire inner surface of a pipe for efficient crack detection, without relying on high-performance capturing equipment. Frontal images taken while passing through the pipe were converted to images of the inner surface of the pipe using IPM. We derived a generalized IPM formula that considers the slope of the image plane to correct the image distortion caused by the tilt of the plane; this IPM formula was derived based on the vanishing point of the perspective image, which was detected using optical flow techniques. Finally, the multiple transformed images with overlapping areas were combined via image stitching to create a panoramic image of the inner pipe surface. To validate our proposed algorithm, we restored images of pipe inner surfaces using a 3D pipe model and used these images for crack detection. The resulting panoramic image of the internal pipe surface accurately demonstrated the positions and shapes of cracks, highlighting its potential utility for crack detection using visual inspection or image-processing techniques. [ABSTRACT FROM AUTHOR]

Today, coronary artery disease (CAD) continues to be a prominent cause of death worldwide. A reliable assessment of coronary stenosis represents a prerequisite for the appropriate management of CAD. Nevertheless, there are still major challenges pertaining to some limitations of current imaging and functional diagnostic modalities. The present review summarizes the current data on invasive functional and intracoronary imaging assessment using optical coherence tomography (OCT), and intravascular ultrasound (IVUS). Amongst the functional parameters—on top of fractional flow reserve (FFR) and instantaneous wave-free ratio (iFR)—we point to novel angiography-based measures such as quantitative flow ratio (QFR), vessel fractional flow reserve (vFFR), angiography-derived fractional flow reserve (FFRangio), and computed tomography-derived flow fractional reserve (FFR-CT), as well as hybrid approaches focusing on optical flow ratio (OFR), computational fluid dynamics and attempts to quantify the forces exaggerated by blood on the coronary plaque and vessel wall. [ABSTRACT FROM AUTHOR]

The free surface at an air–water interface can provide information regarding bathymetric complexities, as well as the subsurface flow. We present a comparison of the performance of two recent advances in light-based optical techniques for free surface measurements, total internal reflection-deflectometry and moon-glade background-oriented schlieren, with the more established method of free surface synthetic schlieren. We make use of an optical flow algorithm over the more traditional digital image correlation, in order to obtain higher spatial resolution data across the imaged free surface domain. The optical flow algorithm presents additional benefits, such as computational efficiency and robustness in capturing large displacements and straining of tracked features. The three optical techniques are assembled in synchronization to image two free surface conditions: (1) a free surface being impinged upon by an underlying turbulent, free-shear flow and (2) a random and irregular wave field induced by a free jet. Using the high-resolution measurements, we provide insight on the emergence of multiple free surface dynamics for a turbulent free surface. We present a comprehensive discussion on the benefits and drawbacks of each technique, including suggestions on the suitability of each technique for several experimental constraints. [ABSTRACT FROM AUTHOR]

Fog and low stratus (FLS) are meteorological phenomena that have a significant impact on all ways of transportation and public safety. Due to their similarity, they are often grouped together as a single category when viewed from a satellite perspective. The early detection of these phenomena is crucial to reduce the negative effects that they can cause. This paper presents an image-based approach for the short-term nighttime forecasting of FLS during the next 5 h over Morocco, based on geostationary satellite observations (Meteosat SEVIRI). To achieve this, a dataset of hourly night microphysics RGB product was generated from native files covering the nighttime cold season (October to April) of the 5-year period (2016–2020). Two optical flow techniques (sparse and dense) and three deep learning techniques (CNN, Unet and ConvLSTM) were used, and the performance of the developed models was assessed using mean squared error (MSE) and structural similarity index measure (SSIM) metrics. Hourly observations from Meteorological Aviation Routine Weather Reports (METAR) over Morocco were used to qualitatively compare the FLS existence in METAR, where it is also shown by the RGB product. Results analysis show that deep learning techniques outperform the traditional optical flow method with SSIM and MSE of about 0.6 and 0.3, respectively. Deep learning techniques show promising results during the first three hours. However, their performance is highly dependent on the number of filters and the computing resources, while sparse optical flow is found to be very sensitive to mask definition on the target phenomenon. [ABSTRACT FROM AUTHOR]

Investigating realistic visual exploration is quite challenging in sport climbing, but it promises a deeper understanding of how performers adjust their perception-action couplings during task completion. However, the samples of participants and the number of trials analyzed in such experiments are often reduced to a minimum because of the time-consuming treatments of the eye-tracking data. Notably, mapping successive points of gaze from local views to the global scene is generally performed manually by watching eye-tracking video data frame by frame. This manual procedure is not suitable for processing a large number of datasets. Consequently, this study developed an automatic method for solving this global point of gaze localization in indoor sport climbing. Particularly, an eye-tracking device was used for acquiring local image frames and points of gaze from a climber's local views. Artificial landmarks, designed as four-color-disk groups, were distributed on the wall to facilitate localization. Global points of gaze were computed based on planar homography transforms between the local and global positions of the detected landmarks. Thirty climbing trials were recorded and processed by the proposed methods. The success rates (Mean ± SD) were up to 85.72% ± 13.90%, and the errors (Mean ± SD) were up to 0. 1 3 0 2 ± 0. 2 0 5 1 m. The proposed method will be employed for computing global points of gaze in our current climbing dataset for understanding the dynamics intertwining of gaze and motor behaviors during the climbs. [ABSTRACT FROM AUTHOR]

Aging society gives rise to the need of fall detection for the elderly. The interference of the environmental noise and the loss of motion information causing fall detection still challenging. In this work, we present a novel two-stream network, called Flow-pose Net (FP-Net), which integrates the optical flow and human pose information to achieve robust and accurate fall detection in videos. Specifically, we use a human pose estimation model to detect the joints of the human body and design a GCN-based network to learn the body appearance feature from human pose. For motion feature extraction, we estimate optical flow from raw videos and utilize a CNN-based network to learn rich motion feature. Finally, the appearance feature and the motion feature are concatenated and then fed into a classifier to perform the classification of fall. To the best of our knowledge, we are the first to combine the optical flow and the human pose to simultaneously extract motion and appearance features for fall detection. Extensive experiments are conducted on two popular datasets URFD and Le2i, and the results show that our FP-Net achieves the state-of-the-art performance and has high robustness. [ABSTRACT FROM AUTHOR]

Rapid, accurate, and convenient diagnosis is essential for effective disease management. Various detection methods, such as enzyme-linked immunosorbent assay, have been extensively used, with lateral flow immunoassay (LFIA) recently emerging as a major diagnostic tool. Nanoparticles (NPs) with characteristic optical properties are used as probes for LFIA, and researchers have presented various types of optical NPs with modified optical properties. Herein, we review the literature on LFIA with optical NPs for the detection of specific targets in the context of diagnostics. [ABSTRACT FROM AUTHOR]

The motion of the camera in a video effectively conveys to the viewers the intention of the director, and is an essential element that enhances their interest. Therefore, detecting the motion of the camera is a very important factor in movie analysis. Existing research to detect the motion of the camera in a video has mainly focused on pan, tilt, and zoom. However, movies use more diverse camera motions to represent complex and varied emotions. Recognizing only pan, tilt, and zoom in a movie has limitations, especially not being able to detect lateral and longitudinal movements of the camera. In this study, a method is proposed to additionally detect boom and truck as well as pan, tilt, and zoom by using deep learning technology to improve this recognition ability. Thus, this study proposes the Improved Extractor of Camera Motion along with the CNN-Based Detector. The Improved Extractor of Camera Motion uses optical flow to extract camera motion vectors from video at eight-frame intervals. The CNN-Based Detector identifies five camera motions by using ResNet-152. As a result, the performance of our proposed method shows accuracy of 86.2%. [ABSTRACT FROM AUTHOR]

In the context of increasing the performance of structured packing used in distillation columns, we develop a state-of-the-art flow diagnostic method to investigate the intricate fluid dynamics of a falling film interacting with a perforation. Perforations are one of the several key microstructure elements in industrial structured packing whose role in mass transfer enhancement and liquid redistribution is yet not well understood. We focus the present investigation on the liquid curtain, or "twin film", blanketing the perforation. Twin films are prone to strong capillary waves and recirculation zones, and experiments of fundamental nature are required to assess the fluid dynamics in the curtain formation in the perforations and ultimately uncover mechanisms that intensify interfacial mass transfer. Here, a three-dimensional two-component (3D-2C PTV) optical-flow-based technique using the concept of defocus is implemented on a test bench designed to study the effect of perforations on falling liquid films. This method provides an opportunity to evaluate the hydrodynamics of the liquid curtain through streamline patterns and dynamic modes and to describe key features that could promote mass transfer intensification in structured packing. • We present experiments of a falling liquid film interacting with a perforation for twin films. • The flow field inside and around the perforation is investigated using a novel single-camera 3D2C flow diagnostic technique. • Detailed flow features such as recirculations and reverse flow areas are observed and quantified. • Dynamic structures triggered by the perforation are analyzed using Dynamic Mode Decomposition. [ABSTRACT FROM AUTHOR]

To improve component design, the fundamental understanding of the fatigue behaviour of gas turbine materials is essential. Since Ni-alloys exhibit pronounced elastic anisotropy, the local grain orientation strongly affects the stress and strain distribution in the material under mechanical loadings. This work addresses the characterisation of anisotropic elastic–plastic deformation and its consequences for crack initiation of nickel-base superalloy IN617 under tensile loading. Samples were loaded in situ in a scanning electron microscope (SEM) to correlate the deformation behaviour with the grain structure and the grain orientation determined by electron backscatter diffraction (EBSD) measurements. To calculate the resulting stresses and strains, the EBSD data were used to develop a model by finite element method (FEM) considering the grain structure and orientation. The results of the elastic–plastic finite element (FE) simulation were compared with the theories of the E⋅m model based on the Schmidt factor (m) and anisotropic Young's modulus (E). A mathematical image registration method called "optical flow method" (OFM), which is capable to calculate the transformation of EBSD measuring points during deformation, was applied to the EBSD data. The strains calculated by the optical flow method and by FE simulation were compared for two samples. The findings revealed large strains in the later crack initiation area found in both the OFM and FEM. The developed FEM model was verified by the successful correlation of hypotheses of the E·m model with the simulated mechanical behaviour. Furthermore, the impact of the microstructural neighbourhood on the mechanical behaviour was emphasised. [ABSTRACT FROM AUTHOR]

Birds of prey rely on vision to execute flight manoeuvres that are key to their survival, such as intercepting fast-moving targets or navigating through clutter. A better understanding of the role played by vision during these manoeuvres is not only relevant within the field of animal behaviour, but could also have applications for autonomous drones. In this paper, we present a novel method that uses computer vision tools to analyse the role of active vision in bird flight, and demonstrate its use to answer behavioural questions. Combining motion capture data from Harris' hawks with a hybrid 3D model of the environment, we render RGB images, semantic maps, depth information and optic flow outputs that characterise the visual experience of the bird in flight. In contrast with previous approaches, our method allows us to consider different camera models and alternative gaze strategies for the purposes of hypothesis testing, allows us to consider visual input over the complete visual field of the bird, and is not limited by the technical specifications and performance of a head-mounted camera light enough to attach to a bird's head in flight. We present pilot data from three sample flights: a pursuit flight, in which a hawk intercepts a moving target, and two obstacle avoidance flights. With this approach, we provide a reproducible method that facilitates the collection of large volumes of data across many individuals, opening up new avenues for data-driven models of animal behaviour. [ABSTRACT FROM AUTHOR]

Tracking points in ultrasound (US) videos can be especially useful to characterize tissues in motion. Tracking algorithms that analyze successive video frames, such as variations of Optical Flow and Lucas–Kanade (LK), exploit frame-to-frame temporal information to track regions of interest. In contrast, convolutional neural-network (CNN) models process each video frame independently of neighboring frames. In this paper, we show that frame-to-frame trackers accumulate error over time. We propose three interpolation-like methods to combat error accumulation and show that all three methods reduce tracking errors in frame-to-frame trackers. On the neural-network end, we show that a CNN-based tracker, DeepLabCut (DLC), outperforms all four frame-to-frame trackers when tracking tissues in motion. DLC is more accurate than the frame-to-frame trackers and less sensitive to variations in types of tissue movement. The only caveat found with DLC comes from its non-temporal tracking strategy, leading to jitter between consecutive frames. Overall, when tracking points in videos of moving tissue, we recommend using DLC when prioritizing accuracy and robustness across movements in videos, and using LK with the proposed error-correction methods for small movements when tracking jitter is unacceptable. [ABSTRACT FROM AUTHOR]

Tracking points in ultrasound (US) videos can be especially useful to characterize tissues in motion. Tracking algorithms that analyze successive video frames, such as variations of Optical Flow and Lucas–Kanade (LK), exploit frame-to-frame temporal information to track regions of interest. In contrast, convolutional neural-network (CNN) models process each video frame independently of neighboring frames. In this paper, we show that frame-to-frame trackers accumulate error over time. We propose three interpolation-like methods to combat error accumulation and show that all three methods reduce tracking errors in frame-to-frame trackers. On the neural-network end, we show that a CNN-based tracker, DeepLabCut (DLC), outperforms all four frame-to-frame trackers when tracking tissues in motion. DLC is more accurate than the frame-to-frame trackers and less sensitive to variations in types of tissue movement. The only caveat found with DLC comes from its non-temporal tracking strategy, leading to jitter between consecutive frames. Overall, when tracking points in videos of moving tissue, we recommend using DLC when prioritizing accuracy and robustness across movements in videos, and using LK with the proposed error-correction methods for small movements when tracking jitter is unacceptable. [ABSTRACT FROM AUTHOR]

The prediction of precipitation patterns up to 2 h ahead, also known as precipitation nowcasting, at high spatiotemporal resolutions is of great relevance in weather-dependent decision-making and early warning systems. In this study, we are aiming to provide an efficient and easy-to-understand deep neural network – CLGAN (convolutional long short-term memory generative adversarial network) – to improve the nowcasting skills of heavy precipitation events. The model constitutes a generative adversarial network (GAN) architecture, whose generator is built upon a u-shaped encoder–decoder network (U-Net) and is equipped with recurrent long short-term memory (LSTM) cells to capture spatiotemporal features. The optical flow model DenseRotation and the competitive video prediction models ConvLSTM (convolutional LSTM) and PredRNN-v2 (predictive recurrent neural network version 2) are used as the competitors. A series of evaluation metrics, including the root mean square error, the critical success index, the fractions skill score, and object-based diagnostic evaluation, are utilized for a comprehensive comparison against competing baseline models. We show that CLGAN outperforms the competitors in terms of scores for dichotomous events and object-based diagnostics. A sensitivity analysis on the weight of the GAN component indicates that the GAN-based architecture helps to capture heavy precipitation events. The results encourage future work based on the proposed CLGAN architecture to improve the precipitation nowcasting and early warning systems. [ABSTRACT FROM AUTHOR]

This paper focuses on moving-target detection and tracking in a three-dimensional (3D) space, and proposes a visual target tracking system only using a two-dimensional (2D) camera. To quickly detect moving targets, an improved optical flow method with detailed modifications in the pyramid, warping, and cost volume network (PWC-Net) is applied. Meanwhile, a clustering algorithm is used to accurately extract the moving target from a noisy background. Then, the target position is estimated using a proposed geometrical pinhole imaging algorithm and cubature Kalman filter (CKF). Specifically, the camera's installation position and inner parameters are applied to calculate the azimuth, elevation angles, and depth of the target while only using 2D measurements. The proposed geometrical solution has a simple structure and fast computational speed. Different simulations and experiments verify the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Reconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately perceive anomalous activities. In this paper, we exploit the idea of a training model based on the "Cloze Test" strategy in natural language processing (NLP) and introduce a novel unsupervised learning framework to encode both motion and appearance information at an object level. Specifically, to store the normal modes of video activity reconstructions, we first design an optical stream memory network with skip connections. Secondly, we build a space–time cube (STC) for use as the basic processing unit of the model and erase a patch in the STC to form the frame to be reconstructed. This enables a so-called "incomplete event (IE)" to be completed. On this basis, a conditional autoencoder is utilized to capture the high correspondence between optical flow and STC. The model predicts erased patches in IEs based on the context of the front and back frames. Finally, we employ a generating adversarial network (GAN)-based training method to improve the performance of VAD. By distinguishing the predicted erased optical flow and erased video frame, the anomaly detection results are shown to be more reliable with our proposed method which can help reconstruct the original video in IE. Comparative experiments conducted on the benchmark UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate AUROC scores reaching 97.7%, 89.7%, and 75.8%, respectively. [ABSTRACT FROM AUTHOR]

Fetal movement (FM) is an important indicator of fetal health. However, the current methods of FM detection are unsuitable for ambulatory or long-term observation. This paper proposes a non-contact method for monitoring FM. We recorded abdominal videos from pregnant women and then detected the maternal abdominal region within each frame. FM signals were acquired by optical flow color-coding, ensemble empirical mode decomposition, energy ratio, and correlation analysis. FM spikes, indicating the occurrence of FMs, were recognized using the differential threshold method. FM parameters including number, interval, duration, and percentage were calculated, and good agreement was found with the manual labeling performed by the professionals, achieving true detection rate, positive predictive value, sensitivity, accuracy, and F1_score of 95.75%, 95.26%, 95.75%, 91.40%, and 95.50%, respectively. The changes in FM parameters with gestational week were consistent with pregnancy progress. In general, this study provides a novel contactless FM monitoring technology for use at home. [ABSTRACT FROM AUTHOR]

Simultaneous Localization and Mapping (SLAM) is one of the most crucial aspects of autonomous navigation in mobile robots. The core components of the SLAM system can be depth perception and pose tracking. However, the existing unsupervised learning visual odometry framework cannot fully meet the actual requirements of the visual odometry information, particularly on the scale uncertainty in visual odometry estimation. It is still lacking in the geometric constraints during the autonomous operation of greenhouse mobile robots. In this study, an unsupervised optical flow-based visual odometry was presented. An optical flow estimation network was trained in an unsupervised manner using image warping. The optical flow between stereo images (disparity) was used to calculate the absolute depth of scenes. The optical flow between adjacent frames of left images was combined with the scene depth, in order to solve the frame-to-frame pose transformation matrix using Perspective-n-Point (PnP) algorithm. The reliable correspondences were selected in the solving process using forward and backward flow consistency checking to recover the absolute pose. A compact deep neural network was built with the convolutional modules to serve as the backbone of the flow model. This improved network was designed, according to the well-established principles: pyramidal processing, warping, and the use of a cost volume. At the same time, the cost volume normalization in the network was estimated with high values to alleviate the feature activations at higher levels than before. Furthermore, the local geometric consistency constraints were designed for the objective function of flow models. Meanwhile, a pyramid distilling loss was introduced to provide the supervision for the intermediate optical flows via distilling the finest final flow field as pseudo labels. A series of experiments were conducted using a wheeled mobile robot in a tomato greenhouse. The results showed that the better performance was achieved in the improved model. The local geometric consistency constraints improved the optical flow estimation accuracy. The endpoint error (EPE) of inter-frame and stereo optical flow was reduced by 8.89% and 8.96%, respectively. The pyramid distillation loss significantly reduced the optical flow estimation error of the flow model, in which the EPEs of the inter-frame and stereo optical flow decreased by 11.76% and 11.45%, respectively. The EPEs of the inter-frame and stereo optical flow were reduced by 12.50% and 7.25%, respectively, after cost volume normalization. Particularly, the price decreased by 1.28% for the calculation speed of the optical flow network. This improved model showed a 9.52% and 9.80% decrease in the root mean square error (RMSE) and mean absolute error (MAE) of relative translation error (RTE), respectively, compared with an existing unsupervised flow model. The decrease was 43.0% and 43.21%, respectively, compared with the Monodepth2. The pose tracking accuracy of this improved model was lower than that of ORB-SLAM3. The pure multi-view geometry shared the predicting dense depth maps of a scene. The relative error of depth estimation was 5.28% higher accuracy than the existing state-of-the-art self-supervised joint depth-pose learning. The accuracy of pose tracking depended mainly on the motion speed of robots. The performance of pose tracking at 0.2 m/s low speed and 0.8 m/s fast speed was significantly lower than that at 0.4-0.6 m/s. The resolution of the input image greatly impacted the pose tracking accuracy, with the errors decreasing gradually as the resolution increased. The MAE of RTE was not higher than 3.6 cm with the input image resolution of 832×512 pixels and the motion scope of 1 m, whereas, the MAE of relative rotation error (RRE) was not higher than 1.3º. These findings can provide technical support to design the vision system of greenhouse mobile robots. [ABSTRACT FROM AUTHOR]

A storm tracking and nowcasting model was developed for the contiguous US (CONUS) by combining observations from the advanced baseline imager (ABI) and numerical weather prediction (NWP) short-range forecast data, along with the precipitation rate from CMORPH (the Climate Prediction Center morphing technique). A random forest based model was adopted by using the maximum precipitation rate as the benchmark for convection intensity, with the location and time of storms optimized by using optical flow (OF) and continuous tracking. Comparative evaluations showed that the optimized models had higher accuracy for severe storms with areas equal to or larger than 5000 km2 over smaller samples, and loweraccuracy for cases smaller than 1000 km2, while models with sample-balancing applied showed higher possibilities of detection (PODs). A typical convective event from August 2019 was presented to illustrate the application of the nowcasting model on local severe storm (LSS) identification and warnings in the pre-convection stage; the model successfully provided warnings with a lead time of 1–2 h before heavy rainfall. Importance score analysis showed that the overall impact from ABI observations was much higher than that from NWP, with the brightness temperature difference between 6.2 and 10.3 microns ranking at the top in terms of feature importance. [ABSTRACT FROM AUTHOR]