Search

Purpose: Kilovoltage cone beam computed tomography (kVCBCT)-guided adaptive radiation therapy (ART) uses daily deformed CT (dCT), which is generated automatically through deformable registration methods. These registration methods may perform poorly in reproducing volumes of the target organ, rectum, and bladder during treatment. We analyzed the registration errors between the daily kVCBCTs and corresponding dCTs for these organs using the default optical flow algorithm and two registration procedures. We validated the effectiveness of these registration methods in replicating the geometry for dose calculation on kVCBCT for ART., Methods: We evaluated three deformable image registration (DIR) methods to assess their registration accuracy and dose calculation effeciency in mapping target and critical organs. The DIR methods include (1) default intensity-based deformable registration, (2) hybrid deformable registration, and (3) a two-step deformable registration process. Each technique was applied to a computerized imaging reference system (CIRS) phantom (Model 062 M) and to five patients who received volumetric modulated arc therapy to the prostate. Registration accuracy was assessed using the 95% Hausdorff distance (HD 95 ) and Dice similarity coefficient (DSC), and each method was compared with the intensity-based registration method. The improvement in the dCT image quality of the CIRS phantom and five patients was assessed by comparing dCT with kVCBCT. Image quality quantitative metrics for the phantom included the signal-to-noise ratio (SNR), uniformity, and contrast-to-noise ratio (CNR), whereas those for the patients included the mean absolute error (MAE), mean error, peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). To determine dose metric differences, we used a dose-volume histogram (DVH) and 3.0%/0.3 mm gamma analysis to compare planning computed tomography (pCT) and kVCBCT recalculations with restimulated CT images used as a reference., Results: The dCT images generated by the hybrid (dCT H ) and two-step (dCT C ) registration methods resulted in significant improvements compared to kVCBCT in the phantom model. Specifically, the SNR improved by 107% and 107.2%, the uniformity improved by 90% and 75%, and the CNR improved by 212.2% and 225.6 for dCT H and dCT C methods, respectively. For the patient images, the MAEs improved by 98% and 94%, the PSNRs improved by 16.3% and 22.9%, and the SSIMs improved by 1% and 1% in the dCT H and dCT C methods, respectively. For the geometric evaluation, only the two-step registration method improved registration accuracy. The dCT H method yielded an average HD 95 of 12 mm and average DSC of 0.73, whereas dCT C yielded an average HD 95 of 2.9 mm and average DSC of 0.902. The DVH showed that the dCT C -based dose calculations differed by <2% from the expected results for treatment targets and volumes of organs at risk. Additionally, gamma indices for dCT C -based treatment plans were >95% at all points, whereas they were <95% for kVCBCT-based treatment plans., Conclusion: The two-step registration method outperforms the intensity-based and hybrid registration methods. While the hybrid and two-step-based methods improved the image quality of kVCBCT in a linear accelerator, only the two-step method improved the registration accuracy of the corresponding structures among the pCT and kVCBCT datasets. A two-step registration process is recommended for applying kVCBCT to ART, which achieves better registration accuracy for local and global image structures. This method appears to be beneficial for radiotherapy dose calculation in patients with pelvic cancer., (© 2024 The Author(s). Journal of Applied Clinical Medical Physics published by Wiley Periodicals, LLC on behalf of The American Association of Physicists in Medicine.)

Human induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs) are an effective tool for studying cardiac function and disease, and hold promise for screening drug effects on human tissue. Understanding alterations in motion patterns within these cells is crucial for comprehending how the administration of a drug or the onset of a disease can impact the rhythm of the human heart. However, quantifying motion accurately and efficiently from optical measurements using microscopy is currently time consuming. In this work, we present a unified framework for performing motion analysis on a sequence of microscopically obtained images of tissues consisting of hiPSC-CMs. We provide validation of our developed software using a synthetic test case and show how it can be used to extract displacements and velocities in hiPSC-CM microtissues. Finally, we show how to apply the framework to quantify the effect of an inotropic compound. The described software system is distributed as a python package that is easy to install, well tested and can be integrated into any python workflow., (Creative Commons Attribution license.)

Data and reports indicate an increasing frequency and intensity of natural disasters worldwide. Buildings play a crucial role in disaster responses and damage assessments, aiding in planning rescue efforts and evaluating losses. Despite advances in applying deep learning to building extraction, challenges remain in handling complex natural disaster scenes and reducing reliance on labeled datasets. Recent advances in satellite video are opening a new avenue for efficient and accurate building extraction research. By thoroughly mining the characteristics of disaster video data, this work provides a new semantic segmentation model for accurate and efficient building extraction based on a limited number of training data, which consists of two parts: the prediction module and the automatic correction module. The prediction module, based on a base encoder-decoder structure, initially extracts buildings using a limited amount of training data that are obtained instantly. Then, the automatic correction module takes the output of the prediction module as input, constructs a criterion for identifying pixels with erroneous semantic information, and uses optical flow values to extract the accurate corresponding semantic information on the corrected frame. The experimental results demonstrate that the proposed method outperforms other methods in accuracy and computational complexity in complicated natural disaster scenes.

Purpose: Wearable ultrasound devices can be used to continuously monitor muscle activity. One possible application is to provide real-time feedback during physiotherapy, to show a patient whether an exercise is performed correctly. Algorithms which automatically analyze the data can be of importance to overcome the need for manual assessment and annotations and speed up evaluations especially when considering real-time video sequences. They even could be used to present feedback in an understandable manner to patients in a home-use scenario. The following work investigates three deep learning based segmentation approaches for abdominal muscles in ultrasound videos during a segmental stabilizing exercise. The segmentations are used to automatically classify the contraction state of the muscles., Methods: The first approach employs a simple 2D network, while the remaining two integrate the time information from the videos either via additional tracking or directly into the network architecture. The contraction state is determined by comparing measures such as muscle thickness and center of mass between rest and exercise. A retrospective analysis is conducted but also a real-time scenario is simulated, where classification is performed during exercise., Results: Using the proposed segmentation algorithms, 71% of the muscle states are classified correctly in the retrospective analysis in comparison to 90% accuracy with manual reference segmentation. For the real-time approach the majority of given feedback during exercise is correct when the retrospective analysis had come to the correct result, too., Conclusion: Both retrospective and real-time analysis prove to be feasible. While no substantial differences between the algorithms were observed regarding classification, the networks incorporating the time information showed temporally more consistent segmentations. Limitations of the approaches as well as reasons for failing cases in segmentation, classification and real-time assessment are discussed and requirements regarding image quality and hardware design are derived., (© 2024. The Author(s).)

Infrastructure operation and maintenance is essential for societal safety, particularly in Japan where the aging of infrastructures built during the period of high economic growth is advancing. However, there are issues such as a shortage of engineers and inefficiencies in work, requiring improvements in efficiency and automation for their resolution. Nevertheless, there are still many inefficiencies in the current procedures for bridge inspections. Usually, inspection engineers check for damage on bridges through close visual inspections at the site, then photograph the damaged parts, measure the size by touch, and create a report. A three-dimensional representation, considering the front and back of the structural elements, is needed for identifying damage, necessitating the creation of multi-directional three-dimensional drawings. However, this process is labor-intensive and prone to errors. Furthermore, due to the lack of uniformity in records, it is challenging to refer to past inspection histories. Especially for long bridges, without resolving such issues, the required labor and the number of mistakes could exceed acceptable limits, making proper management difficult. Therefore, in this study, we developed a method for automatically measuring the position and area of corroded parts by capturing images of the lower surface of the stiffening girder using a bridge inspection vehicle and utilizing image diagnosis technology. By integrating these results into a 3D model called BIM (Building Information Modeling), it becomes possible to manage the bridge more efficiently. We verified this method on actual long bridges and confirmed its effectiveness., Competing Interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (© 2024 The Authors.)

When analyzing microscopic time-lapse observations, frame alignment is an essential task to visually understand the morphological and translation dynamics of cells and tissue. While in traditional single-sample microscopy, the region of interest (RoI) is fixed, multi-sample microscopy often uses a single microscope that scans multiple samples over a long period of time by laterally relocating the sample stage. Hence, the relocation of the optics induces a statistical RoI offset and can introduce jitter as well as drift, which results in a misaligned RoI for each sample's time-lapse observation (stage drift). We introduce a robust approach to automatically align all frames within a time-lapse observation and compensate for frame drift. In this study, we present a sub-pixel precise alignment approach based on recurrent all-pairs field transforms (RAFT); a deep network architecture for optical flow. We show that the RAFT model pre-trained on the Sintel dataset performed with near perfect precision for registration tasks on a set of ten contextually unrelated time-lapse observations containing 250 frames each. Our approach is robust for elastically undistorted and translation displaced (x,y) microscopic time-lapse observations and was tested on multiple samples with varying cell density, obtained using different devices. The approach only performed well for registration and not for tracking of the individual image components like cells and contaminants. We provide an open-source command-line application that corrects for stage drift and jitter.

This research proposes a sensor for tracking the motion of a human head via optical tactile sensing. It implements the use of a fibrescope a non-metal alternative to a webcam. Previous works have included robotics grippers to mimic the sensory features of human skin, that used monochrome cameras and depth cameras. Tactile sensing has shown advantages in feedback-based interactions between robots and their environment. The methodology in this paper is utilised to track motion of objects in physical contact with these sensors to replace external camera based motion capture systems. Our immediate application is related to detection of human head motion during radiotherapy procedures. The motion was analysed in two degrees of freedom, respective to the tactile sensor (translational in z-axis, and rotational around y-axis), to produce repeatable and accurate results. The movements were stimulated by a robot arm, which also provided ground truth values from its end-effector. The fibrescope was implemented to ensure the device's compatibility with electromagnetic waves. The cameras and the ground truth values were time synchronised using robotics operating systems tools. Image processing methods were compared between grayscale and binary image sequences, followed by motion tracking estimation using deterministic approaches. These included Lukas-Kanade Optical Flow and Simple Blob Detection, by OpenCV. The results showed that the grayscale image processing along with the Lukas-Kanade algorithm for motion tracking can produce better tracking abilities, although further exploration to improve the accuracy is still required., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Gandhi, Mihaylova and Dogramadzi.)

Micro-expression recognition (MER) has drawn increasing attention due to its wide application in lie detection, criminal detection and psychological consultation. However, the best recognition accuracy on recent public dataset is still low compared to the accuracy of macro-expression recognition. In this paper, we propose a novel graph convolution network (GCN) for MER achieving state-of-the-art accuracy. Different to existing GCN with fixed graph structure, we define a stochastic graph structure in which some neighbors are selected randomly. As shown by numerical examples, randomness enables better feature characterization while reducing computational complexity. The whole network consists of two branches, one is the spatial branch taking micro-expression images as input, the other is the temporal branch taking optical flow images as input. Because the micro-expression dataset does not have enough images for training the GCN, we employ the transfer learning mechanism. That is, different stochastic GCNs (SGCN) have been trained by the macro-expression dataset in the source network. Then the well-trained SGCNs are transferred to the target network. It is shown that our proposed method achieves the state-of-art performance on all four well-known datasets. This paper explores stochastic GCN and transfer learning with this random structure in the MER task, which is of great importance to improve the recognition performance., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

The growing use of Unmanned Aerial Vehicles (UAVs) raises the need to improve their autonomous navigation capabilities. Visual odometry allows for dispensing positioning systems, such as GPS, especially on indoor flights. This paper reports an effort toward UAV autonomous navigation by proposing a translational velocity observer based on inertial and visual measurements for a quadrotor. The proposed observer complementarily fuses available measurements from different domains and is synthesized following the Immersion and Invariance observer design technique. A formal Lyapunov-based observer error convergence to zero is provided. The proposed observer algorithm is evaluated using numerical simulations in the Parrot Mambo Minidrone App from Simulink-Matlab.

The remarkable damage-tolerance of enamel has been attributed to its hierarchical microstructure and the organized bands of decussated rods. A thorough characterization of the microscale rod evolution within the enamel is needed to elucidate this complex structure. While prior efforts in this area have made use of single particle tracking to track a single rod evolution to various degrees of success, such a process can be both computationally and labor intensive, limited to the evolution path of a single rod, and is therefore prone to error from potentially tracking outliers. Particle image velocimetry (PIV) is a well-established algorithm to derive field information from image sequences for processes that are time-dependent, such as fluid flows and structural deformation. In this work, we demonstrate the use of PIV in extracting the full-field microstructural distribution of rods within the enamel. Enamel samples from a wild African lion were analyzed using high-energy synchrotron X-ray micro-tomography. Results from the PIV analysis provide sufficient full-field information to reconstruct the growth of individual rods that can potentially enable rapid analysis of complex microstructures from high resolution synchrotron datasets. Such information can serve as a template for designing damage-tolerant bioinspired structures for advanced manufacturing. STATEMENT OF SIGNIFICANCE: Thorough characterization and analysis of biological microstructures (viz. dental enamel) allows us to understand the basis of their excellent mechanical properties. Prior efforts have successfully replicated these microstructures via single particle tracking, but the process is computationally and labor intensive. In this work, optical flow imaging algorithms were used to extract full-field microstructural distribution of enamel rods from synchrotron X-ray computed tomography datasets, and a field method was used to reconstruct the growth of individual rods. Such high throughput information allows for the rapid production/prototyping and advanced manufacturing of damage-tolerant bioinspired structures for specific engineering applications. Furthermore, the algorithms used herein are freely available and open source to broaden the availability of the proposed workflow to the general scientific community., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Acta Materialia Inc. All rights reserved.)

The sky may seem big enough for two flying vehicles to collide, but the facts show that mid-air collisions still occur occasionally and are a significant concern. Pilots learn manual tactics to avoid collisions, such as see-and-avoid, but these rules have limitations. Automated solutions have reduced collisions, but these technologies are not mandatory in all countries or airspaces, and they are expensive. These problems have prompted researchers to continue the search for low-cost solutions. One attractive solution is to use computer vision to detect obstacles in the air due to its reduced cost and weight. A well-trained deep learning solution is appealing because object detection is fast in most cases, but it relies entirely on the training data set. The algorithm chosen for this study is optical flow. The optical flow vectors can help us to separate the motion caused by camera motion from the motion caused by incoming objects without relying on training data. This paper describes the development of an optical flow-based airborne obstacle detection algorithm to avoid mid-air collisions. The approach uses the visual information from a monocular camera and detects the obstacles using morphological filters, optical flow, focus of expansion, and a data clustering algorithm. The proposal was evaluated using realistic vision data obtained with a self-developed simulator. The simulator provides different environments, trajectories, and altitudes of flying objects. The results showed that the optical flow-based algorithm detected all incoming obstacles along their trajectories in the experiments. The results showed an F-score greater than 75% and a good balance between precision and recall.

Motion object detection (MOD) with freely moving cameras is a challenging task in computer vision. To extract moving objects, most studies have focused on the difference in motion features between foreground and background, which works well for dynamic scenes with relatively regular movements and variations. However, abrupt illumination changes and occlusions often occur in real-world scenes, and the camera may also pan, tilt, rotate, and jitter, etc., resulting in local irregular variations and global discontinuities in motion features. Such complex and changing scenes bring great difficulty in detecting moving objects. To solve this problem, this paper proposes a new MOD method that effectively leverages local and global visual information for foreground/background segmentation. Specifically, on the global side, to support a wider range of camera motion, the relative inter-frame transformations are optimized to absolute transformations referenced to intermediate frames in a global form after enriching the inter-frame matching pairs. The global transformation is fine-tuned using the spatial transformer network (STN). On the local side, to address the problem of dynamic background scenes, foreground object detection is optimized by utilizing the pixel differences between the current frame and the local background model, as well as the consistency of local spatial variations. Then, the spatial information is combined using optical flow segmentation methods, enhancing the precision of the object information. The experimental results show that our method achieves a detection accuracy improvement of over 1.5% compared with the state-of-the-art methods on the datasets of CDNET2014, FBMS-59, and CBD. It demonstrates significant effectiveness in challenging scenarios such as shadows, abrupt changes in illumination, camera jitter, occlusion, and moving backgrounds.

Objectives: The paralaryngeal muscles are thought to be hyperfunctional with phonation in patients with primary muscle tension dysphonia (pMTD). However, objective, quantitative tools to assess paralaryngeal movement patterns lack. The objectives of this study were to (1) validate the use of optical flow to characterize paralaryngeal movement patterns with phonation, (2) characterize phonatory optical flow velocities and variability of the paralaryngeal muscles before and after a vocal load challenge, and (3) compare phonatory optical flow measures to standard laryngoscopic, acoustic, and self-perceptual assessments., Methods: Phonatory movement velocities and variability of the paralaryngeal muscles at vocal onsets and offsets were quantified from ultrasound videos and optical flow methods across 42 subjects with and without a diagnosis of pMTD, before and after a vocal load challenge. Severity of laryngoscopic mediolateral supraglottic compression, acoustic perturbation, and ratings of vocal effort and discomfort were also obtained at both time points., Results: There were no significant differences in optical flow measures of the paralaryngeal muscles with phonation between patients with pMTD and controls. Patients with pMTD had significantly more supraglottic compression, higher acoustic perturbations, and higher vocal effort and vocal tract discomfort ratings. Vocal load had a significant effect on vocal effort and discomfort but not on supraglottic compression, acoustics, or optical flow measures of the paralaryngeal muscles., Conclusion: Optical flow methods can be used to study paralaryngeal muscle movement velocity and variability patterns during vocal productions, although the role of the paralaryngeal in pMTD diagnostics (e.g., vocal hyperfunction) remains suspect., Level of Evidence: 2 Laryngoscope, 134:1792-1801, 2024., (© 2023 The Authors. The Laryngoscope published by Wiley Periodicals LLC on behalf of The American Laryngological, Rhinological and Otological Society, Inc.)

Background and Objective: Automatic needle tip detection is important in real-time ultrasound (US) images that are utilized to guide interventional needle puncture procedures in clinical settings. However, due to the spatial indiscernibility problem caused by the severe background interferences and the tip characteristics of small size, being grayscale and indistinctive appearance patterns, tip detection in US images is challenging., Methods: To achieve precise tip detection in US images against spatial indiscernibility, a novel multi-keyframe motion-aware framework called TipDet is proposed. It can identify tips based on their short-term spatial-temporal pattern and long-term motion pattern. In TipDet, first, an adaptive keyframe model (AKM) is proposed to decide whether a frame is informative to serve as a keyframe for long-term motion pattern learning. Second, candidate tip detection is conducted using a two-stream backbone (TSB) based on their short-term spatial-temporal pattern. Third, to further identify the true one in the candidate tips, a novel method for learning the long-term motion pattern of the tips is proposed based on the proposed optical-flow-aware multi-head cross-attention (OFA-MHCA)., Results: On the clinical human puncture dataset, which includes 4195 B-mode images, the experimental results show that the proposed TipDet can achieve precise tip detection against the spatial indiscernibility problem, achieving 78.7 % AP 0.1:0.5 and 8.9 % improvement over the base detector at approximately 20 FPS. Moreover, a tip localization error of 1.3±0.6 % is achieved, exceeding the existing method., Conclusions: The proposed TipDet can facilitate a wider and easier application of US-guided interventional procedures by providing robust and precise needle tip localization. The codes and data are available at https://github.com/ResonWang/TipDet., Competing Interests: Declaration of competing interest The authors declare that they have no conflict of interest., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Due to the complexity of real optical flow capture, the existing research still has not performed real optical flow capture of infrared (IR) images with the production of an optical flow based on IR images, which makes the research and application of deep learning-based optical flow computation limited to the field of RGB images only. Therefore, in this paper, we propose a method to produce an optical flow dataset of IR images. We utilize the RGB-IR cross-modal image transformation network to rationally transform existing RGB image optical flow datasets. The RGB-IR cross-modal image transformation is based on the improved Pix2Pix implementation, and in the experiments, the network is validated and evaluated using the RGB-IR aligned bimodal dataset M 3 FD. Then, RGB-IR cross-modal transformation is performed on the existing RGB optical flow dataset KITTI, and the optical flow computation network is trained using the IR images generated by the transformation. Finally, the computational results of the optical flow computation network before and after training are analyzed based on the RGB-IR aligned bimodal data.

Purpose: Specular reflections (SRs) are highlight artifacts commonly found in endoscopy videos that can severely disrupt a surgeon's observation and judgment. Despite numerous attempts to restore SR, existing methods are inefficient and time consuming and can lead to false clinical interpretations. Therefore, we propose the first complete deep-learning solution, SpecReFlow, to detect and restore SR regions from endoscopy video with spatial and temporal coherence., Approach: SpecReFlow consists of three stages: (1) an image preprocessing stage to enhance contrast, (2) a detection stage to indicate where the SR region is present, and (3) a restoration stage in which we replace SR pixels with an accurate underlying tissue structure. Our restoration approach uses optical flow to seamlessly propagate color and structure from other frames of the endoscopy video., Results: Comprehensive quantitative and qualitative tests for each stage reveal that our SpecReFlow solution performs better than previous detection and restoration methods. Our detection stage achieves a Dice score of 82.8% and a sensitivity of 94.6%, and our restoration stage successfully incorporates temporal information with spatial information for more accurate restorations than existing techniques., Conclusions: SpecReFlow is a first-of-its-kind solution that combines temporal and spatial information for effective detection and restoration of SR regions, surpassing previous methods relying on single-frame spatial information. Future work will look to optimizing SpecReFlow for real-time applications. SpecReFlow is a software-only solution for restoring image content lost due to SR, making it readily deployable in existing clinical settings to improve endoscopy video quality for accurate diagnosis and treatment., (© 2024 The Authors.)

Optical identification is often done with spatial or temporal visual pattern recognition and localization. Temporal pattern recognition, depending on the technology, involves a trade-off between communication frequency, range, and accurate tracking. We propose a solution with light-emitting beacons that improves this trade-off by exploiting fast event-based cameras and, for tracking, sparse neuromorphic optical flow computed with spiking neurons. The system is embedded in a simulated drone and evaluated in an asset monitoring use case. It is robust to relative movements and enables simultaneous communication with, and tracking of, multiple moving beacons. Finally, in a hardware lab prototype, we demonstrate for the first time beacon tracking performed simultaneously with state-of-the-art frequency communication in the kHz range., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 von Arnim, Lecomte, Borras, Woźniak and Pantazi.)

Automated assessment of broiler chicken welfare poses particular problems due to the large numbers of birds involved and the variety of different welfare measures that have been proposed. Active (sustained, defect-free) walking is both a universally agreed measure of bird health and a behavior that can be recognized by existing technology. This makes active walking an ideal starting point for automated assessment of chicken welfare at both individual and flock level., Competing Interests: The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Dawkins.)

Detecting violent behavior in videos to ensure public safety and security poses a significant challenge. Precisely identifying and categorizing instances of violence in real-life closed-circuit television, which vary across specifications and locations, requires comprehensive understanding and processing of the sequential information embedded in these videos. This study aims to introduce a model that adeptly grasps the spatiotemporal context of videos within diverse settings and specifications of violent scenarios. We propose a method to accurately capture spatiotemporal features linked to violent behaviors using optical flow and RGB data. The approach leverages a Conv3D-based ResNet-3D model as the foundational network, capable of handling high-dimensional video data. The efficiency and accuracy of violence detection are enhanced by integrating an attention mechanism, which assigns greater weight to the most crucial frames within the RGB and optical-flow sequences during instances of violence. Our model was evaluated on the UBI-Fight, Hockey, Crowd, and Movie-Fights datasets; the proposed method outperformed existing state-of-the-art techniques, achieving area under the curve scores of 95.4, 98.1, 94.5, and 100.0 on the respective datasets. Moreover, this research not only has the potential to be applied in real-time surveillance systems but also promises to contribute to a broader spectrum of research in video analysis and understanding.

Background: Flight accidents caused by spatial disorientation (SD) greatly affect flight safety., Objective: Few studies have been devoted to the evaluation of SD., Methods: 10 pilots and 10 non-pilots were recruited for the experimental induction of SD. Videos for giving optical flow stimuli were played at two different flow speeds to induce SD. Subjective judgment and center of foot pressure (CoP) data were collected from the tests. The data were combined to determine the occurrence of SD and analyze the SD types., Results: The number of self-reported SD events was slightly smaller in the pilots than in the non-pilots. The average upper bound of the confidence interval for the standard deviation of CoP was 0.32 ± 0.09 cm and 0.38 ± 0.12 cm in the pilots and non-pilots, respectively. This indicator was significantly lower in the pilots than in the non-pilots (P= 0.03). The success rate of the experimental induction of unrecognized SD was 26.7% and 45.0% in the pilots and non-pilots, respectively., Conclusion: The method offered a new to analyze unrecognized SD. We could determine the occurrence unrecognized SD. This is an essential means of reducing flight accidents caused by unrecognized SD.

SLAM (simultaneous localization and mapping) plays a crucial role in autonomous robot navigation. A challenging aspect of visual SLAM systems is determining the 3D camera orientation of the motion trajectory. In this paper, we introduce an end-to-end network structure, InertialNet, which establishes the correlation between the image sequence and the IMU signals. Our network model is built upon inertial measurement learning and is employed to predict the camera's general motion pose. By incorporating an optical flow substructure, InertialNet is independent of the appearance of training sets and can be adapted to new environments. It maintains stable predictions even in the presence of image blur, changes in illumination, and low-texture scenes. In our experiments, we evaluated InertialNet on the public EuRoC dataset and our dataset, demonstrating its feasibility with faster training convergence and fewer model parameters for inertial measurement prediction.

In this paper, a brain-like navigation scheme based on fuzzy kernel C-means (FKCM) clustering assisted pyramid Lucas Kanade (LK) optical flow algorithm is developed to measure the position of vehicle. The Speed Cell and Place Cell in animals' brain are introduced to construct the brain-like navigation mechanism which involves the optical flow method and image template matching to imitate the cells above-mentioned separately. To eliminate the singular values during optical flow calculation, the output of pyramid LK algorithm is clustered by FKCM algorithm firstly. Then, the velocity is calculated and integrated to get the position of the vehicle, and the brain-like navigation scheme is introduced to correct the position measurement errors by eliminating the accumulated errors resulting from velocity integration. The prominent advantages of the presented method are: (i) a pure visual brain-like position measurement method based on the concept of speed cells and place cells is proposed, making visual navigation more accurate and intelligent; (ii) the FKCM algorithm is used to eliminate the singular value of the pyramid LK algorithm, which improves the calculated velocity accuracy. Also, experimental comparison with classical pyramid LK algorithm is given to illustrate the superiority of the proposed method in position measurement., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 ISA. Published by Elsevier Ltd. All rights reserved.)

Pig husbandry constitutes a significant segment within the broader framework of livestock farming, with porcine well-being emerging as a paramount concern due to its direct implications on pig breeding and production. An easily observable proxy for assessing the health of pigs lies in their daily patterns of movement. The daily movement patterns of pigs can be used as an indicator of their health, in which more active pigs are usually healthier than those who are not active, providing farmers with knowledge of identifying pigs' health state before they become sick or their condition becomes life-threatening. However, the conventional means of estimating pig mobility largely rely on manual observations by farmers, which is impractical in the context of contemporary centralized and extensive pig farming operations. In response to these challenges, multi-object tracking and pig behavior methods are adopted to monitor pig health and welfare closely. Regrettably, these existing methods frequently fall short of providing precise and quantified measurements of movement distance, thereby yielding a rudimentary metric for assessing pig health. This paper proposes a novel approach that integrates optical flow and a multi-object tracking algorithm to more accurately gauge pig movement based on both qualitative and quantitative analyses of the shortcomings of solely relying on tracking algorithms. The optical flow records accurate movement between two consecutive frames and the multi-object tracking algorithm offers individual tracks for each pig. By combining optical flow and the tracking algorithm, our approach can accurately estimate each pig's movement. Moreover, the incorporation of optical flow affords the capacity to discern partial movements, such as instances where only the pig's head is in motion while the remainder of its body remains stationary. The experimental results show that the proposed method has superiority over the method of solely using tracking results, i.e., bounding boxes. The reason is that the movement calculated based on bounding boxes is easily affected by the size fluctuation while the optical flow data can avoid these drawbacks and even provide more fine-grained motion information. The virtues inherent in the proposed method culminate in the provision of more accurate and comprehensive information, thus enhancing the efficacy of decision-making and management processes within the realm of pig farming.

Objectives: To evaluate the performance of a deep learning-based multi-source model for survival prediction and risk stratification in patients with heart failure., Methods: Patients with heart failure with reduced ejection fraction (HFrEF) who underwent cardiac magnetic resonance between January 2015 and April 2020 were retrospectively included in this study. Baseline electronic health record data, including clinical demographic information, laboratory data, and electrocardiographic information, were collected. Short-axis non-contrast cine images of the whole heart were acquired to estimate the cardiac function parameters and the motion features of the left ventricle. Model accuracy was evaluated using the Harrell's concordance index. All patients were followed up for major adverse cardiac events (MACEs), and survival prediction was assessed using Kaplan-Meier curves., Results: A total of 329 patients were evaluated (age 54 ± 14 years; men, 254) in this study. During a median follow-up period of 1041 days, 62 patients experienced MACEs and their median survival time was 495 days. When compared with conventional Cox hazard prediction models, deep learning models showed better survival prediction performance. Multi-data denoising autoencoder (DAE) model reached the concordance index of 0.8546 (95% CI: 0.7902-0.8883). Furthermore, when divided into phenogroups, the multi-data DAE model could significantly discriminate between the survival outcomes of the high-risk and low-risk groups compared with other models (p < 0.001)., Conclusions: The proposed deep learning (DL) model based on non-contrast cardiac cine magnetic resonance imaging could independently predict the outcome of patients with HFrEF and showed better prediction efficiency than conventional methods., Clinical Relevance Statement: The proposed multi-source deep learning model based on cardiac magnetic resonance enables survival prediction in patients with heart failure., Key Points: • A multi-source deep learning model based on non-contrast cardiovascular magnetic resonance (CMR) cine images was built to make robust survival prediction in patients with heart failure. • The ground truth definition contains electronic health record data as well as DL-based motion data, and cardiac motion information is extracted by optical flow method from non-contrast CMR cine images. • The DL-based model exhibits better prognostic value and stratification performance when compared with conventional prediction models and could aid in the risk stratification in patients with HF., (© 2023. The Author(s), under exclusive licence to European Society of Radiology.)

Atmospheric turbulence (AT) can change the path and direction of light during video capturing of a target in space due to the random motion of the turbulent medium, a phenomenon that is most noticeable when shooting videos at long ranges, resulting in severe video dynamic distortion and blur. To mitigate geometric distortion and reduce spatially and temporally varying blur, we propose a novel Atmospheric Turbulence Video Restoration Generative Adversarial Network (ATVR-GAN) with a specialized Recurrent Neural Network (RNN) generator, which is trained to predict the scene's turbulent optical flow (OF) field and utilizes a recurrent structure to catch both spatial and temporal dependencies. The new architecture is trained using a newly combined loss function that counts for the spatiotemporal distortions, specifically tailored to the AT problem. Our network was tested on synthetic and real imaging data and compared against leading algorithms in the field of AT mitigation and image restoration. The proposed method outperformed these methods for both synthetic and real data examined.

Introduction: Sensory inference and top-down predictive processing, reflected in human neural activity, play a critical role in higher-order cognitive processes, such as language comprehension. However, the neurobiological bases of predictive processing in higher-order cognitive processes are not well-understood., Methods: This study used electroencephalography (EEG) to track participants' cortical dynamics in response to Austrian Sign Language and reversed sign language videos, measuring neural coherence to optical flow in the visual signal. We then used machine learning to assess entropy-based relevance of specific frequencies and regions of interest to brain state classification accuracy., Results: EEG features highly relevant for classification were distributed across language processing-related regions in Deaf signers (frontal cortex and left hemisphere), while in non-signers such features were concentrated in visual and spatial processing regions., Discussion: The results highlight functional significance of predictive processing time windows for sign language comprehension and biological motion processing, and the role of long-term experience (learning) in minimizing prediction error., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2023 Malaia, Borneman, Borneman, Krebs and Wilbur.)

Objectives: To develop a quantitative analysis method for right diaphragm deformation. This method is based on optical flow and applied to diaphragm ultrasound imaging., Methods: This study enrolls six healthy subjects and eight patients under mechanical ventilation. Dynamic images with 3-5 breathing cycles were acquired from three directions of right diaphragm by a portable ultrasound system. Filtering and density clustering algorithms are used for denoising Digital Imaging and Communications in Medicine (DICOM) data. An optical flow based method is applied to track movements of the right diaphragm. An improved drift correction algorithm is used to optimize the results. The method can automatically analyze the respiratory cycle, inter-frame/cumulative vertical and horizontal displacements, and strain of the input right diaphragm ultrasound image., Results: The optical-flow-based diaphragm ultrasound image motion tracking algorithm can accurately track the right diaphragm during respiratory motion. There are significant differences in horizontal and vertical displacements in each section (p-values < 0.05 for all). Significant differences are found between healthy subjects and mechanical ventilation patients for both horizontal and vertical displacements in Section III (p-values < 0.05 for both). There is no significant difference in global strain in each section between healthy subjects and mechanical ventilation patients (p-values > 0.05 for all)., Conclusions: The developed method can quantitatively evaluate the inter-frame/cumulative displacement of the diaphragm in both horizontal and vertical directions, as well as the global strain in three different imaging planes. The above indicators can be used to evaluate diaphragmatic dynamics., (© 2023. BioMed Central Ltd., part of Springer Nature.)

Vidos from a first-person or egocentric perspective offer a promising tool for recognizing various activities related to daily living. In the egocentric perspective, the video is obtained from a wearable camera, and this enables the capture of the person's activities in a consistent viewpoint. Recognition of activity using a wearable sensor is challenging due to various reasons, such as motion blur and large variations. The existing methods are based on extracting handcrafted features from video frames to represent the contents. These features are domain-dependent, where features that are suitable for a specific dataset may not be suitable for others. In this paper, we propose a novel solution to recognize daily living activities from a pre-segmented video clip. The pre-trained convolutional neural network (CNN) model VGG16 is used to extract visual features from sampled video frames and then aggregated by the proposed pooling scheme. The proposed solution combines appearance and motion features extracted from video frames and optical flow images, respectively. The methods of mean and max spatial pooling (MMSP) and max mean temporal pyramid (TPMM) pooling are proposed to compose the final video descriptor. The feature is applied to a linear support vector machine (SVM) to recognize the type of activities observed in the video clip. The evaluation of the proposed solution was performed on three public benchmark datasets. We performed studies to show the advantage of aggregating appearance and motion features for daily activity recognition. The results show that the proposed solution is promising for recognizing activities of daily living. Compared to several methods on three public datasets, the proposed MMSP-TPMM method produces higher classification performance in terms of accuracy (90.38% with LENA dataset, 75.37% with ADL dataset, 96.08% with FPPA dataset) and average per-class precision (AP) (58.42% with ADL dataset and 96.11% with FPPA dataset).

The quantitative measurement of the microvascular blood-flow velocity is critical to the early diagnosis of microvascular dysfunction, yet there are several challenges with the current quantitative flow velocity imaging techniques for the microvasculature. Optical flow analysis allows for the quantitative imaging of the blood-flow velocity with a high spatial resolution, using the variation in pixel brightness between consecutive frames to trace the motion of red blood cells. However, the traditional optical flow algorithm usually suffers from strong noise from the background tissue, and a significant underestimation of the blood-flow speed in blood vessels, due to the errors in detecting the feature points in optical images. Here, we propose a temporal direction filtering and peak interpolation optical flow method (TPIOF) to suppress the background noise, and improve the accuracy of the blood-flow velocity estimation. In vitro phantom experiments and in vivo animal experiments were performed to validate the improvements in our new method.

The development of high-performance, low-cost unmanned aerial vehicles paired with rapid progress in vision-based perception systems herald a new era of autonomous flight systems with mission-ready capabilities. One of the key features of an autonomous UAV is a robust mid-air collision avoidance strategy. This paper proposes a vision-based in-flight collision avoidance system based on background subtraction using an embedded computing system for unmanned aerial vehicles (UAVs). The pipeline of proposed in-flight collision avoidance system is as follows: (i) subtract dynamic background subtraction to remove it and to detect moving objects, (ii) denoise using morphology and binarization methods, (iii) cluster the moving objects and remove noise blobs, using Euclidean clustering, (iv) distinguish independent objects and track the movement using the Kalman filter, and (v) avoid collision, using the proposed decision-making techniques. This work focuses on the design and the demonstration of a vision-based fast-moving object detection and tracking system with decision-making capabilities to perform evasive maneuvers to replace a high-vision system such as event camera. The novelty of our method lies in the motion-compensating moving object detection framework, which accomplishes the task with background subtraction via a two-dimensional transformation approximation. Clustering and tracking algorithms process detection data to track independent objects, and stereo-camera-based distance estimation is conducted to estimate the three-dimensional trajectory, which is then used during decision-making procedures. The examination of the system is conducted with a test quadrotor UAV, and appropriate algorithm parameters for various requirements are deduced.

Purpose: To quantify bowel motility shown on cine MRI using the classical optical flow algorithm and compare it with balloon-assisted enteroscopy (BAE) findings in patients with Crohn's disease (CD)., Methods: This retrospective study included 29 consecutive patients with CD who had undergone MR enterocolonography (MREC) and BAE between March and May 2017. We developed computer software to present motion vector magnitudes between consecutive cine MR images as bowel motility maps via a classical optical flow algorithm using the Horn-Schunck method. Cine MR images were acquired with a balanced steady-state free precession sequence in the coronal direction to capture small bowel motility. The small bowels were divided into three segments. In total, 63 bowel segments were assessed via BAE and MREC. Motility scores on the maps, simplified MR index of activity (sMaRIA), and MREC score derived from a 5-point MR classification were assessed independently by two radiologists and compared with the CD endoscopic index of severity (CDEIS). Correlations were assessed using Spearman's rank coefficient. The areas under the receiver-operating characteristic curve (AUCs) of motility score for differentiating CDEIS was calculated; a P value < 0.05 was considered statistically significant., Results: Motility score was negatively correlated with CDEIS (r = -0.59 [P < 0.001] and -0.54 [P < 0.001]), and the AUCs of motility scores for detecting CDEIS ≥ 3 were 88.2% and 78.6% for observers 1 and 2, respectively. There were no significant differences in the AUC for detecting CDEIS ≥ 3 and CDEIS ≥ 12 between motility and sMaRIA or MREC score., Conclusion: The motility map was feasible for locally quantifying the bowel motility. In addition, the motility score on the map reflected the endoscopic inflammatory activity of each small bowel segment in patients with CD; hence, it could be used as a tool in objectively interpreting cine MREC to predict inflammatory activity in CD.

Despite the continued successes of computationally efficient deep neural network architectures for video object detection, performance continually arrives at the great trilemma of speed versus accuracy versus computational resources (pick two). Current attempts to exploit temporal information in video data to overcome this trilemma are bottlenecked by the state of the art in object detection models. This work presents motion vector extrapolation (MOVEX), a technique which performs video object detection through the use of off-the-shelf object detectors alongside existing optical flow-based motion estimation techniques in parallel. This work demonstrates that this approach significantly reduces the baseline latency of any given object detector without sacrificing accuracy performance. Further latency reductions up to 24 times lower than the original latency can be achieved with minimal accuracy loss. MOVEX enables low-latency video object detection on common CPU-based systems, thus allowing for high-performance video object detection beyond the domain of GPU computing.

Propose: Contrast-enhanced ultrasound has shown great promises for diagnosis and monitoring in a wide range of clinical conditions. Meanwhile, to obtain accurate and effective location of lesion in contrast-enhanced ultrasound videos is the basis for subsequent diagnosis and qualitative treatment, which is a challenging task nowadays. Methods: We propose to upgrade a siamese architecture-based neural network for robust and accurate landmark tracking in contrast-enhanced ultrasound videos. Due to few researches on it, the general inherent assumptions of the constant position model and the missing motion model remain unaddressed limitations. In our proposed model, we overcome these limitations by introducing two modules into the original architecture. We use a temporal motion attention based on Lucas Kanade optic flow and Karman filter to model the regular movement and better instruct location prediction. Moreover, we design a pipeline of template update to ensure timely adaptation to feature changes. Results: Eventually, the whole framework was performed on our collected datasets. It has achieved the average mean IoU values of 86.43% on 33 labeled videos with a total of 37,549 frames. In terms of tracking stability, our model has smaller TE of 19.2 pixels and RMSE of 27.6 with the FPS of 8.36 ± 3.23 compared to other classical tracking models. Conclusion: We designed and implemented a pipeline for tracking focal areas in contrast-enhanced ultrasound videos, which takes the siamese network as the backbone and uses optical flow and Kalman filter algorithm to provide position prior information. It turns out that these two additional modules are helpful for the analysis of CEUS videos. We hope that our work can provide an idea for the analysis of CEUS videos., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2023 Tian, Cai, Ding, Liang, Yu and Huang.)

Pyroelectric infrared sensors (PIR) are widely used as infrared (IR) detectors due to their basic implementation, low cost, low power, and performance. Combined with a Fresnel lens, they can be used as a binary detector in applications of presence and motion control. Furthermore, due to their features, they can be used in autonomous intelligent devices or included in robotics applications or sensor networks. In this work, two neural processing architectures are presented: (1) an analog processing approach to achieve the behavior of a presynaptic neuron from a PIR sensor. An analog circuit similar to the leaky integrate and fire model is implemented to be able to generate spiking rates proportional to the IR stimuli received at a PIR sensor. (2) An embedded postsynaptic neuron where a spiking neural network matrix together with an algorithm based on digital processing techniques is introduced. This structure allows connecting a set of sensors to the post-synaptic circuit emulating an optic nerve. As a case study, the entire neural processing approach presented in this paper is applied to optical flow detection considering a four-PIR array as input. The results validate both the spiking approach for an analog sensor presented and the ability to retrieve the analog information sent as spike trains in a simulated optic nerve.

The blood flow velocity in the nailfold capillary is an important indicator of the status of microcirculation. The conventional manual processing method is both laborious and prone to human artifacts. A feasible way to solve this problem is to use machine learning to assist in image processing and diagnosis. Inspired by the Two-Stream Convolutional Networks, this study proposes an optical flow-assisted two-stream network to segment nailfold blood vessels. Firstly, we use U-Net as the spatial flow network and the dense optical flow as the temporal stream. The results show that the optical flow information can effectively improve the integrity of the segmentation of blood vessels. The overall accuracy is 94.01 %, the Dice score is 0.8099, the IoU score is 0.6806, and the VOE score is 0.3194. Secondly, The flow velocity of the segmented blood vessel is determined by constructing the spatial-temporal (ST) image. The blood flow velocity evaluated is consistent with the typical blood flow speed reported. This study proposes a novel two-stream network for blood vessel segmentation of nailfold capillary images. Combined with ST image and line detection method, it provides an effective workflow for measuring the blood flow velocity of nailfold capillaries., (Creative Commons Attribution license.)

We propose an algorithm for generating a panoramic image of a pipe's inner surface based on inverse perspective mapping (IPM). The objective of this study is to generate a panoramic image of the entire inner surface of a pipe for efficient crack detection, without relying on high-performance capturing equipment. Frontal images taken while passing through the pipe were converted to images of the inner surface of the pipe using IPM. We derived a generalized IPM formula that considers the slope of the image plane to correct the image distortion caused by the tilt of the plane; this IPM formula was derived based on the vanishing point of the perspective image, which was detected using optical flow techniques. Finally, the multiple transformed images with overlapping areas were combined via image stitching to create a panoramic image of the inner pipe surface. To validate our proposed algorithm, we restored images of pipe inner surfaces using a 3D pipe model and used these images for crack detection. The resulting panoramic image of the internal pipe surface accurately demonstrated the positions and shapes of cracks, highlighting its potential utility for crack detection using visual inspection or image-processing techniques.

The aerial interception behaviour of falcons is well modelled by a guidance law called proportional navigation, which commands steering at a rate proportional to the angular rate of the line-of-sight from predator to prey. Because the line-of-sight rate is defined in an inertial frame of reference, proportional navigation must be implemented using visual-inertial sensor fusion. By contrast, the aerial pursuit behaviour of hawks chasing terrestrial targets is better modelled by a mixed guidance law combining information on the line-of-sight rate with information on the deviation angle between the attacker's velocity and the line-of-sight. Here we ask whether this behaviour may be controlled using visual information alone. We use high-speed motion capture to record n = 228 flights from N = 4 Harris' hawks Parabuteo unicinctus , and show that proportional navigation and mixed guidance both model their trajectories well. The mixed guidance law also models the data closely when visual-inertial information on the line-of-sight rate is replaced by visual information on the motion of the target relative to its background. Although the visual-inertial form of the mixed guidance law provides the closest fit, all three guidance laws provide an adequate phenomenological model of the behavioural data, whilst making different predictions on the physiological pathways involved.

This paper focuses on moving-target detection and tracking in a three-dimensional (3D) space, and proposes a visual target tracking system only using a two-dimensional (2D) camera. To quickly detect moving targets, an improved optical flow method with detailed modifications in the pyramid, warping, and cost volume network (PWC-Net) is applied. Meanwhile, a clustering algorithm is used to accurately extract the moving target from a noisy background. Then, the target position is estimated using a proposed geometrical pinhole imaging algorithm and cubature Kalman filter (CKF). Specifically, the camera's installation position and inner parameters are applied to calculate the azimuth, elevation angles, and depth of the target while only using 2D measurements. The proposed geometrical solution has a simple structure and fast computational speed. Different simulations and experiments verify the effectiveness of the proposed method.

Reconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately perceive anomalous activities. In this paper, we exploit the idea of a training model based on the "Cloze Test" strategy in natural language processing (NLP) and introduce a novel unsupervised learning framework to encode both motion and appearance information at an object level. Specifically, to store the normal modes of video activity reconstructions, we first design an optical stream memory network with skip connections. Secondly, we build a space-time cube (STC) for use as the basic processing unit of the model and erase a patch in the STC to form the frame to be reconstructed. This enables a so-called "incomplete event (IE)" to be completed. On this basis, a conditional autoencoder is utilized to capture the high correspondence between optical flow and STC. The model predicts erased patches in IEs based on the context of the front and back frames. Finally, we employ a generating adversarial network (GAN)-based training method to improve the performance of VAD. By distinguishing the predicted erased optical flow and erased video frame, the anomaly detection results are shown to be more reliable with our proposed method which can help reconstruct the original video in IE. Comparative experiments conducted on the benchmark UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate AUROC scores reaching 97.7%, 89.7%, and 75.8%, respectively.

Fetal movement (FM) is an important indicator of fetal health. However, the current methods of FM detection are unsuitable for ambulatory or long-term observation. This paper proposes a non-contact method for monitoring FM. We recorded abdominal videos from pregnant women and then detected the maternal abdominal region within each frame. FM signals were acquired by optical flow color-coding, ensemble empirical mode decomposition, energy ratio, and correlation analysis. FM spikes, indicating the occurrence of FMs, were recognized using the differential threshold method. FM parameters including number, interval, duration, and percentage were calculated, and good agreement was found with the manual labeling performed by the professionals, achieving true detection rate, positive predictive value, sensitivity, accuracy, and F1&#95;score of 95.75%, 95.26%, 95.75%, 91.40%, and 95.50%, respectively. The changes in FM parameters with gestational week were consistent with pregnancy progress. In general, this study provides a novel contactless FM monitoring technology for use at home.

Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or "spikes", when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency, and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2023 Cuadrado, Rançon, Cottereau, Barranco and Masquelier.)

Increasing violence in workplaces such as hospitals seriously challenges public safety. However, it is time- and labor-consuming to visually monitor masses of video data in real time. Therefore, automatic and timely violent activity detection from videos is vital, especially for small monitoring systems. This paper proposes a two-stream deep learning architecture for video violent activity detection named SpikeConvFlowNet. First, RGB frames and their optical flow data are used as inputs for each stream to extract the spatiotemporal features of videos. After that, the spatiotemporal features from the two streams are concatenated and fed to the classifier for the final decision. Each stream utilizes a supervised neural network consisting of multiple convolutional spiking and pooling layers. Convolutional layers are used to extract high-quality spatial features within frames, and spiking neurons can efficiently extract temporal features across frames by remembering historical information. The spiking neuron-based optical flow can strengthen the capability of extracting critical motion information. This method combines their advantages to enhance the performance and efficiency for recognizing violent actions. The experimental results on public datasets demonstrate that, compared with the latest methods, this approach greatly reduces parameters and achieves higher inference efficiency with limited accuracy loss. It is a potential solution for applications in embedded devices that provide low computing power but require fast processing speeds.

Material models are required to solve continuum mechanical problems. These models contain parameters that are usually determined by application-specific test setups. In general, the theoretically developed models and, thus, the parameters to be determined become increasingly complex, e.g., incorporating higher-order motion derivatives, such as the strain or strain rate. Therefore, the strain rate behaviour needs to be extracted from experimental data. Using image data, the most-common way in solid experimental mechanics to do so is digital image correlation. Alternatively, optical flow methods, which allow an adaption to the underlying motion estimation problem, can be applied. In order to robustly estimate the strain rate fields, an optical flow approach implementing higher-order spatial and trajectorial regularisation is proposed. Compared to using a purely spatial variational approach of higher order, the proposed approach is capable of calculating more accurate displacement and strain rate fields. The procedure is finally demonstrated on experimental data of a shear cutting experiment, which exhibited complex deformation patterns under difficult optical conditions.

Hierarchical processing requires activity propagating between higher- and lower-order cortical areas. However, functional neuroimaging studies have chiefly quantified fluctuations within regions over time rather than propagations occurring over space. Here, we leverage advances in neuroimaging and computer vision to track cortical activity propagations in a large sample of youth (n = 388). We delineate cortical propagations that systematically ascend and descend a cortical hierarchy in all individuals in our developmental cohort, as well as in an independent dataset of densely sampled adults. Further, we demonstrate that top-down, descending hierarchical propagations become more prevalent with greater demands for cognitive control as well as with development in youth. These findings emphasize that hierarchical processing is reflected in the directionality of propagating cortical activity and suggest top-down propagations as a potential mechanism of neurocognitive maturation in youth., Competing Interests: Declaration of interests R.T.S. reports consulting income from Octave Biosciences and has received compensation for scientific reviewing from the American Medical Association., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Regularization is an important technique for training deep neural networks. In this paper, we propose a novel shared-weight teacher-student strategy and a content-aware regularization (CAR) module. Based on a tiny, learnable, content-aware mask, CAR is randomly applied to some channels in the convolutional layers during training to be able to guide predictions in a shared-weight teacher-student strategy. CAR prevents motion estimation methods in unsupervised learning from co-adaptation. Extensive experiments on optical flow and scene flow estimation show that our method significantly improves on the performance of the original networks and surpasses other popular regularization methods. The method also surpasses all variants with similar architectures and the supervised PWC-Net on MPI-Sintel and on KITTI. Our method shows strong cross-dataset generalization, i.e., our method solely trained on MPI-Sintel outperforms a similarly trained supervised PWC-Net by 27.9% and 32.9% on KITTI, respectively. Our method uses fewer parameters and less computation, and has faster inference times than the original PWC-Net.

The spatial and temporal evolution of the field backscattered by a beating heart while illuminated with a coherent light reveals its macro- and microvascularization in real time. To perform these vascularization images, we use a recently published method of laser speckle imaging, based on the selective detection of spatially depolarized speckle field that is mainly generated by multiple scattering. We consider the calculation of the speckle contrast, by a spatial or temporal estimation. We show that the signal-to-noise ratio of the observed vascular structure can be noticeably increased by a postprocessing method implying the calculation of a motion field that allows the selection of similar frames extracted from different heartbeat periods. This later optimization reveals vascular microstructures with a spatial resolution of the order of 100    μ m ., (© 2023 The Authors.)

FIB-SEM (Focused Ion Beam-Scanning Electron Microscopy) is an imaging technique that allows 3D ultrastructural analysis of cells and tissues at the nanoscale. The acquired FIB-SEM data are highly noisy, which makes denoising an essential step prior to volume interpretation. Gaussian filtering is a standard method in the field because it is fast and straightforward. However, it tends to blur the biological features due to its linear nature that ignores the rapid changes of the structures throughout the volume. To address this issue, we have developed a new approach to structure-preserving noise reduction for FIB-SEM. It has abilities to locally adapt the filtering to the biological structures while taking advantage of the simplicity of Gaussian filtering. It uses the Optical Flow (OF) to estimate the variations of the structural features across the volume, so that they are compensated before the subsequent filtering with a Gaussian function. As demonstrated qualitatively and objectively with datasets from different samples and acquired under different conditions, our denoising approach outperforms the standard Gaussian filtering and is competitive with state-of-the-art methods in terms of noise reduction and preservation of the sharpness of the structures., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2022 The Author(s). Published by Elsevier B.V. All rights reserved.)

Humans show micro-expressions (MEs) under some circumstances. MEs are a display of emotions that a human wants to conceal. The recognition of MEs has been applied in various fields. However, automatic ME recognition remains a challenging problem due to two major obstacles. As MEs are typically of short duration and low intensity, it is hard to extract discriminative features from ME videos. Moreover, it is tedious to collect ME data. Existing ME datasets usually contain insufficient video samples. In this paper, we propose a deep learning model, double-stream 3D convolutional neural network (DS-3DCNN), for recognizing MEs captured in video. The recognition framework contains two streams of 3D-CNN. The first extracts spatiotemporal features from the raw ME videos. The second extracts variations of the facial motions within the spatiotemporal domain. To facilitate feature extraction, the subtle motion embedded in a ME is amplified. To address the insufficient ME data, a macro-expression dataset is employed to expand the training sample size. Supervised domain adaptation is adopted in model training in order to bridge the difference between ME and macro-expression datasets. The DS-3DCNN model is evaluated on two publicly available ME datasets. The results show that the model outperforms various state-of-the-art models; in particular, the model outperformed the best model presented in MEGC2019 by more than 6%.

Due to the flight characteristics such as small size, low noise, and high efficiency, studies on flapping wing robots are being actively conducted. In particular, the flapping wing robot is in the spotlight in the field of search and reconnaissance. Most of the research focuses on the development of flapping wing robots rather than autonomous flight. However, because of the unique characteristics of flapping wings, it is essential to consider the development of flapping wing robots and autonomous flight simultaneously. In this article, we describe the development of the flapping wing robot and computationally efficient vision-based obstacle avoidance algorithm suitable for the lightweight robot. We developed a 27 cm and 45 g flapping wing robot named CNUX Mini that features an X-type wing and tailed configuration to attenuate oscillation caused by flapping motion. The flight experiment showed that the robot is capable of stable flight for 1.5 min and changing its direction with a small turn radius in a slow forward flight condition. For the obstacle detection algorithm, the appearance variation cue is used with the optical flow-based algorithm to cope robustly with the motion-blurred and feature-less images obtained during flight. If the obstacle is detected during straight flight, the avoidance maneuver is conducted for a certain period, depending on the state machine logic. The proposed obstacle avoidance algorithm was validated in ground tests using a testbed. The experiment shows that the CNUX Mini performs a suitable evasive maneuver with 90.2% success rate in 50 incoming obstacle situations., Competing Interests: Hyondong Oh is an Academic Editor for PeerJ., (© 2023 Park et al.)

Benign paroxysmal positional vertigo (BPPV) is the most common vestibular peripheral vertigo disease characterized by brief recurrent vertigo with positional nystagmus. Clinically, it is common to recognize the patterns of nystagmus by analyzing infrared nystagmus videos of patients. However, the existing approaches cannot effectively recognize different patterns of nystagmus, especially the torsional nystagmus. To improve the performance of recognizing different nystagmus patterns, this paper contributes an automatic recognizing method of BPPV nystagmus patterns based on deep learning and optical flow to assist doctors in analyzing the types of BPPV. Firstly, we present an adaptive method for eliminating invalid frames that caused by eyelid occlusion or blinking in nystagmus videos and an adaptive method for segmenting the iris and pupil area from video frames quickly and efficiently. Then, we use a deep learning-based optical flow method to extract nystagmus information. Finally, we propose a nystagmus video classification network (NVCN) to categorize the patterns of nystagmus. We use ConvNeXt to extract eye movement features and then use LSTM to extract temporal features. Experiments conducted on the clinically collected datasets of infrared nystagmus videos show that the NVCN model achieves an accuracy of 94.91% and an F1 score of 93.70% on nystagmus patterns classification task as well as an accuracy of 97.75% and an F1 score of 97.48% on torsional nystagmus recognition task. The experimental results prove that the framework we propose can effectively recognize different patterns of nystagmus., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2022. Published by Elsevier Ltd.)