Search

Lacking of gesture datasets for the scenarios with multi-person, gesture detection for multi-person is generally used two-stage schemes, thus leading to high computation and time cost. In this paper, we propose a method to automatically create a multi-person gesture dataset, which can quickly generate gestures with annotation information without manual annotation in complex scenarios. Afterwards, we treat different gestures as different categories and use a one-stage approach to detect and classify all the gestures in multi-person scenarios in a single run. Among the popular object detection backbones, the one that performs best on our dataset is pruned for the purpose of computational efficiency, knowledge distillation is further performed on the pruned model to mitigate the accuracy decline caused by pruning, and focal loss function is optimized for improving model performance. Experiments show that our approach significantly reduces the processing time for gesture detection in multi-person scenarios, while just slightly sacrificing the prediction accuracy. In a multi-person scenario with ten hands, the inference time of our algorithm is only 12 ms, outperforming the off-the-shelf methods by at least 4.3 times.

Benign paroxysmal positional vertigo (BPPV) is the most common vestibular peripheral vertigo disease characterized by brief recurrent vertigo with positional nystagmus. Clinically, it is common to recognize the patterns of nystagmus by analyzing infrared nystagmus videos of patients. However, the existing approaches cannot effectively recognize different patterns of nystagmus, especially the torsional nystagmus. To improve the performance of recognizing different nystagmus patterns, this paper contributes an automatic recognizing method of BPPV nystagmus patterns based on deep learning and optical flow to assist doctors in analyzing the types of BPPV. Firstly, we present an adaptive method for eliminating invalid frames that caused by eyelid occlusion or blinking in nystagmus videos and an adaptive method for segmenting the iris and pupil area from video frames quickly and efficiently. Then, we use a deep learning-based optical flow method to extract nystagmus information. Finally, we propose a nystagmus video classification network (NVCN) to categorize the patterns of nystagmus. We use ConvNeXt to extract eye movement features and then use LSTM to extract temporal features. Experiments conducted on the clinically collected datasets of infrared nystagmus videos show that the NVCN model achieves an accuracy of 94.91% and an F1 score of 93.70% on nystagmus patterns classification task as well as an accuracy of 97.75% and an F1 score of 97.48% on torsional nystagmus recognition task. The experimental results prove that the framework we propose can effectively recognize different patterns of nystagmus.

An iterative deviation filter for fixed-valued impulse noise removal is proposed, with the aim to overcome the defects of existing filters, and further improve the denoising performance. In the proposed filter, a noise detection method based on the extreme intensity values and the deviation of neighbor pixels is proposed, i.e., the pixels with the extreme intensity and differ greatly from the mean of neighbor pixels, are identified as noises. A noise removal method based on the minimum deviation of neighbor pixels is proposed, i.e., the intensity of one neighbor noise free pixel, which is closest to the mean of neighbor noise free pixels, is used as estimated intensity of noisy pixel under consideration. Furthermore, the noise removal strategy performs iteratively and takes full advantage of the previous denoising results. Simulation results show that the proposed method has better denoising performance than the existing distinguished filters in terms of visual representation, peak signal to noise ratio and structural similarity index.

In this paper, we propose a gaze-based interactive system for metro map named GazeMetro, which helps explore and interact with the metro map only by eye movements, providing a new experience of interaction. The objective of GazeMetro is to provide metro map viewers with gaze-based interactions to search the metro map without other manual operations. We implement GazeMetro with 4 gaze-based interaction techniques which are Gaze Fisheye, Gaze Scaling and Panning, Gaze Selection and Gaze Hint. We conducted an experiment to evaluate GazeMetro and the results showed a positive evaluation in pragmatic quality and especially in hedonic quality.

The automatic and accurate arrhythmia diagnosis in the electrocardiogram (ECG) signals is significant for cardiac health. Typically, the arrhythmia diagnosis is automatically detected depending on single-lead signals or a simple combination of multilead signals from the ECG. However, it ignores the inter-lead correlation and the significance of different leads for different heart beats detection, which decreases the performance of arrhythmia diagnosis. In this paper, arrhythmia diagnosis is converted to a problem of multigranulation computing in the view of granular computing, and thus different lead signals can be captured to improve the effectiveness of abnormal heart beats detection. To this end, multilead ECG signals are firstly granulated into different fuzzy information granules by the fuzzy equivalence relation. An objective decision-making model based on fuzzy set theory is then proposed for describing and analyzing these granulated multilead ECG signals, which brings a self-adaptive and unsupervised decision making. As a result, the significance and correlation of different leads are analyzed by granularity selection and granular structures to make a better decision for arrhythmia diagnosis. Extensive experimental results show that the proposed algorithm can significantly improve the performance of arrhythmia diagnosis, especially better robustness to several types of cardiac arrhythmia.

Due to the limitation of existing filters in detection and removal of fixed value impulse noise, the authors propose an iterative grouping median filter (IGMF) according to the characteristics of noise intensity and distribution. It sorts the noise-free pixels in neighbourhood by intensity, divides the sorted pixels into groups depending on the intensity differences of adjacent pixels, and finally takes the median of the maximum group as the intensity of noisy pixel. This noise removal strategy is performed iteratively and takes full advantage of the previous denoising results. Experiments show that IGMF outperforms the existing state-of-the-art filters in terms of visual perception, peak signal to noise ratio and structural similarity index at various noise densities.

Radiotherapy is a major treatment for head and neck cancer. Currently, computed tomography (CT) is utilized to delineate the target area and make radiotherapy plans. Compared with CT images, magnetic resonance (MR) images have excellent soft tissue contrast, which can distinguish normal surrounding tissues. It is necessary to register MR and CT images since it helps physicists to improve the accuracy of radiotherapy plans. Most of the current registrations require manual intervention to select regions of interest, which increases the workload of doctors and the time of registration to a certain extent. In this paper, an automatic registration method is proposed to delineate the regions of interest. Herein, surface meshes are extracted from the CT and MR images and utilized to perform the surface matching, then the regions of interest are extracted automatically by calculating the overlapped regions between the surface mesh of the surface-aligned MR images and CT images. Furthermore, a multi-level resolution registration mechanism is utilized to improve the registration speed. Surface matching is performed using low-resolution images to obtain transformation parameters as initial parameters for rigid registration followed by deformable registration. The experiments demonstrate that our proposed method performs better in the registration speed and accuracy over the conventional methods of delineating regions of interest manually.

This paper aims to examine how the trajectory dimension influences sensorimotor control during arm tracking. We designed three trajectories with different dimensions in a three-dimensional (3D) immersive virtual reality environment and instructed the subjects to control a virtual hand to follow a cubic target that moved along the designed trajectories. The position of the virtual hand was determined by the position of the actual hand captured with a high-resolution 3D motion capture system in real time. Five kinematic measures were calculated: the root mean square error (RMSE), the standard deviation of the speed (speedsd), the magnitude of the jerk (Jerkm), the integral of the speed power spectrum (IVPS), and the 3D fuzzy approximate entropy (fApEn3D). All the kinematic measures increased significantly with increasing trajectory dimensions, except for the IVPS between the 1D and 2D conditions and the fApEn3D between the 2D and 3D conditions. The increase in time-domain parameters (i.e., RMSE, speedsd, and Jerkm) showed degradation in accuracy, energy efficiency, and multijoint coordination, respectively, in the higher dimensions. An increase in the frequency-domain measure (i.e., IVPS) in higher dimensional condition reflected an increase of visual feedback-related intermittency in manual control when increasing the trajectory dimension. The larger nonlinear fApEn3D values in the 2D and 3D conditions might have been due to the higher level neuromotor noise and increased sensory inputs. The selected parameters could provide a comprehensive method for evaluating motor performance from different perspectives. The findings in this paper shed light on the underlying sensorimotor control that is caused by the trajectory dimension in arm tracking tasks.

This paper is concerned with a novel extrinsic calibration method consisting of a camera and a LiDAR. A proposed planer calibrator is designed for highly effective calibration together with a feature detection algorithm for extracting the landmark features on the planer calibrator, which are adopted to derive the projection matrix equation between the camera and the LiDAR coordinate systems. The calibration result is further optimized via the cross-correlation equation among the planar calibrator poses in the sensor data frames. Compared to the state-of-the-art extrinsic calibration approaches between a camera and a LiDAR, the proposed approach achieves competitive calibration accuracy and robustness.

Outdoor illumination estimation is particularly essential for augmented reality applications. Traditionally, handcraft illumination features are used to recover outdoor illumination such as shadows and highlight areas of the scenes. These features are so dependent on human experience and auxiliary equipments that the performances of these approaches are limited. This paper presents a lightweight network based on MobileNet V3 to estimate outdoor illumination from a single outdoor image. A physically-based illumination model is fitted to the sky regions of outdoor panoramas to generate illumination parameters, and different field of view images are extracted from outdoor panoramas. The images with illumination parameters annotated are then used to train our lightweight network. Experiments show that our method can perform well and is suitable for mobile-based applications.

Eye movement event detection is a demanding technique in cognitive behavior analysis and HCI. Since an eye movement trajectory is a natural path, we try to introduce path signature (PS) to better explore eye movement events; PS is a feature that can highly summarize path information. For this, a multi-input network (MINN) combining 1D-CNN and bidirectional long short-term memory (BiLSTM) is constructed to classify gaze samples as fixation, saccade, smooth pursuit or noise. MINN requires two inputs of local features and global features respectively. The local features include the speed and direction of the gaze trajectory and the global features are the PS’s of the gaze trajectory. Experiments on GazeCom, the biggest eye movement event detection dataset, show that our approach with PS outperforms the state-of-the-art methods that do not use PS.

With the rapid development of VR head mounted display, VR scenes need to be presented at higher resolutions and higher frame rates. Foveated rendering is an important technique to balance computing resource demand and user satisfaction. In this paper, we propose a perceptual model based foveated rendering. By culling the imperceptible region and adaptively adjusting the tessellation levels based on visual sensitivity, we improve the rendering performance in real-time rendering frameworks.

Polar Fourier analysis can extract orthogonal and rotation invariant features and demonstrate superior performances in many image processing tasks. For real world applications, execution efficiency is always a significant challenge. With widespread use of Graphics Processing Unit (GPU), this research presents GPU accelerated polar Fourier analysis. Proposed parallel algorithm is based on mathematical properties of polar Fourier analysis and optimization techniques of GPU. Optimal parameter selections for GPU execution are also evaluated. In our experiments, with same computation result proposed method is over 170 times faster. Wide range of applications that using polar Fourier analysis will inspired from this work.

Eye tracking is a valuable topic in computer vision. Appearance-based eye tracking is a promising research direction in recent years. Convolutional neural networks (CNN) had been used in gaze estimation, which cover the significant variability in eye appearance caused by unconstrained head motion. With computation capability of consumer devices rapidly evolving, accurate and efficient appearance-based eye tracking has the potential for multipurpose applications. Person-independent networks have limit in improving gaze estimation accuracy. Person-specific network with calibration is more effective than person-independent approaches. Unlike classical eye tracking methods, appearance-based eye tracking has not a clear way to calibration. Our goal is to analyze the impact of calibration data selection and calibration target distribution on person-specific gaze estimation accuracy. We trained person-independent network and use SVR to calibration. We choose two kind of typical distribution targets to evaluation. Use different distribution targets to calibration achieves different accuracy.

Angular radial transform (ART) is a shape descriptor in MPEG-7 for image representation and image analysis. For real world applications, execution efficiency is always a significant challenge. With widespread use of Graphics Processing Unit (GPU) in image processing tasks, this paper presents GPU based angular radial transform (GART). Proposed parallel GART is based on mathematical properties of ART and optimization techniques of GPU. Optimal parameter selections for GPU execution are also evaluated. In our experiments, with same computation result GART is over 1800 times faster than ART. Wide range of applications that using ART will inspired from this work.

To overcome the drawbacks of existing filters for salt and pepper noises, an adaptive probability filter is proposed. For an image, it detects salt and pepper noises based on the characteristic of minimum and maximum intensity values of the images, as well as the distribution of noise. If the noise-free intensities in neighbourhood repeat with a certain probability, the noise-free intensity with highest repeated frequency is used to remove noise based on the statistical significance; otherwise, the median of noise-free pixels in neighbourhood is used to remove noise. Experiments show that the proposed method is capable of detecting noise more accurately and perform much better than the existing distinguished filters in terms of peak-signal-to-noise ratio, image enhancement factor, and visual representation at all the noise densities.