Search

In this study, we address the challenge of 3D human motion prediction from motion capture data, which has become critical in various applications such as autonomous vehicles and human-robot interaction. Previous deep learning-based methods have improved prediction accuracy, but require significant network parameters and do not effectively consider independent joint movements. To overcome the limitations, we propose two lightweight network structures for human motion prediction: LG-Net and LGT-Net, which focus on the individual movements of distinct human limbs and their inter-dependencies. The LG-Net comprises local and global networks, while the LGT-Net combines the proposed LG-Net structure with Long and Short Term Memory (LSTM) cells to exploit temporal information. Our networks, designed to be extremely lightweight with only 0.08M and 0.5M parameters, achieve higher prediction performance compared to state-of-the-art methods. In addition, this study is the first to consider the root joint to improve motion prediction performance. The proposed approach demonstrates the potential for efficient and accurate human motion prediction in various applications.

Memory and language are the two higher-order cognitive abilities intertwined for communication and other cognitive skills. Memory is the storage capacity of all the information we perceive. Where the sensory memory perceives the stimuli, the working memory actively stores the information and passes it to the long-term memory. However, there is a question that how is the continuous perception of stimuli transformed into meaningful information and organized for proper execution and retrieval from the memory? This paper focuses on the episodic memory that perceives information that is spatial and temporal based on our everyday experiences. Though the spatiotemporal information we receive is continuous; the episodic memory arranges the information as to episodes in the working memory before the information is stored for a longer period. The episodic buffer is one of the components of the working memory model which holds the episodic memory that is organized concerning time. To this point, the paper tries to understand the working of the episodic buffer in maintaining the episodic memory and also about the process of episodic events into meaningful units. Further, the paper also concentrates on the hippocampus which is considered to be the location of the episodic buffer. [ABSTRACT FROM AUTHOR]

Remote sensing satellites have been broadly applied to sea ice monitoring. The substantial increase in satellite imagery provides a large amount of data support for deep learning methods in the sea ice classification field. However, there is a lack of public remote sensing datasets to facilitate sea ice classification with spatial and temporal information and to benchmark the deep learning methods. In this paper, we provide a labeled large sea ice dataset derived from time-series sentinel-1 SAR images, dubbed SI-STSAR-7, and a validated dataset construction method for sea ice classification research. The SI-STSAR-7 dataset includes seven different sea ice types corresponding to different sea ice development stages in Hudson Bay during winter, and its samples are time sequences of SAR image patches in order to embody the differences of backscattering intensity and textures between different sea ice types, as well as the change of sea ice with time. We construct the dataset by first performing noise reduction and mitigation of incidence angle dependence on SAR images, and then producing data samples and labeling them based on our proposed sample-producing principles and the weekly regional ice charts provided by Canadian Ice Service. Three baseline classification methods are developed on SI-STSAR-7 to establish benchmarks, which are evaluated with accuracy and kappa coefficient. The sample-producing principles are verified through experiments. Based on the experimental results, sea ice classification can be implemented well on SI-STSAR-7.

In this letter, we propose a joint processing method of spatial and temporal information from L-shaped sparse array (LSA) for two-dimensional (2-D) direction of arrival (DOA) estimation. In particular, either of the two subarrays in LSA can be any type of linear sparse array. With the proposed method, we can construct a virtual planar sparse array first, whose corresponding 2-D difference coarray owns significantly increased degrees of freedom, which can help to enhance the 2-D DOA estimation performance. Thereafter, subspace-based method with 2-D spatial smoothing can be applied to achieve the automatically paired and unambiguous 2-D DOA estimation. Simulation results verify the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Face-based biometric recognition systems that can recognize human faces are widely employed in places such as airports, immigration offices, and companies, and applications such as mobile phones. However, the security of this recognition method can be compromised by attackers (unauthorized persons), who might bypass the recognition system using artificial facial images. In addition, most previous studies on face presentation attack detection have only utilized spatial information. To address this problem, we propose a visible-light camera sensor-based presentation attack detection that is based on both spatial and temporal information, using the deep features extracted by a stacked convolutional neural network (CNN)-recurrent neural network (RNN) along with handcrafted features. Through experiments using two public datasets, we demonstrate that the temporal information is sufficient for detecting attacks using face images. In addition, it is established that the handcrafted image features efficiently enhance the detection performance of deep features, and the proposed method outperforms previous methods.

In this paper, we propose a new and novel modality fusion method designed for combining spatial and temporal fingerprint information to improve video copy detection performance. Most of the previously developed methods have been limited to use only pre-specified weights to combine spatial and temporal modality information. Hence, previous approaches may not adaptively adjust the significance of the temporal fingerprints that depends on the difference between the temporal variances of compared videos, leading to performance degradation in video copy detection. To overcome the aforementioned limitation, the proposed method has been devised to extract two types of fingerprint information: (1) spatial fingerprint that consists of the signs of DCT coefficients in local areas in a keyframe and (2) temporal fingerprint that computes the temporal variances in local areas in consecutive keyframes. In addition, the so-called temporal strength measurement technique is developed to quantitatively represent the amount of the temporal variances; it can be adaptively used to consider the significance of compared temporal fingerprints. The experimental results show that the proposed modality fusion method outperforms other state-of-the-arts fusion methods and popular spatio-temporal fingerprints in terms of video copy detection. Furthermore, the proposed method can save 39.0%, 25.1%, and 46.1% time complexities needed to perform video fingerprint matching without a significant loss of detection accuracy for our synthetic dataset, TRECVID 2009 CCD Task, and MUSCLE-VCD 2007, respectively. This result indicates that our proposed method can be readily incorporated into the real-life video copy detection systems. [ABSTRACT FROM AUTHOR]

The aim of this letter is to present a spatio–temporal pixel-swapping algorithm (STPSA), based on conventional pixel-swapping algorithms (PSAs), in which both spatial and temporal contextual information from previous land cover maps or observed samples are well integrated and utilized to improve subpixel mapping accuracy. Unlike conventional pixel-swapping algorithms, STPSA is capable of utilizing prior information, which was previously ignored, to predict the attractiveness based on pairs of subpixels. This algorithm involves three main steps and operates in an iterative manner: 1) it predicts the maximum and minimum attractiveness of each pair of pixels; 2) ranks the swapping scores based on the attractiveness of all the pairs; and 3) swaps the locations of the pair of pixels with a maximum score to increase the objective function. Experiments with actual satellite images have demonstrated that the proposed algorithm performs better than other algorithms. In comparison, the proposed STPSA's better performance is due to the fact that prior information used in other algorithms is restricted to a percentage level rather than the real subpixel level. [ABSTRACT FROM PUBLISHER]

Remote sensing satellites have been broadly applied to sea ice monitoring. The substantial increase in satellite imagery provides a large amount of data support for deep learning methods in the sea ice classification field. However, there is a lack of public remote sensing datasets to facilitate sea ice classification with spatial and temporal information and to benchmark the deep learning methods. In this paper, we provide a labeled large sea ice dataset derived from time-series sentinel-1 SAR images, dubbed SI-STSAR-7, and a validated dataset construction method for sea ice classification research. The SI-STSAR-7 dataset includes seven different sea ice types corresponding to different sea ice development stages in Hudson Bay during winter, and its samples are time sequences of SAR image patches in order to embody the differences of backscattering intensity and textures between different sea ice types, as well as the change of sea ice with time. We construct the dataset by first performing noise reduction and mitigation of incidence angle dependence on SAR images, and then producing data samples and labeling them based on our proposed sample-producing principles and the weekly regional ice charts provided by Canadian Ice Service. Three baseline classification methods are developed on SI-STSAR-7 to establish benchmarks, which are evaluated with accuracy and kappa coefficient. The sample-producing principles are verified through experiments. Based on the experimental results, sea ice classification can be implemented well on SI-STSAR-7. [ABSTRACT FROM AUTHOR]

• Utilizing temporal information as well as spatial information. • Utilizing pre-beam-formed data instead of beam-formed radio frequency (RF) data. • CNR ans SNR of the estimated strain fields are improved. • Robustness of the alstograms against noise is also increased. This paper introduces a novel technique to estimate tissue displacement in quasi-static elastography. A major challenge in elastography is estimation of displacement (also referred to time-delay estimation) between pre-compressed and post-compressed ultrasound data. Maximizing normalized cross correlation (NCC) of ultrasound radio-frequency (RF) data of the pre- and post-compressed images is a popular technique for strain estimation due to its simplicity and computational efficiency. Several papers have been published to increase the accuracy and quality of displacement estimation based on NCC. All of these methods use 2D spatial windows in RF data to estimate NCC, wherein displacement is assumed to be constant within each window. In this work, we extend this assumption along the third dimension. Two approaches are proposed to get third dimension. In the first approach, we use temporal domain to exploit neighboring samples in both spatial and temporal directions. Considering temporal information is important since traditional and ultrafast ultrasound machines are, respectively, capable of imaging at more than 30 frame per second (fps) and 1000 fps. Another approach is to use time-delayed pre-beam formed data (channel data) instead of RF data. In this method information of all channels that are recorded as pre-beam formed data of each RF line will be considered as 3 rd dimension. We call these methods as spatial temporal normalized cross correlation (STNCC) and channel data normalized cross correlation (CNCC) and show that they substantially outperforms NCC using simulation, phantom and in-vivo experiments. Given substantial improvements of results in addition to the relative simplicity of implementing STNCC and CNCC, the proposed approaches can potentially have a large impact in both academic and commercial work on ultrasound elastography. [ABSTRACT FROM AUTHOR]

Nested arrays have recently attracted considerable attention in the field of direction of arrival (DOA) estimation owing to the hole-free property of their virtual arrays. However, such virtual arrays are confined to difference coarrays as only spatial information of the received signals is exploited. By exploiting the spatial and temporal information jointly, four kinds of novel nested arrays based on the sum-difference coarray (SDCA) concept are proposed. To increase the degrees of freedom (DOFs) of SDCA, a modified translational nested array (MTNA) is introduced first. Then, by analyzing the relationship among sensors in MTNA, we give the specific positions of redundant sensors and remove them later. Finally, we derive the closed-form expressions for the proposed arrays as well as their SDCAs. Meanwhile, different index sets corresponding to the proposed arrays are also designed for their use in obtaining the desirable SDCAs. Moreover, the properties regarding DOFs of SDCAs and physical apertures for the proposed arrays are analyzed, which prove that both the DOFs and physical apertures are improved. Simulation results are provided to verify the superiority of the proposed arrays. [ABSTRACT FROM AUTHOR]

The temporal super-resolution of the dynamic ultrasound imaging, a means to observe rapid heart movements, is considered an important subject in medical diagnosis of cardiac conditions. Here, a new technique based on the acquisition scheme using the matrix completion (MC) theory is offered for the temporal super-resolution of the two-dimensional (2D) and three-dimensional (3D) ultrasound imaging. MC mentions the problem of completing a low-rank matrix when only a subset of its elements can be observed. Here, the lower scan lines are acquired. Whereby, the proposed method uses temporal and spatial information of the radio frequency (RF) image sequences for the reconstruction of skipped RF lines. This is performed using the construction of the MC images and then reconstruction of them by the MC theory. The results of the proposed method are compared with the compressive sensing (CS) reconstruction methods. The qualitative and quantitative evaluations of 2D and 3D data demonstrate that in the proposed method, which uses the spatial and temporal relation of RF images and the MC theory, the reconstruction is more accurate, and the reconstruction error is lower. The computational complexity of this method is very low. It also does not require hardware adjustments. Therefore, it can be easily implemented in current ultrasound-imaging devices with the frame-rate enhancement. For instance, the frame rate up to two times the original sequence is feasible using the proposed methods, while root mean square error is decreased by about 35% and 30% for 2D and 3D data, respectively, compared with the CS reconstruction method.

Face-based biometric recognition systems that can recognize human faces are widely employed in places such as airports, immigration offices, and companies, and applications such as mobile phones. However, the security of this recognition method can be compromised by attackers (unauthorized persons), who might bypass the recognition system using artificial facial images. In addition, most previous studies on face presentation attack detection have only utilized spatial information. To address this problem, we propose a visible-light camera sensor-based presentation attack detection that is based on both spatial and temporal information, using the deep features extracted by a stacked convolutional neural network (CNN)-recurrent neural network (RNN) along with handcrafted features. Through experiments using two public datasets, we demonstrate that the temporal information is sufficient for detecting attacks using face images. In addition, it is established that the handcrafted image features efficiently enhance the detection performance of deep features, and the proposed method outperforms previous methods. [ABSTRACT FROM AUTHOR]