Search

With advancements in remote sensing technology, synthetic aperture radar (SAR) imagery has become one of the main methods to detect surface water bodies. The detection of water bodies in SAR imagery remains a challenging task due to the presence of complex interference. To achieve accurate water body detection, we proposed an attention fusion U-net inspired by the effectiveness of U-net in segmenting small targets with weak edges. First, the spatial attention module and channel attention module are added to the skip connections between encoder and decoder parts to extract useful low- and high-level features, thereby compensating for the loss of semantic information of downsampling. Second, the multiscale convolutional pooling block is introduced into the encoder part to better utilize the contextual information, capturing water and land features at different scales. Third, considering the feature distortion resulting from upsampling, an attentional upsampler (AU) is designed to facilitate lossless feature fusion. Furthermore, an active contour loss is designed as additional regularization to learn more boundary information, improving the model's segmentation performance. The water body detection experiments on the ALOS phased array L-band SAR and Sen1-SAR datasets demonstrate that the presented AFUNet outperforms the related start-of-the-art methods in detection accuracy in terms of five evaluation metrics.

Video compression has become increasingly crucial as video resolution and bitrate have surged in recent years. However, most widely applied video compression methods do not fully exploit the characteristics of the Human Visual System (HVS) to reduce perceptual redundancy in videos. In this paper, we propose a novel video compression method that integrates visual saliency information with foveation to reduce perceptual redundancy. We present a new approach to subsample and restore the input image using saliency data, which allocates more space for salient regions and less for non-salient ones. We analyze the information entropy in video frames before and after applying our algorithm and demonstrate that the proposed method reduces redundancy. Through subjective and objective evaluations, we show that our method produces videos with superior perceptual visual quality. Moreover, our approach can be added to most existing video compression standards without altering their bitstream format.

Convolutional neural networks have demonstrated impressive results in many computer vision tasks. However, the increasing size of these networks raises concerns about the information overload resulting from the large number of network parameters. In this paper, we propose Frequency Regularization to restrict the non-zero elements of the network parameters in the frequency domain. The proposed approach operates at the tensor level, and can be applied to almost all network architectures. Specifically, the tensors of parameters are maintained in the frequency domain, where high-frequency components can be eliminated by zigzag setting tensor elements to zero. Then, the inverse discrete cosine transform (IDCT) is used to reconstruct the spatial tensors for matrix operations during network training. Since high-frequency components of images are known to be less critical, a large proportion of these parameters can be set to zero when networks are trained with the proposed frequency regularization. Comprehensive evaluations on various state-of-the-art network architectures, including LeNet, Alexnet, VGG, Resnet, ViT, UNet, GAN, and VAE, demonstrate the effectiveness of the proposed frequency regularization. For a very small accuracy decrease (less than 2%), a LeNet5 with 0.4M parameters can be represented by only 776 float16 numbers (over $1100\times $ reduction), and a UNet with 34M parameters can be represented by only 759 float16 numbers (over $80000\times $ reduction). In particular, the original size of the UNet model is reduced from 366 Mb to 4.5 Kb.

Hand Gesture Recognition (HGR) plays a crucial role in user-friendly interactions between humans and computers. In recent years, using the Convolutional Neural Network (CNN) has improved the accuracy of image processing problems. Inspired by the high recognition rate of CNN and its efficiency, we propose a model for hand gesture recognition based on CNN and evaluate the results using images with plain and complex backgrounds. Recognizing different hand signs by Two-Dimensional Parallel Spatio-Temporal Pyramid Pooling (2DPSTPP) features with deep learning methods reduces the size of the map, minimizes training complexity, and by paying attention to more details, improves detection performance. The effectiveness of the proposed method is evaluated using regular cross-validation tests on six datasets, namely American Sign Language (ASL), the NUS hand posture dataset I, the NUS hand posture dataset II, the digits dataset, the hand gesture dataset, and the leap gesture recognition dataset.

Abstract Fire detection methods based on the Convolutional Neural Networks (CNN) have advantages of high accuracy, wide coverage and robustness, receiving significant attention from researchers. Among CNN‐based methods, ResNet has achieved better performance than other CNN frameworks in fire detection system, since it uses stacked residual blocks to enlarge the receptive field to overcome the vanishing gradient problem with residual learning. The merits of ResNet can be attributed to the similarity between ResNet and the single‐step explicit solver for Ordinary Differential Equations (ODEs), for example, the Euler method. Motivated by the theory of numerical ODE that a multi‐step implicit solver has higher accuracy than a single‐step explicit solver, the Multi‐step Implicit Adams predictor‐corrector (MIAPC) network for fire detection is proposed. The MIAPC method is first mapped to a corresponding predictor‐corrector Adams block which achieves higher accuracy than a single‐step explicit solver. Then, Adaptive Feature Fusion (AFF) and the Spatial Attention Layer (SAL) are utilized to extract hierarchical features from stacked predictor‐corrector Adams blocks, forming the corresponding Adams module. Finally, the 4 Adams modules which are made of 4, 6, 8, 10 predictor‐corrector Adams blocks and followed by AFF and SAL form the crucial ODE‐based approximation part in the proposed network. By adding a simple feature extraction and detection in front of and after the ODE‐based approximation part, the MIAPC network is built. Experiments demonstrate that the method achieves 87% accuracy in the challenging test dataset, outperforming existing methods by at least 6%. Besides, the 5.3M model size with inference speed of 4.7 frames/second in CPU and 65.7 frames/second in GPU enables the proposed method to be used in practical applications.

Hand gesture recognition is a vital means of communication to convey information between humans and machines. We propose a novel model for hand gesture recognition based on computer vision methods and compare results based on images with complex scenes. While extracting skin color information is an efficient method to determine hand regions, complicated image backgrounds adversely affect recognizing the exact area of the hand shape. Some valuable features like saliency maps, histogram of oriented gradients (HOG), Canny edge detection, and skin color help us maximize the accuracy of hand shape recognition. Considering these features, we proposed an efficient hand posture detection model that improves the test accuracy results to over 99% on the NUS Hand Posture Dataset II and more than 97% on the hand gesture dataset with different challenging backgrounds. In addition, we added noise to around 60% of our datasets. Replicating our experiment, we achieved more than 98% and nearly 97% accuracy on NUS and hand gesture datasets, respectively. Experiments illustrate that the saliency method with HOG has stable performance for a wide range of images with complex backgrounds having varied hand colors and sizes.

To achieve river channel detection in SAR (synthetic aperture radar) images, we developed a level-set-based model (LSBM) guided by a designed data-fitting energy which is called the SoDEF (sum of dual exponential functions)-fitting energy. Firstly, we designed a function by computing the sum of dual exponential functions to substitute for the quadratic function, and used it to construct the data-fitting energy. Secondly, the adaptive area-fitting centers (AFCs) were computed based on two kinds of grayscale characteristics, which are more accurate and more stable. Thirdly, the Dirac function in gradient descent flow was displaced by an edge indicator function to help the evolving level sets stop at the target edges. Moreover, some regularized terms were incorporated into the objective function to guarantee the model’s stability. The river channel detection experiments conducted with real SAR images indicated that the developed model is superior to the related state-of-the-art methods in its detection accuracy and efficiency.

Given the imaging characteristics of synthetic aperture radar (SAR) images and the inherent speckle noise in them, scale-invariant feature transform based algorithms are unable to perform satisfactorily. To improve registration efficiency between SAR images, we propose a robust and efficient registration method with three main contributions. First, considering sudden dark patches appearing in SAR images, we propose the ratio of exponentially weighted average blocks to suppress the sudden dark patches and better adapt to different test images. This new operator called blocks of the ratio of exponentially weighted averages (ROEWA-B) divides the processing windows of ROEWA into blocks, which can not only reduce speckle noise but also retain more edge details compared to ROEWA when sudden dark patches appear. Second, for outlier removal, we present an approach using the minimum moment map to remove erroneous keypoints. Finally, based on the gradient location orientation histogram descriptor, we propose a novel multiscale circle descriptor, which combines scale change information to give weights to feature points at different scales. Experimental results for various thresholds and evaluations demonstrate the advantage and robustness of our method in registration.

The quality of a video clip is considered to be poor if the resolution or the frame rate is low. Video interpolation is thus introduced to enhance video quality and provide a better viewing experience to users. However, there are still some challenges, like the blur caused by motion changes. In this paper, we introduce a dual refinement technique for video interpolation (DRVI). It has three main steps, namely flow refinement, frame synthesis, and Haar refinement. The flow refinement can generate accurate bi-directional flows, which are more suitable for frame interpolation tasks. The Haar refinement uses the Discrete Wavelet Transform (DWT). It can preserve information in different frequency domains and also speed up the learning process. We also add an arbitrary time approximation module to allow multi-frame generation. The number of learnable parameters in our model is much less than existing methods; still, it has excellent performance. Our method is trained on Vimeo90K (Xue et al., 2019) and tested on three well-known datasets to demonstrate its effectiveness.

As a non-invasive medical imaging scanning device, ultrasound has greatly increased the efficiency and accuracy of medical diagnosis. In recent years, portable ultrasound is being more widely used for its convenience and lower cost. Patients and physicians can receive the scanned images on their mobile phones at any time via a wireless network with low latency. However, it is difficult for portable ultrasound devices to capture images with the same quality as standard hospital ultrasound image acquisition systems. Usually, the images captured by portable ultrasound equipment have considerable noise. This noise undoubtedly affects the diagnosis of the physician. It is imperative to develop methods to remove the noise while preserving important information in the image. For this reason, we propose a novel denoising neural network model, called Feature-guided Denoising Convolutional Neural Network (FDCNN), to remove noise while retaining important feature information. In order to achieve high-quality denoising results, we employ a hierarchical denoising framework driven by a feature masking layer for medical images. Furthermore, we propose a feature extraction algorithm based on Explainable Artificial Intelligence (XAI) for medical images. Experimental results show that our medical image feature extraction method outperforms previous methods. Combined with the new denoising neural network architecture, portable ultrasound devices can now achieve better diagnostic performance.

Active contour models driven by local binary fitting energy can segment images with inhomogeneous intensity, while being prone to falling into a local minima. However, the segmentation result largely depends on the location of the initial contour. We propose an active contour model with global and local image information. The local information of the model is obtained by bilateral filters, which can also enhance the edge information while smoothing the image. The local fitting centers are calculated before the contour evolution, which can alleviate the iterative process and achieve fast image segmentation. The global information of the model is obtained by simplifying the C-V model, which can assist contour evolution, thereby increasing accuracy. Experimental results show that our algorithm is insensitive to the initial contour position, and has higher precision and speed.

We propose a new adaptive active contour model (ACM) based on weighted region-based pressure force (RBPF) which is applied to SAR image segmentation. First, the normalized intra-class variances of pixel grayscales of inner and outer areas are used as the new coefficients of the grayscale descriptions of inner and outer areas. Then, the weighted RBPF is constructed to control the curve motion more accurately. Second, when calculating the grayscale descriptions of the inner and outer areas, adaptive weights are introduced to reduce the effect of interference pixels, which improves the accuracy of the grayscale descriptions. Furthermore, some regularized terms are incorporated into the objective functional to ensure the stability of the model. The segmentation results for various kinds of images demonstrate that the proposed model is superior to some state-of-the-art ACMs in segmentation performance and is robust to the initial curve.

Deep convolutional neural networks (CNNs) are effective in image classification, and are widely used in image segmentation tasks. Several neural netowrks have achieved high accuracy in segementation on existing semantic datasets, for instance PASCAL VOC, CamVid, and Cityscapes. However, there are nearly no studies on semantic segmentation from the perspective of a dataset itself. In this paper, we analyzed the characteristics of datasets, and proposed a novel experimental strategy based on bokeh to weaken the impact of futile background information. This crucial bokeh module processed each image in the inference phase by selecting an opportune fuzzy factor σ, so that the attention of our network can focus on the categories of interest. Some networks based on fully convolutional networks (FCNs) were utilized to verify the effectiveness of our method. Extensive experiments demonstrate that our approach can generally improve the segmentation results on existing datasets, such as PASCAL VOC 2012 and CamVid.

With rapid advancements in remote sensing image registration algorithms, comprehensive imaging applications are no longer limited to single-modal remote sensing images. Instead, multi-modal remote sensing (MMRS) image registration has become a research focus in recent years. However, considering multi-source, multi-temporal, and multi-spectrum input introduces significant nonlinear radiation differences in MMRS images for which researchers need to develop novel solutions. At present, comprehensive reviews and analyses of MMRS image registration methods are inadequate in related fields. Thus, this paper introduces three theoretical frameworks: namely, area-based, feature-based and deep learning-based methods. We present a brief review of traditional methods and focus on more advanced methods for MMRS image registration proposed in recent years. Our review or comprehensive analysis is intended to provide researchers in related fields with advanced understanding to achieve further breakthroughs and innovations.

Abstract Image segmentation plays a vital role in MRI abnormality detection. This paper presents a robust MRI segmentation method to outline potential abnormality blobs. Thresholding and boundary tracing strategies are employed to remove background noises, and hence, the ROIs in the whole process are set. Subsequently, a polyfit surface evolution is proposed to approximately estimate bias field, which makes segmentation robust to image noises. Simultaneously, customized initial level set functions are devised so as to detect subtle bright and dark blobs which are highly potential abnormality regions. The proposed method improves bias field estimation and level set method to acquire fine segmentation with low computational complexities. Analysis of experimental results and comparisons with existing algorithms demonstrates that the proposed method can segment weak-edged, low-resolution MR brain images, and its performance prevails in accuracy and effectiveness.

Video has become the most popular medium of communication over the past decade, with nearly 90 percent of the bandwidth on the Internet being used for video transmission. Thus, evaluating the quality of an acquired or compressed video has become increasingly important. The goal of video quality assessment (VQA) is to measure the quality of a video clip as perceived by a human observer. Since manually rating every video clip to evaluate quality is infeasible, researchers have attempted to develop various quantitative metrics that estimate the perceptual quality of video. In this paper, we propose a new region-based average video quality assessment (RAVA) technique extending image quality assessment (IQA) metrics. In our experiments, we extend two full-reference (FR) image quality metrics to measure the feasibility of the proposed RAVA technique. Results on three different datasets show that our RAVA method is practical in predicting objective video scores.

Image registration has always been an important research topic. This paper proposes a novel method of constructing descriptors called the histogram of oriented local binary pattern descriptor (HOLBP) for fast and robust matching. There are three new components in our algorithm. First, we redefined the gradient and angle calculation template to make it more sensitive to edge information. Second, we proposed a new construction method of the HOLBP descriptor and improved the traditional local binary pattern (LBP) computation template. Third, the principle of uniform rotation-invariant LBP was applied to add 10-dimensional gradient direction information to form a 138-dimension HOLBP descriptor vector. The experimental results showed that our method is very stable in terms of accuracy and computational time for different test images.

Preterm births are rising in Canada and worldwide. As clinicians strive to identify preterm neonates at greatest risk of significant developmental or motor problems, accurate predictive tools are required. Infants at highest risk will be able to receive early developmental interventions, and will also enable clinicians to implement and evaluate new methods to improve outcomes. While severe white matter injury (WMI) is associated with adverse developmental outcome, more subtle injuries are difficult to identify and the association with later impairments remains unknown. Thus, our goal was to develop an automated method for detection and visualization of brain abnormalities in MR images acquired in very preterm born neonates. We have developed a technique to detect WMI in T1-weighted images acquired in 177 very preterm born infants (24–32 weeks gestation). Our approach uses a stochastic process that estimates the likelihood of intensity variations in nearby pixels; with small variations being more likely than large variations. We first detect the boundaries between normal and injured regions of the white matter. Following this we use a measure of pixel similarity to identify WMI regions. Our algorithm is able to detect WMI in all of the images in the ground truth dataset with some false positives in situations where the white matter region is not segmented accurately.

Parkinson’s disease (PD) is one of the leading neurological disorders in the world with an increasing incidence rate for the elderly. Freezing of Gait (FOG) is one of the most incapacitating symptoms for PD especially in the later stages of the disease. FOG is a short absence or reduction of ability to walk for PD patients which can cause fall, reduction in patients’ quality of life, and even death. Existing FOG assessments by doctors are based on a patient’s diaries and experts’ manual video analysis which give subjective, inaccurate, and unreliable results. In the present research, an automatic FOG assessment system is designed for PD patients to provide objective information to neurologists about the FOG condition and the symptom’s characteristics. The proposed FOG assessment system uses an RGB-D sensor based on Microsoft Kinect V2 for capturing data for 5 healthy subjects who are trained to imitate the FOG phenomenon. The proposed FOG assessment system is called “Kin-FOG”. The analysis of foot joint trajectory of the motion captured by Kinect is used to find the FOG episodes. The evaluation of Kin-FOG is performed by two types of experiments, including: (1) simple walking (SW); and (2) walking with turning (WWT). Since the standing mode has features similar to a FOG episode, our Kin-FOG system proposes a method to distinguish between the FOG and standing episodes. Therefore, two general groups of experiments are conducted with standing state (WST) and without standing state (WOST). The gradient displacement of the angle between the foot and the ground is used as the feature for discriminating between FOG and standing modes. These experiments are conducted with different numbers of FOGs for getting reliable and general results. The Kin-FOG system reports the number of FOGs, their lengths, and the time slots when they occur. Experimental results demonstrate Kin-FOG has around 90% accuracy rate for FOG prediction in both experiments for different tasks (SW, WWT). The proposed Kin-FOG system can be used as a remote application at a patient’s home or a rehabilitation clinic for sending a neurologist the required FOG information. The reliability and generality of the proposed system will be evaluated for bigger data sets of actual PD subjects.

We present a new method for real-time runway detection embedded in synthetic vision and an ROI (Region of Interest) based level set method. A virtual runway from synthetic vision provides a rough region of an infrared runway. A three-thresholding segmentation is proposed following Otsu’s binarization method to extract a runway subset from this region, which is used to construct an initial level set function. The virtual runway also gives a reference area of the actual runway in an infrared image, which helps us design a stopping criterion for the level set method. In order to meet the needs of real-time processing, the ROI based level set evolution framework is implemented in this paper. Experimental results show that the proposed algorithm is efficient and accurate.

Adaptive search is one of the fastest fractal compression algorithms and has gained great success in many industrial applications. By substituting the luminance offset by the range block mean, the authors create a completely new version for both the encoding and decoding algorithms. In this paper, theoretically, they prove that the proposed decoding algorithm converges at least as fast as the existing decoding algorithms using the luminance offset. In addition, they prove that the attractor of the decoding algorithm can be represented by a linear combination of range-averaged images. These theorems are very important contributions to the theory and applications of fractal image compression. As a result, the decoding image can be represented as the sum of the DC and AC component images, which is similar with discrete cosine transform or wavelet transform. To further speed up this algorithm and reduce the complexity of range and domain blocks matching, they propose two improvements in this paper, that is, employing the post-quantisation and geometric neighbouring local search to replace the currently used pre-quantisation and the global search, respectively. The corresponding experimental results show the proposed encoding and decoding algorithms can provide a better performance compared with the existing algorithms.

We introduce a different approach of applying stereoscopy principles to implement a virtual 3D pointing technique called stereo 3D mouse cursor (S3D-Cursor) based on two or more views of an ordinary mouse cursor. The basics of such an idea have already been applied as a by-product of some stereo-based visualization applications with usually less attention to its strengths or weaknesses as a generic alternative of its 2D counterpart in stereoscopic 3D space. Here, we examine if such an idea satisfies all or the main expected requirements of an abstract 3D cursor. Moreover, we analyze its accuracy and evaluate the applicability of this approach in terms of different efficiency factors. For this purpose, we have adapted a real-time point-based rendering software called QSplat to a multiview rendering version named QSplatMV. We have implemented the S3D-Cursor on top of this new application and developed a simple editing toolset for manipulating the virtual 3D objects. Our user evaluation results suggest the effectiveness of the approach in terms of detection accuracy and user satisfaction compared to using an ordinary mouse cursor on a conventional 2D screen.

We describe transmission of 3D objects represented by texture and mesh over unreliable networks, extending our earlier work for regular mesh structure to arbitrary meshes and considering linear versus cubic interpolation. Our approach to arbitrary meshes considers stripification of the mesh and distributing nearby vertices into different packets, combined with a strategy that does not need texture or mesh packets to be retransmitted. Only the valence (connectivity) packets need to be retransmitted; however, storage of valence information requires only 10% space compared to vertices and even less compared to photorealistic texture. Thus, less than 5% of the packets may need to be retransmitted in the worst case to allow our algorithm to successfully reconstruct an acceptable object under severe packet loss. Even though packet loss during transmission has received limited research attention in the past, this topic is important for improving quality under lossy conditions created by shadowing and interference. Results showing the implementation of the proposed approach using linear, cubic, and Laplacian interpolation are described, and the mesh reconstruction strategy is compared with other methods.

There are several advantages to creating multimedia item types and applying computer-based adaptive testing in education. First is the capability to motivate learning by making the learners feel more engaged and in an interactive environment. Second is a better concept representation, which is not possible in conventional multiple-choice tests. Third is the advantage of individualized curriculum design, rather than a curriculum designed for an average student. Fourth is a good choice of the next question, associated with the appropriate difficulty level based on a student's response to the current question. However, many issues need to be addressed when achieving these goals, including: (a) the large number of item types required to represent the current multiple-choice questions in multimedia formats, (b) the criterion used to determine the difficulty level of a multimedia question item, and (c) the methodology applied to the question selection process for individual students. In this paper, we propose a multimedia item shell design that not only reduces the number of item types required, but also computes difficulty level of an item automatically. The concept of question seed is introduced to make content creation more cost-effective. The proposed item shell framework facilitates efficient communication between user responses at the client, and the scoring agents integrated with a student ability assessor at the server. We also describe approaches for automatically estimating difficulty level of questions, and discuss preliminary evaluation of multimedia item types by students.