Search

Visual surveillance systems have been playing an important role in monitoring and managing at public areas. However, the computational complexity of video compression in these applications is still a great challenge. To meet practical requirements, the authors propose in this paper a low‐complexity surveillance video coding solution in which the most recent Versatile Video Coding (VVC) standard is improved with a novel learning adaptive motion search algorithm. The proposed algorithm is designed based on the temporal motion and spatial texture characteristics of surveillance videos. First, the authors study and define a list of spatial and temporal features which indicates the motion and texture characteristics of surveillance video. These features are used together with a machine learning algorithm to appropriately assign a search range for the VVC motion search. Second, to reduce search points, the authors propose an adaptive Test Zone (TZ) search in which TZ steps are early terminated following the variation of spatial–temporal features. Performance evaluation conducted for a rich set of surveillance videos and relevant benchmarks have shown the superiority of the proposed method, notably with around 33% of encoding time saving when compared with the state‐of‐the art VVC solution and relevant benchmarks while asking for negligible compression loss. [ABSTRACT FROM AUTHOR]

Imbalanced image datasets consist of image datasets where there is a significant disparity in the number of samples across different classes. With imbalanced image datasets, learning algorithms often tend to be biased toward the majority class samples. This leads to poor classification of minority class samples as their training is not properly conducted. It becomes more complicated when the number of samples in the minority class is very low. In this paper, a novel hierarchical algorithm is proposed for generating new data using Capsule Generative Adversarial Networks (CapsGAN) to address the class imbalance problem in imbalanced image datasets. Unlike common GAN models, the proposed method incorporates an auxiliary CapsNet to identify high‐value images in both minority and majority classes. This identification is based on the ability to detect complex relationships between low‐level and high‐level features present in capsule networks. Furthermore, the proposed CapsGAN model is conditioned to generate minority class samples based on feature vectors of last capsule layer to achieve a more balanced image dataset. For evaluating the performance of the proposed model, an image dataset called CICS was collected and introduced. Extensive experiments were also conducted using various online image datasets from different domains, with varying numbers of classes and data sizes. The experimental results demonstrated that the proposed model can generate high‐quality samples in cases where the image dataset or the number of minority class samples is relatively small. Furthermore, the proposed model has maintained an accuracy of over 80% in an imbalanced ratio of 1:60. [ABSTRACT FROM AUTHOR]

Alzheimer's disease, often known as dementia, is a severe neurodegenerative disorder that causes irreversible memory loss by destroying brain cells. People die because there is no specific treatment for this disease. Alzheimer's is most common among seniors 65 years and older. However, the progress of this disease can be reduced if it can be diagnosed earlier. Recently, artificial intelligence has instilled hope in the diagnosis of Alzheimer's disease by performing sophisticated analyses on extensive patient datasets, enabling the identification of subtle patterns that may elude human experts. Researchers have investigated various deep learning and machine learning models to diagnose this disease at an early stage using image datasets. In this paper, a new Deep learning (DL) methodology is proposed, where MRI images are fed into the model after applying various pre‐processing techniques. The proposed Alzheimer's disease detection approach adopts transfer learning for multi‐class classification using brain MRIs. The MRI Images are classified into four categories: mild dementia (MD), moderate dementia (MOD), very mild dementia (VMD), and non‐dementia (ND). The model is implemented and extensive performance analysis is performed. The finding shows that the model obtains 97.31% accuracy. The model outperforms the state‐of‐the‐art models in terms of accuracy, precision, recall, and F‐score. [ABSTRACT FROM AUTHOR]

Court line extraction is one of the important steps in the analysis of sport videos. The court extraction is the foundation of the analysis of badminton video, and an efficient method with horizontal line projection K‐means machine learning algorithm to extract court lines from different broadcast badminton tournament videos is proposed in this paper. The horizontal lines are projected into 1‐D histogram signal; then the signal is trained to learn the intensity of the histogram signal for locating the positions of the horizontal court lines. After the equations of the horizontal court lines and the court lines in the vertical direction have been formulized, the intersection points of the court lines can be calculated and the court line can be extracted. The experimental results show that the proposed method can extract the court lines more efficiently than that done by the Hough transform‐related algorithms, which are widely applied in computer vision and self‐driving car applications. [ABSTRACT FROM AUTHOR]

Security inspection has been playing a critical role in protecting public space from safety threats. As object detection is a fundamental and mature research filed, it still suffers from numerous challenges such as scale, viewpoint and intra‐class variance of X‐ray images. The main reasons are, mis‐modeling of actual security inspection as well as insufficient effective X‐ray samples. To this end, the X‐ray inspection task is reconsidered by predicting the dangerous attributes without boxes and categories, a Dual Multi‐instance Attention network (DMA‐Net) is proposed in this paper to mine the key‐instance from both patch and proposal branch. In patch‐level multi‐instance attention pooling, Recursive Attention Pyramid (RAP) and Spatial‐Channel Attention (SCA) are proposed to effectively learn discriminative representation from hierarchical features. Meanwhile, the proposal‐level multi‐instance attention pooling selectively emphasizes interdependent of crucial regions. The features of dual pooling layers are fused to predict dangerous attributes of X‐ray inspection images directly. Further, this paper contributes a large‐scale and high‐quality X‐ray image dataset from railway stations, named Railway Station X‐ray (RSXray). Enormous experiments on RSXray, GDXray and SIXray demonstrate that the proposed DMA‐Net achieves surprising performance with dual multi instance learning diagrams and attention mechanism. The superiority of the employed framework for applications in real world scenarios are also verified. [ABSTRACT FROM AUTHOR]

Diagnosing Parkinson's disease (PD) in its early stages is a significant challenge in medicine. Hand tremors and dysgraphia, which are typical early motor symptoms of PD, can manifest for decades before a formal diagnosis is made. Therefore, handwriting analysis has become an important tool for detecting PD. While many machine learning algorithms have been applied in this area, they struggle to capture the subtle changes in handwriting and must describe features from various perspectives. To address these issues, this paper proposes a Coordinate Attention Enhanced Swin Transformer (CAS Transformer) model for PD handwriting recognition. It establishes the long‐term dependence of features on the joint coordinate attention application, which enables the model to more accurately localize the important features of handwriting data and also extract the fuzzy edge features of handwriting images.These characteristics of the CAS Transformer enable it to outperform current advanced deep learning methods in classification, with an accuracy of 92.68% in experiments conducted on two handwritten datasets. [ABSTRACT FROM AUTHOR]

In recent years, terahertz imaging technology has been widely used in security, medicine, and other fields. However, the image resolution is low due to the limits of imaging equipment and diffraction. Traditional super‐resolution methods based on machine learning use artificial paired image datasets, and their image degradation process is quite different from the actual terahertz imaging mechanism. In the paper, an unpaired hybrid attention network is proposed, using the real unpaired high‐resolution and low‐resolution terahertz images as the original input, two sub‐networks are trained: degenerate reconstruction network and super‐resolution network. The degenerate reconstruction network is used to learn the degradation process of the real terahertz imaging system, and reconstruct the generated degradation images into the fixed mapping down‐sampled image of high‐resolution terahertz image. The super‐resolution network uses the generated paired image with fixed correspondence to train the super‐resolution network of paired terahertz images, to improve the imaging resolution of the network in the super‐resolution work of real low‐resolution terahertz images. In addition, the no‐reference image quality evaluation system is introduced to objectively evaluate the network performance. The experimental results show that the UHA‐CycleGAN network outperforms the traditional paired super‐resolution network on both open‐source and self‐built datasets. [ABSTRACT FROM AUTHOR]

Images captured through a visual sensory system are degraded in a foggy scene, which negatively influences recognition, tracking, and detection of targets. Efficient tools are needed to detect, pre‐process, and enhance foggy scenes. Machine learning (ML) has a significant role in image defogging domain for tackling adverse issues. Unfortunately, regardless of contributions that were made by ML, little attention has been attributed to this topic. This paper summarizes the role of ML methods and relevant aspects in the image defogging research area. Also, the basic terms and concepts are highlighted in image defogging topic. Feature extraction approaches with a summary of advantages and disadvantages are described. ML algorithms are also summarized that have been used for applications related to image defogging, that is, image denoising, image quality assessment, image segmentation, and foggy image classification. Open datasets are also discussed. Finally, the existing problems of the image defogging domain in general and, specifically related to ML which need to be further studied are discussed. To the best knowledge, this the first review paper which sheds a light on the role of ML and relevant aspects in the image defogging domain. [ABSTRACT FROM AUTHOR]

In the field of target detecting, lots of progress have been made in recent years. Owing to the progress of multiple frames time series data, or video satellites, target detecting from space‐borne satellite videos has been available. However, detecting a slightly moving target from space‐borne videos is still a difficult task, because of the low resolution and illumination variation influence. This paper considers target detecting from time series data as multi‐label problem as there are several different kinds of background objects and targets of interest. To some extent the background of time series data is comparative invariant, using background analysis method to extract the target from the background is promising. This paper proposes a novel target detecting algorithm based on multi‐label learning and Gaussian background description model aiming at extracting slowly moving target. To further enhance performances, multi‐frame fusion and post processing method was utilized to catch the slight difference due to movement. Experimental results on real world datasets indicate that the proposed method outperforms some state‐of‐the‐art algorithms. [ABSTRACT FROM AUTHOR]

Technological advances led to the generation of large scale complex data. Thus, extraction and retrieval of information to automatically discover latent pattern have been largely studied in the various domains of science and technology. Consequently, machine learning experienced tremendous development and various statistical approaches have been suggested. In particular, data clustering has received a lot of attention. Finite mixture models have been revealed to be one of the flexible and popular approaches in data clustering. Considering mixture models, three crucial aspects should be addressed. The first issue is choosing a distribution which is flexible enough to fit the data. In this paper, a model based on multivariate Beta distributions is proposed. The two other challenges in mixture models are estimation of model's parameters and model complexity. To tackle these challenges, variational inference techniques demonstrated considerable robustness. In this paper, two methods are studied, namely, batch and online variational inferences and the models are evaluated on four medical applications including image segmentation of colorectal cancer, multi‐class colon tissue analysis, digital imaging in skin lesion diagnosis and computer aid detection of Malaria. [ABSTRACT FROM AUTHOR]

The Three‐Dimensional High Efficiency Video Coding standard is a video compression standard developed based on the two‐dimensional video coding standard HEVC and used to encode multi‐view plus depth format video. This paper proposes an algorithm based on eXtreme Gradient Boosting to solve the problem of high inter‐frame coding complexity in 3D‐HEVC. Firstly, explore the correlation between the division depth of the inter‐frame coding unit and the texture features in the map, as well as the correlation between the coding unit division structure between each map and each viewpoint. After that, based on the machine learning method, a fast selection mechanism for dividing the depth range of the inter‐frame coding tree unit based on the eXtreme Gradient Boosting algorithm is constructed. Experimental results show that, compared with the reference software HTM‐16.0, this method can save an average of 35.06% of the coding time, with negligible degradation in terms of coding performance. In addition, the proposed algorithm has achieved different degrees of improvement in coding performance compared with the related works. [ABSTRACT FROM AUTHOR]

Due to the low contrast, blurred boundary and intensity inhomogeneity of the images, accurate segmentation of breast cancer lesions with dynamic contrast‐enhanced magnetic resonance imaging (DCE‐MRI) still has great challenges. This paper proposed an improved active contour model (ACM) for segmenting breast cancer lesions in DCE‐MRI images. First, based on the extreme learning machine (ELM) method, a robust function is proposed that combines image intensities and time‐domain features to enhance the difference between the lesions and other tissues. Second, an edge‐stop function (ESF) is introduced by combining the image intensity, time‐domain feature, and Hessian shape index to detect the irregular and blurred boundaries. At the boundary of breast cancer lesions, the energy function of ACM is minimized and the evolution of the contour curve completes, so the accurate lesion region of breast cancer can be segmented. The mean Dice similar coefficient (DICE), Jaccard similarity (JC) and Hausdorff distance (HD) of the segmentation of the proposed model in 50 samples are 85.88±6.62%, 75.72±9.68% and 11.62±4.72 mm, respectively. The results segmented by the proposed ACM are more similar to the manual segmentation than the compared models. [ABSTRACT FROM AUTHOR]

Medical image segmentation is one of the major challenges addressed by machine learning methods. However, these methods profoundly depend on a large amount of annotated data, which is expensive and time‐consuming. Semi‐supervised learning (SSL) approaches this by leveraging an abundant amount of unlabeled data. Recently, MixUp regularizer has been introduced to SSL methods by augmenting the model with new data points through linear interpolation at the input space. While this provides the model with new data, it is limited and may lead to inconsistent soft labels. It is argued that the linear interpolation at different representations provides the network with novel training signals and overcomes the inconsistency of the soft labels. This paper proposes ROAM as an SSL method that explores the manifold and performs linear interpolation on randomly selected layers to generate virtual data that has never been seen before, which encourages the network to be less confident for interpolated points. Hence it avoids overfitting, enhances the generalization, and shows less sensitivity to the domain shift. Extensive experiments are conducted on publicl datasets on whole‐brain and lung segmentation. ROAM achieves state‐of‐the‐art results in fully supervised (89.5%) and semi‐supervised (87.0%) settings with relative improvements up to 2.40% and 16.50%, respectively. [ABSTRACT FROM AUTHOR]

The development of deep learning provides a new way for solving the colorization problem on the grayscale image. Excellent coding‐based methods appear in the automatic image colorization task, avoiding the unsaturated colour effect problem of previous methods based on the L2 loss function. Traditional neural networks come with high computational costs and a large number of parameters. Considering the limitation of memory and computing resources and aiming at lightweight, a novel grey image automatic colorization network is proposed. The basic idea of coding‐based methods is used, regarding the colorization task as a pixel‐level classification problem, meanwhile redesign and improve the colour encoding and decoding process. This network architecture leverages a lightweight convolution to reduce the computation and combines an efficient attention model to form a residual block as the kernel of the backbone network. Furthermore, an efficient image self‐attention mechanism placed at the end of the network is applied to enhance the ultimate colouring results. The method proposed in this paper can maintain the natural colouring effect and significantly reduce the computational amount and network model parameters. [ABSTRACT FROM AUTHOR]

Real‐time facial expression recognition is the basis for computers to understand human emotions and detect abnormalities in time. To effectively solve the problems of server overload and privacy information leakage, a real‐time facial expression recognition method based on iterative transfer learning and efficient attention network (EAN) for edge resource‐constrained scenes is proposed in this paper. Firstly, an EAN is designed with its parameter number and computation amount strictly limited by depth separable convolution and local channel attention mechanism. Then, the soft labels of facial expression data were obtained by EAN based on the idea of knowledge distillation, so as to provide more supervision information for the training process. Finally, an iterative transfer learning method of teacher‐student (T‐S) network was proposed; it refines the soft labels of the teacher network and further improves the recognition accuracy of the student network. The tests on the public datasets, FER2013 and RAF‐DB, show that this method can significantly reduce the model complexity and achieve high recognition accuracy. Compared with other advanced methods, the proposed method strikes a good balance between complexity and accuracy, and well meets the real‐time deployment requirements of facial expression recognition technology for edge resource‐constrained scenes. [ABSTRACT FROM AUTHOR]

Multispectral remote sensing images are a very important data source, but its acquisition process is often affected by haze weather and other factors, resulting in the decline of image quality, blurred details and poor visual effect, which seriously affect their application and interpretation. To reduce the impact of haze on multispectral remote sensing image and improve the image clarity and the use value, a new haze removal method based on rolling deep learning and Retinex theory (RDLRT) was proposed here. It uses the rolling deep learning theory to realize the preliminary removal of image haze, and then calculates the peak signal‐to‐noise ratio (PSNR) value of the processed image, which is used to judge whether the obtained image is a stage effect image (SEI). Then, on the one hand, the SEI image is processed for colour saturation and brightness; on the other hand, the Retinex theory is used to further filter the haze of the SEI image. Finally, the results of the two processing are fused, and it is used as the restored image after removing the haze. A series of validation experiments were carried out with true multispectral remote sensing images, outdoor colour images and different methods, and good experimental results were obtained. The purpose of removing image haze and improving image quality is achieved. Through the comparison of these experiments, it is concluded that the RDLRT algorithm proposed in this paper can not only effectively remove haze in images, but also maintain good details and colour of the image, which has some popularization values for image pre‐processing and restoration. [ABSTRACT FROM AUTHOR]

Automatic generation of calligraphy fonts has attracted broad attention of researchers. However, previous font generation research mainly focused on the known font style imitation based on image to image translation. For poor interpretability, it is hard for deep learning to create new fonts with various font styles and features according to human understanding. To address this issue, the font fusion network based on generative adversarial networks (GANs) and disentangled representation learning is proposed in this paper to generate brand new fonts. It separates font into two understandable disentangled features: stroke style and skeleton shape. According to personal preferences, various new fonts with multiple styles can be generated by fusing the stroke style and skeleton shape of different fonts. First, this task improves the interpretability of deep learning, and is more challenging than simply imitating font styles. Second, considering the robustness of the network, a fuzzy supervised learning skill is proposed to enhance the stability of the fusion of two fonts with considerable discrepancy. Finally, instead of retraining, the authors' trained model can be quickly transferred to other font fusion samples. It improves the efficiency of the model. Qualitative and quantitative results demonstrate that the proposed method is capable of efficiently and stably generating the new font images with multiple styles. The source code and the implementation details of our model are available at https://github.com/Qinmengxi/Fontfusion. [ABSTRACT FROM AUTHOR]

Deraining has been attracting a lot of attention from researchers, and various methods have been proposed, especially deep‐networks are widely adopted in recent years. Their structures and learning become more and more complicated and diverse, making it difficult to analyze the contributions and improvements. In this paper, a comprehensive review for current rain removal methods is first provided to show their contributions. Specifically, they are reviewed in terms of handing rain streaks and rain mist. Second, besides evaluating their rain removal ability, they are also evaluated in terms of their impact on subsequent stereo‐matching task. To this end, a new deraining dataset is first prepared, called Rain‐Kitti2012 and Rain‐Kitti2015. They are created by adding rain part to clean image‐pairs in Kitti2012 and Kitti2015. By then, nine state‐of‐the‐art deraining methods are evaluated with full‐reference and no‐reference image quality assessment metrics. Furthermore, the blurriness and distortion types introduced during deraining are measured. Finally, three learning‐based stereo matching methods are compared, and they take the outputs of deraining methods as inputs. It is further discussed how derained images influence the accuracy of stereo matching, which can provide some insight for jointly handling rain removal and stereo matching. 1: A comprehensive review for the current rain removal methods is provided. They are categorized into rain‐streak‐oriented and rain‐mist‐oriented approaches in terms of degradation type, and are categorized into model‐driven and data‐driven approaches in terms of methodology. 2: A new image deraining dataset is introduced, which is the first dataset that can be used to perform stereo‐matching‐driven evaluation for deraining methods. The dataset is created by adding rain part to clean images in KITTI2012 and KITTI2015. 3: We evaluate 9 deep learning based deraining methods with full‐reference and no‐ reference metrics. In addition, the types of distortions produced by these methods are discussed and measured quantitatively. And, the impact of 9 deraining methods on the subsequent stereo matching task is evaluated, which can provide some insight on how to design stereo matching task‐driven deraining methods. [ABSTRACT FROM AUTHOR]

Cross modal retrieval can retrieve images through a text query and vice versa. In recent years, cross modal retrieval has attracted extensive attention. The purpose of most now available cross modal retrieval methods is to find a common subspace and maximize the different modal correlation. To generate specific representations consistent with cross modal tasks, this paper proposes a novel cross modal retrieval framework, which integrates feature learning and latent space embedding. In detail, we proposed a deep CNN and a shallow CNN to extract the feature of the samples. The deep CNN is used to extract the representation of images, and the shallow CNN uses a multi‐dimensional kernel to extract multi‐level semantic representation of text. Meanwhile, we enhance the semantic manifold by constructing cross modal ranking and within‐modal discriminant loss to improve the division of semantic representation. Moreover, the most representative samples are selected by using online sampling strategy, so that the approach can be implemented on a large‐scale data. This approach not only increases the discriminative ability among different categories, but also maximizes the relativity between different modalities. Experiments on three real word datasets show that the proposed method is superior to the popular methods. [ABSTRACT FROM AUTHOR]