Search

With the development of Earth observation technology, very-high-resolution (VHR) image has become an important data source of change detection. Nowadays, deep learning methods have achieved conspicuous performance in the change detection of VHR images. Nonetheless, most of the existing change detection models based on deep learning require annotated training samples. In this paper, a novel unsupervised model called kernel principal component analysis (KPCA) convolution is proposed for extracting representative features from multi-temporal VHR images. Based on the KPCA convolution, an unsupervised deep siamese KPCA convolutional mapping network (KPCA-MNet) is designed for binary and multi-class change detection. In the KPCA-MNet, the high-level spatial-spectral feature maps are extracted by a deep siamese network consisting of weight-shared PCA convolution layers. Then, the change information in the feature difference map is mapped into a 2-D polar domain. Finally, the change detection results are generated by threshold segmentation and clustering algorithms. All procedures of KPCA-MNet does not require labeled data. The theoretical analysis and experimental results demonstrate the validity, robustness, and potential of the proposed method in two binary change detection data sets and one multi-class change detection data set.

Spherical images or videos, as typical non-Euclidean data, are usually stored in the form of 2D panoramas obtained through an equirectangular projection, which is neither equal area nor conformal. The distortion caused by the projection limits the performance of vanilla Deep Neural Networks (DNNs) designed for traditional Euclidean data. In this paper, we design a novel Spherical Deep Neural Network (DNN) to deal with the distortion caused by the equirectangular projection. Specifically, we customize a set of components, including a spherical convolution, a spherical pooling, a spherical ConvLSTM cell and a spherical MSE loss, as the replacements of their counterparts in vanilla DNNs for spherical data. The core idea is to change the identical behavior of the conventional operations in vanilla DNNs across different feature patches so that they will be adjusted to the distortion caused by the variance of sampling rate among different feature patches. We demonstrate the effectiveness of our Spherical DNNs for saliency detection and gaze estimation in 360° videos. To facilitate the study of the 360 video saliency detection, we further construct a large-scale 360° video saliency detection dataset. Comprehensive experiments validate the effectiveness of our proposed Spherical DNNs for spherical handwritten digit classification and sport classification, saliency detection and gaze tracking in 360° videos.

Varying density of point clouds increases the difficulty of 3D detection. In this paper, we present a context-aware dynamic network (CADNet) to capture the variance of density by considering both point context and semantic context. Point-level contexts are generated from original point clouds to enlarge the effective receptive filed. They are extracted around the voxelized pillars based on our extended voxelization method and processed with the context encoder in parallel with the pillar features. With a large perception range, we are able to capture the variance of features for potential objects and generate attentive spatial guidance to help adjust the strengths for different regions. In the region proposal network, considering the limited representation ability of traditional convolution where same kernels are shared among different samples and positions, we propose a decomposable dynamic convolutional layer to adapt to the variance of input features by learning from the local semantic context. It adaptively generates the position-dependent coefficients for multiple fixed kernels and combines them to convolve with local features. Based on our dynamic convolution, we design a dual-path convolution block to further improve the representation ability. We conduct experiments on KITTI dataset and the proposed CADNet has achieved superior performance of 3D detection outperforming SECOND and PointPillars by a large margin at the speed of 30 FPS.

Loop-closure detection (LCD), which aims to recognize a previously visited location, is a crucial component of the simultaneous localization and mapping system. In this paper, a novel appearance-based LCD method is presented. In particular, we propose a simple yet surprisingly useful feature matching algorithm for real-time geometrical verification of candidate loop-closures, termed as local relative orientation matching (LRO). It aims to efficiently establish reliable feature correspondences based on preserving local topological structures between the query image and candidate frame. To effectively retrieve candidate loop closures, we introduce the aggregated selective match kernel framework into the LCD task, which can effectively represent images and reduce the quantization noise of the traditional bag-of-words framework. In addition, the SuperPoint neural network is employed to extract reliable interest points and feature descriptors. Extensive experimental results demonstrate that our LRO can significantly improve the LCD performance, and the proposed overall LCD method can achieve much better performance over the current state-of-the-art on six publicly available datasets.

This paper attempts at improving the accuracy of Human Action Recognition (HAR) by fusion of depth and inertial sensor data. Firstly, we transform the depth data into Sequential Front view Images(SFI) and fine-tune the pre-trained AlexNet on these images. Then, inertial data is converted into Signal Images (SI) and another convolutional neural network (CNN) is trained on these images. Finally, learned features are extracted from both CNN, fused together to make a shared feature layer, and these features are fed to the classifier. We experiment with two classifiers, namely Support Vector Machines (SVM) and softmax classifier and compare their performances. The recognition accuracies of each modality, depth data alone and sensor data alone are also calculated and compared with fusion based accuracies to highlight the fact that fusion of modalities yields better results than individual modalities. Experimental results on UTD-MHAD and Kinect 2D datasets show that proposed method achieves state of the art results when compared to other recently proposed visual-inertial action recognition methods.

Diabetic Foot Ulcer (DFU) is a complication of diabetes that causes lower limb amputation. In this work, a unique stacked parallel convolution layers-based network (DFU_SPNet) is proposed to perform DFU vs. normal skin classification. The main objective of this work is to design an effective CNN-based classification model, along with proper fine-tuning of optimizer settings. DFU_SPNet consists of 3 blocks of parallel convolution layers with multiple kernel sizes, for local and global feature abstractions. The proposed DFU_SPNet, trained using SGD (with momentum) optimizer with 1 e − 2 learning rate on the DFUNet dataset, outperformed the current state-of-the-art results with an AUC of 0.974.

As a lightweight, flexible and high-performance operating system virtualization, containers are used to speedup the big data platform. However, due to the imperfection of resource isolation mechanism and property of shared kernel, the meltdown and spectre attacks can lead to the information leakage of kernel space and co-resident containers. In this paper, a noise-resilient and real-time detection system, named ContainerGuard, is proposed to detect meltdown and spectre attacks in container-based big data platform. ContainerGuard uses a non-intrusive manner to collect lifecycle multivariate time-series performance events data of processes in containers, then uses ensemble of variational autoencoders as generative neural networks to learn the robust representations of normal patterns. Therefore, ContainerGuard meets the urgent need for information protection in container-based big data platform. Our evaluations using real-world datasets show that ContainerGuard achieves excellent detection performance, only introduces about 4.5% of running performance overhead to the platform.

Social Internet of Things is the fusion carrier of social network and Internet of Things. In the social Internet of Things, millions of different intelligent objects connect and communicate with each other, and social data emerge rapidly, which puts forward higher requirements for the rapid dissemination of valuable information. As an important means of information retrieval, the research and application of multi-label classification technology in social Internet of Things environment is relatively few. Characterized by multi-labels cognitive learning, fast response, concept drifting and huge solution space, the multi-label learning becomes more complicated under data stream environment. To overcome the problems above, this paper proposes a multi-label algorithm based on kernel extreme learning machine. In the training phase, both Cholesky matrix decomposition inverse method and matrix block method are deployed on the Spark platform to solve the inverse problem of large matrix, thus improving the modeling efficiency. In addition, incremental learning of the model is realized from the two aspects of example increment and class increment. The online sequential extreme learning machine is improved to adjust the network weights and realize incremental updating of input dimension. Experimental results demonstrate that the proposed method provides an efficient solution to multi-label classification.

We propose a novel pairwise distance measure between image keypoint sets, for the purpose of large-scale medical image indexing. Our measure generalizes the Jaccard index to account for soft set equivalence (SSE) between keypoint elements, via an adaptive kernel framework modeling uncertainty in keypoint appearance and geometry. A new kernel is proposed to quantify the variability of keypoint geometry in location and scale. Our distance measure may be estimated between $O(N^2)$ image pairs in $O(N~\log~N)$ operations via keypoint indexing. Experiments report the first results for the task of predicting family relationships from medical images, using 1010 T1-weighted MRI brain volumes of 434 families including monozygotic and dizygotic twins, siblings and half-siblings sharing 100%-25% of their polymorphic genes. Soft set equivalence and the keypoint geometry kernel improve upon standard hard set equivalence (HSE) and appearance kernels alone in predicting family relationships. Monozygotic twin identification is near 100%, and three subjects with uncertain genotyping are automatically paired with their self-reported families, the first reported practical application of image-based family identification. Our distance measure can also be used to predict group categories, sex is predicted with an AUC=0.97. Software is provided for efficient fine-grained curation of large, generic image datasets., 10 pages, 8 figures

This brief presents an event-driven spiking multi-kernel convolution architecture for processing address-event representation (AER) streams from dynamic vision sensor (DVS) chips. The processor architecture is designed based on leaky integrate-and-fire (LIF) neural model, and employs pipeline scheme to accelerate data processing. A new scheme for arranging neuron membrane potentials and kernels in memories is proposed, which enables row-by-row kernel processing and accelerates the multi-kernel convolution computation. An FPGA prototype of the proposed architecture is implemented on a Xilinx Zynq FPGA development board with 100 MHz clock frequency. The proposed processor architecture computes 64 filters with configurable kernel size (from 1x1 to 32x32) on input flow, obtaining the latency of 0.10 us to 10.33 us for convoluting an event, and the energy of 0.12 nJ and 12.08 nJ per event per convolution, respectively.

Fine-grained recognition poses the challenge of discriminating categories with only small subtle visual differences, which can be easily overwhelmed by diverse appearance within categories. Conventional approaches generally locate discriminative parts and then recognize the part-based features. However, we find that tuning the effective receptive field (ERF) of the network to the task plays the key role, which enables significant regions to contribute more to the output. Inspired by the receptive field stimulation mechanism of the visual cortex, we propose a Dynamic Perception framework as a solution. Our framework adapts the ERF by considering the image space and the kernel space simultaneously. In the image space, the Spatial Selective Sampling module is adopted to enlarge informative regions locally. In the kernel space, Spatial Selective Kernel convolution is introduced to adapt different kernel sizes for regions of interest and backgrounds by embedding spatial attention in the multi-path convolution. Extensive experiments on challenging benchmarks, including CUB-200-2011, FGVC-Aircraft, and Stanford Cars, demonstrate that our method yields a performance boost over the state-of-the-art methods.

Omni-directional images are becoming more prevalent for understanding the scene of all directions around a camera, as they provide a much wider field-of-view (FoV) compared to conventional images. In this work, we present a novel approach to represent omni-directional images and suggest how to apply CNNs on the proposed image representation. The proposed image representation method utilizes a spherical polyhedron to reduce distortion introduced inevitably when sampling pixels on a non-Euclidean spherical surface around the camera center. To apply convolution operation on our representation of images, we stack the neighboring pixels on top of each pixel and multiply with trainable parameters. This approach enables us to apply the same CNN architectures used in conventional Euclidean 2D images on our proposed method in a straightforward manner. Compared to the previous work, we additionally compare different designs of kernels that can be applied to our proposed method. We also show that our method outperforms in monocular depth estimation task compared to other state-of-the-art representation methods of omni-directional images. In addition, we propose a novel method to fit bounding ellipses of arbitrary orientation using object detection networks and apply it to an omni-directional real-world human detection dataset.

The improved particle swarm optimization algorithm is integrated with variational mode decomposition (VMD) to extract the efficient band-limited intrinsic mode function (BLIMF) of the single and combined power quality events (PQEs). The selected BLIMF of the robust VMD (RVMD) and the privileged Fourier magnitude spectrum (FMS) information are fed to the proposed reduced deep convolutional neural network (RDCNN) for the extraction of the most discriminative unsupervised features. The RVMD-FMS-RDCNN method shows minimum feature overlapping compared with RDCNN and RVMD-RDCNN methods. The feature vector is imported to the novel supervised online kernel random vector functional link network (OKRVFLN) for quick and accurate categorization of complex PQEs. The proposed RVMD-FMS-RDCNN-OKRVFLN method produces excellent recognition capability over RDCNN, RVMD-RDCNN, and RVMD-RDCNN-OKRVFLN methods in noise-free and noisy environments. The unique BLIMF selection, clear detection, descriptive feature extraction, higher learning speed, superior classification accuracy, and robust antinoise performances are considerable importance of the proposed RVMD-FMS-RDCNN-OKRVFLN method. Finally, the proposed method architecture is developed and implemented in a very-high-speed ML506 Virtex-5 FPGA to text, examine, and validate the feasibility, performances, and practicability for online monitoring of the PQEs.

Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. However, existing UDA methods largely ignore the instantaneous data distribution and the sampling strategy during model learning, which could deteriorate the feature representation given large domain shift. In this work, we propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains while considering the instantaneous alignment difficulty. The core of SGA is to calculate "hardness" factors for sample pairs indicating domain distance in a kernel space. With the hardness factor, the proposed SGA adaptively indicates the importance of samples and assigns them different constrains. Indicated by hardness factors, Self-Guided Progressive Sampling (SPS) is implemented in an "easy-to-hard" way during model adaptation. Using multi-stage convolutional features, SGA is further aggregated to fully align hierarchical representations of detection models. Extensive experiments on commonly used benchmarks show that SGA improves the state-of-the-art methods with significant margins, while demonstrating the effectiveness on large domain shift.

Synthetic aperture radar (SAR), due to its inherent characteristics, will produce speckle noise, which results in the deterioration of image quality, so the removal of speckle in SAR image is very important for the subsequent high-level image processing. In order to balance the relationship between denoising and texture preservation, we propose a multiscale residual dense dual attention network (MRDDANet) for SAR image denoising. This algorithm can effectively suppress the speckle while fully retaining the texture details of the image. In MRDDANet, shallow features are extracted from the noisy images by multiscale modules with different kernel sizes, and then, the extracted shallow features are mapped to the residual dense dual-attention network to obtain the deep features of SAR image. Finally, the final denoising image is generated through global residual learning. MRDDANet has advantages of both multiscale blocks and residual dense dual attention networks. The dense connection can fully extract features in the image, and the dual-channel attention enables MRDDANet to pay more attention to noise information, which is beneficial to remove noise and keep the details of the original image at the same time. Compared with state-of-the-art algorithms, the results of the experiment indicate that our method not only improves various objective indicators but also shows great advantages in visual effects.

This research article proposes an automatic frame work for detecting COVID -19 at the early stage using chest X-ray image. It is an undeniable fact that coronovirus is a serious disease but the early detection of the virus present in human bodies can save lives. In recent times, there are somany research solutions that have been presented for early detection, but there is still a lack in need of right and even rich technology for its early detection. The proposed deep learning model analysis the pixels of every image and adjudges the presence of virus. The classifier is designed in such a way so that, it automatically detects the virus present in lungs using chest image. This approach uses an image texture analysis technique called granulometric mathematical model. Selected features are heuristically processed for optimization using novel multi scaling deep learning called light weight residual-atrous spatial pyramid pooling (LightRES-ASPP-Unet) Unet model. The proposed deep LightRES-ASPPUnet technique has a higher level of contracting solution by extracting major level of image features. Moreover, the corona virus has been detected using high resolution output. In the framework, atrous spatial pyramid pooling (ASPP) method is employed at its bottom level for incorporating the deep multi scale features in to the discriminative mode. The architectural working starts from the selecting the features from the image using granulometric mathematical model and the selected features are optimized using LightRESASPP- Unet. ASPP in the analysis of images has performed better than the existing Unet model. The proposed algorithm has achieved 99.6% of accuracy in detecting the virus at its early stage. © 2022 Tech Science Press. All rights reserved.

It is well-known that the convolutional neural network (CNN) is an effective method for synthetic aperture radar (SAR) target classification. In the convolutional layer of CNN, convolutional kernels of different sizes can extract different feature information of the target. The small-size kernel can extract the local texture feature information, and the large-size kernel can extract the global contour feature information. Traditional CNN methods usually use fixed-size kernels for convolution, and they generally lose part of the target's feature information, resulting in the inaccurate classification of the SAR targets. This article proposes a novel CNN model based on multikernel-size feature fusion (MKSFF-CNN) for SAR target classification. MKSFF-CNN designs a convolutional methodology with a multichannel parallel topology, it uses convolutional kernels of different sizes to extract the multikernel-size deep features of the SAR target, and then, these features are fused in an optimal way to acquire the lowest loss. Moreover, MKSFF-CNN concatenates the fused features extracted by the convolutional layers of different dimensions to achieve the finest classification. MKSFF-CNN greatly elevates the feature representation completeness of the SAR targets so that more useful feature information can be exploited for SAR target classification. Undoubtedly, MKSFF-CNN can achieve a better classification performance compared with traditional CNN models with a fixed kernel size. The superiority of MKSFF-CNN is validated on the moving and stationary target acquisition and recognition (MSTAR) dataset with the detailed objective and subjective evaluation.

Convolutional neural networks (CNNs) have been widely used in hyperspectral image (HSI) classification. However, a shape-fixed convolution kernel cannot extract appropriate spatial-spectral features. Thus, we propose a novel two-stage classification method based on kernel-guided deformable convolution networks and double-window joint bilateral filter (KDCDWBF) for HSIs. First, according to the calculated similarity map, the shape of the kernel-guided deformable convolution (KDC) is more consistent with the real shape of land covers, so the KDC can extract more pure neighborhood spatial-spectral information. Then, using the piecewise smoothness property of the HSI, a double-window joint bilateral filter (DWJBF) is designed to complete the coarse-to-fine classification stage, which can solve the misclassification problem of single pixels and small regions. Experiments on two HSI datasets demonstrate that the proposed network can achieve better classification performance when compared with other state-of-the-art methods.

Compressed sensing (CS) demonstrates significant potential to improve image quality in 3-D millimeter-wave imaging compared with conventional matched filtering (MF). However, existing sparsity-driven 3-D imaging algorithms always suffer from large-scale storage, excessive computational cost, and nontrivial tuning of parameters due to the huge-dimensional matrix-vector multiplication in complicated iterative optimization steps. In this article, we present a novel range migration (RM) kernel-based iterative-shrinkage thresholding network, dubbed as RMIST-Net, by combining the traditional model-based CS method and data-driven deep learning method for near-field 3-D millimeter-wave (mmW) sparse imaging. First, the measurement matrices in ISTA optimization steps are replaced by RM kernels, by which matrix-vector multiplication is converted to the Hadamard product. Then, the modified ISTA optimization is unrolled into a deep hierarchical architecture, in which all parameters are learned automatically instead of manually tuned. Subsequently, 1000 pairs of oracle images with randomly distributed targets and their corresponding echoes are simulated to train the network. A well-trained RMIST-Net produces high-quality 3-D images from range-focused echoes. Finally, we experimentally prove that RMIST-Net is capable process 512 x 512 large-scale imaging tasks within 1 s. Besides, we compare RMIST-Net with other state-of-the-art methods in near-field 3-D imaging applications. Both simulations and real-measured experiments demonstrate that RMIST-Net produces impressive reconstruction performance while maintaining high computational speed compared with conventional and sparse imaging algorithms.

Object detection is a fundamental part of the interpretation of remote sensing imagery. The one-stage object detector has been adopted into this field because of its high computational efficiency. However, this detector suffers from the misalignment among predefined anchor, object, and feature extracted by standard convolution kernel both in spatial and scale. It limits the further improvement of performance, especially for the long-narrow and multiscale geospatial objects. In this article, the problem is defined as the feature misalignment problem. To deal with this issue, an efficient feature aligned single-shot detector (ASSD) is proposed, which consists of two modules: a novel pseudo anchor proposal module (PAPM) and a flexible context-based feature alignment module (CFAM). The PAPM replaces the regular anchor group with the proposed core anchor and refines it to get aligned locations. It can tackle the spatial misalignment between anchors and their corresponding objects and alleviate the negative/positive imbalance problem. Then, the CFAM adaptively adjusts the sampling points of the convolution kernel and collects the context information according to the aligned core anchor. This plug-and-play module can effectively rectify the misalignment between kernel and objects and extract aligned and robust features. A series of comprehensive experiments are conducted on two large-scale public remote sensing object detection datasets. Experiment results suggest that the proposed method is effective to alleviate the misalignment problem. Compared with the baseline model, the detection accuracy is improved by 8.5% mAP and 11.0% mAP on the challenging benchmark for object detection in optical remote sensing image (DIOR) and a large-scale dataset for object detection in aerial image (DOTA) dataset, respectively. Our best-resulting model achieves the state-of-the-art performance, surpassing other one-stage detectors both on the two datasets at a high detection speed of 21 FPS.

The fusion of infrared and visible images combines the advantages of thermal radiation information in infrared images and texture information in visible images. To preserve salient targets and obtain better fusion results, this paper proposes a novel infrared and visible image fusion method based on dual-kernel side window filtering and detail optimization with S-shaped curve transformation. First, a dual-kernel side window box filter (DSWBF) with adaptive filter kernel size is designed to extract the base and detail layers of the source images. Then, a saliency-based fusion rule is proposed for the base layers to highlight the salient regions of the infrared image. Next, a detail optimization module is constructed based on an S-shaped curve transformation and guided filter to optimize the detail layer of the infrared image. The optimized detail layer is then merged with the detail layer of the visible image to obtain the fused detail layer. Finally, the fusion image is created by reconstructing the fused base and detail layers. Experimental results on the two public datasets indicate that compared with state-of-the-art methods, the proposed method can obtain better fusion performance in terms of subjective and objective evaluations.

Blind pansharpening addresses the problem of generating a high spatial-resolution multi-spectral (HRMS) image given a low spatial-resolution multi-spectral (LRMS) image with the guidance of its associated spatially misaligned high spatial-resolution panchromatic (PAN) image without parametric side information. In this paper, we propose a fast approach to blind pansharpening and achieve state-of-the-art image reconstruction quality. Typical blind pansharpening algorithms are often computationally intensive since the blur kernel and the target HRMS image are often computed using iterative solvers and in an alternating fashion. To achieve fast blind pansharpening, we decouple the solution of the blur kernel and of the HRMS image. First, we estimate the blur kernel by computing the kernel coefficients with minimum total generalized variation that blur a downsampled version of the PAN image to approximate a linear combination of the LRMS image channels. Then, we estimate each channel of the HRMS image using local Laplacian prior to regularize the relationship between each HRMS channel and the PAN image. Solving the HRMS image is accelerated by both parallelizing across the channels and by fast numerical algorithms for each channel. Due to the fast scheme and the powerful priors we used on the blur kernel coefficients (total generalized variation) and on the cross-channel relationship (local Laplacian prior), numerical experiments demonstrate that our algorithm outperforms state-of-the-art model-based counterparts in terms of both computational time and reconstruction quality of the HRMS images., Comment: 17 pages, 47 figures, journal, accepted by IEEE Transactions on Geoscience and Remote Sensing

Integrating compressed sensing (CS) with millimeter-wave (mmW) holography has shown great potential to achieve lightweight onboard hardware, low sampling ratio, and high-speed sensing. However, conventional CS-driven algorithms are always limited by nontrivial adjusting of parameters and excessive computational cost caused by plenty of iterations. To address this problem, we propose a lightweight model-based deep learning framework (LFIST-Net) for mmW 3-D holography, by combining the interpretability of fast iterative shrinkage-thresholding algorithm (FISTA) and tuning-free merit of data-driven deep neural network. First, the single-frequency (SF) holographic imaging technique is integrated into FISTA, which serves as the sensing kernels, to avoid large-scale matrix multiplications. Subsequently, the kernel-based FISTA (KFISTA) is mapped into layer-fixed and parameter-learnable LFIST-Net, whose weights are relaxed to be layer-varied. The updating of key parameters in LFIST-Net, including step sizes, thresholds, and momentum coefficients, are regularized by soft-plus function to ensure the non-negativity and monotonicity. As for 3-D holography implementation, the ``1-D + 2-D'' scheme is adopted, where the matched filtering (MF) and well-trained LFIST-Net are used for range focusing and reconstructions of azimuth slices. Without losing efficiency, the range-focused subechoes are processed parallelly in 3-D cube form. Experiments, including both simulated and measured tests based on a commercial mmW radar, prove that LFIST-Net is capable of reconstructing the imaging scene precisely. In particular, in near-field mmW 3-D holography tests, both numerical and visual results demonstrate LFIST-Net yields compelling reconstruction performance while maintaining high computational speed compared with MF-based, conventional CS-driven, and network-based methods.

In recent days, facial emotion recognition techniques are considered important and critical due wide array of application domains use facial emotion as part of their workflow and analytics gathering. Reduced recognition rate, inefficient computation and increased time consumption are the major drawbacks of these techniques. To overcome these issues, this paper develops the new facial emotion recognition technique named as Minimal Angular Feature Oriented Neural Network based Emotion Prediction (MAFONN-EP). Initially, the input video sequence are categorized into image frames that are pre-processed by eliminating the noise by using Weighted Median Filtering (WMF) technique and to separate the background and foreground regions using Edge Preserved Background Separation and Foreground Extraction (EPBSFE) technique. Then, the set of texture patterns are extracted based on four key parts such as two eyes, nose and mouth by using the Minimal Angular Deviation (MAD) technique. Particular features are selected by employing the Cuckoo Search based Particle Swarm Optimization (CS-PSO) technique that also reduces the feature dimensionality. Finally, the Weight Based Pointing Kernel Classification (WBPKC) technique is employed for recognizing the emotion. In the experimental results, the performance of the proposed technique is analyzed and compared with different performance measures like accuracy, sensitivity, specificity, precision, recall.

Many data mining workloads are being analyzed in large-scale distributed cloud computing environments which provide nearly infinite resources with diverse hardware configurations. To maintain cost-efficiency in such environments, understanding the characteristics and estimating the overheads of a distributed matrix multiplication task that is a core computation kernel in many machine learning algorithms are essential. This study aims to propose a Matrix Multiplication Performance Estimator on Cloud (MPEC) algorithm. The proposed algorithm predicts the latency incurred when executing distributed matrix multiplication tasks of various input sizes and shapes with diverse instance types and a different number of worker nodes on cloud computing environments. To achieve this goal, we first analyze the characteristics of distributed matrix multiplication tasks. With characteristics generated from qualitative analysis, we propose to apply an ensemble of non-linear regression algorithm to predict the execution time of arbitrary matrix multiplication tasks. Thorough experimental results reveal that the proposed algorithm demonstrates higher accuracy than a state-of-the-art machine learning task performance estimation engine, Ernest, by decreasing the Mean Absolute Percentage Error (MAPE) in half.

Inevitable corruption and degeneration make the performance of subsequent high-level semantic tasks in hyperspectral images (HSIs) unsatisfactory. Despite that many denoising methods have been proposed, significant room for improvement still remains. To better suppress noise and preserve the HSI spatial–spectral structure, we propose an attention-based Octave dense network. A separable spectral feature extraction module is introduced to extract the spatial–spectral features consistent with the structure prior. The extracted features are fine-tuned by the attention module in both channel and spatial domains; then, several dense denoising blocks are elaborately employed to focus on noise feature learning; in order to focus on high-frequency features, which usually have more noise information, we introduce the Octave kernel to implement these blocks. Experiments based on simulated and real-world noisy images demonstrate that the proposed method outperforms the existing traditional and learning-based methods in both quantitative evaluations and visual effects, benefiting the subsequent classification task. In addition, the effectiveness of each module is proven by ablation experiments. Our source code is made available at: https://github.com/LbzSteven/AODN.

Hyperspectral images (HSIs) have gained high spectral resolution due to recent advances in spectral imaging technologies. This incurs problems, such as an increased data scale and an increased number of bands for HSIs, which results in a complex correlation between different bands. In the applications of remote sensing and earth observation, ground objects represented by each HSI pixel are composed of physical and chemical non-Euclidean structures, and HSI classification (HIC) is becoming a more challenging task. To solve the above problems, we propose a framework based on a deep attention graph convolutional network (DAGCN). Specifically, we first integrate an attention mechanism into the spectral similarity measurement to aggregate similar spectra. Therefore, we propose a new similarity measurement method, i.e., the mixed measurement of a kernel spectral angle mapper and spectral information divergence (KSAM-SID), to aggregate similar spectra. Considering the non-Euclidean structural characteristics of HSIs, we design deep graph convolutional networks (DeepGCNs) as a feature extraction method to extract deep abstract features and explore the internal relationship between HSI data. Finally, we dynamically update the attention graph adjacency matrix to adapt to the changes in each feature graph. Experiments on three standard HSI data sets, namely, the Indian Pines, Pavia University, and Salinas data sets, demonstrate that the DAGCN outperforms the baselines in terms of various evaluation criteria. For example, on the Indian Pines data set, the overall accuracy of the proposed method achieves 98.61% when the training sample is 10%.

Currently, the top-performing face detectors use Convolution Neural Networks (CNN). These detectors have high performance but require large computational resources. Hence, it is a challenge to use these for practical applications with limited resources. To address this problem, this research paper proposes a single-stage face detector for practical applications with better accuracy and speed. The proposed face detector is a lightweight and fast CNN-based face detector. To achieve fast detection speed, by swiftly reducing the size of the input feature space through a carefully designed architecture using depth-wise convolution and Receptive Field Blocks (RBF). The high accuracy of the method is achieved by preserving the feature information from the early layers using CReLU (Concatenate Rectified Linear Units) activation in the CNN architecture. To detect faces of different pixel sizes, we used parallel RF blocks, which use multiple filters of different kernel sizes and dilation. The proposed method was tested on benchmark datasets such as WIDER FACE, AFW, and PASCAL faces, and performed well when compared to recent state-of-the-art real-time face detection algorithms.

Gradual advances have occurred in the design of micro-artificial intelligence for resource-limited hardware. A high-resolution (HR) image reconstruction module is indispensable for video analytics chips or devices at the edge. This paper proposes a low-cost and learning-based interpolation method for HR image reconstruction. The proposed method generates reconstructed pixels by processing reference pixels with optimal weights, which are pre-trained by solving the minimum mean square error problem for real images. To reduce the number of computation units and the usage of learned weights, a cross-directional interpolation architecture, which includes a vertical kernel and a horizontal kernel, is adopted. Moreover, a one-dimensional feature discriminator is proposed to improve the quality of up-scaled images efficiently. The main benefit of the proposed method is that it requires a small number of computation units but can still produce high-quality images. The hardware architecture of the proposed method was implemented on a field-programmable gate array (FPGA) by using Xilinx UltraScale+ ZCU102 and an application-specific integrated circuit (ASIC) by using TSMC’s 0.13-μm technology. On the ASIC, the proposed hardware required only approximately 60K gate counts and 50 KBytes of memory. The experimental results indicate that the average peak signal-to-noise ratio of the up-scaled images reached 35.92 dB for the Set-5 dataset. The throughput of the proposed hardware was at least 1000 Mpixels/s on the FPGA and 1200 Mpixels/s on the ASIC, which indicates that the proposed hardware can handle a target resolution higher than 4K in real time.

At present, in the new hardware design work of deep learning, memristor as a non-volatile memory with computing power has become a research hotspot. The weights in the deep neural network are the floating-point number. Writing a floating-point value into a memristor will result in a loss of accuracy, and the writing process will take more time. The binarized neural network (BNN) binarizes the weights and activation values that were originally floating-point numbers to +1 and -1. This will greatly reduce the storage space consumption and time consumption of programming the resistance value of the memristor. Furthermore, this will help to simplify the programming of memristors in deep neural network circuits and speed up the inference process. This paper provides a complete solution for implementing memristive BNN. Furthermore, we improved the design of the memristor crossbar by converting the input feature map and kernel before performing the convolution operation that can ensure the sign of the input voltage of each port constant. Therefore, we do not need to determine the sign of the input voltage required by the port in advance which simplifies the process of inputting the feature map elements to each port of the crossbar in the form of voltage. At the same time, in order to ensure that the output of the current convolution layer can be directly used as the input of the next layer, we have added a corresponding processing circuit, which integrates batch-normalization and binarization operations.

Effectively recognizing different sceneries with complex backgrounds and varied lighting conditions plays an important role in modern AI systems. Competitive performance has recently been achieved by the deep scene categorization models. However, these models implicitly hypothesize that the image-level labels are 100% correct, which is too restrictive. Practically, the image-level labels for massive-scale scenery sets are usually calculated by external predictors such as ImageNet-CN. These labels can easily become contaminated because no predictors are completely accurate. This article proposes a new deep architecture that calculates scene categories by hierarchically deriving stable templates, which are discovered using a generative model. Specifically, we first construct a semantic space by incorporating image-level labels using subspace embedding. Afterward, it is noticeable that in the semantic space, the superpixel distributions from identically labeled images remain unchanged, regardless of the image-level label noises. On the basis of this observation, a probabilistic generative model learns the stable templates for each scene category. To deeply represent each scenery category, a novel aggregation network is developed to statistically concatenate the CNN features learned from scene annotations predicted by HSA. Finally, the learned deep representations are integrated into an image kernel, which is subsequently incorporated into a multiclass SVM for distinguishing scene categories. Thorough experiments have shown the performance of our method. As a byproduct, an empirical study of 33 SIFT-flow categories shows that the learned stable templates remain almost unchanged under a nearly 36% image label contamination rate.

In this paper, an in-depth study and analysis of attribute modelling and knowledge acquisition of massive images are conducted using image recognition. For the complexity of association relationships between attributes of incomplete data, a single-output subnetwork modelling method for incomplete data is proposed to build a neural network model with each missing attribute as output alone and other attributes as input in turn, and the network structure can deeply portray the association relationships between each attribute and other attributes. To address the problem of incomplete model inputs due to the presence of missing values, we propose to treat and describe the missing values as system-level variables and realize the alternate update of network parameters and dynamic filling of missing values through iterative learning among subnets. The method can effectively utilize the information of all the present attribute values in incomplete data, and the obtained subnetwork population model is a fit to the attribute association relationships implied by all the present attribute values in incomplete data. The strengths and weaknesses of existing image semantic modelling algorithms are analysed. To reduce the workload of manually labelling data, this paper proposes the use of a streaming learning algorithm to automatically pass image-level semantic labels to pixel regions of an image, where the algorithm does not need to rely on external detectors and a priori knowledge of the dataset. Then, an efficient deep neural network mapping algorithm is designed and implemented for the microprocessing architecture and software programming framework of this edge processor, and a layout scheme is proposed to place the input feature maps outside the kernel DDR and the reordered convolutional kernel matrices inside the kernel storage body and to design corresponding efficient vectorization algorithms for the multidimensional matrix convolution computation, multidimensional pooling computation, local linear normalization, etc., which exist in the deep convolutional neural network model. The efficient vectorized mapping scheme is designed for the multidimensional matrix convolution computation, multidimensional pooling computation, local linear normalization, etc. in the deep convolutional neural network model so that the utilization of MAC components in the core loop can reach 100%.

Advances in image object detection lead to applying deep convolution neural networks in the document image analysis domain. Unlike general colorful and pattern-rich objects, tables in document images have properties that limit the capacity of deep learning structures. Significant variation in size and aspect ratio and the local similarity among document components are the main challenges that require both global features for detection and local features for the separation of nearby objects. To deal with these challenges, we present TableSegNet, a compact architecture of a fully convolutional network to detect and separate tables simultaneously. TableSegNet consists of a deep convolution path to detect table regions in low resolution and a shallower path to locate table locations in high resolution and split the detected regions into individual tables. To improve the detection and separation capacity, TableSegNet uses convolution blocks of wide kernel sizes in the feature extraction process and an additional table-border class in the main output. With only 8.1 million parameters and trained purely on document images from the beginning, TableSegNet has achieved state-of-the-art F1 score at the IoU threshold of 0.9 on the ICDAR2019 and the highest number of correctly detected tables on the ICDAR2013 table detection datasets.

Deep learning makes radar-based human activity recognition (HAR) attract more attention in the fields of intelligent security, traffic management, medical rehabilitation and military operation, because it has the ability to automatically extract comprehensive features of human activity. This paper proposes a recognition method based on multi-spectrogram and mixed convolutional neural network (MCNN). Specifically, three time-frequency analyses, including short-time Fourier transform (STFT), reduced interference distribution with Hanning kernel (RIDHK) and smoothed pseudo Wigner-Ville distribution (SPWVD), are performed on radar echo data to obtain the time-frequency spectrograms with different feature expressions, and then the spectrograms are fed into the MCNN for recognition and classification. In the MCNN, three two-dimensional CNNs (2DCNNs) are used to extract the independent spatial features from three types of spectrograms, and one three-dimensional CNN (3DCNN) with unit convolution kernel is employed to focus on extracting the correlation features between three kinds of spectrograms. The behavioral features in the spectrograms are characterized comprehensively by fusing these two kinds of features, which is able to improve the recognition accuracy. Experimental results illustrate that the proposed method improves the average recognition accuracy of eight human activities by at least 2.16% compared with other methods.

Subpixel mapping (SPM) is an effective way to solve the mixed pixel problem, which is a ubiquitous phenomenon in remotely sensed imagery, by characterizing subpixel distribution within the mixed pixels. In fact, the majority of the classical and state-of-the-art SPM algorithms can be viewed as a convolution process, but these methods rely heavily on fixed and handcrafted kernels that are insufficient in characterizing a geographically realistic distribution image. In addition, the traditional SPM approach is based on the prerequisite of abundance images derived from spectral unmixing (SU), during which process uncertainty inherently exists and is propagated to the SPM. In this article, a kernel-learnable convolutional neural network (CNN) framework for subpixel mapping (SPMCNN-F) is proposed. In SPMCNN-F, the kernel is learnable during the training stage based on the given training sample pairs of low- and high-resolution patches for learning a geographically realistic prior, instead of fixed priors. The end-to-end mapping structure enables direct subpixel information extraction from the original coarse image, avoiding the uncertainty propagation from the SU. In the experiments undertaken in this study, two state-of-the-art super-resolution networks were selected as application demonstrations of the proposed SPMCNN-F method. In experiment part, three hyperspectral image data sets were adopted, two in a synthetic coarse image approach and one in a real coarse image approach, for the validation. Additionally, a new data set with pairs of Moderate-resolution Imaging Spectroradiometer (MODIS) and Landsat images were adopted in a real coarse image approach, for further validation of SPMCNN-F in large-scale area. The restored fine distribution images obtained in all the experiments showed a perceptually better reconstruction quality, both qualitatively and quantitatively, confirming the superiority of the proposed SPM framework.

For multi-sensor fusion of infrared and visible images, it is difficult to retain the thermal radiation information of the infrared image and the texture information of the visible image in the fused image. To overcome this problem, a novel infrared and visible image fusion method based on a modified side window filter (MSWF) and an intensity transformation is proposed. First, an MSWF with effective edge-preservation ability is developed by adding four additional kernels to better decompose the source images to obtain the base and detail layers. Furthermore, to extract the edge information of the source images, we propose to further decompose the base layers to obtain the low-frequency layers and high-frequency layers (edge information) through the non-subsampled shearlet transform (NSST). Then, an S-shape intensity transformation function (ITF) is proposed to enhance the saliency information and suppress the non-saliency information in the infrared image. In the fusion process, considering the characteristics of the decomposed components, different fusion rules are designed to obtain the fused detail layer and low- and high-frequency layers. Finally, these fused components are reconstructed to obtain the final fusion image. It is experimentally demonstrated that the proposed method is superior to state-of-the-art fusion methods both in terms of subjective evaluation and objective metrics.

Deep learning has attracted rapidly increasing attention in the field of tomographic image reconstruction, especially for CT, MRI, PET/SPECT, ultrasound and optical imaging. Among various topics, sparse-view CT remains a challenge which targets a decent image reconstruction from ultra-sparse projections. To address this challenge, in this article we propose a Dual-domain Residual-based Optimization NEtwork (DRONE). DRONE consists of three modules respectively for embedding, refinement, and awareness. In the embedding module, a sparse sinogram is first extended. Then, sparse-view artifacts are effectively suppressed by the image domain networks. After that, the refinement module focuses on the recovery of image details in the residual data and image domains synergistically. Finally, the results from embedding and refinement components in the data and image domains are regularized for optimized image quality in the awareness module, which ensures the consistency between measurements and images with the kernel awareness of compressed sensing. The DRONE network is trained, validated, and tested on preclinical and clinical datasets, demonstrating its merits in edge preservation, feature recovery, and reconstruction accuracy.

The multi-exposure fusion (MEF) methods have received much attention in recent years due to the importance of constructing high dynamic range images. Among most of the existing studies, multi-scale structural-patch-decomposition-based MEF (MSPD-MEF) has achieved state-of-the-art fusion quality and the fastest running time. However, this method still suffers from detail loss in the fused images. To tackle this issue, we first incorporate the edge-preserving factors into this method to preserve the details in the fused images in a single-scale setting. Then, we develop the novel and flexible bell curve function, which can further preserve the details in both bright and dark regions. After that, we also show that our method can seamlessly plug in to this multi-scale framework. Extensive experimental results indicate that the proposed method can produce pleasing fusion results with little artifacts and low computational cost in both static and dynamic scenes.

Blind image deblurring is a severely ill-posed problem in low-level vision. The success of blind image deblurring relies on statistical priors and well-designed regularizers to obtain a clear image. However, the prior-based method is time-consuming due to a lot of nonlinear calculations. To improve efficiency, this work proposes an enhanced salient edge selection for blind image deblurring. Different from the previous methods that only focus on the salient edge and ignore the image structure, the image sharpening operator is adopted to guide the finer image structure when salient edges provide strong edge information for blur kernel estimation. We find that the latent image restoration can be enhanced by joining the salient edge selection and an image sharpening operator, and the quality of the recovered kernel is hereby improved. By imposing L2 constraints on the salient edge and sharpening operator terms, a new energy function is introduced and an effective alternating optimization strategy is explored. Extensive experiments have been conducted, which shows that our method is more effective compared with state-of-the-art methods.

Motion blur is a common artifact in image processing, specifically in e-health services, which is caused by the motion of a camera or scene. In linear motion cases, the blur kernel, i.e., the function that simulates the linear motion blur process, depends on the length and direction of blur, called linear motion blur parameters. The estimation of blur parameters is a vital and sensitive stage in the process of reconstructing a sharp version of a motion blurred image, i.e., image deblurring. The estimation of blur parameters can also be used in e-health services. Since medical images may be blurry, this method can be used to estimate the blur parameters and then take an action to enhance the image. In this paper, some methods are proposed for estimating the linear motion blur parameters based on the extraction of features from the given single blurred image. The motion blur direction is estimated using the Radon transform of the spectrum of the blurred image. To estimate the motion blur length, the relation between a blur metric, called NIDCT (Noise-Immune Discrete Cosine Transform-based), and the motion blur length is applied. Experiments performed in this study showed that the NIDCT blur metric and the blur length have a monotonic relation. Indeed, an increase in blur length leads to increase in the blurriness value estimated via the NIDCT blur metric. This relation is applied to estimate the motion blur. The efficiency of the proposed method is demonstrated by performing some quantitative and qualitative experiments.

Big challenges are usually occurring in non-rigid 3D shape retrieval, for the shapes undergoing arbitrarily non-affine transformations. In this work, a novel design of feature learning approach is proposed for non-rigid 3D shape retrieval, dubbed Structured Sparsity Regularized Multi-Modality Method (SSR-MM). The shape signatures which capture the deformation-invariant characteristics are averaged and stacked for a multi-modality machine learning approach, and a transform matrix based on the structure sparsity regularization is utilized to map those signatures obtaining the discriminative features for retrieval. The proposed framework is evaluated on the publicly available non-rigid 3D human benchmarks, and the experimental results show the efficacy of our contributions and the advantages of our method over existing ones.

COVID-19 is a virus that has been declared an epidemic by the world health organization and causes more than 2 million deaths in the world. To achieve this, computer-aided automatic diagnosis systems are created on medical images. In this study, an image processing and machine learning-based method is proposed that enables segmenting of CT images taken from COVID-19 patients and automatic detection of the virus through the segmented images. The main purpose of the study is to automatically diagnose the COVID-19 virus. The study consists of three basic steps: preprocessing, segmentation and classification. Image resizing, image sharpening, noise removal, contrast stretching processes are included in the preprocessing phase and segmentation of images with Expectation–Maximization-based Gaussian Mixture Model in the segmentation phase. In the classification stage, COVID-19 is classified as positive and negative by using kNN, decision tree, and two different ensemble methods together with the kernel support vector machines method. In the study, two different CT datasets that are open to the public and a mixed dataset created by combining these datasets were used. The best accuracy values for Dataset-1, Dataset-2 and Mixed Dataset are 98.5%, 86.3%, 94.5%, respectively. The achieved results prove that the proposed approach advances state-of-the-art performance. Within the scope of the study, a GUI that can automatically detect COVID-19 has been created. © 2021, King Fahd University of Petroleum & Minerals. 21.317, NKUBAP.06 This work was supported by Research Fund of the Tekirdag Nam?k Kemal University. Project Number: NKUBAP.06.GA.21.317 This work was supported by Research Fund of the Tekirdag Namık Kemal University. Project Number: NKUBAP.06.GA.21.317

The performance of vision systems can be affected when used in severe weather conditions such as heavy rain or snow. Rain streak removal is an ill posed problem as they can vary in shape, size, and density across the image. In this paper, a single-image deraining network named Factorized Multi-scale Multi-resolution Residual Network (FMMRNet), which follows a U-Net backbone, is proposed. As rain streaks affect non-local regions of the image, larger receptive fields are beneficial to capture these non-local dependencies. We propose the use of multi-scale grouped convolutions to integrate information from global and local scales. In the proposed FMMRNet, multi-scale convolutions are factorized so that they can have large effective kernel sizes while reducing the computational complexity. During training, intermediate multi-resolution outputs are produced for loss computation which improves gradient flow in the deeper layers of the network and promotes better learning. A channel-wise attention mechanism is included to recalibrate feature maps before fusion instead of direct concatenation at each stage of the decoder. A higher-level reconstruction loss called perceptual loss is included for effective training to improve the visual quality of the derained images. The performance of the proposed FMMRNet is quantitatively and qualitatively compared on public benchmark datasets, and it outperforms the state-of-the-art deraining methods. Furthermore, to show the practical applicability of the proposed network, we demonstrate that when FMMRNet is used as a preprocessing step for object-detection methods such as Faster-RCNN and YOLO, it improves their performance on images degraded by rain streaks by 12.6% and 75.2% respectively.

Presently, it is still challenging to obtain satisfied identification results for long-distance safety monitoring with fiber distributed acoustic sensor (DAS) in practical complicated burying environments. Thus, extracting increasingly abundant features has always been the direction of DAS signal recognition. This paper proposes a new recognition method using an end-to-end mCNN-HMM combined model, which can identify the vibration sources more correctly by simultaneously extracting multi-scale structural features and the sequential information of the DAS signals. A modified multi-scale convolution neural network (mCNN) is designed to automatically extract the DAS signals' local structural features from a multilevel perspective and their relationship in the proposed model. A hidden Markov model (HMM) is then used to mine the sequential information of the whole sample's previously extracted features. The test results based on real field data show that it outperforms the HMM model with the all-around hand-crafted features, the CNN-HMM model, and the MS-CNN-HMM model in both the feature extraction ability and the recognition accuracy in the case of little increase in time consumption. Moreover, the Euclidean distance between the posterior probabilities classified correctly and incorrectly is proposed to evaluate the test samples' feature distinguishability for different recognition models. Then the feature extraction capabilities of the models can be measured in an objective parameter.

Achieving the high accuracy of blind roads and crosswalks recognition is important for blind guiding equipment to help blind people sense the surrounding environment. A lightweight semantic segmentation network is proposed to quickly and accurately segment blind roads and crosswalks in a complex road environment. Specifically, a lightweight network with depthwise separable convolution as a component is used as a basic module to reduce the number of parameters of the model and increase the speed of semantic segmentation. In order to ensure the segmentation accuracy of the network, we use a densely connected atrous spatial pyramid pooling module to extract feature information of different angles and context feature modules to enhance the effectiveness of different levels of feature information fusion. To verify the effectiveness of the proposed method, we collect and produce a data set from a real environment, which contains two objects of blind roads and crosswalks 1 . Experimental results demonstrate that, compared to some state-of-the-art approaches, the proposed approach greatly improves the segmentation speed, while achieving better or similar accuracy, which shows that the proposed approach provides a better basis for the application of devices for guiding the blind. 1 https://github.com/qweawq/Blind-road-and-crosswalk-dataset

Deep neural networks (DNNs) and convolutional neural networks (CNNs) have demonstrated greater robustness and accuracy in classifying two-dimensional images and three-dimensional point clouds compared to more traditional machine learning approaches. However, their main drawback is the need for large quantities of semantically labeled training data sets, which are often out of reach for those with resource constraints. In this study, we evaluated the use of simulated 3D point clouds for training a CNN learning algorithm to segment and classify 3D point clouds of real-world urban environments. The simulation involved collecting light detection and ranging (LiDAR) data using a simulated 16 channel laser scanner within the the CARLA (Car Learning to Act) autonomous vehicle gaming environment. We used this labeled data to train the Kernel Point Convolution (KPConv) and KPConv Segmentation Network for Point Clouds (KP-FCNN), which we tested on real-world LiDAR data from the NPM3D benchmark data set. Our results showed that high accuracy can be achieved using data collected in a simulator.

The geometric and color features of agricultural material along with related physical properties are critical to characterize and express its physical quality. The experiments were conducted to classify the physical characteristics like size, shape, color and texture and then workout the relationship between manual observations and using image processing techniques for weight and volume of the four wheat refractions i.e. sound, damaged, shriveled and broken grains of wheat variety PBW 725. A flatbed scanner was used to acquire the images and digital image processing method was used to process the images and output of image analysis was compared with the actual measurements data using digital vernier caliper. A linear relationship was observed between the axial dimensions of refractions between manual measurement and image processing method with R2 in the range of 0.798–0.947. The individual kernel weight and thousand grain weight of the refractions were observed to be in the range of 0.021–0.045 and 12.56–46.32 g respectively. Another linear relationship was found between individual kernel weight and projected area estimated using image processing methodology with R2 in the range of 0.841–0.920. The sphericity of the refractions varied in the range of 0.52–0.71. Analyses of the captured images suggest ellipsoid shape with convex geometry while the same observation was recorded by physical measurements also. A linear relationship was observed between the volume of refractions derived from measured dimensions and calculated from image with R2 in the range of 0.845–0.945. Various color and grey level co-variance matrix texture features were extracted from acquired images using the open-source Python programming language and OpenCV library which can exploit different machine and deep learning algorithms to properly classify these refractions.

Cloud segmentation is one of the hot tasks in the field of weather forecast, environmental monitoring, site selection for observatory, and other areas. In this letter, we proposed a new deep convolutional neural network architecture called CloudU-Net for daytime and nighttime cloud images’ segmentation. The net consists of dilated convolution, activation, batch normalization (BN), max pooling, upsampling, skip connection, and fully connected conditional random field (CRF) layers. The benefits of the net architecture are four aspects: First, the dilated convolution increases the receptive field of the filters to obtain more information of the context without increasing the extra amount of computation and the extra number of parameters. Second, the BN layer increases the speed of network training and prevents over-fitting. Third, the fully connected CRF optimizes the output of the front end of the architecture, and finally gets better segmentation results. Finally, the enhanced optimizer Lookahead improves the learning stability and speeds up model convergence. Compared with the current deep-learning-based state-of-the-art cloud images’ segmentation algorithms, the CloudU-Net demonstrates better segmentation performance for daytime and nighttime cloud images.

Recently, the graph convolutional network (GCN) has drawn increasing attention in the hyperspectral image (HSI) classification. Compared with the convolutional neural network (CNN) with fixed square kernels, GCN can explicitly utilize the correlation between adjacent land covers and conduct flexible convolution on arbitrarily irregular image regions; hence, the HSI spatial contextual structure can be better modeled. However, to reduce the computational complexity and promote the semantic structure learning of land covers, GCN usually works on superpixel-based nodes rather than pixel-based nodes; thus, the pixel-level spectral–spatial features cannot be captured. To fully leverage the advantages of the CNN and GCN, we propose a heterogeneous deep network called CNN-enhanced GCN (CEGCN), in which CNN and GCN branches perform feature learning on small-scale regular regions and large-scale irregular regions, and generate complementary spectral–spatial features at pixel and superpixel levels, respectively. To alleviate the structural incompatibility of the data representation between the Euclidean data-oriented CNN and non-Euclidean data-oriented GCN, we propose the graph encoder and decoder to propagate features between image pixels and graph nodes, thus enabling the CNN and GCN to collaborate in a single network. In contrast to other GCN-based methods that encode HSI into a graph during preprocessing, we integrate the graph encoding process into the network and learn edge weights from training data, which can promote the node feature learning and make the graph more adaptive to HSI content. Extensive experiments on three data sets demonstrate that the proposed CEGCN is both qualitatively and quantitatively competitive compared with other state-of-the-art methods.