Search

This article proposes a CBAM-ASPP-SqueezeNet model based on the attention mechanism and atrous spatial pyramid pooling (CBAM-ASPP) to solve the problem of robot multi-target grasping detection. Firstly, the paper establishes and expends a multi-target grasping dataset, as well as introduces and uses transfer learning to conduct network pre-training on the single-target dataset and slightly modify the model parameters using the multi-target dataset. Secondly, the SqueezeNet model is optimized and improved using the attention mechanism and atrous spatial pyramid pooling module. The paper introduces the attention mechanism network to weight the transmitted feature map in the channel and spatial dimensions. It uses a variety of parallel operations of atrous convolution with different atrous rates to increase the size of the receptive field and preserve features from different ranges. Finally, the CBAM-ASPP-SqueezeNet algorithm is verified using the self-constructed, multi-target capture dataset. When the paper introduces transfer learning, the various indicators converge after training 20 epochs. In the physical grabbing experiment conducted by Kinova and SIASUN Arm, a network grabbing success rate of 93% was achieved.

Infrared small target detection, especially under low SCR conditions and complex backgrounds, is still a challenging research task. Considering the scale change caused by small targets rapidly moving, in this paper, a nonparametric regression-based multi-scale gradient correlation filtering (MGCF) detection method is proposed. First, a nonparametric regression method is applied to calculate the gradient of each point. Then, based on the unique gradient characteristics of small targets, a multi-scale gradient correlation (MGC) template is designed to distinguish small targets from clutter. After that, a multi-scale gradient correlation filtering method is proposed to enhance the target intensity and suppress clutter. At last, based on the obtained filtering response, an adaptive threshold segmentation method is adopted to extract real small targets. Experimental results demonstrate that the proposed method can fully improve the signal-to-clutter ratio (SCR) of small targets under different complex backgrounds. Moreover, compared with other baseline methods, the proposed method exhibits excellent detection performance.

Lightweight semantic segmentation promotes the application of semantic segmentation in tiny devices. The existing lightweight semantic segmentation network (LSNet) has the problems of low precision and a large number of parameters. In response to the above problems, we designed a full 1D convolutional LSNet. The tremendous success of this network is attributed to the following three modules: 1D multi-layer space module (1D-MS), 1D multi-layer channel module (1D-MC), and flow alignment module (FA). The 1D-MS and the 1D-MC add global feature extraction operations based on the multi-layer perceptron (MLP) idea. This module uses 1D convolutional coding, which is more flexible than MLP. It increases the global information operation, improving features’ coding ability. The FA module fuses high-level and low-level semantic information, which solves the problem of precision loss caused by the misalignment of features. We designed a 1D-mixer encoder based on the transformer structure. It performed fusion encoding of the feature space information extracted by the 1D-MS module and the channel information extracted by the 1D-MC module. 1D-mixer obtains high-quality encoded features with very few parameters, which is the key to the network’s success. The attention pyramid with FA (AP-FA) uses an AP to decode features and adds a FA module to solve the problem of feature misalignment. Our network requires no pre-training and only needs a 1080Ti GPU for training. It achieved 72.6 mIoU and 95.6 FPS on the Cityscapes dataset and 70.5 mIoU and 122 FPS on the CamVid dataset. We ported the network trained on the ADE2K dataset to mobile devices, and the latency of 224 ms proves the application value of the network on mobile devices. The results on the three datasets prove that the network generalization ability we designed is powerful. Compared to state-of-the-art lightweight semantic segmentation algorithms, our designed network achieves the best balance between segmentation accuracy and parameters. The parameters of LSNet are only 0.62 M, which is currently the network with the highest segmentation accuracy within 1 M parameters.

Automatic and accurate segmentation of optic disc (OD) and optic cup (OC) in fundus images is a fundamental task in computer-aided ocular pathologies diagnosis. The complex structures, such as blood vessels and macular region, and the existence of lesions in fundus images bring great challenges to the segmentation task. Recently, the convolutional neural network-based methods have exhibited its potential in fundus image analysis. In this paper, we propose a cascaded two-stage network architecture for robust and accurate OD and OC segmentation in fundus images. In the first stage, the U-Net like framework with an improved attention mechanism and focal loss is proposed to detect accurate and reliable OD location from the full-scale resolution fundus images. Based on the outputs of the first stage, a refined segmentation network in the second stage that integrates multi-task framework and adversarial learning is further designed for OD and OC segmentation separately. The multi-task framework is conducted to predict the OD and OC masks by simultaneously estimating contours and distance maps as auxiliary tasks, which can guarantee the smoothness and shape of object in segmentation predictions. The adversarial learning technique is introduced to encourage the segmentation network to produce an output that is consistent with the true labels in space and shape distribution. We evaluate the performance of our method using two public retinal fundus image datasets (RIM-ONE-r3 and REFUGE). Extensive ablation studies and comparison experiments with existing methods demonstrate that our approach can produce competitive performance compared with state-of-the-art methods.

At present, the traditional machine learning methods and convolutional neural network (CNN) methods are mostly used in image recognition. The feature extraction process in traditional machine learning for image recognition is mostly executed by manual, and its generalization ability is not strong enough. The earliest convolutional neural network also has many defects, such as high hardware requirements, large training sample size, long training time, slow convergence speed and low accuracy. To solve the above problems, this paper proposes a novel deep LeNet-5 convolutional neural network model for image recognition. On the basis of Lenet-5 model with the guaranteed recognition rate, the network structure is simplified and the training speed is improved. Meanwhile, we modify the Logarithmic Rectified Linear Unit (L ReLU) of the activation function. Finally, the experiments are carried out on the MINIST character library to verify the improved network structure. The recognition ability of the network structure in different parameters is analyzed compared with the state-of-the-art recognition algorithms. In terms of the recognition rate, the proposed method has exceeded 98%. The results show that the accuracy of the proposed structure is significantly higher than that of the other recognition algorithms, which provides a new reference for the current image recognition.

Monocular depth estimation is critical for endoscopists to perform spatial perception and 3D navigation of surgical sites. However, most of the existing methods ignore the important geometric structural consistency, which inevitably leads to performance degradation and distortion of 3D reconstruction. To address this issue, we introduce a gradient loss to penalize edge fluctuations ambiguous around stepped edge structures and a normal loss to explicitly express the sensitivity to frequently small structures, and propose a geometric consistency loss to spreads the spatial information across the sample grids to constrain the global geometric anatomy structures. In addition, we develop a synthetic RGB-Depth dataset that captures the anatomical structures under reflections and illumination variations. The proposed method is extensively validated across different datasets and clinical images and achieves mean RMSE values of 0.066 (stomach), 0.029 (small intestine), and 0.139 (colon) on the EndoSLAM dataset. The generalizability of the proposed method achieves mean RMSE values of 12.604 (T1-L1), 9.930 (T2-L2), and 13.893 (T3-L3) on the ColonDepth dataset. The experimental results show that our method exceeds previous state-of-the-art competitors and generates more consistent depth maps and reasonable anatomical structures. The quality of intraoperative 3D structure perception from endoscopic videos of the proposed method meets the accuracy requirements of video-CT registration algorithms for endoscopic navigation. The dataset and the source code will be available at https://github.com/YYM-SIA/LINGMI-MR.

Semantic segmentation can address the perceived needs of autonomous driving and micro-robots and is one of the challenging tasks in computer vision. From the application point of view, the difficulty faced by semantic segmentation is how to satisfy inference speed, network parameters, and segmentation accuracy at the same time. This paper proposes a lightweight multi-dimensional dynamic convolutional network (LMDCNet) for real-time semantic segmentation to address this problem. At the core of our architecture is Multidimensional Dynamic Convolution (MDy-Conv), which uses an attention mechanism and factorial convolution to remain efficient while maintaining remarkable accuracy. Specifically, LMDCNet belongs to an asymmetric network architecture. Therefore, we design an encoder module containing MDy-Conv convolution: MS-DAB. The success of this module is attributed to the use of MDy-Conv convolution, which increases the utilization of local and contextual information of features. Furthermore, we design a decoder module containing a feature pyramid and attention: SC-FP, which performs a multi-scale fusion of features accompanied by feature selection. On the Cityscapes and CamVid datasets, LMDCNet achieves accuracies of 73.8 mIoU and 69.6 mIoU at 71.2 FPS and 92.4 FPS, respectively, without pre-training or post-processing. Our designed LMDCNet is trained and inferred only on one 1080Ti GPU. Our experiments show that LMDCNet achieves a good balance between segmentation accuracy and network parameters with only 1.05 M.

Background Adamantinomatous craniopharyngioma (ACP) is a benign tumor with malignant clinical manifestations. ACP adjacent to the hypothalamus often presents with more severe symptoms and higher incidence of hypothalamic dysfunction. However, the mechanism underlying hypothalamic dysfunction remains unclear. Methods Immunostaining was performed to determine the nerve damage to the floor of the third ventricle (3VF) adjacent to ACP and to examine the recruitment and senescence of hypothalamic neural stem cells (htNSCs). The accumulation of lipid droplets (LDs) in htNSCs was evaluated via BODIPY staining, oil red O staining, and transmission electron microscopy. In vitro and in vivo assays were used to evaluate the effect of cystic fluid or oxidized low-density lipoprotein and that of oxytocin (OXT) on htNSC senescence and the hypothalamic function. The protein expression levels were analyzed using western blotting. Results htNSCs with massive LD accumulation were recruited to the damaged 3VF adjacent to ACP. The LDs in htNSCs induced senescence and reduced neuronal differentiation; however, htNSC senescence was effectively prevented by inhibiting either CD36 or integrated stress response (ISR) signaling. Furthermore, OXT pretreatment reduced lipotoxicity via the inhibition of ISR signaling and the repair of the blood–brain barrier. Conclusions Reduced LD aggregation or ISR signaling inhibition prevented senescence in htNSCs and identified molecular pathways and potential therapeutic targets that may improve hypothalamic dysfunction in ACP patients.

Efficient and fast stereo matching is a challenging task due to the presence of occlusion and low texture areas. In stereo matching, the correspondence between left and right images may be difficult owing to the lack of matching information. The cost metrics proposed before are not robust enough or are computationally expensive. In this work, we propose a novel generic tree structure, Pyramid-tree, which improves the single mode of traditional tree, and can achieve cross-regional connection between different regions with similar colors and similar depths. This unique structure can achieve cross-regional cost smoothing, significantly reducing the possibility of mismatch due to occlusion or lack of corresponding matching information, and has stronger robustness to occlusion and low texture regions. In addition, we also propose a new bearings-only cost metric, Log-angle, which is not affected by occlusion, low texture, illumination and other factors. Log-angle combines with traditional metrics can show better performance. We show that the Pyramid-tree structure and Log-angle are very important as it efficiently expands the state-of-the-art stereo matching methods and leads to significant improvements. Qualitative and quantitative experiments on Middlebury data sets verify the superior performance of the algorithm, and very effective compromise between the accuracy and computation load is achieved.

Recently, single image super-resolution (SISR) has been widely applied in the fields of multimedia and computer vision communities and obtained remarkable performance. However, most current methods ignore to utilize multi-granularity features of low-resolution (LR) image to further improve the SISR performance. And the channel and spatial features obtained from original LR images are treated equally, resulting in unnecessary computations for abundant uninformative features, thereby hindering the representational ability of super-resolution (SR) models. In this paper, we present a novel Multi-Granularity Pyramid Attention Network (MGPAN) which fully exploits the multi-granularity perception and attention mechanisms to improve the quality of reconstructed images. We design a multi-branch dilated convolution layer with varied kernels corresponding to receptive fields of different sizes to modulate multi-granularity features for adaptively capturing more important information. Moreover, a novel spatial pyramid pooling attention (SPPA) module is constructed to integrate the channel-wise and multi-scale spatial information, which is beneficial to compute the response values from the multi-scale regions of each neuron, and then establish the accurate mapping from low to high dimensional solution space. Besides, for long-short-term information preservation and information flow enhancement, we adopt the short, long, and global skip connection structures to concatenate and fuse the states of each module, which can improve the SR network performance effectively. Extensive experiments on several standard benchmark datasets show that the proposed MGPAN can provide state-of-the-art or even better performance in both quantitative and qualitative measurements.

Although recently a series of deep‐learning based methods have promoted the performance of face super resolution (FSR), most of these methods cannot recover essential face attributes accurately, especially to super‐resolve a very low‐resolution (LR) face image (16×16 pixels) to its (8×) high‐resolution (HR) version. To address this issue, a novel alternating optimisation algorithm to estimate facial attributes and restore facial images in a single network is presented . Specifically, two convolutional neural modules (denoted as Restorer and Corrector) are constructed and these two modules are alternated repeatedly to form an end‐to‐end trainable network. The Restorer module reconstructs face images based on estimated facial attributes, while the Corrector module corrects the estimated attributes with the help of restored face image. Since Corrector can utilise information from previous estimated attributes and the FSR image, the estimated attributes are iteratively corrected and gradually approach the ground truth. Moreover, a new attribute transformation scheme is designed to introduce attribution information into Restorer, in which the facial attribute vectors as the control conditions can guide the face image restoration explicitly. By conducting extensive experiments on the well‐known CelebA dataset, it is demonstrated that the proposed method can provide superior FSR performance in both quantitative and qualitative measurements.

We consider visual tracking in numerous applications of computer vision and seek to achieve optimal tracking accuracy and robustness based on various evaluation criteria for applications in intelligent monitoring during disaster recovery activities. We propose a novel framework to integrate a Kalman filter (KF) with spatial-temporal regularized correlation filters (STRCF) for visual tracking to overcome the instability problem due to large-scale application variation. To solve the problem of target loss caused by sudden acceleration and steering, we present a stride length control method to limit the maximum amplitude of the output state of the framework, which provides a reasonable constraint based on the laws of motion of objects in real-world scenarios. Moreover, we analyze the attributes influencing the performance of the proposed framework in large-scale experiments. The experimental results illustrate that the proposed framework outperforms STRCF on OTB-2013, OTB-2015 and Temple-Color datasets for some specific attributes and achieves optimal visual tracking for computer vision. Compared with STRCF, our framework achieves AUC gains of 2.8%, 2%, 1.8%, 1.3%, and 2.4% for the background clutter, illumination variation, occlusion, out-of-plane rotation, and out-of-view attributes on the OTB-2015 datasets, respectively. For sporting events, our framework presents much better performance and greater robustness than its competitors.

Blind face restoration (BFR) from severely degraded face images is important in face image processing and has attracted increasing attention due to its wide applications. However, due to the complex unknown degradations in real-world scenarios, existing priors-based methods tend to restore faces with unstable quality. In this article, we propose a multi-prior collaboration network (MPCNet) to seamlessly integrate the advantages of generative priors and face-specific geometry priors. Specifically, we pretrain a high-quality (HQ) face synthesis generative adversarial network (GAN) and a parsing mask prediction network, and then embed them into a U-shaped deep neural network (DNN) as decoder priors to guide face restoration, during which the generative priors can provide adequate details and the parsing map priors provide geometry and semantic information. Furthermore, we design adaptive priors feature fusion (APFF) blocks to incorporate the prior features from pretrained face synthesis GAN and face parsing network in an adaptive and progressive manner, making our MPCNet exhibits good generalization in a real-world application. Experiments demonstrate the superiority of our MPCNet in comparison to state-of-the-arts and also show its potential in handling real-world low-quality (LQ) images from several practical applications.

Salient object detection is getting more and more attention in computer vision field. In this paper, we propose a novel and effective framework for salient object detection. Firstly, we develop a robust background-based map by using spatial prior to remove the foreground noises of image boundary regions. The proposed background-based map and Objectness map are integrated to obtain a coarse saliency map. Then, an effective saliency propagation mechanism is utilized to further highlight salient object and suppress background region by defining a novel graph model, each node connects to its more similar neighbors and nodes with low saliency values in the proposed graph. As a result, the coarse saliency map is optimized to the refined saliency map by novel graph based saliency propagation. Finally, we construct a novel integration framework to further integrate two saliency maps for performance improvement. Experiments on three benchmark datasets are tested, experimental results show the superiority of the proposed algorithm than other state-of-the-art methods.

Low-dose CT (LDCT) images have been widely applied in the medical imaging field due to the potential risk of exposing patients to X-ray radiations. Given the fact that reducing the radiation dose may result in increased noise and artifacts, methods that can eliminate the noise and artifacts in the LDCT image have drawn increasing attentions and produced impressive results over the past decades. However, recent proposed methods mostly suffer from noise remaining, over-smoothing structures or false lesions derived from noise. In this paper, we propose a generative adversarial network (GAN) with novel architecture and loss function for restoring the LDCT image. Firstly, the inception-residual block and residual mapping are incorporated in the U-Net structure. The modified U-Net is applied as the generator of the GAN network so that the noise feature can be eliminated during the forward propagation. Secondly, a novel multi-level joint discriminator is designed by concatenating multiple convolutional neural networks (CNNs) where the output of each deconvolutional layer in the generator is compared with the corresponding down-sampled ground truth image. The adversarial training can be sensitive to noise and artifacts in different scales with this discriminator. Thirdly, we novely define a loss function consisting of the least square adversarial loss, VGG based perceptual loss, MSE based pixel loss and the noise loss, so that the differences in pixel, visual perception and noise distribution are comprehensively considered to optimize the network. Experimental results on both simulated and official simulated clinical images have demonstrated that the proposed method can provide superior performance to the state-of-the-art methods in noise removal, structure preservation and false lesions elimination.

Recently, single image super-resolution (SISR) has been widely applied in the field of remote sensing image processing and obtained remarkable performance. However, existing CNN-based remote sensing image super-resolution methods are unable to exploit shallow visual characteristics at global receptive fields, which results in the limited perceptual capability of these models. Furthermore, the low-resolution inputs and features contain abundant low-frequency information, which are weighed in channels and space equally, hence limiting the representational ability of CNNs. To solve these problems, we propose a non-locally up-down convolutional attention network (NLASR) for remote sensing image super-resolution. First, a non-local features enhancement module (NLEB) is constructed to obtain the spatial context information of high-dimensional feature maps, which allows our network to utilize global information to enhance low-level similar structured texture information with effect, overcoming the defects of deficiency perceptual ability of shallow convolutional layers. Second, an enhanced up-sampling channel-wise attention (EUCA) module and enhanced down-sampling spatial-wise attention (EDSA) module are proposed to weight the features at multiple scales. By integrating the channel-wise and multi-scale spatial information, the attention modules are able to compute the response values from the multi-scale regions of each neuron and then establish the accurate mapping from low to high resolution space. Extensive experiments on NWPU-RESISC45 and UCMerced-LandUse datasets show that the proposed method can provide state-of-the-art or even better performance in both quantitative and qualitative measurements.

Image clustering is a complex procedure that is significantly affected by the choice of the image representation. Generally speaking, image representations are generated by using handcraft features or trained neural networks. When dealing with high dimension data, these two representation methods cause two problems: i) the representation ability of the manually designed features is limited; ii) the non-representative and meaningless feature of a trained deep network may hurt the clustering performance. To overcome these problems, we propose a new clustering method which efficiently builds an image representation and precisely discovers the cluster assignments. Our main tools are an unsupervised representation learning method based on Deep Mutual Information Maximization (DMIM) system, and a clustering method based on self-training algorithm. Specifically speaking, to extract the informative representation of image data, we derive the maximum mutual information theory and propose a system to learn the maximum mutual information between the input images and the latent representations. To discover the clusters and assign each image a clustering label, a self-training mechanism is applied to cluster the learned representations. The superiority and validity of our algorithm are verified in a series of real-world image clustering experiments.

The state-of-the-art trackers using deep learning technology have little special strategy to gain the bounding box well when the target suffers drastic geometric deformation. In this paper, we take full use of the convolutional neural network (CNN) features of the deepest layer to represent the semantic feature model, and affine transformation to be as the space information model. A tracking method based on geometrical transformation region CNN is proposed. Firstly, affine transformation is applied to predict possible locations of a target, and the candidate bounding boxes obtained by affine transformation sampling can locate the possible geometric regions of the target more effectively before extracting features from CNN. Furthermore, RoI pooling with different sizes and shapes are designed to describe the geometric deformation region of the target. Then, multi-tasks loss function including the affine transformation regression is designed to refine the affine bounding box. Finally, the affine transformation NMS (Non-maximum suppression) is used to ensure the tracking bounding box having the largest IoU value. Extensive experimental results show that the proposed algorithm performs favorably against the compared methods in the public benchmarks.

For current learning-based single image deraining methods, deraining networks are usually designed based on a simplified linear additive rain model, which may not only cause unreal synthetic rainy images for both training and testing datasets, but also adversely affect the applicability and generality of corresponding networks. In this paper, we use the screen blend model of Photoshop as the nonlinear rainy image decomposition model. Based on this model, we design a novel channel attention U-DenseNet for rain detection and a residual dense block for rain removal. The detection sub-network not only adjusts channel-wise feature responses by our novel channel attention block to pay more attention to learn the rain map, but also combines the context information with the precise localization by the U-DenseNet to promote pixel-wise estimation accuracy. After rain detection, we use the nonlinear model to get a coarse rain-free image, and then introduce a deraining refinement subnetwork consisted of the residual dense block to obtain a fine rain-free image. For training our network, we apply the nonlinear rain model to synthesize a benchmark dataset called as RITD. It contains 3200 triplets of rainy images, rain maps, and clean background images. Our extensive quantitative and qualitative experimental results show that our method outperforms several state-of-the-art methods on both synthetic and real images.

With the rapid development of the Internet of Things (IoT) and smart cities, more and more types of applications have been emerging. In fact, different applications have different features and different requirements on services. In order to satisfy users' Quality of Service (QoS) requirements, the application-awareness technique should be leveraged to distinguish different applications for providing the differentiated services. However, the traditional Internet only can obtain the local network view, which belongs to the offline awareness mode and cannot adapt to the dynamical network environment. At the right time, Software-Defined Networking (SDN) has been accepted as a new networking paradigm thanks to its network awareness on the global status information, which can greatly facilitate the online application-awareness. At present, three ways, i.e., port number, depth packet inspection and deep learning can be used for the application-awareness. To the best of our knowledge, the deep learning based application-awareness method is the most cutting-edge technique. In spite of this, the previous related schemes fail to effectively guarantee the correctness and stability. To this end, this paper proposes a Convolutional Neural Network (CNN) based deep learning mechanism to do the application-awareness, including three phases, i.e., traffic collection, data pre-processing and application-awareness. The SDN environment is implemented based on the MiniNet and the simulation experiments are made based on the TensorFlow. The experimental results show that the proposed application-awareness mechanism outperforms three benchmarks on recall ratio, precision ratio, F value and stability.

For visual tracking, key factors that affect the performance of trackers are related to whether it can effectively extract the appearance information and spatial information of a target. And most of state-of-the-art trackers either do not model the appearance information and spatial information separately or do not design special strategies to deal with the strong geometric deformation of the target. In this paper, we design an appearance information model and a spatial information model separately, and then combine them to obtain complementary benefits. Firstly, because the features from deeper layers of a convolutional neural network (CNN) can better describe the semantic information of a target while the spatial information becomes less, we adopt the features from the deepest layer as the appearance information model. Secondly, we focus on tracking the target with drastic geometric deformation through utilizing a projection transformation group (SL(3) group) to model the geometric transformation of the target, where SL(3) group can describe the geometric deformation more accurately. Furthermore, a standard discriminative correlation filter is used to develop the effect of convolutional features and is more efficient than other methods used for CNN. Extensive experiments results show that our tracker outperforms all the compared trackers.

Purpose A robotic wheelchair system was designed to assist disabled people with disabilities to walk. Design/methodology/approach An anticipated sharing control strategy based on topological map is proposed in this paper, which is used to assist robotic wheelchairs to realize interactive navigation. Then, a robotic wheelchair navigation control system based on the brain-computer interface and topological map was designed and implemented. Findings In the field of robotic wheelchairs, the problems of poor use, narrow application range and low humanization are still not improved. Originality/value In the system, the topological map construction is not restricted by the environment structure, which helps to expand the scope of application; the shared control system can predict the users’ intention and replace the users’ decision to realize human-machine interactive navigation, which has higher security, robustness and comfort.

Single image snow removal is important since snowy images usually degrade the performance of computer vision systems. In this paper, we deduce a physics-based snow model and propose a novel snow removal method based on the snow model and deep neural networks. Our model decomposes a snowy image into a nonlinear combination of a snow-free image and dynamic snowflakes. Inspired by our model and DenseNet connectivity pattern, we design a novel Multi-scale Stacked Densely Connected Convolutional Network (MS-SDN) to simultaneously detect and remove snowflakes in an image. The MS-SDN is composed of a multi-scale convolutional sub-net for extracting feature maps and two stacked modified DenseNets for snowflakes detection and removal. The snowflake detection sub-net guides snow removal through forward transmission, and the snowflake removal sub-net adjusts snow detection through back transmission. In this way, snowflake detection and removal mutually improve the final results. For training and testing our method, we constructed a large-scale benchmark synthesis dataset which contains 3000 triplets of snowy images, snowflakes, and snow-free images. Specifically, the snow-free images are captured from snow scenes, and the snowy images are synthesized by using our deduced snow model. Our extensive quantitative and qualitative experimental results show that our MS-SDN performs better than several state-of-the-art methods, and the stacked structure is better than multi-branch structures in terms of snow removal.

Computer tomography (CT) has played an essential role in the field of medical diagnosis, but the blurry edges and unclear textures in traditional CT images usually interfere the subsequent judgement from radiologists or clinicians. Deep learning based image super-resolution methods have been applied for CT image restoration recently. However, different levels of information of CT image details are mixed and difficult to be mapped from deep features by traditional convolution operations. Moreover, features representing regions of interest (ROIs) in CT images are treated equally as those for background, resulting in low concentration of meaningful features and high redundancy of computation. To tackle these issues, a CT image super-resolution network is proposed based on hybrid attention mechanism and global feature fusion, which consists of the following three parts: 1) stacked Swin Transformer blocks are used as the backbone to extract initial features from the degraded CT image; 2) a multi-branch hierarchical self-attention module (MHSM) is proposed to adaptively map multi-level features representing different levels of image information from the initial features and establish the relationship between these features through a self-attention mechanism, where three branches apply different strategies of integrating convolution, down-sampling and up-sampling operations according to three different scale factors; 3) a multidimensional local topological feature enhancement module (MLTEM) is proposed and plugged into the end of the backbone to refine features in the channel and spatial dimension simultaneously, so that the features representing ROIs could be enhanced while meaningless ones eliminated. Experimental results demonstrate that our method outperform the state-of-the-art super-resolution methods on restoring CT images with respect to peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indices.

Blind face restoration (BFR) from severely degraded face images is important in face image processing and has attracted increasing attention due to its wide applications. However, due to the complex unknown degradations in real-world scenarios, existing priors-based methods tend to restore faces with unstable quality. In this article, we propose a multi-prior collaboration network (MPCNet) to seamlessly integrate the advantages of generative priors and face-specific geometry priors. Specifically, we pretrain a high-quality (HQ) face synthesis generative adversarial network (GAN) and a parsing mask prediction network, and then embed them into a U-shaped deep neural network (DNN) as decoder priors to guide face restoration, during which the generative priors can provide adequate details and the parsing map priors provide geometry and semantic information. Furthermore, we design adaptive priors feature fusion (APFF) blocks to incorporate the prior features from pretrained face synthesis GAN and face parsing network in an adaptive and progressive manner, making our MPCNet exhibits good generalization in a real-world application. Experiments demonstrate the superiority of our MPCNet in comparison to state-of-the-arts and also show its potential in handling real-world low-quality (LQ) images from several practical applications.

Coronavirus Disease 2019 (COVID-19) has spread globally since the first case was reported in December 2019, becoming a world-wide existential health crisis with over 90 million total confirmed cases. Segmentation of lung infection from computed tomography (CT) scans via deep learning method has a great potential in assisting the diagnosis and healthcare for COVID-19. However, current deep learning methods for segmenting infection regions from lung CT images suffer from three problems: (1) Low differentiation of semantic features between the COVID-19 infection regions, other pneumonia regions and normal lung tissues; (2) High variation of visual characteristics between different COVID-19 cases or stages; (3) High difficulty in constraining the irregular boundaries of the COVID-19 infection regions. To solve these problems, a multi-input directional UNet (MID-UNet) is proposed to segment COVID-19 infections in lung CT images. For the input part of the network, we firstly propose an image blurry descriptor to reflect the texture characteristic of the infections. Then the original CT image, the image enhanced by the adaptive histogram equalization, the image filtered by the non-local means filter and the blurry feature map are adopted together as the input of the proposed network. For the structure of the network, we propose the directional convolution block (DCB) which consist of 4 directional convolution kernels. DCBs are applied on the short-cut connections to refine the extracted features before they are transferred to the de-convolution parts. Furthermore, we propose a contour loss based on local curvature histogram then combine it with the binary cross entropy (BCE) loss and the intersection over union (IOU) loss for better segmentation boundary constraint. Experimental results on the COVID-19-CT-Seg dataset demonstrate that our proposed MID-UNet provides superior performance over the state-of-the-art methods on segmenting COVID-19 infections from CT images.

Salient object detection, which aims at locating the most important object in a scene (or image), has been extensively studied in various tasks, such as robot vision. In this paper, we present an effective salient object detection framework based on a novel bottom-up semi-supervised learning algorithm, which obviously outperforms the existing works in complex scenes. Given an input image, it is firstly segmented into a fixed number of non-overlapping image patches as basic units. A novel segmentation-based sampling method is developed to select a subset of all image patches as training samples. Then, all samples are divided into labeled and unlabeled groups based on multiple prior cues. The labels of all the unlabeled data are inferred by a novel label propagation mechanism. As a result, a complete training set can be obtained and used to train a classifier to classify all image patches into salient object and background. In addition, we also use neighbor-constraint smoothness function to further boost the saliency map. We compare the proposed method with the state-of-the-art approaches on two datasets. Experimental results demonstrate the effectiveness and superiority of the proposed method.

Following the development of autonomous underwater vehicles (AUVs), multiple trajectory-based submarine target information collection constitutes one of the key technologies that significantly influence underwater information collection ability and deployment efficiency. In this paper, we propose an underwater information collection AUV, O-AUV, that can perceive the omnidirectional area and could detect a larger area than the traditional AUV. A 3D sensing model for the O-AUV is proposed to describe the complex underwater information collection spaces. Thereafter, a cube-based environment model involving candidate observation point calculation methods are suggested to adapt the O-AUV model. A voyage cost map is also built according to the multi-AUV path planning for a common submarine mission that must traverse numerous mission targets in complex environments through the R-Dijkstra algorithm. Specifically, the voyage planning problem is solved through a critical algorithm called ANSGA (accelerated NSGA-II algorithm), which in turn, is developed by modifying the non-dominated sorting genetic algorithm (NSGA-II) to accelerate the optimization rate for the Pareto solution. Experiments are carried out in MATLAB, and the results verify the validity of the proposed O-AUV+ANSGA algorithm framework.