Search

Accurate objects detection in remote sensing images is very important, because security, transportation, and rescue applications in military and civilian fields require fully analyzing and using these images. To address the problem that many small-sized objects in remote sensing images are difficult to detect, this paper proposes an improved S 2 ANET-SR model based on S 2 A-NET network. In this paper, the original and reduced image are fed to the detection network at the same time, and then a super-resolution enhancement module for the reduced image is designed to enhance the feature extraction of small objects, after that, the perceptual loss and texture matching loss is proposed as supervision. Extensional experiments are conducted to evaluate the performance on the general remote sensing dataset DOTA, and the results show that our proposed method can achieve 74.47% mAP, which is 0.79% better than the accuracy of S 2 A-NET.

Missing data imputation plays an important role in the data cleansing process. Clustering algorithms have been widely used for missing data imputation, yet, there is little research done on the use of clustering ensemble for missing data imputation, which aggregates multiple clustering results. This paper proposes a novel collaborative clustering-based imputation method, called COLI, which uses the imputation quality as a key criterion for the exchange of information between different clustering results. To the best of our knowledge, this is the first study on the impact of collaborative clustering on imputation performance. The main contributions of this paper are three-fold. A novel missing value imputation based on collaborative clustering is proposed, three amputation strategies are used to induce missingness on various complete and publicly available datasets with different mechanisms, distributions, and ratios, which allows evaluating the imputation quality of the proposed method in estimating missing values of various numerical datasets with different missingness mechanisms, distributions, and ratios. The proposed method is compared to several state-of-the-art imputation methods and attained results demonstrate that the proposed method is an effective method for handling missing data.

Automatic fingerprint identification is a key technology of biological attribute measurement. Due to its sensitivity to fake fingerprint attack, its security has been widely concerned. The fingermark is an important evidence for forensic documents identification. The existence of fake fingermark seriously threatens the fairness and legitimacy in the process of forensic identification. In this paper, in view of the security risks existing in the fingermark identification of forensic documents, a database named JLW-FM-DB is introduced for detecting genuine and fake fingermarks, which consists of two sub databases, signed and unsigned, each of which covers common fake fingermark materials. Based on the database, this paper proposes a method of fingermark anti-counterfeiting based on convolution neural network(CNN). A Patch-Label training strategy is proposed, which uses the unified label as the supervision signal for the class heatmap output by the last convolution layer. This strategy realizes stronger local supervision ability to input fingermark image and enhances local coding ability of CNN feature extraction. The experiments show that our methods are suitable for the detection of genuine and fake fingermarks, achieving 97.999% average accuracy in the signed fingermark database. The effectiveness of Patch-Label training strategy in fingermark anti-counterfeiting is also proved. Moreover, a model fusion of different models can further improve the detection ability of genuine and fake fingermarks, reaching to 98.496% average accuracy.

Recently, deep learning models have achieved great success in computer vision applications, relying on large-scale class-balanced datasets. However, imbalanced class distributions still limit the wide applicability of these models due to degradation in performance. To solve this problem, in this paper, we concentrate on the study of cross entropy which mostly ignores output scores on incorrect classes. This work discovers that neutralizing predicted probabilities on incorrect classes improves the prediction accuracy for imbalanced image classification. This paper proposes a simple but effective loss named complement cross entropy based on this finding. The proposed loss makes the ground truth class overwhelm the other classes in terms of softmax probability, by neutralizing probabilities of incorrect classes, without additional training procedures. Along with it, this loss facilitates the models to learn key information especially from samples on minority classes. It ensures more accurate and robust classification results on imbalanced distributions. Extensive experiments on imbalanced datasets demonstrate the effectiveness of the proposed method., 8 pages, Accepted to Pattern Recognition Letters (PRL), August 2021

This paper presents a new model for the task of scene text visual question answering, in which questions about a given image can only be answered by reading and understanding scene text that is present in it. The proposed model is based on an attention mechanism that attends to multi-modal features conditioned to the question, allowing it to reason jointly about the textual and visual modalities in the scene. The output weights of this attention module over the grid of multi-modal spatial features are interpreted as the probability that a certain spatial location of the image contains the answer text the to the given question. Our experiments demonstrate competitive performance in two standard datasets. Furthermore, this paper provides a novel analysis of the ST-VQA dataset based on a human performance study., Comment: This paper is under consideration at Pattern Recognition Letters

In this paper, we tackle the task of improving biometric verification in the context of Human-Robot Interaction (HRI). A robot that wants to identify a specific person to provide a service can do so by either image verification or, if light conditions are not favourable, through voice verification. In our approach, we will take advantage of the possibility a robot has of recovering further data until it is sure of the identity of the person. The key contribution is that we select from both image and audio signals the parts that are of higher confidence. For images we use a system that looks at the face of each person and selects frames in which the confidence is high while keeping those frames separate in time to avoid using very similar facial appearance. For audio our approach tries to find the parts of the signal that contain a person talking, avoiding those in which noise is present by segmenting the signal. Once the parts of interest are found, each input is described with an independent deep learning architecture that obtains a descriptor for each kind of input (face/voice). We also present in this paper fusion methods that improve performance by combining the features from both face and voice, results to validate this are shown for each independent input and for the fusion methods.

Images with visual and scene text content are ubiquitous in everyday life. However, current image interpretation systems are mostly limited to using only the visual features, neglecting to leverage the scene text content. In this paper, we propose to jointly use scene text and visual channels for robust semantic interpretation of images. We do not only extract and encode visual and scene text cues, but also model their interplay to generate a contextual joint embedding with richer semantics. The contextual embedding thus generated is applied to retrieval and classification tasks on multimedia images, with scene text content, to demonstrate its effectiveness. In the retrieval framework, we augment our learned text-visual semantic representation with scene text cues, to mitigate vocabulary misses that may have occurred during the semantic embedding. To deal with irrelevant or erroneous recognition of scene text, we also apply query-based attention to our text channel. We show how the multi-channel approach, involving visual semantics and scene text, improves upon state of the art., Comment: The paper is under consideration at Pattern Recognition Letters

A typhoon is a destructive weather system that can cause severe casualties and economic losses. Typhoon intensity (TI) is a measurement to evaluate its ruinous degree. Hence, typhoon intensity prediction is an important research problem and many methods have been proposed. However, most of the existing approaches have very limited capability to combine the 2D Typhoon Structure Domain-expert Knowledge (2D-TSDK) and the 3D Typhoon Structure Data-driven Knowledge (3D-TSDK) for the TI prediction. To address this issue, this paper proposes a spatio-temporal deep learning method named Spatial Attention Fusing Network (SAF-Net). The designed model aims to fuse the 2D-TSDK and the 3D-TSDK by developing a specific Wide & Deep framework. In the data-driven component, a special Spatial Attention (SA) module is designed to automatically select high-response wind speed areas and embedded into a three-branch CNN to exploit the 3D-TSDK. Then, the Wide & Deep framework integrates the 2D-TSDK and the 3D-TSDK for the TI prediction. Comprehensive experiments have been conducted on a real-world dataset, and the result shows that the proposed method outperforms state-of-the-art typhoon intensity prediction methods. The code is available in GitHub: https://github.com/xuguangning1218/TI_Prediction

Despite the significant advances in iris segmentation, accomplishing accurate iris segmentation and localization in non-cooperative environment with visible illumination remains a grand challenge. In this paper, we present an end-to-end multi-task deep neural network, referred to as Iris R-CNN, to achieve superior accuracy for iris segmentation and localization. Inspired by instance segmentation, Iris R-CNN seamlessly integrates segmentation and localization in a unified framework and generates a normalized iris image/mask required for iris recognition. It is composed of several novel techniques to carefully explore the unique characteristics of iris. First, we propose two novel networks, (i) Double-Circle Region Proposal Network (DC-RPN) and (ii) Double-Circle Classification and Regression Network (DC-CRN), to efficiently capture pupil and iris circles and enhance the accuracy for iris localization. Second, we propose a novel normalization scheme for Regions of Interest (RoIs) to enable a radically new pooling operation over a double-circle region. To facilitate accurate training and validation, we annotate two public datasets in non-cooperative environment with visible illumination: NICE-II and MICHE. Experimental results on these two challenging datasets demonstrate the superior accuracy of our proposed approach over other state-of-the-art methods.

Face anti-spoofing (FAS) plays a critical role in the face recognition community for securing the face presentation attacks. Many works have been proposed to regard FAS as a domain generalization problem for robust deployment in real-world scenarios. However, existing methods focus on extracting intrinsic spoofing cues to improve the generalization ability, yet neglect to train a robust classifier. In this paper, we propose a framework to improve the generalization ability of face anti-spoofing in two folds:) a generalized feature space is obtained via aggregation of all live faces while dispersing each domain’s spoof faces; and) a domain agnostic classifier is trained through low-rank decomposition. Specifically, a Common Specific Decomposition for Specific (CSD-S) layer is deployed in the last layer of the network to select common features while discarding domain-specific ones among multiple source domains. The above-mentioned two components are integrated into an end-to-end framework, ensuring the generalization ability to unseen scenarios. The extensive experiments demonstrate that the proposed method achieves state-of-the-art results on four public datasets, including CASIA-MFSD, MSU-MFSD, Replay-Attack, and OULU-NPU.

A set of advanced approaches and models on human action, activity, gesture, behavior and related aspects are summarized in this note. There are a number of challenges on these domains, and some of these are addressed here. Notably, Video-based human activity recognition, sensor-based activity analysis, skeleton-based activity recognition, assisted daily living, anomaly detection, facial expression and emotion analysis, gesture and sign language are highlighted in the works. This special issue also introduced six new datasets, while exploring a total of 55 different datasets in 21 papers. However, there are still various challenges that we need to solve in the future. Multi-modality, action quality assessment, real-time applications, and activity and behavior computing on edge devices are some of the dominant future challenges to deal with. We firmly believe that the original works and ideas presented in this special issue will serve as helpful references for the relevant research communities in the journey towards a brighter future.

With the popularity of mobile devices, location-based services, such as Foursquare and Facebook, have attracted increasing attention in recent years. Users share their interests anytime, anywhere with their friends on the social network. Point-of-Interest (POI) recommendation, as one of key services on Location-Based Social Networks (LBSNs), can effectively enhance users’ experience especially when they travel in a new city. Previous studies have made great success on POI recommendation by employing geographical influence and user preference. However, it is believed that the human decision on where to visit is very complex and involves the comprehensive factors such as POI popularity, POI location, user trajectory, and time context. In this paper, we propose a collective POIs recommendation framework which leverages the individual latent preference and contextual information. Firstly, to recommend top-K initial POIs, a scoring prediction model is constructed, which considers the influence of similarity, popularity and location of POIs. Furthermore, a next POI recommendation model based on personalized transfer probability is proposed, and the initial POI recommendation is combined to calculate the user’s score on the next POI. Extensive experiments based on real datasets collected from Foursquare demonstrate the proposed framework outperforms the state-of-art ones.

This paper describes the secure storage and matching of latent fingerprints using phase shifting digital holography. A phase shifting digital holographic system is proposed for imaging of latent fingerprints, which can be applied in the criminal investigation field. Digital holography is one of the accurate and fast technique for measuring 3D information, by acquiring both amplitude and phase information from the object. Digital holography uses a Charge-Coupled Device (CCD) to record the interference pattern. In digital holography, the reconstructed image unavoidably contains the conjugate image and the zeroth order beam, resulting in low resolution. This problem can be overcomed by using phase shifting digital holography. Phase shifting digital holography is a technique by which multiple holograms are taken corresponding to the various phase differences between the object beam and the reference beam. One of the most secure storage methods is digital holographic data storage which provides a credible way in the fingerprint acquisition in the criminal investigation field. The phase shifting digital holograms of the latent fingerprint are acquired and then numerically reconstructed. The reconstructed fingerprint is then compared with the database after further processing to find whether it is a match or no match. The investigations have been conducted through experiments and computer simulations. The parameters used for analyzing the performance of the system are False Matching Rate (FMR), False Non-Matching Rate (FNMR) etc. This system shows a remarkable performance improvement over digital holographic techniques. The method can be explicitly used in forensic applications.

When the images are captured by drones, the effect of oblique angles, distance variations and open environment are the main challenges for successful palm tree detection. This paper presents a method towards palm tree counting in Drone images using a novel idea of detecting dominant points by exploring Generalized Gradient Vector Flow, which defines symmetry based on gradient direction of the pixels. For each dominant point, we use angle information for classifying diagonal dominant points. It is intuition that the direction of the branches of tree converges at center of tree irrespective of the type of tree and plants. This observation motivated us to expand the direction of diagonal dominant points until it finds intersection point with another diagonal dominant point and this results in candidate points. For each candidate point, the proposed method constructs the ring by considering the distance between the intersection point and nearest neighbor candidate point as radius. This outputs region of interest and it includes center of each tree in the image. To ease the effect of complex background, we explore YOLOv5 architecture to remove false region of interests. This step results in counting oil palm trees in the mages irrespective of tree type of palm family. Experimental results on our dataset of the images captured by drones and standard dataset of coconut images captured by unmanned aerial vehicle of different trees show that the proposed method is effective and performs better than SOTA methods.

Automated visual stimuli evoked multi-channel electroencephalograph (EEG) signals classification is in a nascent stage but is receiving progressive attention from researchers. The conventional techniques existing for EEG classification tasks overlook the spatial attributes of EEG signals, which contain spatial data information to characterize an image from visual stimuli evoked EEG signal. In this paper, a Deep learning structure implemented on STFT (Short Term Fourier Transform) generated spectrogram images and a Capsule Network (EEGCapsNet) is proposed. In this architecture, the time and frequency domain as well as, spatial attributes of the multi-channel EEG signals are extracted to build the spectrogram image and are fed to the proposed EEGCapsNet for classifying those EEG signals which are acquired from the stimuli evoked visual experience while seeing an image. For this purpose, two different EEG datasets (namely Perceive and MindBig) are trained on the proposed network. The highest average accuracy of 81.59% and 84.62% is reported for the proposed EEGCapsNet.

Existing human pose estimation methods have some problems, such as key-point representation quantization error, algorithm inconsistency, and easy-to-introduce unequal quantization error. Addressing these issues, this paper proposes a method of human key-point coordinate representation based on probability distribution perception on a depth convolution neural network. In this method, the perceived probability of the key-point coordinate distribution is extracted by a convolutional neural network, and the cross entropy is formed with the prediction probability distribution as the loss function. Under the constraint of minimizing the Kullback-Leibler (KL) distance between the prediction and the ground truth, the convolution neural network is optimized, and the coordinates of the heat map are finally located. The experimental results show that compared with the traditional method based on a heat map, the proposed method can accelerate the reasoning speed of the model in the COCO training set, and at the same time, it can improve the map by 1.7

In this paper, we propose a novel image encryption method based on logistic chaotic systems and deep autoencoder. In the encryption phase, first, the plaintext image is randomly scrambled by a logistic chaotic system. Then, the random scrambled image is encoded by a deep autoencoder to generate the ciphertext image. In order to obtain the ciphertext image with uniform distribution, we incorporated the uniform distribution constraint into the training of the deep autoencoder. The resulting ciphertext image contains high randomness, which is critical for an excellent image encryption algorithm. Histogram analysis, information entropy analysis, key space analysis, key sensitivity analysis, correlation analysis, and ablation experiments show that the proposed encryption algorithm can effectively resist attacks and has excellent encryption performance while providing high security.

Pedestrian detection is a common task in the research area of video analysis and its results lay the foundations of a wide range of applications. It is commonly known that under challenging illumination and weather conditions, conventional visible cameras perform poorly and this limitation could be catered using thermal imagery. But, due to the fact that annotated thermal datasets are less available compared to the visible ones, in this paper we emphasis the need for leveraging information from the visible domain to perform detection in the thermal domain at no additional annotation cost. Precisely, we propose a domain adaptation method by incorporating feature distribution alignments into Faster R-CNN architecture at different levels and at two phases of the network. The resulting proposed adaptive detector has the advantage of covering different aspects of the domain shift in order to improve the overall performance. The proposed detector is evaluated on KAIST multispectral dataset and the obtained results demonstrate its effectiveness by improving the adaptability in the thermal domain. Also, by means of comparisons to other existing works, better results are obtained. Additional experiments are conducted on other datasets to further justify the obtained results.

Convolutional neural networks (CNN) has been widely used in biomedical image segmentation (BIS) tasks for its remarkable feature representation capability, and most of existing CNN-based segmentation networks leverage a down-sampling operation to achieve larger acceptance domain. However, down-sampling operations could inevitably loss the detailed information of images which is very important for the BIS task. In this paper, we propose a full-resolution biomedical image segmentation network(FRNet) that could maintain the integrated detailed information of image while keeping sufficient semantic information and large receptive field. Specifically, the basic semantic feature and non-destructive feature are employed to represent the semantic and detailed information of images, respectively. A backbone network and a new full-feature extraction branch are conducted to extract those two kinds of complementary features. Furthermore, a novel feature fusion module is designed to integrate those complementary features to achieve non-destructive description of images. Finally, in order to further improve the description ability of the integrated feature, a Densely connected Atrous Spatial Pyramid Pooling(DenseASPP) module is arranged at the end of our proposed FRNet to extract the multiscale information of images. Thorough experimental results on several available databases demonstrate the effectiveness and advancement of FRNet.

Telemedicine and various tele-medicinal applications are revolutionizing a variety of healthcare departments through its innovative means of remote diagnosing and faster first aid administration. Digital images play an important role in these applications for ensuring better and faster health carfigure. These digital images, which usually contain confidential and diagnostic information about the patients, are usually transmitted through public networks among hospitals, doctors and patients. Consequently, there is a need to secure them while being stored and in transit to guarantee the privacy of the patient. However, unique properties of digital image data, such as high redundancy and correlation between the pixels of the image and their large size, make the conventional cryptographical algorithms insufficient for ensuring proper security while encrypting them. As the conventional algorithms seize to be a reliable solution, the need to develop improved and dedicated image encryption algorithms also arises. Chaotic systems are systems that appear random and unpredictable from the outside but are governed by deterministic equations or rules on the inside. Due to these properties, along with ergodicity and its heightened sensitivity to the initial conditions, chaotic systems are devised to be one of the best candidates for securing the storage and transmission of digital images. However, the security of a chaotic image encryption system depends upon the chaotic behavior demonstrated by the chaotic map applied in the image encryption system. Because of this, different attacks can be used to break a chaotic encryption system if the scheme is not well-structured. This paper introduces a chaotic image encryption scheme that incorporates two chaotic maps, namely, Arnold's Cat Map and 2D Logistic-Sine-Coupling Map(2D-LSCM), for increased randomness and security of the encrypted image. We also analyze the performance and security of our scheme and compare it with other prominent chaotic image encryption schemes.

Precisely segmenting teeth from digitized 3D dental models is an essential task in computer-aided orthodontic surgical planning. In recent years, various deep learning-based methods have been proposed to process dental models for teeth segmentation, however, these methods usually ignore or coarsely model the dependency between vertices/mesh cells in local space, which fails to exploit local geometric details that are critical to capture complete teeth structure. In this paper, we propose a specific end-to-end network for teeth segmentation on 3D dental models. By constructing a graph for the raw mesh data, our network adopts a series of graph attentional convolution layers and a global structure branch to extract fine-grained local geometric feature and global feature, respectively. Subsequently, these two features are further fused to learn comprehensive information for cell-wise segmentation tasks. We have evaluated our network on a real-patient dataset of dental models acquired through 3D intraoral scanners, and experimental results show that our method outperforms state-of-the-art deep learning methods for 3D shape segmentation.

Over the years world is revolutionized with the advent of technology, the field of transmission too is made easier with the use of different technologies but security during transmission is still a threat. Visual cryptography (VC) is a cryptographic technique that encrypts visual information (pictures, text, etc.) where the decrypted content appears as a visual image. Visual cryptography enables to transmit images securely as well as maintains data confidentiality. The security of multimodal biometric data is a challenging task in the current world scenario as various domains are prone to more threats. In order to secure and authenticate multimodal biometric data a new framework is proposed. The proposed framework provides an optimal solution for securing and authenticating images during transmission. In this paper, three types of biometric inputs were resolute and taken as input: Fingerprint, Face and Iris images. A new shuffling approach which is based on a pixel element is used for creating shares for every plane. These shares are used in the decoding process for reconstruction. The three reconstructed images are passed through the bilateral filter in order to eliminate noise and preserve edges. Each biometric data (BD) utilizes different image segmentation technique: (1) Binary segmentation for finger-print image, (2) face segmentation for face image, and (3) Daugman's integrodifferential operational for iris image. Similarly, feature extraction modules are executed on three segmented images. These features extracted from the reconstructed images are used to train the Visual Sharing Neural Network (VSNN) along with Recurrent Neural Network- Bidirectional Long Short-Term Memory (RNN-BiLSTM) model. In the testing process, the input image is classified using VSNN and it checks for matching BD in the database before providing the authentication to individual. The simulation result, of the proposed module, gives the PSNR as 38.58, MSE as 9, and NCC as 1, 95% of accuracy, 100% of sensitivity, and 5% of error rate, 98.8% of identification rate, 0.45% of false acceptance rate, and 0.97% of false rejection rate.

Visual attributes detection provides rich semantic concepts for image captioning. Some previous methods attempt to directly encode the attributes into vectors and generate the corresponding captions, which ignore the correlations between the image regions and attributes. In this paper, we consider to bridge the gap between visual features and detected attributes: first to look at a specific region of the image and second to decide which attribute to attend to. We propose an attribute-driven image captioning approach consisting of two parts: the visual positioning part and the attribute selection part. Specifically, we introduce the pointer-generator network into the second part of our model as a soft-switch, which determines whether to generate a word through the hidden state or point to a detected attribute at each decoding step. Qualitative and Quantitative experiments show that our model can improve the coverage of key visual attributes and significantly boost the overall performance.

Recommender system has a pivotal role in electronic economy especially for the online shopping platforms. Studies over the past two decades have proved that exploiting the inherent properties of items contributes a lot to the accuracy of multimedia sequential recommendation. There is no doubt that multimedia information including images and texts of a product have an impact on user’s purchase decision. However, modeling user’s dynamic preferences for multimodal (visual and textual in this paper) information over time is still a challenging problem. To solve this problem, we propose a Multimodal-Adaptive Hierarchical Network (MAHN for short) for multimedia sequential recommendation, which includes a hierarchical recurrent neural network and an information modulation module between the hierarchical structure. Specifically, the hierarchical recurrent neural network achieves the re-selection of multimodal information from the first layer to the second layer, the information modulation module realizes the selection of each modal information at time step t based on the previous time steps. Finally, to improve the generalization ability of our model, we adopt the multi-task training style to jointly optimize BPR loss and reconstruction loss of multimodal information. Experiments are conducted on two real world public datasets, and the results demonstrate that our model outperforms the other methods.

E-mail is considered the commonly used and efficient way of communication over the globe. In the corporate sectors, the number of E-mails received every day is considerably high and the timely response to every E-mail is essential. Several researchers believe that natural language processing (NLP) techniques by the use of deep learning (DL) architectures have played a considerable part to reduce manual work for repeated E-mail responses and intended to develop E-mail systems with intelligent response function. In this view, this paper designs an intelligent DL enabled optimal bidirectional long short term memory (Bi-LSTM) technique for an automated E-mail reply (OBiLSTM-AER) of E-mail Client Prototype. The goal of the proposed model is to provide an automated E-mail reply solution for persons as well as corporates which receive massive identical E-mails daily. The presented model employs Glove and OBiLSTM model for feature extraction of receiving and response E-mails respectively. Finally, softmax classifier is applied to allocate the class labels. For improving the performance of the BiLSTM model, the hyperparameter tuning process takes place using an oppositional glowworm swarm optimization (OGSO) algorithm. An extensive set of simulations were performed to highlight the betterment of the proposed method and the results are examined interms of distinct measures.

This paper presents an effective dissimilarity measure for geometric graphs representing shapes. The proposed dissimilarity measure is a distance that combines a sparsification of the geometric graph based on the maximum diversity problem and a new node embedding that captures the topological neighborhood of nodes. The sparsification step aims to reduce the size of the graph and to correct the misdistribution of nodes on the geometric graph induced by the noise of image handling. Experimental evaluation shows that the sparsification algorithm retains the form of the shapes while decreasing the number of processed nodes which reduces the overall matching time. Furthermore, the proposed node embedding and similarity measure give better performance in comparison with existing graph matching approaches.

The data clustering technique plays a significant role for the process of analyzing the data in various fields such as, data mining, big data and image processing. As the health care domain needs various data processing to detecting and diagnosing the disease, in existing image and data mining helps to identify and diagnosis the disease specific to cancer as it need lot of attention for clustering those data with proper detection and accuracy. Apart from the categories of skin cancer types like breast cancer, blood cancer, skin cancer, etc., skin cancer is more complicated disease as it needs proper detection at the earlier stage and treatment. In this paper, we have proposed a combined approach of neural based K means approach and whale data classification-based skin cancer approach. In this approach, we have applied multi-layer (K-Means with whale optimization algorithm) data classification to detect the cluster region. Then whale approach with data classification helps to determine the mass density of user-based data cluster and also train the classified data to optimize along with the predominant features. Finally, this combined approach to classify the skin cancer with respect to segmentation, feature, optimization. This firefly optimization helps to reduce the detection error rate at the early stage along with data accuracy and sensitivity. Regarding the security aspects, the optimization algorithm is secure against DoS and DDoS to ensure data privacy and confidentiality based on the data accuracy and detection time parameters discussed. Our proposed method will be evaluated with the existing methods like K-Means with genetic; K-means with firefly optimization methods.

Efficiency is an important concern for practical applications, therefore, it is of great importance to build effective lightweight networks. This paper proposes a novel lightweight feature self-fusion convolutional (SFC) module, which consists of self-fusion and point-wise convolution. The core of SFC is a three-step self-fusion. First, each input feature map is expanded to a high dimensional space individually, prohibiting connections with other input channels. Then, in the second step, we fuse all features from the same input in the high dimensional space to enhance the representation ability. Finally, we compress high dimensional features to a low dimensional space. After self-fusion, we connect all features by one point-wise convolution. Compared to inverted bottleneck, SFC module decreases the number of parameters by replacing the dense connections among channels with self-fusion. To the best of our knowledge, SFC is the first method to build lightweight networks by feature self-fusion. We then build a new network namely SFC-Net, by stacking SFC modules. Experimental results on the CIFAR and downsampled ImageNet datasets demonstrate our SFC-Net achieves better performance to some previous popular CNNs with fewer number of parameters and achieves comparable performance compared to other previous lightweight architectures. The code is available at https://github.com/Yankeegsj/Self-fusion .

Documents scanning is still one of the widely used documents digitization steps; however, skew in scanned documents is inevitable. If this skew is not corrected, the extraction of region/s of interest (RoI) and further processing like; detection and classification on such RoI becomes difficult. It has been shown that skew detection and correction significantly improve Optical Character Recognition (OCR) systems’ accuracy. This paper introduces a novel, robust and straightforward skew detection method for scanned documents, which uses Probabilistic Hough Transformation (PHT) for line detection in a first step and clusters the lines in a second step based on parallelism. The cluster with maximum parallel lines represents the expected skewed lines. The proposed method is tested on real scanned images taken from the Document Image Skew Estimation Contest (DISEC’13), Pashto, and Tobacco800 datasets. The proposed method performs well both in terms of accuracy and efficiency. It is efficient and robust to noise. Furthermore, we show that it also works on Arabic and Latin scripts.

Neural networks are vulnerable to adversarial input perturbations imperceptible to human, which calls for robust machine learning for safety-critical applications. In this paper, we propose a new neural ODE layer which is inspired by Hopfield-type neural networks. We prove that the proposed ODE layer has global asymptotic stability on the projected space, which implies the existence and uniqueness of its steady state. We further show that the proposed layer satisfies the local stability condition such that the output is Lipschitz continuous in the ODE layer input, guaranteeing that the norm of perturbation on the hidden state does not grow over time. By experiments we show that an appropriate level of stability constraints imposed on the proposed ODE layer can improve the adversarial robustness of ODE layers, and present a heuristic method for finding good hyperparameters for stability constraints.

Accurate object recognition and pose estimation models are essential for practical applications of robot arms, such as picking products on a shelf. Training such a model often requires a large-scale dataset with qualified labels for both object classes and pose parameters, and collecting accurate pose labels is particularly costly. A recent paper Ueno et al.(2019)[28] proposed a triplet learning framework for joint object recognition and pose estimation without explicit pose labels by learning a spatial transformer network to estimate the pose difference of an input image from an anchor image depicting the same object in a reference pose. However, our analysis suggests that the pose estimation accuracy is severely degraded for input images with large pose differences. To address this problem, we propose a new learning approach called multiple-anchor triplet learning. The basic idea is to give dense reference poses by preparing multiple anchors so that there is at least one anchor image having a small pose difference to the input image. Our multiple-anchor triplet learning is an extension of the standard single-anchor triplet learning to the multiple-anchor case. Inspired by the idea of multiple instance learning, we introduce a selection layer that automatically chooses the best anchor for each input image and allows the network to be trained end-to-end to minimize triplet-based losses. Experiments with three benchmark datasets in product picking scenarios demonstrate that our method significantly outperforms existing methods in both object recognition and pose estimation accuracy.

Visual explanations aim at providing an understanding of the inner behavior of convolutional neural networks. Naturally, it is necessary to explore whether these methods themselves are reasonable and reliable. In this paper, we focus on Class Activation Mapping (CAM), a type of attractive explanations that has been widely applied to model diagnosis and weakly supervised tasks. Our contribution is two-fold. First, we identify an important but neglected issue that affects the reliability of CAM results: there is a misalignment between the effective receptive field and the implicit receptive field, where the former is determined by the model and the input, and the latter is determined by the upsampling function in CAM. Occlusion experiments are designed to empirically testify to its existence. Second, based on this finding, an adversarial marginal attack is proposed to fool the CAM-based method and the CNN model simultaneously. Experimental results demonstrate that the provided saliency map can be completely changed to another shape by only perturbing the area with 1-pixel width. The prototype code of the method is now available at https://github.com/xpf/CAM-Adversarial-Marginal-Attack .

Using single-scale prediction kernels or Region of Interest (RoI) pooling in the prediction modules of modern object detectors is not very successful in matching different scales of objects. State-of-the-art detectors with the feature pyramid structure built on different resolutions of feature maps can help alleviate this problem. Although with this structure, single-scale prediction kernels or RoI pooling still struggles to detect small objects, and simultaneously, the former continues to encounter the misalignment problem on very large objects. In this paper, we propose the attention-guided pyramidal prediction kernels module with a customized IoU-adaptive loss function to deal with the misalignment problem between the prediction module and different scales of objects. To mitigate the effect of heavy detection head, we also introduce the salient object regions recognition module to identify these regions that have strong object cues. Additionally, interleaved subsampling, as the proposed feature enhancement approach, is used to generate highly discriminative feature representations. We refer to the detection framework constituted by these proposed methods as SnipeDet. Results show that SnipeDet achieves 41.1 AP at the speed of 15.4 FPS on the MS COCO test-dev set with 512 × 512 input images, which outperforms state-of-the-art one-stage detectors and has a better trade-off between speed and accuracy.

Misinformation is a recurring problem that has experienced a significant growth in recent years due to the rapid development of the Internet. This development has driven the emergence of websites where their content is shared without control. This is even more dangerous in the health domain, given its specific nature and the increasing number of users searching for health-related information on the Internet. For these reasons, this information should be handled with special attention. In this paper, a novel system to detect misinformation in websites related to the health domain is presented. The proposed system uses text mining techniques and visual design features to estimate the trustworthiness of the website. It has been trained using human experts’ knowledge in the selected domain and their visual perception of the website design. Promising results have been obtained during the evaluation in the experimental stage.

Although text detection methods have addressed several challenges in the past, there is a dearth of effective methods for text detection in deformable images, such as images containing text embedded on cloth, banners, rubber, sports jerseys, uniforms, etc. This is because deformable regions contain surfaces of arbitrarily shapes, which lead to poor text quality. This paper presents a new method for deformable text detection in natural scene images. It is observed that although the shapes of characters change in a deformable region, the pixel values and spatial relationship between the pixels do not change. This motivated us to explore extraction of Maximally Stable Extremal Regions (MSER) in an image in which pixels that share common features are grouped into components. The unique character shape variations led us to explore harmonic features to represent the component shape variations, using which a classifier classifies text and non-text components from the output of the MSER step. Additionally, the objective of developing a lightweight method with low computational cost motivated us to introduce a modified Pixel Aggression Network (PAN) for text deformable text detection at a component level. Comprehensive experiments which include experiments on our Deformable Text Dataset (DTD) and standard natural scene text datasets, namely, MSRATD-500, ICDAR 2019 MLT, Total-Text, CTW1500, ICDAR 2019 ArT and DSTA1500 datasets show that the proposed model outperforms the existing methods for our dataset as well as the standard datasets.

Multimodal teaching activity faces significant problems in Visual Question Answering (VQA), which involves simultaneous comprehension with reduced performance fidelity. However, Conventional methods are employed for portrayal and queries in a defined manner, which fails to accomplish the required performance accuracy rate. For elucidating the excellent image and question representation, this paper suggests an Adaptive Deep Concatenated Coder Framework (ADC–CF) that enrolls both the image and question attributes simultaneously with the optimized residual layer. The Coder Framework comprises of cascaded layers of Encoder-Decoder architecture, which captures rich, meaningful query characteristics and image details through the use of keywords employing significant object areas in the picture. ADC–CF layer has an encoder segment that blueprints the self-recognition of queries in which questions are concatenated to limit the answers and decoder segment blueprints the commanded-recognition of images. The simulation results of ADC–CF are tested with both the VQA datasets 1.0 and 2.0 and manifests an improved performance accuracy ratio of 72.45% for 1.0 dataset and 73.57% for 2.0 datasets, thus proving the reliability of the proposed framework.

The current paper aims to analytically visualize the future outcomes that the post-pandemic India might have in store for its citizens. We use time series forecasting on various collected data and combined the statistics of economics-deciding parameters to forecast the trends that might be prevalent in the next year. Since, the data contains a single anomalous trend, even the Prophet model could not learn this property from the data since this trend is not seasonal in nature. The current study proposes a novel architecture to deal with these rare unusual trends by combining two models - one learning normal usual patterns and the other getting trained on usual as well as rare anomalous patterns. It could help in dealing with sudden hike patterns like due to COVID-19 in the data, and lead to better forecasting on future timeframes. We combined the results of two distinct time-forecasting models trained on two sets of data of varying timeline lengths, using parameters obtained from Least Squares Approximation (LSA). The LSA helps us find an approximate vector approximation so as to obtain a model performing closely to the actual.

Nowadays, fake news is widely spreading in various media, and this fake information is causing serious damage in many areas. Therefore, there is an increasing need to accurately detect fake news to prevent such damage. In this paper, we propose a novel method that uses graph and summarization techniques for fake news detection. Our proposed method represents the relationship of all sentences in a graph structure to accurately understand the context information of the document. Accordingly, the relationship between sentences in the graph is calculated as a score through the attention mechanism. Then, the summarization technique is used to reflect the sentence subject information in the graph update process. Our proposed method shows better performance than Karimi’s and BERT based models by approximately 10.34%p and 3.72%p, respectively.

Deep-learning frameworks have been widely used in deformable image registration(DIR), which is a fundamental step in medical image analysis. In traditional deep-learning-based DIR methods, pairwise registration is conducted for each image, cooperated with a dual-input structure. They only considered that the registered images need to be as similar as possible to the target image, while ignored that all registered images to the same target image should have similar appearance too. Therefore, in this paper we add the deformation constraints between different registered images to the registration network in order to enhance the appearance consistency of the registered images and improve the registration accuracy. Specifically, a Triple-Input-Unsupervised neural Network (TIUNet) is proposed to register two source images to a common target image at the same time. Then, a novel deformation-strengthened loss is designed to simulate the deformation constraint between the two warped source images. Finally, the deformation constraints between the source images and the target image are jointly optimized to obtain the deformation field model. The experimental results on the available tissue segmentation database demonstrate that our proposed TIUNet method shows a superior performance of deformation field model over several other advanced deformable image registration algorithms. Moreover, benefited from the triple-input structure, our proposed TIUNet method could expand the available training sets naturally and is suitable for the registration task with fewer training data.

Salient object detection (SOD) focuses on mimicking the attention mechanism in human vision system. Due to the limited information provided from the traditional image data, the image-based SOD advances are very challenging in some complex scenes. Inspired by the emerging event cameras that provide asynchronous measurements of local temporal contrast over a large dynamic range, we propose a new idea to extract more effective information from the combination of event flow and RGB images. In this paper, we construct an end-to-end joint network for salient object detection (ERSOD-Net), which simultaneously supervise the RGB image and the event data within the corresponding image exposure time. To fully exploit temporal information of event data, Long Short-Term Memory module is utilized to effectively process event and learn salient object event surfaces. Moreover, multi-level feature interaction is designed to fuse two complementary branches of image and event and to predict significance mapping. Finally, to demonstrate the effectiveness of our model, a real Event-RGB SOD dataset (ERSOD) is built by DAVIS camera. Experiments on both benchmark and ERSOD datasets show that the proposed event guided network greatly improves the SOD performance in different evaluation metrics. The code and datasets will be released later at https://github.com/jxr326/ERSOD-Net.

Recently, single image super-resolution (SISR), aiming to preserve the lost structural and textural information from the input low resolution image, has witnessed huge demand from the videos and graphics industries. The exceptional success of convolution neural networks (CNNs), has absolutely revolutionized the field of SISR. However, for most of the CNN-based SISR methods, excessive memory consumption in terms of parameters and flops, hinders their application in low-computing power devices. Moreover, different state-of-the-art SR methods collect different features, by treating all the pixels contributing equally to the performance of the network. In this paper, we take into consideration both the performance and the reconstruction efficiency, and propose a Light-weight multi-scale attention residual network (MSAR-Net) for SISR. The proposed MSAR-Net consists of stack of multi-scale attention residual (MSAR) blocks for feature refinement, and an up and down-sampling projection (UDP) block for edge refinement of the extracted multi-scale features. These blocks are capable of effectively exploiting the multi-scale edge information, without increasing the number of parameters. Specially, we design our network in progressive fashion, for substituting the large scale factors ( × 4) combinations, with small scale factor ( × 2) combinations, and thus gradually exploit the hierarchical information. In parallel, for modulation of multi-scale features in global and local manners, channel and spatial attention in MSAR block is being used. Visual results and quantitative metrics of PSNR and SSIM exhibit the accuracy of the proposed approach on synthetic benchmark super-resolution datasets. The experimental analysis shows that the proposed approach outperforms the other existing methods for SISR in terms of memory footprint, inference time and visual quality.

Matrix Factorization (MF) is a widely used collaborative filtering technique for effectively modeling a user-item interaction in recommender system. Despite the successful application of MF and its variants, the method proves to be effective only in situations where there is an abundance of user-item interactions. However, user-item interaction data are usually sparse, limiting the effectiveness of the method. In addressing this problem, recent methods have proposed to use knowledge graphs (KGs) as additional information to complement the sparse user-item interaction data. This has proved challenging given the complexity of the KG structure. In this paper, we propose a collaborative filtering method that takes advantage of knowledge graphs. More specifically, the embedding of a user and item are both grounded on the item’s attributes in the knowledge graph, and are aggregated with generic user and item representations modeled by MF for implicit recommendation. Our model has demonstrated to outperform the recent state-of-the-art method KGCN [18] in very sparse settings, showing an effective integration of KGs in recommender systems.

Graph semi-supervised learning (GSSL) plays an important role in data classification by leveraging the similarity across the graph topology and convex optimization with Laplacian-based regularization. However, the current algorithm to solve the problem is centralized approach calling for heavy computational cost, particularly when the data is of large volume. In this paper, an innovate distributed algorithm is proposed to solve the problem, which is based on the decomposition of the similar graph. Contrary to the centralized approach, the distributed algorithm only requires the neighboring information for solving the optimization. It is proved that difference between the solutions of the distributed algorithm and centralized counterpart is upper bounded. We apply the proposed algorithm to both the synthetic and real-world datasets. The numerical results verify the effectiveness of the proposed distributed algorithm.

Nonlinear dimensionality reduction based on preservation of geometry is an imperative task in many scientific applications such as face recognition, object detection, data visualization, and classification. The traditional methods such as Isomap, locally linear embedding (LLE), and Laplacian eigen map (LEM) preserve local or global geometry or both with several limitations such as topological instability in the presence of noise or large neighborhood, inability to deal with ill-connected data points, high sensitivity of embedding performance to selected neighbor number. They also fail to retain the statistical properties of the data such as mean and variance while reducing the data dimensionality. In this paper, we propose a novel method, referred to as geometry and statistics-preserving embedding (GSE), which preserves both geometry and statistics of the data as well as overcomes some critical limitations of the above-mentioned methods. Moreover, GSE offers nonlinear mapping based on position information of the geometry, which is not possible by any other available method. We run experiments with GSE on several datasets including facial images and single cell RNA sequence data, where the GSE has been found to provide significantly better manifold embedding results than the traditional methods. The implementation codes of GSE are available at https://github.com/tauhidstanford/GSE.

Neural networks have achieved state of the art performance across a wide variety of machine learning tasks, often with large and computation-heavy models. Inducing sparseness as a way to reduce the memory and computation footprint of these models has seen significant research attention in recent years. In this paper, we present a new method for dynamic sparseness, whereby part of the computations are omitted dynamically, based on the input. For efficiency, we combined the idea of dynamic sparseness with block-wise matrix-vector multiplications. In contrast to static sparseness, which permanently zeroes out selected positions in weight matrices, our method preserves the full network capabilities by potentially accessing any trained weights. Yet, matrix vector multiplications are accelerated by omitting a pre-defined fraction of weight blocks from the matrix, based on the input. Experimental results on the task of language modeling, using recurrent and quasi-recurrent models, show that the proposed method can outperform static sparseness baselines. In addition, our method can reach similar language modeling perplexities as the dense baseline, at half the computational cost at inference time.

Point cloud classification has attracted increasing attention due to the outstanding performance of elaborated networks on synthetic datasets. However, rotation invariance has been seldom investigated. In this paper, we propose a straightforward rotation-invariant network called ERINet with a novel enhanced rotation-invariant module for point cloud classification. The enhanced rotation-invariant module is composed of a representation conversion component and a feature aggregation layer. It first takes 12 well-designed rotation-invariant features as the representation of point cloud and leverages the feature aggregation layer to aggregate the features of neighbor points into a discriminative rotation-invariant representation. The enhanced rotation-invariant module is further combined with the multi-layer perceptron and the fully connected layers to form an efficient ERINet. The proposed ERINet demonstrated its advantages with a small model size and high speed. The enhanced rotation-invariant module of our ERINet is also extensible and can be easily integrated with mainstream networks to improve rotation robustness. The experimental results on rotation-augmented datasets demonstrate that our ERINet outperforms other state-of-the-art methods in rotation robustness for point cloud classification.

Scene Text Recognition (STR) task needs to consume large-amount data to develop a powerful recognizer, including visual data like images and linguistic data like texts. However, existing methods mainly leverage a one-stage training manner to train the entire framework end-to-end, which deeply relies on the well-annotated images and does not effectively use the data of the two modalities mentioned above. To solve this, in this paper, we propose a pre-trained multi-modal network (PMMN) that utilizes visual and linguistic data to pre-train the vision model and language model respectively to learn modality-specific knowledge for accurate scene text recognition. In detail, we first pre-train the proposed off-the-shelf vision model and language model to convergence. And then, we combine the pre-trained models in a unified framework for end-to-end fine-tuning and utilize the learned multi-modal information to interact with each other to generate robust features for character prediction. Extensive experiments are conducted to demonstrate the effectiveness of PMMN. The evaluation results on six benchmarks show that our proposed method exceeds most existing methods, achieving state-of-the-art performance.

This paper investigates a novel PDE-constrained optimization model with discontinuous variable exponent p ( x ) identification. Since the parameter p is always related to a better approximation of the image gradient, its computation plays a critical role in preserving the image texture. Analytically, we include results on the approximation of this parameter as well as the resolution of the encountered PDE in a well posed framework. In addition, to resolve the PDE-constrained minimization problem, we proposed a modified primal-dual algorithm. Finally, numerical results are provided to compute the parameter p and also to remove high intensity of noise. The proposed algorithm simultaneously keep safe fine details and important features in medical image applications (Magnetic Resonance Images (MRI)) with numerous comparisons to show the performance of the proposed approach.

The paper investigates the Long short term memory (LSTM) networks for human action recognition in videos. In spite of significant progress in the field, recognizing actions in real-world videos is a challenging task due to the spatial and temporal variations within and across video clips. We propose a novel two-stream deep network for action recognition by applying the LSTM for learning the fusion of spatial and temporal feature streams. The LSTM type of Recurrent neural network by design possess unique capability to preserve long range context in temporal streams. The proposed method capitalizes on LSTMs memory attribute to fuse the input streams in high-dimensional space exploring the spatial and temporal correlations. The temporal stream input is defined on the LSTM learned deep features summarizing the input frame sequence. Our approach of combining the convolutional features based spatial stream and the deep features based temporal stream in LSTM network efficiently captures the long range temporal dependencies in video streams. We perform primary evaluation of the proposed approach on UCF101, HMBD51 and Kinetics400 datasets achieving competitive recognition accuracy of 93.1%, 71.3% and 74.6% respectively.

The semi-supervised learning(SSL) was proposed to deal with the situation that only a few samples were labeled, so how to make the most use of the existing samples is crucial. In general, we apply data augmentation methods, like Generative Adversarial Networks(GAN), to increase the number of data when facing unstructured data. However, things get totally different with structured data, the continuity of neural networks limits their use in category variables. In this paper, we propose a semi-supervised Embedding GAN(EmGAN) to solve that problem. We add an embedding layer before the discriminator to better characterize category features and design a new loss function to further train our model. Moreover, the structures of the generator and discriminator are modified to match structured data. With the experiments of nonparametric statistical test, EmGAN shows its advantages in processing structured samples.