Search

Mobile Edge Computing (MEC) has emerged as a promising paradigm catering to overwhelming explosions of mobile applications, by offloading the compute-intensive tasks to an MEC network for processing. The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and edge intelligence arises to provision real-time deep neural network (DNN) inference services for users. In this paper, we study a novel delay-aware DNN inference throughput maximization problem by accelerating each DNN inference through jointly exploring DNN partitioning and multi-thread parallelism. Specifically, we consider the problem under both offline and online request arrival settings: a set of DNN inference requests is given in advance, and a sequence of DNN inference requests arrives one by one without the knowledge of future arrivals, respectively. We first show that the defined problems are NP-hard. We then devise a novel constant approximation algorithm for the problem under the offline setting. We also propose an online algorithm with a provable competitive ratio for the problem under the online setting. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results demonstrate that the proposed algorithms are promising.

Today there is a rapid development taking place in phenotyping of plants using non-destructive image based machine vision techniques. Machine vision based plant phenotyping ranges from single plant trait estimation to broad assessment of crop canopy for thousands of plants in the field. Plant phenotyping systems either use single imaging method or integrative approach signifying simultaneous use of some of the imaging techniques like visible red, green and blue (RGB) imaging, thermal imaging, chlorophyll fluorescence imaging (CFIM), hyperspectral imaging, 3-dimensional (3-D) imaging or high resolution volumetric imaging. This paper provides an overview of imaging techniques and their applications in the field of plant phenotyping. This paper presents a comprehensive survey on recent machine vision methods for plant trait estimation and classification. In this paper, information about publicly available datasets is provided for uniform comparison among the state-of-the-art phenotyping methods. This paper also presents future research directions related to the use of deep learning based machine vision algorithms for structural (2-D and 3-D), physiological and temporal trait estimation, and classification studies in plants.

Projection techniques are often used to visualize high-dimensional data, allowing users to better understand the overall structure of multi-dimensional spaces on a 2D screen. Although many such methods exist, comparably little work has been done on generalizable methods of inverse-projection -- the process of mapping the projected points, or more generally, the projection space back to the original high-dimensional space. In this paper we present NNInv, a deep learning technique with the ability to approximate the inverse of any projection or mapping. NNInv learns to reconstruct high-dimensional data from any arbitrary point on a 2D projection space, giving users the ability to interact with the learned high-dimensional representation in a visual analytics system. We provide an analysis of the parameter space of NNInv, and offer guidance in selecting these parameters. We extend validation of the effectiveness of NNInv through a series of quantitative and qualitative analyses. We then demonstrate the method's utility by applying it to three visualization tasks: interactive instance interpolation, classifier agreement, and gradient visualization.

Issuing a disaster certificate, which is used to decide the contents of a victim’s support, requires accuracy and rapidity. However, in Japan at large, issuing of damage certificates has taken a long time in past earthquake disasters. Hence, the government needs a more efficient mechanism for issuing damage certificates. This study developed an estimation system of roof-damaged buildings to obtain an overview of earthquake damage based on aero-photo images using deep learning. To provide speedy estimation, this system utilized the trimming algorithm, which automatically generates roof image data using the location information of building polygons on GIS (Geographic Information System). Consequently, the proposed system can estimate, if a house is covered with a blue sheet with 97.57 % accuracy and also detect whether a house is damaged, with 93.51 % accuracy. It would therefore be worth considering the development of an image recognition model and a method of collecting aero-photo data to operate this system during a real earthquake.

Given the prevalence of depression worldwide and its major impact on society, several studies employed artificial intelligence modelling to automatically detect and assess depression. However, interpretation of these models and cues are rarely discussed in detail in the AI community, but have received increased attention lately. In this study, we aim to analyse the commonly selected features using a proposed framework of several feature selection methods and their effect on the classification results, which will provide an interpretation of the depression detection model. The developed framework aggregates and selects the most promising features for modelling depression detection from 38 feature selection algorithms of different categories. Using three real-world depression datasets, 902 behavioural cues were extracted from speech behaviour, speech prosody, eye movement and head pose. To verify the generalisability of the proposed framework, we applied the entire process to depression datasets individually and when combined. The results from the proposed framework showed that speech behaviour features (e.g. pauses) are the most distinctive features of the depression detection model. From the speech prosody modality, the strongest feature groups were F0, HNR, formants, and MFCC, while for the eye activity modality they were left-right eye movement and gaze direction, and for the head modality it was yaw head movement. Modelling depression detection using the selected features (even though there are only 9 features) outperformed using all features in all the individual and combined datasets. Our feature selection framework did not only provide an interpretation of the model, but was also able to produce a higher accuracy of depression detection with a small number of features in varied datasets. This could help to reduce the processing time needed to extract features and creating the model.

Emotion recognition based on electroencephalography (EEG) is a significant task in the brain-computer interface field. Recently, many deep learning-based emotion recognition methods are demonstrated to outperform traditional methods. However, it remains challenging to extract discriminative features for EEG emotion recognition, and most methods ignore useful information in channel and time. This paper proposes an attention-based convolutional recurrent neural network (ACRNN) to extract more discriminative features from EEG signals and improve the accuracy of emotion recognition. First, the proposed ACRNN adopts a channel-wise attention mechanism to adaptively assign the weights of different channels, and a CNN is employed to extract the spatial information of encoded EEG signals. Then, to explore the temporal information of EEG signals, extended self-attention is integrated into an RNN to recode the importance based on intrinsic similarity in EEG signals. We conducted extensive experiments on the DEAP and DREAMER databases. The experimental results demonstrate that the proposed ACRNN outperforms state-of-the-art methods.

It is widely acknowledged that the success of deep learning is built on large-scale training data and tremendous computing power. However, the data and computing power are not always available for many real-world applications. In this paper, we address the machine learning problem where it lacks training data and limits computing power. Specifically, we investigate domain adaptation which is able to transfer knowledge from one labeled source domain to an unlabeled target domain, so that we do not need much training data from the target domain. At the same time, we consider the situation that the running environment is confined, e.g., in edge computing the end device has very limited running resources. Technically, we present the Faster Domain Adaptation (FDA) protocol and further report two paradigms of FDA: early stopping and amid skipping. The former accelerates domain adaptation by multiple early exit points. The latter speeds up the adaptation by wisely skip several amid neural network blocks. Extensive experiments on standard benchmarks verify that our method is able to achieve the comparable and even better accuracy but employ much less computing resources. To the best of our knowledge, there are very few works which investigated accelerating knowledge adaptation in the community.

Hydraulic systems are a class of typical complex nonlinear systems, which have been widely used in manufacturing, metallurgy, energy, and other industries. Nowadays, the intelligent fault diagnosis problem of hydraulic systems has received increasing attention for it can increase operational safety and reliability, reduce maintenance cost, and improve productivity. However, because of the high nonlinear and strong fault concealment, the fault diagnosis of hydraulic systems is still a challenging task. Besides, the data samples collected from the hydraulic system are always in different sampling rates, and the coupling relationship between the components brings difficulties to accurate data acquisition. To solve the above issues, a deep learning model with multirate data samples is proposed in this article, which can extract features from the multirate sampling data automatically without expertise, thus it is more suitable in the industrial situation. Experiment results demonstrate that the proposed method achieves high diagnostic and fault pattern recognition accuracy even when the imbalance degree of sample data is as large as 1:100. Moreover, the proposed method can increase about 10% diagnosis accuracy when compared with some state-of-the-art methods.

Deep learning methods have been very successful at radio frequency fingerprinting tasks, predicting the identity of transmitting devices with high accuracy. We study radio frequency fingerprinting deployments at resource-constrained edge devices. We use structured pruning to jointly train and sparsify neural networks tailored to edge hardware implementations. We compress convolutional layers by a 27.2x factor while incurring a negligible prediction accuracy decrease (less than 1%). We demonstrate the efficacy of our approach over multiple edge hardware platforms, including a Samsung Gallaxy S10 phone and a Xilinx-ZCU104 FPGA. Our method yields significant inference speedups, 11.5x on the FPGA and 3x on the smartphone, as well as high efficiency: the FPGA processing time is 17x smaller than in a V100 GPU. To the best of our knowledge, we are the first to explore the possibility of compressing networks for radio frequency fingerprinting; as such, our experiments can be seen as a means of characterizing the informational capacity associated with this specific learning task.

The advanced metering infrastructure (AMI) in modern networked smart homes brings various advantages. However, smart homes are vulnerable to many cyberattacks, and the most striking one is energy theft. Researchers have developed many countermeasures, fostered by advanced machine learning (ML) techniques. Nevertheless, recent advances are not robust enough in practice, partially due to the vulnerabilities of ML algorithms. In this paper, we present a covert electricity theft strategy through mimicking normal consumption patterns. Such attack is almost impossible to be detected by existing solutions as the manipulated data have little deviation against honest usage records. To address this threat, we initially identify and define two levels of consumption deviations: home-level and interpersonal-level, respectively. Then, we propose a feature extraction method and develop a novel detection model based on deep learning. Extensive experiments show that the presented attack could evade existing mainstream detectors and the proposed countermeasure outperforms existing leading methods.

The staggering innovations and emergence of numerous deep learning (DL) applications have forced researchers to reconsider hardware architecture to accommodate fast and efficient application-specific computations. Applications, such as object detection, image recognition, speech translation, as well as music synthesis and image generation, can be performed with high accuracy at the expense of substantial computational resources using DL. Furthermore, the desire to adopt Industry 4.0 and smart technologies within the Internet of Things infrastructure has initiated several studies to enable on-chip DL capabilities for resource-constrained devices. Specialized DL processors reduce dependence on cloud servers, improve privacy, lessen latency, and mitigate bandwidth congestion. As we reach the limits of shrinking transistors, researchers are exploring various application-specific hardware architectures to meet the performance and efficiency requirements for DL tasks. Over the past few years, several software optimizations and hardware innovations have been proposed to efficiently perform these computations. In this article, we review several DL accelerators, as well as technologies with emerging devices, to highlight their architectural features in application-specific integrated circuit (IC) and field-programmable gate array (FPGA) platforms. Finally, the design considerations for DL hardware in portable applications have been discussed, along with some deductions about the future trends and potential research directions to innovate DL accelerator architectures further. By compiling this review, we expect to help aspiring researchers widen their knowledge in custom hardware architectures for DL. 2012 IEEE. This work was supported in part by the Research University Grant, Universiti Kebangsaan Malaysia, under Grant DPK-2021-001, Grant DIP-2020-004, and Grant MI-2020-002; and in part by the Qatar National Research Foundation (QNRF) under Grant NPRP12s-0227- 190164. Scopus

Many applications utilize sensors on mobile devices and apply deep learning for diverse applications. However, they have rarely enjoyed mainstream adoption due to many different individual conditions users encounter. Individual conditions are characterized by users unique behaviors and different devices they carry, which collectively make sensor inputs different. It is impractical to train countless individual conditions beforehand and we thus argue meta-learning is a great approach in solving this problem. We present MetaSense that leverages seen conditions in training data to adapt to an unseen condition (i.e., the target user). Specifically, we design a meta-learning framework that learns how to adapt to the target via iterative training sessions of adaptation. MetaSense requires very few training examples from the target (e.g., one or two) and thus requires minimal user effort. In addition, we propose a similar condition detector (SCD) that identifies when the unseen condition has similar characteristics to seen conditions and leverages this hint to further improve the accuracy. Our evaluation with 10 different datasets shows that MetaSense improves the accuracy of state-of-the-art transfer learning and meta learning methods by 15% and 11%, respectively. Furthermore, our SCD achieves additional accuracy improvement (e.g., 15% for human activity recognition).

Mobile Deep Learning (MDL) has emerged as a privacy-preserving learning paradigm for mobile devices. This paradigm offers unique features such as privacy preservation, continual learning and low-latency inference to the building of personal mobile sensing applications. However, squeezing Deep Learning to mobile devices is extremely challenging due to resource constraint. Traditional Deep Neural Networks (DNNs) are usually over-parametered, hence incurring huge resource overhead for on-device learning. In this paper, we present a novel on-device deep learning framework named MDLdroidLite that transforms traditional DNNs into resource-efficient model structures for on-device learning. To minimize resource overhead, we propose a novel Release-and-Inhibit Control (RIC) approach based on Model Predictive Control theory to efficiently grow DNNs from tiny to backbone. We also design a gate-based fast adaptation mechanism for channel-level knowledge transformation to quickly adapt new-born neurons with existing neurons, enabling safe parameter adaptation and fast convergence for on-device training. Our evaluations show that MDLdroidLite boosts on-device training on various PMS datasets with 28x to 50x less model parameters, 4x to 10x less floating number operations than the state-of-the-art model structures while keeping the same accuracy level.

We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions (e.g., Fig. 1). These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real vs. fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.

The subtleties of human perception, as measured by vision scientists through the use of psychophysics, are important clues to the internal workings of visual recognition. For instance, measured reaction time can indicate whether a visual stimulus is easy for a subject to recognize, or whether it is hard. In this paper, we consider how to incorporate psychophysical measurements of visual perception into the loss function of a deep neural network being trained for a recognition task, under the assumption that such information can enforce consistency with human behavior. As a case study to assess the viability of this approach, we look at the problem of handwritten document transcription. While good progress has been made towards automatically transcribing modern handwriting, significant challenges remain in transcribing historical documents. Here we describe a general enhancement strategy, underpinned by the new loss formulation, which can be applied to the training regime of any deep learning-based document transcription system. Through experimentation, reliable performance improvement is demonstrated for the standard IAM and RIMES datasets for three different network architectures. Further, we go on to show feasibility for our approach on a new dataset of digitized Latin manuscripts, originally produced by scribes in the Cloister of St. Gall in the the 9th century.

It is of great significance to apply deep learning for the early diagnosis of Alzheimer's disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess mild cognitive impairment (MCI) and AD. By tensorizing a three-player cooperative game-based framework, the proposed model can benefit from the structural information of the brain. By incorporating the high-order pooling scheme into the classifier, the proposed model can make full use of the second-order statistics of holistic magnetic resonance imaging (MRI). To the best of our knowledge, the proposed Tensor-train, High-order pooling and Semisupervised learning-based GAN (THS-GAN) is the first work to deal with classification on MR images for AD diagnosis. Extensive experimental results on Alzheimer's disease neuroimaging initiative (ADNI) data set are reported to demonstrate that the proposed THS-GAN achieves superior performance compared with existing methods, and to show that both tensor-train and high-order pooling can enhance classification performance. The visualization of generated samples also shows that the proposed model can generate plausible samples for semisupervised learning purpose.

Synthetic aperture radar (SAR) has been widely applied in both civilian and military fields because it provides high-resolution images of the ground target regardless of weather conditions, day or night. In SAR imaging, the separation of moving and stationary targets is of great significance as it is capable of removing the ambiguity stemming from inevitable moving targets in stationary scene imaging and suppressing clutter in moving target imaging. The newly emerged generative adversarial networks (GANs) have great performance in many other signal processing areas; however, they have not been introduced to radar imaging tasks. In this work, we propose a novel shuffle GAN with autoencoder separation method to separate the moving and stationary targets in SAR imagery. The proposed algorithm is based on the independence of well-focused stationary targets and blurred moving targets for creating adversarial constraints. Note that the algorithm operates in a totally unsupervised fashion without requiring a sample set that contains mixed and separated SAR images. Experiments are carried out on synthetic and real SAR data to validate the effectiveness of the proposed method.

The Brain Tumor is one of the most serious scenarios associated with the brain where a cluster of abnormal cells grows in an uncontrolled fashion. The field of image processing has experienced remarkable growth in the area of biomedical applications with the invention of different techniques in deep learning. Brain tumor classification and detection is a subject of prime importance where Convolutional Neural Networks (CNN) find application. But the main drawback of the existing technology is that it is complex with a huge number of parameters contributing to high execution time and high system specifications for implementation. In this paper, a novel architecture for Brain tumor classification and tumor type object detection using the RCNN technique is proposed which has been analyzed using two publicly available datasets from Figshare ( Cheng et al., 2017 ) and Kaggle (2020) . Here we aim to decrease the execution time of a conventional RCNN architecture with the use of a low complex framework and propose a system for brain tumor analysis. We first use a Two Channel CNN, a low complex architecture to classify between Glioma and healthy tumor MRI samples which was successfully done with an accuracy of 98.21 percentage. Later this same architecture is used as the feature extractor of an RCNN to detect the tumor regions of the Glioma MRI sample that has been classified from the previous stage and the tumor region is bounded using bounding boxes. Also, this method has been extended to other two types of tumors Meningioma and Pituitary tumor. The methodology was able to achieve very low execution time as compared with the other existing architectures with an average confidence level of 98.83 percentage.

Long short-term memory (LSTM) neural networks and attention mechanism have been widely used in sentiment representation learning and detection of texts. However, most of the existing deep learning models for text sentiment analysis ignore emotion's modulation effect on sentiment feature extraction, and the attention mechanisms of these deep neural network architectures are based on word- or sentence-level abstractions. Ignoring higher level abstractions may pose a negative effect on learning text sentiment features and further degrade sentiment classification performance. To address this issue, in this article, a novel model named AEC-LSTM is proposed for text sentiment detection, which aims to improve the LSTM network by integrating emotional intelligence (EI) and attention mechanism. Specifically, an emotion-enhanced LSTM, named ELSTM, is first devised by utilizing EI to improve the feature learning ability of LSTM networks, which accomplishes its emotion modulation of learning system via the proposed emotion modulator and emotion estimator. In order to better capture various structure patterns in text sequence, ELSTM is further integrated with other operations, including convolution, pooling, and concatenation. Then, topic-level attention mechanism is proposed to adaptively adjust the weight of text hidden representation. With the introduction of EI and attention mechanism, sentiment representation and classification can be more effectively achieved by utilizing sentiment semantic information hidden in text topic and context. Experiments on real-world data sets show that our approach can improve sentiment classification performance effectively and outperform state-of-the-art deep learning-based methods significantly.

In the medical field, Medical diagnosis using Computed tomography (CT) has become increasingly popular due to their non-invasive approach and quick overall turnaround time. 3D visualization for CT Colonography involves the assessment and diagnosis of a patient to find the presence of cancerous polyps in the colon, by taking a Computed Tomography scan of the patient, and evaluating the reports. This technique reduces evaluation time by allowing the doctors themselves to analyze the CT scans without the need for a radiologist to generate an initial report. This technique also avoids an invasive procedure on the patient. This paper gives the insightsof developing computer aided system for the detection of Polyps in CT Colonography images using the principles of Image Processing and the Deep Learning, specifically an ensemble of Convolutional Neural Networks using GoogleNet architecture and 3D reconstruction of the same. The accuracy achieved by the proposed system for region classification, region 1 polyp detection and region 2 polyp detection are 98.75%, 93.75% and 94.03% respectively, and their F1 Scores are 0.88, 0.82 and 0.84 respectively.

Time-series forecasting is one of the most active research topics in artificial intelligence. Applications in real-world time series should consider two factors for achieving reliable predictions: modeling dynamic dependencies among multiple variables and adjusting the model's intrinsic hyperparameters. A still open gap in that literature is that statistical and ensemble learning approaches systematically present lower predictive performance than deep learning methods. They generally disregard the data sequence aspect entangled with multivariate data represented in more than one time series. Conversely, this work presents a novel neural network architecture for time-series forecasting that combines the power of graph evolution with deep recurrent learning on distinct data distributions; we named our method Recurrent Graph Evolution Neural Network (ReGENN). The idea is to infer multiple multivariate relationships between co-occurring time-series by assuming that the temporal data depends not only on inner variables and intra-temporal relationships (i.e., observations from itself) but also on outer variables and inter-temporal relationships (i.e., observations from other-selves). An extensive set of experiments was conducted comparing ReGENN with dozens of ensemble methods and classical statistical ones, showing sound improvement of up to 64.87% over the competing algorithms. Furthermore, we present an analysis of the intermediate weights arising from ReGENN, showing that by looking at inter and intra-temporal relationships simultaneously, time-series forecasting is majorly improved if paying attention to how multiple multivariate data synchronously evolve., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

Large volumes of video data recorded by the increasing mobile devices and embedded sensors can be leveraged to answer queries of our lives, physical world and our evolving society. Especially, the rapid development of convolutional neural networks (CNNs) in the past few years offers the great advantage for multiple tasks in video analysis. However, adopting running CNNs directly on mobile devices and embedded sensors for video analytics brings heavy burden due to their limited capacity, especially for learning a large volume of data. A promising approach is to outsource the computation-intensive part of CNN to cloud. However, the reveal of data to cloud may cause privacy leakage. In addition, the cloud-assisted approach may also bring some communication efficiency challenges for large volume of data. To address both privacy and efficiency issues, we design a privacy-preserving and computationally efficient framework for mobile video analytics. Extensive experiments demonstrate our proposed system PrivacyEye can effectively protect private information while keep the accuracy of the normal tasks with less than 2% drop, and it saves up to 82.9% execution time and 78.8% energy consumption.

The extensive growth of data in the health domain has increased the utility of Deep Learning in health. Deep learning is a highly advanced successor of artificial neural networks, having powerful computing ability. Due to the availability of fast data storage and hardware parallelism its popularity grows in the last five years. This in article presents a comprehensive literature review of research deploying deep learning medical imaging and medical NLP including tasks, pipelines, and challenges. In this work, we have presented an extensive survey of deep learning architecture deployed in the fields of medical imaging and medical natural language processing. This paper helps in identifying suitable combination of Deep learning, Natural language processing and medical imaging to enhance diagnosis. We have highlighted the major challenges in deploying deep learning in medical imaging and medical natural language processing. All the results are presented in pictorial form. This survey is very helpful for novices working in the area of health informatics.

Visual media, in today’s world, has swept across most forms of day to day communication. In the paradigm of generative modelling, restricted Boltzmann machines (RBMs) are used to solve complex tasks such as feature extraction, neuroimaging, collaborative filtering, radar target cognition, etc. In this paper, we implement an unsupervised Boltzmann machine-based time-aware recommendation system (UBMTR) which detects underlying hidden features in user-movie ratings data in connection with the time at which each rating was made (temporal information). The model takes ratings and time as a dual-input and outputs binary values via the contrastive divergence algorithm which samples from a Monte Carlo Markov Chain. Arguably, there exists a correlation between the content requested and the temporal conditions, which is exactly what our model tries to exploit. There is seldom any work in the field of recommender systems built using Boltzmann machines that incorporate temporal information, which necessitates research in this domain. RBMs are adept at pattern completion to tackle missing values, and can deal with imbalanced datasets and unstructured data by encoding raw data into latent variables. Using RBM, the UBMTR outperforms many earlier made attempts made at recommendation systems done through CF and deep learning or their hybridized models.

This paper presents a new contribution to the recommendation system following E-commerce reviews. Here, it implements a dictionary-based sentiment analysis based on four main steps (i) Pre-processing (ii) Semantic word extraction (iii) Feature Extraction (iv) Classification. Initially, the tweets or reviews extracted from the database are subjected to pre-processing, which involves three processes: stop word removal, stemming, and blank space removal. In the semantic word extraction process, the semantic words from the dictionary will be extracted by matching with the extracted keywords. The next process is the feature extraction, which extracts joint holoentropy and cross holoentropy of all keywords, and further proceeds with conditional holoentropy-based feature selection process. Finally, the selected features are subjected to a classification process using Deep Belief Network (DBN), which is a well-performing deep learning algorithm. Moreover, the main contribution of the paper relies on the improvement of DBN architecture, where the number of hidden neurons and activation function is optimally tuned by a hybrid optimization algorithm termed as Grey linked Chick Update-based Chicken Swarm Optimization (GCU-CSO). At last, the comparative analysis of proposed and conventional algorithms in terms of various performance metrics proves the efficacy of the proposed E-commerce recommendation system.

Diabetic retinopathy (DR) is one of the most common retinal diseases that cause preventable blindness in diabetic patients. The timely screening and grading of retinal images minimize the possibility of vision loss. However, manual screening of retinal images, for detecting micro lesions in the early stages of DR, is time-consuming. This paper proposes an ensemble of deep convolutional neural network (CNN) models for accurate detection and grading of DR using fundus images. Each input image is divided into four patches at the first stage and passed on to pre-trained CNN models (InceptionV3, Xception) for training. The relevant features in the shallow-dense layers of CNN models are utilized as prior knowledge. The integration of shallow and dense layer features helps the model learn the significant information of DR images. At the second stage, a classifier based on artificial neural network is trained using the fused probability vectors of four patches. The results of individual CNN models are combined to generate the final decision in the third stage. This ensemble approach of multi-stage deep learning model improves the overall classification accuracy of diabetic retinopathy grading. Among the five different classification schemes presented in this paper, multistage patch-based deep CNN (MPDCNN), in which local patch-based and holistic details of fundus image are concatenated, provides the best classification accuracy. This ensemble classifier exhibits 96.2 % classification accuracy with fivefold cross-validation.

One truism of deep learning is that the automatic feature engineering (seen in the first layers of those networks) excuses data scientists from performing tedious manual feature engineering prior to running DL. For the specific case of deep learning for defect prediction, we show that that truism is false. Specifically, when we preprocess data with a novel oversampling technique called fuzzy sampling, as part of a larger pipeline called GHOST (Goal-oriented Hyper-parameter Optimization for Scalable Training), then we can do significantly better than the prior DL state of the art in 14/20 defect data sets. Our approach yields state-of-the-art results significantly faster deep learners. These results present a cogent case for the use of oversampling prior to applying deep learning on software defect prediction datasets., Comment: v3, revision 2 (minor revision); submitted to TSE

With the fast development of various positioning techniques such as Global Position System (GPS), mobile devices and remote sensing, spatio-temporal data has become increasingly available nowadays. Mining valuable knowledge from spatio-temporal data is critically important to many real world applications including human mobility understanding, smart transportation, urban planning, public safety, health care and environmental management. As the number, volume and resolution of spatio-temporal datasets increase rapidly, traditional data mining methods, especially statistics based methods for dealing with such data are becoming overwhelmed. Recently, with the advances of deep learning techniques, deep leaning models such as convolutional neural network (CNN) and recurrent neural network (RNN) have enjoyed considerable success in various machine learning tasks due to their powerful hierarchical feature learning ability in both spatial and temporal domains, and have been widely applied in various spatio-temporal data mining (STDM) tasks such as predictive learning, representation learning, anomaly detection and classification. In this paper, we provide a comprehensive survey on recent progress in applying deep learning techniques for STDM. We first categorize the types of spatio-temporal data and briefly introduce the popular deep learning models that are used in STDM. Then a framework is introduced to show a general pipeline of the utilization of deep learning models for STDM. Next we classify existing literatures based on the types of ST data, the data mining tasks, and the deep learning models, followed by the applications of deep learning for STDM in different domains including transportation, climate science, human mobility, location based social network, crime analysis, and neuroscience. Finally, we conclude the limitations of current research and point out future research directions., Comment: arXiv admin note: text overlap with arXiv:1711.04710 by other authors

With the aggressive and amazing scaling of the feature size of semiconductors, hotspot detection has become a crucial and challenging problem in the generation of optimized mask design for better printability. Machine learning techniques, especially deep learning, have attained notable success on hotspot detection tasks. However, most existing hotspot detectors suffer from suboptimal performance due to two-stage flow and less efficient representations of layout features. What is more, most works can only solve simple benchmarks with apparent hotspot patterns like ICCAD 2012 Contest benchmarks. In this paper, we firstly develop a new end-to-end hotspot detection flow where layout feature embedding and hotspot detection are jointly performed. An attention mechanism-based deep convolutional neural network is exploited as the backbone to learn embeddings for layout features and classify the hotspots simultaneously. Experimental results demonstrate that our framework achieves accuracy improvement over prior arts with fewer false alarms and faster inference speed on much more challenging benchmarks.

With the advent of ‘heavy Artificial Intelligence’ - big data, deep learning, and ubiquitous use of the internet, ethical considerations are widely dealt with in public discussions and governmental bodies. Within Computational Paralinguistics with its manifold topics and possible applications (modelling of long-term, medium-term, and short-term traits and states such as personality, emotion, or speech pathology), we have not yet seen that many contributions. In this article, we try to set the scene by (1) giving a short overview of ethics and privacy, (2) describing the field of Computational Paralinguistics, its history and exemplary use cases, as well as (de-)anonymisation and peculiarities of speech and text data, and (3) proposing rules for good practice in the field, such as choosing the right performance measure, and accounting for representativity and interpretability.

Affective computing is a field of great interest in many computer vision applications, including video surveillance, behaviour analysis, and human-robot interaction. Most of the existing literature has addressed this field by analysing different sets of face features. However, in the last decade, several studies have shown how body movements can play a key role even in emotion recognition. The majority of these experiments on the body are performed by trained actors whose aim is to simulate emotional reactions. These unnatural expressions differ from the more challenging genuine emotions, thus invalidating the obtained results. In this paper, a solution for basic non-acted emotion recognition based on 3D skeleton and Deep Neural Networks (DNNs) is provided. The proposed work introduces three majors contributions. First, unlike the current state-of-the-art in non-acted body affect recognition, where only static or global body features are considered, in this work also temporal local movements performed by subjects in each frame are examined. Second, an original set of global and time-dependent features for body movement description is provided. Third, to the best of out knowledge, this is the first attempt to use deep learning methods for non-acted body affect recognition. Due to the novelty of the topic, only the UCLIC dataset is currently considered the benchmark for comparative tests. On the latter, the proposed method outperforms all the competitors.

Indonesia is a tropical country that experiences forest fires every year. Forest fires occur due to a prolonged summer season. A major effect of forest fires that commonly occurred in Indonesia is the respiratory disorder and the visual impairment caused by the thick smoke that can experienced by the people who lives around the forest. The effects of forest fire fumes when the forest fire spread across the bigger land or area are also experienced by neighboring countries such as Malaysia, Singapore and Brunei Darussalam. Forest is a habitat for many animals, one of which is a bird. Birds are able to communicate with their colonies through sound. The bird sound during communication to the group can be in the form of calls, marriage invitations, and warnings of danger or threat of forest fires. This paper presents bird sounds classification study using one of Deep Learning (DL) algorithms i.e. Convolutional Neural Network (CNN) method. The CNN method is used to classify bird sounds in the two conditions: (1) under normal circumstances or conditions, and (2) under threated or panic condition. The bird sound data used in this study were collected from the local birds in Indonesia. The classification result of two bird sounds based on CNN method achieved up to 96.45%. This paper is a preliminary study for bio inspired early warning of forest fires based on the sound of birds.

This work presents a hybrid and hierarchical deep learning model for midterm load forecasting. The model combines exponential smoothing (ETS), advanced long short-term memory (LSTM), and ensembling. ETS extracts dynamically the main components of each individual time series and enables the model to learn their representation. Multilayer LSTM is equipped with dilated recurrent skip connections and a spatial shortcut path from lower layers to allow the model to better capture long-term seasonal relationships and ensure more efficient training. A common learning procedure for LSTM and ETS, with a penalized pinball loss, leads to simultaneous optimization of data representation and forecasting performance. In addition, ensembling at three levels ensures a powerful regularization. A simulation study performed on the monthly electricity demand time series for 35 European countries confirmed the high performance of the proposed model and its competitiveness with classical models such as ARIMA and ETS as well as state-of-the-art models based on machine learning.

The increase of cyberattacks by using the Internet of Things devices has exposed multiple vulnerabilities not only on devices but also on IoT infrastructures. Now, small devices can affect different networks and their services. This paper presents a review of the vulnerabilities on wireless technologies that gives connectivity to IoT, and the experiences using Software Defined Radio SDR for implementing wireless attacks to IoT technologies are assessed. A systematic literature review was conducted. The types of vulnerabilities and attacks that can affect the wireless technologies that stand the IoT ecosystem and SDR platforms were compared, and recent methods to overcome these attacks were identified. On the IoT reference model, the perception layer was identified as the most vulnerable. Most attacks at this level occur due to hardware limitations, physical device exposure, and heterogeneity of technologies. Future cybersecurity systems based on SDR technologies have notable advantages due to their flexibility and adaptability to new communication technologies and their potential to develop advanced tools. However, the IoT’s cybersecurity challenges are so complex that merging SDR hardware with cognitive and intelligent techniques is highly advisable. Some of these techniques could include deep learning to adapt mitigation systems to rapid technological changes.

Identifying race, which is a major physical feature in humans, is still a challenging task owing much to the lack of a concrete definition of race and the diversity of population across the globe. In this paper, we try to address the problem of race identification of four broad racial groups namely Caucasian, African, Asian and Indian. The newly developed BUPT Equalised Face dataset bearing about 1.3 M images in unrestricted environment is used to train our deep convolution network (R-Net) which achieves a state-of-the-art accuracy of 97%. To test the validity of this model it is evaluated on other datasets namely UTK and CFD. R-Net is also compared with fine-tuned VGG16 model for race estimation. Experimental results prove the robustness of this model for use in unconstrained environments. And finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is applied to get a visual explanation of the deep learning model.

Security issues have resulted in severe damage to the cloud computing environment, adversely affecting the healthy and sustainable development of cloud computing. Intrusion detection is one of the technologies for protecting the cloud computing environment from malicious attacks. However, network traffic in the cloud computing environment is characterized by large scale, high dimensionality, and high redundancy, these characteristics pose serious challenges to the development of cloud intrusion detection systems. Deep learning technology has shown considerable potential for intrusion detection. Therefore, this study aims to use deep learning to extract essential feature representations automatically and realize high detection performance efficiently. An effective stacked contractive autoencoder (SCAE) method is presented for unsupervised feature extraction. By using the SCAE method, better and robust low-dimensional features can be automatically learned from raw network traffic. A novel cloud intrusion detection system is designed on the basis of the SCAE and support vector machine (SVM) classification algorithm. The SCAE+SVM approach combines both deep and shallow learning techniques, and it fully exploits their advantages to significantly reduce the analytical overhead. Experiments show that the proposed SCAE+SVM method achieves higher detection performance compared to three other state-of-the-art methods on two well-known intrusion detection evaluation datasets, namely KDD Cup 99 and NSL-KDD.

Enabling deep learning technology on the mobile web can improve user's experience for achieving web artificial intelligence in various fields. However, heavy DNN models and limited computing resource of the mobile web are now unable to support executing computationally intensive DNNs when deploying in a cloud computing platform. With the help of promising edge computing, we propose a lightweight collaborative deep neural network for the mobile web, named LcDNN, which contributes to three aspects: (1) we design a composite collaborative DNN that reduces the model size, accelerates inference and reduces mobile energy cost by executing a lightweight binary branch on the mobile web. (2) we provide a jointly training method for LcDNN, and implement an energy-efficient inference library for executing the binary branch on the mobile web. (3) to further promote the resource utilization of the edge cloud, we develop a DRL-based online scheduling scheme to obtain an optimal allocation for LcDNN. The experimental results show that LcDNN outperforms the existing approaches for reducing the model size by about 16x to 29x. It also reduces the end-to-end latency and mobile energy cost with an acceptable accuracy, and improves the throughput and resource utilization of the edge cloud.

Imbalanced class distribution is an inherent problem in many real-world classification tasks where the minority class is the class of interest. Many conventional statistical and machine learning classification algorithms are subject to frequency bias, and learning discriminating boundaries between the minority and majority classes could be challenging. To address the class distribution imbalance in deep learning, we propose a class rebalancing strategy based on a class-balanced dynamically weighted loss function where weights are assigned based on the class frequency and predicted probability of ground-truth class. The ability of dynamic weighting scheme to self-adapt its weights depending on the prediction scores allows the model to adjust for instances with varying levels of difficulty resulting in gradient updates driven by hard minority class samples. We further show that the proposed loss function is classification calibrated. Experiments conducted on highly imbalanced data across different applications of cyber intrusion detection (CICIDS2017 data set) and medical imaging (ISIC2019 data set) show robust generalization. Theoretical results supported by superior empirical performance provide justification for the validity of the proposed dynamically weighted balanced (DWB) loss function.

Most previous EEG-based emotion recognition methods studied hand-crafted EEG features extracted from different electrodes. In this paper, we study the relation among different EEG electrodes and propose a deep learning method to automatically extract the spatial features that characterize the functional relation between EEG signals at different electrodes. Our proposed deep model is called ATtention-based LSTM with Domain Discriminator (ATDD-LSTM) that can characterize nonlinear relations among EEG signals of different electrodes. To achieve state-of-the-art emotion recognition performance, the architecture of ATDD-LSTM has two distinguishing characteristics: (1) By applying the attention mechanism to the feature vectors produced by LSTM, ATDD-LSTM automatically selects suitable EEG channels for emotion recognition, which makes the learned model concentrate on the emotion related channels in response to a given emotion; (2) To minimize the significant feature distribution shift between different sessions and/or subjects, ATDD-LSTM uses a domain discriminator to modify the data representation space and generate domain-invariant features. We evaluate the proposed ATDD-LSTM model on three public EEG emotional databases (DEAP, SEED and CMEED) for emotion recognition. The experimental results demonstrate that our ATDD-LSTM model achieves superior performance on subject-dependent (for the same subject), subject-independent (for different subjects) and cross-session (for the same subject) evaluation.

Recently, deep learning models have been successfully employed in video-based affective computing applications. One key application is automatic depression recognition from facial expressions. State-of-the-art approaches to recognize depression typically explore spatial and temporal information individually, by using convolutional neural networks (CNNs) to analyze appearance information, and then by either mapping feature variations or averaging the depression level over video frames. This approach has limitations to represent dynamic information that can help to discriminate between depression levels. In contrast, 3D CNN-based models can directly encode the spatio-temporal relationships, although these models rely on fixed-range temporal information and single receptive field. This approach limits the ability to capture facial expression variations with diverse ranges, and the exploitation of diverse facial areas. In this paper, a novel 3D CNN architecture the Multiscale Spatiotemporal Network (MSN) is introduced to effectively represent facial information related to depressive behaviours. The basic structure of the model is composed of parallel convolutional layers with different temporal depths and sizes of receptive field, which allows the MSN to explore a wide range of spatio-temporal variations in facial expressions. Experimental results on two benchmark datasets show that our MSN is effective, outperforming state-of-the-art methods in automatic depression recognition.

Detection of Alzheimer's Disease (AD) from neuroimaging data such as MRI through machine learning have been a subject of intense research in recent years. Recent success of deep learning in computer vision have progressed such research further. However, common limitations with such algorithms are reliance on a large number of training images, and requirement of careful optimization of the architecture of deep networks. In this paper, we attempt solving these issues with transfer learning, where state-of-the-art architectures such as VGG and Inception are initialized with pre-trained weights from large benchmark datasets consisting of natural images, and the fully-connected layer is re-trained with only a small number of MRI images. We employ image entropy to select the most informative slices for training. Through experimentation on the OASIS MRI dataset, we show that with training size almost 10 times smaller than the state-of-the-art, we reach comparable or even better performance than current deep-learning based methods., Comment: Presented at IEEE BIBM 2017

Stress analysis and assessment of affective states of mind using ECG as a physiological signal is a burning research topic in biomedical signal processing. However, existing literature provides only binary assessment of stress, while multiple levels of assessment may be more beneficial for healthcare applications. Furthermore, in present research, ECG signal for stress analysis is examined independently in spatial domain or in transform domains but the advantage of fusing these domains has not been fully utilized. To get the maximum advantage of fusing diferent domains, we introduce a dataset with multiple stress levels and then classify these levels using a novel deep learning approach by converting ECG signal into signal images based on R-R peaks without any feature extraction. Moreover, We made signal images multimodal and multidomain by converting them into time-frequency and frequency domain using Gabor wavelet transform (GWT) and Discrete Fourier Transform (DFT) respectively. Convolutional Neural networks (CNNs) are used to extract features from different modalities and then decision level fusion is performed for improving the classification accuracy. The experimental results on an in-house dataset collected with 15 users show that with proposed fusion framework and using ECG signal to image conversion, we reach an average accuracy of 85.45%.

Detection of Alzheimer's Disease (AD) from neuroimaging data such as MRI through machine learning has been a subject of intense research in recent years. Recent success of deep learning in computer vision has progressed such research further. However, common limitations with such algorithms are reliance on a large number of training images, and requirement of careful optimization of the architecture of deep networks. In this paper, we attempt solving these issues with transfer learning, where the state-of-the-art VGG architecture is initialized with pre-trained weights from large benchmark datasets consisting of natural images. The network is then fine-tuned with layer-wise tuning, where only a pre-defined group of layers are trained on MRI images. To shrink the training data size, we employ image entropy to select the most informative slices. Through experimentation on the ADNI dataset, we show that with training size of 10 to 20 times smaller than the other contemporary methods, we reach state-of-the-art performance in AD vs. NC, AD vs. MCI, and MCI vs. NC classification problems, with a 4% and a 7% increase in accuracy over the state-of-the-art for AD vs. MCI and MCI vs. NC, respectively. We also provide detailed analysis of the effect of the intelligent training data selection method, changing the training size, and changing the number of layers to be fine-tuned. Finally, we provide Class Activation Maps (CAM) that demonstrate how the proposed model focuses on discriminative image regions that are neuropathologically relevant, and can help the healthcare practitioner in interpreting the model's decision making process.

Deep latent generative models have attracted increasing attention due to the capacity of combining the strengths of deep learning and probabilistic models in an elegant way. The data representations learned with the models are often continuous and dense. However in many applications, sparse representations are expected, such as learning sparse high dimensional embedding of data in an unsupervised setting, and learning multi-labels from thousands of candidate tags in a supervised setting. In some scenarios, there could be further restriction on degree of sparsity: the number of non-zero features of a representation cannot be larger than a pre-defined threshold $L_{0}$ . In this paper we propose a sparse deep latent generative model SDLGM to explicitly model degree of sparsity and thus enable to learn the sparse structure of the data with the quantified sparsity constraint. The resulting sparsity of a representation is not fixed, but fits to the observation itself under the pre-defined restriction. In particular, we introduce to each observation $i$ an auxiliary random variable $L_{i}$ , which models the sparsity of its representation. The sparse representations are then generated with a two-step sampling process via two Gumbel-Softmax distributions. For inference and learning, we develop an amortized variational method based on MC gradient estimator. The resulting sparse representations are differentiable with backpropagation. The experimental evaluation on multiple datasets for unsupervised and supervised learning problems shows the benefits of the proposed method.

Deep neural networks (DNNs) achieve impressive results for complicated tasks like object detection on images and speech recognition. Motivated by this practical success, there is now a strong interest in showing good theoretical properties of DNNs. To describe for which tasks DNNs perform well and when they fail, it is a key challenge to understand their performance. The aim of this paper is to contribute to the current statistical theory of DNNs. We apply DNNs on high dimensional data and we show that the least squares regression estimates using DNNs are able to achieve dimensionality reduction in case that the regression function has locally low dimensionality. Consequently, the rate of convergence of the estimate does not depend on its input dimension $d$, but on its local dimension $d^*$ and the DNNs are able to circumvent the curse of dimensionality in case that $d^*$ is much smaller than $d$. In our simulation study we provide numerical experiments to support our theoretical result and we compare our estimate with other conventional nonparametric regression estimates. The performance of our estimates is also validated in experiments with real data.

Nowadays, “aging” becomes a challenging problem for face recognition systems. The challenge is particularly due to age-related biological changes, which can lead to significant variations in facial characteristics between two images taken at different ages of the same person. As the face is the most heavily part affected by aging, there is a growing need to extracting robust face features for age-invariant face recognition, particularly, in the presence of large age differences of the same person face images. The aim of this paper is to examine effectiveness of deep-learning based methods as features extraction tool for age-invariant face recognition. In this study, we evaluate five popular pre-trained deep-convolutional neural network (CNN) models that are AlexNet, GoogleNet, Inception V3, ResNet50 and SqueezeNet, on a widely used face-aging database, namely FG-NET, using K-nearest neighbors (K-NN), discriminant analysis, and support vector machines (SVM) classifiers. Further, a statistical analysis test is performed to confirm the statistical significance of the obtained results. Experimental results on this database show the promise of using Convolutional Neural Networks(CNN) for face recognition across age progression. Also the AlexNet model appears to be most promising for age-invariant face recognition, as highest mean accuracy rate is always achieved with feature extraction using the AlexNet model. These results are more significant, according to a 95% confidence level hypothesis test.

The detection of Brain cancer is an essential process, which is based on the clinician’s knowledge and experience. An automatic tumor classification model is important to handle radiologists to detect the brain tumors. However, the precision of present model should be enhanced for appropriate treatments. Numerous computer-aided diagnosis (CAD) models are offered in the literary works of medical imaging to help radiologists concerning their patients. This paper proposes an optimization-driven technique, namely Whale Harris Hawks optimization (WHHO) for brain tumor detection using MR images. Here, segmentation is performed using cellular automata and rough set theory. In addition, the features are extracted from the segments, which include tumor size, Local Optical Oriented Pattern (LOOP), Mean, Variance, and Kurtosis. In addition, the brain tumor detection is carried out using deep convolutional neural network (DeepCNN), wherein the training is performed using proposed WHHO. The proposed WHHO is designed by integrating Whale optimization algorithm (WOA) and Harris hawks optimization (HHO) algorithm. The proposed WHHO-based DeepCNN outperformed other methods with maximal accuracy of 0.816, maximal specificity of 0.791, and maximal sensitivity of 0.974, respectively.

In today’s digital era, the recognition of handwritten documents is highly demanding due to the widespread applications. Bangla is one of the most used languages in the world, which consists of 50 basic alphabets. Many compound characters exist in Bangla, which is the combination of two or more basic characters. The recognition of Bangla handwritten characters is a challenging task due to their various sizes, sheer, diversity, lots of turns, similarities between alphabets, and different writing patterns. This paper presents a deep CNN (Convolutional Neural Network) model using the SE-ResNeXt. The squeeze and excitation (SE) blocks are added with existing ResNeXt to address the Bangla handwritten compound character recognition. Usually, CNN extracted the spatial features in the lower-layers and complex features in the upper layers. The SE blocks are added to improve the performance of usual deep CNN by automatically fusing the channel-wise spatial information and inter-channel dependencies through squeezing and excitation within local receptive fields, respectively. To validate the performance of the proposed model, we have utilized Mendeley BanglaLekha-Isolated 2 dataset. The experimental results demonstrate that the proposed model exhibits an average accuracy of 99.82% in recognizing Bangla handwritten compound characters. In addition, the proposed model outperforms the state-of-the-art models by demonstrating higher results.

The ubiquitous deep learning (DL) in remote sensing (RS) motivates the most challenging problem of crop classification. To perpetrate such an exigent task, an attempt is made to prepare a novel dataset, the CaneSat dataset, in two formats: RGB color space and geo-tiff images, covering the region of four talukas in Karnataka, India. This research aims to build a model for sugarcane classification using two-dimensional convolutional neural network (CNN or ConvNet) applying RS time series data. Further, the study intents to evaluate competency of four state-of-the-art deep CNNs namely AlexNet, GoogLeNet, ResNet50 and DenseNet201 using fine tuning and deep CNNs as feature extractors to classify sugarcane and non-sugarcane areas from Sentinel-2 data. The results of the research are expressive on CaneSat dataset. It shows that the CNN model performs significantly good producing 88.46% accuracy, whereas all deep networks exhibit more than 73.00% overall accuracy. When used as feature extractors, ResNet50 and DenseNet201 outperform all other models with precision of 85.65% and 87.70%, respectively. Noticeably, the results indicate that 2D CNN model and features extracted using CNNs with SVM classifier are efficient methods for sugarcane classification from Sentinel-2 time series data in peninsular zone of India.