Search

Land-Use and Land-Cover (LULC) mapping is relevant for many applications, from Earth system and climate modelling to territorial and urban planning. Global LULC products are continuously developing as remote sensing data and methods grow. However, there still exists low consistency among LULC products due to low accuracy in some regions and LULC types. Here, we introduce Sentinel2GlobalLULC, a Sentinel-2 RGB image dataset, built from the spatial-temporal consensus of up to 15 global LULC maps available in Google Earth Engine. Sentinel2GlobalLULC v2.1 contains 194877 single-class RGB image tiles organized into 29 LULC classes. Each image is a 224 × 224 pixels tile at 10 × 10 m resolution built as a cloud-free composite from Sentinel-2 images acquired between June 2015 and October 2020. Metadata includes a unique LULC annotation per image, together with level of consensus, reverse geo-referencing, global human modification index, and number of dates used in the composite. Sentinel2GlobalLULC is designed for training deep learning models aiming to build precise and robust global or regional LULC maps. Measurement(s) land cover • land use Technology Type(s) satellite imaging of a planet [ABSTRACT FROM AUTHOR]

Land use and land cover (LULC) mapping are of paramount importance to monitor and understand the structure and dynamics of the Earth system. One of the most promising ways to create accurate global LULC maps is by building good quality state-of-the-art machine learning models. Building such models requires large and global datasets of annotated time series of satellite images, which are not available yet. This paper presents TimeSpec4LULC 10.5281/zenodo.5913554; , a smart open-source global dataset of multispectral time series for 29 LULC classes ready to train machine learning models. TimeSpec4LULC was built based on the seven spectral bands of the MODIS sensors at 500 m resolution, from 2000 to 2021, and was annotated using spatial–temporal agreement across the 15 global LULC products available in Google Earth Engine (GEE). The 22-year monthly time series of the seven bands were created globally by (1) applying different spatial–temporal quality assessment filters on MODIS Terra and Aqua satellites; (2) aggregating their original 8 d temporal granularity into monthly composites; (3) merging Terra + Aqua data into a combined time series; and (4) extracting, at the pixel level, 6 076 531 time series of size 262 for the seven bands along with a set of metadata: geographic coordinates, country and departmental divisions, spatial–temporal consistency across LULC products, temporal data availability, and the global human modification index. A balanced subset of the original dataset was also provided by selecting 1000 evenly distributed samples from each class such that they are representative of the entire globe. To assess the annotation quality of the dataset, a sample of pixels, evenly distributed around the world from each LULC class, was selected and validated by experts using very high resolution images from both Google Earth and Bing Maps imagery. This smartly, pre-processed, and annotated dataset is targeted towards scientific users interested in developing various machine learning models, including deep learning networks, to perform global LULC mapping. [ABSTRACT FROM AUTHOR]

Land Use and Land Cover (LULCs) mapping and change detection are of paramount importance to understand the distribution and effectively monitor the dynamics of the Earth's system. An unexplored way to create global LULC maps is by building good quality LULC-models based on state-of-the-art deep learning networks. Building such models requires large global good quality time series LULC datasets, which are not available yet. This paper presents TimeSpec4LULC (Khaldi et al., 2021), a smart open-source global dataset of multi-Spectral Time series for 29 LULC classes. TimeSpec4LULC was built based on the 7 spectral bands of MODIS sensor at 500 m resolution from 2002 to 2021, and was annotated using a spatial agreement across the 15 global LULC products available in Google Earth Engine. The 19-year monthly time series of the seven bands were created globally by: (1) applying different spatio-temporal quality assessment filters on MODIS Terra and Aqua satellites, (2) aggregating their original 8-day temporal granularity into monthly composites, (3) merging their data into a Terra+Aqua combined time series, and (4) extracting, at the pixel level, 11.85 million time series for the 7 bands along with a set of metadata about geographic coordinates, country and departmental divisions, spatio-temporal consistency across LULC products, temporal data availability, and the global human modification index. To assess the annotation quality of the dataset, a sample of 100 pixels, evenly distributed around the world, from each LULC class, was selected and validated by experts using very high resolution images from both Google Earth and Bing Maps imagery. This smartly, pre-processed, and annotated dataset is targeted towards scientific users interested in developing and evaluating various machine learning models, including deep learning networks, to perform global LULC mapping and change detection. [ABSTRACT FROM AUTHOR]

Aspect-based sentiment analysis enables the extraction of fine-grained information, as it connects specific aspects that appear in reviews with a polarity. Although we detect that the information from these algorithms is very accurate at local level, it does not contribute to obtain an overall understanding of reviews. To fill this gap, we propose a methodology to portray opinions through the most relevant associations between aspects and polarities. Our methodology combines three off-the-shelf algorithms: (1) deep learning for extracting aspects, (2) clustering for joining together similar aspects, and (3) subgroup discovery for obtaining descriptive rules that summarize the polarity information of set of reviews. Concretely, we aim at depicting negative opinions from three cultural monuments in order to detect those features that need to be improved. Experimental results show that our approach clearly gives an overview of negative aspects, therefore it will be able to attain a better comprehension of opinions. [ABSTRACT FROM AUTHOR]

The current evolution in multidisciplinary learning analytics research poses significant challenges for the exploitation of behavior analysis by fusing data streams toward advanced decision‐making. The identification of students that are at risk of withdrawals in higher education is connected to numerous educational policies, to enhance their competencies and skills through timely interventions by academia. Predicting student performance is a vital decision‐making problem including data from various environment modules that can be fused into a homogenous vector to ascertain decision‐making. This research study exploits a temporal sequential classification problem to predict early withdrawal of students, by tapping the power of actionable smart data in the form of students' interactional activities with the online educational system, using the freely available Open University Learning Analytics data set by employing deep long short‐term memory (LSTM) model. The deployed LSTM model outperforms baseline logistic regression and artificial neural networks by 10.31% and 6.48% respectively with 97.25% learning accuracy, 92.79% precision, and 85.92% recall. [ABSTRACT FROM AUTHOR]

Fingerprint classification is one of the most common approaches to accelerate the identification in large databases of fingerprints. Fingerprints are grouped into disjoint classes, so that an input fingerprint is compared only with those belonging to the predicted class, reducing the penetration rate of the search. The classification procedure usually starts by the extraction of features from the fingerprint image, frequently based on visual characteristics. In this work, we propose an approach to fingerprint classification using convolutional neural networks, which avoid the necessity of an explicit feature extraction process by incorporating the image processing within the training of the classifier. Furthermore, such an approach is able to predict a class even for low-quality fingerprints that are rejected by commonly used algorithms, such as FingerCode. The study gives special importance to the robustness of the classification for different impressions of the same fingerprint, aiming to minimize the penetration in the database. In our experiments, convolutional neural networks yielded better accuracy and penetration rate than state-of-the-art classifiers based on explicit feature extraction. The tested networks also improved on the runtime, as a result of the joint optimization of both feature extraction and classification. [ABSTRACT FROM AUTHOR]

Accurate tree cover mapping is of paramount importance in many fields, from biodiversity conservation to carbon stock estimation, ecohydrology, erosion control, or Earth system modelling. Despite this importance, there is still uncertainty about global forest cover, particularly in drylands. Recently, the Food and Agriculture Organization of the United Nations (FAO) conducted a costly global assessment of dryland forest cover through the visual interpretation of orthoimages using the Collect Earth software, involving hundreds of operators from around the world. Our study proposes a new automatic method for estimating tree cover using artificial intelligence and free orthoimages. Our results show that our tree cover classification model, based on convolutional neural networks (CNN), is 23% more accurate than the manual visual interpretation used by FAO, reaching up to 79% overall accuracy. The smallest differences between the two methods occurred in the driest regions, but disagreement increased with the percentage of tree cover. The application of CNNs could be used to improve and reduce the cost of tree cover maps from the local to the global scale, with broad implications for research and management. [ABSTRACT FROM AUTHOR]

Invasion of the Polygraphus proximus Blandford bark beetle causes catastrophic damage to forests with firs (Abies sibirica Ledeb) in Russia, especially in Central Siberia. Determining tree damage stage based on the shape, texture and colour of tree crown in unmanned aerial vehicle (UAV) images could help to assess forest health in a faster and cheaper way. However, this task is challenging since (i) fir trees at different damage stages coexist and overlap in the canopy, (ii) the distribution of fir trees in nature is irregular and hence distinguishing between different crowns is hard, even for the human eye. Motivated by the latest advances in computer vision and machine learning, this work proposes a two-stage solution: In a first stage, we built a detection strategy that finds the regions of the input UAV image that are more likely to contain a crown, in the second stage, we developed a new convolutional neural network (CNN) architecture that predicts the fir tree damage stage in each candidate region. Our experiments show that the proposed approach shows satisfactory results on UAV Red, Green, Blue (RGB) images of forest areas in the state nature reserve "Stolby" (Krasnoyarsk, Russia). [ABSTRACT FROM AUTHOR]

There are several breast cancer datasets for building Computer Aided Diagnosis systems (CADs) using either deep learning or traditional models. However, most of these datasets impose various trade-offs on practitioners related to their availability or inner clinical value. Recently, a public dataset called BreakHis has been released to overcome these limitations. BreakHis is organized into four magnification levels, each image is labeled according to its main category (Benign/Malignant) and its subcategory (A/F/PT/TA/PC/DC/LC/MC). This organization allows practitioners to address this problem either as a binary or a multi-category classification task with either a magnification dependent or independent training approach. In this work, we define a taxonomy that categorize this problem into four different reformulations: Magnification-Specific Binary (MSB), Magnification-Independent Binary (MIB), Magnification-Specific Multi-category (MSM) and Magnification-Independent Multi-category (MIM) classifications. We provide a comprehensive survey of all related works. We identify the best reformulation from clinical and practical standpoints. Finally, we explore for the first time the MIM approach using deep learning and draw the learnt lessons. [ABSTRACT FROM AUTHOR]

Current surveillance and control systems still require human supervision and intervention. This work presents a novel automatic handgun detection system in videos appropriate for both, surveillance and control purposes. We reformulate this detection problem into the problem of minimizing false positives and solve it by i) building the key training data-set guided by the results of a deep Convolutional Neural Networks (CNN) classifier and ii) assessing the best classification model under two approaches, the sliding window approach and region proposal approach. The most promising results are obtained by Faster R-CNN based model trained on our new database. The best detector shows a high potential even in low quality youtube videos and provides satisfactory results as automatic alarm system. Among 30 scenes, it successfully activates the alarm after five successive true positives in a time interval smaller than 0.2 s, in 27 scenes. We also define a new metric, Alarm Activation Time per Interval (AATpI), to assess the performance of a detection model as an automatic detection system in videos. [ABSTRACT FROM AUTHOR]

In recent years, Deep Learning models have shown a great performance in complex optimization problems. They generally require large training datasets, which is a limitation in most practical cases. Transfer learning allows importing the first layers of a pre-trained architecture and connecting them to fully-connected layers to adapt them to a new problem. Consequently, the configuration of the these layers becomes crucial for the performance of the model. Unfortunately, the optimization of these models is usually a computationally demanding task. One strategy to optimize Deep Learning models is the pruning scheme. Pruning methods are focused on reducing the complexity of the network, assuming an expected performance penalty of the model once pruned. However, the pruning could potentially be used to improve the performance, using an optimization algorithm to identify and eventually remove unnecessary connections among neurons. This work proposes EvoPruneDeepTL, an evolutionary pruning model for Transfer Learning based Deep Neural Networks which replaces the last fully-connected layers with sparse layers optimized by a genetic algorithm. Depending on its solution encoding strategy, our proposed model can either perform optimized pruning or feature selection over the densely connected part of the neural network. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show the contribution of EvoPruneDeepTL and feature selection to the overall computational efficiency of the network as a result of the optimization process. In particular, the accuracy is improved, reducing at the same time the number of active neurons in the final layers. [ABSTRACT FROM AUTHOR]

The combination of convolutional and recurrent neural networks is a promising framework. This arrangement allows the extraction of high-quality spatio-temporal features together with their temporal dependencies. This fact is key for time series prediction problems such as forecasting, classification or anomaly detection, amongst others. In this paper, the TSFE DL library is introduced. It compiles 22 state-of-the-art methods for both time series feature extraction and prediction, employing convolutional and recurrent deep neural networks for its use in several data mining tasks. The library is built upon a set of Tensorflow+Keras and PyTorch modules under the AGPLv3 license. The performance validation of the architectures included in this proposal confirms the usefulness of this Python package. [ABSTRACT FROM AUTHOR]

The latest Deep Learning (DL) models for detection and classification have achieved an unprecedented performance over classical machine learning algorithms. However, DL models are black-box methods hard to debug, interpret, and certify. DL alone cannot provide explanations that can be validated by a non technical audience such as end-users or domain experts. In contrast, symbolic AI systems that convert concepts into rules or symbols – such as knowledge graphs – are easier to explain. However, they present lower generalization and scaling capabilities. A very important challenge is to fuse DL representations with expert knowledge. One way to address this challenge, as well as the performance-explainability trade-off is by leveraging the best of both streams without obviating domain expert knowledge. In this paper, we tackle such problem by considering the symbolic knowledge is expressed in form of a domain expert knowledge graph. We present the eXplainable Neural-symbolic learning (X-NeSyL) methodology, designed to learn both symbolic and deep representations, together with an explainability metric to assess the level of alignment of machine and human expert explanations. The ultimate objective is to fuse DL representations with expert domain knowledge during the learning process so it serves as a sound basis for explainability. In particular, X-NeSyL methodology involves the concrete use of two notions of explanation, both at inference and training time respectively: (1) EXPLANet : Expert-aligned eXplainable Part-based cLAssifier NETwork Architecture, a compositional convolutional neural network that makes use of symbolic representations, and (2) SHAP-Backprop , an explainable AI-informed training procedure that corrects and guides the DL process to align with such symbolic representations in form of knowledge graphs. We showcase X-NeSyL methodology using MonuMAI dataset for monument facade image classification, and demonstrate that with our approach, it is possible to improve explainability at the same time as performance. • EXplainable Neural-symbolic Learning methodology fuses deep learning and symbolic representations. • EXPLANet's compositional part-based object detection and classification outperforms regular classification. • SHAP-Backprop aligns model output with expert knowledge in a knowledge graph. • SHAP Graph Edit Distance quantifies the alignment between a knowledge graph and neural representations. • X-NeSyL shows it is possible to improve over both explainability and performance. [ABSTRACT FROM AUTHOR]

Image segmentation is an important issue in many industrial processes, with high potential to enhance the manufacturing process derived from raw material imaging. For example, metal phases contained in microstructures yield information on the physical properties of the steel. Existing prior literature has been devoted to develop specific computer vision techniques able to tackle a single problem involving a particular type of metallographic image. However, the field lacks a comprehensive tutorial on the different types of techniques, methodologies, their generalizations and the algorithms that can be applied in each scenario. This paper aims to fill this gap. First, the typologies of computer vision techniques to perform the segmentation of metallographic images are reviewed and categorized in a taxonomy. Second, the potential utilization of pixel similarity is discussed by introducing novel deep learning-based ensemble techniques that exploit this information. Third, a thorough comparison of the reviewed techniques is carried out in two openly available real-world datasets, one of them being a newly published dataset directly provided by ArcelorMittal, which opens up the discussion on the strengths and weaknesses of each technique and the appropriate application framework for each one. Finally, the open challenges in the topic are discussed, aiming to provide guidance in future research to cover the existing gaps. [Display omitted] • We create a metallography dataset from additive manufacturing of steels (MetalDAM). • We provide an updated taxonomy of segmentation methods. • We propose a new DL-based ensemble specialized in the semantic segmentation task. • We compare state-of-the-art models and the new ensembles with UHCS and MetalDAM. • We present a thorough analysis of the current difficulties and challenges. [ABSTRACT FROM AUTHOR]

Decision making models are constrained by taking the expert evaluations with pre-defined numerical or linguistic terms. We claim that the use of sentiment analysis will allow decision making models to consider expert evaluations in natural language. Accordingly, we propose the Sentiment Analysis based Multi-person Multi-criteria Decision Making (SA-MpMcDM) methodology for smarter decision aid, which builds the expert evaluations from their natural language reviews, and even from their numerical ratings if they are available. The SA-MpMcDM methodology incorporates an end-to-end multi-task deep learning model for aspect based sentiment analysis, named DOC-ABSADeepL model, able to identify the aspect categories mentioned in an expert review, and to distill their opinions and criteria. The individual evaluations are aggregated via the procedure named criteria weighting through the attention of the experts. We evaluate the methodology in a case study of restaurant choice using TripAdvisor reviews, hence we build, manually annotate, and release the TripR-2020 dataset of restaurant reviews. We analyze the SA-MpMcDM methodology in different scenarios using and not using natural language and numerical evaluations. The analysis shows that the combination of both sources of information results in a higher quality preference vector. • Decision making models are limited by pre-defined numerical and linguistic terms. • We propose a methodology to deal with natural language and numerical assessments. • We design a deep learning model for distilling opinions from written assessments. • We present and release a dataset, which can be used for evaluating MpMcDM models. [ABSTRACT FROM AUTHOR]

Much has been said about the fusion of bio-inspired optimization algorithms and Deep Learning models for several purposes: from the discovery of network topologies and hyperparametric configurations with improved performance for a given task, to the optimization of the model's parameters as a replacement for gradient-based solvers. Indeed, the literature is rich in proposals showcasing the application of assorted nature-inspired approaches for these tasks. In this work we comprehensively review and critically examine contributions made so far based on three axes, each addressing a fundamental question in this research avenue: (a) optimization and taxonomy (Why?), including a historical perspective, definitions of optimization problems in Deep Learning, and a taxonomy associated with an in-depth analysis of the literature, (b) critical methodological analysis (How?), which together with two case studies, allows us to address learned lessons and recommendations for good practices following the analysis of the literature, and (c) challenges and new directions of research (What can be done, and what for?). In summary, three axes – optimization and taxonomy, critical analysis, and challenges – which outline a complete vision of a merger of two technologies drawing up an exciting future for this area of fusion research. • We thoroughly examine the fusion between Deep Learning and bioinspired optimization. • Definitions and a taxonomy of Deep Learning optimization problems are provided. • We perform a critical methodological analysis of contributions made so far. • Learned lessons and recommendations are drawn from our analysis and two study cases. • Challenges and research directions are given in this fusion of technologies. [ABSTRACT FROM AUTHOR]

An important part of art history can be discovered through the visual information in monument facades. However, the analysis of this visual information, i.e, morphology and architectural elements, requires high expert knowledge. An automatic system for identifying the architectural style or detecting the architectural elements of a monument based on one image will certainly help improving our knowledge in art and history. Building such tool is challenging as some styles share architectural elements, the bad conservation state of some monuments and the noise included in the image itself. The aim of this paper is to introduce MonuMAI (Monument with Mathematics and Artificial Intelligence) framework. In particular, (i) we designed MonuMAI dataset rich with expert knowledge considering the proposed architectural styles taxonomy and key elements relationship, which allows addressing several tasks, e.g., monument style classification and architectural elements detection, (ii) we developed MonuMAI deep learning pipeline based on lightweight MonuNet architecture for monument style classification and MonuMAI Key Elements Detection (MonuMAI-KED) model, and (iii) we built citizen science based MonuMAI mobile app that uses the proposed MonuMAI deep learning pipeline trained on MonuMAI dataset for performing in real life conditions. Our experiments show that both MonuNet architecture and the detection model achieve very good results under real life conditions. [ABSTRACT FROM AUTHOR]

Crowd behaviour analysis is an emerging research area. Due to its novelty, a proper taxonomy to organise its different sub-tasks is still missing. This paper proposes a taxonomic organisation of existing works following a pipeline, where sub-problems in last stages benefit from the results in previous ones. Models that employ Deep Learning to solve crowd anomaly detection, one of the proposed stages, are reviewed in depth, and the few works that address emotional aspects of crowds are outlined. The importance of bringing emotional aspects into the study of crowd behaviour is remarked, together with the necessity of producing real-world, challenging datasets in order to improve the current solutions. Opportunities for fusing these models into already functioning video analytics systems are proposed. • Proposal of hierarchical taxonomy for crowd behaviour analysis subtasks. • Review and numeric comparison of Deep Learning models for crowd anomaly detection. • Discussion of current limitations in datasets and importance of going beyond. • Discussion of the importance of using emotional aspects in crowd behaviour analysis. • Proposals of fusion crowd analysis models into existing video analytics solutions. [ABSTRACT FROM AUTHOR]

• It provides an overview of the most popular ensemble methods. • It analyzes several fusion schemes using MNIST as guiding thread. • It introduces MNIST-NET10, a complex heterogeneous fusion architecture. • MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Ensemble methods have been widely used for improving the results of the best single classification model. A large body of works have achieved better performance mainly by applying one specific ensemble method. However, very few works have explored complex fusion schemes using heterogeneous ensembles with new aggregation strategies. This paper is three-fold: 1) It provides an overview of the most popular ensemble methods, 2) analyzes several fusion schemes using MNIST as guiding thread and 3) introduces MNIST-NET10, a complex heterogeneous fusion architecture based on a degree of certainty aggregation approach; it combines two heterogeneous schemes from the perspective of data, model and fusion strategy. MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Our analysis shows that such complex heterogeneous fusion architectures based on the degree of certainty can be considered as a way of taking benefit from diversity. [ABSTRACT FROM AUTHOR]

In many machine learning tasks, learning a good representation of the data can be the key to building a well-performant solution. This is because most learning algorithms operate with the features in order to find models for the data. For instance, classification performance can improve if the data is mapped to a space where classes are easily separated, and regression can be facilitated by finding a manifold of data in the feature space. As a general rule, features are transformed by means of statistical methods such as principal component analysis, or manifold learning techniques such as Isomap or locally linear embedding. From a plethora of representation learning methods, one of the most versatile tools is the autoencoder. In this paper we aim to demonstrate how to influence its learned representations to achieve the desired learning behavior. To this end, we present a series of learning tasks: data embedding for visualization, image denoising, semantic hashing, detection of abnormal behaviors and instance generation. We model them from the representation learning perspective, following the state of the art methodologies in each field. A solution is proposed for each task employing autoencoders as the only learning method. The theoretical developments are put into practice using a selection of datasets for the different problems and implementing each solution, followed by a discussion of the results in each case study and a brief explanation of other six learning applications. We also explore the current challenges and approaches to explainability in the context of autoencoders. All of this helps conclude that, thanks to alterations in their structure as well as their objective function, autoencoders may be the core of a possible solution to many problems which can be modeled as a transformation of the feature space. [ABSTRACT FROM AUTHOR]

• We review concepts related to the explainability of AI methods (XAI). • We comprehensive analyze the XAI literature organized in two taxonomies. • We identify future research directions of the XAI field. • We discuss potential implications of XAI and privacy in data fusion contexts. • We identify Responsible AI as a concept promoting XAI and other AI principles in practical settings. In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence , namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability. [ABSTRACT FROM AUTHOR]

The problem of Multiple Object Tracking (MOT) consists in following the trajectory of different objects in a sequence, usually a video. In recent years, with the rise of Deep Learning, the algorithms that provide a solution to this problem have benefited from the representational power of deep models. This paper provides a comprehensive survey on works that employ Deep Learning models to solve the task of MOT on single-camera videos. Four main steps in MOT algorithms are identified, and an in-depth review of how Deep Learning was employed in each one of these stages is presented. A complete experimental comparison of the presented works on the three MOTChallenge datasets is also provided, identifying a number of similarities among the top-performing methods and presenting some possible future research directions. [ABSTRACT FROM AUTHOR]

Anomaly detection aims to identify observations that differ significantly from the majority of the data. Time series, which are data with a temporal component, is often used for anomaly detection. Identifying anomalies is not perfect and may produce many false positives, which labels standard data as anomalous. In this context, false positive mitigation is the task of reducing the number of false positives tagged by the anomaly detector, and thus both problems are closely linked. Moreover, current techniques for false positive mitigation are ad-hoc solutions for specific data sets. In this paper, we propose a novel two-stage methodology for Multivariate Anomaly Detection for Time Series and False Positive Mitigation, namely F A D F P M methodology, which creates the fusion of two learning models. The first stage is a multivariate anomaly detection stage. The second stage consists of training a new classifier on the false and true positives from the anomaly detector, which refines the observations labeled as anomalous by the anomaly detector to obtain more accurate and higher-quality results. Experiments using two benchmark data sets, as well as a real-world case study have shown the performance and validity of the proposal. • Proposed methodology decreases impact from FPs in anomaly detection. • A thorough comparison with latest SOA methods is performed. • We also provide a series of hints for applying the methodology. • High sensitivity methods are more benefited from the proposal. • A real-world case of study provided by ArcelorMIttal is analyzed. [ABSTRACT FROM AUTHOR]

Highlights • This paper proposes a novel binocular image approach that makes the detection model focus on the area of interest. • We built a low cost symmetric dual camera system to compute the disparity map and exploit that information to improve the selection of candidate regions in the input frames. • The proposed approach reduces the number of false positives in the test videos by 49.47%. Abstract Object detection models have known important improvements in the recent years. The state-of-the art detectors are end-to-end Convolutional Neural Network based models that reach good mean average precisions, around 73%, on benchmarks of high quality images. However, these models still produce a large number of false positives in low quality videos such as, surveillance videos. This paper proposes a novel image fusion approach to make the detection model focus on the area of interest where the action is more likely to happen in the scene. We propose building a low cost symmetric dual camera system to compute the disparity map and exploit this information to improve the selection of candidate regions from the input frames. From our results, the proposed approach not only reduces the number of false positives but also improves the overall performance of the detection model which make it appropriate for object detection in surveillance videos. [ABSTRACT FROM AUTHOR]

Abstract A particular challenge in Natural Language Processing is the disambiguation of polysemic words. The great availability, diversity and the speed of changing of the data from on-line sources force the development of disambiguation systems with a reduced dependency on linguistic resources. We argue that the contextual neural encoding of a specific entity avoids the need of using external linguistic resources like knowledge bases. Hence, we propose a neural network architecture grounded in the use of Long Short-Term Memory Recurrent Neural Network for encoding the context of a target geographical entity, specifically Two k-Contextual Windows model for the disambiguation of the geographical entity Granada. We generate two annotated corpora of texts from social media written in English and Spanish, which we use to evaluate our proposal. The results show that our claim holds. Highlights • A challenge in Natural Language Processing is the disambiguation of polysemic words. • Named entities are also polysemic, so their disambiguation is required. • Disambiguation methods highly depend on linguistic resources. • We propose a deep recurrent neural network that does not use any linguistic resourse. • The results show that our model can disambiguate the target entity. [ABSTRACT FROM AUTHOR]

Highlights • Study the performance of promising CNNs in the classification of coral texture images. • Analyze different types of transfer learning. • Analyze data augmentation on the performance of the coral classification model. • Experimental results outperform state-of-the-art methods needing human intervention. • Generalize the best approach to other coral texture datasets. Graphical abstract Abstract The recognition of coral species based on underwater texture images poses a significant difficulty for machine learning algorithms, due to the three following challenges embedded in the nature of this data: (1) datasets do not include information about the global structure of the coral; (2) several species of coral have very similar characteristics; and (3) defining the spatial borders between classes is difficult as many corals tend to appear together in groups. For this reasons, the classification of coral species has always required an aid from a domain expert. The objective of this paper is to develop an accurate classification model for coral texture images. Current datasets contain a large number of imbalanced classes, while the images are subject to inter-class variation. We have focused on the current small datasets and analyzed (1) several Convolutional Neural Network (CNN) architectures, (2) data augmentation techniques and (3) transfer learning approaches. We have achieved the state-of-the art accuracies using different variations of ResNet on the two small coral texture datasets, EILAT and RSMAS. [ABSTRACT FROM AUTHOR]

Highlights • A labeled database for cold steel detection. • Selection of the best model for cold steel weapon detection. • A new brightness guided preprocessing procedure, called Darkening and Contrast at Learning and Test (DaCoLT). • A real time cold steel detection system for surveillance videos. Abstract The automatic detection of cold steel weapons handled by one or multiple persons in surveillance videos can help reducing crimes. However, the detection of these metallic objects in videos faces an important problem: their surface reflectance under medium to high illumination conditions blurs their shapes in the image and hence makes their detection impossible. The objective of this work is two-fold: (i) To develop an automatic cold steel weapon detection model for video surveillance using Convolutional Neural Networks(CNN) and (ii) strengthen its robustness to light conditions by proposing a brightness guided preprocessing procedure called DaCoLT (Darkening and Contrast at Learning and Test stages). The obtained detection model provides excellent results as cold steel weapon detector and as automatic alarm system in video surveillance. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) is currently being utilized in a wide range of sophisticated applications, but the outcomes of many AI models are challenging to comprehend and trust due to their black-box nature. Usually, it is essential to understand the reasoning behind an AI model's decision-making. Thus, the need for eXplainable AI (XAI) methods for improving trust in AI models has arisen. XAI has become a popular research subject within the AI field in recent years. Existing survey papers have tackled the concepts of XAI, its general terms, and post-hoc explainability methods but there have not been any reviews that have looked at the assessment methods, available tools, XAI datasets, and other related aspects. Therefore, in this comprehensive study, we provide readers with an overview of the current research and trends in this rapidly emerging area with a case study example. The study starts by explaining the background of XAI, common definitions, and summarizing recently proposed techniques in XAI for supervised machine learning. The review divides XAI techniques into four axes using a hierarchical categorization system: (i) data explainability, (ii) model explainability, (iii) post-hoc explainability, and (iv) assessment of explanations. We also introduce available evaluation metrics as well as open-source packages and datasets with future research directions. Then, the significance of explainability in terms of legal demands, user viewpoints, and application orientation is outlined, termed as XAI concerns. This paper advocates for tailoring explanation content to specific user types. An examination of XAI techniques and evaluation was conducted by looking at 410 critical articles, published between January 2016 and October 2022, in reputed journals and using a wide range of research databases as a source of information. The article is aimed at XAI researchers who are interested in making their AI models more trustworthy, as well as towards researchers from other disciplines who are looking for effective XAI methods to complete tasks with confidence while communicating meaning from data. • A novel four-axis framework to examine a model for robustness and explainability. • Formulation of research questions at each axis and its corresponding taxonomy. • Discussion of different explainability assessment methods. • A novel methodological workflow for determining the model and explainability criteria. • Revisited discussion on challenges and future directions of XAI and Trustworthy AI. [ABSTRACT FROM AUTHOR]

There exist a high demand to provide explainability to artificial intelligence systems, where decision making models are included. This paper focuses on crowd decision making using natural language evaluations from social media with the aim to provide explainability. We present the Explainable Crowd Decision Making based on Subgroup Discovery and Attention Mechanisms (ECDM-SDAM) methodology as an a posteriori explainable process that captures the wisdom of crowds that is naturally provided in social media opinions. It extracts the opinions from social media texts using a deep learning based sentiment analysis approach called Attention based Sentiment Analysis Method. The methodology includes a backward process that provides explanations to justify its sense-making procedure by applying mainly the attention mechanism on texts and subgroup discovery on opinions. We evaluate the methodology in the real case study of the TripR-2020Large dataset for restaurant choice. The results show that the ECDM-SDAM methodology provides easy understandable explanations that elucidates the key reasons that support the output of the decision process. • Explainability in decision making is essential to increase its use and understanding. • Attention mechanisms and subgroup discovery can generate explainable decision making. • We propose a methodology that offers explanations of its internal decision mechanism. • The proposed methodology captures the wisdom of crowds from social media. • Natural language with sentiment analysis and deep learning enriches expert evaluation. [ABSTRACT FROM AUTHOR]

Many of the existing machine learning algorithms, both supervised and unsupervised, depend on the quality of the input characteristics to generate a good model. The amount of these variables is also important, since performance tends to decline as the input dimensionality increases, hence the interest in using feature fusion techniques, able to produce feature sets that are more compact and higher level. A plethora of procedures to fuse original variables for producing new ones has been developed in the past decades. The most basic ones use linear combinations of the original variables, such as PCA ( Principal Component Analysis ) and LDA ( Linear Discriminant Analysis ), while others find manifold embeddings of lower dimensionality based on non-linear combinations, such as Isomap or LLE ( Linear Locally Embedding ) techniques. More recently, autoencoders (AEs) have emerged as an alternative to manifold learning for conducting nonlinear feature fusion. Dozens of AE models have been proposed lately, each with its own specific traits. Although many of them can be used to generate reduced feature sets through the fusion of the original ones, there also AEs designed with other applications in mind. The goal of this paper is to provide the reader with a broad view of what an AE is, how they are used for feature fusion, a taxonomy gathering a broad range of models, and how they relate to other classical techniques. In addition, a set of didactic guidelines on how to choose the proper AE for a given task is supplied, together with a discussion of the software tools available. Finally, two case studies illustrate the usage of AEs with datasets of handwritten digits and breast cancer. [ABSTRACT FROM AUTHOR]

Opinion summarisation is concerned with generating structured summaries of multiple opinions in order to provide insightful knowledge to end users. We present the Aspect Discovery for OPinion Summarisation (ADOPS) methodology, which is aimed at generating explainable and structured opinion summaries. ADOPS is built upon aspect-based sentiment analysis methods based on deep learning and Subgroup Discovery techniques. The resultant opinion summaries are presented as interesting rules, which summarise in explainable terms for humans the state of the opinion about the aspects of a specific entity. We annotate and release a new dataset of opinions about a single entity on the restaurant review domain for assessing the ADOPS methodology, and we call it ORCo. The results show that ADOPS is able to generate interesting rules with high values of support and confidence, which provide explainable and insightful knowledge about the state of the opinion of a certain entity. • We present a novel methodology for aspect-based opinion summarisation. • Our methodology combines deep learning and subgroup discovery methods. • We categorise the aspects of restaurant reviews and classify their opinion values. • The summaries are presented in explainable terms for humans as interesting rules. • We release a new dataset for assessing opinion summarisation models. [ABSTRACT FROM AUTHOR]

In Deep Learning, training a model properly with a high quantity and quality of data is crucial in order to achieve a good performance. In some tasks, however, the necessary data is not available at a particular moment and only becomes available over time. In which case, incremental learning is used to train the model correctly. An open problem remains, however, in the form of the stability–plasticity dilemma: how to incrementally train a model that is able to respond well to new data (plasticity) while also retaining previous knowledge (stability). In this paper, an incremental learning model inspired in Rehearsal (recall of past memories based on a subset of data) named CRIF is proposed, and two instances for the framework are employed — one using a random-based selection of representative samples (Naive Incremental Learning, NIL), the other using Crowding Distance and Best vs. Second Best metrics in conjunction for this task (RILBC). The experiments were performed on five datasets — MNIST, Fashion-MNIST, CIFAR-10, Caltech 101, and Tiny ImageNet, in two different incremental scenarios: a strictly class-incremental scenario, and a pseudo class-incremental scenario with unbalanced data. In Caltech 101, Transfer Learning was used, and in this scenario as well as in the other three datasets, the proposed method, NIL, achieved better results in most of the quality metrics than comparison algorithms such as RMSProp Inc (base line) and iCaRL (state-of-the-art proposal) and outperformed the other proposed method, RILBC. NIL also requires less time to achieve these results. • An incremental learning model inspired in Rehearsal (recall of past memories based on a subset of data) is proposed. • Experiments were performed over MNIST, Fashion-MNIST, CIFAR-10 and Caltech 101 in two different scenarios. • Several metrics were used to compare learning quality results when each new megabatch of data is used. • Friedman's non-parametric statistical test and Holm post-hoc test were used for supporting the analysis of the results. • Random-based selection of representative samples obtains the best results. [ABSTRACT FROM AUTHOR]

Corals are crucial animals as they support a large part of marine life. The automatic classification of corals species based on underwater images is important as it can help experts to track and detect threatened and vulnerable coral species. However, this classification is complicated due to the nature of coral underwater images and the fact that current underwater coral datasets are unrealistic as they contain only texture images, while the images taken by autonomous underwater vehicles show the complete coral structure. The objective of this paper is two-fold. The first is to build a dataset that is representative of the problem of classifying underwater coral images, the StructureRSMAS dataset. The second is to build a classifier capable of resolving the real problem of classifying corals, based either on texture or structure images. We have achieved this by using a two-level classifier composed of three ResNet models. The first level recognizes whether the input image is a texture or a structure image. Then, the second level identifies the coral species. To do this, we have used a known texture dataset, RSMAS, and StructureRSMAS. • Creation of a new coral structure dataset. • Study the performance of promising CNNs in the classification of coral images. • Analyse data augmentation on the performance of the coral classification models. • Evaluate image enhancement techniques on the performance of the models. • Creation of a two-level model able to classify any coral image, texture or structure. [ABSTRACT FROM AUTHOR]