Search

The latest Deep Learning (DL) models for detection and classification have achieved an unprecedented performance over classical machine learning algorithms. However, DL models are black-box methods hard to debug, interpret, and certify. DL alone cannot provide explanations that can be validated by a non technical audience such as end-users or domain experts. In contrast, symbolic AI systems that convert concepts into rules or symbols – such as knowledge graphs – are easier to explain. However, they present lower generalization and scaling capabilities. A very important challenge is to fuse DL representations with expert knowledge. One way to address this challenge, as well as the performance-explainability trade-off is by leveraging the best of both streams without obviating domain expert knowledge. In this paper, we tackle such problem by considering the symbolic knowledge is expressed in form of a domain expert knowledge graph. We present the eXplainable Neural-symbolic learning (X-NeSyL) methodology, designed to learn both symbolic and deep representations, together with an explainability metric to assess the level of alignment of machine and human expert explanations. The ultimate objective is to fuse DL representations with expert domain knowledge during the learning process so it serves as a sound basis for explainability. In particular, X-NeSyL methodology involves the concrete use of two notions of explanation, both at inference and training time respectively: (1) EXPLANet : Expert-aligned eXplainable Part-based cLAssifier NETwork Architecture, a compositional convolutional neural network that makes use of symbolic representations, and (2) SHAP-Backprop , an explainable AI-informed training procedure that corrects and guides the DL process to align with such symbolic representations in form of knowledge graphs. We showcase X-NeSyL methodology using MonuMAI dataset for monument facade image classification, and demonstrate that with our approach, it is possible to improve explainability at the same time as performance. • EXplainable Neural-symbolic Learning methodology fuses deep learning and symbolic representations. • EXPLANet's compositional part-based object detection and classification outperforms regular classification. • SHAP-Backprop aligns model output with expert knowledge in a knowledge graph. • SHAP Graph Edit Distance quantifies the alignment between a knowledge graph and neural representations. • X-NeSyL shows it is possible to improve over both explainability and performance. [ABSTRACT FROM AUTHOR]

Land use and land cover (LULC) mapping are of paramount importance to monitor and understand the structure and dynamics of the Earth system. One of the most promising ways to create accurate global LULC maps is by building good quality state-of-the-art machine learning models. Building such models requires large and global datasets of annotated time series of satellite images, which are not available yet. This paper presents TimeSpec4LULC 10.5281/zenodo.5913554; , a smart open-source global dataset of multispectral time series for 29 LULC classes ready to train machine learning models. TimeSpec4LULC was built based on the seven spectral bands of the MODIS sensors at 500 m resolution, from 2000 to 2021, and was annotated using spatial–temporal agreement across the 15 global LULC products available in Google Earth Engine (GEE). The 22-year monthly time series of the seven bands were created globally by (1) applying different spatial–temporal quality assessment filters on MODIS Terra and Aqua satellites; (2) aggregating their original 8 d temporal granularity into monthly composites; (3) merging Terra + Aqua data into a combined time series; and (4) extracting, at the pixel level, 6 076 531 time series of size 262 for the seven bands along with a set of metadata: geographic coordinates, country and departmental divisions, spatial–temporal consistency across LULC products, temporal data availability, and the global human modification index. A balanced subset of the original dataset was also provided by selecting 1000 evenly distributed samples from each class such that they are representative of the entire globe. To assess the annotation quality of the dataset, a sample of pixels, evenly distributed around the world from each LULC class, was selected and validated by experts using very high resolution images from both Google Earth and Bing Maps imagery. This smartly, pre-processed, and annotated dataset is targeted towards scientific users interested in developing various machine learning models, including deep learning networks, to perform global LULC mapping. [ABSTRACT FROM AUTHOR]

Image segmentation is an important issue in many industrial processes, with high potential to enhance the manufacturing process derived from raw material imaging. For example, metal phases contained in microstructures yield information on the physical properties of the steel. Existing prior literature has been devoted to develop specific computer vision techniques able to tackle a single problem involving a particular type of metallographic image. However, the field lacks a comprehensive tutorial on the different types of techniques, methodologies, their generalizations and the algorithms that can be applied in each scenario. This paper aims to fill this gap. First, the typologies of computer vision techniques to perform the segmentation of metallographic images are reviewed and categorized in a taxonomy. Second, the potential utilization of pixel similarity is discussed by introducing novel deep learning-based ensemble techniques that exploit this information. Third, a thorough comparison of the reviewed techniques is carried out in two openly available real-world datasets, one of them being a newly published dataset directly provided by ArcelorMittal, which opens up the discussion on the strengths and weaknesses of each technique and the appropriate application framework for each one. Finally, the open challenges in the topic are discussed, aiming to provide guidance in future research to cover the existing gaps. [Display omitted] • We create a metallography dataset from additive manufacturing of steels (MetalDAM). • We provide an updated taxonomy of segmentation methods. • We propose a new DL-based ensemble specialized in the semantic segmentation task. • We compare state-of-the-art models and the new ensembles with UHCS and MetalDAM. • We present a thorough analysis of the current difficulties and challenges. [ABSTRACT FROM AUTHOR]

Land Use and Land Cover (LULCs) mapping and change detection are of paramount importance to understand the distribution and effectively monitor the dynamics of the Earth's system. An unexplored way to create global LULC maps is by building good quality LULC-models based on state-of-the-art deep learning networks. Building such models requires large global good quality time series LULC datasets, which are not available yet. This paper presents TimeSpec4LULC (Khaldi et al., 2021), a smart open-source global dataset of multi-Spectral Time series for 29 LULC classes. TimeSpec4LULC was built based on the 7 spectral bands of MODIS sensor at 500 m resolution from 2002 to 2021, and was annotated using a spatial agreement across the 15 global LULC products available in Google Earth Engine. The 19-year monthly time series of the seven bands were created globally by: (1) applying different spatio-temporal quality assessment filters on MODIS Terra and Aqua satellites, (2) aggregating their original 8-day temporal granularity into monthly composites, (3) merging their data into a Terra+Aqua combined time series, and (4) extracting, at the pixel level, 11.85 million time series for the 7 bands along with a set of metadata about geographic coordinates, country and departmental divisions, spatio-temporal consistency across LULC products, temporal data availability, and the global human modification index. To assess the annotation quality of the dataset, a sample of 100 pixels, evenly distributed around the world, from each LULC class, was selected and validated by experts using very high resolution images from both Google Earth and Bing Maps imagery. This smartly, pre-processed, and annotated dataset is targeted towards scientific users interested in developing and evaluating various machine learning models, including deep learning networks, to perform global LULC mapping and change detection. [ABSTRACT FROM AUTHOR]

Decision making models are constrained by taking the expert evaluations with pre-defined numerical or linguistic terms. We claim that the use of sentiment analysis will allow decision making models to consider expert evaluations in natural language. Accordingly, we propose the Sentiment Analysis based Multi-person Multi-criteria Decision Making (SA-MpMcDM) methodology for smarter decision aid, which builds the expert evaluations from their natural language reviews, and even from their numerical ratings if they are available. The SA-MpMcDM methodology incorporates an end-to-end multi-task deep learning model for aspect based sentiment analysis, named DOC-ABSADeepL model, able to identify the aspect categories mentioned in an expert review, and to distill their opinions and criteria. The individual evaluations are aggregated via the procedure named criteria weighting through the attention of the experts. We evaluate the methodology in a case study of restaurant choice using TripAdvisor reviews, hence we build, manually annotate, and release the TripR-2020 dataset of restaurant reviews. We analyze the SA-MpMcDM methodology in different scenarios using and not using natural language and numerical evaluations. The analysis shows that the combination of both sources of information results in a higher quality preference vector. • Decision making models are limited by pre-defined numerical and linguistic terms. • We propose a methodology to deal with natural language and numerical assessments. • We design a deep learning model for distilling opinions from written assessments. • We present and release a dataset, which can be used for evaluating MpMcDM models. [ABSTRACT FROM AUTHOR]

Much has been said about the fusion of bio-inspired optimization algorithms and Deep Learning models for several purposes: from the discovery of network topologies and hyperparametric configurations with improved performance for a given task, to the optimization of the model's parameters as a replacement for gradient-based solvers. Indeed, the literature is rich in proposals showcasing the application of assorted nature-inspired approaches for these tasks. In this work we comprehensively review and critically examine contributions made so far based on three axes, each addressing a fundamental question in this research avenue: (a) optimization and taxonomy (Why?), including a historical perspective, definitions of optimization problems in Deep Learning, and a taxonomy associated with an in-depth analysis of the literature, (b) critical methodological analysis (How?), which together with two case studies, allows us to address learned lessons and recommendations for good practices following the analysis of the literature, and (c) challenges and new directions of research (What can be done, and what for?). In summary, three axes – optimization and taxonomy, critical analysis, and challenges – which outline a complete vision of a merger of two technologies drawing up an exciting future for this area of fusion research. • We thoroughly examine the fusion between Deep Learning and bioinspired optimization. • Definitions and a taxonomy of Deep Learning optimization problems are provided. • We perform a critical methodological analysis of contributions made so far. • Learned lessons and recommendations are drawn from our analysis and two study cases. • Challenges and research directions are given in this fusion of technologies. [ABSTRACT FROM AUTHOR]

Anomaly detection aims to identify observations that differ significantly from the majority of the data. Time series, which are data with a temporal component, is often used for anomaly detection. Identifying anomalies is not perfect and may produce many false positives, which labels standard data as anomalous. In this context, false positive mitigation is the task of reducing the number of false positives tagged by the anomaly detector, and thus both problems are closely linked. Moreover, current techniques for false positive mitigation are ad-hoc solutions for specific data sets. In this paper, we propose a novel two-stage methodology for Multivariate Anomaly Detection for Time Series and False Positive Mitigation, namely F A D F P M methodology, which creates the fusion of two learning models. The first stage is a multivariate anomaly detection stage. The second stage consists of training a new classifier on the false and true positives from the anomaly detector, which refines the observations labeled as anomalous by the anomaly detector to obtain more accurate and higher-quality results. Experiments using two benchmark data sets, as well as a real-world case study have shown the performance and validity of the proposal. • Proposed methodology decreases impact from FPs in anomaly detection. • A thorough comparison with latest SOA methods is performed. • We also provide a series of hints for applying the methodology. • High sensitivity methods are more benefited from the proposal. • A real-world case of study provided by ArcelorMIttal is analyzed. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) is currently being utilized in a wide range of sophisticated applications, but the outcomes of many AI models are challenging to comprehend and trust due to their black-box nature. Usually, it is essential to understand the reasoning behind an AI model's decision-making. Thus, the need for eXplainable AI (XAI) methods for improving trust in AI models has arisen. XAI has become a popular research subject within the AI field in recent years. Existing survey papers have tackled the concepts of XAI, its general terms, and post-hoc explainability methods but there have not been any reviews that have looked at the assessment methods, available tools, XAI datasets, and other related aspects. Therefore, in this comprehensive study, we provide readers with an overview of the current research and trends in this rapidly emerging area with a case study example. The study starts by explaining the background of XAI, common definitions, and summarizing recently proposed techniques in XAI for supervised machine learning. The review divides XAI techniques into four axes using a hierarchical categorization system: (i) data explainability, (ii) model explainability, (iii) post-hoc explainability, and (iv) assessment of explanations. We also introduce available evaluation metrics as well as open-source packages and datasets with future research directions. Then, the significance of explainability in terms of legal demands, user viewpoints, and application orientation is outlined, termed as XAI concerns. This paper advocates for tailoring explanation content to specific user types. An examination of XAI techniques and evaluation was conducted by looking at 410 critical articles, published between January 2016 and October 2022, in reputed journals and using a wide range of research databases as a source of information. The article is aimed at XAI researchers who are interested in making their AI models more trustworthy, as well as towards researchers from other disciplines who are looking for effective XAI methods to complete tasks with confidence while communicating meaning from data. • A novel four-axis framework to examine a model for robustness and explainability. • Formulation of research questions at each axis and its corresponding taxonomy. • Discussion of different explainability assessment methods. • A novel methodological workflow for determining the model and explainability criteria. • Revisited discussion on challenges and future directions of XAI and Trustworthy AI. [ABSTRACT FROM AUTHOR]

Crowd behaviour analysis is an emerging research area. Due to its novelty, a proper taxonomy to organise its different sub-tasks is still missing. This paper proposes a taxonomic organisation of existing works following a pipeline, where sub-problems in last stages benefit from the results in previous ones. Models that employ Deep Learning to solve crowd anomaly detection, one of the proposed stages, are reviewed in depth, and the few works that address emotional aspects of crowds are outlined. The importance of bringing emotional aspects into the study of crowd behaviour is remarked, together with the necessity of producing real-world, challenging datasets in order to improve the current solutions. Opportunities for fusing these models into already functioning video analytics systems are proposed. • Proposal of hierarchical taxonomy for crowd behaviour analysis subtasks. • Review and numeric comparison of Deep Learning models for crowd anomaly detection. • Discussion of current limitations in datasets and importance of going beyond. • Discussion of the importance of using emotional aspects in crowd behaviour analysis. • Proposals of fusion crowd analysis models into existing video analytics solutions. [ABSTRACT FROM AUTHOR]

• It provides an overview of the most popular ensemble methods. • It analyzes several fusion schemes using MNIST as guiding thread. • It introduces MNIST-NET10, a complex heterogeneous fusion architecture. • MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Ensemble methods have been widely used for improving the results of the best single classification model. A large body of works have achieved better performance mainly by applying one specific ensemble method. However, very few works have explored complex fusion schemes using heterogeneous ensembles with new aggregation strategies. This paper is three-fold: 1) It provides an overview of the most popular ensemble methods, 2) analyzes several fusion schemes using MNIST as guiding thread and 3) introduces MNIST-NET10, a complex heterogeneous fusion architecture based on a degree of certainty aggregation approach; it combines two heterogeneous schemes from the perspective of data, model and fusion strategy. MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Our analysis shows that such complex heterogeneous fusion architectures based on the degree of certainty can be considered as a way of taking benefit from diversity. [ABSTRACT FROM AUTHOR]

• We review concepts related to the explainability of AI methods (XAI). • We comprehensive analyze the XAI literature organized in two taxonomies. • We identify future research directions of the XAI field. • We discuss potential implications of XAI and privacy in data fusion contexts. • We identify Responsible AI as a concept promoting XAI and other AI principles in practical settings. In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence , namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability. [ABSTRACT FROM AUTHOR]

Highlights • This paper proposes a novel binocular image approach that makes the detection model focus on the area of interest. • We built a low cost symmetric dual camera system to compute the disparity map and exploit that information to improve the selection of candidate regions in the input frames. • The proposed approach reduces the number of false positives in the test videos by 49.47%. Abstract Object detection models have known important improvements in the recent years. The state-of-the art detectors are end-to-end Convolutional Neural Network based models that reach good mean average precisions, around 73%, on benchmarks of high quality images. However, these models still produce a large number of false positives in low quality videos such as, surveillance videos. This paper proposes a novel image fusion approach to make the detection model focus on the area of interest where the action is more likely to happen in the scene. We propose building a low cost symmetric dual camera system to compute the disparity map and exploit this information to improve the selection of candidate regions from the input frames. From our results, the proposed approach not only reduces the number of false positives but also improves the overall performance of the detection model which make it appropriate for object detection in surveillance videos. [ABSTRACT FROM AUTHOR]

There exist a high demand to provide explainability to artificial intelligence systems, where decision making models are included. This paper focuses on crowd decision making using natural language evaluations from social media with the aim to provide explainability. We present the Explainable Crowd Decision Making based on Subgroup Discovery and Attention Mechanisms (ECDM-SDAM) methodology as an a posteriori explainable process that captures the wisdom of crowds that is naturally provided in social media opinions. It extracts the opinions from social media texts using a deep learning based sentiment analysis approach called Attention based Sentiment Analysis Method. The methodology includes a backward process that provides explanations to justify its sense-making procedure by applying mainly the attention mechanism on texts and subgroup discovery on opinions. We evaluate the methodology in the real case study of the TripR-2020Large dataset for restaurant choice. The results show that the ECDM-SDAM methodology provides easy understandable explanations that elucidates the key reasons that support the output of the decision process. • Explainability in decision making is essential to increase its use and understanding. • Attention mechanisms and subgroup discovery can generate explainable decision making. • We propose a methodology that offers explanations of its internal decision mechanism. • The proposed methodology captures the wisdom of crowds from social media. • Natural language with sentiment analysis and deep learning enriches expert evaluation. [ABSTRACT FROM AUTHOR]

Highlights • Study the performance of promising CNNs in the classification of coral texture images. • Analyze different types of transfer learning. • Analyze data augmentation on the performance of the coral classification model. • Experimental results outperform state-of-the-art methods needing human intervention. • Generalize the best approach to other coral texture datasets. Graphical abstract Abstract The recognition of coral species based on underwater texture images poses a significant difficulty for machine learning algorithms, due to the three following challenges embedded in the nature of this data: (1) datasets do not include information about the global structure of the coral; (2) several species of coral have very similar characteristics; and (3) defining the spatial borders between classes is difficult as many corals tend to appear together in groups. For this reasons, the classification of coral species has always required an aid from a domain expert. The objective of this paper is to develop an accurate classification model for coral texture images. Current datasets contain a large number of imbalanced classes, while the images are subject to inter-class variation. We have focused on the current small datasets and analyzed (1) several Convolutional Neural Network (CNN) architectures, (2) data augmentation techniques and (3) transfer learning approaches. We have achieved the state-of-the art accuracies using different variations of ResNet on the two small coral texture datasets, EILAT and RSMAS. [ABSTRACT FROM AUTHOR]

Many of the existing machine learning algorithms, both supervised and unsupervised, depend on the quality of the input characteristics to generate a good model. The amount of these variables is also important, since performance tends to decline as the input dimensionality increases, hence the interest in using feature fusion techniques, able to produce feature sets that are more compact and higher level. A plethora of procedures to fuse original variables for producing new ones has been developed in the past decades. The most basic ones use linear combinations of the original variables, such as PCA ( Principal Component Analysis ) and LDA ( Linear Discriminant Analysis ), while others find manifold embeddings of lower dimensionality based on non-linear combinations, such as Isomap or LLE ( Linear Locally Embedding ) techniques. More recently, autoencoders (AEs) have emerged as an alternative to manifold learning for conducting nonlinear feature fusion. Dozens of AE models have been proposed lately, each with its own specific traits. Although many of them can be used to generate reduced feature sets through the fusion of the original ones, there also AEs designed with other applications in mind. The goal of this paper is to provide the reader with a broad view of what an AE is, how they are used for feature fusion, a taxonomy gathering a broad range of models, and how they relate to other classical techniques. In addition, a set of didactic guidelines on how to choose the proper AE for a given task is supplied, together with a discussion of the software tools available. Finally, two case studies illustrate the usage of AEs with datasets of handwritten digits and breast cancer. [ABSTRACT FROM AUTHOR]

Accurate tree cover mapping is of paramount importance in many fields, from biodiversity conservation to carbon stock estimation, ecohydrology, erosion control, or Earth system modelling. Despite this importance, there is still uncertainty about global forest cover, particularly in drylands. Recently, the Food and Agriculture Organization of the United Nations (FAO) conducted a costly global assessment of dryland forest cover through the visual interpretation of orthoimages using the Collect Earth software, involving hundreds of operators from around the world. Our study proposes a new automatic method for estimating tree cover using artificial intelligence and free orthoimages. Our results show that our tree cover classification model, based on convolutional neural networks (CNN), is 23% more accurate than the manual visual interpretation used by FAO, reaching up to 79% overall accuracy. The smallest differences between the two methods occurred in the driest regions, but disagreement increased with the percentage of tree cover. The application of CNNs could be used to improve and reduce the cost of tree cover maps from the local to the global scale, with broad implications for research and management. [ABSTRACT FROM AUTHOR]

Invasion of the Polygraphus proximus Blandford bark beetle causes catastrophic damage to forests with firs (Abies sibirica Ledeb) in Russia, especially in Central Siberia. Determining tree damage stage based on the shape, texture and colour of tree crown in unmanned aerial vehicle (UAV) images could help to assess forest health in a faster and cheaper way. However, this task is challenging since (i) fir trees at different damage stages coexist and overlap in the canopy, (ii) the distribution of fir trees in nature is irregular and hence distinguishing between different crowns is hard, even for the human eye. Motivated by the latest advances in computer vision and machine learning, this work proposes a two-stage solution: In a first stage, we built a detection strategy that finds the regions of the input UAV image that are more likely to contain a crown, in the second stage, we developed a new convolutional neural network (CNN) architecture that predicts the fir tree damage stage in each candidate region. Our experiments show that the proposed approach shows satisfactory results on UAV Red, Green, Blue (RGB) images of forest areas in the state nature reserve "Stolby" (Krasnoyarsk, Russia). [ABSTRACT FROM AUTHOR]