Search

Background Discussing patient preferences for cardio‐pulmonary resuscitation (CPR) is routine in hospital admission for older people. The way the conversation is conducted plays an important role for patient comprehension and the ethics of decision making. Objective The objective was to examine how CPR is explained in geriatric rehabilitation hospital admission interviews, focussing on circumstances in which physicians explain CPR and the content of these explanations. Method We recorded forty‐three physician‐patient admission interviews taking place in a hospital in French‐speaking Switzerland, during which CPR was discussed. Data were analysed in French with thematic and conversation analysis, and the extracts used for publication were translated into English. Results Mean patient age was 83.7 years; 53.5% were admitted for rehabilitation after surgery or traumatism. CPR was explained in 53.8% of the conversations. Most explanations were brief and concerned the technical procedures, mentioning only rarely potential outcome. With one exception, medical indication and prognosis of CPR did not feature in these explanations. Explanations occurred either before the patient's answer (as part of the question about CPR preferences) or after the patient's answer, generated by patients' indecision, misunderstanding and by the need to clarify answers. Discussion and conclusions The scarcity and simplicity of CPR explanations highlight a reluctance to have in‐depth discussions and reflect the assumption that CPR does not need explaining. Providing patients with accurate information about the outcomes and risks of CPR is incremental for reaching informed decisions and patient‐centred care. Patient contribution Patients were involved in the data collection stage of the study.

In this paper we argue that transparency of machine learning algorithms, just as explanation, can be defined at different levels of abstraction. We criticize recent attempts to identify the explanation of black box algorithms with making their decisions (post-hoc) interpretable, focusing our discussion on counterfactual explanations. These approaches to explanation simplify the real nature of the black boxes and risk misleading the public about the normative features of a model. We propose a new form of algorithmic transparency, that consists in explaining algorithms as an intentional product, that serves a particular goal, or multiple goals (Daniel Dennet’s design stance) in a given domain of applicability, and that provides a measure of the extent to which such a goal is achieved, and evidence about the way that measure has been reached. We call such idea of algorithmic transparency “design publicity.” We argue that design publicity can be more easily linked with the justification of the use and of the design of the algorithm, and of each individual decision following from it. In comparison to post-hoc explanations of individual algorithmic decisions, design publicity meets a different demand (the demand for impersonal justification) of the explainee. Finally, we argue that when models that pursue justifiable goals (which may include fairness as avoidance of bias towards specific groups) to a justifiable degree are used consistently, the resulting decisions are all justified even if some of them are (unavoidably) based on incorrect predictions. For this argument, we rely on John Rawls’s idea of procedural justice applied to algorithms conceived as institutions.

Although Analytical Sociology is not often used in the mainstream Sociology, its history is however, traceable to the classical works of scholars such as Emile Durkheim, Max Weber, Alexis de Tocqueville as well as contemporary sociological thinkers like Talcott Parsons and Robert Merton, among others. This paper provides a contemporary argument for the application of mechanistic explanation in the overall understanding of Analytical Sociology using relevant and practical examples. In the course of this, attention has been paid to the concept of explanation and its various types in a sociological discourse. This paper therefore argues that social reality can significantly be understood only when explanations are systematic and detailed in content and context. The conclusion is that analytical sociology has the capacity to explain the actions of social actors within the social environment beyond some social doubts, even though, not all situations, can be sufficiently explained with the strategy.

Advances in artificial intelligence, sensors and big data man-agement have far-reaching societal impacts. As these sys-tems augment our everyday lives, it becomes increasingly important for people to understand them and remain in con-trol. We investigate how HCI researchers can help to develop accountable systems by performing a literature analysis of 289 core papers on explanations and explainable systems, as well as 12,412 citing papers. Using topic modeling, co-oc-currence and network analysis, we mapped the research space from diverse domains, such as algorithmic accounta-bility, interpretable machine learning, context-awareness, cognitive psychology, and software learnability. We reveal fading and burgeoning trends in explainable systems, and identify domains that are closely connected or mostly iso-lated. The time is ripe for the HCI community to ensure that the powerful new autonomous systems have intelligible in-terfaces built-in. From our results, we propose several impli-cations and directions for future research towards this goal.

Recent research claims that information cues and system attributes of algorithmic decision-making processes affect decision subjects’ fairness perceptions. However, little is still known about how these factors interact. This paper presents a user study (N = 267) investigating the individual and combined effects of explanations, human oversight, and contestability on informational and procedural fairness perceptions for high- and low-stakes decisions in a loan approval scenario. We find that explanations and contestability contribute to informational and procedural fairness perceptions, respectively, but we find no evidence for an effect of human oversight. Our results further show that both informational and procedural fairness perceptions contribute positively to overall fairness perceptions but we do not find an interaction effect between them. A qualitative analysis exposes tensions between information overload and understanding, human involvement and timely decision-making, and accounting for personal circumstances while maintaining procedural consistency. Our results have important design implications for algorithmic decision-making processes that meet decision subjects’ standards of justice.

The paper documents beliefs that manifest themselves through newspaper items and elaborates on their potential to enhance a sustainable learning environment in a school science lesson. “Learning environment” is depicted from different angles and includes virtual and real learning environments, school environments and classroom environments. Descriptive and item analyses were conducted on sixty-eight newspaper items that were identified. The nature of problems and prescriptions/solutions was categorised for each item and the paper further provides elaboration on the types of problems and recommended solutions. The results show that the “believed” structure contents in their newspaper items to catch the attention of the “believer”. Lessons on the power of belief must be learnt by school science teachers if they are to succeed in creating a sustainable learning environment with improved performance in school science.

Querying inconsistent knowledge bases is a problem that has attracted a great deal of interest over the last decades. While several semantics of query answering have been proposed, and their complexity is rather well-understood, little attention has been paid to the problem of explaining query answers. Explainability has recently become a prominent problem in different areas of AI. In particular, explaining query answers allows users to understand not only what is entailed by an inconsistent knowledge base, but also why. In this paper, we address the problem of explaining query answers for existential rules under three popular inconsistency-tolerant semantics, namely, the ABox repair, the intersection of repairs, and the intersection of closed repairs semantics. We provide a thorough complexity analysis for a wide range of existential rule languages and for different complexity measures.

Summary Stable operation of an electric power system requires strict operational limits for the grid frequency. Fluctuations and external impacts can cause large frequency deviations and increased control efforts. Although these complex interdependencies can be modeled using machine learning algorithms, the black box character of many models limits insights and applicability. In this article, we introduce an explainable machine learning model that accurately predicts frequency stability indicators for three European synchronous areas. Using Shapley additive explanations, we identify key features and risk factors for frequency stability. We show how load and generation ramps determine frequency gradients, and we identify three classes of generation technologies with converse impacts. Control efforts vary strongly depending on the grid and time of day and are driven by ramps as well as electricity prices. Notably, renewable power generation is central only in the British grid, while forecasting errors play a major role in the Nordic grid., Graphical abstract, Highlights • Power grid frequency stability is analyzed via explainable artificial intelligence • Effect of generation ramps differ between Continental Europe, Britain, Nordic grids • Control efforts are driven by electricity prices and load ramps • Renewable generation and forecasting errors dominate Britain and Nordic grids, The bigger picture The transition to a sustainable energy system is challenging for the operation and stability of electric power systems as power generation becomes increasingly uncertain, grid loads increase, and their dynamical properties fundamentally change. At the same time, operational data are available at an unprecedented level of detail, enabling new methods of monitoring and control. To fully harness these data, advanced methods from machine learning must be used. In this paper, we present explainable artificial intelligence (XAI) as a tool to quantify, predict, and explain essential aspects of power system operation and stability in three major European synchronous areas. We focus on the power grid frequency, which measures the balance of generation and load and thus provides the central observable for control and balancing. Combining XAI with domain knowledge, we identify the main drivers and stability risks, while our model and open dataset may enable further XAI research on power systems., The transition to a sustainable energy system is one of the greatest challenges of our time. With operational data becoming available at an unprecedented level of detail, advanced methods from machine learning must be used to fully harness these data.In our article, we present explainable artificial intelligence (XAI) as a tool to quantify, predict, and explain essential aspects of power system operation and stability in three major European areas, giving indications of how to stably operate future power systems.

In our current society, it is very common to have access to a huge amount of information and scenarios in which it is possible to choose among a great number of alternatives. Therefore, systems that automate or assist with the decision-making process are becoming crucial. Those systems should allow users to represent their preferences accurately, even in scenarios where the user has many forms of comparing the available alternatives. In this paper, we propose an argument-based framework for multi-criteria decision-making based on conditional rules, that allows users to encode their preferences among the comparison criteria in terms of requirement and tolerance values. From these rules, arguments that are involved in the comparison process are generated. In order to comply with users’ demands for explanations of the given answers, our proposed framework can provide qualitative explanations about the argumentation process, and also an explanatory graph that shows how all the alternatives were ranked and the connections with the user's rules that were used. We also show the results of an empirical analysis of the performance of a developed prototype. Fil: Buron Brarda, Martín Ezequiel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Bahía Blanca. Instituto de Ciencias e Ingeniería de la Computación. Universidad Nacional del Sur. Departamento de Ciencias e Ingeniería de la Computación. Instituto de Ciencias e Ingeniería de la Computación; Argentina Fil: Tamargo, Luciano Héctor. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Bahía Blanca. Instituto de Ciencias e Ingeniería de la Computación. Universidad Nacional del Sur. Departamento de Ciencias e Ingeniería de la Computación. Instituto de Ciencias e Ingeniería de la Computación; Argentina Fil: García, Alejandro Jorge. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Bahía Blanca. Instituto de Ciencias e Ingeniería de la Computación. Universidad Nacional del Sur. Departamento de Ciencias e Ingeniería de la Computación. Instituto de Ciencias e Ingeniería de la Computación; Argentina

This paper provides an extensive overview of the use of knowledge graphs in the context of Explainable Machine Learning. As of late, explainable AI has become a very active field of research by addressing the limitations of the latest machine learning solutions that often provide highly accurate, but hardly scrutable and interpretable decisions. An increasing interest has also been shown in the integration of Knowledge Representation techniques in Machine Learning applications, mostly motivated by the complementary strengths and weaknesses that could lead to a new generation of hybrid intelligent systems. Following this idea, we hypothesise that knowledge graphs, which naturally provide domain background knowledge in a machine-readable format, could be integrated in Explainable Machine Learning approaches to help them provide more meaningful, insightful and trustworthy explanations. Using a systematic literature review methodology we designed an analytical framework to explore the current landscape of Explainable Machine Learning. We focus particularly on the integration with structured knowledge at large scale, and use our framework to analyse a variety of Machine Learning domains, identifying the main characteristics of such knowledge-based, explainable systems from different perspectives. We then summarise the strengths of such hybrid systems, such as improved understandability, reactivity, and accuracy, as well as their limitations, e.g. in handling noise or extracting knowledge efficiently. We conclude by discussing a list of open challenges left for future research.

Contains fulltext : 234229.pdf (Publisher’s version ) (Open Access) Research in the social sciences has shown that expectations are an important factor in explanations as used between humans: rather than explaining the cause of an event per se, the explainer will often address another event that did not occur but that the explainee might have expected. For AI-powered systems, this finding suggests that explanation-generating systems may need to identify such end user expectations. In general, this is a challenging task, not the least because users often keep them implicit; there is thus a need to investigate the importance of such an ability. In this paper, we report an empirical study with 181 participants who were shown outputs from a text classifier system along with an explanation of why the system chose a particular class for each text. Explanations were both factual, explaining why the system produced a certain output or counterfactual, explaining why the system produced one output instead of another. Our main hypothesis was explanations should align with end user expectations; that is, a factual explanation should be given when the system's output is in line with end user expectations, and a counterfactual explanation when it is not. We find that factual explanations are indeed appropriate when expectations and output match. When they do not, neither factual nor counterfactual explanations appear appropriate, although we do find indications that our counterfactual explanations contained at least some necessary elements. Overall, this suggests that it is important for systems that create explanations of AI systems to infer what outputs the end user expected so that factual explanations can be generated at the appropriate moments. At the same time, this information is, by itself, not sufficient to also create appropriate explanations when the output and user expectations do not match. This is somewhat surprising given investigations of explanations in the social sciences, and will need more scrutiny in future studies. 27 p.

The main purpose of this paper is to present Slovenian and Finnish prospective chemistry teachers? perceptions of their future profession, especially with regard to their understanding of the role of the triple nature of chemical concepts (macro, submicro and symbolic) and their representations in chemistry learning. A total of 19 prospective teachers (10 Slovenian, 9 Finnish) at master?s level in chemical education participated in the research. The prospective teachers? opinions were gathered using an electronic questionnaire comprising six open-ended questions. The study revealed many parallels between Slovenian and Finnish prospective chemistry teachers? perceptions of their future profession and their understanding of the role of the triple nature of chemical concepts, especially particle representations, in chemistry learning. The majority of the prospective teachers from both countries believe that personal characteristics are the most important attribute of a successful chemistry teacher. Thus, they highly value teachers? enthusiasm for teaching and the use of contemporary teaching approaches in chemistry. The prospective teachers displayed an adequate understanding of the role of the triple nature of chemical concepts (i.e., particle representations) in the planning and implementation of a specific chemistry lesson.

ResumenLas explicaciones en los sistemas de recomendación se han utilizado para mejorar componentes de la experiencia del usuario (EU). Se ha probado que las explicaciones son útiles para mejorar la percepción del usuario respecto a la satisfacción, y para persuadir al usuario de adoptar el sistema de recomendación. Este artículo se enfoca en la evaluación de la confianza del usuario de acuerdo con el sistema y sus relaciones con la comprensión y esfuerzo cognitivo que percibe, mediante el uso de explicaciones en un tipo específico de sistema de recomendación: sensible al contexto. Se realizó un experimento de usuario con un sistema de recomendación organizacional que provee recomendaciones de personas, lugares, eventos, objetos de conocimiento e infraestructura, usando tres tipos de explicación: mapas conceptuales, mapas mentales y explicaciones textuales. Los resultados muestran que, aunque los mapas conceptuales y mentales incrementan la comprensión de las recomendaciones y reducen el esfuerzo cognitivo percibido, las explicaciones textuales mejoran la confianza del usuario respecto al sistema, aun cuando requieren mayor esfuerzo cognitivo para comprenderlas.AbstractExplanations on recommender systems have been used to improve components of user experience. It has been proven that explanations are useful to enhance users’ perception on satisfaction and persuade users to adopt the recommender system. This paper focuses on evaluating user trust and the trade-offs with understandability and cognitive effort by using explanations in a specific type of recommender system: context-aware. We performed an user experiment with an organizational recommender system that gives recommendations of people, places, events, knowledge objects and infrastructure, using three different types of explanations: conceptual maps, mind maps and textual explanation. The results show that although mind and conceptual maps increases users’ understanding of recommendations and decreases cognitive effort, textual explanations improves the users’ trust on the system even when they require much cognitive effort to be understood.

Ontology-mediated query answering is an extensively studied paradigm, which aims at improving query answers with the use of a logical theory. As a form of logical entailment, ontology-mediated query answering is fully interpretable, which makes it possible to derive explanations for query answers. Surprisingly, however, explaining answers for ontology-mediated queries has received little attention for ontology languages based on existential rules. In this paper, we close this gap, and study the problem of explaining query answers in terms of minimal subsets of database facts. We provide a thorough complexity analysis for several decision problems associated with minimal explanations under existential rules.

Part 5: MAKE Explainable AI; International audience; The transparent AI initiative has ignited several academic and industrial endeavors and produced some impressive technologies and results thus far. Many state-of-the-art methods provide explanations that mostly target the needs of AI engineers. However, there is very little work on providing explanations that support the needs of business owners, software developers, and consumers who all play significant roles in the service development and use cycle. By considering the overall context in which an explanation is presented, including the role played by the human-in-the-loop, we can hope to craft effective explanations. In this paper, we introduce the notion of the “cognitive value” of an explanation and describe its role in providing effective explanations within a given context. Specifically, we consider the scenario of a business owner seeking to improve sales of their product, and compare explanations provided by some existing interpretable machine learning algorithms (random forests, scalable Bayesian Rules, causal models) in terms of the cognitive value they offer to the business owner. We hope that our work will foster future research in the field of transparent AI to incorporate the cognitive value of explanations in crafting and evaluating explanations.

Graph navigational languages allow to specify pairs of nodes in a graph subject to the existence of paths satisfying a certain regular expression. Under this evaluation semantics, connectivity information in terms of intermediate nodes/edges that contributed to the answer is lost. The goal of this paper is to introduce the GeL language, which provides query evaluation semantics able to also capture connectivity information and output graphs. We show how this is useful to produce query explanations. We present efficient algorithms to produce explanations and discuss their complexity. GeL machineries are made available into existing SPARQL processors thanks to a translation from GeL queries into CONSTRUCT SPARQL queries. We outline examples of explanations obtained with a tool implementing our framework and report on an experimental evaluation that investigates the overhead of producing explanations.

Myriad research establishes that Easterners think more holistically (i.e., attend to the “big picture” of how the world fits together and rely more on intuition) than Westerners do. Yet little is known about how Easterners integrate, structure, and make sense the information gleaned through holistic thinking. This paper proposes that, compared to Westerners, Easterners may rely more on analogical reasoning to integrate information and gain a sense of understanding their worlds. Four studies provide evidence that Easterners rely more on analogical reasoning that Westerners do. Compared to Westerners, Easterners found it easier to generate analogical explanations for understanding an outcome, preferred analogical to non-analogical explanations, showed some evidence of being more likely to spontaneously apply analogical solutions to the Duncker ray tumor problem, and tended to use more individual analogies in cultural products like children’s stories. Easterners, however, did not show evidence of being better at highly structured analogical reasoning (e.g., the Raven’s matrices) than Westerners. These findings suggest that Easterners and Westerners may both be able to reason analogically when required, but Easterners may spontaneously rely more on analogical reasoning to gain a sense of understanding of the world.

The modelling of complex, dynamic and uncertain socioenvironmental systems requires close collaboration between research disciplines and stakeholders at all levels, for if such models are representations of aspects of reality, how can it be possible to build them without inputs from people who interact with the systems in reality? This paper reflects on findings of case study research involving stakeholders in knowledge creation through conceptual and formal model building to support upland water catchment management. The uncertainty, multiple scales and conflicting understandings of stakeholders inherent in natural resource management necessitate a strong focus on participatory processes in integrated modelling. This leads to the recognition that problems and solutions should be identified by the stakeholders themselves, emphasising the intersection and complementarity of lay and expert knowledge. Stakeholders in this context comprise varied groups such as land managers ( e. g. grouse moor managers and sheep farmers), water companies and water users, tourists, residents, policy-makers and researchers. Models are used at multiple stages to help formulate problems, create and compare conceptual understanding, and explore implications. This requires the involvement of stakeholders early in the problem formulation and research process to help answer the 'right' questions and provide 'relevant' outputs. This poses a number of interesting new challenges for the organisation of research. In contrast to traditional approaches of matching model components to expertise in the research team, modelling expertise must adapt to answer the questions and priorities that emerge from stakeholder engagement.

As an operator telecontrols real time industrial devices, his difficulties consist partly in apprehending the dynamic of the device. He has to solve problems in a context evolving not only under the influence of his own actions but also by new faults and reactions of automatic devices. In this paper we present DIAPASON, a simulation-based training system for telecontrol operators responsible for the management of the French medium voltage electrical power systems. During the training session, the trainee solves a problem without being interrupted by the system. He acts on a simulated power system without explaining his actions. On request, during the trainee's solving process, DIAPASON explains how the device runs and comments the trainee's solution. The design of DIAPASON is focussed on dynamic as all the functionalities of the system integrate it : simulation, explanation and assessment., La complexité de la conduite de dispositifs industriels temps réel provient en partie de la difficulté de l'opérateur à appréhender la dynamique du dispositif. Il doit résoudre un problème dans un contexte qui évolue sous l'effet de ses propres actions et de l'apparition d'anomalies et des réactions des automatismes. Nous présentons ici DIAPASON, un système d'entraînement à la conduite de réseaux électriques moyenne tension fondé sur la simulation où le stagiaire résout un cas sans être interrompu par le système. Il agit sur le réseau simulé sans justifier ses actions. En cours de résolution, DIAPASON lui fournit des explications sur le dispositif et des commentaires sur sa résolution. La dimension dynamique est au cœur de DIAPASON car elle a été intégrée dans toutes les fonctions du système : simulation, explication, évaluation. Elle se reflète dans la conception générale du système et dans la conception détaillée de chacun de ses modules., Joab Michelle, Auzende Odette, Moinard Claudine. La prise en compte de la dynamique dans la conception d'un système de formation professionnelle fondé sur la simulation. In: Sciences et techniques éducatives, volume 6 n°1, 1999. Simulation et formation professionnelle dans l'industrie, sous la direction de Michelle Joab et Guy Gouardères. pp. 61-93.

Managing large-scale transportation infrastructure projects is difficult due to frequent misinformation about the costs which results in large cost overruns that often threaten the overall project viability. This paper investigates the explanations for cost overruns that are given in the literature. Overall, four categories of explanations can be distinguished: technical, economic, psychological, and political. Political explanations have been seen to be the most dominant explanations for cost overruns. Agency theory is considered the most interesting for political explanations and an eclectic theory is also considered possible. Non-political explanations are diverse in character, therefore a range of different theories (including rational choice theory and prospect theory), depending on the kind of explanation is considered more appropriate than one all-embracing theory., European Journal of Transport and Infrastructure Research, Vol 10 No 1 (2010)

En este trabajo se estudia, usando una metodología cualitativa, la forma en que un equipo de estudiantes de primaria y su profesora construyen explicaciones de la visión humana usando diversas representaciones (dibujos y maqueta). Los resultados muestran que las representaciones se vuelven más abstractas, integrando ideas construidas con antelación y entidades del modelo teórico de referencia. Las ideas representadas están influenciadas por la intención de la profesora, el tipo de registro semiótico usado, la negociación social y el contexto de la actividad. El conjunto de representaciones genera un historia narrativa que permite a los alumnos explicar teóricamente el fenómeno, modelizándolo. In this paper we study, using a qualitative method, how elementary students and their teacher construct explanations of the human vision through a variety of representations (drawings and a 3D model). The results shows that the representations became increasingly abstract, integrating ideas built beforehand and entities of the theoretical model of reference. The ideas represented are influenced by the intention of the teacher, the kind of semiotic resource, the social negotiation and the context of the activity. The ensemble of representations creates a narrative that allows students to explain theoretically and modeling the phenomena studied.

The goal of predictive toxicology is the automatic construction of carcinogenecity models. Most common artificial intelligence techniques used to construct these models are inductive learning methods. In a previous work we presented an approach that uses lazy learning methods for solving the problem of predicting carcinogenecity. Lazy learning methods solve new problems based on their similarity to already solved problems. Nevertheless, a weakness of these kind of methods is that sometimes the result is not completely understandable by the user. In this paper we propose an explanation scheme for a concrete lazy learning method. This scheme is particularly interesting to justify the predictions about the carcinogenesis of chemical compounds. In addition we propose that these explanations could be used to build a partial domain knowledge. In our particular case, we use the explanations for building general knowledge about carcinogenesis. © 2008 Elsevier Ltd. All rights reserved., This work has been supported by the MCYT-FEDER Project MID-CBR (TIN2006-15140-C03-01) and also by the Generalitat de Catalunya under the Grant 2005-SGR-00093.

Network security tests should be periodically conducted to detect vulnerabilities before they are exploited. However, analysis of testing results is resource intensive with many data and requires expertise because it is an unsupervised domain. This paper presents how to automate and improve this analysis through the identification and explanation of device groups with similar vulnerabilities. Clustering is used for discovering hidden patterns and abnormal behaviors. Self-organizing maps are preferred due to their soft computing capabilities. Explanations based on anti-unification give comprehensive descriptions of clustering results to analysts. This approach is integrated in Consensus, a computer-aided system to detect network vulnerabilities. © 2009 Elsevier B.V., This work has been partially supported by the MCYT-FEDER project called MID-CBR (TIN2006-15140-C03-01 and TIN2006-15140-C03-03) and by the Generalitat de Catalunya under Grants 2005SGR-302 and 2005-SGR-00093. We would also like to thank La Salle-Universitat Ramon Llull for the support to our research group

Machine Learning (ML) is increasingly prominent in or- ganizations. While those algorithms can provide near perfect accuracy, their decision-making process remains opaque. In a context of accelerating regulation in Artificial Intelligence (AI) and deepening user awareness, explainability has become a priority notably in critical healthcare and financial environ- ments. The various frameworks developed often overlook their integration into operational applications as discovered with our industrial partner. In this paper, explainability in ML and its relevance to our industrial partner is presented. We then dis- cuss the main challenges to the integration of ex- plainability frameworks in production we have faced. Finally, we provide recommendations given those challenges.