Search

[Abstract]: Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them.

[Abstract]: Human-in-the-Loop Machine Learning (HITL-ML) is a set of techniques that attempt to actively involve experts into the learning loop of machine learning (ML) models. One of these techniques is Machine Teaching (MT) which tries to apply techniques that come from the world of didactics within machine learning (ML), such as sorting the dataset according to its difficulty and presenting the cases to the model in incremental levels of complexity. In this work we propose a new twist to MT: since its foundation is to bring didactic techniques to ML, why not use this technique as a didactic method itself? In this case we propose the creation of an ML model for the identification of diphthongs and hiatuses in the Spanish language. The first step is to develop a deep learning model to identify diphthongs and hiatuses using Curriculum Learning (CL) and a sorted dataset that identifies simple and complex cases. The accuracy of this model identifies the upper limit of efficiency that we can obtain by training the model. The next step is to reset the weights of the model but retain its architecture and offer the model to the students for its training. The idea is that students use MT techniques to make the model learn again, but the ultimate goal is that students learn by teaching in an informal and gamified learning environment. The results show how a HITL strategy can make a model learn iteratively to identify diphthongs and hiatuses and a workflow is proposed to include this technique in the classroom.

[Abstract]: Any machine learning (ML) model is highly dependent on the data it uses for learning, and this is even more important in the case of deep learning models. The problem is a data bottleneck, i.e. the difficulty in obtaining an adequate number of cases and quality data. Another issue is improving the learning process, which can be done by actively introducing experts into the learning loop, in what is known as human-in-the-loop (HITL) ML. We describe an ML model based on a neural network in which HITL techniques were used to resolve the data bottleneck problem for the treatment of pancreatic cancer. We first augmented the dataset using synthetic cases created by a generative adversarial network. We then launched an active learning (AL) process involving human experts as oracles to label both new cases and cases by the network found to be suspect. This AL process was carried out simultaneously with an interactive ML process in which feedback was obtained from humans in order to develop better synthetic cases for each iteration of training. We discuss the challenges involved in including humans in the learning process, especially in relation to human–computer interaction, which is acquiring great importance in building ML models and can condition the success of a HITL approach. This paper also discusses the methodological approach adopted to address these challenges.

[Abstract] The increasing adoption of artificial intelligent systems at sensitive domains where humans are particularly, such as medicine, has provided the context to deeply explore ways of making machine learning models (ML) understandable for their final users. The success of such systems require the trust of their users, and thus there is a need to design and provide methods to understand the decisions made by such systems. We start from a public Pancreatic Cancer dataset and experiment with different ML models on a diagnosis scenario with the goal to decide whether a patient should be prescribed with a chemotherapy treatment. To validate the diagnosis results we explore different explainability approaches: Decision Tree, Random Forest, and model agnostic ad-hoc models, and compare them against a standard Pancreatic Cancer treatment set of rules. The increasing adoption of artificial intelligent systems at sensitive domains where humans are particularly, such as medicine, has provided the context to deeply explore ways of making machine learning models (ML) understandable for their final users. The success of such systems require the trust of their users, and thus there is a need to design and provide methods to understand the decisions made by such systems. We start from a public Pancreatic Cancer dataset and experiment with different ML models. To validate the diagnostic results we explore different explainability approaches: Decision Tree based approach, Random Forest based approach, and different model agnostic ad-hoc approaches, and we compare them against a standard Pancreatic Cancer treatment set of rules

[Abstract] In this paper we aim to bring some order to the myriad of tools that have emerged in the field of Artificial Intelligence (AI), focusing on the field of Machine Learning (ML). For this purpose, we suggest a classification of the tools in which the categories are organized following the development lifecycle of an ML system and we make a review of the existing tools within each section of the classification. We believe this will help to better understand the ecosystem of tools currently available and will also allow us to identify niches in which to develop new tools to aid in the development of AI and ML systems. After reviewing the state-of-the-art of the tools, we have identified three trends in them: the incorporation of humans into the loop of the machine learning process, the movement from ad-hoc and experimental approaches to a more engineering perspective and the ability to make it easier to develop intelligent systems for people without an educational background in the area, in order to move the focus from the technical environment to the domain-specific problem.

[Resumen]: El Aprendizaje Máquina es usado por gran parte de la comunidad científica como una caja negra: datos de entrada, ajuste de hiperparámetros, entrenamiento del modelo y obtención de resultados. Los modelos que se desarrollen bajo este escenario corren el riesgo de no escalar bien, volverse estáticos o ser difíciles de evaluar. El Aprendizaje Máquina Interactivo son un conjunto de técnicas que intentan introducir activamente a los humanos en el bucle de aprendizaje de los modelos, funciona especialmente bien cuando el dominio es complejo, no hay suficientes datos o son muy costosos de anotar. El Aprendizaje Máquina Interactivo se divide dependiendo del papel que ocupa el humano dentro del ciclo de aprendizaje del modelo. En nuestro proyecto el Aprendizaje Activo es la aproximación desarrollada que consiste en que el modelo consulta al humano (oráculo) los casos que generan mayor incertidumbre en sus predicciones. En este proyecto aplicamos las técnicas de Aprendizaje Máquina Interactivo a un problema real con datos de pacientes oncológicos con cáncer de páncreas, donde los datos y el tiempo de etiquetado de los médicos son limitados. Para aplicar la solución fue necesario utilizar técnicas de aumento de datos generando casos sintéticos de cáncer de páncreas mediante una Red Generativa Antagónica. Los resultados muestran como la incorporación del humano al bucle de entrenamiento provoca que el modelo converja de manera más rápida y eficiente con un menor coste humano que las aproximaciones clásicas., [Abstract]: Machine Learning is used by most of the scientific community as a black box: input data, hyperparameters optimization, training the model and obtaining results. Models developed under this scenario run the risk of not scaling well, becoming static or difficult to evaluate. Interactive Machine Learning are a set of techniques that attempt to actively introduce humans into the learning loop of models, works especially well when the domain is complex, not enough data or is too costly to annotate. Interactive Machine Learning is divided depending on the role of the human within the model learning loop. In our project, Active Learning is the approach developed which consists of the model consulting the human (oracle) for the cases that generate greater uncertainty in its predictions. In this project we applied the Interactive Machine Learning techniques to a real problem with data from oncological patients with pancreatic cancer, where data and labeling time of physicians are limited. To apply the solution it was necessary to use data augmentation techniques generating synthetic cases of pancreatic cancer using an Generative Adversarial Network. The results show how the incorporation of the human into the training loop causes the model to converge faster and more efficiently with a lower human cost than classical approaches.

[Abstract] At present, the great majority of Artificial Intelligence (AI) systems require the participation of humans in their development, tuning, and maintenance. Particularly, Machine Learning (ML) systems could greatly benefit from their expertise or knowledge. Thus, there is an increasing interest around how humans interact with those systems to obtain the best performance for both the AI system and the humans involved. Several approaches have been studied and proposed in the literature that can be gathered under the umbrella term of Human-in-the-Loop Machine Learning. The application of those techniques to the health informatics environment could provide a great value on prognosis and diagnosis tasks contributing to develop a better health service for Cancer related diseases.

[Abstract] Scholars and practitioners are defining new types of interactions between humans and machine learning algorithms that we can group under the umbrella term of Human-in-the-Loop Machine Learning (HITL-ML). This paper is focused on implementing two approaches to this topic—Iterative Machine Teaching (iMT) and Active Learning (AL)—and analyzing how to integrate them in the learning loop. iMT is a variation of Machine Teaching in which a machine acts as a teacher that tries to transfer knowledge to a machine learning model. The focus of the problem in iMT is how to obtain the optimal training set given a machine learning algorithm and a target model. The idea is to learn a target concept with a minimal number of iterations with the smallest dataset. Active Learning, in contrast, is a specialized type of supervised learning in which humans are incorporated in the loop to act as oracles that analyze unlabeled data. AL allows us to achieve greater accuracy with less data and less training. Our proposal to incorporate iMT and AL into the machine learning loop is to use iMT as a technique to obtain the “Minimum Viable Data (MVD)” for training a learning model, that is, a dataset that allows us to increase speed and reduce complexity in the learning process by allowing to build early prototypes. Next, we will use AL to refine this first prototype by adding new data in an iterative and incremental way. We carried out several experiments to test the feasibility of our proposed approach. They show that the algorithms trained with the teachers converge faster than those that have been trained in a conventional way. Also, AL helps the model to avoid getting stuck and to keep improving after the first few iterations. The two approaches investigated in this paper can be considered complementary, as they correspond to different stages in the learning process.

[Abstract] The usability of Domain-Specific Languages (DSLs) has been attracting considerable interest from researchers lately. In particular, our literature review found many usability studies that make use of subjective and empirical methods. However, we noted a lack of heuristic methods in the literature. In comparison, there exist several usability studies of Application Programming Interfaces (APIs) that have used heuristics with success, so we argue that this approach would be also useful for DSLs. Therefore, this paper proposes a set of usability heuristics for DSLs and illustrates the approach with a case study. We show how our heuristics helped us identify many actual usability problems, even for a simple DSL.

[Abstract]: Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.

[Abstract]: CONTEXT. The currently existing literature about Application Program Interface (API) usability is heterogeneous in terms of goals, scope, and audience; and its connection to accepted definitions of usability is rarely made explicit. The use of metrics to measure API usability is focused only on measurable characteristics excluding those usability aspects that are related to the subjectivity of human opinions. OBJECTIVE. Our objective is to build a comprehensive set of heuristics and guidelines for API usability that is a structured synthesis of the existing literature on API usability but which also covers other aspects that have been neglected so far. This set is explicitly connected with a usability model, something that allows us to check if we are addressing actual usability problems. METHOD. Our approach is to follow a systematic approach based on a comprehensive model of usability and context-of-use. From this comprehensive model we derived the set of heuristics and guidelines that are used to carry out a heuristic evaluation with usability experts and a subjective analysis with users. The influence of the context of use, something that is normally ignored, is explicitly analyzed. RESULTS. Our heuristics and guidelines were integrated into a usability study of a sleep medicine API. In this study, we were able to identify several usability issues of the proposed API that are not explicitly addressed in the existing literature. The context of use helped us to identify those categories that were more relevant to consider in order to improve API usability. CONCLUSION. The literature on API usability is very technically-minded and tends to neglect the subjective component of usability. We contribute to a more global and comprehensive view of the usability of APIs that is not contradictory but complementary with metrics. Our criteria ease the always necessary usability evaluation with human evaluators and users.

[Resumen] Un importante obstáculo para la implantación del Diseño Centrado en el Usuario en el mundo real es que no existe una definición del concepto de usabilidad que sea precisa y comúnnmente aceptada y aplicada en la práctica. La literatura tiende a definir la usabilidad en términos demasiado escuetos y ambiguos, y a describir su aplicación en términos informales. Idénticas críticas pueden hacerse al concepto del contexto de uso, al cual la usabilidad es siempre relativa. Como consecuencia de esto, las técnicas ad hoc predominan en las metodologías de usabilidad. Esta tesis propone taxonomías detalladas para los conceptos de usabilidad y contexto de uso. Las taxonomías están organizadas jerárquicamente y contienen descripciones precisas de sus atributos y subatributos. Para ilustrar la utilidad práctica de las taxonomías, se describe cómo fueron integradas en el ciclo de vida de desarrollo de dos productos reales en el campo de los Sistemas Inteligentes. Concretamente, un dispositivo de Adaptación Inteligente de la Velocidad y un generador automático de interfaces de usuario. En un punto específico de cada proyecto se realizó un estudio de usabilidad, usando las taxonomías para estructurar y guiar actividades de usabilidad como el análisis de los requisitos de usabilidad, la evaluación heurística y el análisis subjetivo., [Abstract] A major obstacle to the implantation of User-Centered Design in the real world is the fact that no precise de nition of the concept of usability exists that is widely accepted and applied in practice. Generally speaking, the literature tends to de ne usability in overly brief and ambiguous terms and to describe its application in informal terms. The same criticisms can be leveled at the concept of context of use, to which usability is always relative. As a consequence of these drawbacks, ad hoc techniques predominate in usability study methodologies. This thesis proposes detailed taxonomies for the concepts of usability and the context of use. The taxonomies are organized hierarchically and contain precise descriptions of their attributes and subattributes. In order to illustrate the practical usefulness of the taxonomies, this thesis describes and discusses how the taxonomies were integrated into the development life cycle of two real-world projects in the eld of Intelligent Systems, namely, an Intelligent Speed Adaptation device and an automatic generator of user interfaces. At a speci c point of each project, a usability study was conducted, in which the taxonomies were used to structure and guide well-known usability activities such as usability requirements analysis, heuristic evaluation, and subjective analysis, [Resumo] Un importante obstáculo para a implantación do Deseño Centrado no Usuario no mundo real é que non existe unha definición do concepto de usabilidade que sexa precisa e comunmente aceptada e aplicada na práctica. A literatura tende a definir a usabilidade en termos demasiado concisos e ambiguos, e a describir a súa aplicación en termos informais. Idénticas críticas poden facerse ao concepto do contexto de uso, ao cal a usabilidade de sempre relativa. Como consecuencia disto, as técnicas ad hoc predominan nas metodoloxías de usabilidade. Esta tese propón taxonomías detalladas para os conceptos de usabilidade e contexto de uso. As taxonomías están organizadas xerárquicamente e conteñen descricións precisas dos seus atributos e subatributos. Para ilustrar a utilidade práctica das taxonomías, descríbese como foron integradas no ciclo de vida de desenvolvemento de dous produtos reais no campo dos Sistemas Intelixentes. Concretamente, un dispositivo de Adaptación Intelixente da Velocidade e un xerador automático de interfaces de usuario. Nun punto específico de cada proxecto realizouse un estudo de usabilidade, usando as taxonomías para estruturar e guiar actividades de usabilidade como a análise dos requisitos de usabilidade, a avaliación heurística e a análise subxectiva.

[Abstract] Usability studies are often based on ad hoc definitions of usability. These studies can be difficult to generalize, they might have a steep learning curve, and there is always the danger of being inconsistent with the concept of usability as defined in standards and the literature. This alternative approach involves comprehensive, general-purpose, and hierarchically structured taxonomies that follow closely the main usability literature. These taxonomies are then instantiated for a specific product. To illustrate this approach, a usability study for a prototype of an Intelligent Speed Adaptation device is described. The usability study consists of usability requirements analysis, heuristic evaluation, and subjective analysis, which helped identify problems of clarity, operability, robustness, safety, and aesthetics. As a context-specific usability taxonomy for this particular field of application happened to exist, the way that real-world usability results can be mapped to that taxonomy compared to the taxonomy in this article is examined, with the argument that this study’s taxonomy is more complete and generalizable.