Search

To fully understand human memory, it is necessary to understand its lifespan development. However, memory assessments often rely on significantly different methodologies for different age groups, and their results are typically not directly comparable. In this paper, we present a quantitative assessment of memory function spanning an age range of five to 85 years that is based on a model-based memory assessment. This approach yields a uniform metric that is directly interpretable and can be compared across different tasks and materials that are appropriate for different age groups. The results show a robust U-shape function, with long-term memory function at age 5 being comparable to that of cognitively impaired elderly individuals. These results and the method utilized could provide a new foundation for future studies on memory development across life stages.

Expanding on previous research highlighting the learning benefits of errors, this study explores the enduring effects of error-induced learning. Using an adaptive fact-learning system, 23 participants engaged in recognition, recall, and error tasks, with repeated testing for memory assessment. Initial findings echoed previous results: items learned through errors initially took longer to retrieve. However, a significant shift occurred over time; error items demonstrated faster retrieval speeds compared to study items, and, most notably, they exhibited greater resilience against forgetting. This study reaffirms the positive role of errors in learning and uncovers their contribution to enhanced long-term memory retention. These insights challenge traditional learning paradigms, advocating for an educational approach that recognizes and leverages the value of errors in learning processes.

In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations, given the operational context of the system. However, developing a safety monitor for practical deployment in real-world applications is challenging. This is due to limited access to internal workings and training data of the learned component. Furthermore, safety monitors should predict safety violations with low latency, while consuming a reasonable amount of computation. To address the challenges, we propose a safety monitoring method based on probabilistic time series forecasting. Given the learned component outputs and an operational context, we empirically investigate different Deep Learning (DL)-based probabilistic forecasting to predict the objective measure capturing the satisfaction or violation of a safety requirement (safety metric). We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models, with varying horizons, using an autonomous aviation case study. Our results suggest that probabilistic forecasting of safety metrics, given learned component outputs and scenarios, is effective for safety monitoring. Furthermore, for the autonomous aviation case study, Temporal Fusion Transformer (TFT) was the most accurate model for predicting imminent safety violations, with acceptable latency and resource consumption.

Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic representations of real-world conditions. However, while promising, these techniques may potentially introduce artifacts, distortions, or inconsistencies in the generated data that can affect the effectiveness of ADS testing. In our empirical study, we investigated how the quality of image-to-image (I2I) techniques influences the mitigation of the sim2real gap, using a set of established metrics from the literature. We evaluated two popular generative I2I architectures, pix2pix, and CycleGAN, across two ADS perception tasks at a model level, namely vehicle detection and end-to-end lane keeping, using paired simulated and real-world datasets. Our findings reveal that the effectiveness of I2I architectures varies across different ADS tasks, and existing evaluation metrics do not consistently align with the ADS behavior. Thus, we conducted task-specific fine-tuning of perception metrics, which yielded a stronger correlation. Our findings indicate that a perception metric that incorporates semantic elements, tailored to each task, can facilitate selecting the most appropriate I2I technique for a reliable assessment of the sim2real gap mitigation., Comment: In Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation 2024 (ICST '24)

The automated real-time recognition of unexpected situations plays a crucial role in the safety of autonomous vehicles, especially in unsupported and unpredictable scenarios. This paper evaluates different Bayesian uncertainty quantification methods from the deep learning domain for the anticipatory testing of safety-critical misbehaviours during system-level simulation-based testing. Specifically, we compute uncertainty scores as the vehicle executes, following the intuition that high uncertainty scores are indicative of unsupported runtime conditions that can be used to distinguish safe from failure-inducing driving behaviors. In our study, we conducted an evaluation of the effectiveness and computational overhead associated with two Bayesian uncertainty quantification methods, namely MC- Dropout and Deep Ensembles, for misbehaviour avoidance. Overall, for three benchmarks from the Udacity simulator comprising both out-of-distribution and unsafe conditions introduced via mutation testing, both methods successfully detected a high number of out-of-bounds episodes providing early warnings several seconds in advance, outperforming two state-of-the-art misbehaviour prediction methods based on autoencoders and attention maps in terms of effectiveness and efficiency. Notably, Deep Ensembles detected most misbehaviours without any false alarms and did so even when employing a relatively small number of models, making them computationally feasible for real-time detection. Our findings suggest that incorporating uncertainty quantification methods is a viable approach for building fail-safe mechanisms in deep neural network-based autonomous vehicles., Comment: In Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation 2024 (ICST '24)

Computational models of memory used in adaptive learning settings trace a learner's memory capacities. However, less work has been done on the implementation of these models in the clinical realm. Current assessment tools lack the reliable, convenient, and repeatable qualities needed to capture the individualized and evolving nature of memory decline. The goal of this project was to predict and track memory decline in subjectively- or mildly cognitively impaired (MCI) individuals by using a model-based, adaptive fact-learning system. Here, we present data demonstrating that these tools can diagnose mild memory impairment with over 80% accuracy after a single 8-minute learning session. These findings provide new insights into the nature and progression of memory decline and may have implications for the early detection and management of Alzheimer's disease and other forms of dementia.

Recent studies suggest that errors facilitate learning in certain conditions. Despite this, reinforcement paradigms dominate learning methods, subscribing to the narrative that errorless learning is the foundation of an ideal learning environment. If we continue to view learning from this restrictive perspective, we may fail to capture and apply the benefits of errors. In this paper, we investigate two potential mechanisms of post-error learning. Participants (N = 61) learned word pairs in either a study or error trial before taking a final test. Supporting past error learning literature, errors before a study opportunity led to better performance on a final test. Differences in reaction times between conditions support the theory that errors increase learning by acting as a mediator, or secondary cue, to the correct answer on subsequent tests.

Simulation-based testing represents an important step to ensure the reliability of autonomous driving software. In practice, when companies rely on third-party general-purpose simulators, either for in-house or outsourced testing, the generalizability of testing results to real autonomous vehicles is at stake. In this paper, we enhance simulation-based testing by introducing the notion of digital siblings, a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators built with different technologies, that operate collectively as an ensemble in the testing process. We exemplify our approach on a case study focused on testing the lane-keeping component of an autonomous vehicle. We use two open-source simulators as digital siblings, and we empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases. Our approach requires generating and running test cases for each individual simulator, in the form of sequences of road points. Then, test cases are migrated between simulators, using feature maps to characterize the exercised driving conditions. Finally, the joint predicted failure probability is computed, and a failure is reported only in cases of agreement among the siblings. Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin. We discuss the findings of our case study and detail how our approach can help researchers interested in automated testing of autonomous driving software.

Web test automation techniques employ web crawlers to automatically produce a web app model that is used for test generation. Existing crawlers rely on app-specific, threshold-based, algorithms to assess state equivalence. Such algorithms are hard to tune in the general case and cannot accurately identify and remove near-duplicate web pages from crawl models. Failing to retrieve an accurate web app model results in automated test generation solutions that produce redundant test cases and inadequate test suites that do not cover the web app functionalities adequately. In this paper, we propose WEBEMBED, a novel abstraction function based on neural network embeddings and threshold-free classifiers that can be used to produce accurate web app models during model-based test generation. Our evaluation on nine web apps shows that WEBEMBED outperforms state-of-the-art techniques by detecting near-duplicates more accurately, inferring better web app models that exhibit 22% more precision, and 24% more recall on average. Consequently, the test suites generated from these models achieve higher code coverage, with improvements ranging from 2% to 59% on an app-wise basis and averaging at 23%., Comment: 12 pages; in revision

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/ input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

The basal ganglia (BG) performs an important functional role in cognition, but models disagree about the nature of the relationship between BG activity and activity in other cortical areas. Previous computational models can be categorized as implementing the effects of the BG on prefrontal cortex as either local and direct, or involving other regions and, therefore, modulatory. To test which of these two effects best represents the role of the BG, a large fMRI dataset of 200 participants performing six, representative cognitive tasks was analyzed through Dynamic Causal Modeling (DCM). To ensure that DCM models were realistic and representative of a general brain architecture, the models were implemented within the putative neural underpinnings of the Common Model of Cognition, an abstract blueprint for cognition. The comparison showed that Mixed model, including both Direct and Modulatory connectivity, consistently outperformed models that included only direct or modulatory connections. It was also found that the relative rankings of the Direct and Modulatory models depended on the specific task, suggesting that the BG is a flexible system that adapts to task demands.

The basal ganglia (BG) performs an important functional role in cognition, but models disagree about the nature of the relationship between BG activity and activity in other cortical areas. Previous computational models can be categorized as implementing the effects of the BG on prefrontal cortex as either local and direct, or involving other regions and, therefore, modulatory. To test which of these two effects best represents the role of the BG, a large fMRI dataset of 200 participants performing six, representative cognitive tasks was analyzed through Dynamic Causal Modeling (DCM). To ensure that DCM models were realistic and representative of a general brain architecture, the models were implemented within the putative neural underpinnings of the Common Model of Cognition, an abstract blueprint for cognition. The comparison showed that Mixed model, including both Direct and Modulatory connectivity, consistently outperformed models that included only direct or modulatory connections. It was also found that the relative rankings of the Direct and Modulatory models depended on the specific task, suggesting that the BG is a flexible system that adapts to task demands.

Safe deployment of self-driving cars (SDC) necessitates thorough simulated and in-field testing. Most testing techniques consider virtualized SDCs within a simulation environment, whereas less effort has been directed towards assessing whether such techniques transfer to and are effective with a physical real-world vehicle. In this paper, we shed light on the problem of generalizing testing results obtained in a driving simulator to a physical platform and provide a characterization and quantification of the sim2real gap affecting SDC testing. In our empirical study, we compare SDC testing when deployed on a physical small-scale vehicle vs its digital twin. Due to the unavailability of driving quality indicators from the physical platform, we use neural rendering to estimate them through visual odometry, hence allowing full comparability with the digital twin. Then, we investigate the transferability of behavior and failure exposure between virtual and real-world environments, targeting both unintended abnormal test data and intended adversarial examples. Our study shows that, despite the usage of a faithful digital twin, there are still critical shortcomings that contribute to the reality gap between the virtual and physical world, threatening existing testing solutions that only consider virtual SDCs. On the positive side, our results present the test configurations for which physical testing can be avoided, either because their outcome does transfer between virtual and physical environments, or because the uncertainty profiles in the simulator can help predict their outcome in the real world., Comment: 13 pages; Accepted for publication in the IEEE Transactions of Software Engineering (TSE)

In a recent paper, Laird, Lebiere, and Rosenbloom (2017)highlight how 40 years of research on cognitive architectureshas begun to yield a dramatic convergence of different ap-proaches towards a set of basic assumptions that they calledthe “Standard Model of the Mind” (SMM), in analogy to theStandard Model of particle physics. The SMM was designedto capture a consensus view of “human-like minds”, whetherfrom AI or cognitive science, which if valid must also be trueof the human brain. Here, we provide a preliminary test ofthis hypothesis based on a re-analysis of fMRI data from fourtasks that span a wide range of cognitive functions and cog-nitive complexity, and are representative of the specific formof intelligence and flexibility that is associated with higher-level human cognition. Using an established method (DynamicCausal Modeling) to examine functional connectivity betweenbrain regions, the SMM was compared against two alternativemodels that violate either functional or structural assumptionsof the SMM. The results show that, in every dataset, the SMMsignificantly outperforms the other models, suggesting that theSMM best captures the functional requirements of brain dy-namics in fMRI data among these alternatives.

Behavioral data, though has been an influential index oncognitive processes, is under scrutiny for having poorreliability as a result of noise or lacking replications ofreliable effects. Here, we argue that cognitive modeling canbe used to enhance the test-retest reliability of the behavioralmeasures by recovering individual-level parameters frombehavioral data. We tested this empirically with theProbabilistic Stimulus Selection (PSS) task, which is used tomeasure a participant’s sensitivity to positive or negativereinforcement. An analysis of 400,000 simulations from anAdaptive Control of Thought - Rational (ACT-R) model ofthis task showed that the poor reliability of the task is due tothe instability of the end-estimates: because of the way thetask works, the same participants might sometimes end uphaving apparently opposite scores. To recover the underlyinginterpretable parameters and enhance reliability, we used aBayesian Maximum A Posteriori (MAP) procedure. We wereable to obtain reliable parameters across sessions (IntraclassCorrelation Coefficient ~ 0.5), and showed that this approachcan further be used to provide superior measures in terms ofreliability, and bring greater insights into individualdifferences.

A complete understanding of cognitive function in humansmust incorporate a model of interactions between networkedbrain regions. Alterations to these network interactions under-lie cognitive impairment in many neurodegenerative diseases,providing an important physiological link between brain struc-ture and cognitive function. Cognitive architectures have of-ten been used to explain how healthy brains function, typi-cally using task-based activity. However, this description isincomplete. Most systems-level brain activity is spontaneous,or intrinsic, and occurs whether or not a subject is performinga task. Here, we provide evidence that the Common Modelof Cognition, a consensus model derived from an analysis ofexisting cognitive architectures, can (a) be generalized to ac-count for brain activity at rest, rather than during tasks, and (b)correctly identify differences in basal ganglia connectivity inParkinson’s Disease.

Methodological advances have made it possible to generatefMRI predictions for cognitive architectures, such as ACT-R, thus expanding the range of model predictions and mak-ing it possible to distinguish between alternative models thatproduce otherwise identical behavioral patterns. However, fortasks associated with relatively brief response times, fMRI pre-dictions are often not sufficient to compare alternative models.In this paper, we outline a method based on effective connec-tivity, which significantly augments the amount of informationthat can be extracted from fMRI data to distinguish betweenmodels. We show the application of this method in the caseof two competing ACT-R models of the Stroop task. Althoughthe models make, predictably, identical behavioral and BOLDtime-course predictions, patterns of functional connectivity fa-vor one model over the other. Finally, we show that the samedata suggests directions in which both models should be re-vised.

The feedback-related negativity (FRN) is an event-related potential that differentiates between positive and negative feed-back, occurring most prominently at frontocentral electrodes 200-300ms after delivery of feedback. The FRN seems tobe reflective of a reward prediction error, as the magnitude of the ERP component has been related to the magnitude ofprediction error estimated through reinforcement learning (RL) models. We aim to further understanding of the FRNand its relationship to behavior by replicating the study of Reinhart & Woodman (2014), replacing tDCS with focal, tar-geted transcranial magnetic stimulation (TMS) over the frontocentral region. Preliminary data shows that our participantsreliably generate a FRN when presented with incorrect feedback, and that single-trial estimates of theta power are signifi-cantly correlated with RL-derived single-trial estimates of prediction error for correct trials. We will examine the effect ofstimulation both on participant behavior as well as on RL parameter estimates.

The current study aimed to elucidate the contributions of thesubcortical basal ganglia to human language by adopting theview that these structures engage in a basic neurocomputationthat may account for its involvement across a wide range of lin-guistic phenomena. Specifically, we tested the hypothesis thatbasal ganglia reinforcement learning mechanisms may accountfor variability in semantic selection processes necessary forambiguity resolution. To test this, we used a biased homographlexical ambiguity priming task that allowed us to measure au-tomatic processes for resolving ambiguity towards high fre-quency word meanings. Individual differences in task perfor-mance were then related to indices of basal ganglia function-ing and reinforcement learning, which were used to group sub-jects by learning style: primarily from choosing positive feed-back (Choosers), primarily from avoiding negative feedback(Avoiders), and balanced participants who learned equally wellfrom both (Balanced). The pattern of results suggests that bal-anced individuals, whom learn from both positive and negativereward equally well, had significantly lower access to the sub-ordinate homograph word meaning. Choosers and Avoiders,on the other hand, had higher access to the subordinate wordmeaning even after a long delay between prime and target. Ex-perimental findings were then tested using an ACT-R compu-tational model of reinforcement learning that learns from bothpositive and negative feedback. Results from the computa-tional model confirm and extend the pattern of behavioral find-ings, and provide a reinforcement learning account of lexicalpriming processes in human linguistic abilities, where a dual-path reinforcement learning system is necessary for preciselymapping out word co-occurrence probabilities.

The growing application of deep neural networks in safety-critical domains makes the analysis of faults that occur in such systems of enormous importance. In this paper we introduce a large taxonomy of faults in deep learning (DL) systems. We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts. Structured interviews with 20 researchers and practitioners describing the problems they have encountered in their experience have enriched our taxonomy with a variety of additional faults that did not emerge from the other two sources. Our final taxonomy was validated with a survey involving an additional set of 21 developers, confirming that almost all fault categories (13/15) were experienced by at least 50% of the survey participants.

E2E web test suites are prone to test dependencies due to the heterogeneous multi-tiered nature of modern web apps, which makes it difficult for developers to create isolated program states for each test case. In this paper, we present the first approach for detecting and validating test dependencies present in E2E web test suites. Our approach employs string analysis to extract an approximated set of dependencies from the test code. It then filters potential false dependencies through natural language processing of test names. Finally, it validates all dependencies, and uses a novel recovery algorithm to ensure no true dependencies are missed in the final test dependency graph. Our approach is implemented in a tool called TEDD and evaluated on the test suites of six open-source web apps. Our results show that TEDD can correctly detect and validate test dependencies up to 72% faster than the baseline with the original test ordering in which the graph contains all possible dependencies. The test dependency graphs produced by TEDD enable test execution parallelization, with a speed-up factor of up to 7x., Comment: 11 pages, published in the Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019), pp. 154-164

Behavior that is contingent on conditional rules necessitatesan abstraction away from concrete stimulus-response identi-ties in order to form a rule template, but also a subsequenttransformation of representation back into sensorimotor for-mat in order to produce concrete behavior. Evidence suggeststhat dorsal premotor cortex (PMd) is well-positioned to me-diate such an operation. We utilized repetitive transcranialmagnetic stimulation, a non-invasive manner of perturbing thefunctioning of targeted cortical regions, to investigate the roleof dorsal premotor cortex during performance of a Rapid In-structed Task Learning paradigm. The task required partici-pants to form conditional associations between stimuli and re-sponses carrying varying levels of abstraction. Selective inter-ference of response times to stimuli presentation was observedonly when the task necessitated the participants to resolve aconditional response referring to an internally-produced repre-sentation of a rule element with relatively abstracted quality.We conclude that PMd specifically supports conditional rulebehavior through transformation of abstract representations toconcrete response, when the conditional rule necessary to re-solve includes abstract, internally-produced identities.

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three human subjects to collaborate and solve a task using direct brain-to-brain communication. Two of the three subjects are "Senders" whose brain signals are decoded using real-time EEG data analysis to extract decisions about whether to rotate a block in a Tetris-like game before it is dropped to fill a line. The Senders' decisions are transmitted via the Internet to the brain of a third subject, the "Receiver," who cannot see the game screen. The decisions are delivered to the Receiver's brain via magnetic stimulation of the occipital cortex. The Receiver integrates the information received and makes a decision using an EEG interface about either turning the block or keeping it in the same position. A second round of the game gives the Senders one more chance to validate and provide feedback to the Receiver's action. We evaluated the performance of BrainNet in terms of (1) Group-level performance during the game; (2) True/False positive rates of subjects' decisions; (3) Mutual information between subjects. Five groups of three subjects successfully used BrainNet to perform the Tetris task, with an average accuracy of 0.813. Furthermore, by varying the information reliability of the Senders by artificially injecting noise into one Sender's signal, we found that Receivers are able to learn which Sender is more reliable based solely on the information transmitted to their brains. Our results raise the possibility of future brain-to-brain interfaces that enable cooperative problem solving by humans using a "social network" of connected brains.

The approach presented in this paper addresses the question of the proper scientific basis for Human Factors modeling and proposes an architectural framework for integrating federated models and simulations. It is intended to be applicable to a broad range of scenarios across domains. Indeed, broad applicability and integration of modeling and simulation techniques are essential to their effectiveness and validation. Our approach is grounded in the concept of unified theories of cognition, implemented computationally as cognitive architectures. However, cognitive architectures also need to be constrained by our knowledge of neural processes in order to properly account for all cognitive, perceptual and motor factors. Despite that attention to the small-scale basis of cognition, cognitive modeling can scale up to social situations and large-scale network settings through a process of abstraction and integration. Key to that process is the availability of easily accessible resources in the form of existing cognitive models, implemented tasks, simulation environments adhering to a common standard, and human performance data to constrain and validate models. Investment in that infrastructure is essential to ensure growth and scalability in the application of cognitive models and their proper integration in military simulations., See also ADA564696. Human Modelling for Military Application (Applications militaires de la modelisation humaine). Supported in part by ONR Grant no. N00014-07-C-0912

Introduction The project s goal was to reproduce robust and intelligent decision making capabilities in artificial agents by integrating two successful cognitive architectures, ACT-R (Anderson, 2007) and Leabra (O Reiily & Munakata, 2000). The rationale was that such an integration effort would yield insights on the general mechanisms that allow rapid decision-making in real-time. Taken separately, ACTR and Leabra incorporate different views of how decision-making and robust behavior occur. The two architectures have different and complementary strengths and weaknesses and work at different levels of abstractions. Thus, an integration of the two would possibly yield a uniform framework for understanding the computational basis of robust intelligence and decision-making in humans., The original document contains color images.

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/ input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.