Search

Metamorphic Relations (MRs) play a key role in determining the fault detection capability of Metamorphic Testing (MT). As human judgement is required for MR identification, systematic MR generation has long been an important research area in MT. Additionally, due to the extra program executions required for follow-up test cases, some concerns have been raised about MT cost-effectiveness. Consequently, the reduction in testing costs associated with MT has become another important issue to be addressed. MR composition can address both of these problems. This technique can automatically generate new MRs by composing existing ones, thereby reducing the number of follow-up test cases. Despite this advantage, previous studies on MR composition have empirically shown that some composite MRs have lower fault detection capability than their corresponding component MRs. To investigate this issue, we performed theoretical and empirical analyses to identify what characteristics component MRs should possess so that their corresponding composite MR has at least the same fault detection capability as the component MRs do. We have also derived a convenient, but effective guideline so that the fault detection capability of MT will most likely not be reduced after composition.

Big data validation and system verification are crucial for ensuring the quality of big data applications. However, a rigorous technique for such tasks is yet to emerge. During the past decade, we have developed a big data system called CMA for investigating the classification of biological cells based on cell morphology which is captured in diffraction images. CMA includes a collection of scientific software tools, machine learning algorithms, and a large-scale cell image repository. In order to ensure the quality of big data system CMA, we developed a framework for rigorously validating the massive scale image data as well as adequately verifying both the software tools and machine learning algorithms. The validation of big data is conducted by iteratively selecting the data using a machine learning approach. An experimental approach guided by a feature selection algorithm is introduced in the framework to select an optimal feature set for improving the machine learning performance. The verification of software and algorithms is developed on the iterative metamorphic testing approach due to the non-testable property of the software and algorithms. A machine learning approach is introduced for developing test oracles iteratively to ensure the adequacy of the test coverage criteria. Performance of the machine learning algorithm is evaluated with a stratified N-fold cross validation and confusion matrix. We describe the design of the proposed big data verification and validation framework with CMA as the case study, and demonstrate its effectiveness through verifying and validating the dataset, the software and the algorithms in CMA.

Modern-day demands for services often require an availability on a 24/7 basis as well as online accessibility around the globe. For this sake, personalized virtual assistants, called chatbots, are implemented. Such systems offer services, goods or information in natural language. These natural language processing (NLP) programs respond to the user in real time and offer an intuitive and simple interface to interact with. Advantages like these make them increasingly popular. Therefore, ensuring correct functionality of chatbots is of increasing importance. However, since different implementations and user behaviour result in unpredictable results, the chatbot’s input and output data are difficult to predict and classify as well. Under such circumstances, test cases can be inferred from the domain of possible inputs of a system under test (SUT). Ontologies are concepts used in AI to provide formal representations of knowledge for a specific domain. Such ontological models contain structured information that is used for test generation. On the other hand, testing of chatbots represents a challenge because of the absence of a test oracle. In this paper, both challenges are addressed by conceptualizing ontologies for input generation and output processing in form of a metamorphic testing approach. In this scenario, both concepts are applied for automated testing of chatbots. The approach is demonstrated on a real system from the tourism domain, thereby discussing the obtained results.

Test-suite-based automated program repair (APR) techniques acquire information from an input test suite to guide the repair process, aiming to produce a repair that can pass all test cases of the input test suite. Obviously, the input test suite has a critical impact on the repair effectiveness of APR techniques. This article reports on a study of the APR input test suites from a new perspective. We first propose a novel method of constructing the APR input test suites, using information derived from violated metamorphic relations. We then empirically evaluate our construction method using three APR techniques (Angelix, CETI, and GenProg), comparing it with random and code-coverage-based construction methods that are used as the experimental control. The results show that our approach is complementary to these two input test suite construction methods. This article illustrates a new use of metamorphic relations for program repair.

The validation of a cloud system can be complicated by the size of the system, the number of users that can concurrently request services, and the virtualization used to give the illusion of using dedicated machines. Unfortunately, it is not feasible to use conventional testing methods with cloud systems. This article proposes a framework, called TEA- Cloud , that integrates simulation with testing methods for validating cloud system designs. Testing is applied on both functional and nonfunctional aspects of the cloud, like performance and cost. The aim of the framework is to provide a complete methodology to help users to model both software and hardware parts of cloud systems and automatically test the validity of these clouds using a cost-effective approach. Metamorphic testing is used to overcome the lack of an oracle that checks whether the behavior observed in testing is allowed. Metamorphic testing is based on metamorphic relations (MRs). We define three families of MRs, which target issues such as performance, resource provisioning, and cost. TEA- Cloud was evaluated through an empirical study that used fault seeding (mutation) and ten MRs for testing different cloud configurations. The results were promising, with TEA- Cloud finding all seeded faults.

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction.

With recent breakthroughs in Deep Learning (DL), DL systems are increasingly deployed in safety-critical fields. Hence, some software testing methods are required to ensure the reliability and safety of DL systems. Since the rules of DL systems are inferred from training data, it is difficult to know the implementation rules about each behavior of DL systems. At the same time, Random Testing (RT) is a popular testing method and the knowledge about software implementation is not needed when we use RT. Therefore, RT is very suitable for the testing of DL systems. And the existing mechanisms for testing DL systems also depend heavily on RT by the labeled test data. In order to increase the effectiveness of RT for DL systems, we design, implement and evaluate the Adaptive Random Testing for DL systems (ARTDL), which is the first Adaptive Random Testing (ART) method to improve the effectiveness of RT for DL systems. ARTDL refers to the idea of ART. That is, fewer test cases are needed to detect failures by selecting the test case with the furthest distance from non-failure-causing test cases. Firstly, we propose the Feature-based Euclidean Distance (FED) as the distance metric that can be used to measure the difference between failure-causing inputs and non-failure-causing inputs. Secondly, we verify the availability of FED by presenting the failure pattern of DL models. Finally, we design ARTDL algorithm to generate the test cases that are more likely to cause failures based on the FED. We implement ARTDL to test top performing DL models in the field of image classification and automatic driving. The results show that, on average, the number of test cases used to find the first bug is reduced by 62.74% through ARTDL, compared with RT.

Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles. To alleviate this situation, we propose an automated technique for building metamorphic testing environments for DSLs. Metamorphic testing identifies expected relationships between the outputs of two consecutive tests, reducing the effort in specifying oracles and creating test cases manually. This new ideas paper presents the overarching concepts, the architecture and a prototype implementation. We illustrate our proposal using a DSL to model and simulate data centres.

Enhancing the trust of machine learning-based classifiers with large input spaces is a desirable goal; however, due to high labeling costs and limited resources, this is a challenging task. One solution is to use test input prioritization techniques that aim to identify and label only the most effective test instances. These prioritized test inputs can then be used with some popular testing techniques e.g., Metamorphic testing (MT) to test and uncover implementation bugs in computationally complex machine learning classifiers that suffer from the oracle problem. However, there are certain limitations involved with this approach, (i) using a small number of prioritized test inputs may not be enough to check the program correctness over a large variety of input scenarios, and (ii) traditional MT approaches become infeasible when the programs under test exhibit a non-deterministic behavior during training e.g., Neural Network-based classifiers. Therefore, instead of using MT for testing purposes, we propose a metamorphic relation to solve a data generation/labeling problem; that is, enhancing the test inputs effectiveness by extending the prioritized test set with new tests without incurring additional labeling costs. Further, we leverage the prioritized test inputs (both source and follow-up data sets) and propose a statistical hypothesis testing (for detection) and machine learning-based approach (for prediction) of faulty behavior in two other machine learning classifiers (Neural Network-based Intrusion Detection Systems). In our case, the problem is interesting in the sense that injected bugs represent the high accuracy producing mutated program versions that may be difficult to detect by a software developer. The results indicate that (i) the proposed statistical hypothesis testing is able to identify the induced buggy behavior, and (ii) Random Forest outperforms and achieves the best performance over SVM and k-NN algorithms.

Datalog is a popular query language with applications in several domains. Like any complex piece of software, Datalog engines may contain bugs. The most critical ones manifest as incorrect results when evaluating queries—we refer to these as query bugs. Given the wide applicability of the language, query bugs may have detrimental consequences, for instance, by compromising the soundness of a program analysis that is implemented and formalized in Datalog. In this paper, we present the first metamorphic-testing approach for detecting query bugs in Datalog engines. We ran our tool on three mature engines and found 13 previously unknown query bugs, some of which are deep and revealed critical semantic issues.

Ensuring the equality of SMT solvers is critical due to its broad spectrum of applications in academia and industry, such as symbolic execution and program verification. Existing approaches to testing SMT solvers are either too costly or find difficulties generalizing to different solvers and theories, due to the test oracle problem. To complement existing approaches and overcome their weaknesses, this paper introduces skeletal approximation enumeration (SAE), a novel lightweight and general testing technique for all first-order theories. To demonstrate its practical utility, we have applied the SAE technique to test Z3 and CVC4, two comprehensively tested, state-of-the-art SMT solvers. By the time of writing, our approach had found 71 confirmed bugs in Z3 and CVC4,55 of which had already been fixed.

Machine Reading Comprehension (MRC) in Natural Language Processing has seen great progress recently. But almost all the current MRC software is validated with a reference-based method, which requires well-annotated labels for test cases and tests the software by checking the consistency between the labels and the outputs. However, labeling test cases of MRC could be very costly due to their complexity, which makes reference-based validation hard to be extensible and sufficient. Furthermore, solely checking the consistency and measuring the overall score may not be sensible and flexible for assessing the language understanding capability. In this paper, we propose a property-based validation method for MRC software with Metamorphic Testing to supplement the reference-based validation. It does not refer to the labels and hence can make much data available for testing. Besides, it validates MRC software against various linguistic properties to give a specific and in-depth picture on linguistic capabilities of MRC software. Comprehensive experimental results show that our method can successfully reveal violations to the target linguistic properties without the labels. Moreover, it can reveal problems that have been concealed by the traditional validation. Comparison according to the properties provides deeper and more concrete ideas about different language understanding capabilities of the MRC software.

Testing deep learning systems requires expensive labeled data. In recent years, researchers began to leverage metamorphic testing to address this issue. However, metamorphic relations on image data remain poorly understood. To gain a deeper understanding of these metamorphic relations, we survey common image operations modeling covariate shift, manually classify and categorize the underlying metamorphic relations, and conduct experiments to validate our classifications. In our experiments, we train three popular convolutional neural network architectures on an image classification task. Next, we apply metamorphic operations on input test images and measure the change in classification accuracy and cross-entropy loss. A hierarchical clustering algorithm clusters these results and plots a dendrogram. We compare the groups from manual classification and the clusters from the algorithm to provide key insights. We find that Affine and Noise relations are consistent. Furthermore, we recommend metamorphic relationships to save time and better test deep learning systems in the future.

Testing computationally complex neural network-based applications (i.e. network intrusion detection systems) is a challenging task due to the absence of a test oracle. Metamorphic testing is a method to potentially solve the oracle problem when the correctness of individual output is difficult to determine. However, due to the stochastic nature of these applications, multiple runs with the same input can produce slightly different results; thus rendering traditional metamorphic testing technique inadequate. To address this problem, this paper proposes a statistical metamorphic testing technique to test neural network based Network Intrusion Detection Systems (N-IDSs) in a nondeterministic environment. We also performed mutation analysis to show the effectiveness of the proposed approach. The results show that the proposed method has a strong defect detection capability and is able to kill 100% implementation bugs in two neural network-based N-IDSs, and 66.66% in a neural network-based cancer prediction system.

Proper validation of a simulation model is essential for confidence in its accuracy and credibility. However, many of the most effective approaches for simulation validation require access to data that may be unavailable. Metamorphic Testing (MT), an approach from traditional software testing, has been shown to be useful for verification of simulation software in similar situations. Recent research shows that MT can be applied to the validation of agent based and discrete event simulation models as well. In this paper we build on that work and show how MT can be applied to hybrid simulation models. We demonstrate the effectiveness of our approach by applying it on a case study of helping behavior among servers in a restaurant.

Metamorphic testing seeks to verify software in the absence of test oracles. Our application domain is ocean system modeling, where test oracles rarely exist, but where symmetries of the simulated physical systems are known. The input data set is large owing to the requirements of the application domain. This paper presents work in progress for the automated generation of metamorphic test scenarios using machine learning. We extended our previously proposed method [1] to identify metamorphic relations with reduced computational complexity. Initially, we represent metamorphic relations as identity maps. We construct a cost function that minimizes for identifying a metamorphic relation orthogonal to previously found metamorphic relations and penalize for the identity map. A machine learning algorithm is used to identify all possible metamorphic relations minimizing the defined cost function. We propose applying dimensionality reduction techniques to identify attributes in the input which have high variance among the identified metamorphic relations. We apply mutation on these selected attributes to identify distinct metamorphic relations with reduced computational complexity. For experimental evaluation, we subject the two implementations of an ocean-modeling application to the proposed method to present the use of metamorphic relations to test the two implementations of this application., 5 Pages, 1 Figure

Since the popularization of social media, news has entered our lives digitally. While news is spreading broader and faster, fake news is becoming an increasingly popular topic. Fake news detection is therefore important in both social media and research areas. With artificial intelligence technology, software engineers have developed a lot of fake news detection systems. One of the biggest challenges for such systems is that they may face the oracle problem, which means that there may not be a way, or it may take too long time, to confirm the correctness of a specific output. Metamorphic Testing has been applied successfully to alleviate the oracle problem in many different areas, including in artificial intelligence. In this paper, we propose several metamorphic relations for fake news detection and report on experiments using metamorphic testing on fake news detection applications.

Deep Neural Network (DNN) models are widely used for image classification. While they offer high performance in terms of accuracy, researchers are concerned about if these models inappropriately make inferences using features irrelevant to the target object in a given image. To address this concern, we propose a metamorphic testing approach that assesses if a given inference is made based on irrelevant features. Specifically, we propose two metamorphic relations (MRs) to detect such unreliable inferences. These relations expect (a) the classification results with different labels or the same labels but less certainty from models after corrupting the relevant features of images, and (b) the classification results with the same labels after corrupting irrelevant features. The inferences that violate the metamorphic relations are regarded as unreliable inferences. Our evaluation demonstrated that our approach can effectively identify unreliable inferences for single-label classification models with an average precision of 64.1% and 96.4% for the two MRs, respectively. As for multi-label classification models, the corresponding precision for MR-1 and MR-2 is 78.2% and 86.5%, respectively. Further, we conducted an empirical study to understand the problem of unreliable inferences in practice. Specifically, we applied our approach to 18 pre-trained single-label image classification models and 3 multi-label classification models, and then examined their inferences on the ImageNet and COCO datasets. We found that unreliable inferences are pervasive. Specifically, for each model, more than thousands of correct classifications are actually made using irrelevant features. Next, we investigated the effect of such pervasive unreliable inferences, and found that they can cause significant degradation of a model’s overall accuracy. After including these unreliable inferences from the test set, the model’s accuracy can be significantly changed. Therefore, we recommend that developers should pay more attention to these unreliable inferences during the model evaluations. We also explored the correlation between model accuracy and the size of unreliable inferences. We found the inferences of the input with smaller objects are easier to be unreliable. Lastly, we found that the current model training methodologies can guide the models to learn object-relevant features to certain extent, but may not necessarily prevent the model from making unreliable inferences. We encourage the community to propose more effective training methodologies to address this issue.

Deep Neural Network based models are widely used for perception and control in autonomous driving. Recent work leverages metamorphic testing to improve defect detection but is limited to using only an equality-based metamorphic relation. Thus, it does not provide sufficient expressiveness for users to define custom metamorphic relations nor means to automatically generate meaningful inputs based on such expressive metamorphic relations that reflect real-world traffic behaviors. In this paper, we preliminarily design and evaluate a declarative Behaviour-Driven Development (BDD)-based metamorphic testing framework BMT, which enables domain experts to specify custom traffic behaviors—a car shall decelerate by x% when a bicycle is in front, etc. It then automatically translates a human-written behavior to a corresponding metamorphic relation and synthesizes meaningful test inputs using a variety of image and graphics processing techniques. Our preliminary evaluation shows that BMT can detect a significant number of erroneous predictions of three driving models for speed predictions. These detected erroneous predictions are manually examined and confirmed by six human judges as meaningful traffic violations. By automating test generation from custom behaviors, BMT enables experts to easily express domain-specific constraints and finds violations of such constraints.

Recently, metamorphic testing (MT) has been used to augment test datasets by inclusion of new (follow-up) test cases constructed from existing (source) test cases using metamorphic relations (MRs). It has been reported that the augmented test datasets usually have higher fault detection capabilities. It is natural to ask which contributes to the improvement of the fault detection capabilities. To investigate this issue, we conducted an empirical study on three DNN models by feeding 70,000 handwritten digits images, in which six sets of MRs were designed. We found that follow-up test cases have better fault detection capabilities than source test cases. Furthermore, the impacts of the amounts of follow-up test cases on the fault detection capabilities of the augmented test datasets were investigated.

Visual question answering (VQA) takes an image and a natural-language question as input and returns a natural-language answer. To date, VQA models are primarily assessed by their accuracy on high-level reasoning questions. Nevertheless, Given that perception tasks (e.g., recognizing objects) are the building blocks in the compositional process required by high-level reasoning, there is a demanding need to gain insights into how much of a problem low-level perception is. Inspired by the principles of software metamorphic testing, we introduce MetaVQA, a model-agnostic framework for benchmarking perception capability of VQA models. Given an image i, MetaVQA is able to synthesize a low-level perception question q. It then jointly transforms (i, q) to one or a set of sub-questions and sub-images. MetaVQA checks whether the answer to (i, q) satisfies metamorphic relationships (MRs), denoting perception consistency, with the composed answers of transformed questions and images. Violating MRs denotes a failure of answering perception questions. MetaVQA successfully detects over 4.9 million perception failures made by popular VQA models with metamorphic testing. The state-of-the-art VQA models (e.g., the champion of VQA 2020 Challenge) suffer from perception consistency problems. In contrast, the Oscar VQA models, by using anchor points to align questions and images, show generally better consistency in perception tasks. We hope MetaVQA will revitalize interest in enhancing the low-level perceptual abilities of VQA models, a cornerstone of high-level reasoning.

Advanced Driver-Assistance Systems (ADASs) have become increasingly popular. To ensure safety, simulation testing is essential. In this research, we conducted a case study to investigate the fault-detection effectiveness of existing ADAS testing standards: We tested the Lane Keeping Assist System (LKAS), which is a prebuilt ADAS module of MATLAB and Simulink. We first tested LKAS using the European New Car Assessment Programme (Euro NCAP), which contains 40 prebuilt driving scenarios in MATLAB. Our results show that none of the 40 scenarios detected any failure. We then continued the tests by applying a simple metamorphic relation "equivalence under geometric transformation," and a previously unknown real-life bug in LKAS was immediately revealed. We reported this finding to the MATLAB team in the US, who then confirmed the bug and corrected the LKAS code. This research provides a strong case for incorporating metamorphic testing into ADAS testing standards and protocols.

The BRAIN platform that we developed over the last 2.5 years is a collaborative platform for the neuroscience research community. Such platforms are crucial for expediting research in data-heavy research communities such as the neuroscience optical imaging community. However, it is important to also build-in quality assurance within the platform to ensure that the results can be trusted. Testing the accuracy of scientific algorithms by comparing the actual results to the expected requires tedious manual efforts, which typically has to be repeated for each new context. Metamorphic testing has been shown to be a powerful technique for scientific data analysis applications where asserting correctness is usually a manual, laborious, and challenge task. It uses relations between a set of inputs and a set of outputs to detect issues instead of comparing actual and expected results and thus addresses the above problem. In this paper we report on a feasibility study where we applied metamorphic testing to one step in the calcium imaging pipeline analysis, namely the step to extract the locations of the neuron cells. We tested the cell registration module of CaImAn, one popular software library for calcium imaging analysis, for robustness against slight changes in the input images, i.e. we applied linear transformations such as small degrees of rotation and blurring to the input images, and then evaluated the neurons identified by the cell registration module for both the original and the transformed inputs. Our results show that for transformations such as rotation and translation, CaImAn’s cell registration unexpectedly identified more neurons for the transformed input than for the original image. Meanwhile, for transformations such as padding, CaImAn identified fewer neurons when run on the the transformed inputs. Our results demonstrate the need for evaluating third party libraries for confidence. Furthermore, it also demonstrates that metamorphic testing is a viable quality assurance technique for the BRAIN platform as well as for similar scientific applications where it is difficult to detect analysis inaccuracies.

Synthesising photo-realistic images from natural language is one of the challenging problems in computer vision. Over the past decade, a number of approaches have been proposed, of which the improved Stacked Generative Adversarial Network (StackGAN-v2) has proven capable of generating high resolution images that reflect the details specified in the input text descriptions. In this paper, we aim to assess the robustness and fault-tolerance capability of the StackGAN-v2 model by introducing variations in the training data. However, due to the working principle of Generative Adversarial Network (GAN), it is difficult to predict the output of the model when the training data are modified. Hence, in this work, we adopt Metamorphic Testing technique to evaluate the robustness of the model with a variety of unexpected training dataset. As such, we first implement StackGAN-v2 algorithm and test the pre-trained model provided by the original authors to establish a ground truth for our experiments. We then identify a metamorphic relation, from which test cases are generated. Further, metamorphic relations were derived successively based on the observations of prior test results. Finally, we synthesise the results from our experiment of all the metamorphic relations and found that StackGAN-v2 algorithm is susceptible to input images with obtrusive objects, even if it overlaps with the main object minimally, which was not reported by the authors and users of StackGAN-v2 model. The proposed metamorphic relations can be applied to other text-to-image synthesis models to not only verify the robustness but also to help researchers understand and interpret the results made by the machine learning models., 8 pages, accepted at the 6th International Workshop on Metamorphic Testing (MET'21)

Machine learning-based malware detection systems are often vulnerable to evasion attacks, in which a malware developer manipulates their malicious software such that it is misclassified as benign. Such software hides some properties of the real class or adopts some properties of a different class by applying small perturbations. A special case of evasive malware hides by repackaging a bonafide benign mobile app to contain malware in addition to the original functionality of the app, thus retaining most of the benign properties of the original app. We present a novel malware detection system based on metamorphic testing principles that can detect such benign-seeming malware apps. We apply metamorphic testing to the feature representation of the mobile app, rather than to the app itself. That is, the source input is the original feature vector for the app and the derived input is that vector with selected features removed. If the app was originally classified benign, and is indeed benign, the output for the source and derived inputs should be the same class, i.e., benign, but if they differ, then the app is exposed as (likely) malware. Malware apps originally classified as malware should retain that classification, since only features prevalent in benign apps are removed. This approach enables the machine learning model to classify repackaged malware with reasonably few false negatives and false positives. Our training pipeline is simpler than many existing ML-based malware detection methods, as the network is trained end-to-end to jointly learn appropriate features and to perform classification. We pre-trained our classifier model on 3 million apps collected from the widely-used AndroZoo dataset.1 We perform an extensive study on other publicly available datasets to show our approach’s effectiveness in detecting repackaged malware with more than 94% accuracy, 0.98 precision, 0.95 recall, and 0.96 F1 score.

Simulation validation, demonstrating that a simulation accurately represents the system it studies, poses a particularly potent form of the oracle problem. A simulation must be validated for its results to be reliable, but often no oracle exists due to the nature of simulations. We propose taking the metamorphic testing (MT) approach from the realm of software verification and applying it for validating executable simulation models. By establishing pseudo-oracles based on metamorphic relations between parameters and behaviors within an executable model, we can create a methodical approach for validating simulation models. We propose an overall framework and guidelines to apply MT for simulation validation, with details for two prevalent simulation approaches: agent-based simulation models (ABM), and discrete-event simulation (DES) modeling. Through three case studies, we demonstrate the successful application of MT toward validating three different simulation models: an ABM of gossip propagation, an ABM of cancer, and a DES of airport check-in and security.

Context In model-based software development, bidirectional model transformation (BX) provides a fundamental solution to model synchronization that can retain the consistency among models. Similar to conventional programs, a BX program may also contain bugs. Accordingly, a BX program must be tested prior to being used in practice. A challenging problem of testing BX programs is to construct test oracles (e.g., assertions and expected output models), which are usually difficult and/or expensive to manually specify. Objective In we paper, we investigate how to alleviate the oracle problem in BX testing via reducing the costs of developing test oracles. Method We propose a metamorphic testing approach for BX. First, we identify three generic metamorphic relations for BX. Afterwards, we define a metamodel MT4MT to establish metamorphic test groups and test scripts. We also propose a testing framework to support metamorphic testing based on MT4MT. Results We conducted an experimental study of mutation analysis and a case study on three ATL-based ad-hoc BXs. The results of the experimental study and the case study showed that our approach killed 79.38% mutants and enabled us to test real-world ATL-based ad-hoc BXs. We also demonstrated that MT4MT can be used to test the semantics properties of BXs. Conclusion Our approach is an effective and practical approach with lower costs of developing test oracles.

Scientific software can be used for decades and is constantly evolving. Recently, metamorphic testing, a property-based testing technique, has shown to be effective in testing scientific software, and the necessary properties are expressed as metamorphic relations. However, the development of metamorphic relations is difficult: it requires considerable practical expertise for the software tester. In this article, we report our experience of uncovering metamorphic relations from a user forum's questions of the United States Environmental Protection Agency's Storm Water Management Model (SWMM). Our study not only illustrates a wealth of end users’ expertise in interpreting software results, but also demonstrates the usefulness of classifying the user-oriented metamorphic relations into a nominal, ordinal, and functional hierarchy mainly from the software output perspective.

Metamorphic testing uncovers defects by checking whether a relation holds among multiple software executions. These relations are known as metamorphic relations (MRs). For scientific software operating in a large multi-parameter input space, identifying MRs that determine the simultaneous changes among multiple variables is challenging. In this poster, we propose a fully automatic approach to classifying input and output variables from scientific software’s user manual, mining these variables’ associations from the user forum to generate MRs, and validating the MRs with existing regression tests. Preliminary results of our end-to-end MR support for the Storm Water Management Model (SWMM) are reported.

A common task in search-based testing is automatically identifying valuable test cases for software systems. However, existing approaches tend to either search for unique tests with regard to inputs or outputs (i.e., novelty search) or search for inputs that invalidate some expected proposition regarding the software (i.e., metamorphic testing). Problematically, verifying unique tests induces the oracle problem while an invalidated proposition results in a single test case. In this paper we utilize novelty search and metamorphic testing to discover a broad range of unique test cases that are directly verifiable via a metamorphic relation and invalidate such an expected proposition in fewer generations of an evolutionary algorithm than direct search. We apply this novelty search and metamorphic testing combination to discover errors in identifying the midpoint of a geodesic as a proof-of-concept.

Autonomous Vehicles (AVs) will revolutionize the way people travel by car. However, in order to deploy autonomous vehicles, effective testing techniques are required. The driving quality of an AV should definitely be considered when testing such systems. However, as in other complex systems, determining the outcome of a test in the driving quality on an AV can be extremely complex. To solve this issue, in this paper we explore the application of Quality-of-Service (QoS) aware metamorphic testing to test AVs modeled in MATLAB/Simulink, one of the predominant modeling tools in the market. We first defined a set of QoS measures applied to AVs by considering as input a recent study. With them, we define metamorphic relations. Lastly we assess the approach in an AV modeled in Simulink by using mutation testing. The results suggests that our approach is effective at detecting faults.

We report on Facebook's deployment of MIA (Metamorphic Interaction Automaton). MIA is used to test Facebook's Web Enabled Simulation, built on a web infrastructure of hundreds of millions of lines of code. MIA tackles the twin problems of test flakiness and the unknowable oracle problem. It uses metamorphic testing to automate continuous integration and regression test execution. MIA also plays the role of a test bot, automatically commenting on all relevant changes submitted for code review. It currently uses a suite of over 40 metamorphic test cases. Even at this extreme scale, a non–trivial metamorphic test suite subset yields outcomes within 20 minutes (sufficient for continuous integration and review processes). Furthermore, our offline mode simulation reduces test flakiness from approximately 50% (of all online tests) to 0% (offline). Metamorphic testing has been widely-studied for 22 years. This paper is the first reported deployment into an industrial continuous integration system.

Metamorphic Testing is a software testing paradigm which aims at using necessary properties of a system-under-test, called metamorphic relations, to either check its expected outputs, or to generate new test cases. Metamorphic Testing has been successful to test programs for which a full oracle is not available or to test programs for which there are uncertainties on expected outputs such as learning systems. In this article, we propose Adaptive Metamorphic Testing as a generalization of a simple yet powerful reinforcement learning technique, namely contextual bandits, to select one of the multiple metamorphic relations available for a program. By using contextual bandits, Adaptive Metamorphic Testing learns which metamorphic relations are likely to transform a source test case, such that it has higher chance to discover faults. We present experimental results over two major case studies in machine learning, namely image classification and object detection, and identify weaknesses and robustness boundaries. Adaptive Metamorphic Testing efficiently identifies weaknesses of the tested systems in context of the source test case.

Recent transformation-based approaches to compiler testing look for mismatches between the results of pairs of equivalent programs, where one program is derived from the other by randomly applying semantics-preserving transformations. We present a formulation of transformation-based compiler testing that provides effective test-case reduction almost for free: if transformations are designed to be as small and independent as possible, standard delta debugging can be used to shrink a bug-inducing transformation sequence to a smaller subsequence that still triggers the bug. The bug can then be reported as a delta between an original and minimally-transformed program. Minimized transformation sequences can also be used to heuristically deduplicate a set of bug-inducing tests, recommending manual investigation of those that involve disparate types of transformations and thus may have different root causes. We demonstrate the effectiveness of our approach via a new tool, spirv-fuzz, the first compiler-testing tool for the SPIR-V intermediate representation that underpins the Vulkan GPU programming model.

Metamorphic testing requires the availability of a suitable set of metamorphic relations (MRs) for the application domain of interest. A software testing practitioner interested in using metamorphic testing is thus blocked unless they can devise a suitable set of MRs. In this paper we offer some practical advice on sources of inspiration for MRs, based on our experience building three fuzzing tools based on metamorphic testing: MF++, which supports automated testing of C++11 libraries, C4, which tests concurrency support in C11 compilers, and spirv-fuzz, which aims to find bugs in compilers for the SPIR-V programming language (mainly used in computer graphics). The MRs we have devised have taken inspiration from three main sources: (1) careful study of specification documents related to the libraries and programming languages that these tools target, (2) consultation of prior work and discussion with domain experts, and (3) manual inspection of the results of automated code coverage analysis on the systems under test. We describe these sources of inspiration in detail, giving a range of concrete examples for each. We hope that this experience report will help to inform developers of future metamorphic testing tools as to the steps they can take to discover MRs in their domains of interest.

With the advent of Bigdata analytics or machine learning software, the characteristics of test target programs become more divergent than before. It brings about two issues on the test oracle problems, uncertainties of the testing conditions and unknown correctness criteria. This paper proposes a new testing framework, which is general enough to account for the existing statistical metamorphic testing and is further amenable to adapt itself to the machine learning software testing. The proposed approach is illustrated with an experiment of testing neural network programs.

Understanding which input and output variables are related to each other is important for metamorphic testing, a simple and effective approach for testing scientific software. We report in this paper a quantitative analysis of input/output (I/O) associations based on co-occurrence statistics of the user manual, as well as association rule mining of a user forum, of the Storm Water Management Model (SWMM). The results show a positive correlation of the identified I/O pairs, and further reveal the complementary aspects of the user manual and user forum in supporting scientific software engineering tasks.

Regression is one of the most commonly used statistical techniques. However, testing regression systems is a great challenge because of the absence of test oracle in general. In this paper, we show that Metamorphic Testing is an effective approach to test multiple linear regression systems. In doing so, we identify intrinsic mathematical properties of linear regression, and then propose 11 Metamorphic Relations to be used for testing. Their effectiveness is examined using mutation analysis with a range of different regression programs. We further look at how the testing could be adopted in a more effective way. Our work is applicable to examine the reliability of predictive systems based on regression that has been widely used in economics, engineering and science, as well as of the regression calculation manipulated by statistical users., Comment: 24 pages, 5 figures, 7 tables. The Journal of Systems and Software (2021)

Software testing is often hindered where it is impossible or impractical to determine the correctness of the behaviour or output of the software under test (SUT), a situation known as the oracle problem. An example of an area facing the oracle problem is automatic image classification, using machine learning to classify an input image as one of a set of predefined classes. An approach to software testing that alleviates the oracle problem is metamorphic testing (MT). While traditional software testing examines the correctness of individual test cases, MT instead examines the relations amongst multiple executions of test cases and their outputs. These relations are called metamorphic relations (MRs): if an MR is found to be violated, then a fault must exist in the SUT. This paper examines the problem of classifying images containing visually hidden markers called Artcodes, and applies MT to verify and enhance the trained classifiers. This paper further examines two MRs, Separation and Occlusion, and reports on their capability in verifying the image classification using one-way analysis of variance (ANOVA) in conjunction with three other statistical analysis methods: t-test (for unequal variances), Kruskal-Wallis test, and Dunnett's test. In addition to our previously-studied classifier, that used Random Forests, we introduce a new classifier that uses a support vector machine, and present its MR-augmented version. Experimental evaluations across a number of performance metrics show that the augmented classifiers can achieve better performance than non-augmented classifiers. This paper also analyses how the enhanced performance is obtained., Comment: 32 pages, 11 figures

Recent advances in deep neural networks (DNNs) have led to object detectors (ODs) that can rapidly process pictures or videos, and recognize the objects that they contain. Despite the promising progress by industrial manufacturers such as Amazon and Google in commercializing deep learning-based ODs as a standard computer vision service, ODs --- similar to traditional software --- may still produce incorrect results. These errors, in turn, can lead to severe negative outcomes for the users. For instance, an autonomous driving system that fails to detect pedestrians can cause accidents or even fatalities. However, despite their importance, principled, systematic methods for testing ODs do not yet exist. To fill this critical gap, we introduce the design and realization of MetaOD, a metamorphic testing system specifically designed for ODs to effectively uncover erroneous detection results. To this end, we (1) synthesize natural-looking images by inserting extra object instances into background images, and (2) design metamorphic conditions asserting the equivalence of OD results between the original and synthetic images after excluding the prediction results on the inserted objects. MetaOD is designed as a streamlined workflow that performs object extraction, selection, and insertion. We develop a set of practical techniques to realize an effective workflow, and generate diverse, natural-looking images for testing. Evaluated on four commercial OD services and four pretrained models provided by the TensorFlow API, MetaOD found tens of thousands of detection failures. To further demonstrate the practical usage of MetaOD, we use the synthetic images that cause erroneous detection results to retrain the model. Our results show that the model performance is significantly increased, from an mAP score of 9.3 to an mAP score of 10.5.

Recommender system has been widely used in many Internet-based applications such as e-commerce and content provision. In the existing studies, the focus is mainly on its accuracy, while its robustness and trustworthiness are often ignored. Targeting at the “non-testable” problem of recommender systems, this paper proposes a solution of applying metamorphic testing to validate their robustness. According to the characteristics of most prediction programs in recommender systems, five types of metamorphic relations are proposed. Meanwhile, taking a well-known recommender library LibRec as an example, empirical experiments are conducted to verify the proposed solution and metamorphic relations. The results show that they are effective in revealing the robustness problem of recommender systems, and metamorphic testing can also be regarded as a quality evaluation approach for recommendation algorithms or models.

Machine learning techniques have been successfully employed in various areas and, in particular, for the development of healthcare applications, aiming to support in more effective and faster diagnostics (such as cancer diagnosis). However, machine learning models may present uncertainties and errors. Errors in the training process, classification, and evaluation can generate incorrect results and, consequently, to wrong clinical decisions, reducing the professionals' confidence in the use of such techniques. Similar to other application domains, the quality should be guaranteed to produce more reliable models capable of assisting health professionals in their daily activities. Metamorphic testing can be an interesting option to validate machine learning applications. Using this testing approach is possible to define relationships that define changes to be made in the application's input data to identify faults. This paper presents an experimental study to evaluate the effectiveness of metamorphic testing to validate machine learning applications. A Machine learning application to verify breast cancer diagnostic was developed, using an available dataset composed of 569 samples whose data were taken from breast cancer images, and used as the software under test, in which the metamorphic testing was applied. The results indicate that metamorphic testing can be an alternative to support the validation of machine learning applications.

Generating test cases and further evaluating their “quality” are two critical topics in the area of Deep Neural Networks (DNNs). In this domain, different studies (e.g., [1], [2]) have reported that metamorphic testing (MT) serves as an effective test case generation method, where an initial set of source test cases is augmented with identified metamorphic relations (MRs) to produce the corresponding set of follow-up test cases. As a result, the fault detection effectiveness (and, hence, the “quality”) of the resulting test suite T, containing these source and follow-up test cases, will most likely be increased.

Context: Artificial Intelligence (AI) applications appear as one of the main demands for the software industry nowadays; within this context, speech recognition and acoustic scene detection and classification achieve near-human performance. However, performing systematic testing on these applications is challenging and very costly if we follow traditional testing methodologies. In this scenario, Metamorphic Testing presents an efficient approach to ensuring the quality of machine learning-based systems. Objective: analyze techniques and applications of metamorphic testing and propose metamorphic relations to perform verification and validation of acoustic scene classifiers. Method: the use of Design Science Research to provide iterative and incremental research development, through which the results achieved in the first cycle of research are presented. Results: in the first design cycle, the use of two metamorphic relations focused on attributes and samples permutation were adopted to verify and validate 6 learning algorithms in an acoustic scene classification system, wherein one of the applied relations, on the random forest algorithm, presented a violation, leading to prediction errors and 2.34% drop in its accuracy in one of the tests performed. In the second design cycle, three new relations based on acoustic variations were proposed to validate the audio attributes, where in all of them the ZCR attribute was more effective to deal with the proposed variations. Conclusion: At the end of the two cycles, our approach has revealed verification flaws and has also proven effective for validation purposes of the systems under test, allowing developers of acoustic scene classification systems to apply them to their learning components, audio extraction processes, and to the test and training dataset

Natural language processing (NLP) models have been increasingly used in sensitive application domains including credit scoring, insurance, and loan assessment. Hence, it is critical to know that the decisions made by NLP models are free of unfair bias toward certain subpopulation groups. In this paper, we propose a novel framework employing metamorphic testing, a well-established software testing scheme, to test NLP models and find discriminatory inputs that provoke fairness violations. Furthermore, inspired by recent breakthroughs in the certified robustness of machine learning, we formulate NLP model fairness in a practical setting as (ε, k)-fairness and accordingly smooth the model predictions to mitigate fairness violations. We demonstrate our technique using popular (commercial) NLP models, and successfully flag thousands of discriminatory inputs that can cause fairness violations. We further enhance the evaluated models by adding certified fairness guarantee at a modest cost.

Combined sewer overflows represent significant risks to human health as untreated water is discharged to the environment. Municipalities recently began collecting large amounts of water-related data and considering the adoption of deep learning solutions like recurrent neural network (RNN) for overflow prediction. In this paper, we contribute a novel metamorphic relation to characterize RNN robustness in the presence of missing data. We show how this relation drives automated testing of three implementation variants: LSTM, GRU, and IndRNN thereby uncovering deficiencies and suggesting more robust solutions for overflow prediction.

Metamorphic testing (MT) is getting increasingly popular by exhibiting test effectiveness for a wide range of subjects, from compilers to machine learning programs. However, the central part of MT, i.e., the derivation of useful metamorphic relations (MRs), still falls behind MT's rapid applications. In this paper, we propose M.R. Hunter, an interactive online game for attracting users, especially those who are not familiar with, or even reluctant to, learning intrinsic complexities behind MT, to participate into the MR derivation process in a puzzle-solving way. The game design carefully considers how to guide users to participate actively, how to present conjectured MRs intuitively, and how to validate MRs effectively. So far, we have built and deployed a preliminary version of the game, and received active feedbacks, suggesting both promising results and useful advices for future improvement.

We present the design of MTKeras, a generic metamorphic testing framework for machine learning, and demonstrate its effectiveness through case studies in image classification and sentiment analysis.

Current research on the testing of machine translation software mainly focuses on functional correctness for valid, well-formed inputs. By contrast, robustness testing, which involves the ability of the software to handle erroneous or unanticipated inputs, is often overlooked. In this paper, we propose to address this important shortcoming. Using the metamorphic robustness testing approach, we compare the translations of original inputs with those of follow-up inputs having different categories of minor typos. Our empirical results reveal a lack of robustness in Google Translate, thereby opening a new research direction for the quality assurance of neural machine translators.

Metamorphic relations (MRs) are an essential component of metamorphic testing (MT) that highly affects its fault detection effectiveness. MRs are usually identified with the help of a domain expert, which is a labor-intensive task. In this work, we explore the feasibility of a text classification-based machine learning approach to predict MRs using their program documentation as the sole input. We compare our method to our previously developed graph kernelbased machine learning approach and demonstrate that textual features extracted from program documentation are highly effective for predicting metamorphic relations for matrix calculation programs.