Search

In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intractability of neural machine translation (NMT) models that power modern machine translation make the robustness of these systems difficult to even assess, much less guarantee. Machine translation systems can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation systems is very difficult and has, therefore, been much under-explored. To tackle this challenge, we introduce structure-invariant testing (SIT), a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of "similar" source sentences should typically exhibit similar sentence structures. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words; (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing); (3) reports sentence pairs whose structures differ quantitatively by more than some threshold. To evaluate SIT, we use it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy issues with 69.5% and 70% top-1 accuracy, respectively. The translation errors are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic. [ABSTRACT FROM AUTHOR]

Central Processing Unit (CPU) scheduling is used to allocate CPU for multiple processes. CPU is one of the most important resources in the computer system, and its scheduling is vital and influential in operating systems. Thus, it is necessary to ensure the correctness of the CPU scheduling program. However, testing the correctness of a scheduling program is difficult because it is hard to verify the correctness of its output, which is known as the test oracle problem in software testing. Metamorphic Testing (MT) which has been recently proposed to alleviate the test oracle problem, is applied to test the CPU scheduling program. In this paper, we use MT to test the Highest Response Ratio Next (HRRN) scheduling algorithm. Two simulators of HRRN scheduler are used in the evaluation of our method. Surprisingly, some real life faults in one open source simulator are detected by MT. Further experiments are performed based on mutants, and the experimental results show that MT is an effective strategy to test CPU scheduler. [ABSTRACT FROM PUBLISHER]

The correctness of mission-critical software is an important part of information security and oracle problem[1] is often a great constraint for their testing. Metamorphic testing(MT) is practical for oracle problem, but calls for more executions and only focuses on program's mathematics properties in most situations. This article provides the Path-Combination-Based MT method, which mines the relationships among inputs that could execute different paths and their corresponding outputs based on the analysis of program structure, and then tests the program with these relationships. The experimental results prove its efficiency. [ABSTRACT FROM PUBLISHER]

The existence of an oracle is often assumed in software testing. But in many situations, especially for scientific programs, oracles do not exist or they are too hard to implement. This paper examines three techniques that are used to test programs without oracles: (1) Metamorphic testing, (2) Run-time Assertions and (3) Developing test oracles using machine learning. We examine these methods in terms of their (1) fault finding ability, (2) automation, and (3) required domain knowledge. Several case studies apply these three techniques to effectively test scientific programs that do not have oracles. Certain techniques have reported a better fault finding ability than the others when testing specific programs. Finally, there is potential to increase the level of automation of these techniques, thereby reducing the required level of domain knowledge. Techniques that can potentially be automated include (1) detection of likely metamorphic relations, (2) static analyses to eliminate spurious invariants and (3) structural analyses to develop machine learning generated oracles. [ABSTRACT FROM PUBLISHER]

Much software lacks test oracles, which limits automated testing. Metamorphic testing is one proposed method for automating the testing process for programs without test oracles. Unfortunately, finding the appropriate metamorphic relations required for use in metamorphic testing remains a labor intensive task, which is generally performed by a domain expert or a programmer. In this work we present a novel approach for automatically predicting metamorphic relations using machine learning techniques. Our approach uses a set of features developed using the control flow graph of a function for predicting likely metamorphic relations. We show the effectiveness of our method using a set of real world functions often used in scientific applications. [ABSTRACT FROM PUBLISHER]

A fundamental problem for software testing is the oracle problem, which means that in many practical situations, it is extremely expensive, if not impossible, to verify the test result given any possible program input. Metamorphic testing is an approach to alleviating the oracle problem. The key part of metamorphic testing is a set of necessary properties of the software under test, namely metamorphic relations. Metamorphic relations not only help generate test cases, but also provide a mechanism to partially verify the test results without the need of oracle. In most previous studies, metamorphic relations were identified manually by testers in an ad hoc way. There is no systematic methodology that helps us identify metamorphic relations. In this paper, we propose a simple method, namely, the composition of metamorphic relations, for systematically constructing new metamorphic relations based on the already identified metamorphic relations. We conduct a case study and show that new metamorphic relations can be easily constructed by compositing some existing metamorphic relations. It is also observed that the new metamorphic relations are very likely to deliver a higher cost-effectiveness of metamorphic testing than the original metamorphic relations. [ABSTRACT FROM PUBLISHER]

Photon propagation in biological tissue can be equivalently modeled with Monte Carlo simulations numerically or by the Radiative Transfer Equation (RTE) analytically. However, testing of a Monte Carlo program modeling photon propagation in biological tissue is difficult due to the unknown character of the test oracles. Although approaches based on Beer-Lambert law, van de Hulst's table or RTE can be used for testing the Monte Carlo modeling program, these approaches are only applied to the program that is designed for homogeneous media. A rigorous way for testing the Monte Carlo modeling program for heterogeneous media is needed. In this paper, we investigate the effectiveness of the metamorphic testing approach to test a Monte Carlo modeling program for heterogeneous media. In addition, the metamorphic testing is extended with the evaluation of the adequacy of testing coverage criteria to measure the quality of the metamorphic testing, to guide the creation of metamorphic relations, to generate testing inputs, and to investigate the found exceptions. The effectiveness of the approach has been demonstrated through testing a Monte Carlo modeling program we developed for simulating photon propagation in human skins. [ABSTRACT FROM AUTHOR]

Health care professionals rely on software to simulate anatomical and physiological elements of the human body for purposes of training, prototyping, and decision making. Software can also be used to simulate medical processes and protocols to measure cost effectiveness and resource utilization. Whereas much of the software engineering research into simulation software focuses on validation (determining that the simulation accurately models real-world activity), to date there has been little investigation into the testing of simulation software itself, that is, the ability to effectively search for errors in the implementation. This is particularly challenging because often there is no test oracle to indicate whether the results of the simulation are correct. In this paper, we present an approach to systematically testing simulation software in the absence of test oracles, and evaluate the effectiveness of the technique. [ABSTRACT FROM AUTHOR]

Two of the key challenges in software testing are the automated generation of test cases, and the identification of failures by checking test outputs. Both challenges are effectively addressed by metamorphic testing (MT), a software testing technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the software under test. Two decades after its introduction, MT is becoming a fully-fledged testing paradigm with successful applications in multiple domains including, among others, big data engineering, simulation and modeling, compilers, machine learning programs, autonomous cars and drones, and cybersecurity. This technical briefing will provide an introduction to MT from a double perspective. First, we will present the technique and the results of a novel survey outlining its main trends and lessons learned. Then, we will go deeper and present some of the successful applications of the technique, as well as challenges and opportunities on the topic. The briefing will be complemented with practical exercises on testing real web applications and APIs. [ABSTRACT FROM AUTHOR]