Search

Manual testing is a machine testing method in which testing cases are carried out manually rather than utilizing an automatic process. Both test cases were handled manually by the tester from the perspective of the end consumer. It guarantees that the program performs, regardless of whether or not it is specified in the prerequisite text. In order to complete approximately 100% of the application, test cases are planned and implemented. Manually produced test case reports are common. Since it can spot both visible and hidden program bugs, manual testing is one of the most effective testing techniques. A defect is described by the software as the difference between expected and actual performance. The bugs were corrected and turned over to the tester by the creator. Manual training is expected before automatic testing for any newly created software. This research requires a lot of time and resources, but it ensures the entire program is bug-free. Manual research entails knowledge of manual testing methods but does not require the use of an automated evaluation method. Manual checking is necessary since one of the fundamental software testing concepts is that "100% automation is not feasible." [ABSTRACT FROM AUTHOR]

This article proposes a hybrid test scheme based on interleaved test applications to increase the efficiency of multisite testing of integrated circuits (ICs) by solving problems related to test coverage, which is an important feature in automotive IC, and test costs, including the number of test pins and the volume of test data. To solve the problems, a hybrid test scheme that combines the merits of broadcast-based scan compression and built-in self-test (BIST) is proposed. The proposed scheme not only generates deterministic test patterns, similar to scan compression, but also requires fewer test pins, similar to BIST, thereby increasing the efficiency of multisite testing. With appropriately designed linear-feedback shift registers (LFSRs) and seeds periodically inserted through the reduced test pins, the test application process combines deterministic testing and random testing. The random testing stage can compensate the seed-insertion time for the subsequent deterministic testing stage and reduce the number of required deterministic patterns by detecting easy-to-detect faults. For further reduction of the volume of test data and hardware overhead, some compatible scan chains (SCs) are grouped and receive test data from an LFSR. The experimental results show that the proposed method increases the efficiency of multisite testing with a reduced pin count. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool testRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that testRNN has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, testRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing. [ABSTRACT FROM AUTHOR]

In an era of connectivity and automation, the vehicle industry is adopting numerous technologies to transform driver-centric vehicles into intelligent mechanical devices driven by software components. Software integration and network connectivity inherit numerous security issues that open the door for malicious attacks. Software security testing is a scalable and practical approach to identify systems’ weaknesses and vulnerabilities at an early stage and throughout their life-cycle. Security specialists recommend fuzz testing to identify vulnerabilities within vehicle software systems. Nevertheless, the randomness and blindness of fuzzing hinder it from becoming a reliable security tool. This article presents a vulnerability-oriented fuzz (VulFuzz) testing framework that utilizes security vulnerability metrics designed particularly for connected and autonomous vehicles to direct and prioritize the fuzz testing toward the most vulnerable components. While most gray-box fuzzing techniques aim solely to expand code coverage, the proposed approach assigns weights to ensure a thorough examination of the most vulnerable components. Moreover, we employ an input structure-aware mutation technique that can bypass vehicle software systems’ input formats to boost test performance and avoid dropped test cases. Such a testing technique will contribute to the quality assurance of vehicle software engineering. We implemented the proposed approach on OpenPilot, a driver assistance system, and compared our results to American fuzzy lop (AFL) and an unguided mutation-based fuzzer. Within 16.8 h, VulFuzz exposed 335 crashes, 41 times more than AFL and two times more than an unguided mutation-based fuzzer. VulFuzz is explicitly efficient for automotive systems, reaching the same code coverage as AFL but with more exposed crashes and fewer dropped messages. [ABSTRACT FROM AUTHOR]

Symbolic execution is a powerful technique for automating software testing to detect many types of errors such as memory errors and assertion violations. However, it encounters the problem of path explosion, and by using only assertions, it still lacks the capability of going deep into checking the functional correctness of a path based on corresponding formal specifications. To address these problems, we propose a specification-based incremental testing method with symbolic execution, called SIT-SE, providing a much more rigorous way to automatically check the functional correctness of all the discovered program paths, by introducing theorems (instead of assertions) for path correctness and branch sequence coverage algorithm for guiding a moderate path exploration. Compared with Hoare logic for proving the correctness of an entire program, a theorem in the SIT-SE is made for verifying the correctness of a program path. The proposed method carefully treats the relationship between a path condition and the specification in a theorem to restrict the monotonous path exploration, whereas traditional concolic testing methods roughly use one test data to determine the path correctness by assertions during long path searching. We use a classic case to demonstrate how the method works and conduct an experiment to evaluate the performance of both the proposed method and the commonly used well-known concolic testing tool KLEE. The experimental results show that our method SIT-SE is effective and outperforms KLEE in detecting faulty paths based on specifications. [ABSTRACT FROM AUTHOR]

Though Deep Neural Networks (DNNs) have been widely deployed and achieved great success in many domains, they have severe safety and reliability concerns. To provide testing evidence for DNNs' reliable behaviors, various coverage testing techniques inspired by traditional software testing have been proposed. However, the coverage criteria in these techniques are either not fine enough to capture subtle behaviors of DNNs, or too time-consuming to be applied on large-scale DNNs. In this paper, we propose a coverage testing framework named HashC, which makes mainstream coverage criteria (e.g., NC and KMNC) much finer. Meanwhile, HashC reduces the time complexity of combinatorial coverage testing from polynomial time to linear time. We also develop the corresponding test sample selection method. Our experiments show that, (1) the existing mainstream coverage criteria are becoming finer after being equipped with HashC, (2) HashC greatly accelerates combinatorial coverage testing and can handle the testing of large-scale DNNs. [ABSTRACT FROM AUTHOR]

Spectrum-based fault localization (SBFL) techniques can automatically localize software faults. They employ the program spectrum, such as code coverage profile with test verdicts, to rank the program entities based on their code suspiciousness. In the past decades, researchers have proposed many approaches to optimize these techniques; however, the program structure, which can influence their performance, is not taken into consideration in developing and improving these techniques. In this article, we identify and analyze the effect of the program structure on the application of SBFL techniques. We observe that some specific program structures may introduce structure bias to code suspiciousness and negatively influence the output of SBFL techniques. To mitigate these effects and improve the performance of fault localization, we propose Delta4Ts, a structure-aware technique. Delta4Ts references debugging history to alleviate the impact of structure bias in the calculation of code suspiciousness. It reasons from the observable suspicious value towards the desired suspicious value and the impact of structure bias. To evaluate Delta4Ts under practical constraints, we conduct a controlled experiment using nine widely-studied SBFL formulae on 12 C programs and 6 Java programs. The experiment results show that Delta4Ts can significantly improve the accuracy of the studied SBFL formulae by an average of 34.8% on 12 C programs and 30.6% on 6 Java programs, and improve more on subject programs associated with more history versions or having larger code sizes. [ABSTRACT FROM AUTHOR]

Test case prioritization (TCP) is a popular regression testing technique in software engineering field. The task of TCP is to schedule the execution order of test cases so that certain objective (e.g., code coverage) can be achieved quickly. In this article, we propose an efficient ant colony system framework for the TCP problem, with the aim of maximizing the code coverage as soon as possible. In the proposed framework, an effective heuristic function is proposed to guide the ants to construct solutions based on additional statement coverage among remaining test cases. Besides, a sorting-based local search mechanism is proposed to further accelerate the convergence speed of the algorithm. Experimental results on different benchmark problems, and a real-world application, have shown that the proposed framework can outperform several state-of-the-art methods, in terms of solution quality and search efficiency. [ABSTRACT FROM AUTHOR]

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction. [ABSTRACT FROM AUTHOR]

Software failures caused by data race bugs have always been major concerns in parallel and distributed systems, despite significant efforts spent in software testing. Due to their nondeterministic and hard-to-reproduce features, when evaluating systems’ operational reliability, a rather long period of experimental execution time is expected to be spent on observing failures caused by data race conditions. To address this problem, in this paper, we make two contributions. First, this paper proposes stress testing with influencing factors, in which the system runs under certain workloads for a long time with controlled stress conditions to accelerate the occurrence of data race failures. Second, it explores and formulates mathematical relationship models between data races’ statistical characteristics of time to failure (TTF) or mean TTF (MTTF) and the influencing factors. Such relationship models are used for TTF/MTTF extrapolation under different operational conditions and are essential to reduce systems’ reliability evaluation time. The proposed method is empirically evaluated on six applications suffering from failures caused by real-world data race bugs. Through analysis of the experimental results, we obtain several important findings: First, the reduction in the manifestation time to data race failures achieved by controlling the influencing factors is statistically significant. Second, Power model is the best-fitting model of the relationship between the MTTF and the influencing factors. Third, Power Weibull distribution is the best-fitting probability distribution between the TTF and the influencing factors. Finally, the TTF/MTTF can be accurately estimated with the approach proposed in this paper. [ABSTRACT FROM AUTHOR]

Adaptive random testing (ART) was developed as an enhanced version of random testing to increase the effectiveness of detecting failures in programs by spreading the test cases evenly over the input space. However, heavy computation may be incurred. In this paper, three enhanced algorithms for fixed-size-candidate-set ART (FSCS-ART) are proposed based on the $k$ -dimensional tree (KD-tree) structure. The first algorithm Naive-KDFC constructs a KD-tree by splitting the input space with respect to every dimension successively in a round-robin fashion. The second algorithm SemiBal-KDFC improves the balance of the KD-tree by prioritizing the splitting according to the spread in each dimension. In order to control the number of traversed nodes in backtracking, the third algorithm LimBal-KDFC introduces an upper bound for the nodes involved. Simulation and empirical studies have been conducted to investigate the efficiency and effectiveness of the three algorithms. The experimental results show that these algorithms significantly reduce the computation time of the original FSCS-ART for low dimensions and for the case of high dimensions with low failure rates. The efficiency of SemiBal-KDFC is better than that of Naive-KDFC when the dimension is no more than 8, but LimBal-KDFC is the most efficient of all three. Although the limited backtracking leads only to an approximate nearest neighbor in LimBal-KDFC, its failure-detection effectiveness is, in fact, better than FSCS-ART in high-dimensional input spaces and has no significant deterioration in low-dimensional spaces. [ABSTRACT FROM AUTHOR]

Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing, where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing, to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing. [ABSTRACT FROM AUTHOR]

New processing models are being adopted in Big Data engineering to overcome the limitations of traditional technology. Among them, MapReduce stands out by allowing for the processing of large volumes of data over a distributed infrastructure that can change during runtime. The developer only designs the functionality of the program and its execution is managed by a distributed system. As a consequence, a program can behave differently at each execution because it is automatically adapted to the resources available at each moment. Therefore, when the program has a design fault, this could be revealed in some executions and masked in others. However, during testing, these faults are usually masked because the test infrastructure is stable, and they are only revealed in production because the environment is more aggressive with infrastructure failures, among other reasons. This paper proposes new testing techniques that aimed to detect these design faults by simulating different infrastructure configurations. The testing techniques generate a representative set of infrastructure configurations that as whole are more likely to reveal failures using random testing, and partition testing together with combinatorial testing. The techniques are automated by using a test execution engine called MRTest that is able to detect these faults using only the test input data, regardless of the expected output. Our empirical evaluation shows that MRTest can automatically detect these design faults within a reasonable time. [ABSTRACT FROM AUTHOR]

During the unit testing of model-driven tools, a large number of models and test classes needs to be managed and maintained. Typically, some of these artifacts are specified manually, some are generated automatically. Existing approaches to test management rely on the available visual and textual modeling notations. As these notations are not tailored to unit testing, distinct maintainability trade-offs arise. In this paper, we propose a notation that aims to combine the benefits of visual and text-based approaches. The notation is at the same time visual and text-based, as it uses ASCII characters to emulate the familiar graphical notations. In our evaluation based on real models, we identify problematic model shapes challenging the scalability our notation, while finding that it is well-suited to capture typical test models. [ABSTRACT FROM AUTHOR]

Previous research in defect prediction has proposed approaches to determine which files require additional testing resources. However, practitioners typically create tests at a higher level of abstraction, which may span across many files. In this paper, we study software testing, especially test resource prioritization, from a different perspective. We use topic models to generate topics that provide a high-level view of a system, allowing developers to look at the test case coverage from a different angle. We propose measures of how well tested and defect prone a topic is, allowing us to discover which topics are well tested and which are defect prone. We conduct case studies on the histories of Mylyn, Eclipse, and NetBeans. We find that 34–78% of topics are shared between source code and test files, indicating that we can use topic models to study testing; well-tested topics are usually less defect prone, defect-prone topics are usually undertested; we can predict which topics are defect prone but not well tested with an average precision and recall of 75% and 77%, respectively; our approach complements traditional prediction-based approaches by saving testing and code inspection effort; and our approach is not particularly sensitive to the parameters that we use. [ABSTRACT FROM PUBLISHER]

Software testing enables development teams to maintain the quality of a software system while it evolves. The database manipulation code requires special attention in this context. However, it is often neglected and suffers from software maintenance problems. In this paper, we study the current state-of-the-practice in testing database manipulation code. We first analysed the tests of 72 open-source projects to gain insight into the coverage of database access code. The database was poorly tested: 46% of the projects did not cover with tests half of their database access methods, and 33% did not cover the database code at all. This poor coverage motivated us to study developers' challenges and best practices. (i) First, we analysed 532 questions on Stack Exchange sites and deduced a taxonomy of issues. Developers mostly looked for general best practices to test database access code. Their technical questions were related to database management, mocking, parallelisation, or framework/tool usage. (ii) Next, we examined the answers to these questions. We manually labelled 598 answers to 255 questions. We distinguished 363 solutions and organised them in a taxonomy of best practices. Most of the suggestions considered the testing environment and recommended various tools or configurations. The second largest category was database management, where many addressed database initialisation and clean-up between tests. Other categories pertained to code structure or design, concepts, performance, processes, test characteristics, test code, and mocking. We illustrate the two taxonomies through intriguing examples. • A study on the coverage of database access code in the tests of open-source projects. • A qualitative assessment of developers' issues when testing database access code. • An investigation of developers' best practices when testing database access code. • Taxonomies of issues and best practices when testing database access code. • The full dataset of the studies is publicly available as an online appendix. [ABSTRACT FROM AUTHOR]

To ensure that an access control (AC) system is safe, there must be a reliable means to verify that the specified AC policy model conforms to the safety requirements and policy author's intentions. A general verification approach includes black-box and white-box testing, as well as the generation of sufficient test cases to check the correctness of model implementations. [ABSTRACT FROM AUTHOR]

Metamorphic testing (MT) can enhance security testing by providing an alternative to using a test oracle, which is often unavailable or impractical. The authors report how MT detected previously unknown bugs in real-world critical applications such as code obfuscators, giving evidence that software testing requires diverse perspectives to achieve greater cybersecurity. [ABSTRACT FROM AUTHOR]

Random testing is a fundamental testing technique that can be used to generate test cases for both hardware and software systems. Quasi-random testing was proposed as an enhancement to the cost-effectiveness of random testing: In addition to having similar computation overheads to random testing, it makes use of quasi-random sequences to generate low-discrepancy and low-dispersion test cases that help deliver high failure-detection effectiveness. Currently, few algorithms exist to generate quasi-random sequences, and these are mostly deterministic, rather than random. A previous study of quasi-random testing has examined two methods for randomizing quasi-random sequences to improve their applicability in testing. However, these randomization methods still have shortcomings—one method does not introduce much randomness to the test cases, while the other does not support incremental test case generation. In this paper, we present an innovative approach to incrementally randomizing quasi-random sequences. The test cases generated by this new approach show a high degree of randomness and evenness in distribution. We also conduct simulations and empirical studies to demonstrate the applicability and effectiveness of our approach in software testing. [ABSTRACT FROM PUBLISHER]

We show how to write concise and readable specifications of linked data structures that are applicable for both formal deductive verification and testing. A singly linked list and a binary search tree are provided as examples. The main characteristic of the specifications is the use of observer methods, in particular to express reachability of elements in a data structure. The specifications are written in the Java Modeling Language (JML) and do not require extensions of that language. This paper addresses a mixed audience of users and developers in the fields of formal verification, testing, and specification language design. We provide an in-depth description of the proposed specifications and analyze the implications both for verification and testing. Based on this analysis we have developed verification techniques that are implemented in the deductive verification tool KeY and enable fully automatic verification of the linked list example. This techniques are also described in this paper. [ABSTRACT FROM AUTHOR]

Model-based testing employs models for testing. Model-based mutation testing (MBMT) additionally involves fault models, called mutants, by applying mutation operators to the original model. A problem encountered with MBMT is the elimination of equivalent mutants and multiple mutants modeling the same faults. Another problem is the need to compare a mutant to the original model for test generation. This paper proposes an event-based approach to MBMT that is not fixed on single events and a single model but rather operates on sequences of events of length k ≥ 1 and invokes a sequence of models that are derived from the original one by varying its morphology based on k. The approach employs formal grammars, related mutation operators, and algorithms to generate test cases, enabling the following: (1) the exclusion of equivalent mutants and multiple mutants; (2) the generation of a test case in linear time to kill a selected mutant without comparing it to the original model; (3) the analysis of morphologically different models enabling the systematic generation of mutants, thereby extending the set of fault models studied in related literature. Three case studies validate the approach and analyze its characteristics in comparison to random testing and another MBMT approach. [ABSTRACT FROM PUBLISHER]

Testing is a key activity of software development and maintenance that determines the level of reliability. Traceability is the ability to describe and follow the life of software artifacts, and has been promoted as a means for supporting various activities, most importantly testing. Traceability information facilitates the testing and debugging of complex software by modeling the dependencies between code and tests. Actively supplementing traceability to testing enables rectifying defects more reliably and efficiently. Despite its importance, the development of test-to-code traceability has not been sufficiently addressed in the literature, and even worse there is currently no organized review of traceability studies in this field. In this work, we have investigated the main conferences, workshops, and journals of the requirements engineering, testing, and reliability, and identified those contributions that refer to traceability topics. From that starting point, we characterized and analyzed the chosen contributions against three research questions by utilizing a comparative framework including nine criteria. As a result, our study arrives to some interesting points, and outlines a number of potential research directions. This, in turn, can pave the way for facilitating and empowering traceability research in this domain to assist software engineers and testers in test management. [ABSTRACT FROM AUTHOR]

In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC's characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer's Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution. [ABSTRACT FROM AUTHOR]

Transaction-level modeling (TLM) has become the de-facto reference modeling style for system-level design and verification of embedded systems. It allows designers to implement high-level communication protocols for simulations up to 1000 \times faster than at register-transfer level (RTL). To guarantee interoperability between TLM IP suppliers and users, designers implement the TLM communication protocols by relying on a reference standard, such as the standard OSCI for SystemC TLM. Functional correctness of such protocols as well as their compliance to the reference TLM standard are usually verified through user-defined testbenches, whose high quality and completeness play a key role for an efficient TLM design and verification flow. This article presents a methodology to apply mutation analysis, a technique applied in literature for SW testing, for measuring the testbench quality in verifying TLM protocols. In particular, the methodology aims at (i) qualifying the testbenches by considering both the TLM protocol correctness and their compliance to a defined standard (i.e., OSCI TLM), (ii) optimizing the simulation time during mutation analysis by avoiding mutation redundancies, and (iii) driving the designers in the testbench improvement. Experimental results on benchmarks of different complexity and architectural characteristics are reported to analyze the methodology applicability. [ABSTRACT FROM AUTHOR]

Context. The equivalent mutant problem (EMP) is one of the crucial problems in mutation testing widely studied over decades. Objectives. The objectives are: to present a systematic literature review (SLR) in the field of EMP; to identify, classify and improve the existing, or implement new, methods which try to overcome EMP and evaluate them. Method. We performed SLR based on the search of digital libraries. We implemented four second order mutation (SOM) strategies, in addition to first order mutation (FOM), and compared them from different perspectives. Results. Our SLR identified 17 relevant techniques (in 22 articles) and three categories of techniques: detecting (DEM); suggesting (SEM); and avoiding equivalent mutant generation (AEMG). The experiment indicated that SOM in general and JudyDiffOp strategy in particular provide the best results in the following areas: total number of mutants generated; the association between the type of mutation strategy and whether the generated mutants were equivalent or not; the number of not killed mutants; mutation testing time; time needed for manual classification. Conclusions . The results in the DEM category are still far from perfect. Thus, the SEM and AEMG categories have been developed. The JudyDiffOp algorithm achieved good results in many areas. [ABSTRACT FROM PUBLISHER]

This paper introduces a new metric to characterize test sets in terms of their diagnostic power. Our method uses much less space compared to the existing ones and is quite accurate. The metric can be utilized to increase the diagnosability of incompletely specified test sets via don't care filling. The X-filling approach can be integrated with test pattern generation tools to aid in better diagnostic pattern set generation. [ABSTRACT FROM PUBLISHER]

Background: Cross-company defect prediction (CCDP) is a field of study where an organization lacking enough local data can use data from other organizations for building defect predictors. To support CCDP, data must be shared. Such shared data must be privatized, but that privatization could severely damage the utility of the data. Aim: To enable effective defect prediction from shared data while preserving privacy. Method: We explore privatization algorithms that maintain class boundaries in a dataset. CLIFF is an instance pruner that deletes irrelevant examples. MORPH is a data mutator that moves the data a random distance, taking care not to cross class boundaries. CLIFF+MORPH are tested in a CCDP study among 10 defect datasets from the PROMISE data repository. Results: We find: 1) The CLIFFed+MORPHed algorithms provide more privacy than the state-of-the-art privacy algorithms; 2) in terms of utility measured by defect prediction, we find that CLIFF+MORPH performs significantly better. Conclusions: For the OO defect data studied here, data can be privatized and shared without a significant degradation in utility. To the best of our knowledge, this is the first published result where privatization does not compromise defect prediction. [ABSTRACT FROM AUTHOR]

Testing of a software product is the key activity before deploying it in the real-time environment. Therefore testing should be introduced into the product development process as early as possible in order to decrease the costs of repairing the damage in later phases. In this paper we use Coloured Petri Nets (CPN) for the system specification and also as a system to be tested instead of its implementation. This specification also serves as the basis for the test suite generation. Test cases are provided in the language of Testing and Test Control Notation version 3 (TTCN-3) due to its general application area and platform independence. Stop-and-wait communication protocol has been used as a tutorial example of our approach. [ABSTRACT FROM AUTHOR]

Modern vehicles have evolved to support connected and self-driving capabilities. The concepts such as connected driving, cooperative driving, and intelligent transportation systems have resulted in an increase in the connectivity of vehicles and subsequently created new information security risks. The original vehicular ad-hoc network term is now emerged to a new term, Internet of Vehicles (IoV), which is a typical application of symmetry of Internet of Things (IoT). Vehicle manufacturers address some critical issues such as software bugs or security issues through remote updates, and this gives rise to concerns regarding the security of updated components. Moreover, aftermarket units such as those imposed by transportation authorities or insurance companies expose vehicles to high risk. Software testing aims to ensure that software products are reliable and behave as expected. Many commercial and open-source software products undergo formal certifications to increase users' confidence in their accuracy, reliability, and security. There are different techniques for software certification, including test-based certification. Testcase repositories are available to support software testing and certification, such as the Linux Test Project for Linux kernel testing. Previous studies performed various testing and experimental evaluation of different parts of modern vehicles to assess the security risks. Due to the lack of trusted testcase repositories and a common approach for testing, testing efforts are performed individually. In this paper, we propose a blockchain-based approach for a testcase repository to support test-based software and security testing and overcome the lack of trusted testcase repositories. The novel concept Proof-of-Validation to manage global state is proposed to manage updates to the repository. The initial work in this study considers the LTP test suite as a use case for the testcase repository. This research work is expected to contribute to the further development in including evidence generation for testing verification. [ABSTRACT FROM AUTHOR]

Testing provides a primary means for assuring software in safety-critical systems. To demonstrate, particularly to a certification authority, that sufficient testing has been performed, it is necessary to achieve the test coverage levels recommended or mandated by safety standards and industry guidelines. Mutation testing provides an alternative or complementary method of measuring test sufficiency, but has not been widely adopted in the safety-critical industry. In this study, we provide an empirical evaluation of the application of mutation testing to airborne software systems which have already satisfied the coverage requirements for certification. Specifically, we apply mutation testing to safety-critical software developed using high-integrity subsets of C and Ada, identify the most effective mutant types, and analyze the root causes of failures in test cases. Our findings show how mutation testing could be effective where traditional structural coverage analysis and manual peer review have failed. They also show that several testing issues have origins beyond the test activity, and this suggests improvements to the requirements definition and coding process. Our study also examines the relationship between program characteristics and mutation survival and considers how program size can provide a means for targeting test areas most likely to have dormant faults. Industry feedback is also provided, particularly on how mutation testing can be integrated into a typical verification life cycle of airborne software. [ABSTRACT FROM PUBLISHER]

This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity. The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity. However, subgroup analysis has found both the quality improvement and the productivity drop to be much larger in industrial studies in comparison with academic studies. A larger drop of productivity was found in studies where the difference in test effort between the TDD and the control group's process was significant. A larger improvement in quality was also found in the academic studies when the difference in test effort is substantial; however, no conclusion could be derived regarding the industrial studies due to the lack of data. Finally, the influence of developer experience and task size as moderator variables was investigated, and a statistically significant positive correlation was found between task size and the magnitude of the improvement in quality. [ABSTRACT FROM PUBLISHER]

Random testing is a basic software testing technique that can be used to assess the software reliability as well as to detect software failures. Adaptive random testing has been proposed to enhance the failure-detection capability of random testing. Previous studies have shown that adaptive random testing can use fewer test cases than random testing to detect the first software failure. In this paper, we evaluate and compare the performance of adaptive random testing and random testing from another perspective, that of code coverage. As shown in various investigations, a higher code coverage not only brings a higher failure-detection capability, but also improves the effectiveness of software reliability estimation. We conduct a series of experiments based on two categories of code coverage criteria: structure-based coverage, and fault-based coverage. Adaptive random testing can achieve higher code coverage than random testing with the same number of test cases. Our experimental results imply that, in addition to having a better failure-detection capability than random testing, adaptive random testing also delivers a higher effectiveness in assessing software reliability, and a higher confidence in the reliability of the software under test even when no failure is detected. [ABSTRACT FROM PUBLISHER]

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image annotations and gene function analysis. Current research on multilabel classification focuses on supervised settings which assume existence of large amounts of labeled training data. However, in many applications, the labeling of multilabeled data is extremely expensive and time consuming, while there are often abundant unlabeled data available. In this paper, we study the problem of transductive multilabel learning and propose a novel solution, called Trasductive Multilabel Classification (TraM), to effectively assign a set of multiple labels to each instance. Different from supervised multilabel learning methods, we estimate the label sets of the unlabeled instances effectively by utilizing the information from both labeled and unlabeled data. We first formulate the transductive multilabel learning as an optimization problem of estimating label concept compositions. Then, we derive a closed-form solution to this optimization problem and propose an effective algorithm to assign label sets to the unlabeled instances. Empirical studies on several real-world multilabel learning tasks demonstrate that our TraM method can effectively boost the performance of multilabel classification by using both labeled and unlabeled data. [ABSTRACT FROM AUTHOR]

Test case prioritization is the process of ordering the execution of test cases to achieve a certain goal, such as increasing the rate of fault detection. Increasing the rate of fault detection can provide earlier feedback to system developers, improving fault fixing activity and, ultimately, software delivery. Many existing test case prioritization techniques consider that tests can be run in any order. However, due to functional dependencies that may exist between some test cases—that is, one test case must be executed before another—this is often not the case. In this paper, we present a family of test case prioritization techniques that use the dependency information from a test suite to prioritize that test suite. The nature of the techniques preserves the dependencies in the test ordering. The hypothesis of this work is that dependencies between tests are representative of interactions in the system under test, and executing complex interactions earlier is likely to increase the fault detection rate, compared to arbitrary test orderings. Empirical evaluations on six systems built toward industry use demonstrate that these techniques increase the rate of fault detection compared to the rates achieved by the untreated order, random orders, and test suites ordered using existing "coarse-grained” techniques based on function coverage. [ABSTRACT FROM AUTHOR]

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes. [ABSTRACT FROM AUTHOR]

Testing is a destructive task in which the goal is to find relevant defects as early as possible. It requires automation to reduce cost and ensure high regression, thus delivering determined quality. This article reviews technologies for test automation. [ABSTRACT FROM PUBLISHER]

Dynamic software updating (DSU) systems patch programs on the fly without incurring downtime. To avoid failures due to the updating process itself, many DSU systems employ timing restrictions. However, timing restrictions are theoretically imperfect, and their practical effectiveness is an open question. This paper presents the first significant empirical evaluation of three popular timing restrictions: activeness safety (AS), which prevents updates to active functions, con-freeness safety (CFS), which only allows modifications to active functions when doing so is provably type-safe, and manual identification of the event-handling loops during which an update may occur. We evaluated these timing restrictions using a series of DSU patches to three programs: OpenSSH, vsftpd, and ngIRCd. We systematically applied updates at each distinct update point reached during execution of a suite of system tests for these programs to determine which updates pass and which fail. We found that all three timing restrictions prevented most failures, but only manual identification allowed none. Further, although CFS and AS allowed many more update points, manual identification still supported updates with minimal delay. Finally, we found that manual identification required the least developer effort. Overall, we conclude that manual identification is most effective. [ABSTRACT FROM AUTHOR]

The growing dependence of society on software systems places a high premium on their reliable operation. Moreover, the stringent reliability expectations imposed on these systems must be achieved despite their increasing size and complexity, and decreasing resources available for their development and maintenance. To mitigate these dual challenges, a systematic approach to guide the allocation of resources to the components of a software system is necessary. This paper presents an optimization framework which considers the contribution of each component to system reliability to determine the amount of effort to be allocated to each component, towards the ultimate objective of achieving the specified system reliability target with minimal effort. We assume that the contribution of a component to system reliability is governed by two factors: the system architecture, and the effort-reliability relationship of the component. This characterization is referred to as “architecture-based optimization” because it considers the system architecture explicitly in the effort allocation process. It is demonstrated that the architecture-based optimization framework outperforms other effort allocation strategies, including equal component weighting, and expert opinion. [ABSTRACT FROM AUTHOR]

Abstract: When computational methods are developed, the efficiency of the novel methods should be compared to the existing ones. This can be done using, e.g., analytical methods and benchmark test patterns. In addition, the comparison of the best and the worst case performance is commonly of interest. In this paper, methodologies of genetic algorithm based software testing are adopted to the comparative computational testing of three varieties of dynamic two-dimensional nearest point algorithms. The extreme performances of the algorithms are searched for by optimizing the shape of two-dimensional Gaussian distributions, from which the test patterns are drawn. In particular, an approach to pairwise comparisons of computational complexities of algorithms is proposed. The test case algorithms can be sorted with respect to their computational complexity by the proposed method. [Copyright &y& Elsevier]

Software testing is important, but many of us don't do it because it is frustrating and boring. test that is a new testing framework for R that is easy learn and use, and integrates with your existing workflow. This paper shows how, with illustrations from existing packages. [ABSTRACT FROM AUTHOR]

This paper describes the authors' experience of teaching agile software development to students of computer science, software engineering, and other related disciplines, and comments on the implications of this and the lessons learned. It is based on the authors' eight years of experience in teaching agile software methodologies to various groups of students at different universities, in different cultural settings, and in a number of courses and seminars. It specifically discusses three different courses on agile software development, given in different teaching settings and at different levels, and briefly surveys variations to these courses given elsewhere. Based on the experience acquired, analyses and evaluations conducted, and current pedagogical trends at relevant university departments, the authors provide recommendations on how to overcome potential problems in teaching agile software development and make their adoption more effective. [ABSTRACT FROM AUTHOR]

Abstract: Models are abstractions of reality that have predetermined limits (often not consciously thought through) on what problem domains the models can be used to explore. These limits are determined by the range of observed data used to construct and validate the model. However, it is important to remember that operating the model beyond these limits, one of the reasons for building the model in the first place, potentially brings unwanted behaviour and thus reduces the usefulness of the model. Our experience with the Agricultural Production Systems Simulator (APSIM), a farming systems model, has led us to adapt techniques from the disciplines of modelling and software development to create a model development process. This process is simple, easy to follow, and brings a much higher level of stability to the development effort, which then delivers a much more useful model. A major part of the process relies on having a range of detailed model tests (unit, simulation, sensibility, validation) that exercise a model at various levels (sub-model, model and simulation). To underline the usefulness of testing, we examine several case studies where simulated output can be compared with simple relationships. For example, output is compared with crop water use efficiency relationships gleaned from the literature to check that the model reproduces the expected function. Similarly, another case study attempts to reproduce generalised hydrological relationships found in the literature. This paper then describes a simple model development process (using version control, automated testing and differencing tools), that will enhance the reliability and usefulness of a model. [ABSTRACT FROM AUTHOR]

With the growing size and complexity of software applications, traditional software reliability methods are insufficient to analyze inter-component interactions of modular software systems. The number of test cases may be extremely large for this application; therefore, it is hard for us to extensively test each software component given resource limitations. In this paper, we propose an adaptive framework of incorporating path testing into reliability estimation for modular software systems. Three estimated methods based on common program structures, namely, sequence, branch, and loop structures, are proposed to calculate the path reliability. Consequently, the derived path reliabilities can be applied to the estimates of software reliability. Some experiments are performed based on two real systems. In addition, the accuracy and correlation with respect to the experiments are investigated by simulation and sensitivity analysis. Experimental results show that the path reliability has a high correlation to the actual software reliability. For software with loop structures, a smaller loop number can be assigned to derive an acceptable estimation of path reliability. Further, the sensitivity analysis can be used to identify critical modules and paths for resource allocation. It can be concluded that the proposed methods are useful and helpful for estimating software reliability and can be adaptively used in the early stages of software development. [ABSTRACT FROM AUTHOR]

In this paper, we propose two general frameworks for deriving several software reliability growth models based on a non-homogeneous Poisson process (NHPP) in the presence of imperfect debugging and error generation. The proposed models are initially formulated for the case when there is no differentiation between failure observation and fault removal testing processes, and then extended for the case when there is a clear differentiation between failure observation and fault removal testing processes. During the last three decades, many software reliability growth models (SRGM) have been developed to describe software failures as a random process, and can be used to evaluate development status during testing. With SRGM, software engineers can easily measure (or forecast) the software reliability (or quality), and plot software reliability growth charts. It is not easy to select the best model from a plethora of models available. There are few SRGM in the literature of software engineering that differentiates between failure observation and fault removal processes. In real software development environments, the number of failures observed need not be the same as the number of faults removed. Due to the complexity of software systems, and an incomplete understanding of software, the testing team may not be able to remove the fault perfectly on observation of a failure, and the original fault may remain, resulting in a phenomenon known as imperfect debugging, or get replaced by another fault causing error generation. In the case of imperfect debugging, the fault content of the software remains the same; while in the case of error generation, the fault content increases as the testing progresses. Removal of observed faults may result in the introduction of new faults. [ABSTRACT FROM AUTHOR]

Software testing is an important issue in software engineering. As software systems become increasingly large and complex, the problem of how to optimally allocate the limited testing resource during the testing phase has become more important, and difficult. Traditional Optimal Testing Resource Allocation Problems (OTRAPs) involve seeking an optimal allocation of a limited amount of testing resource to a number of activities with respect to some objectives (e.g., reliability, or cost). We suggest solving OTRAPs with Multi-Objective Evolutionary Algorithms (MOEAs). Specifically, we formulate OTRAPs as two types of multi-objective problems. First, we consider the reliability of the system and the testing cost as two objectives. Second, the total testing resource consumed is also taken into account as the third objective. The advantages of MOEAs over state-of-the-art single objective approaches to OTRAPs will be shown through empirical studies. Our study has revealed that a well-known MOEA, namely Nondominated Sorting Genetic Algorithm II (NSGA-II), performs well on the first problem formulation, but fails on the second one. Hence, a Harmonic Distance Based Multi-Objective Evolutionary Algorithm (HaD-MOEA) is proposed and evaluated in this paper. Comprehensive experimental studies on both parallel-series, and star-structure modular software systems have shown the superiority of HaD-MOEA over NSGA-II for OTRAPs. [ABSTRACT FROM AUTHOR]

Recent advances in mechanical techniques for systematic testing have increased our ability to automatically find subtle bugs, and hence, to deploy more dependable software. This paper builds on one such systematic technique, scope-bounded testing, to develop a novel specification-based approach for efficiently generating tests for products in a software product line. Given properties of features as first-order logic formulas in Alloy, our approach uses SAT-based analysis to automatically generate test inputs for each product in a product line. To ensure soundness of generation, we introduce an automatic technique for mapping a formula that specifies a feature into a transformation that defines incremental refinement of test suites. Our experimental results using different data structure product lines show that an incremental approach can provide an order of magnitude speedup over conventional techniques. We also present a further optimization using dedicated integer constraint solvers for feature properties that introduce integer constraints, and show how to use a combination of solvers in tandem for solving Alloy formulas. [ABSTRACT FROM AUTHOR]

The difficulty of developing wireless sensor systems is widely recognized. Problems associated with testing, debugging, and profiling are key contributing factors. While network simulators have proven useful, they are unable to capture the subtleties of underlying hardware, nor the dynamics of wireless signal propagation and interference; and physical experimentation remains a necessity. To this end, developers increasingly rely on shared deployments exposed for physical experimentation. Sensor network testbeds are under development across the world. We present a complementary testbed architecture that derives its novelty from three characteristics. First, the system is interactive; users can profile source and network level components across a network in real time, as well as inject transient state faults and external network traffic. Second, the system is source-centric; it enables automated source code analysis, instrumentation, and compilation. Finally, the design is open; developers can extend the set of exposed inter faces as appropriate to particular projects without modifying the underlying middleware. We present the testbed design and implementation, a graphical user interface, a shell-based macro programming interface, example scenarios that illustrate their use, and discuss the testbed's application in the research and teaching activities at client institutions. [ABSTRACT FROM AUTHOR]

With feature sizes steadily shrinking, manufacturing defects and parameter variations often cause design timing failures. It is essential that those errors be correctly and quickly diagnosed. In this paper, we analyze the multiple-delay-fault diagnosis problem and propose a novel approach to solve it. We enhance the diagnostic resolution by processing failure logs at various slower-than-nominal clock frequencies. We evaluate the utility of n-detection and timing-aware automatic-test-pattern-generated (ATPG) sets. Experimental results show that using timing-aware ATPG sets yields better diagnostic resolution and results in better delay-defect-size estimations compared to n-detection ATPG sets. We experimentally determined our diagnosis algorithm's sensitivity to delay variations. [ABSTRACT FROM AUTHOR]

The article presents information on an computer processor, Bugnet which helps the software developers to deterministically replay and debug an application while executing an computer program. The article author says that the difficult part of debugging is reproducing the bugs at a customer's site because it relys on core dumps which contain the final state of the system before the crash. The article authors' say that they propose the BugNet architecture to assist program developers to continuously record a program's execution as it runs. The Deterministic Replay Debugging technique helps to replay the exact same sequence of instructions that led up to the crash. The processor Bugnet continuously records a program's execution without interfering with the execution of the monitored program. The BugNet writes the logs to a fixed-sized buffer allocated in main memory so that software developers can then use these logs at the time of crash to deterministically replay the last part of execution as many times as required to fix the bug.