Search

There currently exists a need for a platform that provides users with the ability to perform extensive, repeatable and meaningful simulation and testing for the hardware and software which compose vehicle/autonomous vehicle systems whilst being broadly accessible, widely supported and provides robust features and development tools. The contemporary implementations of similar systems are either financially exorbitant or highly contained. The system reflected in this paper aims to fill a gap in the industry of vehicle/autonomous vehicle development by extending on currently existing open-source software to provide a highly streamlined platform to support the production of general road vehicle and autonomous vehicle driving systems. The software tools and hardware components chosen for the system will be discussed, followed by the features constructed throughout the development process. The end result of the system is a platform that allows for quick, repeatable, accurate, and nearly endless testing of a digital twin of real life vehicles. This system will allow users to gain valuable simulation and testing data of hardware and software components in a manner which is not always feasible using the traditional methods of autonomous vehicle testing. [ABSTRACT FROM AUTHOR]

This paper describes a hardware/software strategy for the effective and efficient management of several distributed Memory Built-In Self-Test (MBIST) units orchestrated by a single CPU to enable the parallel testing of several memory banks. Experimental testing of the implementation on an Infineon chip shows up to a 25% test time reduction compared to traditional strategies, especially in cases for which there are a large number of failures affecting several banks. Additionally, it permits balanced failure collection from different banks in cases for which there are limitations to the storage of failure-related information. [ABSTRACT FROM AUTHOR]

Software testing and debugging are standard practices of software quality assurance since they enable the identification and correction of failures. Benchmarks have been used in that context as a group of programs to support the comparison of different techniques according to pre-established parameters. However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains. The study was carried out using systematic mapping procedures according to the guidelines widely followed by software engineering literature. The search identified 1674 studies, from which, 25 were selected for analysis. A list of benchmarks is provided and descriptively mapped according to their characteristics, motivations, and scope of use for their creation. The lack of data to support the comparison between available and novel software testing and debugging techniques is the main motivation for the proposition of benchmarks. Advancements in the standardization and prescription of benchmark structure and composition are still required. Establishing such a standard could foster benchmark reuse, thereby saving time and effort in the engineering of benchmarks for software testing and debugging. [ABSTRACT FROM AUTHOR]

Manual testing is a machine testing method in which testing cases are carried out manually rather than utilizing an automatic process. Both test cases were handled manually by the tester from the perspective of the end consumer. It guarantees that the program performs, regardless of whether or not it is specified in the prerequisite text. In order to complete approximately 100% of the application, test cases are planned and implemented. Manually produced test case reports are common. Since it can spot both visible and hidden program bugs, manual testing is one of the most effective testing techniques. A defect is described by the software as the difference between expected and actual performance. The bugs were corrected and turned over to the tester by the creator. Manual training is expected before automatic testing for any newly created software. This research requires a lot of time and resources, but it ensures the entire program is bug-free. Manual research entails knowledge of manual testing methods but does not require the use of an automated evaluation method. Manual checking is necessary since one of the fundamental software testing concepts is that "100% automation is not feasible." [ABSTRACT FROM AUTHOR]

With the widespread use of embedded software in consumer electronics, automotive industry, medical devices, and industrial environments, embedded software testing is gaining significance as an indispensable part of development and deployment of embedded products. With more than 20 years of research, development, and testing of various consumer technologies and products based on digital signal processors (DSPs) and advanced reduced instruction set computers (ARMs), we obtained insight into typical embedded development process and testing, and the pros and cons of various testing approaches and environments. In this paper, we propose the Smart Multi-Agent Framework based on IoT and Jenkins agents, customised for audio technologies in the Home Audio domain. We evaluated our solution on several complex immersive audio technologies implemented on a multicore DSP. Our uniform, customised, fully automated approach proved to be time efficient, error resilient, easy to replicate and use across all development, certification, and deployment phases of the product life cycle. [ABSTRACT FROM AUTHOR]

This article proposes a hybrid test scheme based on interleaved test applications to increase the efficiency of multisite testing of integrated circuits (ICs) by solving problems related to test coverage, which is an important feature in automotive IC, and test costs, including the number of test pins and the volume of test data. To solve the problems, a hybrid test scheme that combines the merits of broadcast-based scan compression and built-in self-test (BIST) is proposed. The proposed scheme not only generates deterministic test patterns, similar to scan compression, but also requires fewer test pins, similar to BIST, thereby increasing the efficiency of multisite testing. With appropriately designed linear-feedback shift registers (LFSRs) and seeds periodically inserted through the reduced test pins, the test application process combines deterministic testing and random testing. The random testing stage can compensate the seed-insertion time for the subsequent deterministic testing stage and reduce the number of required deterministic patterns by detecting easy-to-detect faults. For further reduction of the volume of test data and hardware overhead, some compatible scan chains (SCs) are grouped and receive test data from an LFSR. The experimental results show that the proposed method increases the efficiency of multisite testing with a reduced pin count. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool testRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that testRNN has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, testRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing. [ABSTRACT FROM AUTHOR]

Software industry has adopted automated testing widely. The most common method adopted is graphical user interface test automation for the functional scenarios to reduce manual testing and increase the repeatability. Reducing execution time and redundancy to achieve quality "go/no go decisions" provides rational for the executive management to allocate funds to adopt automation and invest in the setup including people, process, and tools to achieve faster time to market. There are a variety of practices engaged by testers, like frameworks, tools, methods, procedures, models, and technologies to achieve automation. Nonetheless, the actual effectiveness in terms of return on investment (ROI) is not known, though there are various formulas to calculate ROI of test automation. The factors that determine the ROI are maturity of test automation, purpose, or intent, picking right tests for automation, knowledge of the tester to derive test coverage, domain expertise, defining right metrics like defects found by automation runs, cost versus reduction of time, labor versus quality of scripts, repeatability, and results. These factors form the base of the questions designed for the survey. The paper presents a survey and analysis to understand the ROI of test automation from industry test professionals from both product and services organizations. [ABSTRACT FROM AUTHOR]

In an era of connectivity and automation, the vehicle industry is adopting numerous technologies to transform driver-centric vehicles into intelligent mechanical devices driven by software components. Software integration and network connectivity inherit numerous security issues that open the door for malicious attacks. Software security testing is a scalable and practical approach to identify systems’ weaknesses and vulnerabilities at an early stage and throughout their life-cycle. Security specialists recommend fuzz testing to identify vulnerabilities within vehicle software systems. Nevertheless, the randomness and blindness of fuzzing hinder it from becoming a reliable security tool. This article presents a vulnerability-oriented fuzz (VulFuzz) testing framework that utilizes security vulnerability metrics designed particularly for connected and autonomous vehicles to direct and prioritize the fuzz testing toward the most vulnerable components. While most gray-box fuzzing techniques aim solely to expand code coverage, the proposed approach assigns weights to ensure a thorough examination of the most vulnerable components. Moreover, we employ an input structure-aware mutation technique that can bypass vehicle software systems’ input formats to boost test performance and avoid dropped test cases. Such a testing technique will contribute to the quality assurance of vehicle software engineering. We implemented the proposed approach on OpenPilot, a driver assistance system, and compared our results to American fuzzy lop (AFL) and an unguided mutation-based fuzzer. Within 16.8 h, VulFuzz exposed 335 crashes, 41 times more than AFL and two times more than an unguided mutation-based fuzzer. VulFuzz is explicitly efficient for automotive systems, reaching the same code coverage as AFL but with more exposed crashes and fewer dropped messages. [ABSTRACT FROM AUTHOR]

Symbolic execution is a powerful technique for automating software testing to detect many types of errors such as memory errors and assertion violations. However, it encounters the problem of path explosion, and by using only assertions, it still lacks the capability of going deep into checking the functional correctness of a path based on corresponding formal specifications. To address these problems, we propose a specification-based incremental testing method with symbolic execution, called SIT-SE, providing a much more rigorous way to automatically check the functional correctness of all the discovered program paths, by introducing theorems (instead of assertions) for path correctness and branch sequence coverage algorithm for guiding a moderate path exploration. Compared with Hoare logic for proving the correctness of an entire program, a theorem in the SIT-SE is made for verifying the correctness of a program path. The proposed method carefully treats the relationship between a path condition and the specification in a theorem to restrict the monotonous path exploration, whereas traditional concolic testing methods roughly use one test data to determine the path correctness by assertions during long path searching. We use a classic case to demonstrate how the method works and conduct an experiment to evaluate the performance of both the proposed method and the commonly used well-known concolic testing tool KLEE. The experimental results show that our method SIT-SE is effective and outperforms KLEE in detecting faulty paths based on specifications. [ABSTRACT FROM AUTHOR]

Copyright of Investigación e Innovación en Ingenierías is the property of Universidad Simon Bolivar and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Spectrum-based fault localization (SBFL) techniques can automatically localize software faults. They employ the program spectrum, such as code coverage profile with test verdicts, to rank the program entities based on their code suspiciousness. In the past decades, researchers have proposed many approaches to optimize these techniques; however, the program structure, which can influence their performance, is not taken into consideration in developing and improving these techniques. In this article, we identify and analyze the effect of the program structure on the application of SBFL techniques. We observe that some specific program structures may introduce structure bias to code suspiciousness and negatively influence the output of SBFL techniques. To mitigate these effects and improve the performance of fault localization, we propose Delta4Ts, a structure-aware technique. Delta4Ts references debugging history to alleviate the impact of structure bias in the calculation of code suspiciousness. It reasons from the observable suspicious value towards the desired suspicious value and the impact of structure bias. To evaluate Delta4Ts under practical constraints, we conduct a controlled experiment using nine widely-studied SBFL formulae on 12 C programs and 6 Java programs. The experiment results show that Delta4Ts can significantly improve the accuracy of the studied SBFL formulae by an average of 34.8% on 12 C programs and 30.6% on 6 Java programs, and improve more on subject programs associated with more history versions or having larger code sizes. [ABSTRACT FROM AUTHOR]

Test case prioritization (TCP) is a popular regression testing technique in software engineering field. The task of TCP is to schedule the execution order of test cases so that certain objective (e.g., code coverage) can be achieved quickly. In this article, we propose an efficient ant colony system framework for the TCP problem, with the aim of maximizing the code coverage as soon as possible. In the proposed framework, an effective heuristic function is proposed to guide the ants to construct solutions based on additional statement coverage among remaining test cases. Besides, a sorting-based local search mechanism is proposed to further accelerate the convergence speed of the algorithm. Experimental results on different benchmark problems, and a real-world application, have shown that the proposed framework can outperform several state-of-the-art methods, in terms of solution quality and search efficiency. [ABSTRACT FROM AUTHOR]

One of the major challenges in software testing is the generation of test data automatically that satisfy a specified adequacy criterion. This paper presents a GA-based approach and a supporting tool for data-flow test data generation for ASP.NET web applications. The proposed tool accepts as input the web application under test, instruments it, and performs static analysis to compute the definition-use pairs. The proposed GA conducts its search by constructing new test data from previously generated test data that are evaluated as effective test data. In this GA, the chromosome is a collection of user interface control objects, where each control is considered as a gene. Therefore, novel crossover and mutation operators are developed to manipulate the chromosome, which are called block crossover and control-based mutation operators. The proposed GA accepts as input the instrumented version, the list of definition-use pairs to be covered, and input controls related information. The tool produces a set of test cases, the set of definition-use pairs covered by each test case, and a list of uncovered definition-use pairs, if any. Also the paper presents a case study to illustrate how the tool works. Finally, it presents the results of the empirical evaluation that is performed to evaluate the effectiveness of the generated test data in exposing web application errors. [ABSTRACT FROM AUTHOR]

For the test case prioritization problems, the average percent of faults detected (APFD) and its variant versions are widely used as metrics to evaluate prioritized test suite's efficiency of fault detection. By a revisit of metrics for test case prioritization, we observe that APFD is only available for the scenarios where all test suites under evaluation contain the same number of test cases. Such a limitation is often overlooked, and lead to incorrect results when comparing fault detection efficiency of test suites with different sizes. Moreover, APFD cannot precisely illustrate the process of fault detection in the real world. Besides the APFD, most of its variants, including the NAPFD and the APFD C W , have similar problems. This paper points out these limitations in detail by analyzing the physical explanation of APFD series metrics formally. In order to eliminate these limitations, we propose a series of improved metrics, including the relative average percent of faults detected (RAPFD) and the relative cost-cognizant weighted average percent of faults detected (RAPFD C W ), to evaluate the efficiency of the test suite. Furthermore, for the scenario of parallel testing, a series of metrics including the relative average percent of faults detected in parallel testing (P -RAPFD) and the relative cost-cognizant weighted average percent of faults detected in parallel testing (P -RAPFD C W ) are proposed too. All the proposed metrics refer to both the speed of fault detection and the constraint of the testing resource. A formal analysis and some examples show that all the proposed metrics provide much more precise illustrations of the fault detection process. [ABSTRACT FROM AUTHOR]

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction. [ABSTRACT FROM AUTHOR]

Software failures caused by data race bugs have always been major concerns in parallel and distributed systems, despite significant efforts spent in software testing. Due to their nondeterministic and hard-to-reproduce features, when evaluating systems’ operational reliability, a rather long period of experimental execution time is expected to be spent on observing failures caused by data race conditions. To address this problem, in this paper, we make two contributions. First, this paper proposes stress testing with influencing factors, in which the system runs under certain workloads for a long time with controlled stress conditions to accelerate the occurrence of data race failures. Second, it explores and formulates mathematical relationship models between data races’ statistical characteristics of time to failure (TTF) or mean TTF (MTTF) and the influencing factors. Such relationship models are used for TTF/MTTF extrapolation under different operational conditions and are essential to reduce systems’ reliability evaluation time. The proposed method is empirically evaluated on six applications suffering from failures caused by real-world data race bugs. Through analysis of the experimental results, we obtain several important findings: First, the reduction in the manifestation time to data race failures achieved by controlling the influencing factors is statistically significant. Second, Power model is the best-fitting model of the relationship between the MTTF and the influencing factors. Third, Power Weibull distribution is the best-fitting probability distribution between the TTF and the influencing factors. Finally, the TTF/MTTF can be accurately estimated with the approach proposed in this paper. [ABSTRACT FROM AUTHOR]

Adaptive random testing (ART) was developed as an enhanced version of random testing to increase the effectiveness of detecting failures in programs by spreading the test cases evenly over the input space. However, heavy computation may be incurred. In this paper, three enhanced algorithms for fixed-size-candidate-set ART (FSCS-ART) are proposed based on the $k$ -dimensional tree (KD-tree) structure. The first algorithm Naive-KDFC constructs a KD-tree by splitting the input space with respect to every dimension successively in a round-robin fashion. The second algorithm SemiBal-KDFC improves the balance of the KD-tree by prioritizing the splitting according to the spread in each dimension. In order to control the number of traversed nodes in backtracking, the third algorithm LimBal-KDFC introduces an upper bound for the nodes involved. Simulation and empirical studies have been conducted to investigate the efficiency and effectiveness of the three algorithms. The experimental results show that these algorithms significantly reduce the computation time of the original FSCS-ART for low dimensions and for the case of high dimensions with low failure rates. The efficiency of SemiBal-KDFC is better than that of Naive-KDFC when the dimension is no more than 8, but LimBal-KDFC is the most efficient of all three. Although the limited backtracking leads only to an approximate nearest neighbor in LimBal-KDFC, its failure-detection effectiveness is, in fact, better than FSCS-ART in high-dimensional input spaces and has no significant deterioration in low-dimensional spaces. [ABSTRACT FROM AUTHOR]

Though Deep Neural Networks (DNNs) have been widely deployed and achieved great success in many domains, they have severe safety and reliability concerns. To provide testing evidence for DNNs' reliable behaviors, various coverage testing techniques inspired by traditional software testing have been proposed. However, the coverage criteria in these techniques are either not fine enough to capture subtle behaviors of DNNs, or too time-consuming to be applied on large-scale DNNs. In this paper, we propose a coverage testing framework named HashC, which makes mainstream coverage criteria (e.g., NC and KMNC) much finer. Meanwhile, HashC reduces the time complexity of combinatorial coverage testing from polynomial time to linear time. We also develop the corresponding test sample selection method. Our experiments show that, (1) the existing mainstream coverage criteria are becoming finer after being equipped with HashC, (2) HashC greatly accelerates combinatorial coverage testing and can handle the testing of large-scale DNNs. [ABSTRACT FROM AUTHOR]

Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing, where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing, to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing. [ABSTRACT FROM AUTHOR]

New processing models are being adopted in Big Data engineering to overcome the limitations of traditional technology. Among them, MapReduce stands out by allowing for the processing of large volumes of data over a distributed infrastructure that can change during runtime. The developer only designs the functionality of the program and its execution is managed by a distributed system. As a consequence, a program can behave differently at each execution because it is automatically adapted to the resources available at each moment. Therefore, when the program has a design fault, this could be revealed in some executions and masked in others. However, during testing, these faults are usually masked because the test infrastructure is stable, and they are only revealed in production because the environment is more aggressive with infrastructure failures, among other reasons. This paper proposes new testing techniques that aimed to detect these design faults by simulating different infrastructure configurations. The testing techniques generate a representative set of infrastructure configurations that as whole are more likely to reveal failures using random testing, and partition testing together with combinatorial testing. The techniques are automated by using a test execution engine called MRTest that is able to detect these faults using only the test input data, regardless of the expected output. Our empirical evaluation shows that MRTest can automatically detect these design faults within a reasonable time. [ABSTRACT FROM AUTHOR]

Research regarding MAT (Mobile Application Testing) in Pakistan is hard to discover and to the best of our knowledge, no work has been done in surveying MAT in Pakistan. In this work, we have examined the current trend and status of MAT in Pakistan. Main objective was to investigate to what extent MAT is currently applied in Pakistan software companies and what experience the companies have with using MAT. Furthermore, efforts were made to find out what testers think about MAT, e.g. issue, advantages and disadvantages of MAT, what factors affects MAT and how they plan to improve MAT. In order to achieve our objectives, we used a comprehensive online survey so we converted our research questions into correspondence survey questions. We served a questionnaire of the survey to 66 testing relevant officials of leading software companies in different cities of Pakistan to develop a model study about general trend and status of MAT which can be generalized all over Pakistan. We received 56 replies in total after over 2 months. After that, we used SPSS tool to analyze the replies of this questionnaire. Cross-Tabulation Analysis and Pearson Chi-square tests have been computed to examine the results. We found some interesting results on current status and practice of MAT in Pakistan software companies. [ABSTRACT FROM AUTHOR]

Demand for highly reliable software is increasing day by day which in turn has increased the pressure on the software firms to provide reliable software in no time. Ensuring high reliability of the software can only be done by prolonged testing which in result consumes more resources which is not feasible in the existing market situation. To overcome this, software firms are providing patches after software release so as to fix the remaining number of bugs and to give better product experience to users. An update/fix is a minor portion of software to repair the bugs. With such patches, organizations enhance the performance of the software. Delivering patches after release demands extra effort and resources which are costly and hence not economical for the firms. Also, early patch release might cause improper fixation of bugs, on the other hand, delayed release may increase the chances of more failure during the operational phase. Therefore, determining optimal patch release time is imperative. To overcome the above issues we have formulated a two-dimensional time and effort-based cost model to regulate the optimum release and patch time of software, so that the total cost is minimized. Proposed model is validated on a real life data set. [ABSTRACT FROM AUTHOR]

Previous research in defect prediction has proposed approaches to determine which files require additional testing resources. However, practitioners typically create tests at a higher level of abstraction, which may span across many files. In this paper, we study software testing, especially test resource prioritization, from a different perspective. We use topic models to generate topics that provide a high-level view of a system, allowing developers to look at the test case coverage from a different angle. We propose measures of how well tested and defect prone a topic is, allowing us to discover which topics are well tested and which are defect prone. We conduct case studies on the histories of Mylyn, Eclipse, and NetBeans. We find that 34–78% of topics are shared between source code and test files, indicating that we can use topic models to study testing; well-tested topics are usually less defect prone, defect-prone topics are usually undertested; we can predict which topics are defect prone but not well tested with an average precision and recall of 75% and 77%, respectively; our approach complements traditional prediction-based approaches by saving testing and code inspection effort; and our approach is not particularly sensitive to the parameters that we use. [ABSTRACT FROM PUBLISHER]

During the unit testing of model-driven tools, a large number of models and test classes needs to be managed and maintained. Typically, some of these artifacts are specified manually, some are generated automatically. Existing approaches to test management rely on the available visual and textual modeling notations. As these notations are not tailored to unit testing, distinct maintainability trade-offs arise. In this paper, we propose a notation that aims to combine the benefits of visual and text-based approaches. The notation is at the same time visual and text-based, as it uses ASCII characters to emulate the familiar graphical notations. In our evaluation based on real models, we identify problematic model shapes challenging the scalability our notation, while finding that it is well-suited to capture typical test models. [ABSTRACT FROM AUTHOR]

Software testing enables development teams to maintain the quality of a software system while it evolves. The database manipulation code requires special attention in this context. However, it is often neglected and suffers from software maintenance problems. In this paper, we study the current state-of-the-practice in testing database manipulation code. We first analysed the tests of 72 open-source projects to gain insight into the coverage of database access code. The database was poorly tested: 46% of the projects did not cover with tests half of their database access methods, and 33% did not cover the database code at all. This poor coverage motivated us to study developers' challenges and best practices. (i) First, we analysed 532 questions on Stack Exchange sites and deduced a taxonomy of issues. Developers mostly looked for general best practices to test database access code. Their technical questions were related to database management, mocking, parallelisation, or framework/tool usage. (ii) Next, we examined the answers to these questions. We manually labelled 598 answers to 255 questions. We distinguished 363 solutions and organised them in a taxonomy of best practices. Most of the suggestions considered the testing environment and recommended various tools or configurations. The second largest category was database management, where many addressed database initialisation and clean-up between tests. Other categories pertained to code structure or design, concepts, performance, processes, test characteristics, test code, and mocking. We illustrate the two taxonomies through intriguing examples. • A study on the coverage of database access code in the tests of open-source projects. • A qualitative assessment of developers' issues when testing database access code. • An investigation of developers' best practices when testing database access code. • Taxonomies of issues and best practices when testing database access code. • The full dataset of the studies is publicly available as an online appendix. [ABSTRACT FROM AUTHOR]

To ensure that an access control (AC) system is safe, there must be a reliable means to verify that the specified AC policy model conforms to the safety requirements and policy author's intentions. A general verification approach includes black-box and white-box testing, as well as the generation of sufficient test cases to check the correctness of model implementations. [ABSTRACT FROM AUTHOR]

The focus of this article has been about automated testing in practice. Using testing tools is quite common phenomenon in software companies. The article studied the use of software testing tools. Testing software is enough ordinary things in the IT software development company. But the using automated testing tools only develop in the educational training of IT professionals. Information study of the use of testing tools for the control of software quality IT professionals and IT students was held. Statistical analysis of the data shows the percentage of how many people use the testing tools, but most still depends on manual testing. The main purpose of the article is to find out the facts related to the research questions of creation of new testing tools for educational purposes of highly qualified IT professionals. [ABSTRACT FROM AUTHOR]

Random testing is a fundamental testing technique that can be used to generate test cases for both hardware and software systems. Quasi-random testing was proposed as an enhancement to the cost-effectiveness of random testing: In addition to having similar computation overheads to random testing, it makes use of quasi-random sequences to generate low-discrepancy and low-dispersion test cases that help deliver high failure-detection effectiveness. Currently, few algorithms exist to generate quasi-random sequences, and these are mostly deterministic, rather than random. A previous study of quasi-random testing has examined two methods for randomizing quasi-random sequences to improve their applicability in testing. However, these randomization methods still have shortcomings—one method does not introduce much randomness to the test cases, while the other does not support incremental test case generation. In this paper, we present an innovative approach to incrementally randomizing quasi-random sequences. The test cases generated by this new approach show a high degree of randomness and evenness in distribution. We also conduct simulations and empirical studies to demonstrate the applicability and effectiveness of our approach in software testing. [ABSTRACT FROM PUBLISHER]

Metamorphic testing (MT) can enhance security testing by providing an alternative to using a test oracle, which is often unavailable or impractical. The authors report how MT detected previously unknown bugs in real-world critical applications such as code obfuscators, giving evidence that software testing requires diverse perspectives to achieve greater cybersecurity. [ABSTRACT FROM AUTHOR]

Software testing is an important step in the life cycle of agile development; it represents an efficient way to ensure the good functioning of the product. But as the complexity of a system increases, the effort and expertise to test it also increases. To significantly reduce these efforts, and reduce the cost and time; several studies have been carried out and various tools and test automation techniques have been proposed. In this paper, we present an approach to automatic generation of test cases from UML 2 Models at the Scrum agile process. This approach automates two important steps: the transformation of design models into test models and generating test cases, based on an open source MDA framework. [ABSTRACT FROM AUTHOR]

The Linux kernel does not export a stable, well-defined kernel interface, complicating the development of kernel-level services, such as device drivers and file systems. While there does exist a set of functions that are exported to external modules, this set of functions frequently changes, and the functions have implicit, ill-documented preconditions. No specific debugging support is provided. We present Diagnosys, an approach to automatically constructing a debugging interface for the Linux kernel. First, a designated kernel maintainer uses Diagnosys to identify constraints on the use of the exported functions. Based on this information, developers of kernel services can then use Diagnosys to generate a debugging interface specialized to their code. When a service including this interface is tested, it records information about potential problems. This information is preserved following a kernel crash or hang. Our experiments show that the generated debugging interface provides useful log information and incurs a low performance penalty. [ABSTRACT FROM AUTHOR]

Students with special needs often require special teaching tools; physical and sensory impaired students can benefit from using computer-aided assessment methods that address their fully developed abilities. Easy-E is an assessment software tool designed to improve the learning abilities in the field of basic electronics. The friendly user interface provides easy access to all sections (circuits with diodes, resistors, capacitors, operational amplifiers). Theoretical presentations, schematics, waveforms, and transfer characteristics are available and the user can change the values of components and signal parameters. The acquired knowledge can be tested, in the Test section. The tests cover the entire area, and consist of ten multiple choice questions. The proposed software tool is a standalone application with low computational requirements, which can be used for both online and offline learning. Due to the fact that our tool offers theoretical fundamentals, active learning through solved problems and final testing, it can be seen as a complete instrument for home schooling in the field of basic electronics. [ABSTRACT FROM AUTHOR]

We show how to write concise and readable specifications of linked data structures that are applicable for both formal deductive verification and testing. A singly linked list and a binary search tree are provided as examples. The main characteristic of the specifications is the use of observer methods, in particular to express reachability of elements in a data structure. The specifications are written in the Java Modeling Language (JML) and do not require extensions of that language. This paper addresses a mixed audience of users and developers in the fields of formal verification, testing, and specification language design. We provide an in-depth description of the proposed specifications and analyze the implications both for verification and testing. Based on this analysis we have developed verification techniques that are implemented in the deductive verification tool KeY and enable fully automatic verification of the linked list example. This techniques are also described in this paper. [ABSTRACT FROM AUTHOR]

Web Services are a technology designed to support the invocation of remote elements by client applications, with the goal of providing interoperable application-to-application interaction while assuring vendor and platform independence. In business-critical environments, the occurrence of interoperability issues can have disastrous consequences, including direct financial costs, reputation, and client fidelity losses. Despite this, experience shows that services interoperability is still quite difficult to achieve. The goal of this paper is to propose a practical testing process to understand the real level of interoperability provided by web services platforms. An extensible tool, that implements the proposed approach, has been used to run a large campaign during which we have tested the interoperability of a large number of web services, comprising both homeimplemented and publicly available services, deployed on top of several web service platforms and against 11 client-side web service platforms. Numerous issues have been disclosed, showing the effectiveness of the proposed approach and the importance of having an automatic tool for testing web services for interoperability. [ABSTRACT FROM AUTHOR]

Test case prioritization is the process of ordering the test cases to be conducted eventually. Prioritizing test cases aids to meet two important constraints namely time and budget in software testing in order to enhance the fault detection rate as early as possible. This results us finding the problem earlier due to improved fault detection rate at early stage and deliver the system at short period of time .Interactions and relationships between a systems or sub system of an user is called as dependency. Few techniques that have been used in the past are namely Random order, greedy technique as well as fine grained and coarse grained. But all these techniques have considered the test cases in a test suite as an independent way. Few techniques have been derived dependency structures among test cases manually. Hence, this paper focuses to provide a set of algorithms to derive dependency structures among functions in a system and find exact number of dependents of each function in a system to prioritize the test cases in automated manner. High priority is assigned to test cases that are more dependent in using graph coverage values. By executing high priority (more complex) test cases earlier, the rate of fault detection will be improved at earlier stage and help to fix the fault earlier. For this experimentation purposes, Siemens test suite has been taken to establish the truth of verifying the given proposed technique. [ABSTRACT FROM AUTHOR]

Software testing is a very delicate aspect of software development, since designing good test sets is a non-trivial task. In this article, we describe a testing technique for testing business rules using property-based testing and the property-based automatic testing tool QuickCheck. Systematic, effective, and efficient testing of business rules increases the confidence on the validation of business concepts and domain rules which are specifically critical to data consistency. The approach is presented on the basis of small but representative examples in order to facilitate the readers' understanding, but it has been successfully evaluated in a number of different industrial examples, demonstrating that it generalises to much larger systems and is, thus, broadly applicable. [ABSTRACT FROM AUTHOR]

Model-based testing employs models for testing. Model-based mutation testing (MBMT) additionally involves fault models, called mutants, by applying mutation operators to the original model. A problem encountered with MBMT is the elimination of equivalent mutants and multiple mutants modeling the same faults. Another problem is the need to compare a mutant to the original model for test generation. This paper proposes an event-based approach to MBMT that is not fixed on single events and a single model but rather operates on sequences of events of length k ≥ 1 and invokes a sequence of models that are derived from the original one by varying its morphology based on k. The approach employs formal grammars, related mutation operators, and algorithms to generate test cases, enabling the following: (1) the exclusion of equivalent mutants and multiple mutants; (2) the generation of a test case in linear time to kill a selected mutant without comparing it to the original model; (3) the analysis of morphologically different models enabling the systematic generation of mutants, thereby extending the set of fault models studied in related literature. Three case studies validate the approach and analyze its characteristics in comparison to random testing and another MBMT approach. [ABSTRACT FROM PUBLISHER]

Testing is a key activity of software development and maintenance that determines the level of reliability. Traceability is the ability to describe and follow the life of software artifacts, and has been promoted as a means for supporting various activities, most importantly testing. Traceability information facilitates the testing and debugging of complex software by modeling the dependencies between code and tests. Actively supplementing traceability to testing enables rectifying defects more reliably and efficiently. Despite its importance, the development of test-to-code traceability has not been sufficiently addressed in the literature, and even worse there is currently no organized review of traceability studies in this field. In this work, we have investigated the main conferences, workshops, and journals of the requirements engineering, testing, and reliability, and identified those contributions that refer to traceability topics. From that starting point, we characterized and analyzed the chosen contributions against three research questions by utilizing a comparative framework including nine criteria. As a result, our study arrives to some interesting points, and outlines a number of potential research directions. This, in turn, can pave the way for facilitating and empowering traceability research in this domain to assist software engineers and testers in test management. [ABSTRACT FROM AUTHOR]

In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC's characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer's Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution. [ABSTRACT FROM AUTHOR]

Transaction-level modeling (TLM) has become the de-facto reference modeling style for system-level design and verification of embedded systems. It allows designers to implement high-level communication protocols for simulations up to 1000 \times faster than at register-transfer level (RTL). To guarantee interoperability between TLM IP suppliers and users, designers implement the TLM communication protocols by relying on a reference standard, such as the standard OSCI for SystemC TLM. Functional correctness of such protocols as well as their compliance to the reference TLM standard are usually verified through user-defined testbenches, whose high quality and completeness play a key role for an efficient TLM design and verification flow. This article presents a methodology to apply mutation analysis, a technique applied in literature for SW testing, for measuring the testbench quality in verifying TLM protocols. In particular, the methodology aims at (i) qualifying the testbenches by considering both the TLM protocol correctness and their compliance to a defined standard (i.e., OSCI TLM), (ii) optimizing the simulation time during mutation analysis by avoiding mutation redundancies, and (iii) driving the designers in the testbench improvement. Experimental results on benchmarks of different complexity and architectural characteristics are reported to analyze the methodology applicability. [ABSTRACT FROM AUTHOR]

Context. The equivalent mutant problem (EMP) is one of the crucial problems in mutation testing widely studied over decades. Objectives. The objectives are: to present a systematic literature review (SLR) in the field of EMP; to identify, classify and improve the existing, or implement new, methods which try to overcome EMP and evaluate them. Method. We performed SLR based on the search of digital libraries. We implemented four second order mutation (SOM) strategies, in addition to first order mutation (FOM), and compared them from different perspectives. Results. Our SLR identified 17 relevant techniques (in 22 articles) and three categories of techniques: detecting (DEM); suggesting (SEM); and avoiding equivalent mutant generation (AEMG). The experiment indicated that SOM in general and JudyDiffOp strategy in particular provide the best results in the following areas: total number of mutants generated; the association between the type of mutation strategy and whether the generated mutants were equivalent or not; the number of not killed mutants; mutation testing time; time needed for manual classification. Conclusions . The results in the DEM category are still far from perfect. Thus, the SEM and AEMG categories have been developed. The JudyDiffOp algorithm achieved good results in many areas. [ABSTRACT FROM PUBLISHER]

Abstract: The goal of this paper is to review the literature in order to understand the implications of accessibility testing processes with the objective to detect potential improvements and developments in the field. Thus, a brief review is presented of the fundamental test processes proposed by the International Software Testing Qualification Board (ISTQB) and the currently available literature about testing processes for evaluating the accessibility of web applications. The result of the review reflects an array of proposals to incorporate accessibility requirements and evaluation tools, but they do not describe a comprehensive testing process at each phase of the development lifecycle of accessible web applications. [Copyright &y& Elsevier]

This paper introduces a new metric to characterize test sets in terms of their diagnostic power. Our method uses much less space compared to the existing ones and is quite accurate. The metric can be utilized to increase the diagnosability of incompletely specified test sets via don't care filling. The X-filling approach can be integrated with test pattern generation tools to aid in better diagnostic pattern set generation. [ABSTRACT FROM PUBLISHER]

Background: Cross-company defect prediction (CCDP) is a field of study where an organization lacking enough local data can use data from other organizations for building defect predictors. To support CCDP, data must be shared. Such shared data must be privatized, but that privatization could severely damage the utility of the data. Aim: To enable effective defect prediction from shared data while preserving privacy. Method: We explore privatization algorithms that maintain class boundaries in a dataset. CLIFF is an instance pruner that deletes irrelevant examples. MORPH is a data mutator that moves the data a random distance, taking care not to cross class boundaries. CLIFF+MORPH are tested in a CCDP study among 10 defect datasets from the PROMISE data repository. Results: We find: 1) The CLIFFed+MORPHed algorithms provide more privacy than the state-of-the-art privacy algorithms; 2) in terms of utility measured by defect prediction, we find that CLIFF+MORPH performs significantly better. Conclusions: For the OO defect data studied here, data can be privatized and shared without a significant degradation in utility. To the best of our knowledge, this is the first published result where privatization does not compromise defect prediction. [ABSTRACT FROM AUTHOR]

Testing of a software product is the key activity before deploying it in the real-time environment. Therefore testing should be introduced into the product development process as early as possible in order to decrease the costs of repairing the damage in later phases. In this paper we use Coloured Petri Nets (CPN) for the system specification and also as a system to be tested instead of its implementation. This specification also serves as the basis for the test suite generation. Test cases are provided in the language of Testing and Test Control Notation version 3 (TTCN-3) due to its general application area and platform independence. Stop-and-wait communication protocol has been used as a tutorial example of our approach. [ABSTRACT FROM AUTHOR]

Testing provides a primary means for assuring software in safety-critical systems. To demonstrate, particularly to a certification authority, that sufficient testing has been performed, it is necessary to achieve the test coverage levels recommended or mandated by safety standards and industry guidelines. Mutation testing provides an alternative or complementary method of measuring test sufficiency, but has not been widely adopted in the safety-critical industry. In this study, we provide an empirical evaluation of the application of mutation testing to airborne software systems which have already satisfied the coverage requirements for certification. Specifically, we apply mutation testing to safety-critical software developed using high-integrity subsets of C and Ada, identify the most effective mutant types, and analyze the root causes of failures in test cases. Our findings show how mutation testing could be effective where traditional structural coverage analysis and manual peer review have failed. They also show that several testing issues have origins beyond the test activity, and this suggests improvements to the requirements definition and coding process. Our study also examines the relationship between program characteristics and mutation survival and considers how program size can provide a means for targeting test areas most likely to have dormant faults. Industry feedback is also provided, particularly on how mutation testing can be integrated into a typical verification life cycle of airborne software. [ABSTRACT FROM PUBLISHER]

This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity. The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity. However, subgroup analysis has found both the quality improvement and the productivity drop to be much larger in industrial studies in comparison with academic studies. A larger drop of productivity was found in studies where the difference in test effort between the TDD and the control group's process was significant. A larger improvement in quality was also found in the academic studies when the difference in test effort is substantial; however, no conclusion could be derived regarding the industrial studies due to the lack of data. Finally, the influence of developer experience and task size as moderator variables was investigated, and a statistically significant positive correlation was found between task size and the magnitude of the improvement in quality. [ABSTRACT FROM PUBLISHER]

Random testing is a basic software testing technique that can be used to assess the software reliability as well as to detect software failures. Adaptive random testing has been proposed to enhance the failure-detection capability of random testing. Previous studies have shown that adaptive random testing can use fewer test cases than random testing to detect the first software failure. In this paper, we evaluate and compare the performance of adaptive random testing and random testing from another perspective, that of code coverage. As shown in various investigations, a higher code coverage not only brings a higher failure-detection capability, but also improves the effectiveness of software reliability estimation. We conduct a series of experiments based on two categories of code coverage criteria: structure-based coverage, and fault-based coverage. Adaptive random testing can achieve higher code coverage than random testing with the same number of test cases. Our experimental results imply that, in addition to having a better failure-detection capability than random testing, adaptive random testing also delivers a higher effectiveness in assessing software reliability, and a higher confidence in the reliability of the software under test even when no failure is detected. [ABSTRACT FROM PUBLISHER]

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image annotations and gene function analysis. Current research on multilabel classification focuses on supervised settings which assume existence of large amounts of labeled training data. However, in many applications, the labeling of multilabeled data is extremely expensive and time consuming, while there are often abundant unlabeled data available. In this paper, we study the problem of transductive multilabel learning and propose a novel solution, called Trasductive Multilabel Classification (TraM), to effectively assign a set of multiple labels to each instance. Different from supervised multilabel learning methods, we estimate the label sets of the unlabeled instances effectively by utilizing the information from both labeled and unlabeled data. We first formulate the transductive multilabel learning as an optimization problem of estimating label concept compositions. Then, we derive a closed-form solution to this optimization problem and propose an effective algorithm to assign label sets to the unlabeled instances. Empirical studies on several real-world multilabel learning tasks demonstrate that our TraM method can effectively boost the performance of multilabel classification by using both labeled and unlabeled data. [ABSTRACT FROM AUTHOR]