Search

Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the specification of the system requirements, and (b) manual fitness function design. Both techniques have advantages. The former uses information from the system requirements to guide the search toward portions of the input domain more likely to contain failures. The latter uses the engineers' domain knowledge. We propose ATheNA, a novel SBST framework that combines fitness functions automatically generated from requirements specifications and those manually defined by engineers. We design and implement ATheNA-S, an instance of ATheNA that targets Simulink® models. We evaluate ATheNA-S by considering a large set of models from different domains. Our results show that ATheNA-S generates more failure-revealing test cases than existing baseline tools and that the difference between the runtime performance of ATheNA-S and the baseline tools is not statistically significant. We also assess whether ATheNA-S could generate failure-revealing test cases when applied to two representative case studies: one from the automotive domain and one from the medical domain. Our results show that ATheNA-S successfully revealed a requirement violation in our case studies. [ABSTRACT FROM AUTHOR]

Software quality is becoming a momentous challenge in software engineering processes, and software testing has a pivotal role in its measurements. Nature-inspired metaheuristic methods play an essential role in software testing, in which various studies have been conducted in this field; however, due to a lack of wide-ranging papers reviewing these methods, conducting a comprehensive systematic review to examine an array of crucial mechanisms in this field has become a necessity. This study aims to present a detailed analysis and taxonomically classifies the metaheuristic methods inspired by nature. This paper compromises a systematic literature review of 65 chosen studies published between 2015 and 2022. Genetic algorithm-based, hybrid, ant colony optimization-based, cuckoo search-based, firefly algorithm-based, artificial bee colony-based, and other metaheuristic methods constitute this systematic study's stratification. Evaluation methods, applied tools, merits, and demerits of each reviewed article are investigated. Additionally, future directions and open issues are addressed. This conducted paper not only expounds on software testing strengths, open issues, and future works, but also recognizes the quest for optimizing the insufficient metrics in software testing, such as mutation score, complexity, and scalability, which would be the propulsion of the testing process if consummated. [ABSTRACT FROM AUTHOR]

This paper describes a hardware/software strategy for the effective and efficient management of several distributed Memory Built-In Self-Test (MBIST) units orchestrated by a single CPU to enable the parallel testing of several memory banks. Experimental testing of the implementation on an Infineon chip shows up to a 25% test time reduction compared to traditional strategies, especially in cases for which there are a large number of failures affecting several banks. Additionally, it permits balanced failure collection from different banks in cases for which there are limitations to the storage of failure-related information. [ABSTRACT FROM AUTHOR]

Purpose: The aim of the article is to provide an in-depth discussion of traditional and modern methods of software diagnostics and testing. The basic ways of implementing these processes will be discussed. We will also discuss the concept of automation of the testing process and its impact on modern software development with usage of Artificial Intelligence. Design/methodology/approach: The analysis of the article's topic is based on the analysis of sources discussing methods of software diagnostics and testing. Findings: The importance of analytical thinking and predictive skills. Originality/value: Presentation of various factors influencing the quality of software production with a particular focus on software test automation. [ABSTRACT FROM AUTHOR]

Software testing and debugging are standard practices of software quality assurance since they enable the identification and correction of failures. Benchmarks have been used in that context as a group of programs to support the comparison of different techniques according to pre-established parameters. However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains. The study was carried out using systematic mapping procedures according to the guidelines widely followed by software engineering literature. The search identified 1674 studies, from which, 25 were selected for analysis. A list of benchmarks is provided and descriptively mapped according to their characteristics, motivations, and scope of use for their creation. The lack of data to support the comparison between available and novel software testing and debugging techniques is the main motivation for the proposition of benchmarks. Advancements in the standardization and prescription of benchmark structure and composition are still required. Establishing such a standard could foster benchmark reuse, thereby saving time and effort in the engineering of benchmarks for software testing and debugging. [ABSTRACT FROM AUTHOR]

The paper proposes further development of formal method for obtaining the values of software quality attributes and assessing the quality of software used for controlling and parameters monitoring of autonomous electric power systems. For this purpose, dynamic software testing was used with the involvement of a group of experts. The results of the conducted expert assessment were used as initial data. The attributes of software quality indicators, such as functionality, practicality, maintainability, reliability, were calculated using the method of summarizing and grouping the results of statistical observation that allowed to check the software compliance with quality standards. Additional weighting coefficients that describe the importance of individual software quality attributes were introduced, and additive convolution is used to calculate the values of various software quality indicators. Minimax criterion was used to find the best solution that maximizes the quality of the software and minimizes possible losses due to errors. The technique proposed in the paper makes it possible to obtain quantitative assessments of software quality based on statistical processing of testing results and expert assessments. This allows to select specific software characteristics for improvement without affecting others, to predict software failureless time and to minimize the subjective factor during testing. [ABSTRACT FROM AUTHOR]

The object of research in the article is the process of testing and operating software with cost minimization. In the Software Development Life Cycle, depending on the chosen option of the flexible methodology, special attention is focused on testing software versions both in the process of passing iterations and in the process of releasing alpha, beta and production versions. This article is devoted to the problem of developing a method for software testing cost optimization method that estimates the test cost function and the losses cost function from the occurrence of an error. Using the optimization method (for example, the first-order descent method) from the two functions of testing costs and estimating the losses caused during operation, it is possible to calculate the optimal cost of testing and operating the software product. The results obtained show that with the correct assessment of a cost function and a loss function such calculations allow to significantly save money and time for the production of the next version of the software product. These results are explained by the fact that the method of optimizing the cost function finds the optimum point and allows to pre-estimate the budget and risks during the development and operation of the software. The article provides several examples of the calculation and optimization of testing costs within the proposed concept for one iteration in a flexible software development cycle. The results of the study can be used in practice, provided that the functions of estimating costs for testing and compensation for losses caused during the operation of the software are set correctly. Experienced managers and project supervisors determine these functions quite accurately for a certain number of iterations, which makes it possible to apply the method of finding the minimum budget costs for testing and operating a software product. [ABSTRACT FROM AUTHOR]

Manual testing is a machine testing method in which testing cases are carried out manually rather than utilizing an automatic process. Both test cases were handled manually by the tester from the perspective of the end consumer. It guarantees that the program performs, regardless of whether or not it is specified in the prerequisite text. In order to complete approximately 100% of the application, test cases are planned and implemented. Manually produced test case reports are common. Since it can spot both visible and hidden program bugs, manual testing is one of the most effective testing techniques. A defect is described by the software as the difference between expected and actual performance. The bugs were corrected and turned over to the tester by the creator. Manual training is expected before automatic testing for any newly created software. This research requires a lot of time and resources, but it ensures the entire program is bug-free. Manual research entails knowledge of manual testing methods but does not require the use of an automated evaluation method. Manual checking is necessary since one of the fundamental software testing concepts is that "100% automation is not feasible." [ABSTRACT FROM AUTHOR]

With the widespread use of embedded software in consumer electronics, automotive industry, medical devices, and industrial environments, embedded software testing is gaining significance as an indispensable part of development and deployment of embedded products. With more than 20 years of research, development, and testing of various consumer technologies and products based on digital signal processors (DSPs) and advanced reduced instruction set computers (ARMs), we obtained insight into typical embedded development process and testing, and the pros and cons of various testing approaches and environments. In this paper, we propose the Smart Multi-Agent Framework based on IoT and Jenkins agents, customised for audio technologies in the Home Audio domain. We evaluated our solution on several complex immersive audio technologies implemented on a multicore DSP. Our uniform, customised, fully automated approach proved to be time efficient, error resilient, easy to replicate and use across all development, certification, and deployment phases of the product life cycle. [ABSTRACT FROM AUTHOR]

This article proposes a hybrid test scheme based on interleaved test applications to increase the efficiency of multisite testing of integrated circuits (ICs) by solving problems related to test coverage, which is an important feature in automotive IC, and test costs, including the number of test pins and the volume of test data. To solve the problems, a hybrid test scheme that combines the merits of broadcast-based scan compression and built-in self-test (BIST) is proposed. The proposed scheme not only generates deterministic test patterns, similar to scan compression, but also requires fewer test pins, similar to BIST, thereby increasing the efficiency of multisite testing. With appropriately designed linear-feedback shift registers (LFSRs) and seeds periodically inserted through the reduced test pins, the test application process combines deterministic testing and random testing. The random testing stage can compensate the seed-insertion time for the subsequent deterministic testing stage and reduce the number of required deterministic patterns by detecting easy-to-detect faults. For further reduction of the volume of test data and hardware overhead, some compatible scan chains (SCs) are grouped and receive test data from an LFSR. The experimental results show that the proposed method increases the efficiency of multisite testing with a reduced pin count. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool testRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that testRNN has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, testRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing. [ABSTRACT FROM AUTHOR]

Composite materials are obtained by combining two or more materials with different physical properties, which maintain their properties at the macromolecular level, thus retaining their identity, to obtain a resulting material with characteristics clearly superior to the constituent elements, comparable to conventional materials in terms of performance. The specimens are made either by injection molding, machined from the center sections of a standard test sample (see ISO 20735: 2018), processed from semifinished, finished, rolled, or extruded products. In this sense, the article briefly presents the activity of creating a modeling, simulation. and tensile testing software program that can acquire control and process data, taken from an experimental stand or from a software simulation. Thus, the following elements are realized: software for retrieving data from the software equipment of the Zwick/Roell Z020 traction test machine; taking data from an experimental stand; simulation of tensile testing; creating reports, management of the production logistics chain of the benchmark. The experimental and theoretical results are further input in the design of the presented software. [ABSTRACT FROM AUTHOR]

To overcome the financial and operational barriers often associated with district metered areas in North America, and to develop relevant benchmarking metrics, a multiyear water loss testing program using a mobile testing unit was undertaken in Ontario, Canada. [ABSTRACT FROM AUTHOR]

Software testing is gaining much more importance in the software development life cycle. Many companies are investing in several testing methods and especially in test automation. Nevertheless, it is quite difficult to choose the best tool or framework to use. In this study, we will do an evaluation and comparison of Katalon Studio, Selenium and UFT in a web application. This is the preliminary work to achieve some solid outcomes in order to propose and create a new general framework for test and code optimisation. [ABSTRACT FROM AUTHOR]

Software industry has adopted automated testing widely. The most common method adopted is graphical user interface test automation for the functional scenarios to reduce manual testing and increase the repeatability. Reducing execution time and redundancy to achieve quality "go/no go decisions" provides rational for the executive management to allocate funds to adopt automation and invest in the setup including people, process, and tools to achieve faster time to market. There are a variety of practices engaged by testers, like frameworks, tools, methods, procedures, models, and technologies to achieve automation. Nonetheless, the actual effectiveness in terms of return on investment (ROI) is not known, though there are various formulas to calculate ROI of test automation. The factors that determine the ROI are maturity of test automation, purpose, or intent, picking right tests for automation, knowledge of the tester to derive test coverage, domain expertise, defining right metrics like defects found by automation runs, cost versus reduction of time, labor versus quality of scripts, repeatability, and results. These factors form the base of the questions designed for the survey. The paper presents a survey and analysis to understand the ROI of test automation from industry test professionals from both product and services organizations. [ABSTRACT FROM AUTHOR]

In an era of connectivity and automation, the vehicle industry is adopting numerous technologies to transform driver-centric vehicles into intelligent mechanical devices driven by software components. Software integration and network connectivity inherit numerous security issues that open the door for malicious attacks. Software security testing is a scalable and practical approach to identify systems’ weaknesses and vulnerabilities at an early stage and throughout their life-cycle. Security specialists recommend fuzz testing to identify vulnerabilities within vehicle software systems. Nevertheless, the randomness and blindness of fuzzing hinder it from becoming a reliable security tool. This article presents a vulnerability-oriented fuzz (VulFuzz) testing framework that utilizes security vulnerability metrics designed particularly for connected and autonomous vehicles to direct and prioritize the fuzz testing toward the most vulnerable components. While most gray-box fuzzing techniques aim solely to expand code coverage, the proposed approach assigns weights to ensure a thorough examination of the most vulnerable components. Moreover, we employ an input structure-aware mutation technique that can bypass vehicle software systems’ input formats to boost test performance and avoid dropped test cases. Such a testing technique will contribute to the quality assurance of vehicle software engineering. We implemented the proposed approach on OpenPilot, a driver assistance system, and compared our results to American fuzzy lop (AFL) and an unguided mutation-based fuzzer. Within 16.8 h, VulFuzz exposed 335 crashes, 41 times more than AFL and two times more than an unguided mutation-based fuzzer. VulFuzz is explicitly efficient for automotive systems, reaching the same code coverage as AFL but with more exposed crashes and fewer dropped messages. [ABSTRACT FROM AUTHOR]

Symbolic execution is a powerful technique for automating software testing to detect many types of errors such as memory errors and assertion violations. However, it encounters the problem of path explosion, and by using only assertions, it still lacks the capability of going deep into checking the functional correctness of a path based on corresponding formal specifications. To address these problems, we propose a specification-based incremental testing method with symbolic execution, called SIT-SE, providing a much more rigorous way to automatically check the functional correctness of all the discovered program paths, by introducing theorems (instead of assertions) for path correctness and branch sequence coverage algorithm for guiding a moderate path exploration. Compared with Hoare logic for proving the correctness of an entire program, a theorem in the SIT-SE is made for verifying the correctness of a program path. The proposed method carefully treats the relationship between a path condition and the specification in a theorem to restrict the monotonous path exploration, whereas traditional concolic testing methods roughly use one test data to determine the path correctness by assertions during long path searching. We use a classic case to demonstrate how the method works and conduct an experiment to evaluate the performance of both the proposed method and the commonly used well-known concolic testing tool KLEE. The experimental results show that our method SIT-SE is effective and outperforms KLEE in detecting faulty paths based on specifications. [ABSTRACT FROM AUTHOR]

Software changes constantly, because developers add new features or modifications. This directly affects the effectiveness of the test suite associated with that software, especially when these new modifications are in a specific area that no test case covers. This article tackles the problem of generating a high-quality test suite to cover repeatedly a given point in a program, with the ultimate goal of exposing faults possibly affecting the given program point. Both search-based software testing and constraint solving offer ready, but low-quality, solutions to this: Ideally, a maximally diverse covering test set is required, whereas search and constraint solving tend to generate test sets with biased distributions. Our approach, Diversified Focused Testing (DFT), uses a search strategy inspired by GödelTest. We artificially inject parameters into the code branching conditions and use a bi-objective search algorithm to find diverse inputs by perturbing the injected parameters, while keeping the path conditions still satisfiable. Our results demonstrate that our technique, DFT, is able to cover a desired point in the code at least 90% of the time. Moreover, adding diversity improves the bug detection and themutation killing abilities of the test suites. We show that DFT achieves better results than focused testing, symbolic execution, and random testing by achieving from 3% to 70% improvement in mutation score and up to 100% improvement in fault detection across 105 software subjects. [ABSTRACT FROM AUTHOR]

Copyright of Investigación e Innovación en Ingenierías is the property of Universidad Simon Bolivar and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Measuring and testing the scalability and performance of cloud-based software services is critical for the delivery of such services, and the development of cloud computing. There are three interconnected Cloud-based software services' performance aspects; both elasticity and efficiency are depending on the delivery of a sufficient level of scalability performance. In this work, we focused on testing and measuring the cloud-based software services scalability from a technical perspective. This paper uses technical scalability metrics that address both volume and quality scaling, inspired by earlier technical metrics of elasticity. We demonstrate the application of the metrics using a practical example and three demand scenarios and discuss the importance of these metrics. We show how our technical scalability metrics integrated into an earlier utility-oriented metric of scalability, in order to enable the scalability analysis from technical and production-driven perspectives. This work is related to the delivery of cloud-based software services from scalability perspective. We demonstrate our metrics by testing a cloud-based software service connecting with both AWS auto-scaling and load-balancing services. Measurements have been collected and analysis, and then been used to calculate the values of our scalability metrics. [ABSTRACT FROM AUTHOR]

The benefit of exploratory testing and ad hoc testing by tester's experience is that crucial bugs are found quickly. Regression testing and test case prioritization are important processes of software testing when software functions have been changed. We propose a test path prioritization method to generate a sequence of test paths that would match the testers' interests and focuses on the target area of interest or on the changed area. We generate test paths form the activity diagrams and survey the test path prioritization from testers. We define node and edge weight to the symbols of activity diagrams by applying Time management, Pareto, Buffett, Binary, and Bipolar method. Then we propose a test path score equation to prioritize test paths. We also propose evaluation methods i.e., the difference and the similarity of test path prioritization to testers' interests. Our proposed method had the least average of the difference and the most average of the similarity compare with the tester's prioritization of test paths. The Bipolar method was the most suitable for assigning weights to match test path rank by the tester. Our proposed method also has given the affected path by changing area higher priority than the other test path. [ABSTRACT FROM AUTHOR]

Though Deep Neural Networks (DNNs) have been widely deployed and achieved great success in many domains, they have severe safety and reliability concerns. To provide testing evidence for DNNs' reliable behaviors, various coverage testing techniques inspired by traditional software testing have been proposed. However, the coverage criteria in these techniques are either not fine enough to capture subtle behaviors of DNNs, or too time-consuming to be applied on large-scale DNNs. In this paper, we propose a coverage testing framework named HashC, which makes mainstream coverage criteria (e.g., NC and KMNC) much finer. Meanwhile, HashC reduces the time complexity of combinatorial coverage testing from polynomial time to linear time. We also develop the corresponding test sample selection method. Our experiments show that, (1) the existing mainstream coverage criteria are becoming finer after being equipped with HashC, (2) HashC greatly accelerates combinatorial coverage testing and can handle the testing of large-scale DNNs. [ABSTRACT FROM AUTHOR]

Spectrum-based fault localization (SBFL) techniques can automatically localize software faults. They employ the program spectrum, such as code coverage profile with test verdicts, to rank the program entities based on their code suspiciousness. In the past decades, researchers have proposed many approaches to optimize these techniques; however, the program structure, which can influence their performance, is not taken into consideration in developing and improving these techniques. In this article, we identify and analyze the effect of the program structure on the application of SBFL techniques. We observe that some specific program structures may introduce structure bias to code suspiciousness and negatively influence the output of SBFL techniques. To mitigate these effects and improve the performance of fault localization, we propose Delta4Ts, a structure-aware technique. Delta4Ts references debugging history to alleviate the impact of structure bias in the calculation of code suspiciousness. It reasons from the observable suspicious value towards the desired suspicious value and the impact of structure bias. To evaluate Delta4Ts under practical constraints, we conduct a controlled experiment using nine widely-studied SBFL formulae on 12 C programs and 6 Java programs. The experiment results show that Delta4Ts can significantly improve the accuracy of the studied SBFL formulae by an average of 34.8% on 12 C programs and 30.6% on 6 Java programs, and improve more on subject programs associated with more history versions or having larger code sizes. [ABSTRACT FROM AUTHOR]

Test case prioritization (TCP) is a popular regression testing technique in software engineering field. The task of TCP is to schedule the execution order of test cases so that certain objective (e.g., code coverage) can be achieved quickly. In this article, we propose an efficient ant colony system framework for the TCP problem, with the aim of maximizing the code coverage as soon as possible. In the proposed framework, an effective heuristic function is proposed to guide the ants to construct solutions based on additional statement coverage among remaining test cases. Besides, a sorting-based local search mechanism is proposed to further accelerate the convergence speed of the algorithm. Experimental results on different benchmark problems, and a real-world application, have shown that the proposed framework can outperform several state-of-the-art methods, in terms of solution quality and search efficiency. [ABSTRACT FROM AUTHOR]

One of the major challenges in software testing is the generation of test data automatically that satisfy a specified adequacy criterion. This paper presents a GA-based approach and a supporting tool for data-flow test data generation for ASP.NET web applications. The proposed tool accepts as input the web application under test, instruments it, and performs static analysis to compute the definition-use pairs. The proposed GA conducts its search by constructing new test data from previously generated test data that are evaluated as effective test data. In this GA, the chromosome is a collection of user interface control objects, where each control is considered as a gene. Therefore, novel crossover and mutation operators are developed to manipulate the chromosome, which are called block crossover and control-based mutation operators. The proposed GA accepts as input the instrumented version, the list of definition-use pairs to be covered, and input controls related information. The tool produces a set of test cases, the set of definition-use pairs covered by each test case, and a list of uncovered definition-use pairs, if any. Also the paper presents a case study to illustrate how the tool works. Finally, it presents the results of the empirical evaluation that is performed to evaluate the effectiveness of the generated test data in exposing web application errors. [ABSTRACT FROM AUTHOR]

For the test case prioritization problems, the average percent of faults detected (APFD) and its variant versions are widely used as metrics to evaluate prioritized test suite's efficiency of fault detection. By a revisit of metrics for test case prioritization, we observe that APFD is only available for the scenarios where all test suites under evaluation contain the same number of test cases. Such a limitation is often overlooked, and lead to incorrect results when comparing fault detection efficiency of test suites with different sizes. Moreover, APFD cannot precisely illustrate the process of fault detection in the real world. Besides the APFD, most of its variants, including the NAPFD and the APFD C W , have similar problems. This paper points out these limitations in detail by analyzing the physical explanation of APFD series metrics formally. In order to eliminate these limitations, we propose a series of improved metrics, including the relative average percent of faults detected (RAPFD) and the relative cost-cognizant weighted average percent of faults detected (RAPFD C W ), to evaluate the efficiency of the test suite. Furthermore, for the scenario of parallel testing, a series of metrics including the relative average percent of faults detected in parallel testing (P -RAPFD) and the relative cost-cognizant weighted average percent of faults detected in parallel testing (P -RAPFD C W ) are proposed too. All the proposed metrics refer to both the speed of fault detection and the constraint of the testing resource. A formal analysis and some examples show that all the proposed metrics provide much more precise illustrations of the fault detection process. [ABSTRACT FROM AUTHOR]

Goal. Development of the experimental installation for testing of the software and hardware tools of management of a photovoltaic system of a local facility connected to the grid, which, along with checking its operability, will allow analyzing the operation and adjusting the settings of systems with different parameters under different weather conditions, load schedules and operating modes. Method. Experimental research with preliminary justification of settings based on calculations and simulation data obtained in the Matlab software package. Results. The structure and experimental installation on the base of the photovoltaic system with storage battery and using a photovoltaic battery emulator with software control of generation and load power were developed. The software modules of the system have been developed. According to the results of the simulation, the moments of time are determined when the operating mode is switched in the experimental setup. The experimental setup was tested in static and dynamic modes. Static tests were aimed at checking the functioning of the photovoltaic battery emulator in accordance with the specified solar radiation power and the temperature of the photo module, as well as checking the correctness of the model used in the program. Tests in statics with fixed parameters and in dynamics in the daily cycle, in general, confirmed the efficiency of the proposed solutions. Scientific novelty. The method of experimental research in the daily cycle of photovoltaic systems control with the use of software-controlled emulators of a photovoltaic battery and load with power and time scaling was developed. This, along with the acceleration of the experiment, allows us to estimate the effectiveness of the tested systems at different ratios of system parameters and adjust the functioning algorithms. Practical importance. The proposed solutions for the implementation of the experimental installation allow minimizing material costs and time costs in the process of testing and development of software and hardware control tools of a photovoltaic system with storage battery. [ABSTRACT FROM AUTHOR]

Today, there are millions of third-party Android applications. Some of them are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic analysis are being developed by the Android community. Code coverage is one of the most common metrics for evaluating effectiveness of these frameworks. Furthermore, code coverage is used as a fitness function for guiding evolutionary and fuzzy testing techniques. However, there are no reliable tools for measuring fine-grained code coverage in black-box Android app testing. We present the Android Code coVerage Tool, ACVTool for short, that instruments Android apps and measures code coverage in the black-box setting at class, method and instruction granularity. ACVTool has successfully instrumented 96.9% of apps in our experiments. It introduces a negligible instrumentation time overhead, and its runtime overhead is acceptable for automated testing tools. We demonstrate practical value of ACVTool in a large-scale experiment with Sapienz, a state-of-the-art automated testing tool. Using ACVTool on the same cohort of apps, we have compared different coverage granularities applied by Sapienz in terms of the found amount of crashes. Our results showthat none of the applied coverage granularities clearly outperforms others in this aspect. [ABSTRACT FROM AUTHOR]

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction. [ABSTRACT FROM AUTHOR]

Software failures caused by data race bugs have always been major concerns in parallel and distributed systems, despite significant efforts spent in software testing. Due to their nondeterministic and hard-to-reproduce features, when evaluating systems’ operational reliability, a rather long period of experimental execution time is expected to be spent on observing failures caused by data race conditions. To address this problem, in this paper, we make two contributions. First, this paper proposes stress testing with influencing factors, in which the system runs under certain workloads for a long time with controlled stress conditions to accelerate the occurrence of data race failures. Second, it explores and formulates mathematical relationship models between data races’ statistical characteristics of time to failure (TTF) or mean TTF (MTTF) and the influencing factors. Such relationship models are used for TTF/MTTF extrapolation under different operational conditions and are essential to reduce systems’ reliability evaluation time. The proposed method is empirically evaluated on six applications suffering from failures caused by real-world data race bugs. Through analysis of the experimental results, we obtain several important findings: First, the reduction in the manifestation time to data race failures achieved by controlling the influencing factors is statistically significant. Second, Power model is the best-fitting model of the relationship between the MTTF and the influencing factors. Third, Power Weibull distribution is the best-fitting probability distribution between the TTF and the influencing factors. Finally, the TTF/MTTF can be accurately estimated with the approach proposed in this paper. [ABSTRACT FROM AUTHOR]

Adaptive random testing (ART) was developed as an enhanced version of random testing to increase the effectiveness of detecting failures in programs by spreading the test cases evenly over the input space. However, heavy computation may be incurred. In this paper, three enhanced algorithms for fixed-size-candidate-set ART (FSCS-ART) are proposed based on the $k$ -dimensional tree (KD-tree) structure. The first algorithm Naive-KDFC constructs a KD-tree by splitting the input space with respect to every dimension successively in a round-robin fashion. The second algorithm SemiBal-KDFC improves the balance of the KD-tree by prioritizing the splitting according to the spread in each dimension. In order to control the number of traversed nodes in backtracking, the third algorithm LimBal-KDFC introduces an upper bound for the nodes involved. Simulation and empirical studies have been conducted to investigate the efficiency and effectiveness of the three algorithms. The experimental results show that these algorithms significantly reduce the computation time of the original FSCS-ART for low dimensions and for the case of high dimensions with low failure rates. The efficiency of SemiBal-KDFC is better than that of Naive-KDFC when the dimension is no more than 8, but LimBal-KDFC is the most efficient of all three. Although the limited backtracking leads only to an approximate nearest neighbor in LimBal-KDFC, its failure-detection effectiveness is, in fact, better than FSCS-ART in high-dimensional input spaces and has no significant deterioration in low-dimensional spaces. [ABSTRACT FROM AUTHOR]

There are multiple software testing techniques in the literature. These are white, black and gray box test techniques. Each of these has different tasks and purposes. Technical selection is decisive for these tasks and objectives. In this paper, information about white box technique and approaches are described. In addition, multiple test instruments that are currently in use are examined and a comparative view is provided. As a result of this comparison, a tabular chart is presented with an information set. This information set is intended to be valuable for software testing professionals and researchers to appreciate many tools at a glance. [ABSTRACT FROM AUTHOR]

• A custom made measurement system for on-line testing of electromechanical relays. • Measurement of essential magnetic and contact properties in less than two seconds. • Measurement algorithms based on simultaneous waveform generation and acquisition. • The comprehensive relay test software application. An automatic measurement system has been developed for high-speed testing of the most important magnetic and contact properties for elementary monostable and bistable miniature electromechanical relays. The system design is based on the virtual instrumentation approach, which is a combination of software and hardware. The comprehensive test application was developed using the LabVIEW graphical programming environment, while the used hardware is based on commercial measurement equipment and a custom developed device. The presented measurement methods are based on simultaneous generation and acquisition of analog and digital waveforms with the high-speed data acquisition device. The implemented automatic relay measurement system was validated metrologically and tested successfully with all supported relay types. The results show that the developed system is capable of performing accurate measurements of the coil resistance, contact resistance, time parameters, and operate/release values in less than two seconds. [ABSTRACT FROM AUTHOR]

Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing, where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing, to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing. [ABSTRACT FROM AUTHOR]

The expansion of Internet of Things and migration to cloud computing, highlights the need for testing from the software engineering phase and each module that is part of them. This comes from the fact that each application, or module, or device is developed by a different provider, making software testing more challenging. We will evaluate the need of unit testing for Internet of Things from the point of view of a tester, especially a QA (quality assurance) tester. In the conclusion we will highlight the need of testing for IoT and further research. [ABSTRACT FROM AUTHOR]

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different state-of-the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method. [ABSTRACT FROM AUTHOR]

Abstract: The problem of testing and debugging learning neural network systems is discussed. Differences of these systems from program implementations of algorithms from the point of view of testing are noted. Requirements to the testing systems are identified. Specific features of various neural network models from the point of view of selection of the testing technique and determination of tested parameters are analyzed. It is discussed how to get rid of the noted drawbacks of the systems under study. The discussion is illustrated by an example. [ABSTRACT FROM AUTHOR]

New processing models are being adopted in Big Data engineering to overcome the limitations of traditional technology. Among them, MapReduce stands out by allowing for the processing of large volumes of data over a distributed infrastructure that can change during runtime. The developer only designs the functionality of the program and its execution is managed by a distributed system. As a consequence, a program can behave differently at each execution because it is automatically adapted to the resources available at each moment. Therefore, when the program has a design fault, this could be revealed in some executions and masked in others. However, during testing, these faults are usually masked because the test infrastructure is stable, and they are only revealed in production because the environment is more aggressive with infrastructure failures, among other reasons. This paper proposes new testing techniques that aimed to detect these design faults by simulating different infrastructure configurations. The testing techniques generate a representative set of infrastructure configurations that as whole are more likely to reveal failures using random testing, and partition testing together with combinatorial testing. The techniques are automated by using a test execution engine called MRTest that is able to detect these faults using only the test input data, regardless of the expected output. Our empirical evaluation shows that MRTest can automatically detect these design faults within a reasonable time. [ABSTRACT FROM AUTHOR]

Combinatorial testing can test software that has various configurations for multiple parameters efficiently. This method is based on a set of test cases that guarantee a certain level of interaction among parameters. Mixed covering array (MCA) can be used to represent a test-suite. Each row of the array corresponds to a test case. In general, a smaller size of MCA does not necessarily imply less testing time. There are certain combinations of parameter values which would take much longer time than other cases. Based on this observation, it is more valuable to construct MCAs that are better in terms of testing effort characterization other than size. We present a method to find cost-aware MCAs. The method contains two steps. First, simulated annealing algorithm is used to get an MCA with a small size. Then we propose a novel nested differential evolution algorithm to improve the solution with its testing effort. The experimental results indicate that our method succeeds in constructing cost-aware MCAs for real-world applications. The testing effort is significantly reduced compared with representative state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Research regarding MAT (Mobile Application Testing) in Pakistan is hard to discover and to the best of our knowledge, no work has been done in surveying MAT in Pakistan. In this work, we have examined the current trend and status of MAT in Pakistan. Main objective was to investigate to what extent MAT is currently applied in Pakistan software companies and what experience the companies have with using MAT. Furthermore, efforts were made to find out what testers think about MAT, e.g. issue, advantages and disadvantages of MAT, what factors affects MAT and how they plan to improve MAT. In order to achieve our objectives, we used a comprehensive online survey so we converted our research questions into correspondence survey questions. We served a questionnaire of the survey to 66 testing relevant officials of leading software companies in different cities of Pakistan to develop a model study about general trend and status of MAT which can be generalized all over Pakistan. We received 56 replies in total after over 2 months. After that, we used SPSS tool to analyze the replies of this questionnaire. Cross-Tabulation Analysis and Pearson Chi-square tests have been computed to examine the results. We found some interesting results on current status and practice of MAT in Pakistan software companies. [ABSTRACT FROM AUTHOR]

Demand for highly reliable software is increasing day by day which in turn has increased the pressure on the software firms to provide reliable software in no time. Ensuring high reliability of the software can only be done by prolonged testing which in result consumes more resources which is not feasible in the existing market situation. To overcome this, software firms are providing patches after software release so as to fix the remaining number of bugs and to give better product experience to users. An update/fix is a minor portion of software to repair the bugs. With such patches, organizations enhance the performance of the software. Delivering patches after release demands extra effort and resources which are costly and hence not economical for the firms. Also, early patch release might cause improper fixation of bugs, on the other hand, delayed release may increase the chances of more failure during the operational phase. Therefore, determining optimal patch release time is imperative. To overcome the above issues we have formulated a two-dimensional time and effort-based cost model to regulate the optimum release and patch time of software, so that the total cost is minimized. Proposed model is validated on a real life data set. [ABSTRACT FROM AUTHOR]

With the intensive and large-scale development of cloud computing, software testing has become one of the most important problems. How to evaluate and assess the testing process of a cloud computing system is a key to the deployment and use of it. This paper proposes a method to generate and optimize test paths of cloud application based on SDN, which separates the cloud service testing process from the underlying execution logic, thus improving the scalability of the testing process. The finite state machine (FSM) is used to establish the formal description language of the cloud application testing process, and is also used to model the basic elements, such as cloud services, jobs, test cases and cloud applications, and construct a test model for cloud application. The related theory of FSM is used to analyze the effectiveness and correctness of cloud application test model. Based on the actual mapping of the model, the generation method of test path is also given. In addition, we analyze the coverage of test cases and propose the test path optimization methods to improve the test efficiency. The specific examples and simulations show that this method can simplify the design and analysis of the cloud application testing process and effectively improve the generation of test paths. [ABSTRACT FROM AUTHOR]

Software testing is an important means to ensure the quality of software. However, there is a lack of effective modeling and optimization of CPS software testing. In this paper, Petri nets are used to model the underlying devices, components, connectors and test cases of CPS software. Aspect oriented programming extracts the crosscutting concerns of CPS software testing. The behaviors and their relationships are described based on AOP, and a weaving mechanism is used to dynamically integrate these models into the test model of CPS. The correctness of the model is analyzed by using the operation semantics and related theories of Petri nets. Based on the state space of constructed model, a strategy is proposed to dynamically select the test suite. The experiment results show that this method can effectively describe the CPS software testing process, which can improve the quality of software testing. [ABSTRACT FROM AUTHOR]

Comparing behaviours of program models has become an important task in software maintenance and regression testing. Combinatorial testing focuses on recognizing faults that happen due to interaction of values of a small number of input parameters. In this paper presents the Black-Box Regression Maximization (BBM) Algorithm with Density-based Spatial Clustering Algorithm (DSC) using Greedy Search optimization algorithm focuses on combinatorial testing and proactively exposes behavioural deviations by checking inside block transitions. In this method presents new approach of BBM with Internal block transitions to measure the dissimilarity statements in large program data. To identify specific faults, an adaptive testing rule repeatedly constructs and tests configurations in order to determine, for each interaction of interest, whether it is faulty or not. [ABSTRACT FROM AUTHOR]

During the unit testing of model-driven tools, a large number of models and test classes needs to be managed and maintained. Typically, some of these artifacts are specified manually, some are generated automatically. Existing approaches to test management rely on the available visual and textual modeling notations. As these notations are not tailored to unit testing, distinct maintainability trade-offs arise. In this paper, we propose a notation that aims to combine the benefits of visual and text-based approaches. The notation is at the same time visual and text-based, as it uses ASCII characters to emulate the familiar graphical notations. In our evaluation based on real models, we identify problematic model shapes challenging the scalability our notation, while finding that it is well-suited to capture typical test models. [ABSTRACT FROM AUTHOR]

Previous research in defect prediction has proposed approaches to determine which files require additional testing resources. However, practitioners typically create tests at a higher level of abstraction, which may span across many files. In this paper, we study software testing, especially test resource prioritization, from a different perspective. We use topic models to generate topics that provide a high-level view of a system, allowing developers to look at the test case coverage from a different angle. We propose measures of how well tested and defect prone a topic is, allowing us to discover which topics are well tested and which are defect prone. We conduct case studies on the histories of Mylyn, Eclipse, and NetBeans. We find that 34–78% of topics are shared between source code and test files, indicating that we can use topic models to study testing; well-tested topics are usually less defect prone, defect-prone topics are usually undertested; we can predict which topics are defect prone but not well tested with an average precision and recall of 75% and 77%, respectively; our approach complements traditional prediction-based approaches by saving testing and code inspection effort; and our approach is not particularly sensitive to the parameters that we use. [ABSTRACT FROM PUBLISHER]