Search

Test suite reduction is an effective way to reduce the cost of regression testing by identifying and removing redundant test cases from the original test suite. In this paper, we propose a novel cluster test suite reduction using an evolutionary multi-objective optimization algorithm. Specifically, we use a K-means algorithm to group similar test cases to the same cluster. Then the evolutionary algorithm is used to remove redundant test cases based on the clustering results, and optimization objects are represented as the coverage-related criteria, fault-related criteria and cost-related criteria. The experimental results involving eight subject programs show that the proposed method can outperform the other three state-of-the-arts with respect to both fault detection (4.61% -9.44%) and reduction ratio (4.10% -10.64%). Meanwhile, the experiments also prove that our method has a better performance of missing failure rate (0.049% -0.132%) and code coverage rate (3.34% -6.10%). Besides, the proposed method costs are found to be comparable to the other techniques.

Software defect prediction (SDP) plays a significant part in identifying the most defect-prone modules before software testing and allocating limited testing resources. One of the most commonly used classifiers in SDP is naive Bayes (NB). Despite the simplicity of the NB classifier, it can often perform better than more complicated classification models. In NB, the features are assumed to be equally important, and the numeric features are assumed to have a normal distribution. However, the features often do not contribute equivalently to the classification, and they usually do not have a normal distribution after performing a Kolmogorov-Smirnov test; this may harm the performance of the NB classifier. Therefore, this paper proposes a new weighted naive Bayes method based on information diffusion (WNB-ID) for SDP. More specifically, for the equal importance assumption, we investigate six weight assignment methods for setting the feature weights and then choose the most suitable one based on the F-measure. For the normal distribution assumption, we apply the information diffusion model (IDM) to compute the probability density of each feature instead of the acquiescent probability density function of the normal distribution. We carry out experiments on 10 software defect data sets of three types of projects in three different programming languages provided by the PROMISE repository. Several well-known classifiers and ensemble methods are included for comparison. The final experimental results demonstrate the effectiveness and practicability of the proposed method.

The number of mobile applications has increased geometrically nowadays, but how to ensure their quality and conduct adequate and effective testing is still a challenge for developers. On the one hand, the number of mobile apps is increasing, and the update speed is faster and faster. Many small and medium-sized companies can hardly test the app adequately before each release. On the other hand, mobile apps play more and more important roles in people’s life, such as financial payment. For the sake of company security and user privacy, most companies will encrypt key codes in their APP. Even third-party testers cannot get source code, which also leads to many researchers cannot carry out further research and effective testing for these widely used mainstream APPs. Code coverage is an important indicator to guide software testing, which plays a crucial role in ensuring the quality of testing. However, it is an urgent problem to find accurate coverage indictors to evaluate these tests. And when testing those existing widely used mainstream closed-source apps, we find that the existing coarse-grained coverage metrics like method coverage is bad coverage indictors for app testing that can exaggerate or minimize the actual coverage rate, which cannot obtain satisfactory results for the evaluation of test effects. To find a more reliable coverage indictor, this paper demonstrates the correctness of instruction coverage indictor and the inaccurate of method coverage in evaluating the test of closed-source APP from the perspective of probability and statistics. Then we shows how inaccurate the method coverage can be through an empirical evaluation on datasets of closed source APPs and open source APPs respectively. It is further verified that instruction coverage is a more effective evaluation indictor than methods coverage or activity coverage in the test of closed source APP for the first time.

With the rise of crowdsourced software testing in recent years, the issuers of crowd test tasks can usually collect a large number of test reports after the end of the task. These reports have insufficient validity and completeness, and manual review often takes a lot of time and effort. The crowdsourced test task publisher hopes that after the crowdsourced platform collects the test report, it can analyze the validity and completeness of the report to determine the severity of the report and improve the efficiency of crowdsourced software testing. In the past ten years, researchers have used various technologies (such as natural language processing, information retrieval, machine learning, deep learning) to assist in analyzing reports to improve the efficiency of report review. We have summarized the relevant literature of report analysis in the past ten years, and then classified from report classification, duplicate report detection, report prioritization, report refactoring, and summarized the most important research work in each area. Finally, we propose research trends in these areas and analyze the challenges and opportunities facing crowdsourced test report analysis.

The generation of multiple-path test cases can greatly enhance the efficiency of path-wise testing. Various methods adopting meta-heuristic algorithm to generate multiple-path test cases have been proposed, but existing methods focus on improving the meta-heuristic algorithm to get better test case generation efficiency, and test cases covering each path needs to be generated by meta-heuristic algorithm searching. To improve efficiency, a test case generation method for multiple-path coverage is proposed in this study, which combines a particle swarm optimisation (PSO) algorithm with metamorphic relations (MRs). The method first generates a test case using the PSO algorithm, and then generates new test cases by repeatedly using MRs between test cases. This method reduces evolving numbers of PSO algorithm. The experimental results show that the proposed method can significantly enhance the efficiency in terms of fitness evaluations and time consumption.

It is difficult to apply traditional testing adequacy criteria when measuring the adequacy of convolutional neural network applications. However, only a small number of test cases applied to the CNN model can achieve neuron coverage of almost 100%, overturning the effectiveness of the neuronal coverage criteria. In this paper, we propose a model coverage criterion based on mutation testing for CNN, and applying model coverage criterion to a common CNN image classification models (LeNet-5). we focus on the testing accuracy of model. Experiments show that our method can find the local optimal model and play an important role in improving the testing adequacy of the set of models.

As a vehicle for recording and tracking defects, bug report provides basis for solving software quality problems. However, moving fake or duplication bug report in multi-person and parallel software testing project is a labor-intensive job. Therefore, this paper proposes a method based on vector space model for automatic dealing with this problem. We built a matching library according to the test requirements and confirmed bug reports and used vector space model to calculate the similarity between the bug report and the matching library. Then the correctness of the bug report is detected based on this similarity.

One of the key problems in app testing is to improve the test coverage of apps. However, current testing techniques, whether in code coverage or activity coverage, are not satisfactory. To address this limitation, we present an algorithm for generating test runtime-environment-set to exercise mobile apps. Our approach is based on code analysis to systematically test the targeted code of the Android apps. It analyzes the decompiled code that identifies the code related to Android SDK version, generating the corresponding test cases with the runtime-environment set. We also implement our approach on Android, and validate the method with the existing widely used strategies. An empirical study of the practical usefulness of the technique has been presented on 6 widely-used industrial apps. 18 unique crashes have been found, and the method coverage has been increased by far 9.8% to 130.4% on those apps.

In order to enhance efficiency, a novel test cases generation method for multiple paths is proposed in this paper. It utilizes metamorphic relation to generate other test cases, after a test case is generated by PSO algorithm. The experimental results show that the method has better performance

Software defect prediction (SDP) plays a significant part in identifying the most defect-prone modules before software testing. Software attributes are the key to improve performance indices in the process of SDP. However, there exists redundant information in those attributes, and this may do harm to the accuracy of predictor. To address this issue, a new attribute selection method (NASM) is proposed in this paper. Firstly, we compute the maximal information coefficient matrix between the attributes in the method, and then these attributes are clustered by spectral clustering according to the maximal information coefficient matrix. In order to implement automatic clustering, Calinski-Harabasz criterion is used to find optimal number of clusters in the process of clustering. Consequently, the best set of attributes is selected. The proposed method is evaluated on five public datasets respectively, and compared with other four attribute selection strategies. The experimental results show that the proposed method NASM outperforms the other three attribute selection strategies on the five standard datasets.

A software security testing behavior model, SETAM, was proposed in our previous work as the integrated model for describing software security testing requirements behavior, which is not only compatible with security functions and latent typical misuse behaviors, but also with the interaction of them. In this paper, we analyze the differences between SETAM with security use case and security misuse case in different types of security test requirements. To illustrate the effectiveness of SETAM, we compare them in a practical case study by the number of test cases and the number of faults detected by them. The results show that SETAM could decrease about 34.87% use cases on average, and the number of faults detected by SETAM increased by 71.67% in average, which means that our model can detect more faults with fewer test cases for software security testing.

Using functional diagram to represent GUI (Graphical User Interface) can promote the efficiency of GUI testing effectively. In this paper, a method of extracting formalized GUI and coverage criteria based on function-diagram are suggested, and the corresponding coverage percentage algorithm is designed. It is proved that the more massive of the GUI windows are, the higher the coverage and implementation efficiency is.

Due to the increasing complexity of software applications, traditional function security testing ways, which only test and validate software security mechanisms, are becoming ineffective to detect latent software security defects (SSD). However, the most of vulnerabilities result from some typical SSD. According to CERT/CC, ten defects known are responsible for 75% of security breaches in today software applications. On the base of threat tree modeling, we use it in the integrated software security test model. For introducing the usefulness of the method, we use the test model in M3TR software security test.

Due to the increasing complexity of Web applications, traditional function security testing, which only tests and validates software security mechanisms, is becoming ineffective to detect latent software security defects (SSD). The number of reported web application vulnerabilities is increasing dramatically. However, the most of vulnerabilities result from some typical SSD. Based on SSD, this paper presents an effective software security testing (SST) model, which extends traditional security testing process to defects behavior analysis which incorporates advantages of traditional testing method and SSD-based security testing methodology. Primary applications show the effectiveness of our test model.