Search

The complex and dynamic nature of Systems of Systems (SoSs) poses many challenges on their validation and testing, but so far few effective test strategies exist to address them. On the other hand, extensive research has been conducted in the testing of Software Product Lines (SPLs), which present interesting convergence points with SoSs, as both disciplines aim at reducing development costs and time-to-market thanks to extensive reuse of existing artifacts. In this paper, we outline commonalities and differences between the SoS and SPL paradigms from the point of view of testing and investigate how existing methods and tools from SPL testing could be leveraged to address the challenges of SoS testing.

Quality of Experience (QoE) refers to the end users level of satisfaction with a real-time service, in particular in relation to its audio and video quality. Advances in WebRTC technology have favored the spread of multimedia services through use of any browser. Provision of adequate QoE in such services is of paramount importance. The assessment of QoE is costly and can be done only late in the service lifecycle. In this work we propose a simple approach for QoE-driven non-functional testing of WebRTC services that relies on the ElasTest open-source platform for end-to-end testing of large complex systems. We describe the ElasTest platform, the proposed approach and an experimental study. In this study, we compared qualitatively and quantitatively the effort required in the ElasTest supported scenario with respect to a "traditional" solution, showing great savings in terms of effort and time.

International audience; The recent push towards test automation and test-driven development continues to scale up the dimensions of test code that needs to be maintained, analysed, and processed side-by-side with production code. As a consequence, on the one side regression testing techniques, e.g., for test suite prioritization or test case selection, capable to handle such large-scale test suites become indispensable; on the other side, as test code exposes own characteristics, specific techniques for its analysis and refactoring are actively sought. We present JTeC, a large-scale dataset of test cases that researchers can use for benchmarking the above techniques or any other type of tool expressly targeting test code. JTeC collects more than 2.5M test classes belonging to 31K+ GitHub projects and summing up to more than 430 Million SLOCs of ready-to-use real-world test code.

We propose a factory of educational System of Systems (SoS) case studies that can be used for evaluating SoS research results, in particular in SoS testing. The factory includes a first set of constituent systems that can collaborate within different SoS architectures to accomplish different missions. In the paper, we introduce three possible SoSs and outline their missions. For more detailed descriptions, diagrams and the source code, we refer to the online repository of EDUFYSoS. The factory is meant to provide an extensible playground, which we aim to grow to include more systems and other missions with the support of the community. Ministero dell'Università e della Ricerca (Italia) SISMA 201752ENYB

Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research activity on this topic, no effort has been made on understanding the vocabulary of flaky tests. This work proposes to automatically classify tests as flaky or not based on their vocabulary. Static classification of flaky tests is important, for example, to detect the introduction of flaky tests and to search for flaky tests after they are introduced in regression test suites. We evaluated performance of various machine learning algorithms to solve this problem. We constructed a data set of flaky and non-flaky tests by running every test case, in a set of 64k tests, 100 times (6.4 million test executions). We then used machine learning techniques on the resulting data set to predict which tests are flaky from their source code. Based on features, such as counting stemmed tokens extracted from source code identifiers, we achieved an F-measure of 0.95 for the identification of flaky tests. The best prediction performance was obtained when using Random Forest and Support Vector Machines. In terms of the code identifiers that are most strongly associated with test flakiness, we noted that job, action, and services are commonly associated with flaky tests. Overall, our results provides initial yet strong evidence that static detection of flaky tests is effective.

Functional, structural and operational testing are three broad categories of software testing methods driven by the product functionalities, the way it is implemented, and the way it is expected to be used, respectively. A large body of the software testing literature is devoted to evaluate and compare test techniques in these categories. Although it appears reasonable to devise hybrid methods to merge their different strengths - because different techniques may complement each other by targeting different types of faults and/or using different artifacts - we still miss clear guidelines on how to best combine them. We discuss differences and limitations of two popular testing approaches, namely coverage-driven and operational-profile testing, belonging to structural and operational testing, respectively. We show why and how test coverage and operational profile can cross-fertilize each other, improving the effectiveness of structural testing or, conversely, the product reliability achievable by operational testing. © 2019, Springer Nature Switzerland AG.

Great advances in network technology and software engineering have triggered the development and spread of Systems of Systems (SoSs). The dynamic and evolvable nature of SoSs poses important challenges on the validation of such systems and in particular on their regression testing, aiming at assessing that run-time changes and evolutions do not introduce regression in SoS behavior. This paper outlines issues and challenges of regression testing of SoSs, identifying the main kinds of evolution that can impact on their regression testing activity. Furthermore, it presents a conceptual framework for governing the regression testing of SoSs. The proposed framework leverages the concept of an orchestration graph that describes the flow of test cases and sketches a solution for deriving a regression test plan according to test cases dependencies.

The introduction of new generation ICT systems into vehicles makes them highly connected with the external World. As drawback, vehicle becomes potentially vulnerable to security attacks. Here, we consider a scenario in which Vehicular Networks and a Urban Network work together to realize a defence mechanism based on Reputation Systems. In this way, we are able to identify and isolate possible malicious vehicles acting that could send messages with the aim of reducing the availability of the network. We propose Context Aware Reputation Systems, CARS, able to identify insider attackers and isolate them taking into account contextual conditions derived from sensors spread along the entire urban network. Then, we experimentally evaluate CARS on a real data-set of mobility traces of taxis in Rome to compare the proposed systems with existing ones that do not consider contextual conditions. The preliminary results obtained are promising and show the feasibility and potentiality of CARS.

White paper by the ElasTest team that provides an overview of the ElasTest project and its goals, while providing answers to industrial and research challenges.

While the relation between code coverage measures and fault detection is actively studied, only few works have investigated the correlation between measures of coverage and of reliability. In this work, we introduce a novel approach to measuring code coverage, called the operational coverage, that takes into account how much the program's entities are exercised so to reflect the profile of usage into the measure of coverage. Operational coverage is proposed as (i) an adequacy criterion, i.e., to assess the thoroughness of a black box test suite derived from the operational profile, and as (ii) a selection criterion, i.e., to select test cases for operational profile-based testing. Our empirical evaluation showed that operational coverage is better correlated than traditional coverage with the probability that the next test case derived according to the user's profile will not fail. This result suggests that our approach could provide a good stopping rule for operational profile-based testing. With respect to test case selection, our investigations revealed that operational coverage outperformed the traditional one in terms of test suite size and fault detection capability when we look at the average results.

In contemporary IT and Internet society, Cloud computing brings a paradigm shift not only for successfully evolving large distributed software applications but also for delivering a number of innovative services and applications. Cloud computing is an umbrella enveloping several models of service, like SaaS, IaaS, PaaS, TaaS. In particular, software testing in/on/over cloud is the focus of this paper. We see software testing in the Cloud and TaaS as a novel domain-specific approach to software testing. However, the field is still immature and there does not yet exist a clear understanding of its concepts and constituent technologies. In this paper we provide an introductory overview of software testing in the cloud, based also on a systematic search of literature, along with its buzzwords and tools.

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections. The techniques presented include: (a) structural testing using symbolic execution, (b) model-based testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) search-based testing. Each section is contributed by world-renowned active researchers on the technique, and briefly covers the basic ideas underlying the method, the current state of the art, a discussion of the open research problems, and a perspective of the future development of the approach. As a whole, the paper aims at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation, while ensuring a comprehensive and authoritative treatment.

With the aim of identifying good structures and examples for papers in the software engineering field, we conducted a study of the type of papers accepted along four decades in the Research Track of the International Conference on Software Engineering (ICSE). We used for this purpose a categorization scheme for Software Engineering papers that was obtained by merging, extending and revising a few existing paper scheme proposals. This paper summarizes some outcomes relative to what topics and problems are addressed, what types of contribution are presented and how they are validated. Insights from the study could help ICSE authors, reviewers and conference organizers in focusing and improving future efforts.

Having robust systems that behave properly even in presence of faults is becoming increasingly important. This is the case of the system we investigate in this paper, which is an embedded distributed system consisting of components that communicate with each other via messages exchange in the RBS (Radio Based Station) at Ericsson AB in Gothenburg, Sweden. Specifically, this paper describes a novel fault injection approach for testing the robustness of distributed embedded systems with very limited computation power. The new approach is inspired by Netflix's ChaosMonkey, a fault injection approach that has been developed for testing distributed systems hosted in the cloud. However, ChaosMonkey cannot be used in the context of RBS since the latter consists of small-embedded components with specific requirements of performance, programming language, and communication paradigm. This paper reports about the approach called Postmonkey we developed, illustrates the results of applying it to RBS, and discusses the potential of utilizing fault injection to test complex, embedded, and distributed systems. The approach and tool are now adopted by Ericsson.

We introduce covrel, an adaptive software testing approach based on the combined use of operational profile and coverage spectrum, with the ultimate goal of improving the delivered reliability of the program under test. Operational profile-based testing is a black-box technique that selects test cases having the largest impact on failure probability in operation; as such, it is considered well suited when reliability is a major concern. Program spectrum is a characterization of a program’s behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The driving idea of covrel is to complement operational profile information with white-box coverage measures based on count spectra, so as to dynamically select the most effective test cases for reliability improvement. In particular, we bias operational profile-based test selection towards those entities covered less frequently. We assess the approach by experiments with 18 versions from 4 subjects commonly used in software testing research, comparing results with traditional operational and coverage testing. Results show that exploiting operational and coverage data in a combined adaptive way actually pays in terms of reliability improvement, with covrel overcoming conventional operational testing in more than 80% of the cases.

2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), 25-29 July 2017, Prague (Czech Republic), Big Data programs are those that process large data exceeding the capabilities of traditional technologies. Among newly proposed processing models, MapReduce stands out as it allows the analysis of schema-less data in large distributed environments with frequent infrastructure failures. Functional faults in MapReduce are hard to detect in a testing/preproduction environment due to its distributed characteristics. We propose an automatic test framework implementing a novel testing approach called Ex Vivo. The framework employs data from production but executes the tests in a laboratory to avoid side-effects on the application. Faults are detected automatically without human intervention by checking if the same data would generate different outputs with different infrastructure configurations. The framework (MrExist) is validated with a real-world program. MrExist can identify a fault in a few seconds, then the program can be stopped, not only avoiding an incorrect output, but also saving money, time and energy of production resources, This work was supported in part by PERTEST (TIN2013-46928-C3-1-R), project funded by the Spanish Ministry of Science and Technology; TestEAMoS (TIN2016-76956-C3-1-R) and POLOLAS (TIN2016-76956-C3-2-R), projects funded by the Spanish Ministry of Economy and Competitiveness; GRUPIN14-007, funded by the Principality of Asturias (Spain); GAUSS (PRIN 2015, 2015KWREMX), funded by Italian MIUR; and ERDF funds

We introduce a new coverage measure, called the operational coverage, which is customized to the usage profile (count spectrum) of the entities to be covered. Operational coverage is proposed as an adequacy criterion for operational profile based testing, i.e., to assess the thoroughness of a black box test suite derived from the operational profile. To validate the approach we study the correlation between operational coverage of branches, statements, and functions, and the probability that the next test input will not fail. On the three subjects considered, we observed a moderate correlation in all cases (except a low correlation for function coverage for one subject), and consistently better results than traditional coverage measure.

Monitoring of business process execution has been proposed for the evaluation of business process performance. An important aspect to assess the thoroughness of the business process execution is to monitor if some entities have not been observed for some time and timely check if something is going wrong. We propose in this paper business process execution adequacy criteria and provide a proof-of-concept monitoring framework for their assessment. Similar to testing adequacy, the purpose of our approach is to identify the main entities of the business process that are covered during its execution and raise a warning if some entities are not covered. We provide a first assessment of the proposed approach on a case study in the learning context.

Test coverage information can be very useful for guiding testers in enhancing their test suites to exercise possible uncovered entities and in deciding when to stop testing. Since the concept of test criterion was born, several contributions have been made by both academia and industry in the definition and adaptation of adequacy criteria aiming at ensuring the discovery of more failures. Numerous contributions have also been done in the development of coverage tools. However, for complex applications that are reused in different contexts and for emerging paradigms (e.g., component-based development, service-oriented architecture, and cloud computing), traditional coverage metrics may no longer provide meaningful information to help testers on these tasks. Inspired by the idea of relative coverage this research focuses on the introduction of meaningful coverage metrics to cope with the challenges imposed by the current programming paradigms as well as on the definition of a theoretical framework for the development of relative coverage metrics.

Access Control is among the most important security mechanisms to put in place in order to secure applications. XACML is the de facto standard for storing and deploying access control policies. However, due to the complexity of the XACML language, policy definition becomes a difficult and error prone process. In recent years, the combined use of models for the access control policy specification, and the model-to-code facilities, for the automatic transformation of the model into the XACML language, has been proposed as a possible solution. These model-driven methodologies and facilities need to be thoroughly validated and verified. In this paper we provide an integrated framework for testing the automatic translation of the specification of an access control model into an XACML policy. The framework includes different test strategies for the derivation of test cases and some facilities for making easier their execution against the XACML policy and the test results collection and analysis. In addition, we illustrate the use of the framework on a case study.

XACML is the de facto standard for implementing access control policies. Testing the correctness of policies is a critical task. The test of XACML policies involves running requests and checking manually the correct response. It is therefore important to reduce the manual test effort by automatically selecting the most important requests to be tested. This paper introduces the XACML smart coverage selection approach, based on a proposed XACML policy coverage criterion. The approach is evaluated using mutation analysis and is compared on the one side with a not-reduced test suite, on the other with random and greedy optimal test selection approaches. We performed the evaluation on a set of six real world policies. The results show that our selection approach can reach good mutation scores, while significantly reducing the number of tests to be run. © 2014 IEEE.

Testing takes a considerable amount of the time and resources spent on producing software. It would therefore be useful to have ways 1) to reduce the cost of testing and 2) to estimate this cost. In particular, the number of tests to be executed is an important and useful attribute of the entity "testing effort". All-uses coverage is a data flow testing strategy widely researched in recent years. In this paper we present spanning sets of duas for the all-uses coverage criterion. A spanning set of duas is a minimum set of duas (definition-use associations) such that a set of test paths covering them covers every dua in the program. We give a method to find a spanning set of duas using the relation of subsumption between duas. Intuitively, there exists a natural ordering between the duas in a program: some duas are covered more easily than others, since coverage of the former is automatically guaranteed whenever the latter are covered. Those duas that are the most difficult to be covered according to this ordering are called unconstrained. A spanning set of duas is composed of unconstrained duas. Our results are useful for reducing the cost of testing, since the generation of test paths can be targeted to cover the smaller spanning set of duas, rather than all those in a program. On the other hand, assuming that a different path is taken to cover each dua in a spanning set, the cardinality of spanning sets can be used to estimate the cost of testing. Other interesting uses of spanning sets of duas are also discussed.