Search

In recent years, Service Oriented Architecture (SOA) has been increasingly adopted to develop distributed applications in the context of the Internet. To develop reliable SOA-based applications, an important issue is how to ensure the quality of web services. In this paper, we propose a dynamic random testing (DRT) technique for web services, which is an improvement over the widely-practiced random testing (RT) and partition testing (PT). We examine key issues when adapting DRT to the context of SOA, including a framework, guidelines for parameter settings, and a prototype for such an adaptation. Empirical studies are reported where DRT is used to test three real-life web services, and mutation analysis is employed to measure the effectiveness. Our experimental results show that, compared with the three baseline techniques, RT, Adaptive Testing (AT) and Random Partition Testing (RPT), DRT demonstrates higher fault-detection effectiveness with a lower test case selection overhead. Furthermore, the theoretical guidelines of parameter setting for DRT are confirmed to be effective. The proposed DRT and the prototype provide an effective and efficient approach for testing web services. IEEE

Software reliability assessment is a key step that determines whether the software can be delivered for the use or not. Because only limited testing resources are allowed for software reliability assessment, this paper proposes a software cybernetics perspective that introduces feedback and self-adaptive control to guide the software testing process in the software reliability assessment. Also, an adaptive testing strategy based on gradient descent is proposed for software reliability assessment. The theoretical and experimental results show that reliability assessment methods based on software cybernetics can achieve highly precise and stable software-reliability-assessment results using restricted testing resources.

Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing , where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing , to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing.

Service-oriented architecture (SOA) has become an application development paradigm widely recognized both in academia and in industry. Although SOA-based applications may vertically implement business processes through the composition of loosely coupled services, they have to face frequent changes, such as unavailability of service (due to the uncontrollable, dynamic, and distributed environments) or dynamic replacement of service (due to specific user requirements). This indicates that the service compositions are expected to be adaptable enough to cater to such changing situations. In this paper, we propose an approach to supporting unplanned dynamic changes of service compositions by combining variability management and dynamic binding. The proposed approach introduces the concept of abstract proxy services in a variability-supporting service composition language, namely VxBPEL, and provides a mechanism to support variation design and dynamic binding for unplanned changes at run time. To realize the proposed approach, we have developed a service composition engine that supports abstract proxy services and their run-time replacement via service discovery or user intervention. Finally, a case study has been conducted to demonstrate the feasibility of the proposed approach and quantify the performance overhead resulting from runtime variability management and dynamic binding. The experimental results show that the proposed approach overcomes the limitation of imperative variability-based approaches in handling unplanned dynamic changes and further enhances the dynamic adaptability of VxBPEL-based service compositions.

Metamorphic testing is a technique that makes use of some necessary properties of the software under test, termed as metamorphic relations, to construct new test cases, namely follow-up test cases, based on some existing test cases, namely source test cases. Due to the ability of verifying testing results without the need of test oracles, it has been widely used in many application domains and detected lots of real-life faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing, most of which were focused on the identification and selection of “good” metamorphic relations. Recently, a few studies emerged on the research direction of how to generate and select source test cases that are effective in fault detection. In this paper, we propose a novel approach to generating source test cases based on their associated path constraints, which are obtained through symbolic execution. The path distance among test cases is leveraged to guide the prioritization of source test cases, which further improve the efficiency. A tool has been developed to automate the proposed approach as much as possible. Empirical studies have also been conducted to evaluate the fault-detection effectiveness of the approach. The results show that this approach enhances both the performance and automation of metamorphic testing. It also highlights interesting research directions for further improving metamorphic testing.

Nowadays, service oriented architecture (SOA) has become one mainstream paradigm for developing distributed applications. As the basic unit in SOA, web services can be composed to construct complex applications. The quality of web services and their compositions is critical to the success of SOA applications. Testing, as a major quality assurance technique, is confronted with new challenges in the context of service compositions. In this paper, we propose a scenario-oriented testing approach that can automatically generate test cases for service compositions. Our approach is particularly focused on the service compositions specified by Business Process Execution Language for web services (WS-BPEL), a widely recognized executable service composition language. In the approach, a WS-BPEL service composition is first abstracted into a graph model; test scenarios are then derived from the model; finally, test cases are generated according to different scenarios. We also developed a prototype tool implementing the proposed approach, and an empirical study was conducted to demonstrate the applicability and effectiveness of our approach. The experimental results show that the automatic scenario-oriented testing approach is effective in detecting many types of faults seeded in the service compositions.

A path-aware approach to mutant reduction which explores mutant reduction from the perspective of the path depth in the program under test.Four mutant reduction strategies which are developed with different priorities among the four heuristic rules for mutant reduction.An empirical study which evaluates effectiveness of the mutant reduction strategies with 11 C programs. Context: Mutation testing, which systematically generates a set of mutants by seeding various faults into the base program under test, is a popular technique for evaluating the effectiveness of a testing method. However, it normally requires the execution of a large amount of mutants and thus incurs a high cost.Objective: A common way to decrease the cost of mutation testing is mutant reduction, which selects a subset of representative mutants. In this paper, we propose a new mutant reduction approach from the perspective of program structure.Method: Our approach attempts to explore path information of the program under test, and select mutants that are as diverse as possible with respect to the paths they cover. We define two path-aware heuristic rules, namely module-depth and loop-depth rules, and combine them with statement- and operator-based mutation selection to develop four mutant reduction strategies.Results: We evaluated the cost-effectiveness of our mutant reduction strategies against random mutant selection on 11 real-life C programs with varying sizes and sampling ratios. Our empirical studies show that two of our mutant reduction strategies, which primarily rely on the path-aware heuristic rules, are more effective and systematic than pure random mutant selection strategy in terms of selecting more representative mutants. In addition, among all four strategies, the one giving loop-depth the highest priority has the highest effectiveness.Conclusion: In general, our path-aware approach can reduce the number of mutants without jeopardizing its effectiveness, and thus significantly enhance the overall cost-effectiveness of mutation testing. Our approach is particularly useful for the mutation testing on large-scale complex programs that normally involve a huge amount of mutants with diverse fault characteristics.

The publish/subscribe paradigm provides a loosely-coupled and scalable communication model for the large-scale IoT service systems, such as supervisory control and data acquisition (SCADA). Data confidentiality and service privacy are two crucial security issues for the publish/subscribe model deployed in different domains. The existing access control schemes for such model cannot address both the issues at the same time. In this paper, we propose a comprehensive access control framework (CACF) to bridge this gap. The design principle of the proposed framework is twofold: (a) a bi-directional policy matching scheme for protecting the privacy of IoT services; and (b) a fully homomorphic encryption scheme for encrypting published events to protect data confidentiality. We analyze the correctness and security of the CACF, moreover, we prototype CACF based on Apache ActiveMQ, an open source message broker, and evaluate its performance. The experimental results indicate that our security system meets the latency requirements for very high-quality SCADA services.

Nowadays, applications are increasingly deployed as Web services in the globally distributed cloud computing environment. Multiple services are normally composed to fulfill complex functionalities. Business Process Execution Language for Web Services (WS-BPEL) is an XML-based service composition language that is used to define a complex business process by orchestrating multiple services. Compared with traditional applications, WS-BPEL programs pose many new challenges to the quality assurance, especially testing, of service compositions. A number of techniques have been proposed for testing WS-BPEL programs, but only a few studies have been conducted to systematically evaluate the effectiveness of these techniques. Mutation testing has been widely acknowledged as not only a testing method in its own right but also a popular technique for measuring the fault-detection effectiveness of other testing methods. Several previous studies have proposed a family of mutation operators for generating mutants by seeding various faults into WS-BPEL programs. In this study, we conduct a series of empirical studies to evaluate the applicability and effectiveness of various mutation operators for WS-BPEL programs. The experimental results provide insightful and comprehensive guidance for mutation testing of WS-BPEL programs in practice. In particular, our work is the systematic study in the selection of effective mutation operators specifically for WS-BPEL programs.

Metamorphic testing (MT) is an innovative approach to alleviating the oracle problem in software testing, which uses metamorphic relations of the program under test, instead of the test oracles, to verify its outputs. To alleviate the oracle problem of testing web services, we had previously proposed an MT framework for web services. In this paper, we further improve the efficiency and automation of this framework by leveraging metamorphic relations to iteratively generate test cases. We present a fixed-size iterative MT algorithm and implement it in the MT framework. We conduct three case studies to evaluate the fault detection effectiveness and efficiency of the proposed approach. Experimental results suggest that, compared with the conventional MT, iterative MT can achieve a comparable fault detection effectiveness, but with significantly fewer resources. Observations and limitations are summarised to provide new insights into the application of iterative MT.

Service-Oriented Architecture (SOA) enables the coordination of multiple loosely coupled services. This allows users to choose any service provided by the SOA without knowing implementation details, thus making coding easier and more flexible. Web services are basic units of SOA. However, the functionality of a single Web service is limited, and usually cannot completely satisfy the actual demand. Hence, it is necessary to coordinate multiple independent Web services to achieve complex business processes. Business Process Execution Language for Web Services (WS-BPEL) makes the coordination possible, by helping the integration of multiple Web services and providing an interface for users to invoke. When coordinating these services, however, illegal or faulty operations may be encountered, but current tools are not yet powerful enough to support the localisation and removal of these problems. In this paper, we propose a fault localisation technique for WS-BPEL programs based on predicate switching and program slicing, allowing developers to more precisely locate the suspicious faulty code. Case studies were conducted to investigate the effectiveness of the proposed technique, which was compared with predicate switching only, slicing only, and one existing fault localisation technique, namely Tarantula. The experimental results show that the proposed technique has a higher fault localisation effectiveness and precision than the baseline techniques.

Unified Modeling Language UML activity diagrams are widely used to model concurrent interaction among multiple objects. In this paper, we propose a transformation-based approach to generating scenario-oriented test cases for applications modeled by UML activity diagrams. Using a set of transformation rules, the proposed approach first transforms a UML activity diagram specification into an intermediate representation, from which it then constructs test scenarios with respect to the given concurrency coverage criteria. The approach then finally derives a set of test cases for the constructed test scenarios. The approach resolves the difficulties associated with fork and join concurrency in the UML activity diagram and enables control over the number of the resulting test cases. We further implemented a tool to automate the proposed approach and studied its feasibility and effectiveness using a case study. Experimental results show that the approach can generate test cases on demand to satisfy a given concurrency coverage criterion and can detect up to 76.5% of seeded faults when a weak coverage criterion is used. With the approach, testers can not only schedule the software test process earlier, but can also better allocate the testing resources for testing concurrent applications. Copyright © 2015John Wiley & Sons, Ltd.

A great amount of fault-based testing strategies have been proposed to generate test cases for detecting certain types of faults in Boolean specifications. However, most of the previous studies on these strategies were focused on the Boolean expressions in the disjunctive normal form (DNF), even the irredundant DNF (IDNF)-little work has been conducted to comprehensively investigate their performance on general Boolean specifications. In this study, we conducted a series of experiments to evaluate and compare 18 fault-based testing strategies using over 4000 randomly generated fault-seeded Boolean expressions. In the experiments, a testing strategy is regarded as effective and efficient if it can detect most of the seeded faults using a small number of test cases. Our experimental results show that if a testing strategy is highly effective and efficient when testing the Boolean expressions in the IDNF, it also shows high effectiveness and efficiency on general Boolean expressions. It is found that one family of fault-based testing strategies, namely MUMCUT, normally deliver the best performance among all the 18 strategies. Our study provides an in-depth understanding and insight of fault-based testing for general Boolean expressions.

Metamorphic testing is well known for its ability to alleviate the oracle problem in software testing. The main idea of metamorphic testing is to test a software system by checking whether each identified metamorphic relation (MR) holds among several executions. In this regard, identifying MRs is an essential task in metamorphic testing. In view of the importance of this identification task, METRIC (METamorphic Relation Identification based on Category-choice framework) was developed to help software testers identify MRs from a given set of complete test frames. However, during MR identification, METRIC primarily focuses on the input domain without sufficient attention given to the output domain, thereby hindering the effectiveness of METRIC. Inspired by this problem, we have extended METRIC into METRIC $^{+}$ + by incorporating the information derived from the output domain for MR identification. A tool implementing METRIC $^{+}$ + has also been developed. Two rounds of experiments, involving four real-life specifications, have been conducted to evaluate the effectiveness and efficiency of METRIC $^{+}$ + . The results have confirmed that METRIC $^{+}$ + is highly effective and efficient in MR identification. Additional experiments have been performed to compare the fault detection capability of the MRs generated by METRIC $^{+}$ + and those by $\mu$ μ MT (another MR identification technique). The comparison results have confirmed that the MRs generated by METRIC $^{+}$ + are highly effective in fault detection.

软件测试通过运行测试用例, 观察被测软件的输出结果是否符合预期结果来发现潜藏的故障。 然而, 判定测试输出是否正确往往会成为软件测试面临的一个挑战(称为测试预期问题)。 蜕变测试通过建立被测软件输入变量域和输出结果域的关系(称为蜕变关系), 以蜕变关系判定测试执行的正确性。 蜕变测试不仅具有良好的故障检测能力, 也是一种测试用例选择方法。 本文侧重报告蜕变测试作为一种测试用例选择策略的几个成功案例。

Service Oriented Architecture (SOA) has become a major application development paradigm. As a basic unit of SOA applications, Web services significantly affect the quality of the applications constructed from them. In the context of SOA, the specification and implementation of Web services are completely separated. The lack of source code and the restricted control of Web services limit the testability of Web services, and make the oracle problem prominent. In this context, can one alleviate the test oracle problem, or effectively and efficiently test such Web services even without oracles? It is an important issue which has not been yet adequately addressed. To address the challenge of testing Web services, the authors propose a metamorphic relation-based approach to testing Web services without oracles. The proposed approach leverages so-called metamorphic relations to generate test cases and evaluate test results. To make the proposed approach practical and effective, the authors proposed a framework taking into account the unique features of SOA, and developed a prototype which partially automates the framework. Three case studies are conducted to validate the feasibility and effectiveness of the proposed approach. The work presented in the paper not only alleviates the test oracle problem of testing Web services, but also delivers an effective and efficient test technique without oracles.

Web service-based systems are built orchestrating loosely coupled, standardized, and internetworked programs. If on the one hand, Web services address the interoperability issues of modern information systems, on the other hand, they enable the development of software systems on the basis of reuse, greatly limiting the necessity for reimplementation. Techniques and methodologies to gain the maximum from this emerging computing paradigm are in great need. In particular, a way to explicitly model and manage variability would greatly facilitate the creation and customization of Web service-based systems. By variability we mean the ability of a software system to be extended, changed, customized or configured for use in a specific context.We present a framework and related tool suite for modeling and managing the variability of Web service-based systems for design and run-time, respectively. It is an extension of the COVAMOF framework for the variability management of software product families, which was developed at the University of Groningen. Among the novelties and advantages of the approach are the full modeling of variability via UML diagrams, the run-time support, and the low involvement of the user. All of which leads to a great deal of automation in the management of all kinds of variability. (C) 2009 Elsevier Inc. All rights reserved.

Web services provide a way to facilitate the business integration over the Internet. Flexibility is an important and desirable property of Web service-based systems due to dynamic business environments. The flexibility can be provided or addressed by incorporating variability into a system. In this study, we investigate how variability can be incorporated into service-based systems. We propose a language, VxBPEL, which is an adaptation of an existing language, BPEL, and able to capture variability in these systems. We develop a prototype to interpret this language. Finally, we illustrate our method by using it to handle variability of an example. (C) 2008 Elsevier B.V. All rights reserved.

Business Process Execution Language (BPEL) is a widely recognized executable service composition language. Since BPEL integrates services of desired functionality to compose business processes, it is significantly different from typical programming languages. How to effectively locate the integration-level faults in BPEL programs is an open issue. In this paper, we propose the BPEL fault localization guidelines based on the characteristics of BPEL programs, and adapt Tarantula, a traditional fault localization technique, to locate the integration-level faults in BPEL programs. We also conducted an empirical study to demonstrate the feasibility of our methodology.

Service Oriented Architecture (SOA) has become a major application development paradigm. As a basic unit of SOA applications, Web services significantly affect the quality of the applications constructed from them. Since the development and consumption of Web services are completely separated under SOA environment, the consumers are normally provided with limited knowledge of the services and thus have little information about test oracles. The lack of source code and the restricted control of Web services limit the testability of Web services. To address the prominent oracle problem when testing Web services, we propose a metamorphic testing framework for Web services taking into account the unique features of SOA. We conduct a case study where the new metamorphic testing framework is employed to test a Web service that implements the electronic payment. The results of case study show the feasibility of the framework for web services, and also the efficiency of metamorphic testing. The work presented in the paper alleviates the test oracle problem when testing Web services under SOA.

In regression testing, an important problem is how to select a smaller size of test set for execution. We present a novel constraint-oriented test suite reduction method for conservative regression testing by which we mean that all bugs discovered must be revealed by the reduced test suite. A test constraint for a bug is Boolean formulas defined over the input variables of program under test. The reduced test constraints for a pool of bugs are constructed using the subsumption relationship between test constraint conditions. Test case selection is based on the reduced test constraint set. A test case is selected into the test suite when and only when it satisfies one or more test constraints. The selection process is completed when all test constraint conditions are satisfied by the selected test cases. An empirical study is conducted and the experimental results show that our method can significantly save efforts for the conservative regression testing.

Nowadays applications are increasingly developed based on remote Web services and service composition has become a powerful novel development paradigm. Due to the fact that such applications in the context of Internet are deployed and executed in an open and dynamic environment, adaptability is one of the crucial requirements for developing such applications. In our previous work, we presented a variability-based approach to address the adaptability issue of service compositions, and developed the VxBPEL, an extension of BPEL with respect to variability. This paper presents a variability analysis tool for variability-based adaptive service compositions called ValySeC. ValySeC extracts the variation from service compositions specified using VxBPEL and provides a variation view. With the view, the designer can better understand variation points and the possible instances, and efficiently maintain the variation within service compositions. The paper presents the design and implementation of ValySeC using a case study to handle service compositions with variations to illustrate the main concepts.

The Business Process Execution Language (BPEL) is a widely recognized executable language supporting the specification of process-oriented service compositions. However, the language is limited in addressing variable requirements in the description of business processes. We propose to construct variable and maintainable Web services compositions with VxBPEL, an extension to BPEL we developed to define variability in business process specification. We present the main concepts of VxBPEL and show how to achieve better adaptation and variability maintenance of service compositions, which is particularly desired in the context of dynamically changing business goals and processes.

As service centric (SC) systems are being increasingly adopted, new challenges and possibilities emerge. Business processes are now able to execute seamlessly across organizations and to coordinate the interaction of loosely coupled services. Often it is necessary to have transactionality for a set of business operations, but the loosely nature of such systems calls for techniques and principles that go beyond traditional ACID transactions. By analyzing existing service composition languages, tools, and needs on a classical example, we provide requirements for transactionality in service centric systems and indications for developing SC systems transactionally capable.