Search

The complex and dynamic nature of Systems of Systems (SoSs) poses many challenges on their validation and testing, but so far few effective test strategies exist to address them. On the other hand, extensive research has been conducted in the testing of Software Product Lines (SPLs), which present interesting convergence points with SoSs, as both disciplines aim at reducing development costs and time-to-market thanks to extensive reuse of existing artifacts. In this paper, we outline commonalities and differences between the SoS and SPL paradigms from the point of view of testing and investigate how existing methods and tools from SPL testing could be leveraged to address the challenges of SoS testing.

Background In Software Engineering (SE), conference publications have high importance both in effective communication and in academic careers. Researchers actively discuss how a paper should be organized to be accepted in mainstream conferences. Aiming This work tackles the problem of generalizing and characterizing the type of papers accepted at SE conferences. Method The paper offers a new perspective in the analysis of SE literature: a categorization scheme for SE papers is obtained by merging, extending and revising related proposals from a few existing studies. The categorization scheme is used to classify the papers accepted at three top-tier SE conferences during five years (2012–2016). Results While a broader experience is certainly needed for validation and fine-tuning, preliminary outcomes can be observed relative to what problems and topics are addressed, what types of contributions are presented and how they are validated. Conclusions The results provide insights to paper writers, paper reviewers and conference organizers in focusing their future efforts, without any intent to provide judgments or authoritative guidelines.

International audience; The recent push towards test automation and test-driven development continues to scale up the dimensions of test code that needs to be maintained, analysed, and processed side-by-side with production code. As a consequence, on the one side regression testing techniques, e.g., for test suite prioritization or test case selection, capable to handle such large-scale test suites become indispensable; on the other side, as test code exposes own characteristics, specific techniques for its analysis and refactoring are actively sought. We present JTeC, a large-scale dataset of test cases that researchers can use for benchmarking the above techniques or any other type of tool expressly targeting test code. JTeC collects more than 2.5M test classes belonging to 31K+ GitHub projects and summing up to more than 430 Million SLOCs of ready-to-use real-world test code.

We propose a factory of educational System of Systems (SoS) case studies that can be used for evaluating SoS research results, in particular in SoS testing. The factory includes a first set of constituent systems that can collaborate within different SoS architectures to accomplish different missions. In the paper, we introduce three possible SoSs and outline their missions. For more detailed descriptions, diagrams and the source code, we refer to the online repository of EDUFYSoS. The factory is meant to provide an extensible playground, which we aim to grow to include more systems and other missions with the support of the community. Ministero dell'Università e della Ricerca (Italia) SISMA 201752ENYB

International Conference on Web Information Systems and Technologies, WEBIST (15th. 2019. Vienna, Austria), This work was supported in part by the Spanish Ministry of Economy and Competitiveness under TestEAMoS (TIN2016-76956-C3-1-R) project and ERDF funds, and by the European Project ElasTest in the Horizon 2020 research and innovation program (GA No. 731535).

Model-driven engineering (MDE) promotes automated model transformations along the entire development process. Guaranteeing the quality of early models is essential for a successful application of MDE techniques and related tool-supported model refinements. Do these models properly reflect the requirements elicited from the owners of the problem domain? Ultimately, this question needs to be asked to the domain experts. The problem is that a gap exists between the respective backgrounds of modeling experts and domain experts. MDE developers cannot show a model to the domain experts and simply ask them whether it is correct with respect to the requirements they had in mind. To facilitate their interaction and make such validation more systematic, we propose a methodology and a tool that derive a set of customizable questionnaires expressed in natural language from each model to be validated. Unexpected answers by domain experts help to identify those portions of the models requiring deeper attention. We illustrate the methodology and the current status of the developed tool MOTHIA, which can handle UML Use Case, Class, and Activity diagrams. We assess MOTHIA effectiveness in reducing the gap between domain and modeling experts, and in detecting modeling faults on the European Project CHOReOS.

The growing number of vehicles daily moving on roads increases the need of protecting the safety and security of passengers, pedestrians, and vehicles themselves. This need is intensified when considering the pervasive introduction of Information and Communication Technologies (ICT) systems into modern vehicles, because this makes such vehicles potentially vulnerable from the point of view of security. The convergence of safety and security requirements is one of the main outstanding research challenges in software-intensive systems. This work reviews existing methodologies and solutions addressing security issues in the automotive domain with a focus on the integration between safety and security aspects. In particular, we identify the main security issues with vehicular communication technologies and existing gaps between state-of-the-art methodologies and their implementation in the real world. Starting from a literature survey and referring to widely accepted standards of the domain, such as AUTOSAR and ISO 26262, we discuss research challenges and set baselines for a holistic secure-by-design approach targeting safety and security aspects all along the different phases of the development process of automotive software.

In the context of XACML-based access control systems, an intensive testing activity is among the most adopted means to assure that sensible information or resources are correctly accessed. Unfortunately, it requires a huge effort for manual inspection of results: thus automated verdict derivation is a key aspect for improving the cost-effectiveness of testing. To this purpose, we introduce XACMET, a novel approach for automated model-based oracle definition. XACMET defines a typed graph, called the XAC-Graph, that models the XACML policy evaluation. The expected verdict of a specific request execution can thus be automatically derived by executing the corresponding path in such graph. Our validation of the XACMET prototype implementation confirms the effectiveness of the proposed approach., Comment: 7 pages

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections. The techniques presented include: (a) structural testing using symbolic execution, (b) model-based testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) search-based testing. Each section is contributed by world-renowned active researchers on the technique, and briefly covers the basic ideas underlying the method, the current state of the art, a discussion of the open research problems, and a perspective of the future development of the approach. As a whole, the paper aims at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation, while ensuring a comprehensive and authoritative treatment.

With the aim of identifying good structures and examples for papers in the software engineering field, we conducted a study of the type of papers accepted along four decades in the Research Track of the International Conference on Software Engineering (ICSE). We used for this purpose a categorization scheme for Software Engineering papers that was obtained by merging, extending and revising a few existing paper scheme proposals. This paper summarizes some outcomes relative to what topics and problems are addressed, what types of contribution are presented and how they are validated. Insights from the study could help ICSE authors, reviewers and conference organizers in focusing and improving future efforts.

Software measurement is a crucial technology along the software development process, and notably for project management. Decision-making in project management requires in fact, accurate estimations and the support of metrics to monitor and control the varying factors affecting the development process. Managers must make reliable schedule predictions and optimize personnel utilization. They therefore must be able to dynamically evaluate whether the resources assigned to a job are sufficient and whether the organization structure is adequate to meet the scheduled deadlines. To support managers in these tasks, we propose a method called Propean (for PROject PErformance ANalysis) based on the combination of classical performance engineering techniques and the Unified Modelling Language (UML). The combined application of these disciplines guarantees on one hand, a validated approach for modelling and estimating the ‘Quality of Service’ parameters when they come to the development process. Performance engineering techniques are in fact based on rigorous mathematical formalisms such as queueing networks, stochastic Petri nets and Markov models. On the other hand, the adoption of UML provides managers with an easy-to-use front-end representation of the process closer to the design notations they routinely employ. To illustrate Propean application, in this article, we model the case of a manager who must decide a release date for a product undergoing the testing phase. Copyright © 2007 John Wiley & Sons, Ltd.

A large gap exists between the state-of-the-art in software testing literature, and the state of software testing practice. Empirical research should (and could) play a first class role for bridging this gap. Empirical studies in software testing have focused mainly on the evaluation of techniques for test case selection. But effective selection of test cases by itself is not sufficient to warrant successful testing: we need also empirical studies to start collecting proven patterns that test practitioners can use to predictably solve software testing problems.

Software engineering for Internet computing involves the architecting, development, deployment, management, and quality assurance of software supporting Internet-based systems. It also addresses global-development issues such as communication complexity, distributed control, governance policies, and cultural differences. This issue presents a selection of exciting and representative research on this topic. The Web extra at http://youtu.be/UA7rLAwFbwY is an audio recording in which author IEEE Software Multimedia Editor Davide Falessi interviews Antonia Bertolino, Hong Mei, and Tao Xie, guest editors of the magazine's January/February 2015 issue on Software Engineering for Internet Computing: Internetware and Beyond.

Test coverage adequacy measures provide a widely used stopping criterion. Engineering of modern software-intensive systems emphasizes reuse. In the case that a program uses reused code or third-party components in a context that is different from the original one, some of their entities (e.g. branches) might never be exercised, thus producing a code coverage level far from full and not meaningful anymore as a stopping rule for the program at hand. We introduce a new coverage criterion, called "Relevant Coverage", that in each testing context in which a code is reused calculates coverage measures over the set of relevant entities for that context. We provide an approach for identifying relevant entities using dynamic symbolic execution. The introduced coverage adequacy criterion is assessed in an exploratory study against traditional coverage in terms of test suite size reduction factor, cost-effectiveness ratio and rate of fault detection. The results of our study showed that relevant coverage can considerably reduce the test suite size while preserving a high cost-effectiveness ratio with respect to the traditional approach.

Access Control is among the most important security mechanisms to put in place in order to secure applications. XACML is the de facto standard for storing and deploying access control policies. However, due to the complexity of the XACML language, policy definition becomes a difficult and error prone process. In recent years, the combined use of models for the access control policy specification, and the model-to-code facilities, for the automatic transformation of the model into the XACML language, has been proposed as a possible solution. These model-driven methodologies and facilities need to be thoroughly validated and verified. In this paper we provide an integrated framework for testing the automatic translation of the specification of an access control model into an XACML policy. The framework includes different test strategies for the derivation of test cases and some facilities for making easier their execution against the XACML policy and the test results collection and analysis. In addition, we illustrate the use of the framework on a case study.

Security is an important aspect of modern information management systems. The crucial role of security in this systems demands the use of tools and applications that are thoroughly validated and verified. However, the testing phase is an effort consuming activity that requires reliable supporting tools for speeding up this costly stage. Access control systems, based on the integration of new and existing tools are available in the Service Development Environment (SDE). We introduce an Access Control Testing toolchain (ACT) for designing and testing access control policies that includes the following features: (i) the graphical specification of an access control model and its translation into an XACML policy; (ii) the derivation of test cases and their execution against the XACML policy; (iii) the assessment of compliance between the XACML policy execution and the access control model. In addition, we illustrate the use of the ACT toolchain on a case study.

[Context and motivation] Choreographies are a form of service composition in which partner services interact in a global scenario without a single point of control. The absence of an explicitly specified orchestration requires changes to requirements practices to recognize the need to optimize software services choreography and monitoring for satisfaction with system requirements. [Question/problem] We developed a requirements-led approach that aims to provide tools and processes to transform requirements expressed on service-based systems to QoS-aware choreography specifications. [Principal ideas/results] The approach is used by domain experts to specify natural language requirements on a service-based system, and by choreography designers to adapt their models to satisfy requirements more effectively. Non-functional requirements are mapped to BPMN choreography diagrams as quality properties, using the Q4BPMN notation, that support analysis and monitoring facilities. [Contribution] We report the new integrated approach and provide lessons learned from applying it to a real-world example of dynamic taxi management.

Service choreographies present numerous engineering challenges, particularly with respect to testing activities, that traditional design-time approaches cannot properly address. A proposed online testing solution offers a powerful, extensible framework to effectively assess service compositions, leading to a more trustworthy and reliable service ecosystem.

The term software architecture (SA) has been introduced to denote the high level structure of a software system. SA has been proposed as a means for managing complexity and improving reuse, by supporting the decomposition of a system into its high level components and their interconnections. SA became prevalent in the beginning of the ’90s when its main role was to describe the system structure (by identifying architectural components and connectors) and the required system behavior. Over the years, the SA scope has evolved, and today it also captures the architecturally-relevant decisions behind design [50] taken by a variety of stakeholders to satisfy their own specific concerns, and codified into different views and viewpoints [46]. Nowadays, the relevance of SA in both academic and industrial worlds is unquestionable, and SAs are used for documenting and communicating design decisions and architectural solutions [24], for driving analysis techniques [57,62,63,79], for code generation purposes in model-driven engineering [3,37], for product line engineering [15], for risks and cost estimation [36,67], and further on. SAs have also been advocated since the 1990s as a means for improving the dependability of complex software systems. In this light, different methods have been

International audience; Monitoring is an essential means in the management of service-oriented applications. Here, event correlation results crucial when monitoring rules aim at checking the exposed levels of Quality of Service against the Service Level Agreements established among the choreography participants. However, when choreographies are enacted over distributed networks or clouds, the relevant monitoring rules might not be completely defined a-priori, as they may need to be adapted to the specific infrastructure and to the evolution of events. This paper presents an adaptive multi-source monitoring architecture synthesizing instances of rules at run-time and shows examples of use on a demonstration scenario from the European Project CHOReOS.

In modern pervasive application domains, such as Service Oriented Architectures (SOAs) and Peer-to-Peer (P2P) systems, security aspects are critical. Justified confidence in the security mechanisms that are implemented for assuring proper data access is a key point. In the last years XACML has become the de facto standard for specifying policies for access control decisions in many application domains. Briefly, an XACML policy defines the constraints and conditions that a subject needs to comply with for accessing a resource and doing an action in a given environment. Due to the complexity of the language, XACML policy specification is a difficult and error prone process that requires specific knowledge and a high effort to be properly managed.

International audience; The deployment and the execution of applications on dynamic Cloud infrastructures introduces new requirements of adaptability with respect to monitoring. Specifically, the governance of service choreographies enacted over Cloud-based solutions relies on the observation and analysis of events happening at different abstraction layers. Adaptability requirements are even more evident when monitoring deals with Service Level Agreements (SLA) established among the choreography participants. In fact, as the Cloud paradigm offers on-demand solutions as a service, often monitoring rules cannot be completely defined off-line. Thus also the monitoring infrastructure must keep track of the continuous evolution of the underlying environment, and adapt itself accordingly. This paper proposes an adaptive multi-source monitoring architecture that can synthesize on-the-fly SLA monitoring rules following the evolution of the Cloud infrastructure. We demonstrate the idea on a case study and discuss limitations as well as planned further advancements.

International audience; Modern software applications are more and more conceived as distributed service compositions deployed over Grid and Cloud technologies. Choreographies provide abstract specifications of such compositions, by modeling message-based multi-party interactions without assuming any central coordination. To enable the management and dynamic adaptation of choreographies, it is essential to keep track of events and exchanged messages and to monitor the status of the underlying platform, and combine these different levels of information into complex events meaningful at the application level. Towards this goal, we propose a Multi-source Monitoring Framework that we are developing within the EU Project CHOReOS, which can correlate the messages passed at business-service level with observations relative to the infrastructure resources. We present the monitor architecture and illustrate it on a use-case excerpted from the CHOReOS project.

International audience; In service-oriented systems non-functional properties become very important to support run-time service discovery and composition. Software engineers should take care of them for guaranteeing the service quality in all the software life-cycle phases, from requirements speci cation to design, to system deployment and execution monitoring. This wide scope and the criticality of non-functional properties demand that they are expressed in a language which is intuitive and easy to use for the service quality specifi cation, and at the same time is machine-processable to be automatically handled at run-time. In this paper we present a Property Meta-Model that aims to reach these two main objectives and show as a proof of concept its use for the modeling of two di fferent properties.

According to a study commissioned by the National Institute of Standards and Technology in 2002, software bugs cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the nation's gross domestic product (GDP). The same study also found that more than one-third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure. These numbers would be significantly higher if the study were conducted today.

The attractive feature of Service Oriented Architecture (SOA) is that pieces of software conceived and developed by independent organizations can be dynamically composed to provide richer functionality. The same reasons that enable flexible compositions, however, also prevent the application of some traditional testing approaches, making SOA validation challenging and costly. Web services usually expose just an interface, enough to invoke them and develop some general (black-box) tests, but insufficient for a tester to develop an adequate understanding of the integration quality between the application and the independent web services. To address this lack we propose an approach that makes web services more transparent to testers through the addition of an intermediary service that provides coverage information. The approach, named Service Oriented Coverage Testing (SOCT), provides testers with feedback about how much a service is exercised by their tests without revealing the service internals. In SOCT, testing feedback is offered itself as a service, thus preserving SOA founding principles of loose coupling and implementation neutrality. In this paper we motivate and define the SOCT approach, and implement an instance of it. We also perform a study to asses SOCT feasibility and provide a preliminary evaluation of its viability and value.

Defining a domain model is a costly and error-prone process. It requires that the knowledge possessed by domain experts be suitably captured by modeling experts. Eliciting what is in the domain expert's mind and expressing it using a modeling language involve substantial human effort. In the process, conceptual errors may be introduced that are hard to detect without a suitable validation methodology. This paper proposes an approach to support such validation, by reducing the knowledge gap that separates modeling experts and domain experts. While our methodology still requires the domain expert's judgement, it partially automates the validation process by generating a set of yes/no questions from the model. Answers differing from expected ones point to elements in the model which require further consideration and can be used to guide the dialogue between domain experts and modeling experts. Our methodology was implemented as a tool and was applied to a real case study, within the IPERMOB project.

The Sixth International Workshop on Automation of Software Test (AST 2011) is associated with the 33rd International Conference on Software Engineering (ICSE 2011). This edition of AST was focused on the special theme of Software Design and the Automation of Software Test and authors were encouraged to submit work in this area. The workshop covers two days with presentations of regular research papers, industrial case studies and experience reports. The workshop also aims to have extensive discussions on collaborative solutions in the form of charette sessions. This paper summarizes the organization of the workshop, the special theme, as well as the sessions.

Recently, a growing interest on robustness testing of services composition is emerging. In this paper, we propose a robustness testing approach of services composition by means of fault injection of the BPEL process. The proposed solution is implemented into the existing TASSA framework aimed at functional and non-functional testing of service applications. A case study is provided for assessing the effectiveness of the proposed approach.

International audience; To respond to the growing needs of evolution and adaptation coming from the modern open connected world, applications must continuously monitor their own execution and the surrounding context. The events to be observed, belonging to guaranteed functional and non-functional properties, can themselves vary in scope and along time. Therefore the monitor must be easily con figurable and able to serve differing event consumers. To address these requirements, we developed the Glimpse monitoring infrastructure conceived having flexibility and generality as main concerns. The paper introduces the architecture of Glimpse and shows how it can support runtime performance analysis through a simple example.

International audience; The CONNECT European project that started in February 2009 aims at dropping the interoperability barrier faced by today's distributed systems. It does so by adopting a revolutionary approach to the seamless networking of digital systems, that is, synthesizing on the fly the connectors via which networked systems communicate. CONNECT investigates formal foundations for connectors together with associated automated support for learning, reasoning about and adapting the interaction behavior of networked systems.

International audience; Web-services are broadly considered as an effective means to achieve interoperability between heterogeneous parties of a business process and offer an open platform for developing new composite web-services out of existing ones. In the literature many approaches have been proposed with the aim to automatically compose web-services. All of them assume that, along with the webservice signature, some information is provided about how clients interacting with the web-service should behave when invoking it. We call this piece of information the web-service behavior protocol. Unfortunately, in the practice this assumption turns out to be unfounded. To address this need, in this paper we propose a method to automatically derive from the web-service signature an automaton modeling its behavior protocol. The method, called StrawBerry, combines synthesis and testing techniques. In particular, synthesis is based on data type analysis. The conformance between the synthesized automaton and the implementation of the corresponding web-service is checked by means of testing. The application of StrawBerry to the Amazon E-Commerce Service shows that it is practical and realistic.

The European Project TAS3 addresses the challenge of combining the openness, flexibility, dynamicity offered by service-oriented applications, together with privacy, security and reliability characteristics that are required when personal information is handled. In addition to appropriate authorization and authentication mechanisms, it is important to put in place appropriate validation procedures that can check the trustworthiness and dependability of services provided. We outline the Audition framework and discuss how such on-line validation approach can fit within the TAS3 vision of a seamless connected world, where the final users remain in charge for releasing and administering their data.

In every application domain, achieving interoperability among heterogenous information systems is a crucial challenge and alliances are formed to standardize data-exchange formats. In the healthcare sector, HL7-V3 provides the current international reference models for clinical and administrative documents. Codices, an Italian company, provides the PICASSO platform that uses HL7-V3 as the pivot format to fast achieve a highly integrated degree of interoperability among health-related applications. Given the XML structure of HL7-V3, PICASSO can exploit the XSLT technology to flexibly transform documents. However, Codices spends a large part of the PICASSO deployment workflow for manually validating the required XSL stylesheets. In this paper, we describe a pilot experience in test automation, based on the TAXI tool that applies systematic black-box techniques to generate a set of XML instances from a schema. Observed benefits to Codices development process are reported and discussed.

Service Oriented Architectures (SOAs) are becoming increasingly popular and powerful. Fueling that growth is the availability of independent web services that can be costeffectively composed with other services to provide richer functionality. The reasons that make these systems easier to build, however, also make them more challenging to test. Independent web services usually provide just an interface, enough to invoke them and develop some general (black-box) tests, but insufficient for a tester to develop an adequate understanding of the integration quality between the application and independent web services. To address this lack we propose a "whitening" approach to make web services more transparent through the addition of an intermediate coverage service. The approach, named Service Oriented Coverage Testing (SOCT), provides a tester with feedback about how a whitened service, called a Testable Service, is exercised. In this paper we introduce the SOCT approach, implement an instance of it, and perform a preliminary study to show its feasibility and potential value. SOCT enables SOA white-box testing, while maintaining SOA flexibility, dynamism and loose coupling.

The attractiveness and popularity of Service-Oriented Software Systems (SOSSs) stem from the growing availability of independent services that can be cost-effectively composed with other services to dynamically provide richer functionality. Service-orientation however poses new and difficult challenges to testers, especially when it comes to testing the interactions between heterogeneous, loosely coupled and independently developed services. Service integration testing requires discipline, standardized processes, and agreed policies to be put in place, which we referred to as SOA (Service Oriented Architecture) Test Governance (STG). Discovered services usually provide just a syntactical interface, enabling some general black-box tests, but insufficient to develop an adequate understanding of the integration quality between the interacting services. Besides, testing for the functional and extra-functional properties of a composite SOSS cannot generally rely on the ready or full availability, for testing purposes, of all invoked services (e.g., their usage might bring unwanted side effects). In this talk we will survey some of our recent results on SOSSs testing that span over the above needs. We will first discuss how the STG concept is implicit behind any approach to testing composite SOSSs and then give an overview of three different, complementary SOSS test approaches realizing different grades of STG, namely: the state-of-practice prototype tool WS-TAXI, for fully automatic generation of black-box test inputs; the novel SOCT approach allowing for test coverage measurement of independent services without loosing their implementation neutrality; the PUPPET tool for model-based generation of a testbed simulating the functional and extra-functional behavior of invoked external services.

Software testing is a very complex activity deserving a first-class role in software development. Testing related activities encompass the entire development process and may consume a large part of the effort required for producing software. In this talk, I will first organize into a coherent framework the many topics and tasks forming the software testing discipline, pointing at relevant open issues [1]. Then, among the outlined challenges, I will focus on some hot ones posed by the testing of modern complex and highly dynamic systems [2]. What is assured is that software testers do not risk to remain without their job, and testing researchers are not at short of puzzles. Software testing is and will forever be a fundamental activity of software engineering: notwithstanding the revolutionary advances in the way it is built and employed (or perhaps exactly because of), the software will always need to be eventually tried and monitored. In the years, software testing has evolved from an "art" to a discipline, but test practice largely remains a trial-and-error methodology. We will never find a test approach that is guaranteed to deliver a "perfect" product, whichever is the effort we employ. However, what we can and must pursue is to transform testing from "trial-and-error" to a systematic, cost-effective and predictable engineering practice.

Composition of Web Services (WSs) is anticipated as the future standard way to dynamically build distributed applications, and hence their verification and validation is attracting great attention. The standardization of BPEL as a composition language and of WSDL as a WS interface definition language has led researchers to investigate verification and validation techniques mainly focusing on the sequence of events in the composition, while minor attention has been paid to the validation of the data flow exchange. In this chapter we study the potential of using data flow modelling for testing composite WSs. After an exhaustive exploration of the issues on testing based on data-related models, we schematically settle future research issues on the perspectives opened by data flow-based validation and present examples for some of them, illustrated on the case study of a composite WS that we have developed, the Virtual Scientific Book Store.

This year ESCE/FSE launches the new Widened Software Engineering (WISE) track with an explicit aim to widen international participation, especially from countries which are usually under-represented in the conference audience.

This paper presents Puppet (Pick UP Performance Evaluation Test-bed), an approach for the automatic generation of test-beds to empirically evaluate different QoS features of a Web Service under development. Specifically, the generation exploits the information about the coordinating scenario, the service description and the specification of the agreements that the roles will abide. The approach is supported by a proof-of-concept tool to validate the feasibility of the idea.

Component-based development has emerged as a system engineering approach that promises rapid software development with fewer resources. Yet, improved reuse and reduced cost benefits from software components can only be achieved in practice if the components provide reliable services, thereby rendering component analysis and testing a key activity. This paper discusses various issues that can arise in component testing by the component user at the stage of its integration within the target system. The crucial problem is the lack of information for analysis and testing of externally developed components. Several testing techniques for component integration have recently been proposed. These techniques are surveyed here and classified according to a proposed set of relevant attributes. The paper thus provides a comprehensive overview which can be useful as introductory reading for newcomers in this research field, as well as to stimulate further investigation. Copyright © 2006 John Wiley & Sons, Ltd.

Software engineering comprehends several disciplines devoted to prevent and remedy malfunctions and to warrant adequate behaviour. Testing, the subject of this paper, is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Indeed, software testing is a broad term encompassing a variety of activities along the development cycle and beyond, aimed at different goals. Hence, software testing research faces a collection of challenges. A consistent roadmap of the most relevant challenges to be addressed is here proposed. In it, the starting point is constituted by some important past achievements, while the destination consists of four identified goals to which research ultimately tends, but which remain as unreachable as dreams. The routes from the achievements to the dreams are paved by the outstanding research challenges, which are discussed in the paper along with interesting ongoing work.

In the last years both industry and academia have shown a great interest in ensuring consistent cooperation for business-critical services, with contractually agreed levels of Quality of Service. Service Level Agreement specifications as well as techniques for their evaluation are nowadays irremissible assets. This paper presents Puppet (Pick UP Performance Evaluation Test-bed), an approach and a tool for the automatic generation of test-beds to empirically evaluate the QoS features of a Web Service under development. Specifically, the generation exploits the information about the coordinating scenario (be it choreography or orchestration), the service description (WSDL) and the specification of the agreements (WS-Agreement).