Search

Microservice-based applications consist of multiple services that can evolve independently. When services are modified, they are typically tested before being deployed. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field, therefore they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at runtime in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest exploits cloud technologies, containers in particular, to generate a mocked environment that fully isolates the service under test from the rest of the system. It then reproduces service interactions as previously analyzed, thus testing the new version of the service against usage scenarios that capture the field usages of its earlier versions. We evaluate our approach on an open sourced microservices application and show that ExVivoMicroTest can effectively reveal faults based on automatically collected data.

The ecosystem in which mobile applications run is highly heterogeneous and configurable. All layers upon which mobile apps are built offer wide possibilities of variations, from the device and the hardware, to the operating system and middleware, up to the user preferences and settings. Testing all possible configurations exhaustively, before releasing the app, is unaffordable. As a consequence, the app may exhibit different, including faulty, behaviours when executed in the field, under specific configurations. In this paper, we describe a framework that can be instantiated to support in-vivo testing of a mobile app. The framework monitors the configuration in the field and triggers in-vivo testing when an untested configuration is recognized. Experimental results show that the overhead introduced by monitoring is unnoticeable to negligible (i.e., 0-6%) depending on the device being used (high- vs. low-end). In-vivo test execution required on average 3s: if performed upon screen lock activation, it introduces just a slight delay before locking the device., Research paper accepted to ICST'20, 10+1 pages

Despite their growing complexity and increasing size, modern software applications must satisfy strict release requirements that impose short bug fixing and maintenance cycles, putting significant pressure on developers who are responsible for timely producing high-quality software. To reduce developers workload, repairing and healing techniques have been extensively investigated as solutions for efficiently repairing and maintaining software in the last few years. In particular, repairing solutions have been able to automatically produce useful fixes for several classes of bugs that might be present in software programs. A range of algorithms, techniques, and heuristics have been integrated, experimented, and studied, producing a heterogeneous and articulated research framework where automatic repair techniques are proliferating. This paper organizes the knowledge in the area by surveying a body of 108 papers about automatic software repair techniques, illustrating the algorithms and the approaches, comparing them on representative examples, and discussing the open challenges and the empirical evidence reported so far.

Field failures, that is, failures caused by faults that escape the testing phase leading to failures in the field, are unavoidable. Improving verification and validation activities before deployment can identify and timely remove many but not all faults, and users may still experience a number of annoying problems while using their software systems. This paper investigates the nature of field failures, to understand to what extent further improving in-house verification and validation activities can reduce the number of failures in the field, and frames the need of new approaches that operate in the field. We report the results of the analysis of the bug reports of five applications belonging to three different ecosystems, propose a taxonomy of field failures, and discuss the reasons why failures belonging to the identified classes cannot be detected at design time but shall be addressed at runtime. We observe that many faults (70%) are intrinsically hard to detect at design-time.

Quando interagiscono con sistemi software, gli utenti potrebbero dover affrontare problemi come crash, fallimenti e instabilità del programma. Il software difettoso in esecuzione sul campo non è solo la conseguenza di tecniche di verifica inefficaci, ma è anche dovuto alla complessità e alla diversità delle interazioni tra un'applicazione e il suo ambiente. Molte di queste interazioni sono difficilmente previste al momento del test, e anche quando potrebbero essere previste, spesso ci sono così tanti casi da testare che non possono essere affrontati in modo fattibile prima che il software sia rilasciato. Il testing sul campo si propone di affrontare il problema dei fallimenti delle applicazioni sul campo spostando la fase di test direttamente nell'ambiente di produzione. Ciò rende possibile sfruttare diversi scenari che altrimenti sarebbero difficili da catturare con test tradizionali. In questa tesi esploriamo l'area del testing sul campo del software, presentiamo uno studio che caratterizza il problema delle applicazioni che falliscono sul campo, un'architettura client-server che può essere sfruttata per organizzare e controllare il processo di test sul campo e un approccio di test che sfrutta l’ambiente di produzione come banco di prova per l'esecuzione dei test case. L'approccio presentato viene valutato empiricamente su un dataset di errori del software, dimostrando che il 35% dei guasti non rilevati internamente potrebbe essere stato rivelato con test sul campo. When interacting with their software systems, users may have to deal with problems like crashes, failures, and program instability. Faulty software running in the field is not only the consequence of ineffective in-house verification and validation techniques, but it is also due to the complexity and diversity of the interactions between an application and its environment. Many of these interactions can be hardly predicted at testing time, and even when they could be predicted, often there are so many cases to be tested that they cannot be all feasibly addressed before the software is released. Field testing aims to tackle the problem of applications failing in the field by moving the testing phase directly in the field environment. This makes it possible to exploit different scenarios that would otherwise be difficult to capture with in-house testing. In this Ph.D. thesis we explore the area of software field testing, we present a study that characterizes the problem of applications failing in the field, a client-server architecture that can be exploited to organize and control the field testing process and a testing approach that exploits the field itself as testbed for running the test cases. The presented approach is empirically evaluated on a popular dataset of software faults demonstrating that 35% of the faults that were not discovered in-house could have been revealed with field testing.

Mobile apps can be executed with an extremely large set of partially unpredictable configurations.Indeed, they can be executed on an unbounded combination of devices, operating systems, settings, and user preferences since apps may also interact with other apps or devices that were not even available when they were released.This results in a virtually infinite set of configurations that might be responsible for unexpected behaviors which can be validated in-house only to a negligible extent. To address this challenge, this paper discusses the application of in-vivo testing to mobile apps. The main idea is to run test cases in the field, where we exploit the intrinsic heterogeneity and variety of the end-user environment to dramatically increase the range of validated configurations. Actually, the many devices available in-the-field generate a naturally distributed and highly scalable environment that can be exploited to timely validate many configurations as soon as they are observed.&nbsp