Search

The detection of performance bugs, like those causing an unexpected execution time, has gained much attention in the last years due to their potential impact in safety-critical and resource-constrained applications. Much effort has been put on trying to understand the nature of performance bugs in different domains as a starting point for the development of effective testing techniques. However, the lack of a widely accepted classification scheme of performance faults and, more importantly, the lack of well-documented and understandable datasets makes it difficult to draw rigorous and verifiable conclusions widely accepted by the community. In this paper, we present TANDEM, a dual contribution related to real-world performance bugs. Firstly, we propose a taxonomy of performance bugs based on a thorough systematic review of the related literature, divided into three main categories: effects, causes and contexts of bugs. Secondly, we provide a complete collection of fully documented real-world performance bugs. Together, these contributions pave the way for the development of stronger and reproducible research results on performance testing.

Performance bugs are known to be a major threat to the success of software products. Performance tests aim to detect performance bugs by executing the program through test cases and checking whether it exhibits a noticeable performance degradation. The principles of mutation testing, a well-established testing technique for the assessment of test suites through the injection of artificial faults, could be exploited to evaluate and improve the detection power of performance tests. However, the application of mutation testing to assess performance tests, henceforth called performance mutation testing (PMT), is a novel research topic with numerous open challenges. In previous papers, we identified some key challenges related to PMT. In this work, we go a step further and explore the feasibility of applying PMT at the source-code level in general-purpose languages. To do so, we revisit concepts associated with classical mutation testing, and design seven novel mutation operators to model known bug-inducing patterns. As a proof of concept, we applied traditional mutation operators as well as performance mutation operators to open-source C++ programs. The results reveal the potential of the new performance-mutants to help assess and enhance performance tests when compared to traditional mutants. A review of live mutants in these programs suggests that they can induce the design of special test inputs. In addition to these promising results, our work brings a whole new set of challenges related to PMT, which will hopefully serve as a starting point for new contributions in the area.

Mutation testing is a suitable technique to determine the quality of test suites designed for a certain program. The set of mutation operators and the overall technique should be developed around each programming language in particular. The structures related to the object-oriented paradigm require a tailored analysis addressing them. However, class mutation operators for these languages have not been analyzed at the same extent as traditional operators for procedural languages in the literature. The purpose of the chapter is to look in depth at the development and the current state of mutation testing, and more specifically, with regard to object-oriented programming languages.

For software quality assurance, many safety-critical industries appeal to the use of dynamic testing and structural coverage criteria. However, there are reasons to doubt the adequacy of such practices. Mutation testing has been suggested as an alternative or complementary approach but its cost has traditionally hindered its adoption by industry, and there are limited studies applying it to real safety-critical code. This paper evaluates the effectiveness of state-of-the-art mutation testing on safety-critical code from within the U.K. nuclear industry, in terms of revealing flaws in test suites that already meet the structural coverage criteria recommended by relevant safety standards. It also assesses the practical feasibility of implementing such mutation testing in a real setting. We applied a conventional selective mutation approach to a C codebase supplied by a nuclear industry partner and measured the mutation score achieved by the existing test suite. We repeated the experiment using trivial compiler equivalence (TCE) to assess the benefit that it might provide. Using a conventional approach, it first appeared that the existing test suite only killed 82% of the mutants, but applying TCE revealed that it killed 92%. The difference was due to equivalent or duplicate mutants that TCE eliminated. We then added new tests to kill all the surviving mutants, increasing the test suite size by 18% in the process. In conclusion, mutation testing can potentially improve fault detection compared to structural-coverage-guided testing, and may be affordable in a nuclear industry context. The industry feedback on our results was positive, although further evidence is needed from application of mutation testing to software with known real faults.

Los defectos de rendimiento del software pueden causar una importante degradación en la experiencia de usuario y dar lugar a pro blemas muy costosos de detectar y resolver. Las pruebas de rendimiento persiguen detectar estos defectos y reducir su impacto. Sin embargo, no existen mecanismos para evaluar la calidad de las pruebas de rendimien to, causando en muchos casos, que estos problemas pasen desapercibidos. La prueba de mutación es una técnica para evaluar y mejorar las pruebas funcionales a través de la introducción de fallos artificiales en el programa bajo prueba. En este artículo, exploramos la aplicabilidad de la prueba de mutación junto con el empleo de un algoritmo evolutivo para buscar mutantes que simulen errores de rendimiento. Esta propuesta novedosa contribuye a mejorar la confianza en las pruebas de rendimiento al mismo tiempo que reduce el coste de la prueba de mutación. Ministerio de Economía y Competitividad TIN2015-65845-C3-3-R Ministerio de Economía y Competitividad BELI (TIN2015-70560-R) Ministerio de Ciencia, Innovación y Universidades TIN2015-71841-REDT

Performance bugs are common, costly, and elusive. Performance tests aim to detect performance bugs by running the program with specific inputs and determining whether the observed behaviour is acceptable. There not exist mechanisms, however, to assess the effectiveness of performance tests. Mutation testing is a technique to evaluate and enhance functional test suites by seeding artificial faults in the program under test. In this new idea paper, we explore the applicability of mutation testing to assess and improve performance tests. This novel approach is motivated with examples and open questions.

La enseñanza y aprendizaje de habilidades de programación informática son procesos complejos para el profesor y para el alumno respectivamente. Como tal, haciendo uso de avances tecnológicos, en los últimos años han surgido diversas propuestas para apoyar al alumno en esta tarea, así como para ayudar al profesor en la evaluación de asignaturas en las que se imparten estas habilidades. Entre estas propuestas es popular la ejecución de conjuntos de casos de prueba, lo cual permite comprobar que el programa que escribe el alumno en las prácticas satisface la funcionalidad requerida. Sin embargo, este enfoque no permite detectar si el alumno cumple ciertos requisitos que estaban establecidos. Este capítulo se centra en el análisis directo del código para la comprobación del cumplimiento de los requisitos requeridos en los enunciados de las prácticas. Este análisis, conocido como estático, es automático y cuenta con varios beneficios tanto para el alumno como para el profesor, destacando que se acelera la retroalimentación que obtiene el estudiante en todo momento. El objetivo del capítulo es describir la experiencia en una asignatura de programación orientada a objetos durante dos cursos aplicando dichas comprobaciones sobre el código, mostrando en qué consiste y cómo se ha desarrollado tal innovación. Además, se muestran resultados orientativos de su aceptación por parte de los alumnos y su utilidad.

Mutation testing is a method used to assess and improve the fault detection capability of a test suite by creating faulty versions, called mutants, of the system under test. Evolutionary Mutation Testing (EMT), like selective mutation or mutant sampling, was proposed to reduce the computational cost, which is a major concern when applying mutation testing. This technique implements an evolutionary algorithm to produce a reduced subset of mutants but with a high proportion of mutants that can help the tester derive new test cases (strong mutants). In this paper, we go a step further in estimating the ability of this technique to induce the generation of test cases. Instead of measuring the percentage of strong mutants within the subset of generated mutants, we compute how much the test suite is actually improved thanks to those mutants. In our experiments, we have compared the extent to which EMT and the random selection of mutants help to find missing test cases in C++ object-oriented systems. We can conclude from our results that the percentage of mutants generated with EMT is lower than with the random strategy to obtain a test suite of the same size and that the technique scales better for complex programs.

The reduction of the expenses of mutation testing should be based on well-studied cost reduction techniques to avoid bi ased results. Evolutionary Mutation Testing (EMT) aims at generating a reduced set of mutants by means of an evolu tionary algorithm, which searches for potentially equivalent and difficult to kill mutants to help improve the test suite. However, there is little evidence of its applicability to other contexts beyond WS-BPEL compositions. This study ex plores its performance when applied to C++ object-oriented programs thanks to a newly developed system, GiGAn. The conducted experiments reveal that EMT shows stable be havior in all the case studies, where the best results are ob tained when a low percentage of the mutants is generated. They also support previous studies of EMT when compared to random mutant selection, reinforcing its use for the goal of improving the fault detection capability of the test suite. Universidad de Cádiz PU-EPIF-FPI-PPI-BC 2012-037 Ministerio de Economía y Competitividad TIN2015-65845-C3-3-R (DarDos) Ministerio de Economía y Competitividad TIN2015-70560-R (BELI) Ministerio de Ciencia, Innovación y Universidades TIN2015-71841-REDT (SEBASEnet) Junta de Andalucía P12-TIC-1867 (COPAS)

A pesar del beneficio que puede reportar la prueba de muta ciones en el proceso de prueba de software, el coste que supone su apli cación siempre ha sido visto como un obstáculo para una mayor acogida por parte de la industria. Por esta razón, se han desarrollado diversas técnicas que tratan de paliar el problema, principalmente mediante la reducción del número de mutantes que son generados. Entre ellas se en cuentra la Prueba de Mutación Evolutiva (PME), que propone el empleo de algoritmos evolutivos para encontrar un subconjunto de mutantes que presenta mayor posibilidad de ayudar a refinar el conjunto de casos de prueba empleado. La técnica solo había sido probada con éxito en ope radores para el lenguaje de programación WS-BPEL. En este artículo se presentan los experimentos llevados a cabo aplicando la técnica de PME con mutantes generados por operadores de mutación para C++ relacio nados con la orientación a objetos. Los resultados obtenidos, usando los parámetros considerados como más apropiados para la configuración del algoritmo, revelan que la técnica también es más efectiva que una estra tegia aleatoria con operadores de clase para sistemas en C++. Universidad de Cádiz PU-EPIF-FPI-PPI-BC 2012-037 Ministerio de Ciencia, Innovación y Universidades TIN2015-71841-REDT Ministerio de Economía y Competitividad TIN2015-65845-C3-3-R (DarDos) Ministerio de Economía y Competitividad TIN2012-32273 (TAPAS) Junta de Andalucía TIC-5906 (THEOS) Junta de Andalucía P12-TIC-1867 (COPAS)