Search

Gamification is an established practice in Software Engineering to increase effectiveness and engagement in many practices. This manuscript provides a characterization of the application of gamification to the Software Testing area. Such practice in fact reportedly suffers from low engagement by both personnel in industrial contexts and learners in educational contexts. Our goal is to identify the application areas and utilized gamified techniques and mechanics, the provided benefits and drawbacks, as well as the open challenges in the field. To this purpose, we conducted a Multivocal Literature Review to identify white and grey literature sources addressing gamified software testing. We analyzed 73 contributions and summarized the most common gamified mechanics, concepts, tools, and domains where they are mostly applied. We conclude that gamification in software testing is mostly applied to the test creation phase with simple white-box unit or mutation testing tools and is mostly used to foster good behaviors by promoting the testers' accomplishment. Key research areas and main challenges in the field are: careful design of tailored gamified mechanics for specific testing techniques; the need for technological improvements to enable crowdsourcing, cooperation, and concurrency; the necessity for empirical and largescale evaluation of the benefits delivered by gamification mechanics. [ABSTRACT FROM AUTHOR]

Context: Software regression testing refers to rerunning test cases after the system under test is modified, ascertaining that the changes have not (re-)introduced failures. Not all researchers' approaches consider applicability and scalability concerns, and not many have produced an impact in practice. Objective: One goal is to investigate industrial relevance and applicability of proposed approaches. Another is providing a live review, open to continuous updates by the community. Method: A systematic review of regression testing studies that are clearly motivated by or validated against industrial relevance and applicability is conducted. It is complemented by follow-up surveys with authors of the selected papers and 23 practitioners. Results: A set of 79 primary studies published between 2016-2022 is collected and classified according to approaches and metrics. Aspects relative to their relevance and impact are discussed, also based on their authors' feedback. All the data are made available from the live repository that accompanies the study. Conclusions: While widely motivated by industrial relevance and applicability, not many approaches are evaluated in industrial or large-scale open-source systems, and even fewer approaches have been adopted in practice. Some challenges hindering the implementation of relevant approaches are synthesized, also based on the practitioners' feedback. [ABSTRACT FROM AUTHOR]

Instance Space Analysis (ISA) is a recently developed methodology to (a) support objective testing of algorithms and (b) assess the diversity of test instances. Representing test instances as feature vectors, the ISA methodology extends Rice’s 1976 Algorithm Selection Problem framework to enable visualization of the entire space of possible test instances, and gain insights into how algorithm performance is affected by instance properties. Rather than reporting algorithm performance on average across a chosen set of test problems, as is standard practice, the ISA methodology offers a more nuanced understanding of the unique strengths and weaknesses of algorithms across different regions of the instance space that may otherwise be hidden on average. It also facilitates objective assessment of any bias in the chosen test instances and provides guidance about the adequacy of benchmark test suites. This article is a comprehensive tutorial on the ISA methodology that has been evolving over several years, and includes details of all algorithms and software tools that are enabling its worldwide adoption in many disciplines. A case study comparing algorithms for university timetabling is presented to illustrate the methodology and tools. [ABSTRACT FROM AUTHOR]

Machine learning (ML) techniques increase the effectiveness of software engineering (SE) lifecycle activities. We systematically collected, quality-assessed, summarized, and categorized 83 reviews in ML for SE published between 2009 and 2022, covering 6,117 primary studies. The SE areas most tackled with ML are software quality and testing, while human-centered areas appear more challenging for ML. We propose a number of ML for SE research challenges and actions, including conducting further empirical validation and industrial studies on ML, reconsidering deficient SE methods, documenting and automating data collection and pipeline processes, reexamining how industrial practitioners distribute their proprietary data, and implementing incremental ML approaches. [ABSTRACT FROM AUTHOR]

Security attacks abuse software vulnerabilities of IoT devices; hence, detecting and eliminating these vulnerabilities immediately are crucial. Fuzzing is an efficient method to identify vulnerabilities automatically, and many publications have been released to date. However, fuzzing for embedded systems has not been studied extensively owing to various obstacles, such as multi-architecture support, crash detection difficulties, and limited resources. Thus, the article introduces fuzzing techniques for embedded systems and the fuzzing differences for desktop and embedded systems. Further, we collect state-of-the-art technologies, discuss their advantages and disadvantages, and classify embedded system fuzzing tools. Finally, future directions for fuzzing research of embedded systems are predicted and discussed. [ABSTRACT FROM AUTHOR]

Bayesian data analysis is gaining traction in many fields, including empirical studies in software engineering. Bayesian approaches provide many advantages over traditional, or frequentist, data analysis, but the mechanics often remain opaque to beginners due to the underlying computational complexity. Introductory articles, while successful in explaining the theory and principles, fail to provide a totally transparent operationalization. To address this gap, this tutorial provides a step-by-step illustration of Bayesian hypothesis testing in the context of software engineering research using a fully developed example and in comparison to the frequentist hypothesis testing approach. It shows how Bayesian analysis can help build evidence over time incrementally through a family of experiments. It also discusses chief advantages and disadvantages in an applied manner. A figshare package is provided for reproducing all calculations. [ABSTRACT FROM AUTHOR]

Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements, and how field tests are managed, and identify many challenging research directions. [ABSTRACT FROM AUTHOR]

Although testing effort estimation is a very important task in software project management, it is rarely described in the literature. There are many difficulties in finding any useful methods or tools for this purpose. Solutions to many other problems related to testing effort calculation are published much more often. There is also no research focusing on both testing effort estimation and all related areas of software engineering. To fill this gap, we performed a systematic literature review on both questions. Although our primary objective was to find some tools or implementable metods for test effort estimation, we have quickly discovered many other interesting topics related to the main one. The main contribution of this work is the presentation of the testing effort estimation task in a very wide context, indicating the relations with other research fields. This systematic literature review presents a detailed overview of testing effort estimation task, including challenges and approaches to automating it and the solutions proposed in the literature. It also exhaustively investigates related research topics, classifying publications that can be found in connection to the testing effort according to seven criteria formulated on the basis of our research questions. We present here both synthesis of our finding and the deep analysis of the stated research problems. [ABSTRACT FROM AUTHOR]

Context: The mobile app market is continually growing offering solutions to almost all aspects of people's lives, e.g., healthcare, business, entertainment, as well as the stakeholders' demand for apps that are more secure, portable, easy to use, among other non-functional requirements (NFRs). Therefore, manufacturers should guarantee that their mobile apps achieve high-quality levels. A good strategy is to include software testing and quality assurance activities during the whole life cycle of such solutions. Problem: Systematically warranting NFRs is not an easy task for any software product. Software engineers must take important decisions before adopting testing techniques and automation tools to support such endeavors. Proposal: To provide to the software engineers with a broad overview of existing dynamic techniques and automation tools for testing mobile apps regarding NFRs. Methods: We planned and conducted a Systematic Mapping Study (SMS) following well-established guidelines for executing secondary studies in software engineering. Results: We found 56 primary studies and characterized their contributions based on testing strategies, testing approaches, explored mobile platforms, and the proposed tools. Conclusions: The characterization allowed us to identify and discuss important trends and opportunities that can benefit both academics and practitioners. [ABSTRACT FROM AUTHOR]

Equipped with an abundance of small-scale microelectromechanical sensors, modern mobile devices such as smartphones and smartwatches can now offer context-aware services to users in mobile environments. Although advances in mobile context-aware applications have made our everyday environments increasingly intelligent, these applications are prone to bugs that are highly difficult to reproduce and repair. Compared to conventional computer software, mobile context-aware applications often have more complex structures to process a wide variety of dynamic context data in specific scenarios. Accordingly, researchers have proposed diverse context simulation techniques to enable low-cost and effective tests instead of conducting costly and time-consuming real-world experiments. This article aims to give a comprehensive overview of the state-of-the-art context simulation methods for testing mobile context-aware applications. In particular, this article highlights the technical distinctions and commonalities in previous research conducted across multiple disciplines, particularly at the intersection of software testing, ubiquitous computing, and mobile computing. This article also discusses how each method can be implemented and deployed by testing tool developers and mobile application testers. Finally, this article identifies several unexplored issues and directions for further advancements in this field. [ABSTRACT FROM AUTHOR]

Testing is one of the most important phases in the development of any product or software. Various types of software testing exist that have to be done to meet the need of the software. Regression testing is one of the crucial phases of testing where testing of a program is done for the original test build along with the modifications. In this article, various studies proposed by the authors have been analysed focusing on test cases generation and their approach toward web application. A detailed study was conducted on Regression Test Case Generation and its approaches toward web application. From our detailed study, we have found that very few approaches and methodologies have been found that provide the real tool for test case generation. There is a need of an automated regression testing tool to generate the regression test cases directly based on user requirements. These test cases have to be generated and implemented by the tool so that the reduction in the overall effort and cost can be achieved. From our study, we have also found that regression testing for web applications was not investigated much, but in today's scenario web applications are an integral part of our daily life and so that needs to be tested for regression testing. [ABSTRACT FROM AUTHOR]

Many security and software testing applications require checking whether certain properties of a program hold for any possible usage scenario. For instance, a tool for identifying software vulnerabilities may need to rule out the existence of any backdoor to bypass a program's authentication. One approach would be to test the program using different, possibly random inputs. As the backdoor may only be hit for very specific program workloads, automated exploration of the space of possible inputs is of the essence. Symbolic execution provides an elegant solution to the problem, by systematically exploring many possible execution paths at the same time without necessarily requiring concrete inputs. Rather than taking on fully specified input values, the technique abstractly represents them as symbols, resorting to constraint solvers to construct actual instances that would cause property violations. Symbolic execution has been incubated in dozens of tools developed over the past four decades, leading to major practical breakthroughs in a number of prominent software reliability applications. The goal of this survey is to provide an overview of the main ideas, challenges, and solutions developed in the area, distilling them for a broad audience. [ABSTRACT FROM AUTHOR]

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research. [ABSTRACT FROM AUTHOR]

Regression test case selection techniques attempt to increase the testing effectiveness based on the measurement capabilities, such as cost, coverage, and fault detection. This systematic literature review presents state-of-the-art research in effective regression test case selection techniques. We examined 47 empirical studies published between 2007 and 2015. The selected studies are categorized according to the selection procedure, empirical study design, and adequacy criteria with respect to their effectiveness measurement capability and methods used to measure the validity of these results. The results showed that mining and learning-based regression test case selection was reported in 39% of the studies, unit level testing was reported in 18% of the studies, and object-oriented environment (Java) was used in 26% of the studies. Structural faults, the most common target, was used in 55% of the studies. Overall, only 39% of the studies conducted followed experimental guidelines and are reproducible. There are 7 different cost measures, 13 different coverage types, and 5 fault-detection metrics reported in these studies. It is also observed that 70% of the studies being analyzed used cost as the effectiveness measure compared to 31% that used fault-detection capability and 16% that used coverage. [ABSTRACT FROM AUTHOR]

Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques. [ABSTRACT FROM AUTHOR]

The functional correctness technique is presented and discussed. It is also explained that the underlying theory has an implication for the derivation of loop invariants. The functional verification conditions concerning program loops are then shown to be a specialization of the commonly used inductive assertion verification conditions. Next, the function technique is compared and contrasted with subgoal induction. Finally, the difficulty of proving initialized loop programs is examined in light of both the inductive assertion and functional correctness theories. [ABSTRACT FROM AUTHOR]

Several software quality assessment methods which span the software life cycle are discussed. The quality of a system design can be estimated by measuring the system entropy function or the system work function. The quality improvement due to reconfiguration can be determined by calculating system entropy loading measures. Software science and Zipf's law are shown to be useful for estimating program length and implementation time. Deterministic and statistical methods are presented for predicting the number of errors. Testing theory is useful in planning the program test process; as discussed in this paper, it includes measurement of program structural characteristics to determine test effectiveness and test planning. Statistical models for estimating software reliability are also discussed. [ABSTRACT FROM AUTHOR]

Testing is the process of determining whether a task has been correctly carried out. Testing should be performed throughout the software life cycle; it should not be restricted to verification and validation. The correction of a fault exposed by testing is termed debugging. The goal of testing is to reveal faults. There are two types of testing: execution-based testing and nonexecution-based testing. It is impossible to execute a specification document; instead, it has to be reviewed carefully. Once executable code has been written it is possible to run test cases, that is, perform execution-based testing. Nevertheless, the existence of code does not preclude nonexecution-based testing because carefully reviewing code will uncover at least as many faults as running test cases. Computer software is tested in order to detect faults. However, instead of using better techniques for detecting faults, it is more effective to employ software-development approaches that reduce the number of faults in the software. Thus, the role of testing will be to prevent faults rather than to detect them.