Search

Enhancing the trust of machine learning-based classifiers with large input spaces is a desirable goal; however, due to high labeling costs and limited resources, this is a challenging task. One solution is to use test input prioritization techniques that aim to identify and label only the most effective test instances. These prioritized test inputs can then be used with some popular testing techniques e.g., Metamorphic testing (MT) to test and uncover implementation bugs in computationally complex machine learning classifiers that suffer from the oracle problem. However, there are certain limitations involved with this approach, (i) using a small number of prioritized test inputs may not be enough to check the program correctness over a large variety of input scenarios, and (ii) traditional MT approaches become infeasible when the programs under test exhibit a non-deterministic behavior during training e.g., Neural Network-based classifiers. Therefore, instead of using MT for testing purposes, we propose a metamorphic relation to solve a data generation/labeling problem; that is, enhancing the test inputs effectiveness by extending the prioritized test set with new tests without incurring additional labeling costs. Further, we leverage the prioritized test inputs (both source and follow-up data sets) and propose a statistical hypothesis testing (for detection) and machine learning-based approach (for prediction) of faulty behavior in two other machine learning classifiers (Neural Network-based Intrusion Detection Systems). In our case, the problem is interesting in the sense that injected bugs represent the high accuracy producing mutated program versions that may be difficult to detect by a software developer. The results indicate that (i) the proposed statistical hypothesis testing is able to identify the induced buggy behavior, and (ii) Random Forest outperforms and achieves the best performance over SVM and k-NN algorithms.

Testing computationally complex neural network-based applications (i.e. network intrusion detection systems) is a challenging task due to the absence of a test oracle. Metamorphic testing is a method to potentially solve the oracle problem when the correctness of individual output is difficult to determine. However, due to the stochastic nature of these applications, multiple runs with the same input can produce slightly different results; thus rendering traditional metamorphic testing technique inadequate. To address this problem, this paper proposes a statistical metamorphic testing technique to test neural network based Network Intrusion Detection Systems (N-IDSs) in a nondeterministic environment. We also performed mutation analysis to show the effectiveness of the proposed approach. The results show that the proposed method has a strong defect detection capability and is able to kill 100% implementation bugs in two neural network-based N-IDSs, and 66.66% in a neural network-based cancer prediction system.

Context: The technical debt (TD) metaphor helps to conceptualize the pending issues and trade-offs made during software development. Knowing TD causes can support in defining preventive actions and having information about effects aids in the prioritization of TD payment. Goal: To investigate the impact of the experience level on how practitioners perceive the most likely causes that lead to TD and the effects of TD that have the highest impacts on software projects. Method: We approach this topic by surveying 227 practitioners. Results: While experienced software developers focus on human factors as TD causes and external quality attributes as TD effects, low experienced developers seem to concentrate on technical issues as causes and internal quality issues and increased project effort as effects. Missing any of these types of causes could lead a team to miss the identification of important TD, or miss opportunities to preempt TD. On the other hand, missing important effects could hamper effective planning or erode the effectiveness of decisions about prioritizing TD items. Conclusion: Having software development teams composed of practitioners with a homogeneous experience level can erode the team's ability to effectively manage TD.

Context: Knowing the effects of technical debt (TD) can support software development teams in the prioritization of TD items to pay off. However, little is known about the relations between the effects of TD and TD payment practices. Having this knowledge can provide valuable information for decision making about which payment practice can be applied given the presence of specific effects of TD. Aims: To investigate, from the point of view of software practitioners, (i) which TD payment practices have been used when certain effects of the presence of debt are felt in software projects and (ii) the reasons for not paying debt items despite the effects they are causing to the project. Method: We analyze quantitatively and qualitatively data collected from a survey with 432 practitioners across four countries. Results: Among the identified relations, the practice "code refactoring" is commonly used to pay debt items off when the effects "delivery delay" and "rework" are felt in software projects. On the other hand, when practitioners face the TD effects "low external quality" and "delivery delay", ，they commonly justify the non- payment of the debt items indicating the need of "focusing on short term goals". Conclusion: We organize the relationship between TD effects, and payment practices and reasons for not eliminating debt items. All this information is structured in an alluvial diagram, which can facilitate the visualization of the identified relations.

Context: Managing technical debt (TD) associated with external cybersecurity attacks on an organization can significantly improve decisions made when prioritizing which security weaknesses require attention. Whilst source code vulnerabilities can be found using static analysis techniques, malicious external attacks expose the vulnerabilities of a system at runtime and can sometimes remain hidden for long periods of time. By mapping malicious attack tactics to the consequences of weaknesses (i.e. exploitable source code vulnerabilities) we can begin to understand and prioritize the refactoring of the source code vulnerabilities that cause the greatest amount of technical debt on a system. Goal: To establish an approach that maps common external attack tactics to system weaknesses. The consequences of a weakness associated with a specific attack technique can then be used to determine the technical debt principal of said violation; which can be measured in terms of loss of business rather than source code maintenance. Method: We present a position study that uses Jaccard similarity scoring to examine how 11 malicious attack tactics can relate to Common Weakness Enumerations (CWEs). Results: We conduct a study to simulate attacks, and generate dependency graphs between external attacks and the technical consequences associated with CWEs. Conclusion: The mapping of cyber security attacks to weaknesses allows operational staff (SecDevOps) to focus on deploying appropriate countermeasures and allows developers to focus on refactoring the vulnerabilities with the greatest potential for technical debt.

Context: We investigate the different perceptions of quality provided by leading operational quality models when used to evaluate software systems from an industry perspective. Goal: To compare and evaluate the quality assessments of two competing quality models and to develop an extensible solution to meet the quality assurance measurement needs of an industry stakeholder -The Construction Engineering Research Laboratory (CERL). Method: In cooperation with our industry partner TechLink, we operationalize the Quamoco quality model and employ a multiple case study design comparing the results of Quamoco and SQALE, two implementations of well known quality models. The study is conducted across current versions of several open source software projects sampled from GitHub and commercial software for sustainment management systems implemented in the C# language from our industry partner. Each project represents a separate embedded unit of study in a given context -open source or commercial. We employ inter-rater agreement and correlation analysis to compare the results of both models, focusing on Maintainability, Reliability, and Security assessments. Results: Our observations suggest that there is a significant disconnect between the assessments of quality under both quality models. Conclusion: In order to support industry adoption, additional work is required to bring competing implementations of quality models into alignment. This exploratory case study helps us shed light into this problem.

Providing software developers and researchers with useful technical debt analysis tools is an instrumental outcome of software engineering and technical debt research. Such tools aggregate industry best practices to provide users with organized and quantifiable metrics that can help minimize the time it takes to synthesize and make an intelligent future decision regarding a system. Today, most tools rely primarily on structural measurements from static analysis to generate results. However, it is also necessary to consider measurements that capture the behavior of software, as these represent additional complexities within a system that structural measurements are incapable of detecting. Herein, we present our position, that more effort needs to be placed towards understanding software behavior so that technical debt analysis tools can begin supporting them, in order to provide tool users with a more accurate and complete view of their system. In this paper, we describe this problem in the context of design patterns and outline an effective method to talk about behaviors in the future. We create and classify two example behaviors using our method, both of which increase the technical debt in their respective design pattern applications.

Context: Software maintenance is important to keep existing software systems functional for organizations or users that depend on that software. Goal: We aim to identify the factors, i.e., software characteristics such as code complexity, leading to maintenance problems. Method: We present a Mapping Study (MS) on controlled experiments that investigated software characteristics related to defects during maintenance. Results: The search strategy identified 78 papers, of which 9 have been included in our study, dated from 1985 to 2013, after applying our inclusion and exclusion criteria. We extracted data from these papers to identify the research methods, and the independent, dependent, blocked, and measured variables. Conclusions: Our MS results point to a weak evidence on software factors causing defects during maintenance. Stronger evidence can be developed via more controlled experiments that address multiple independent variables and hold the software objects constant.

Technical debt has recently become a major concern in the software industry. While it has been shown that technical debt has an adverse effect on the quality of a software system, there has been little work to explore this relationship. Further, with the growing number of approaches to estimate the technical debt principal of a software system, there is a dearth of work to empirically validate the relationship between technical debt scores produced by practical tools against established theoretical quality models. We conducted a case study across 10 releases of 10 open source systems in order to evaluate three proposed methods of technical debt principal estimation. The evaluation compares each technique against an external quality model. We found that only one estimation technique had a strong correlation to the quality attributes reusability and understand ability. In a multiple linear regression analysis we also found that a different estimation technique had a significant relationship to the quality attributes effectiveness and functionality. These results indicate that it is important that industry practitioners, ensure that the technical debt estimate they employ accurately depicts the effects of technical debt as viewed from their quality model.

Data in the natural sciences can often be dense and difficult to understand. The process of visualizing this data helps to alleviate these issues. In this paper, we demonstrate how using digital theater software built for planetaria can be an appropriate medium in which to facilitate these visualizations; not only because of the software's complex and comprehensive graphics engine, but also as a means to convey the information this data is representing to the general public in way that is understandable, accessible and enjoyable. We exemplify the use of this technology through a case study where the simulation of water molecules in the Nyack Floodplain on the Middle Fork Flathead River, are visualized with dome technology.

Measurements are subject to random and systematic errors, yet almost no study in software engineering makes significant efforts in reporting these errors. Whilst established statistical techniques are well suited for the analysis of random error, such techniques are not valid in the presence of systematic errors. We propose a departure from de- facto methods of reporting results of technical debt measurements for more rigorous techniques drawn from established methods in the physical sciences. This line of inquiry focuses on technical debt calculations; however it can be generalized to quantitative software engineering studies. We pose research questions and seek answers to the identification of systematic errors in metric-based tools, as well as the reporting of such errors when subjected to propagation. Exploratory investigations reveal that the techniques suggested allow for the comparison of uncertainties that come from differing sources. We suggest the study of error propagation of technical debt is a worthwhile subject for further research and techniques seeded from the physical sciences present viable options that can be used in software engineering reporting.

Network Exchange Objects (NEO) is a new software framework designed to facilitate development of complex natural or built distributed system models, where the system model is represented as a graph, through which currencies (e.g., coding information) flux. This paper introduces "system-level hypothesis (SLH) testing" as a form of computational thinking that will drive integration of computational and empirical sciences to promote efficient, self- correcting inquiry into the operations and behavior of complex systems. To demonstrate NEO, we examine the problem of maximizing the productivity of a software development organization by measuring growth in the total lines of code (LOC) contributed by developers. We develop a software framework (NEO) that allows rapid creation of model variants representing alternative SLHs. NEO is designed to investigate systems we describe as "complex adaptive hierarchical networks" (CAHNs - complex systems represented as networks that route and store multiple interactive currencies). Models built atop NEO, are organized collections of individual values (model variables) and algorithms (model logic). Modelers systematically combine algorithms to create alternative model formulations at runtime. Thus, NEO is a simulation framework that can be used in any domain of expertise, where systems are represented as interdependent entities that store and flux multiple currencies.

We created an application that facilitates improved knowledge discovery from aircraft maintenance data by transforming transactional database records into ontology-based event graphs, and then providing a filterable visualization of event sequences through time. We developed OWL ontologies based on formally defined IEEE standards, and use these ontologies to guide the data mining and data transformation processes. Our application removes much of the users burden for data look-up and greatly increases the potential for knowledge discovery from data (KDD) in this field. We provide an easy-to-use interface that generates relevant sequences of data in a meaningful context in a fraction of the time it would take domain experts to retrieve and display similar information.

In model-based diagnostic algorithms, it is assumed that the model is correct. If the model is incorrect, the diagnostic algorithm may diagnose the wrong fault, which can be costly and time consuming. Using past maintenance events, one should be able to make corrections to the model in order for diagnostic algorithm to correctly diagnosis faults. In this paper, a maturation approach is proposed which uses the graph-theoretic representations of Timed Failure Propagation Graph (TFPG) models and diagnostic sessions based on recently standardized diagnostic ontologies to determine statistical discrepancies between that which is expected by the models and that which has been encountered in practice. These discrepancies are then analyzed to generate recommendations for maturing the diagnostic models. Maturation recommendations include identifying new dependencies and erroneous or tenuous dependencies. 1 2

Evidence suggests that as software ages the original realizations of design patterns remain in place, and participants in design pattern realizations accumulate "grime" - non-pattern-related code. This research examines the consequences that grime buildup has on the testability of general purpose design patterns. Test cases put in place during the design phase and initial implementation of a project can become ineffective as the system matures. The evolution of a design due to added functionality or defect fixing increases the coupling and dependencies between many classes that must be tested. We show that as systems age, the growth of grime and the appearance of anti-patterns increase testing requirements. Early recognition and removal of grime and anti-patterns can potentially improve system testability.