Search

We present an empirical study in which model-based testing (MBT) was applied to the software bus of NASA’s Goddard Mission Service Evolution Center (GMSEC), a reusable software framework. The goal was to study the feasibility of using MBT on a real-world software system that was designed to be flexible. GMSEC has three levels of flexibility: 1) loose application coupling through a software bus based on the publish–subscribe architectural style, 2) language independence by providing APIs to the bus in several programming languages, 3) middleware independence by providing wrappers for several middlewares that are supported by the software bus. The novelty brought forward in this paper is that one model and one set of generated test cases were used as the basis to test the software bus for behavioral consistency across multiple programming languages and middleware wrappers. The comparison of costs and benefits from using finite state machines (FSM) vs. extended FSMs (EFSM) when used for MBT on a real-world system is also novel. The case study shows that it was feasible, even for a programmer who neither knew MBT nor the system under test, to successfully apply MBT to a flexible system such as GMSEC and that MBT could within reasonable effort detect non-trivial defects in a fielded system.

There are many technical challenges in ensuring high life-time quality of NASA's systems. Some of NASA's software-related challenges could potentially be addressed by the many powerful technologies that are being developed in software research laboratories. However, most such research technologies do not make the transition from the research lab to the software lab because research infusion and technology transfer is difficult. For example, there must be evidence that the technology works in the practitioner's particular domain, and there must be a potential for great improvements and enhanced competitive edge for the practitioner, for such infusion to take place. NASA IV&V's Research Infusion initiative strives to facilitate such infusion. In 2006, a research infusion project involving Johns Hopkins University Applied Physics Laboratory (JHU/APL) and the Fraunhofer Center for Experimental Software Engineering Maryland, was successfully completed infusing Fraunhofer's s oftware architecture visualization and evaluation (SAVE) tool. The infusion project helped improve JHU/APL's software architecture and produced evidence that SAVE is applicable to software architecture problems in the aerospace domain, spawning a series of related research infusion projects. The project also led to the discovery of other needs that could not be addressed by current technologies and, therefore, spawned the research and development of a new technology that will be ready for infusion in the future. This paper describes the SAVE technology followed by a description of the infusion of SAVE at JHU/APL and the other projects that followed, as well as the newly started Dynamic SAVE research and development project. Lessons learned related to various aspects of research infusion conclude the paper.

This article introduces the Architecture Discovery and Analysis Method (ADAM). ADAM supports the discovery of module and runtime views as well as the analysis of quality attributes, such as testability, performance, and maintainability, of software systems. The premise of ADAM is that the implementation constructs, architecture constructs, concerns, and quality attributes are all influenced by the external entities (e.g., libraries, frameworks, COTS software) used by the system under analysis. The analysis uses such external dependencies to identify, classify, and review a minimal set of key source-code files supported by a knowledge base of the external entities. Given the benefits of analyzing external dependencies as a way to discover architectures and potential risks, it is demonstrated that dependencies to external entities are useful not only for architecture discovery but also for analysis of quality attributes. ADAM is evaluated using the NASA's Space Network Access System (SNAS). The results show that this method offers systematic guidelines for discovering the architecture and locating potential risks (e.g., low testability and decreased performance) that are hidden deep inside the system implementation. Some generally applicable lessons for developers and analysts, as well as threats to validity are also discussed.

We present a case study that evaluates the applicability and effectiveness of model-based testing in detecting bugs in real-world, mission-critical systems. NASA's Operating System ion Layer (OSAL) is the subject system of this paper. The OSAL is a reusable framework that wraps several operating systems (OS) and is used extensively in NASA's flight software missions. We developed a suite of behavioral models, represented as hierarchical finite state machines (FSMs), of the core file system API and generated a large number of test cases automatically. We then automatically executed these test cases against the OSAL. The results show that the OSAL is a high quality product. Naturally, due to the systematic and rigorous nature of MBT, we detected a few previously unknown “corner-case” bugs and issues, which escaped traditional manual testing and code reviews. We discuss the MBT architecture, the detected bugs, the code coverage of generated tests, as well as threats to validity of the study.

In this paper, we report on our experiences with architecture compliance checking - the process of checking whether the planned or specified software architecture is obeyed by the running system - of an OSGi-based, dynamically evolving application in the office domain. To that end, we first show how to dynamically instrument a running system in the context of OSGi in order to collect run-time traces. Second, we explain how to bridge the abstraction gap between run-time traces and software architectures, through the construction of hierarchical Colored Petri nets (CP-nets). In addition, we demonstrate how to design reusable hierarchical CP-nets. In an industry example, we were able to extract views that helped us to identify a number of architecturally relevant issues (e.g., architectural style violations, behavior violations) that would not have been detected otherwise, and could have caused serious problems like system malfunctioning or unauthorized access to sensitive data. Finally, we package valuable experiences and lessons learned from this endeavor.

Engine Control Systems (ECS) for automobiles have many variants for many manufactures and several markets. To improve their development efficiency, exploiting ECS commonalities and predicting their variability are mandatory. The concept of software product line engineering meets this ECS business background. However, we should carefully investigate the expected technical, economical, and organizational effects of introducing the strategy into existing products. Thereafter, a strategy will be derived systematically and realize the desired benefits.This paper reports an experience with the up-front investigation performed for Hitachi's ECS. We focus on the approach to plan the migration of the existing family of individual systems into a future product line. The approach assesses potential ways of merging software from existing variants and eventually defines a procedure for performing the migration. To get a high quality strategy, we integrate the approach of software measurement, the expertise of software architects, and reverse engineering techniques.

Engine Control Systems (ECS) for automobiles have numerous variants for many manufactures and different markets. To improve development efficiency, exploiting ECS commonalities and predicting their variability are mandatory. The concept of software product line engineering meets the business background of ECS. However, we should carefully investigate the expected technical, economical, and organizational effects of introducing this strategy into existing products.This paper explains an approach for assessing the potential of merging existing embedded software into a product line approach. The definition of an economically useful product line approach requires two things: analyzing return on investment (ROI) expectations of a product line and understanding the effort required for building reusable assets. We did a clone analysis to provide the basis for effort estimation for merge potential assessment of existing variants. We also report on a case study with ECS. We package the lessons learned and open issues that arose during the case study.

To establish software reuse successfully in the long run, it is crucial for providers of reusable components to continuously react on problems or future trends arising around their component. In practice, however, many providers of reusable components are not able to do so due to insufficient feedback and details from reusers. Additionally, they often have too little knowledge on system context and constraints that may lead to major deficits of the reusable component especially with respect to non-functional aspects. This paper presents an approach for systematically engineering performance of reusable components that has been validated in an industrial context.