Search

Software engineering for big data systems is complex and faces challenges including pervasive distribution, write-heavy workloads, variable request loads, computation-intensive analytics, and high availability. The articles in this theme issue examine several facets of this complicated puzzle. The Web extra at https://youtu.be/YKBGf9EOBUo is an audio recording of Davide Falessi speaking with Ayse Basar Bener and Audris Mockus about the authors, articles, and discussions that went into the IEEE Software March/April 2016 theme issue on software engineering for big data systems.

Agile teams strive to balance short-term feature development with longer-term quality concerns. These evolutionary approaches often hit a "complexity wall"' from the cumulative effects of unplanned changes, resulting in unreliable, poorly performing software. So, the agile community is refocusing on approaches to address architectural concerns. Researchers analyzed quality attribute concerns from 15 years of Architecture Trade-Off Analysis Method data, gathered from 31 projects. Modifiability was the dominant concern across all project types; performance, availability, and interoperability also received considerable attention. For IT projects, a relatively new quality-deployability-emerged as a key concern. The study results provide insights for agile teams allocating architecture-related tasks to iterations. For example, teams can use these results to create checklists for release planning or retrospectives to help assess whether to address a given quality to support future needs. This article is part of a special issue on Software Architecture.

Exponential data growth from the Internet, low-cost sensors, and high-fidelity instruments have fueled the development of advanced analytics operating on vast data repositories. These analytics bring business benefits ranging from Web content personalization to predictive maintenance of aircraft components. To construct the data repositories underpinning these systems, rapid innovation has occurred in distributed-data-management technologies, employing schemaless data models and relaxing consistency guarantees to satisfy scalability and availability requirements. These big data systems present many challenges to software architects. Distributed-software architecture quality attributes are tightly linked to both the data and deployment architectures. This causes a consolidation of concerns, and designs must be closely harmonized across these three architectures to satisfy quality requirements.

Many science domains need to build computationally efficient and accurate representations of high fidelity, computationally expensive simulations known as reduced-order models (ROMs). This article presents the design and implementation of the Reveal toolset, a ROM builder that generates ROMs based on science- and engineering-domain-specific simulations executed on high-performance computing (HPC) platforms. The toolset encompasses a range of sampling and regression methods for ROM generation, automatically quantifies ROM accuracy, and supports an iterative approach to improve ROM accuracy. Reveal is designed to be extensible for any simulator that has published input and output formats. It also defines programmatic interfaces to include new sampling and regression techniques so users can mix and match mathematical techniques best suited to their model characteristics. The article describes the architecture of Reveal and demonstrates its use with a computational fluid dynamics model used in carbon capture.

This is the first article we're fortunate to have from the SATURN 2012 conference. This issue highlights a compelling story of crisis, larceny, and, of course, Fortran-the first programming language I learned, which I did by teaching it to undergrad engineering students. Because I never saw those engineers after they left my class, it's heartening to see that others like them learned some good lessons and developed useful insights. I'm sure you will enjoy reading this architectural tale.

Velo is a reusable, domain-independent knowledge-management infrastructure for modeling and simulation. Velo leverages, integrates, and extends Web-based open source collaborative and data-management technologies to create a scalable and flexible core platform tailored to specific scientific domains. As the examples here describe, Velo has been used in both the carbon sequestration and climate modeling domains.

State-of-the-art scientific instruments and simulations routinely produce massive datasets requiring intensive processing to disclose key features of the artifact or model under study. Scientists commonly call these data-processing pipelines, which are structured according to the pipe and-filter architecture pattern.1 Different stages typically communicate using files; each stage is an executable program that performs the processing needed at that point in the pipeline.The MeDICi (Middleware for Data-Intensive Computing) Integration Framework supports constructing complex software pipelines from distributed heterogeneous components and controlling qualities of service to meet performance, reliability and communication requirements.

Through the development of new classes of software, algorithms, and hardware, data-intensive applications provide timely and meaningful analytical results in response to exponentially growing data complexity and associated analysis requirements.

In this paper, having morphed their external personae to become almost mainstream, the future looks bright for real programmers. The author discusses the evolution of the programming profession.

Server-side component technologies such as Enterprise JavaBeans (EJBs), .NET, and CORBA are commonly used in enterprise applications that have requirements for high performance and scalability. When designing such applications, architects must select suitable component technology platform and application architecture to provide the required performance. This is challenging as no methods or tools exist to predict application performance without building a significant prototype version for subsequent benchmarking. In this paper, we present an approach to predict the performance of component-based server-side applications during the design phase of software development. The approach constructs a quantitative performance model for a proposed application. The model requires inputs from an application-independent performance profile of the underlying component technology platform, and a design description of the application. The results from the model allow the architect to make early decisions between alternative application architectures in terms of their performance and scalability. We demonstrate the method using an EJB application and validate predictions from the model by implementing two different application architectures and measuring their performance on two different implementations of the EJB platform.

As part of the Middleware Technology Evaluation (MTE) project, we conducted several experiments to explore the performance implications of two common application architectures supported by J2EE's enterprise JavaBean (EJB) component technology. One architecture promises simpler engineering and maintenance of the resulting component collection. For applications that require high performance and scalability, however, the alternative architecture might offer a better solution. Such knowledge is crucial to software architects, who must make initial design decisions early in a project, before extensive engineering has begun. An examination of two EJB-based architectures reveals differences that can significantly affect the performance and scalability of applications built on them.

COTS middleware speeds e-business application deployment but can be difficult to select. The i-Mate tool provides a proven, structured software engineering process for COTS middleware acquisition. Using i-Mate in six major projects for a wide range of organizations led to highly visible, accountable, and ultimately reliable selections of COTS middleware products in greatly compressed time scales. This significantly reduced the risks associated with inappropriate product selections and made these projects more likely to succeed.

What are the main issues in software engineering for parallel and distributed systems? This question, deceptively simple perhaps, produced a range of views, some convergent, others governed by the needs of specialist domains and the writer's particular expertise. These various views are expressed in a series of articles which also appear on the Web.

The Parse Project has been investigating software development issues covering a range of parallel applications. Parse itself is an object-based design methodology that incorporates design management strategies based on data and function encapsulation, hierarchical decomposition, and staged refinement. Parse represents parallel software designs with a graphical notation called process graphs. After capturing a design's important structural features, this notation systematically derives the design's skeletal dynamic properties. For this, it can use either a behavioral specification language or formal methods such as CSP (Communicating Sequential Processes) or Petri nets. Parse builds on existing parallel software design techniques that are based on dataflow and object-oriented approaches. Parse attempts to capture precise object interactions and synchronization in an abstract, architecture-independent notation. To do so, it combines the strengths of various object-oriented techniques and extends the abilities of real-time design methodologies. The Parse project has tried to promote recognized software engineering techniques and extend them to incorporate high-level abstractions for parallel software design. Using systematic transformation techniques, system designers then can methodically refine their designs into formal notations to promote verification and into programming languages for easy implementation. >