Search

Industrial applications tend to rely increasingly on large datasets for regular operations. In order to facilitate that need, we unite the increasingly available hardware resources with fundamental problems found in classical algorithms. We show solutions to the following problems: power flow and island detection in power networks, and the more general graph sparsification. At their core lie respectively algorithms for solving systems of linear equations, graph connectivity and matrix multiplication, and spectral sparsification of graphs, which are applicable on their own to a far greater spectrum of problems. The novelty of our approach lies in developing the first open source and distributed solutions, capable of handling large datasets. Such solutions constitute a toolkit, which, aside from the initial purpose, can be used for the development of unrelated applications and for educational purposes in the study of distributed algorithms.

The process of handing over of expert knowledge to younger colleagues is an error prone, difficult and time consuming process. This handover is most often done either trough conversation, a document or in better cases trough some video material. All of the aforementioned methods are slow and can mislead the less experienced colleagues. This paper gives a proposal on how to gather and transfer expert knowledge with the aid of artificial intelligence and presents a few examples where it would be applicable.

Coping with ever increasing complexity is one of the main challenges of software engineering. Applying proven practices through established software processes is a viable way to introduce consistency and tame complexity. The SEMAT is a novel meta-process aiming to encompass commonalities across a disparate set of different software processes. The goal is to create a framework, giving proper jumpstart to any proprietary customisation. This is crucial, as various types of software demand diverse software processes (there is no one-size-fits-all method). On the other hand, all processes strive for a disciplined application of valid development and maintenance routines. This paper analyses a set of methodologies for several software development lines to recognise common and specific development practices. A special emphasise is put on the test-driven development (TDD), showcasing its significant contribution in improving the total quality of the end product. The SEMAT Kernel is extended with additional set of alphas, which are elaborated in detail including containing states, transitions and dependencies. The TDD practice activities were analysed for each introduced alpha as state transition triggers. Consequently, the high-level development plan is expanded with newly defined alphas, providing a powerful tool for performing all sorts of system analysis.

Let me start by making an analogy between software engineering and data science. Software engineering may be summarized as the application of engineering principles and methods to the development of software. The aim is to produce a dependable software product. In a similar vein, data science may be described as the application of scientific principles and methods in working with data. The goal is to synthesize reliable and actionable insights from data (sometimes referred as data product). To continue with our analogy, the systems/software development life cycle (SDLC) prescribes the major phases of a software development process: project initiation, requirements engineering, design, construction, testing, deployment, and maintenance. The data science process also encompasses multiple phases: project initiation, data acquisition, data preparation, data analysis, reporting, and execution of actions (another “phase” is data exploration, which is more of an all-embracing activity than a stand-alone phase). As in software development, these phases are quite interwoven, and the process is inherently iterative and incremental. An overarching activity that is indispensable in both software engineering and data science (and any other iterative and incremental endeavor) is retrospection, which involves reviewing a project or process to determine what was successful and what could be improved. Another similarity to software engineering is that data science also relies on a multidimensional team or team of teams. A typical project requires domain experts, software engineers specializing in various technologies, and mathematicians (a single person may take different roles at various times). Yet another common denominator with software engineering is a penchant for automation (via programmability of most activities) to increase productivity, reproducibility, and quality. The aim of this chapter is to explain the key concepts regarding data science and put them into proper context.

As a data scientist, you must solve essential problems in an economical fashion, taking into account all constraints and functional requirements. The biggest mistake is to get emotionally attached to some particular technique and try to find “suitable” problems to which to apply it; this is pseudo-science, at best. A similar issue arises when you attempt to apply some convoluted approach where a simpler and more elegant method exists. I’ve seen these two recurring themes myriad times. For example, logistic regression (an intuitive technique) often can outperform kernelized support vector machines in classification tasks, and a bit of cleverness in feature engineering may completely obviate the need for an opaque neural network–oriented approach. Start simple and keep it simple is perhaps the best advice for any data scientist. In order to achieve simplicity, you must invest an enormous amount of energy and time, but the end result will be more than rewarding (especially when including long-term benefits). Something that goes along with simplicity is pragmatism—good enough is usually better than perfect in most scenarios. This chapter will exemplify an attitude toward simplicity and pragmatism as well as acquaint you with the notion of complexity and heuristic. It contains many small use cases, each highlighting a particular topic pertaining to succinctness and effectiveness of a data product.

Data science and scientific computing are human-centered, collaborative endeavors that gather teams of experts covering multiple domains. You will rarely perform any serious data analysis task alone. Therefore, efficient intra-team communication that ensures proper information exchange within a team is required. There is also a need to convey all details of an analysis to relevant external parties. Your team is part of a larger scientific community, so others must be able to easily validate and verify your team’s findings. Reproducibility of an analysis is as important as the result itself. Achieving this requirement—to effectively deliver data, programs, and associated narrative as an interactive bundle—is not a trivial task. You cannot assume that everybody who wants to peek into your analysis is an experienced software engineer. On the other hand, all stakeholders aspire to make decisions based on available data. Fortunately, there is a powerful open-source solution for reconciling differences in individuals’ skill sets. This chapter introduces the project Jupyter (see https://jupyter.org ), the most popular ecosystem for documenting and sharing data science work.

Ad-hoc approaches to building Internet of Things (IoT) solutions in the past had mostly failed. The sheer complexity of managing a mesh of distributed interconnected devices was unbearable. The appearance of IoT platforms has enabled the proliferation of successful products in this area. A full-scale, comprehensive IoT platform embraces the following three major capabilities: application enablement, data storage, and connectivity management. This paper presents an overview of technologies and techniques to realize such an IoT platform. The focus is on the platform that unifies edge and cloud computing. The paper refers to Mainflux as such reference example. Mainflux serves as a software infrastructure, that is built around the microservices architectural style, for development of IoT solutions, and deployment of intelligent products. It is conceived as an opensource project, and is already used in production settings. Therefore, the paper describes a real product, whose chosen set of methods, technologies and techniques are feasible and effective in crafting IoT systems. The Mainflux project contains an extensive set of automated tests, that could serve as an additional documentation regarding its usage.

Graphs are the cornerstone of many algorithms pertaining to various network analyses. When the problem's dimensionality is relatively small, expressed in the number of vertices and edges of a graph, then most methods perform adequately well. As the problem size increases, more compute power is required. Distributed computing is a one viable option to address this issue, but it cannot scale indefinitely. At one point, it is necessary to turn to heuristic approaches. Spectral graph theory is an example of such approximate scheme. In this paper, we combine spectral analysis with distributed computing using Apache Spark. The paper is accompanied with a publicly available proof of concept implementation. The system was extensively performance tested, and the results show a superb fit of Apache Spark to the purpose of spectral graph analysis. Furthermore, the resulting code is straightforward thankfully to Spark's intuitive distributed programming model, and well-designed APIs.

This paper tackles an important and challenging problem of protecting players from irresponsible gambling behavior. Such prevention is a crucial and mandatory obligation for major gambling providers. The paper presents a novel machine learning driven solution for implementing the responsible gaming facility. The engine leverages two powerful machine learning algorithms: random forest and gradient boosting. The tests were actualized by reusing a publicly available dataset provided by Transparency Project. The final results confirm that the proposed implementation of the framework passes the criteria as a proof of concept solution.

The intense push toward the Internet of Everything (IoE) paradigm, and the desire to raise the level of efficiency of socio-technical systems engineering, entails reconsidering a traditional SCADA system. Any sort of intelligence in system behavior should start with high quality data, and this is where SCADA plays the central role. Data is necessarily coupled with facilities to process and store them, and SCADA's capabilities must be in par with such elevated expectations pertaining to overall system behavior. The goal of this paper is to propose a modern and novel data handling approach in a SCADA system. The aim is to boost data storage and retrieval performance to seek out extra opportunities for finding patterns and clues in data. The overarching goal is to build the next generation of SCADA based solutions, that would deliver the promises embraced by IoE. Our proposal revolves around the usage of distributed hash tables to disseminate data, and switch from a classical relational database model toward the one utilized by many NoSQL databases. The paper contains both the description of an architecture as well as implementation considerations related to a new SCADA system. The paper concludes with a summary of further research directions that needs to be addressed for a production ready variant.

We propose an architecture that aims to improve the execution times of compute intensive calculations that are part of prominent power system analysis functions. The approach uses the distributed framework Apache Spark to target the common problems in linear algebra, like the solving of a matrix equation, that increase the complexity of said functions. Our analysis focuses on the load flow study as a common and important procedure that could benefit from distributed computing.

The architecture plays a central role in software engineering. This chapter will demonstrate through a case study how an architecture enables evolution. The aim is to demonstrate the following topics: Architectural Pattern, Software Reuse, Software Framework, Inversion of Control, Dependency Injection, and Test-Driven Development.

Quality is usually invisible (from the viewpoint of customers and management), and many problems are only noticeable by software engineers. This chapter presents a long software development/evolution story to highlight many problems that will surface when quality is not treated seriously. It showcases how ignoring maintainability instigates an exponential growth in expenses.

During development and maintenance, software engineers spend most of their time reasoning about the system. To speed up this activity, we need to employ a proper architecture and a straightforward implementation. This chapter shows how maintainability as a quality attribute directly supports safety and demonstrates the following concepts: Single Level of Abstraction, Layered Design, Cyclomatic Complexity, Safety Argument, and Programming in Intentions.

Reverse engineering is about uncovering the secrets behind the product so that you may change it according to your needs. Reengineering is the process of amending the product to some new form. This chapter unifies reengineering and reverse engineering maintenance techniques into a case study and presents a flow that can be applied in practice. The aim is to illuminate how to properly proceed in extensive restructuring work.

Context awareness is like defensive programming in a sense that it creates a powerful feedback loop and control mechanism to reinforce or reject an action or idea. This chapter explains the interplay between context and knowledge (partly embodied as a set of rules and principles), as well as give examples of how things can turn upside down when actions are thoughtlessly executed. The aim of this chapter is to demonstrate the following concepts: context, rules, and principles.

Coupling as a notion has many sides (common, data, control, environmental, etc.); thus, dozens of metrics are associated with this concept. From the perspective of automated deployment/delivery, we are most interested in how to differentiate between logical and physical coupling. This chapter revisits what the preconditions are to attain physically loosely coupled solutions and what the benefits are.

DevOps are an abyss-crossing movement, like Test-Driven Development regarding testability, where deployability and observability are treated as architecturally significant quality attributes. This entails connecting construction with deployment and operational aspects of the produced system. This chapter introduces some key technologies and tools to scale the deployment of a system that is developed using the DevOps paradigm.

In production, we must have visibility of what is going on within our system. Part of the DevOps culture is to share the burden of providing such supervising capability by both development and operations teams. In parallel to shipping metrics and log messages to a central location, we must also have facilities to trigger alarms if something goes awry. This chapter primarily illustrates how to properly collect/send metrics in an application and how to structure/send logs toward the appropriate infrastructure node. The aim is to showcase techniques of squeezing out the maximum insight with minimal effort (expressed in data collection, transmission, and processing time as well as cost).

It is important to understand the role of HTTP and what it brings into the picture. The HTTP’s uniform interface relieves clients from learning fundamentally new ways of handling resources. Everything happens via URLs (these are identifiers and handles of resources at the same time) and HTTP regular methods (see the sidebar “Identifier vs. Handle” later in this chapter for more details about them).

There are two broad categories of API quality attributes: usability and power. A similar relation holds with the synchronous and asynchronous programming models. The former is surely easier to understand, but the latter could be critical to achieving the desired performance level.

For a REST API we can produce an executable version of the documentation and allow users to interact with a system through its own documentation. In this sense, the documentation wouldn’t be a passive dust collector artifact (if stored in a printed form), but a handy UI to communicate with software.

Reuse is reified in various forms across many dimensions. We can reuse code (by importing a class or module into our code base), code and design (by using a full-fledged framework), applications or services (e.g., by creating mashups combining different services into a unified portal), and so on. Software processes are also amenable for reuse, as are requirements and conceptual domain models. Moreover, we might want to reuse existent stuff, or we could also opt to build artifacts for reuse. The topic is really broad. Therefore, we limit ourselves here purely to aspects of reuse pertaining to OO systems, specifically focusing on APIs. In other words, we analyze how APIs can help or hinder efficient reuse.

XML and JSON are language-independent and generic message formats, but they lag behind binary formats in efficiency. When you need to craft a system capable of processing high volumes of data, switching to a binary representation is a viable approach.

In an ideal world, we wouldn’t need explicit version numbers (or any version identifiers for that matter) in our REST APIs. If you think that such a world doesn’t exist, well, it is all around us. It is the Web! Our web browsers can handle all sorts of web sites, usually without significant problems (we assume that our modern browser understands different dialects of HTML, and the site is not specifically targeted for another browser). Part of the magic stems from hypermedia controls embedded in an HTML content. Take a look at the source HTML (try to load some complex web page into your browser, and ask it to show the page source), and you will encounter all sorts of links to other web pages, various forms for uploading data, as well as forms for performing custom searches, and so on.

TDD is usually associated with the phrase “Write tests first, and then implement.” However, this is a rather distorted and reduced view of TDD.

Smoke tests are good to quickly assess whether the system functions or not (they are usually performed before deployment as a swift sanity check). The main problem is that running tests manually isn’t a scalable tactic (you cannot perform fast regression tests after each change by relying on manual labor). This is the reason we need to seek a better alternative.

A message-oriented middleware (MOM) with an associated system built on top of it is targeting scenarios where the action–reaction cycle is desirable to be separated both in space and time. This allows clients and servers to be detached from each other, and follow their intrinsic life cycle. Apache Kafka is a high performant and fault-tolerant MOM.

If a message structure isn’t aligned to support distributed development and deployment, then we cannot qualify that system as supportable. Such a system cannot expand beyond some level. This chapter is about important techniques and principles for crafting evolvable message structures.