Search

Slides for our presentation on command horizons at WAMTA 2023. Abstract In runtime systems for distributed memory parallel computing which automatically manage dependencies and data transfers, a fundamental trade-off exists between the fidelity of dependency tracking and the overhead incurred by its implementation. Precise tracking of data state allows for effective scheduling, which can leverage opportunities for compute and transfer parallelism. However, it also induces more overhead, and with some data access patterns this overhead can grow with e.g. the number of iterations of an algorithm. We present the concept of command horizons, which allow coalescing of previous fine-grained tracking information while maintaining an easily configurable scheduling window with full information precision. Furthermore, they enable consistent cluster-wide decision points without requiring any inter-node communication, and effectively cap the size of state tracking data structures even in the presence of problematic access patterns. Experimental evaluation on microbenchmarks demonstrates that horizons are effective in keeping the scheduling complexity constant, while their own overhead is negligible -- below $10 \mu s$ per horizon when building a command graph for 512 GPUs. We additionally demonstrate the performance impact of horizons -- as well as their low overhead -- on a real-world application. &nbsp

This paper describes two-fold approach towards utilizing Triple Modular Redundancy (TMR) in Wireless Adhoc Network (AdocNet). A distributed checkpointing and recovery protocol is proposed. The protocol eliminates useless checkpoints and helps in selecting only dependent processes in the concerned checkpointing interval, to recover. A process starts recovery from its last checkpoint only if it finds that it is dependent (directly or indirectly) on the faulty process. The recovery protocol also prevents the occurrence of missing or orphan messages. In AdocNet, a set of three nodes (connected to each other) is considered to form a TMR set, being designated as main, primary and secondary. A main node in one set may serve as primary or secondary in another. Computation is not triplicated, but checkpoint by main is duplicated in its primary so that primary can continue if main fails. Checkpoint by primary is then duplicated in secondary if primary fails too.

In today’s Configuration Management system, it becomes extremely necessary to be able to support multiple projects, branches and versions of the source code. As an institution’s build system becomes more complex, management of the source code becomes very time-consuming and complex if the build system is not mature enough. This puts pressure on several entities, especially the build and release management teams, as well as the quality assurance and product testing teams. This thesis introduces a technique to obtain dependency information from build systems, mapping executables to source files and analyzing their dependencies. Dependency information can be used to reduce the testing effort, automate the entire sustaining production process (service packs and patches) to customers, prevent test escapes, highlight the relationships between various components of a system as well as provide product and system model views of the internal representation of a product’s architecture. This thesis proposes a method to gather dependency information of a system by extracting certain metadata from executables. After some in-depth research, we identified that source files can be initially injected with certain information that remains embedded in it even after a system has been built into executables. This information can also be extracted and compiled to create dependency information that maps executables, static and dynamic shared libraries as well as software and enterprise archives to the source files that were used to build them

Current cluster architectures provide the ideal environment to run federations of main-memory database systems (FMMDBs). In FMMDBs, data resides in the main memory of the federation servers. FMMDBs significantly improve performance by avoiding I/O during the execution of read operations. To maximize the performance of update transactions as well, some applications recur to deferred disk writes. This means that update transactions commit before their modifications are written on stable storage and durability must be ensured outside the database. While deferred disk writes in centralized MMDBs relax the durability property of transactions only, in FMMDBs transaction atomicity may be also violated in case of failures. We address this issue from the perspective of log-based rollback-recovery in distributed systems and provide an efficient solution to the problem. Besides presenting a mechanism to ensure atomicity in FMMDBs, the paper bridges the gap between rollback-recovery in message-passing distributed systems and distributed transaction processing, and shows how results developed in the first context can be exploited in the second.