Search

The Data Activated Liu1Graph Engine – DALiuGE2– is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE’s scalability to very large numbers of tasks on two supercomputing facilities.

The Square Kilometre Array (SKA) project is an international cooperation project to build the largest radio telescope worldwide. Data processing is one of the biggest challenges of building the SKA telescope. As a distributed execution framework, the Data Activated Liu Graph Engine (DALiuGE) was proposed to be one of the candidates for addressing the massive data of the SKA. DALiuGE has many distinctive features, but its actual ability to handle scientific data is still not clear. In this paper, we perform an objective evaluation of the usability of DALiuGE for the execution performance, developer workload, and implementation difficulty of porting the SAGECal to DALiuGE. The evaluation results show that the DALiuGE enables fast integration of astronomical software, but there are significant differences in the efficiency of different parallel granularities. Even with the deep optimization of the program, there is still a gap between the current DALiuGE and the traditional MPI in execution performance. Therefore, we come to a preliminary conclusion that the DALiuGE has no performance advantage in batch processing of massive data. At the same time, it may be more suitable for application scenarios with more customized computational tasks, such as SKA science regional centers.

We measured the growth, nutrition, and N assimilation of arbuscular-mycorrhizal and non-mycorrhizal lettuce (Lactuca sativa L.) as affected by forms of N and drought. Moisture was maintained at 80% water-holding capacity, and N was applied as NOinf3sup-, NHinf4sup+, or NOinf3sup-/NHinf4sup+(3:1, 1:1, or 1:3). The growth of Glomus fasciculatum-colonized plants was comparable to that of uncolonized P-supplemented plants when N was provided as NHinf4sup+or combined NOinf3sup-/NHinf4sup+. When N was supplied solely as NOinf3sup-, G. fasciculatum-colonized plants produced a higher yield than P-fertilized plants, suggesting that the uptake and/or assimilation of NOinf3sup-was particularly affected by mycorrhizal status in this water-limited situation. Nutrient availability, except Ca, was less limited for mycorrhizal plants than for P-fertilized plants. P fertilization increased the growth, glutamine synthetase activity, and protein content of lettuce to the same extent that G. fasciculatum colonization did when N was applied as NHinf4sup+. With NOinf3sup--fertilization, G. fasciculatum-colonized plants showed increased growth, nitrate reductase activity, and protein content compared to P-fertilizer treatment. Plants colonized by G. mosseae showed increased photosynthetic activity and proline acumulation, and these mechanisms may be important in adaptation by the plant to drought conditions. The present results confirmed that under drought conditions, the uptake or metabolism of N forms is particularly affected in mycorrhizal fungi-colonized plants, depending on the mycorrhizal endophyte and the N source added. Thus the significance of arbuscular-mycorrhizal fungus selection for plant growth in drought conditions is a consideration for management strategy.

The ability of the external mycelium in arbuscular mycorrhiza for N uptake and transport was studied. The contribution of the fungal symbiont to N acquisition by plants was studied mainly under waterstressed conditions using 15N. Lettuce (Lactuca sativa L) was the host for two isolates of the arbuscular mycorrhizal fungi Glomus mosseae and G. fasciculatum. The experimental pots had two soil compartments separated by a fine mesh screen (60 µm). The root system was restricted to one of these compartments, while the fungal mycelium was able to cross the screen and colonize the soil in the hyphal compartment. A trace amount of 15NH4+was applied to the hyphal compartment 1 week before harvest. Under water-stressed conditions both endophytes increased the 15N enrichment of plant tissues; this was negligible in nonmycorrhizal control plants. This indicates a direct effect of arbuscular mycorrhizal fungi on N acquisition in relatively dry soils. G. mosseae had more effect on N uptake and G. fasciculatum on P uptake under the water-limited conditions tested, but both fungi improved plant biomass production relative to nonmycorrhizal plants to a similar extent.