Search

Researchers in computational science are regularly posed with the challenge of managing and analyzing large, heterogeneous, and highly dynamic data spaces. We present signac, an open-source Python framework that enables researchers to efficiently operate on primarily file-based data spaces while keeping track of all relevant metadata. The signac framework provides all components required to create a well-defined, collectively accessible data space and to implement reproducible workflows. The software is designed to be highly modular, decoupling its data and workflow management components so as to minimize the effort required for integration into existing workflows. The serverless data management and lightweight workflow model ensure that workflows are just as easily executed on laptops as in high-performance computing environments. Using signac not only increases research efficiency, it also improves reproducibility and lowers barriers for data sharing by transparently enabling the robust tracking, selection, and searching of data by its metadata.

The HammerCloud system is used by CERN IT to monitor the status of the Worldwide LHC Computing Grid (WLCG). HammerCloud automatically submits jobs to WLCG computing resources, closely replicating the workflow of Grid users (e.g. physicists analyzing data). This allows computation nodes and storage resources to be monitored, software to be tested (somewhat like continuous integration), and new sites to be stress tested with a heavy job load before commissioning. The HammerCloud system has been in use for ATLAS and CMS experiments for about five years. This summer's work involved porting the HammerCloud suite of tools to the LHCb experiment. The HammerCloud software runs functional tests and provides data visualizations. HammerCloud's LHCb variant is written in Python, using the Django web framework and Ganga/DIRAC for job management.