Search

Containers became the de-facto standard to package and distribute modern applications and their dependencies. The HEP community demonstrates an increasing interest in such technology, with scientists encapsulating their analysis workflow and code inside a container image. The analysis is first validated on a small dataset and minimal hardware resources to then run at scale on the massive computing capacity provided by the grid. The typical approach for distributing containers consists of pulling their image from a remote registry and extracting it on the node where the container runtime (e.g., Docker, Singularity) runs. This approach, however, does not easily scale to large images and thousands of nodes. CVMFS has long been used for the efficient distribution of software directory trees at a global scale. In order to extend its optimized caching and network utilization to the distribution of containers, CVMFS recently implemented a dedicated container image ingestion service together with container runtime integrations. CVMFS ingestion is based on per-file deduplication, instead of the per-layer deduplication adopted by traditional container registries. On the client-side, CVMFS implements on-demand fetching of the chunks required for the execution of the container instead of the whole image.

The CernVM File System provides the software and container distribution backbone for most High Energy and Nuclear Physics experiments. It is implemented as a file system in user-space (Fuse) module, which permits its execution without any elevated privileges. Yet, mounting the file system in the first place is handled by a privileged suid helper program that is installed by the Fuse package on most systems. The privileged nature of the mount system call is a serious hindrance to running CernVM-FS on opportunistic resource and supercomputers. Fortunately, recent developments in the Linux kernel and in the Fuse user-space libraries enabled fully unprivileged mounting for Fuse file systems (as of RHEL 8), or at least outsourcing the privileged mount system call to a custom, external process. This opens the door to several, very appealing new ways to use CernVM-FS, such as a generally usable "super pilot" consisting of the pilot code bundled with Singularity and CernVM-FS, or the on-demand instantiation of unprivileged, ephemeral containers to publish new CernVM-FS content from anywhere. In this contribution, we discuss the integration of these new Linux features with CernVM-FS and show some of its most promising, new applications. The CernVM File System provides the software and container distribution backbone for most High Energy and Nuclear Physics experiments. It is implemented as a file system in user-space (Fuse) module, which permits its execution without any elevated privileges. Yet, mounting the file system in the first place is handled by a privileged suid helper program that is installed by the Fuse package on most systems. The privileged nature of the mount system call is a serious hindrance to running CernVM-FS on opportunistic resource and supercomputers. Fortunately, recent developments in the Linux kernel and in the Fuse user-space libraries enabled fully unprivileged mounting for Fuse file systems (as of RHEL 8), or at least outsourcing the privileged mount system call to a custom, external process. This opens the door to several, very appealing new ways to use CernVM-FS, such as a generally usable “super pilot” consisting of the pilot code bundled with Singularity and CernVM-FS, or the on-demand instantiation of unprivileged, ephemeral containers to publish new CernVM-FS content from anywhere. In this contribution, we discuss the integration of these new Linux features with CernVM-FS and show some of its most promising, new applications.

The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution and—to some extent—a data distribution service. It gives POSIX access to more than a billion binary files of experiment application software stacks and operating system containers to end user devices, grids, clouds, and supercomputers. Increasingly, CernVM-FSalso provides access to certain classes of data, such as detector conditions data, genomics reference sets, or gravitational wave detector experiment data. For most of the high- energy physics experiments, an underlying HTTP content distribution infrastructure is jointly provided by universities and research institutes around the world. In this contribution, we will present recent developments and future plans. For future developments, we put a focus on evolving the content distribution infrastructure and at lowering the barrier for publishing into CernVM-FS. Through so-called serverless computing, we envision cloud hosted CernVM-FS repositories without the need to operate dedicated servers or virtual machines. An S3 compatible service in conjunction with a content delivery network takes on data provisioning, replication, and caching. A chainof time-limited and resource-limited functions (so called “lambda function” or “function-as- a-service”) operate on the repository and stage the updates. As a result, any CernVM-FS client should be able to turn intoawriter, possession of suitable keys provided. For repository owners, we aim at providing cost transparency and seamless scalability from very small to very large CernVM-FS installations.

Linux containers have gained widespread use in high energy physics, be it for services using container engines such as containerd/kubernetes, for production jobs using container engines such as Singularity or Shifter, or for development workflows using Docker as a local container engine. Thus the efficient distribution of the container images, whose size usually ranges from a few hundred megabytes to a few tens of gigabytes, is becoming a pressing concern. Because container images show similar characteristics than scientific application stacks, unpacking the images in CernVM-FS can remedy the distribution issues provided that the container engine at hand is able to use such unpacked images from CernVM-FS. In this contribution, we willl report on recent advances in the integration of Singularity, Docker, and containerd with CernVM-FS. We show improvements in the publishing of container images from a Docker registry that rely on new means of directly ingesting image tarballs. Well also show a repository file system structure for storing container images that are optimized for storing both container engines using flat root file systems (Singularity) as well as container engines using layers (containerd, Docker). To evaluate the benefits of our approach, we show concrete use cases and figures for production and development images from LHC experiments stored in the recently created unpacked.cern.ch repository.