120 results on '"Blomer, Jakob"'
Search Results
2. A caching mechanism to exploit object store speed in High Energy Physics analysis
- Author
-
Padulano, Vincenzo Eduardo, Tejedor Saavedra, Enric, Alonso-Jordá, Pedro, López Gómez, Javier, and Blomer, Jakob
- Published
- 2023
- Full Text
- View/download PDF
3. HL-LHC Analysis With ROOT
- Author
-
Naumann, Axel, Canal, Philippe, Tejedor, Enric, Guiraud, Enrico, Moneta, Lorenzo, Bellenot, Bertrand, Couet, Olivier, Tadel, Alja Mrak, Tadel, Matevz, Linev, Sergey, Gomez, Javier Lopez, Rembser, Jonas, Padulano, Vincenzo Eduardo, Blomer, Jakob, Hahnfeld, Jonas, Gruber, Bernhard Manfred, and Vassilev, Vassil
- Subjects
High Energy Physics - Experiment ,Physics - Data Analysis, Statistics and Probability - Abstract
ROOT is high energy physics' software for storing and mining data in a statistically sound way, to publish results with scientific graphics. It is evolving since 25 years, now providing the storage format for more than one exabyte of data; virtually all high energy physics experiments use ROOT. With another significant increase in the amount of data to be handled scheduled to arrive in 2027, ROOT is preparing for a massive upgrade of its core ingredients. As part of a review of crucial software for high energy physics, the ROOT team has documented its R&D plans for the coming years.
- Published
- 2022
4. ROOT for the HL-LHC: data format
- Author
-
Naumann, Axel, Canal, Philippe, Tejedor, Enric, Guiraud, Enrico, Moneta, Lorenzo, Bellenot, Bertrand, Couet, Olivier, Tadel, Alja Mrak, Tadel, Matevz, Linev, Sergey, Gomez, Javier Lopez, Rembser, Jonas, Padulano, Vincenzo Eduardo, Blomer, Jakob, Hahnfeld, Jonas, Gruber, Bernhard Manfred, and Vassilev, Vassil
- Subjects
High Energy Physics - Experiment ,Physics - Data Analysis, Statistics and Probability - Abstract
This document discusses the state, roadmap, and risks of the foundational components of ROOT with respect to the experiments at the HL-LHC (Run 4 and beyond). As foundational components, the document considers in particular the ROOT input/output (I/O) subsystem. The current HEP I/O is based on the TFile container file format and the TTree binary event data format. The work going into the new RNTuple event data format aims at superseding TTree, to make RNTuple the production ROOT event data I/O that meets the requirements of Run 4 and beyond.
- Published
- 2022
5. RNTuple performance: Status and Outlook
- Author
-
Lopez-Gomez, Javier and Blomer, Jakob
- Subjects
Physics - Data Analysis, Statistics and Probability ,Computer Science - Databases ,Physics - Accelerator Physics ,Physics - Computational Physics ,H.3.0 ,E.2 ,J.2 - Abstract
Upcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes critical. To this end, the ROOT RNTuple I/O subsystem has been designed to address performance bottlenecks and shortcomings of ROOT's current state of the art TTree I/O subsystem. RNTuple provides a backwards-incompatible redesign of the TTree binary format and access API that evolves the ROOT event data I/O for the challenges of the upcoming decades. It focuses on a compact data format, on performance engineering for modern storage hardware, for instance through making parallel and asynchronous I/O calls by default, and on robust interfaces that are easy to use correctly. In this contribution, we evaluate the RNTuple performance for typical HEP analysis tasks. We compare the throughput delivered by RNTuple to popular I/O libraries outside HEP, such as HDF5 and Apache Parquet. We demonstrate the advantages of RNTuple for HEP analysis workflows and provide an outlook on the road to its use in production., Comment: 5 pages, 5 figures; submitted to proceedings of 20th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2021)
- Published
- 2022
- Full Text
- View/download PDF
6. Exploring Object Stores for High-Energy Physics Data Storage
- Author
-
López-Gómez, Javier and Blomer, Jakob
- Subjects
Computer Science - Databases ,High Energy Physics - Experiment - Abstract
Over the last two decades, ROOT TTree has been used for storing over one exabyte of High-Energy Physics (HEP) events. The TTree columnar on-disk layout has been proved to be ideal for analyses of HEP data that typically require access to many events, but only a subset of the information stored for each of them. Future colliders, and particularly HL-LHC, will bring an increase of at least one order of magnitude in the volume of generated data. Therefore, the use of modern storage hardware, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes more important. However, TTree was not designed to optimally exploit modern hardware and may become a bottleneck for data retrieval. The ROOT RNTuple I/O system aims at overcoming TTree's limitations and at providing improved efficiency for modern storage systems. In this paper, we extend RNTuple with a backend that uses Intel DAOS as the underlying storage, demonstrating that the RNTuple architecture can accommodate high-performance object stores. From the user perspective, data can be accessed with minimal changes to the code, that is by replacing a filesystem path by a DAOS URI. Our performance evaluation shows that the new backend can be used for realistic analyses, while outperforming the compatibility solution provided by the DAOS project., Comment: Accepted for Proceedings of 25th International Conference on Computing in High-Energy and Nuclear Physics
- Published
- 2021
- Full Text
- View/download PDF
7. LLAMA: The Low-Level Abstraction For Memory Access
- Author
-
Gruber, Bernhard Manfred, Amadio, Guilherme, Blomer, Jakob, Matthes, Alexander, Widera, René, and Bussmann, Michael
- Subjects
Computer Science - Performance - Abstract
The performance gap between CPU and memory widens continuously. Choosing the best memory layout for each hardware architecture is increasingly important as more and more programs become memory bound. For portable codes that run across heterogeneous hardware architectures, the choice of the memory layout for data structures is ideally decoupled from the rest of a program. This can be accomplished via a zero-runtime-overhead abstraction layer, underneath which memory layouts can be freely exchanged. We present the Low-Level Abstraction of Memory Access (LLAMA), a C++ library that provides such a data structure abstraction layer with example implementations for multidimensional arrays of nested, structured data. LLAMA provides fully C++ compliant methods for defining and switching custom memory layouts for user-defined data types. The library is extensible with third-party allocators. Providing two close-to-life examples, we show that the LLAMA-generated AoS (Array of Structs) and SoA (Struct of Arrays) layouts produce identical code with the same performance characteristics as manually written data structures. Integrations into the SPEC CPU\textsuperscript{\textregistered} lbm benchmark and the particle-in-cell simulation PIConGPU demonstrate LLAMA's abilities in real-world applications. LLAMA's layout-aware copy routines can significantly speed up transfer and reshuffling of data between layouts compared with naive element-wise copying. LLAMA provides a novel tool for the development of high-performance C++ applications in a heterogeneous environment., Comment: 39 pages, 10 figures, 11 listings
- Published
- 2021
- Full Text
- View/download PDF
8. Software Challenges For HL-LHC Data Analysis
- Author
-
ROOT Team, Brann, Kim Albertsson, Amadio, Guilherme, An, Sitong, Bellenot, Bertrand, Blomer, Jakob, Canal, Philippe, Couet, Olivier, Galli, Massimiliano, Guiraud, Enrico, Hageboeck, Stephan, Linev, Sergey, Vila, Pere Mato, Moneta, Lorenzo, Naumann, Axel, Tadel, Alja Mrak, Padulano, Vincenzo Eduardo, Rademakers, Fons, Shadura, Oksana, Tadel, Matevz, Saavedra, Enric Tejedor, Pla, Xavier Valls, Vassilev, Vassil, and Wunsch, Stefan
- Subjects
Physics - Data Analysis, Statistics and Probability - Abstract
The high energy physics community is discussing where investment is needed to prepare software for the HL-LHC and its unprecedented challenges. The ROOT project is one of the central software players in high energy physics since decades. From its experience and expectations, the ROOT team has distilled a comprehensive set of areas that should see research and development in the context of data analysis software, for making best use of HL-LHC's physics potential. This work shows what these areas could be, why the ROOT team believes investing in them is needed, which gains are expected, and where related work is ongoing. It can serve as an indication for future research proposals and cooperations.
- Published
- 2020
9. Evolution of the ROOT Tree I/O
- Author
-
Blomer, Jakob, Canal, Philippe, Naumann, Axel, and Piparo, Danilo
- Subjects
Computer Science - Databases ,High Energy Physics - Experiment ,E.2 ,J.2 - Abstract
The ROOT TTree data format encodes hundreds of petabytes of High Energy and Nuclear Physics events. Its columnar layout drives rapid analyses, as only those parts ("branches") that are really used in a given analysis need to be read from storage. Its unique feature is the seamless C++ integration, which allows users to directly store their event classes without explicitly defining data schemas. In this contribution, we present the status and plans of the future ROOT 7 event I/O. Along with the ROOT 7 interface modernization, we aim for robust, where possible compile-time safe C++ interfaces to read and write event data. On the performance side, we show first benchmarks using ROOT's new experimental I/O subsystem that combines the best of TTrees with recent advances in columnar data formats. A core ingredient is a strong separation of the high-level logical data layout (C++ classes) from the low-level physical data layout (storage backed nested vectors of simple types). We show how the new, optimized physical data layout speeds up serialization and deserialization and facilitates parallel, vectorized and bulk operations. This lets ROOT I/O run optimally on the upcoming ultra-fast NVRAM storage devices, as well as file-less storage systems such as object stores., Comment: 9 pages, 8 figures, 1 table, submitted to 24th Computing in High Energy and Nuclear Physics (CHEP'19)
- Published
- 2020
- Full Text
- View/download PDF
10. CernVM-FS at Extreme Scales
- Author
-
Promberger Laura, Blomer Jakob, Völkl Valentin, and Harvey Matt
- Subjects
Physics ,QC1-999 - Abstract
The CernVM File System (CVMFS) provides the software distribution backbone for High Energy and Nuclear Physics experiments and many other scientific communities in the form of a globally available shared software area. It has been designed for the software distribution problem of experiment software for LHC Runs 1 and 2. For LHC Run 3 and even more so for HL-LHC (Runs 4-6), the complexity of the experiment software stacks and their build pipelines is substantially larger. For instance, software is being distributed for several CPU architectures, often in the form of containers which includes base and operating system libraries, the number of external packages such as machine learning libraries has multiplied, and there is a shift from C++ to more Python-heavy software stacks that results in more and smaller files needing to be distributed. For CVMFS, the new software landscape means an order of magnitude increase of scale in several key metrics. This contribution reports on the performance and reliability engineering on the file system client to sustain current and expected future software access load. Concretely, the impact of the newly designed file system cache management is shown, including significant performance improvements for HEP-representative benchmark workloads, and an up to 25% performance increase in software built-time when the build tools reside on CVMFS. Operational improvements presented include better network failure handling, error reporting, and integration with container runtimes. And a pilot study using zstd as compression algorithm shows that it could bring significant improvements for remote data access times.
- Published
- 2024
- Full Text
- View/download PDF
11. Integration of RNTuple in ATLAS Athena
- Author
-
de Geus Florine, López-Gómez Javier, Blomer Jakob, Nowak Marcin, and van Gemmeren Peter
- Subjects
Physics ,QC1-999 - Abstract
After using ROOT’s TTree I/O subsystem for over two decades and storing more than an exabyte of compressed High Energy Physics (HEP) data, advances in technology have motivated a complete redesign, RNTuple, which breaks backward-compatibility to take better advantage of these storage options. The RNTuple I/O subsystem has been designed to address performance bottlenecks and other shortcomings of TTree. Specifically, RNTuple comes with an updated, more compact binary data format that can be stored both in ROOT files and natively in object stores. It is designed for modern storage hardware (e.g. high-throughput low-latency NVMe SSDs), and provides robust and easy to use interfaces. The binary format of RNTuple is scheduled to become production grade in 2024, and recently has become mature enough to start exploring the integration into software used by HEP experiments. In this contribution, we discuss the developments to support the features as required by the ATLAS analysis Event Data Model (EDM) in RNTuple, which will enable its integration into the Athena software framework. With these developments in place, we evaluate the performance of the current most recent versions of RNTuple-based ATLAS data sets and compare this to that of TTree.
- Published
- 2024
- Full Text
- View/download PDF
12. ROOT’s RNTuple I/O Subsystem: The Path to Production
- Author
-
Blomer Jakob, Canal Philippe, de Geus Florine, Hahnfeld Jonas, Naumann Axel, Lopez-Gomez Javier, Lazzari Miotto Giovanna, and Padulano Vincenzo Eduardo
- Subjects
Physics ,QC1-999 - Abstract
The RNTuple I/O subsystem is ROOT’s future event data file format and access API. It is driven by the expected data volume increase at upcoming HEP experiments, e.g. at the HL-LHC, and recent opportunities in the storage hardware and software landscape such as NVMe drives and distributed object stores. RNTuple is a redesign of the TTree binary format and API and has shown to deliver substantially faster data throughput and better data compression both compared to TTree and to industry standard formats. In order to let HENP computing workflows benefit from RNTuple’s superior performance, however, the I/O stack needs to connect efficiently to the rest of the ecosystem, from grid storage to (distributed) analysis frameworks to (multithreaded) experiment frameworks for reconstruction and ntuple derivation. With the RNTuple binary format soon arriving at its first production release, we present RNTuple’s feature set, integration efforts, and its performance impact on the time-to-solution. We show the latest performance figures of RDataFrame analysis code of realistic complexity, comparing RNTuple and TTree as data sources. We discuss RNTuple’s approach to functionality critical to the HENP I/O (such as multithreaded writes, fast data merging, schema evolution) and we provide an outlook on the road to its use in production.
- Published
- 2024
- Full Text
- View/download PDF
13. Boosting RDataFrame performance with transparent bulk event processing
- Author
-
Guiraud Enrico, Blomer Jakob, Canal Philippe, and Naumann Axel
- Subjects
Physics ,QC1-999 - Abstract
RDataFrame is ROOT’s high-level interface for Python and C++ data analysis. Since it first became available, RDataFrame adoption has grown steadily and it is now poised to be a major component of analysis software pipelines for LHC Run 3 and beyond. Thanks to its design inspired by declarative programming principles, RDataFrame enables the development of highperformance, highly parallel analyses without requiring expert knowledge of multi-threading and I/O: user logic is expressed in terms of self-contained, small computation kernels tied together by a high-level API. This design completely decouples analysis logic from its actual execution, and opens several interesting avenues for workflow optimization. In particular, in this work we explore the benefits of moving internal data processing from an event-by-event to a bulkby-bulk loop. This refactoring dramatically reduces the framework’s runtime overheads; in collaboration with the I/O layer it improves data access patterns; it exposes information that optimizing compilers might use to auto-vectorize the invocation of user-defined computations; finally, while existing user-facing interfaces remain unaffected, it becomes possible to additionally offer interfaces that explicitly expose bulks of events, useful e.g. for the injection of GPU kernels into the analysis workflow. In order to inform similar future R&D, design challenges will be presented, as well as an investigation of the relevant timememory trade-off backed by novel performance benchmarks.
- Published
- 2024
- Full Text
- View/download PDF
14. I/O performance studies of analysis workloads on production and dedicated resources at CERN
- Author
-
Sciabà Andrea, Blomer Jakob, Canal Philippe, Duellmann Dirk, Guiraud Enrico, Naumann Axel, Padulano Vincenzo Eduardo, Panzer-Steindel Bernd, Peters Andreas, Schulz Markus, and Smith David
- Subjects
Physics ,QC1-999 - Abstract
The recent evolutions of the analysis frameworks and physics data formats of the LHC experiments provide the opportunity of using central analysis facilities with a strong focus on interactivity and short turnaround times, to complement the more common distributed analysis on the Grid. In order to plan for such facilities, it is essential to know in detail the performance of the combination of a given analysis framework, of a specific analysis and of the installed computing and storage resources. This contribution describes performance studies performed at CERN, using the EOS disk-based storage, either directly or through an XCache instance, from both batch resources and highperformance compute nodes which could be used to build an analysis facility. A variety of benchmarks, both synthetic and based on real-world physics analyses and their corresponding input datasets, are utilized. In particular, the RNTuple format from the ROOT project is put to the test and compared to the latest version of the TTree format, and the impact of caches is assessed. In addition, we assessed the difference in performance between the use of storage system specific protocols, like XRootd, and FUSE. The results of this study are intended to be a valuable input in the design of analysis facilities, at CERN and elsewhere.
- Published
- 2024
- Full Text
- View/download PDF
15. A Roadmap for HEP Software and Computing R&D for the 2020s
- Author
-
Albrecht, Johannes, Alves, Antonio Augusto, Amadio, Guilherme, Andronico, Giuseppe, Anh-Ky, Nguyen, Aphecetche, Laurent, Apostolakis, John, Asai, Makoto, Atzori, Luca, Babik, Marian, Bagliesi, Giuseppe, Bandieramonte, Marilena, Banerjee, Sunanda, Barisits, Martin, Bauerdick, Lothar AT, Belforte, Stefano, Benjamin, Douglas, Bernius, Catrin, Bhimji, Wahid, Bianchi, Riccardo Maria, Bird, Ian, Biscarat, Catherine, Blomer, Jakob, Bloom, Kenneth, Boccali, Tommaso, Bockelman, Brian, Bold, Tomasz, Bonacorsi, Daniele, Boveia, Antonio, Bozzi, Concezio, Bracko, Marko, Britton, David, Buckley, Andy, Buncic, Predrag, Calafiura, Paolo, Campana, Simone, Canal, Philippe, Canali, Luca, Carlino, Gianpaolo, Castro, Nuno, Cattaneo, Marco, Cerminara, Gianluca, Cervantes Villanueva, Javier, Chang, Philip, Chapman, John, Chen, Gang, Childers, Taylor, Clarke, Peter, Clemencic, Marco, Cogneras, Eric, Coles, Jeremy, Collier, Ian, Colling, David, Corti, Gloria, Cosmo, Gabriele, Costanzo, Davide, Couturier, Ben, Cranmer, Kyle, Cranshaw, Jack, Cristella, Leonardo, Crooks, David, Crépé-Renaudin, Sabine, Currie, Robert, Dallmeier-Tiessen, Sünje, De, Kaushik, De Cian, Michel, De Roeck, Albert, Delgado Peris, Antonio, Derue, Frédéric, Di Girolamo, Alessandro, Di Guida, Salvatore, Dimitrov, Gancho, Doglioni, Caterina, Dotti, Andrea, Duellmann, Dirk, Duflot, Laurent, Dykstra, Dave, Dziedziniewicz-Wojcik, Katarzyna, Dziurda, Agnieszka, Egede, Ulrik, Elmer, Peter, Elmsheuser, Johannes, Elvira, V Daniel, Eulisse, Giulio, Farrell, Steven, Ferber, Torben, Filipcic, Andrej, Fisk, Ian, Fitzpatrick, Conor, Flix, José, Formica, Andrea, Forti, Alessandra, Franzoni, Giovanni, Frost, James, Fuess, Stu, Gaede, Frank, Ganis, Gerardo, Gardner, Robert, Garonne, Vincent, and Gellrich, Andreas
- Subjects
Information and Computing Sciences ,Physical Sciences ,Software Engineering ,physics.comp-ph ,hep-ex - Abstract
Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.
- Published
- 2019
16. A Roadmap for HEP Software and Computing R&D for the 2020s
- Author
-
Albrecht, Johannes, Alves Jr, Antonio Augusto, Amadio, Guilherme, Andronico, Giuseppe, Anh-Ky, Nguyen, Aphecetche, Laurent, Apostolakis, John, Asai, Makoto, Atzori, Luca, Babik, Marian, Bagliesi, Giuseppe, Bandieramonte, Marilena, Banerjee, Sunanda, Barisits, Martin, Bauerdick, Lothar A. T., Belforte, Stefano, Benjamin, Douglas, Bernius, Catrin, Bhimji, Wahid, Bianchi, Riccardo Maria, Bird, Ian, Biscarat, Catherine, Blomer, Jakob, Bloom, Kenneth, Boccali, Tommaso, Bockelman, Brian, Bold, Tomasz, Bonacorsi, Daniele, Boveia, Antonio, Bozzi, Concezio, Bracko, Marko, Britton, David, Buckley, Andy, Buncic, Predrag, Calafiura, Paolo, Campana, Simone, Canal, Philippe, Canali, Luca, Carlino, Gianpaolo, Castro, Nuno, Cattaneo, Marco, Cerminara, Gianluca, Villanueva, Javier Cervantes, Chang, Philip, Chapman, John, Chen, Gang, Childers, Taylor, Clarke, Peter, Clemencic, Marco, Cogneras, Eric, Coles, Jeremy, Collier, Ian, Colling, David, Corti, Gloria, Cosmo, Gabriele, Costanzo, Davide, Couturier, Ben, Cranmer, Kyle, Cranshaw, Jack, Cristella, Leonardo, Crooks, David, Crépé-Renaudin, Sabine, Currie, Robert, Dallmeier-Tiessen, Sünje, De, Kaushik, De Cian, Michel, De Roeck, Albert, Peris, Antonio Delgado, Derue, Frédéric, Di Girolamo, Alessandro, Di Guida, Salvatore, Dimitrov, Gancho, Doglioni, Caterina, Dotti, Andrea, Duellmann, Dirk, Duflot, Laurent, Dykstra, Dave, Dziedziniewicz-Wojcik, Katarzyna, Dziurda, Agnieszka, Egede, Ulrik, Elmer, Peter, Elmsheuser, Johannes, Elvira, V. Daniel, Eulisse, Giulio, Farrell, Steven, Ferber, Torben, Filipcic, Andrej, Fisk, Ian, Fitzpatrick, Conor, Flix, José, Formica, Andrea, Forti, Alessandra, Franzoni, Giovanni, Frost, James, Fuess, Stu, Gaede, Frank, Ganis, Gerardo, Gardner, Robert, Garonne, Vincent, Gellrich, Andreas, Genser, Krzysztof, George, Simon, Geurts, Frank, Gheata, Andrei, Gheata, Mihaela, Giacomini, Francesco, Giagu, Stefano, Giffels, Manuel, Gingrich, Douglas, Girone, Maria, Gligorov, Vladimir V., Glushkov, Ivan, Gohn, Wesley, Lopez, Jose Benito Gonzalez, Caballero, Isidro González, Fernández, Juan R. González, Govi, Giacomo, Grandi, Claudio, Grasland, Hadrien, Gray, Heather, Grillo, Lucia, Guan, Wen, Gutsche, Oliver, Gyurjyan, Vardan, Hanushevsky, Andrew, Hariri, Farah, Hartmann, Thomas, Harvey, John, Hauth, Thomas, Hegner, Benedikt, Heinemann, Beate, Heinrich, Lukas, Heiss, Andreas, Hernández, José M., Hildreth, Michael, Hodgkinson, Mark, Hoeche, Stefan, Holzman, Burt, Hristov, Peter, Huang, Xingtao, Ivanchenko, Vladimir N., Ivanov, Todor, Iven, Jan, Jashal, Brij, Jayatilaka, Bodhitha, Jones, Roger, Jouvin, Michel, Jun, Soon Yung, Kagan, Michael, Kalderon, Charles William, Kane, Meghan, Karavakis, Edward, Katz, Daniel S., Kcira, Dorian, Keeble, Oliver, Kersevan, Borut Paul, Kirby, Michael, Klimentov, Alexei, Klute, Markus, Komarov, Ilya, Konstantinov, Dmitri, Koppenburg, Patrick, Kowalkowski, Jim, Kreczko, Luke, Kuhr, Thomas, Kutschke, Robert, Kuznetsov, Valentin, Lampl, Walter, Lancon, Eric, Lange, David, Lassnig, Mario, Laycock, Paul, Leggett, Charles, Letts, James, Lewendel, Birgit, Li, Teng, Lima, Guilherme, Linacre, Jacob, Linden, Tomas, Livny, Miron, Presti, Giuseppe Lo, Lopienski, Sebastian, Love, Peter, Lyon, Adam, Magini, Nicolò, Marshall, Zachary L., Martelli, Edoardo, Martin-Haugh, Stewart, Mato, Pere, Mazumdar, Kajari, McCauley, Thomas, McFayden, Josh, McKee, Shawn, McNab, Andrew, Mehdiyev, Rashid, Meinhard, Helge, Menasce, Dario, Lorenzo, Patricia Mendez, Mete, Alaettin Serhan, Michelotto, Michele, Mitrevski, Jovan, Moneta, Lorenzo, Morgan, Ben, Mount, Richard, Moyse, Edward, Murray, Sean, Nairz, Armin, Neubauer, Mark S., Norman, Andrew, Novaes, Sérgio, Novak, Mihaly, Oyanguren, Arantza, Ozturk, Nurcan, Pages, Andres Pacheco, Paganini, Michela, Pansanel, Jerome, Pascuzzi, Vincent R., Patrick, Glenn, Pearce, Alex, Pearson, Ben, Pedro, Kevin, Perdue, Gabriel, Yzquierdo, Antonio Perez-Calero, Perrozzi, Luca, Petersen, Troels, Petric, Marko, Petzold, Andreas, Piedra, Jónatan, Piilonen, Leo, Piparo, Danilo, Pivarski, Jim, Pokorski, Witold, Polci, Francesco, Potamianos, Karolos, Psihas, Fernanda, Navarro, Albert Puig, Quast, Günter, Raven, Gerhard, Reuter, Jürgen, Ribon, Alberto, Rinaldi, Lorenzo, Ritter, Martin, Robinson, James, Rodrigues, Eduardo, Roiser, Stefan, Rousseau, David, Roy, Gareth, Rybkine, Grigori, Sailer, Andre, Sakuma, Tai, Santana, Renato, Sartirana, Andrea, Schellman, Heidi, Schovancová, Jaroslava, Schramm, Steven, Schulz, Markus, Sciabà, Andrea, Seidel, Sally, Sekmen, Sezen, Serfon, Cedric, Severini, Horst, Sexton-Kennedy, Elizabeth, Seymour, Michael, Sgalaberna, Davide, Shapoval, Illya, Shiers, Jamie, Shiu, Jing-Ge, Short, Hannah, Siroli, Gian Piero, Skipsey, Sam, Smith, Tim, Snyder, Scott, Sokoloff, Michael D., Spentzouris, Panagiotis, Stadie, Hartmut, Stark, Giordon, Stewart, Gordon, Stewart, Graeme A., Sánchez, Arturo, Sánchez-Hernández, Alberto, Taffard, Anyes, Tamponi, Umberto, Templon, Jeff, Tenaglia, Giacomo, Tsulaia, Vakhtang, Tunnell, Christopher, Vaandering, Eric, Valassi, Andrea, Vallecorsa, Sofia, Valsan, Liviu, Van Gemmeren, Peter, Vernet, Renaud, Viren, Brett, Vlimant, Jean-Roch, Voss, Christian, Votava, Margaret, Vuosalo, Carl, Sierra, Carlos Vázquez, Wartel, Romain, Watts, Gordon T., Wenaus, Torre, Wenzel, Sandro, Williams, Mike, Winklmeier, Frank, Wissing, Christoph, Wuerthwein, Frank, Wynne, Benjamin, Xiaomei, Zhang, Yang, Wei, and Yazgan, Efe
- Subjects
Physics - Computational Physics ,High Energy Physics - Experiment - Abstract
Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.
- Published
- 2017
- Full Text
- View/download PDF
17. Status Report of the DPHEP Collaboration: A Global Effort for Sustainable Data Preservation in High Energy Physics
- Author
-
DPHEP Collaboration, Amerio, Silvia, Barbera, Roberto, Berghaus, Frank, Blomer, Jakob, Branson, Andrew, Cancio, Germán, Cartaro, Concetta, Chen, Gang, Dallmeier-Tiessen, Sünje, Diaconu, Cristinel, Ganis, Gerardo, Gheata, Mihaela, Hara, Takanori, Herner, Ken, Hildreth, Mike, Jones, Roger, Kluth, Stefan, Krücker, Dirk, Lassila-Perini, Kati, Maggi, Marcello, de Lucas, Jesus Marco, Mele, Salvatore, Pace, Alberto, Schröder, Matthias, Shamdasani, Jetendr, Shiers, Jamie, Smith, Tim, Sobie, Randall, South, David Michael, Verbytskyi, Andrii, Viljoen, Matthew, Wang, Lu, and Zimmermann, Markus
- Subjects
High Energy Physics - Experiment ,Computer Science - Digital Libraries - Abstract
Data from High Energy Physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organizational aspects of HEP data preservation. An intermediate report was released in November 2009 addressing the general issues of data preservation in HEP and an extended blueprint paper was published in 2012. In July 2014 the DPHEP collaboration was formed as a result of the signature of the Collaboration Agreement by seven large funding agencies (others have since joined or are in the process of acquisition) and in June 2015 the first DPHEP Collaboration Workshop and Collaboration Board meeting took place. This status report of the DPHEP collaboration details the progress during the period from 2013 to 2015 inclusive., Comment: report, 60 pages
- Published
- 2015
- Full Text
- View/download PDF
18. The Need for a Versioned Data Analysis Software Environment
- Author
-
Blomer, Jakob, Berzano, Dario, Buncic, Predrag, Charalampidis, Ioannis, Ganis, Gerardo, Lestaris, George, and Meusel, René
- Subjects
Computer Science - Software Engineering - Abstract
Scientific results in high-energy physics and in many other fields often rely on complex software stacks. In order to support reproducibility and scrutiny of the results, it is good practice to use open source software and to cite software packages and versions. With ever-growing complexity of scientific software on one side and with IT life-cycles of only a few years on the other side, however, it turns out that despite source code availability the setup and the validation of a minimal usable analysis environment can easily become prohibitively expensive. We argue that there is a substantial gap between merely having access to versioned source code and the ability to create a data analysis runtime environment. In order to preserve all the different variants of the data analysis runtime environment, we developed a snapshotting file system optimized for software distribution. We report on our experience in preserving the analysis environment for high-energy physics such as the software landscape used to discover the Higgs boson at the Large Hadron Collider.
- Published
- 2014
19. PROOF as a Service on the Cloud: a Virtual Analysis Facility based on the CernVM ecosystem
- Author
-
Berzano, Dario, Blomer, Jakob, Buncic, Predrag, Charalampidis, Ioannis, Ganis, Gerardo, Lestaris, Georgios, and Meusel, René
- Subjects
Computer Science - Distributed, Parallel, and Cluster Computing ,Physics - Data Analysis, Statistics and Probability - Abstract
PROOF, the Parallel ROOT Facility, is a ROOT-based framework which enables interactive parallelism for event-based tasks on a cluster of computing nodes. Although PROOF can be used simply from within a ROOT session with no additional requirements, deploying and configuring a PROOF cluster used to be not as straightforward. Recently great efforts have been spent to make the provisioning of generic PROOF analysis facilities with zero configuration, with the added advantages of positively affecting both stability and scalability, making the deployment operations feasible even for the end user. Since a growing amount of large-scale computing resources are nowadays made available by Cloud providers in a virtualized form, we have developed the Virtual PROOF-based Analysis Facility: a cluster appliance combining the solid CernVM ecosystem and PoD (PROOF on Demand), ready to be deployed on the Cloud and leveraging some peculiar Cloud features such as elasticity. We will show how this approach is effective both for sysadmins, who will have little or no configuration to do to run it on their Clouds, and for the end users, who are ultimately in full control of their PROOF cluster and can even easily restart it by themselves in the unfortunate event of a major failure. We will also show how elasticity leads to a more optimal and uniform usage of Cloud resources., Comment: Talk from Computing in High Energy and Nuclear Physics 2013 (CHEP2013), Amsterdam (NL), October 2013, 7 pages, 4 figures
- Published
- 2014
- Full Text
- View/download PDF
20. HL-LHC Analysis With ROOT
- Author
-
Naumann, Axel, primary, Canal, Philippe, additional, Tejedor, Enric, additional, Guiraud, Enrico, additional, Moneta, Lorenzo, additional, Bellenot, Bertrand, additional, Couet, Olivier, additional, Tadel, Alja, additional, Tadel, Matevz, additional, Linev, Sergey, additional, Gomez, Javier, additional, Rembser, Jonas, additional, Padulano, Vincenzo, additional, Blomer, Jakob, additional, Hahnfeld, Jonas, additional, Gruber, Bernhard, additional, and Vassilev, Vassil, additional
- Published
- 2022
- Full Text
- View/download PDF
21. ROOT for the HL-LHC: data format
- Author
-
Naumann, Axel, primary, Canal, Philippe, additional, Tejedor, Enric, additional, Guiraud, Enrico, additional, Moneta, Lorenzo, additional, Bellenot, Bertrand, additional, Couet, Olivier, additional, Tadel, Alja, additional, Tadel, Matevz, additional, Linev, Sergey, additional, Gomez, Javier, additional, Rembser, Jonas, additional, Padulano, Vincenzo, additional, Blomer, Jakob, additional, Hahnfeld, Jonas, additional, Gruber, Bernhard, additional, and Vassilev, Vassil, additional
- Published
- 2022
- Full Text
- View/download PDF
22. Delivering LHC Software to HPC Compute Elements with CernVM-FS
- Author
-
Blomer, Jakob, Ganis, Gerardo, Hardi, Nikola, Popescu, Radu, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Kunkel, Julian M., editor, Yokota, Rio, editor, Taufer, Michela, editor, and Shalf, John, editor
- Published
- 2017
- Full Text
- View/download PDF
23. RNTuple performance: Status and Outlook
- Author
-
Lopez-Gomez, Javier, primary and Blomer, Jakob, additional
- Published
- 2023
- Full Text
- View/download PDF
24. CernVM-FS ephemeral publishers on Kubernetes
- Author
-
Valenzuela, Andrea, primary and Blomer, Jakob, additional
- Published
- 2023
- Full Text
- View/download PDF
25. Annual Report 2022
- Author
-
Aglieri, Gianluca, Aleksa, Martin, Feito, Diego Alvarez, Carballo, Aitor Amatriain, Andorno, Marco, Angeletti, Massimo, Antoszczuk, Pablo, Zarate, Fernando Aretio, Tortajada, I. Asensi, Auffray, Etiennette, Ballabriga, Rafael, Balutto, Mattia, Bandi, Franco, Higueras, Maria Barba, Barney, David, Baron, Sophie, Baszczyk, Mateusz Karol, Bialas, Wojciech, Biereigel, Stefan, Blomer, Jakob, Bordelius, Aleksandar, Borghello, G., Bouvier, Philippe, Boyer, Francois, Braach, Justus, De Souza Mendes, Eduardo Brandao, Brondolin, Erica, Brunbauer, Florian, Buschmann, Eric, Buytaert, Jan, Byczynski, Wiktor, Cala', Roberto, Campbell, Michael, Caratelli, Alessandro, Carnesecchi, Ricardo, Catinaccio, Andrea, Cecconi, Leonardo, Ceresa, Davide, Lemos, Edgar Cid, Coco, Victor, Collins, Paula, Contiero, Luca, Arena, Maria Cristina, Cure, Benoit, Rivera, Esteban Curras, Dachs, Florian, Dall'Omo, Frederik, D'Ambrosio, Carmelo, Dannheim, Dominik, Dao, Valerio, De Melo, Joao, De Oliveira, Rui, Deng, Wenjing, Detraz, Stephane, Dhaliwal, Jashandeep, Di Castro, Mario, Di Mauro, Antonello, Dias, Matheo, Dobrijevic, Dominik, Martin, Ana Dorda, Dorosz, Piotr, Dort, Katharina, Dudarev, Alexey, Dumps, Raphael, Emiliani, Simone, Faccio, Federico, Francois, Brieuc, Bulling, Anthony Frederick, Frei, Christoph, Gabrielli, Andrea, Gargiulo, Corrado, Gessinger-Befurt, Paul, Gkougkousis, Vagelis, Gluchowska, Weronika, Gose, Melwin, Guida, Roberto, Gustavino, Carlo, Haimberger, Jakob, Halvorsen, Marius, Hasenbichler, Jan, Hawkings, Richard, Hong, Geun Hee, Hegner, Benedikt, Hellenschmidt, Desiree, Heribi, Quassim, Hillemans, Hartmut, Himmerlich, Anja, Ijzermans, Pieter, Jaekel, Martin, Jakobsen, Sune, Jama, Kacper, Janot, Patrick, Janssens, Djunes, Floethner, Karl Jonathan, Joram, Christian, Junique, Antoine, Jurco, Robert, Kaplon, Jan, Keizer, Floris, Kiehn, Moritz, Klekotko, Adam, Kloukinas, Kostas, Kluge, Alex, Krammer, Manfred, Kratochwil, Nicolaus, Kremastiotis, Iraklis, Kucharska, Gabriela, Kugathasan, Thanushan, Kühn, Susanne, Kulis, Szymon, La Rosa, Alessandro, Lalovic, Milana, Laudi, Elisa, Le Blanc, M., Ledey, Gael, Miotto, Giovanna Lehmann, Lemoine, Corentin, Linssen, Lucie, Lisowska, Marta, Gomez, Javier Lopez, Mager, Magnus, Malentacca, Lorenzo, Mandelli, Beatrice, Manolescu, Florentian, Martina, Francesco, Martinazzoli, Loris, Martinengo, Paolo, Vila, Pere Mato, Maulerova, Vendula, Mazzei, Francesco, McAlpine, Lee, Mehl, Bertrand, Mentink, Matthias, Michelis, Stephano, Carceller, Juan Miguel, Leitao, Pedro Miguel Vicente, Mlynarikova, Michaela, Moll, Michael, Moneta, Lorenzo, Muller, Hans, Musa, Luciano, Bandi, Franco Nahuel, Nauman, Axel, Nookala, Anvesh, Olantera, Lauri, Oliveri, Eraldo, Orlandini, Giorgio, Pacifico, Nicola, Padulano, Vincenzo, Pandey, Awanish, Pantaleo, Felice, Pape, Sebastian, Pejasinovic, Risto, Pernegger, Heinz, Petagna, Paolo, Piedigrossi, Didier, Diaz, Francisco Piernas, Piller, Markus, Piro, Francesco, Pizzichemi, Marco, Pizzirusso, Olivier, Poblocki, Marcin, Prousalidi, Thenia, Rebane, Karolina, Reichenbach, Leonhard, Reidt, Felix, Rembser, Christoph, Riedler, Petra, Riegler, Werner, Rigoletti, Gianluca, Moreira, Paulo Rodrigues Simoes, Rodrigues, Alexis, Roloff, Philipp, Ropelewski, Leszek, Rovere, Marco, Sailer, André, Salamani, Dalila, Salomoni, Matteo, Salzburger, Andreas, Sanna, Isabella, Sasikumar, Swathi, Sauli, Fabio, Scarcella, Carmelo, Scharenberg, Lucian, Schindler, Heinrich, Schmidt, Burkhard, Schmidt, Janis, Schneider, Thomas, Schopper, Andreas, Secouet, Pascal, Sharma, Abhishek, Sicking, Eva, Sigaud, Christophe, Singh, Shuvay, Sirskaran, Viros, Snoeys, Walter, Solans, Carlos, Gonzalez, Maria Soledad Molina, Soos, Csaba, Stewart, Graeme A., Suljic, Miljenko, Svhira, Peter, Teixeira, Antonio, Teofili, Lorenzo, Termo, Gennaro, Troska, Jan, Utrobicic, Antonija, Van Beelen, Jacob, Van Rijnbach, Milou, Van Stenis, Miranda, Vasey, Francois, Vaskuri, Anna, Nunez, Marcos Vazquez, Veenhof, Rob, Verzeroli, Mattia, Pinto, Mateus Vicente Barreto, Leitao, Pedro Vicente, Volker, Alexander, Volkl, Valentin, Wanotayaroj, Chaowaroj, Weick, Julian, Wiehe, Moritz, Wilkens, Henric, and Zaborowska, Anna
- Abstract
This report summarises the activities and main achievements of the CERN strategic R&D programme on technologies for future experiments during the year 2022
- Published
- 2023
26. Extension of the R&D Programme on Technologies for Future Experiments
- Author
-
Joram, Christian, Aglieri, Gianluca, Aleksa, Martin, Feito, Diego Alvarez, Carballo, Aitor Amatriain, Andorno, Marco, Angeletti, Massimo, Antoszczuk, Pablo, Zarate, Fernando Aretio, Tortajada, I. Asensi, Auffray, Etiennette, Ballabriga, Rafael, Balutto, Mattia, Bandi, Franco, Higueras, Maria Barba, Barney, David, Baron, Sophie, Baszczyk, Mateusz Karol, Bialas, Wojciech, Biereigel, Stefan, Blomer, Jakob, Bordelius, Aleksandar, Borghello, G., Bouvier, Philippe, Boyer, Francois, Braach, Justus, De Souza Mendes, Eduardo Brandao, Brondolin, Erica, Brunbauer, Florian, Buschmann, Eric, Buytaert, Jan, Byczynski, Wiktor, Cala', Roberto, Campbell, Michael, Caratelli, Alessandro, Carnesecchi, Ricardo, Catinaccio, Andrea, Cecconi, Leonardo, Ceresa, Davide, Lemos, Edgar Cid, Coco, Victor, Collins, Paula, Contiero, Luca, Arena, Maria Cristina, Cure, Benoit, Rivera, Esteban Curras, Dachs, Florian, Dall'Omo, Frederik, D'Ambrosio, Carmelo, Dannheim, Dominik, Dao, Valerio, De Melo, Joao, De Oliveira, Rui, Deng, Wenjing, Detraz, Stephane, Dhaliwal, Jashandeep, Di Castro, Mario, Di Mauro, Antonello, Dias, Matheo, Dobrijevic, Dominik, Martin, Ana Dorda, Dorosz, Piotr, Dort, Katharina, Dudarev, Alexey, Dumps, Raphael, Emiliani, Simone, Faccio, Federico, Francois, Brieuc, Bulling, Anthony Frederick, Frei, Christoph, Gabrielli, Andrea, Gargiulo, Corrado, Gessinger-Befurt, Paul, Gkougkousis, Vagelis, Gluchowska, Weronika, Gose, Melwin, Guida, Roberto, Gustavino, Carlo, Haimberger, Jakob, Halvorsen, Marius, Hasenbichler, Jan, Hawkings, Richard, Hong, Geun Hee, Hegner, Benedikt, Hellenschmidt, Desiree, Heribi, Quassim, Hillemans, Hartmut, Himmerlich, Anja, Ijzermans, Pieter, Jaekel, Martin, Jakobsen, Sune, Jama, Kacper, Janot, Patrick, Janssens, Djunes, Floethner, Karl Jonathan, Junique, Antoine, Jurco, Robert, Kaplon, Jan, Keizer, Floris, Kiehn, Moritz, Klekotko, Adam, Kloukinas, Kostas, Kluge, Alex, Krammer, Manfred, Kratochwil, Nicolaus, Kremastiotis, Iraklis, Kucharska, Gabriela, Kugathasan, Thanushan, Kühn, Susanne, Kulis, Szymon, La Rosa, Alessandro, Lalovic, Milana, Laudi, Elisa, Le Blanc, M., Ledey, Gael, Miotto, Giovanna Lehmann, Lemoine, Corentin, Linssen, Lucie, Lisowska, Marta, Gomez, Javier Lopez, Mager, Magnus, Malentacca, Lorenzo, Mandelli, Beatrice, Manolescu, Florentian, Martina, Francesco, Martinazzoli, Loris, Martinengo, Paolo, Vila, Pere Mato, Maulerova, Vendula, Mazzei, Francesco, McAlpine, Lee, Mehl, Bertrand, Mentink, Matthias, Michelis, Stephano, Carceller, Juan Miguel, Leitao, Pedro Miguel Vicente, Mlynarikova, Michaela, Moll, Michael, Moneta, Lorenzo, Muller, Hans, Musa, Luciano, Bandi, Franco Nahuel, Nauman, Axel, Nookala, Anvesh, Olantera, Lauri, Oliveri, Eraldo, Orlandini, Giorgio, Pacifico, Nicola, Padulano, Vincenzo, Pandey, Awanish, Pantaleo, Felice, Pape, Sebastian, Pejasinovic, Risto, Pernegger, Heinz, Petagna, Paolo, Piedigrossi, Didier, Diaz, Francisco Piernas, Piller, Markus, Piro, Francesco, Pizzichemi, Marco, Pizzirusso, Olivier, Poblocki, Marcin, Prousalidi, Thenia, Rebane, Karolina, Reichenbach, Leonhard, Reidt, Felix, Rembser, Christoph, Riedler, Petra, Riegler, Werner, Rigoletti, Gianluca, Moreira, Paulo Rodrigues Simoes, Rodrigues, Alexis, Roloff, Philipp, Ropelewski, Leszek, Rovere, Marco, Sailer, André, Salamani, Dalila, Salomoni, Matteo, Salzburger, Andreas, Sanna, Isabella, Sasikumar, Swathi, Sauli, Fabio, Scarcella, Carmelo, Scharenberg, Lucian, Schindler, Heinrich, Schmidt, Burkhard, Schmidt, Janis, Schneider, Thomas, Schopper, Andreas, Secouet, Pascal, Sharma, Abhishek, Sicking, Eva, Sigaud, Christophe, Singh, Shuvay, Sirskaran, Viros, Snoeys, Walter, Solans, Carlos, Gonzalez, Maria Soledad Molina, Soos, Csaba, Stewart, Graeme A., Suljic, Miljenko, Svhira, Peter, Teixeira, Antonio, Teofili, Lorenzo, Termo, Gennaro, Troska, Jan, Utrobicic, Antonija, Van Beelen, Jacob, Van Rijnbach, Milou, Van Stenis, Miranda, Vasey, Francois, Vaskuri, Anna, Nunez, Marcos Vazquez, Veenhof, Rob, Verzeroli, Mattia, Pinto, Mateus Vicente Barreto, Leitao, Pedro Vicente, Volker, Alexander, Volkl, Valentin, Wanotayaroj, Chaowaroj, Weick, Julian, Wiehe, Moritz, Wilkens, Henric, and Zaborowska, Anna
- Abstract
we have conceived an extension of the R&D programme covering the period 2024 to 2028, i.e. again a 5-year period, however with 2024 as overlap year. This step was encouraged by the success of the current programme but also by the Europe-wide efforts to launch new Detector R&D collaborations in the framework of the ECFA Detector R&D Roadmap. We propose to continue our R&D programme with the main activities in essentially the same areas. All activities are fully aligned with the ECFA Roadmap and in most cases will be carried out under the umbrella of one of the new DRD collaborations. The program is a mix of natural continuations of the current activities and a couple of very innovative new developments, such as a radiation hard embedded FPGA implemented in an ASIC based on System-on-Chip technology. A special and urgent topic is the fabrication of Al-reinforced super-conducting cables. Such cables are a core ingredient of any new superconducting magnet such as BabyIAXO, PANDA, EIC, ALICE-3 etc. Production volumes are small and demands come in irregular intervals. Industry (world-wide) is no longer able and willing to fabricate such cables. The most effective approach (technically and financially) may be to re-invent the process at CERN, together with interested partners, and offer this service to the community.
- Published
- 2023
27. ZERO-COPY MERGE FOR ROOT RNTUPLE
- Author
-
Marinelli, Eugenio, Blomer, Jakob, and Lopez Gomez, Javier
- Subjects
CERN openlab, CERN ,summer-student programme - Abstract
Data sets in High-Energy Physics (HEP) are often produced in a parallel manner where tens or hundreds of processes produce partial results that are merged into a single file at the end of the data processing workflow. This is enabled by the parallel nature of HEP data, where individual collisions can be processed independently from each other. The predominant file format for storing HEP data is provided by the ROOT libraries. Modern file systems allow for zero-copy cloning of data blocks. This feature has a potentially high impact for the merging of HEP data, as it may allow creation of merged results without physically copying the partial results. The aim of this project is to evaluate the zero-copy merging functionality in modern file systems as well as the requirements to use it at the file format level (e.g., relocatability). The results of this project have an impact on ROOT’s object store support as well, as HEP data in object stores such as Intel DAOS is by nature prepared to exploit zero-copy merging.
- Published
- 2023
- Full Text
- View/download PDF
28. Preparing for HL-LHC: Increasing the LHCb software publication rate to CVMFS by an order of magnitude
- Author
-
Bocchi Enrico, Blomer Jakob, Couturier Benjamin, Burr Christopher, and van der Ster Dan
- Subjects
Physics ,QC1-999 - Abstract
In the HEP community, software plays a central role in the operation of experiments’ facilities and for reconstruction jobs, with CVMFS being the service enabling the distribution of software at scale. In view of High Luminosity LHC, CVMFS developers investigated how to improve the publication workflow to support the most demanding use cases. This paper reports about recent CVMFS developments and infrastructural updates that enable faster publication into existing repositories. A new CVMFS component, the CVMFS Gateway, allows for concurrent transactions and the use of multiple publishers, increasing the overall publication rate on a single repository. Also, the repository data has been migrated to Ceph-based S3 object storage, which brings a relevant performance enhancement over the previously-used Cinder volumes. We demonstrate how recent improvements allow for faster publication of software releases in CVMFS repositories by focusing on the LHCb nightly builds use case, which is currently by far the most demanding one for the CVMFS infrastructure at CERN. The publication of nightly builds is characterized by a high churn rate, needs regular garbage collection, and requires the ability to ingest a huge amount of software files over a limited period of time.
- Published
- 2021
- Full Text
- View/download PDF
29. CernVM-FS powered container hub
- Author
-
Bocchi Enrico, Blomer Jakob, Mosciatti Simone, and Valenzuela Andrea
- Subjects
Physics ,QC1-999 - Abstract
Containers became the de-facto standard to package and distribute modern applications and their dependencies. The HEP community demonstrates an increasing interest in such technology, with scientists encapsulating their analysis workflow and code inside a container image. The analysis is first validated on a small dataset and minimal hardware resources to then run at scale on the massive computing capacity provided by the grid. The typical approach for distributing containers consists of pulling their image from a remote registry and extracting it on the node where the container runtime (e.g., Docker, Singularity) runs. This approach, however, does not easily scale to large images and thousands of nodes. CVMFS has long been used for the efficient distribution of software directory trees at a global scale. In order to extend its optimized caching and network utilization to the distribution of containers, CVMFS recently implemented a dedicated container image ingestion service together with container runtime integrations. CVMFS ingestion is based on per-file deduplication, instead of the per-layer deduplication adopted by traditional container registries. On the client-side, CVMFS implements on-demand fetching of the chunks required for the execution of the container instead of the whole image.
- Published
- 2021
- Full Text
- View/download PDF
30. A caching mechanism to exploit object store speed in High Energy Physics analysis
- Author
-
Padulano, Vincenzo Eduardo, primary, Tejedor Saavedra, Enric, additional, Alonso-Jordá, Pedro, additional, López Gómez, Javier, additional, and Blomer, Jakob, additional
- Published
- 2022
- Full Text
- View/download PDF
31. A fully unprivileged CernVM-FS
- Author
-
Blomer Jakob, Dykstra Dave, Ganis Gerardo, Mosciatti Simone, and Priessnitz Jan
- Subjects
Physics ,QC1-999 - Abstract
The CernVM File System provides the software and container distribution backbone for most High Energy and Nuclear Physics experiments. It is implemented as a file system in user-space (Fuse) module, which permits its execution without any elevated privileges. Yet, mounting the file system in the first place is handled by a privileged suid helper program that is installed by the Fuse package on most systems. The privileged nature of the mount system call is a serious hindrance to running CernVM-FS on opportunistic resource and supercomputers. Fortunately, recent developments in the Linux kernel and in the Fuse user-space libraries enabled fully unprivileged mounting for Fuse file systems (as of RHEL 8), or at least outsourcing the privileged mount system call to a custom, external process. This opens the door to several, very appealing new ways to use CernVM-FS, such as a generally usable “super pilot” consisting of the pilot code bundled with Singularity and CernVM-FS, or the on-demand instantiation of unprivileged, ephemeral containers to publish new CernVM-FS content from anywhere. In this contribution, we discuss the integration of these new Linux features with CernVM-FS and show some of its most promising, new applications.
- Published
- 2020
- Full Text
- View/download PDF
32. Strategic R&D Programme on Technologies for Future Experiments - Annual Report 2021
- Author
-
Aglieri Rinella, Gianluca, Aleksa,Martin, Alvarez Feito,Diego, Andorno,Marco, Angeletti,Massimo, Antoszczuk,Pablo, Asensi Tortajada,I., Auffray Hillemanns, Etiennette, Ballabriga,Raphael, Balutto,Mattia, Bandi,Franco, Barba Higueras,Maria, Barney,David, Baron,Sophie, Barreto Pinto,Mateus, Bastian Van Beelen,Jacob, Baszczyk,Mateusz Karol, Beguin,Marina, Bialas,W., Biereigel,Stefan, Blomer,Jakob, Borghello,G., Braach,Justus, Brandao De Souza Mendes,Eduardo, Brondolin,Erica, Brunbauer,Florian, Buschmann,Eric, Buytaert,Jan, Byczynski,Wiktor, Cala',Roberto, Cambie,Federico, Campbell,Michael, Caratelli,Alessandro, Carnesecchi,F., Catinaccio,Andrea, Cecconi,Leonardo, Ceresa,Davide, Cid Lemos,Edgar, Coco,Victor, Collins,Paula, Contiero,Luca, Corbetta,Mara, Cure,Benoit, Curras Rivera,Esteban, Dachs,Florian, Dall'Omo,Frederik, D'Ambrosio,Carmelo, Dannheim,Dominik, Dao,V., De Melo,Joao, De Oliveira,Rui, Deng,W., Deng,Wenjing, Detraz,Stephane, Di Mauro,Antonello, Dias,Matheo, Dobrijevic,D., Dobrijevic,Dominik, Dorda Martin,A., Dorda Martin,Ana, Dorosz,P., Dort,Katharina, Dudarev,Alexey, Dumps,Raphael, Faccio,Federico, Fernandez Declara,Placido, Fiorenza,Gabriele, Francois,Brieuc, Frederick Bulling ,Anthony, Frei,Christoph, Gabrielli,A., Gargiulo,Corrado, Gessinger-Befurt,Paul, Gkougkousis,Vagelis, Gluchowska,Weronika, Gose,Melwin, Guida,Roberto, Gustavino,C., Haimberger,Jakob, Halvorsen,Marius, Hasenbichler,J., Hawkings,Richard, Hee Hong,Geun, Hellenschmidt,Desiree, Heribi,Quassim, Hillemans,H., Himmerlich,Anja, Ijzermans,Pieter, Janot,Patrick, Janssens,Djunes, Jonathan Floethner ,Karl, Joram,Christian, Kaplon,Jan, Keizer,Floris, Kiehn,Moritz, Klekotko,Adam, Kloukinas,Kostas, Kluge,Alex, Krammer,Manfred, Kratochwil,Nicolaus, Kremastiotis, Iraklis, Kroeger,Jens, Kugathasan,T., La Rosa,Alessandro, Lafuente,Antonio, Lalovic,Milana, Laudi,Elisa, Le Blanc,M., Ledey,Gael, Leitao,M., Linssen,Lucie, Lisowska,Marta, Lopez Gomez,Javier, Maciej Malinowski,Filip, Magatti,Demetrio, Mager,Magnus, Mandelli,Beatrice, Martinazzoli,Loris, Martinengo,Paolo, Mato Vila,Pere, Maulerova,Vendula, Mehl,Bertrand, Mentink,Matthias, Moll,Michael, Muller,Hans, Musa,Luciano, Neroni,Michela, Olantera ,Lauri, Oliveri,Eraldo, Orlandini,Giorgio, Padulano,Vincenzo, Pantaleo,Felice, Pape,Sebastian, Pejasinovic,Risto, Pernegger,Heinz, Petagna,Paolo, Piedigrossi,Didier, Piernas Diaz,Francisco, Piller,Markus, Piro,Francesco, Pizzichemi, Marco, Pizzirusso,Olivier, Prousalidi ,Thenia, Rebane,K., Reidt,F., Rembser,Christoph, Riedler,Petra, Riegler,Werner, Rigoletti,Gianluca, Rodrigues Simoes Moreira,Paulo, Roloff, Philipp Gerhard, Ropelewski,Leszek, Rovere,Marco, Sailer,André, Salamani,Dalila, Salomoni,Matteo, Salzburger,Andreas, Sanna,I., Scarcella ,Carmelo, Scharenberg,Lucian, Schindler,Heinrich, Schmidt,Burkhard, Schmidt,Janis, Schneider,Thomas, Schopper,Andreas, Secouet,Pascal, Sharma,A., Singh,Shuvay, Sirskaran,Viros, Snoeys,Walter, Solans,Carlos, Soledad Molina Gonzalez,Maria, Soos,Csaba, Stewart,Graeme A, Suljic,M., Svhira,Peter, Teixeira,Antonio, Teofili,Lorenzo, Termo,Gennaro, Troska,Jan, Utrobicic,Antonija, Van Rijnbach,Milou, Van Stenis,Miranda, Vasey,Francois, Vaskuri,Anna, Veenhof,Rob, Verzeroli,Mattia, Vicente Leitao,Pedro, Vicente,M., Volkl,Valentin, Wanotayaroj,Chaowaroj, Weick,Julian, Wiehe,Moritz, and Zaborowska,Anna
- Abstract
This report summarises the activities and main achievements of the CERN strategic R&D programme on technologies for future experiments during the year 2021.
- Published
- 2022
33. Towards a responsive CernVM-FS architecture
- Author
-
Popescu Radu, Blomer Jakob, and Ganis Gerardo
- Subjects
Physics ,QC1-999 - Abstract
The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service implemented as a POSIX read-only filesystem in user space (FUSE). It was originally developed at CERN to assist High Energy Physics (HEP) collaborations in deploying software on the worldwide distributed computing infrastructure for data processing applications. Files are stored remotely as content-addressed blocks on standard web servers and are retrieved and cached on-demand through outgoing HTTP connections only. Repository metadata is recorded in SQLite catalogs, which represent implicit Merkle treeencodings of the repository state. For writing, CernVM-FS follows a publish-subscribe pattern with a single source of new content that is propagated to a large number of readers. This paper focuses on the work to move the CernVM-FS architecturein the direction of a responsive data distribution system. A new distributed publication backend allows scaling out large publication tasks across multiple machines, reducing the time to publish. For the faster propagation of new published content, the addition of a notification system allows clients to subscribe to messages about changes in the repository and to request new root catalogs as soon as they become available. These devel-opments make CernVM-FS more responsive and are particularly relevant for use cases where a short propagation delay from repository down to individual clients is important, such as using CernVM-FS as an AFS replacement for distributing software stacks. Additionally, they permit the implementation of more complex workflows, with producer-consumer pipelines, as for example in the ALICE analysis trains system.
- Published
- 2019
- Full Text
- View/download PDF
34. Towards a serverless CernVM-FS
- Author
-
Blomer Jakob, Ganis Gerardo, Mosciatti Simone, and Popescu Radu
- Subjects
Physics ,QC1-999 - Abstract
The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution and—to some extent—a data distribution service. It gives POSIX access to more than a billion binary files of experiment application software stacks and operating system containers to end user devices, grids, clouds, and supercomputers. Increasingly, CernVM-FSalso provides access to certain classes of data, such as detector conditions data, genomics reference sets, or gravitational wave detector experiment data. For most of the high- energy physics experiments, an underlying HTTP content distribution infrastructure is jointly provided by universities and research institutes around the world. In this contribution, we will present recent developments and future plans. For future developments, we put a focus on evolving the content distribution infrastructure and at lowering the barrier for publishing into CernVM-FS. Through so-called serverless computing, we envision cloud hosted CernVM-FS repositories without the need to operate dedicated servers or virtual machines. An S3 compatible service in conjunction with a content delivery network takes on data provisioning, replication, and caching. A chainof time-limited and resource-limited functions (so called “lambda function” or “function-as- a-service”) operate on the repository and stage the updates. As a result, any CernVM-FS client should be able to turn intoawriter, possession of suitable keys provided. For repository owners, we aim at providing cost transparency and seamless scalability from very small to very large CernVM-FS installations.
- Published
- 2019
- Full Text
- View/download PDF
35. The Open High Throughput Computing Content Delivery Network
- Author
-
Dykstra Dave, Bockelman Brian, Blomer Jakob, and Field Laurence
- Subjects
Physics ,QC1-999 - Abstract
LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases where experiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.
- Published
- 2019
- Full Text
- View/download PDF
36. LLAMA: The low‐level abstraction for memory access.
- Author
-
Gruber, Bernhard Manfred, Amadio, Guilherme, Blomer, Jakob, Matthes, Alexander, Widera, René, and Bussmann, Michael
- Subjects
DATA structures ,MEMORY ,C++ - Abstract
The performance gap between CPU and memory widens continuously. Choosing the best memory layout for each hardware architecture is increasingly important as more and more programs become memory bound. For portable codes that run across heterogeneous hardware architectures, the choice of the memory layout for data structures is ideally decoupled from the rest of a program. This can be accomplished via a zero‐runtime‐overhead abstraction layer, underneath which memory layouts can be freely exchanged. We present the low‐level abstraction of memory access (LLAMA), a C++ library that provides such a data structure abstraction layer with example implementations for multidimensional arrays of nested, structured data. LLAMA provides fully C++ compliant methods for defining and switching custom memory layouts for user‐defined data types. The library is extensible with third‐party allocators. Providing two close‐to‐life examples, we show that the LLAMA‐generated array of structs and struct of arrays layouts produce identical code with the same performance characteristics as manually written data structures. Integrations into the SPEC CPU® lbm benchmark and the particle‐in‐cell simulation PIConGPU demonstrate LLAMA's abilities in real‐world applications. LLAMA's layout‐aware copy routines can significantly speed up transfer and reshuffling of data between layouts compared with naive element‐wise copying. LLAMA provides a novel tool for the development of high‐performance C++ applications in a heterogeneous environment. [ABSTRACT FROM AUTHOR]
- Published
- 2023
- Full Text
- View/download PDF
37. LLAMA: The low‐level abstraction for memory access
- Author
-
Gruber, Bernhard Manfred, primary, Amadio, Guilherme, additional, Blomer, Jakob, additional, Matthes, Alexander, additional, Widera, René, additional, and Bussmann, Michael, additional
- Published
- 2022
- Full Text
- View/download PDF
38. Increasing the Execution Speed of Containerized Analysis Workflows Using an Image Snapshotter in Combination With CVMFS
- Author
-
Mosciatti, Simone, primary, Lange, Clemens, additional, and Blomer, Jakob, additional
- Published
- 2021
- Full Text
- View/download PDF
39. Upcoming Storage Features in ROOT [Slides]
- Author
-
Canal, Philippe, primary and Blomer, Jakob, additional
- Published
- 2021
- Full Text
- View/download PDF
40. Exploring Object Stores for High-Energy Physics Data Storage
- Author
-
López-Gómez, Javier, primary and Blomer, Jakob, additional
- Published
- 2021
- Full Text
- View/download PDF
41. Strategic R&D Programme on Technologies for Future Experiments - Annual Report 2020
- Author
-
Aglieri, Gianluca, Aleksa, Martin, Feito, Diego Alvarez, Angeletti, Massimo, Antoszczuk, Pablo, Ballabriga, Rafael, Balutto, Mattia, Higueras, Maria Barba, Barney, David, Baron, Sophie, Pinto, Mateus Barreto, Van Beelen, Jacob Bastian, Baszczyk, Mateusz Karol, Beguin, Marina, Biereigel, Stefan, Blomer, Jakob, De Souza Mendes, Eduardo Brandao, Brondolin, Erica, Brunbauer, Florian, Buschmann, Eric, Buytaert, Jan, Cala', Roberto, Campbell, Michael, Catinaccio, Andrea, Cecconi, Leonardo, Ceresa, Davide, Lemos, Edgar Cid, Coco, Victor, Collins, Paula, Corbetta, Mara, Cure, Benoit, Dachs, Florian, D'Ambrosio, Carmelo, Dannheim, Dominik, De Melo, Joao, De Oliveira, Rui, Deng, Wenjing, Detraz, Stephane, Dias, Matheo, Dobrijevic, Dominik, Martin, Ana Dorda, Dort, Katharina, Dumps, Raphael, Faccio, Federico, Declara, Placido Fernandez, Fiorenza, Gabriele, Forty, Roger, Francois, Brieuc, Bulling, Anthony Frederick, Gargiulo, Corrado, Gessinger-Befurt, Paul, Gkougkousis, Vagelis, Gose, Melwin, Guida, Roberto, Hahnfeld, Jonas, Haimberger, Jakob, Halvorsen, Marius, Hong, Geun Hee, Hellenschmidt, Desiree, Ijzermans, Pieter, Janot, Patrick, Janssens, Djunes, Williams, Morag Jean, Joram, Christian, Kiehn, Moritz, Klekotko, Adam, Kloukinas, Kostas, Kluge, Alex, Krammer, Manfred, Kroeger, Jens, Lafuente, Antonio, Lalovic, Milana, Laudi, Elisa, Ledey, Gael, Linssen, Lucie, Lisowska, Marta, Gomez, Javier Lopez, Malinowski, Filip Maciej, Magatti, Demetrio, Mandelli, Beatrice, Martinazzoli, Loris, Vila, Pere Mato, Maulerova, Vendula, Mentink, Matthias, Gonzalez, Maria Molina, Moll, Michael, Muller, Hans, Munker, Magdalena, Musa, Luciano, Neroni, Michela, Olantera, Lauri, Oliveri, Eraldo, Padulano, Vincenzo, Pejasinovic, Risto, Pernegger, Heinz, Petagna, Paolo, Piro, Francesco, Pizzirusso, Olivier, Prousalidi, Thenia, Quast, Thorben, Rembser, Christoph, Riedler, Petra, Riegler, Werner, Rigoletti, Gianluca, Rivera, Esteban Curras, Moreira, Paulo Rodrigues Simoes, Ropelewski, Leszek, Salamani, Dalila, Salomoni, Matteo, Scarcella, Carmelo, Scharenberg, Lucian, Schindler, Heinrich, Schmidt, Burkhard, Schneider, Thomas, Schopper, Andreas, Secouet, Pascal, Singh, Shuvay, Snoeys, Walter, Soos, Csaba, Stewart, Graeme, Teixeira, Antonio, Teofili, Lorenzo, Termo, Gennaro, Troska, Jan, Utrobicic, Antonija, Van Stenis, Miranda, Vanat, Tomas, Vasey, Francois, Veenhof, Rob, Leitao, Pedro Vicente, Vicente, Mateus, Volkl, Valentin, Wanotayaroj, Chaowaroj, Wiehe, Moritz, Wunsch, Stefan, Auffray-Hillemanns, Etiennette, Borghello, Giulio, Michelis, Stefano, Pantaleo, Felice, Pizzichemi, Marco, Ripamonti, Giacomo, Roloff, Philipp, Rovere, Marco, Sailer, André, and Salzburger, Andreas
- Abstract
This report summarises the activities and achievements of the strategic R&D programme on technologies for future experiments in the year 2020.
- Published
- 2020
42. Solving the Container Explosion Problem for Distributed High Throughput Computing
- Author
-
Shaffer, Tim, primary, Hazekamp, Nicholas, additional, Blomer, Jakob, additional, and Thain, Douglas, additional
- Published
- 2020
- Full Text
- View/download PDF
43. CernVM-FS Container Image Integration
- Author
-
Mosciatti, Simone, primary, Blomer, Jakob, additional, Ganis, Gerardo, additional, and Popescu, Radu, additional
- Published
- 2020
- Full Text
- View/download PDF
44. Evolution of the ROOT Tree I/O
- Author
-
Blomer, Jakob, primary, Canal, Philippe, additional, Naumann, Axel, additional, and Piparo, Danilo, additional
- Published
- 2020
- Full Text
- View/download PDF
45. Strategic R&D Programme on Technologies for Future Experiments
- Author
-
Aleksa, Martin, Blomer, Jakob, Cure, Benoit, Campbell, Michael, D'Ambrosio, Carmelo, Dannheim, Dominik, Doser, Michael, Faccio, Federico, Farthouat, Philippe, Gargiulo, Corrado, Janot, Patrick, Joram, Christian, Krammer, Manfred, Linssen, Lucie, Mato Vila, Pere, Rodrigues Simoes Moreira, Paulo, Musa, Luciano, Oliveri, Eraldo, Onnela, Antti, Pernegger, Heinz, Riedler, Petra, Rembser, Christoph, Stewart, Graeme, Ten Kate, Herman, and Vasey, Francois
- Subjects
Detectors and Experimental Techniques - Abstract
Instrumentation is a key ingredient for progress in experimental high energy physics. The Experimental Physics Department of CERN has defined a strategic R&D (Research and Development) programme on technologies for future experiments. Provided the required resources can be made available it will start in 2020 and initially extend over five years. The selection of topics and the established work plans are the result of a transparent and open process, which lasted 14 months and involved several hundred of physicists and engineers at CERN and in the broader HEP community.
- Published
- 2018
46. A fully unprivileged CernVM-FS.
- Author
-
Doglioni, C., Kim, D., Stewart, G.A., Silvestris, L., Jackson, P., Kamleh, W., Blomer, Jakob, Dykstra, Dave, Ganis, Gerardo, Mosciatti, Simone, and Priessnitz, Jan
- Subjects
NUCLEAR physics experiments ,SUPERCOMPUTERS ,LINUX operating systems ,DATA integration ,COMPUTER software packaging - Abstract
The CernVM File System provides the software and container distribution backbone for most High Energy and Nuclear Physics experiments. It is implemented as a file system in user-space (Fuse) module, which permits its execution without any elevated privileges. Yet, mounting the file system in the first place is handled by a privileged suid helper program that is installed by the Fuse package on most systems. The privileged nature of the mount system call is a serious hindrance to running CernVM-FS on opportunistic resource and supercomputers. Fortunately, recent developments in the Linux kernel and in the Fuse user-space libraries enabled fully unprivileged mounting for Fuse file systems (as of RHEL 8), or at least outsourcing the privileged mount system call to a custom, external process. This opens the door to several, very appealing new ways to use CernVM-FS, such as a generally usable "super pilot" consisting of the pilot code bundled with Singularity and CernVM-FS, or the on-demand instantiation of unprivileged, ephemeral containers to publish new CernVM-FS content from anywhere. In this contribution, we discuss the integration of these new Linux features with CernVM-FS and show some of its most promising, new applications. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF
47. Evolution of the ROOT Tree I/O.
- Author
-
Doglioni, C., Kim, D., Stewart, G.A., Silvestris, L., Jackson, P., Kamleh, W., Blomer, Jakob, Canal, Philippe, Naumann, Axel, and Piparo, Danilo
- Subjects
NUCLEAR physics ,COMPUTER input-output equipment ,C++ ,RANDOM access memory ,DATA analysis - Abstract
The ROOT TTree data format encodes hundreds of petabytes of High Energy and Nuclear Physics events. Its columnar layout drives rapid analyses, as only those parts ("branches") that are really used in a given analysis need to be read from storage. Its unique feature is the seamless C++ integration, which allows users to directly store their event classes without explicitly defining data schemas. In this contribution, we present the status and plans of the future ROOT 7 event I/O. Along with the ROOT 7 interface modernization, we aim for robust, where possible compile-time safe C++ interfaces to read and write event data. On the performance side, we show first benchmarks using ROOT's new experimental I/O subsystem that combines the best of TTrees with recent advances in columnar data formats. A core ingredient is a strong separation of the high-level logical data layout (C++ classes) from the low-level physical data layout (storage backed nested vectors of simple types). We show how the new, optimized physical data layout speeds up serialization and deserialization and facilitates parallel, vectorized and bulk operations. This lets ROOT I/O run optimally on the upcoming ultra-fast NVRAM storage devices, as well as file-less storage systems such as object stores. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF
48. CERN Services for Long Term Data Preservation: Paper - iPRES 2016 - Swiss National Library, Bern
- Author
-
Berghaus, Frank, Simko, Tibor, Shiers, Jamie, Ganis, Gerardo, Dallmeier Tiessen, Sünje, Melia, Germán Cancio, and Blomer, Jakob
- Abstract
In this paper we describe the services that are offered by CERN [3] for Long Term preservation of High Energy Physics (HEP) data, with the Large Hadron Collider (LHC) as a key use case. Data preservation is a strategic goal for European High Energy Physics (HEP) [9], as well as for the HEP community worldwide and we position our work in this global content. Specifically, we target the preservation of the scientific data, together with the software, documentation and computing environment needed to process, (re-)analyse or otherwise (re-)use the data. The target data volumes range from hundreds of petabytes (PB – 1015 bytes) to hundreds of exabytes (EB – 1018 bytes) for a target duration of several decades. The Use Cases driving data preservation are presented together with metrics that allow us to measure how close we are to meeting our goals, including the possibility for formal certification for at least part of this work. Almost all of the services that we describe are fully generic – the exception being Analysis Preservation that has some domain-specific aspects (where the basic technology could nonetheless be adapted).
- Published
- 2017
49. New directions in the CernVM file system
- Author
-
Blomer, Jakob, primary, Buncic, Predrag, additional, Ganis, Gerardo, additional, Hardi, Nikola, additional, Meusel, Rene, additional, and Popescu, Radu, additional
- Published
- 2017
- Full Text
- View/download PDF
50. Accessing Data Federations with CVMFS
- Author
-
Weitzel, Derek, primary, Bockelman, Brian, additional, Dykstra, Dave, additional, Blomer, Jakob, additional, and Meusel, Ren, additional
- Published
- 2017
- Full Text
- View/download PDF
Catalog
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.