Search

Software quality monitoring and analysis are among the most productive topics in software engineering research. Their results may be effectively employed by engineers during software development life cycle. Open source software constitutes a valid test case for the assessment of software characteristics. The data mining approach has been proposed in literature to extract software characteristics from software engineering data. This paper aims at comparing diverse data mining techniques (e.g., derived from machine learning) for developing effective software quality prediction models. To achieve this goal, we tackled various issues, such as the collection of software metrics from open source repositories, the assessment of prediction models to detect software issues and the adoption of statistical methods to evaluate data mining techniques. The results of this study aspire to identify the data mining techniques that perform better amongst all the ones used in this paper for software quality prediction models. [ABSTRACT FROM AUTHOR]

This paper explores the dynamic interplay between cybersecurity, legislation, and SaaS platforms, focusing on the current landscape, potential future threats, and emerging legal challenges. SaaS providers, acting as custodians of considerable amounts of personal data, are significantly impacted by diverse legal frameworks such as the GDPR, CCPA, HIPAA, and PCI-DSS. These frameworks mandate stringent data protection and security measures. The paper further elucidates the sector-specific regulations and discusses the complex issue of cross-border data transfers, underscoring the need for adequate safeguards. This work examines how laws deal with cybersecurity threats and emphasizes the importance of complying with them to reduce these risks. It addresses challenges and conflicts that arise when SaaS and cybersecurity laws intersect, including issues related to data ownership and protection responsibilities. This paper makes suggestions to improve legal protections through international collaboration, adapting laws to keep up with technology, and ensuring transparency in data processing. It also explores potential future threats and legal challenges, such as cyber threats powered by AI, quantum computing, privacy paradoxes, and questions about data sovereignty. Therefore, the paper stresses the necessity of staying ahead of regulatory changes and fostering a comprehensive cybersecurity strategy that blends technology, people, processes, and legislation. [ABSTRACT FROM AUTHOR]

In the past couple of years, we have been actively developing the Dynamic On-Demand Analysis Service (DODAS) as an enabling technology to deploy container-based clusters over hybrid, private or public, Cloud infrastructures with almost zero effort. DODAS is particularly suitable for harvesting opportunistic computing resources; this is why several scientific communities already integrated their computing use cases into DODAS-instantiated clusters, automating the instantiation, management and federation of HTCondor batch systems. The increasing demand, availability and utilization of HPC resources by and for multidisciplinary user communities, often mandates the possibility to transparently integrate, manage and mix HTC and HPC resources. In this paper, we discuss our experience extending and using DODAS to connect HPC and HTC resources in the context of a distributed Italian regional infrastructure involving multiple sites and communities. In this use case, DODAS automatically generates HTCondor batch system on-demand. Moreover it dynamically and transparently federates sites that may also include HPC resources managed by SLURM; DODAS allows user workloads to make opportunistic and automated use of both HPC and HTC resources, thus effectively maximizing and optimizing resource utilization. We also report on our experience of using and federating HTCondor batch systems exploiting the JSON Web Token capabilities introduced in recent HTCondor versions, replacing the traditional X509 certificates in the whole chain of workload authorization. In this respect we also report on how we integrated HTCondor using OAuth with the INDIGO IAM service. [ABSTRACT FROM AUTHOR]

Developing an Open Source Software application is a challenge. Mainly because there are commercial alternatives that have an army of expert developers behind them, experienced supporters and wellestablished business processes in their development and promotion. Nevertheless, web-based applications, that securely handle the users' personal data are an area of freedom and ease of use, features that make such applications very attractive. The "ease-of-use" part is very hard to achieve, for the developers and the end-users. Dependencies change often in OSS packages, so the fear that something breaks is always around the corner. If the application looks attractive, additional user requirements fall like rain. This poses a problem of continuity, maintenance and operational quality of the packages. In this paper and presentation we shall share our experience in building such a tool, using https://cern.ch/slides, as a showcase and a learning exercise. We shall describe what was available, what was missing, how it was put together, how much effort it took, and what was achieved. [ABSTRACT FROM AUTHOR]

HPC systems have seen impressive growth in terms of performance over a period of many years. Soon the next milestone is expected to be reached with the deployment of exascale systems in 2021. In this paper, we provide an overview of the exascale challenges from a computer architecture's perspective and explore technological and other constraints. The analysis of upcoming architectural options and emerging technologies allow for setting expectations for application developers, which will have to cope with heterogeneous architectures, increasingly diverse compute technologies as well as deeper memory and storage hierarchies. Finally, needs resulting from changing science and engineering workflows will be discussed, which need to be addressed by making HPC systems available as part of more open e-infrastructures that provide also other compute and storage services. [ABSTRACT FROM AUTHOR]

The physics software stack of LHCb is based on Gaudi and is comprised of about 20 interdependent projects, managed across multiple GitLab repositories. At present, the continuous integration (CI) system used for regular building and testing of this software is implemented using Jenkins and runs on a cluster of about 300 cores. LHCb CI pipelines are python-based and relatively modern with some degree of modularity, i.e. the separation of test jobs from build jobs. However, these still suffer from obsoleted design choices that prevent improvements to scalability and reporting. In particular, the resource use and speed have not been thoroughly optimized due to the predominant use of the system for nightly builds, where a feedback time of 8 hours is acceptable. We describe recent work on speeding up pipelines by aggressively splitting and parallelizing checkout, build and test jobs and caching their artifacts. The current state of automatic code quality integration, such as coverage reports, is shown. This paper presents how feedback time from change (merge request) submission to build and test reports is reduced from "next day" to a few hours by dedicated on-demand pipelines. Custom GitLab integration allows easy triggering of pipelines, including linked changes to multiple projects, and provides immediate feedback as soon as ready. Reporting includes a comparison to tests on a unique stable reference build, dynamically chosen for every set of changes under testing. This work enables isolated testing of changes that integrates well into the development workflow, leaving nightly testing primarily for integration tests. [ABSTRACT FROM AUTHOR]

The BM@N experiment (Dubna, JINR) within the NICA complex is a running fixed target experiment. This paper mainly discusses the software support of the experimen and presents the status of the development and testing algorithms to be used for the reconstruction of heavy-ion collisions. Algorithms of reconstruction are described and tested with Monte Carlo input demonstrating a reasonable level of quality assurance. First results on particle identification being a good probe to estimate quality of reconstruction are also mentioned. [ABSTRACT FROM AUTHOR]

Creating software releases is one of the more tedious occupations in the life of a software developer. For this purpose we have tried to automate as many of the repetitive tasks involved as possible from getting the commits to running the software. For this simplification we rely in large parts on free collaborative services available around GitHub: issue tracking, code review (GitHub), continuous integration (Travis-CI), static code analysis (coverity). The dependencies and compilers used in the continuous integration are obtained by mounting CVMFS into a docker container. This enables running any desired compiler version (e.g., gcc 6.2, llvm 3.9) or tool (e.g, clang-format, pylint). To create tags for the software package the powerful GitHub API is used. A script was developed that first collates the release notes from the description of each pull request, commits the release notes file, and finally makes a tag. This moves the burden of writing release notesfrom the package maintainer to the individual developer. The deployment of software releases to CVMFS is handled via GitLab-CI. When a tag is made the software is built and automatically deployed. In this paper we will describe the software infrastructure used for the iLCSoft and iLCDirac projects, which are used by CLICdp and the ILC detector collaborations, and give examples of automation which might be useful for others. [ABSTRACT FROM AUTHOR]

Detector simulation has become fundamental to the success of modern high-energy physics (HEP) experiments. For example, the Geant4-based simulation applications developed by the ATLAS and CMS experiments played a major role for them to produce physics measurements of unprecedented quality and precision with faster turnaround, from data taking to journal submission, than any previous hadron collider experiment. The material presented here contains highlights of a recent review on the impact of detector simulation in particle physics collider experiments published in Ref. [1]. It includes examples of applications to detector design and optimization, software development and testing of computing infrastructure, and modeling of physics objects and their kinematics. The cost and economic impact of simulation in the CMS experiment is also presented. A discussion on future detector simulation needs, challenges and potential solutions to address them is included at the end. [ABSTRACT FROM AUTHOR]

In the paper, the main accent is put on development of software to be used in the BM@N experiment. The experiment is considered as a first step towards a realization of fixed target program at the NICA complex. A brief description of software used for reconstruction of track parameters in the inner tracker of the experiment is presented. The alignment procedure in automatic mode was made using the Millepede package fully integrated in the software. The obtained results illustrate a quality assurance of alignment performed using existing experimental data from experimental runs and recent results, including methodological ones from a tracking procedure, which were used for event reconstruction in the inner tracker. The importance of a precise geometry description and a realistic detector response via micro-simulations done for the GEM part of the inner tracker used for processing experimental data is emphasized. Our recent progress in this activity is discussed. [ABSTRACT FROM AUTHOR]

The management of separate memory spaces of CPUs and GPUs brings an additional burden to the development of software for GPUs. To help with this, CUDA unified memory provides a single address space that can be accessed from both CPU and GPU. The automatic data transfer mechanism is based on page faults generated by the memory accesses. This mechanism has a performance cost, that can be with explicit memory prefetch requests. Various hints on the inteded usage of the memory regions can also be given to further improve the performance. The overall effect of unified memory compared to an explicit memory management can depend heavily on the application. In this paper we evaluate the performance impact of CUDA unified memory using the heterogeneous pixel reconstruction code from the CMS experiment as a realistic use case of a GPU-targeting HEP reconstruction software. We also compare the programming model using CUDA unified memory to the explicit management of separate CPU and GPU memory spaces. [ABSTRACT FROM AUTHOR]

The project of Super Charm-Tau (SCT) factory — a high-luminosity electron-positron collider for studying charmed hadrons and tau lepton — is proposed by Budker INP. The project implies single collision point equipped with a universal particle detector. The Aurora software framework has been developed for the SCT detector. It is based on trusted and widely used in high energy physics software packages, such as Gaudi, Geant4, and ROOT. At the same time, new ideas and developments are employed, in particular the Aurora project benefits a lot from the turnkey software for future colliders (Key4HEP) initiative. This paper describes the first release of the Aurora framework, summarizes its core technologies, structure and roadmap for the near future. [ABSTRACT FROM AUTHOR]

Infrastructures supporting distributed scientific collaborations must address competing goals in both providing high performance access to resources while simultaneously securing the infrastructure against security threats. The NetBASILISK project is attempting to improve the security of such infrastructures while not adversely impacting their performance. This paper will present our work to create a benchmark and monitoring infrastructure that allows us to test for any degradation in transferring data into a NetBASILISK protected site. [ABSTRACT FROM AUTHOR]

Particle physics experiments rely extensively on computing and data services, making e-infrastructure an integral part of the research collaboration. Constructing and operating distributed computing can however be challenging for a smaller-scale collaboration. The Light Dark Matter eXperiment (LDMX) is a planned small-scale accelerator-based experiment to search for dark matter in the sub-GeV mass region. Finalizing the design of the detector relies on Monte-Carlo simulation of expected physics processes. A distributed computing pilot project was proposed to better utilize available resources at the collaborating institutes, and to improve scalability and reproducibility. This paper outlines the chosen lightweight distributed solution, presenting requirements, the component integration steps, and the experiences using a pilot system for tests with large-scale simulations. The system leverages existing technologies wherever possible, minimizing the need for software development, and deploys only non-intrusive components at the participating sites. The pilot proved that integrating existing components can dramatically reduce the effort needed to build and operate a distributed e-infrastructure, making it attainable even for smaller research collaborations. [ABSTRACT FROM AUTHOR]

The Cherenkov Telescope Array (CTA) is the next-generation instrument in the very-high energy gamma ray astronomy domain. It will consist of tens of Cherenkov telescopes deployed in 2 arrays at La Palma (Spain) and Paranal (ESO, Chile) respectively. Currently under construction, CTA will start operations around 2023 for a duration of about 30 years. During operations CTA is expected to produce about 2 PB of raw data per year plus 5-20 PB of Monte Carlo data. The global data volume to be managed by the CTA archive, including all versions and copies, is of the order of 100 PB with a smooth growing profile. The associated processing needs are also very high, of the order of hundreds of millions of CPU HS06 hours per year. In order to optimize the instrument design and study its performances, during the preparatory phase (2010-2017) and the current construction phase, the CTA consortium has run massive Monte Carlo productions on the EGI grid infrastructure. In order to handle these productions and the future data processing, we have developed a production system based on the DIRAC framework. The current system is the result of several years of hardware infrastructure upgrades, software development and integration of different services like CVMFS and FTS. In this paper we present the current status of the CTA production system and its exploitation during the latest large-scale Monte Carlo campaigns. [ABSTRACT FROM AUTHOR]

DIRAC and Rucio are two standard pieces of software widely used in the HEP domain. DIRAC provides Workload and Data Management functionalities, among other things, while Rucio is a dedicated, advanced Distributed Data Management system. Many communities that already use DIRAC have expressed their interest in using DIRAC for Workload Management in combination with Rucio for Data Management. In this paper, we describe the integration of the Rucio File Catalog into DIRAC that was initially developed for the Belle II collaboration. [ABSTRACT FROM AUTHOR]

The ATLAS experiment's software production and distribution on the grid benefits from a semi-automated infrastructure that provides up-to-date information about software usability and availability through the CVMFS distribution service for all relevant systems. The software development process uses a Continuous Integration pipeline involving testing, validation, packaging and installation steps. For opportunistic sites that can not access CVMFS, containerized releases are needed. These standalone containers are currently created manually to support Monte-Carlo data production at such sites. In this paper we will describe an automated procedure for the containerization of ATLAS software releases in the existing software development infrastructure, its motivation, integration and testing in the distributed computing system. [ABSTRACT FROM AUTHOR]

In 2016, CERN decided to phase out the legacy OpenAFS storage service due to concerns for the upstream project's longevity, and the potential impact of disorderly service stop on CERN's computing services. Early 2019, the OpenAFS risks of the project collapsing have been reassessed and several early concerns have been allayed. In this paper we recap the work done so far, highlight some of the issues encountered, and present current state and planning. [ABSTRACT FROM AUTHOR]

GlideinWMS is a pilot framework to provide uniform and reliable HTCondor clusters using heterogeneous resources. The Glideins are pilot jobs that are sent to the selected nodes, test them, set them up as desired by the user jobs, and ultimately start an HTCondor schedd to join an elastic pool. These Glideins collect information that is very useful to evaluate the health and efficiency of the worker nodes and invaluable to troubleshoot when something goes wrong. This data, including local stats, the results of all the tests, and the HTCondor log files, is packed and sent to the GlideinWMS Factory. To access this information, developers and troubleshooters must exchange emails with Factory operators and dig manually into files. Furthermore, these files contain also information like email and IP addresses, and user IDs, that we want to protect and limit access to. GlideinMonitor is a Web application to make these logs more accessible and useful: it organizes the logs in an efficient compressed archive; it allows to search, unpack, and inspect them, all in a convenient and secure Web interface; via plugins like the log anonymizer, it can redact protected information preserving the parts useful for troubleshooting. [ABSTRACT FROM AUTHOR]

With the ever-increasing size of scientific collaborations and complexity of scientific instruments, the software needed to acquire, process and analyze the gathered data is increasing in both complexity and size. Unfortunately the role and career path of scientists and engineers working on software R&D and developing scientific software are neither clearly established nor defined in many fields of natural science. In addition, the exchange of information between scientific software development and computer science departments at universities or computing schools is scattered and fragmented into individual initiatives. To address the above issues we propose a new effort on a European level, which concentrates on strengthening the role of software developers in natural sciences, acts as a hub for the exchange of ideas among different stake-holders in computer science and scientific software and forms a lobbying forum for software engineering in natural sciences on an international level. This contribution discusses in detail the motivation, role and interplay with other initiatives of a "Software Institute for Data-Intensive Sciences", which is currently being discussed between research institutes, universities and funding agencies in Europe. In addition to the current status, an outlook on future prospects of this initiative will be given. [ABSTRACT FROM AUTHOR]

The CMS collaboration at the CERN LHC has made more than one petabyte of open data available to the public, including large parts of the data which formed the basis for the discovery of the Higgs boson in 2012. Apart from their scientific value, these data can be used not only for education and outreach, but also for software development. However, in their original format, the data cannot be accessed easily without experiment-specific knowledge and skills. Work is presented that allows to set up open analyses that are performed close to the published ones, but which meet minimum requirements for experiment-specific knowledge and software. The suitability of this approach for education and outreach is demonstrated with analyses that have been made fully accessible to the public via the CERN Open Data portal. Further, the value of these data for software development and as basis for benchmarks of analysis software under realistic conditions of a high-energy physics experiment is discussed. [ABSTRACT FROM AUTHOR]

The diversity of the scientific goals across HEP experiments necessitates unique bodies of software tailored for achieving particular physics results. The challenge, however, is to identify the software that must be unique, and the code that is unnecessarily duplicated, which results in wasted effort and inhibits code maintainability. Fermilab has a history of supporting and developing software projects that are shared among HEP experiments. Fermilab's scientific computing division currently expends effort in maintaining and developing the LArSoft toolkit, used by liquid argon TPC experiments, as well as the event-processing framework technologies used by LArSoft, CMS, DUNE, and the majority of Fermilab-hosted experiments. As computing needs for DUNE and the HL-LHC become clearer, the computing models are being rethought. Presented here are Fermilab's plans for addressing the evolving software landscape as it relates to LArSoft and the data-processing frameworks, specifically as it relates to the DUNE experiment. [ABSTRACT FROM AUTHOR]

In the near future, large scientific collaborations will face unprecedented computing challenges. Processing and storing exabyte datasets require a federated infrastructure of distributed computing resources. The current systems have proven to be mature and capable of meeting the experiment goals, by allowing timely delivery of scientific results. However, a substantial amount of interventions from software developers, shifters and operational teams is needed to efficiently manage such heterogeneous infrastructures. A wealth of operational data can be exploited to increase the level of automation in computing operations by using adequate techniques, such as machine learning (ML), tailored to solve specific problems. The Operational Intelligence project is a joint effort from various WLCG communities aimed at increasing the level of automation in computing operations. We discuss how state-of-the-art technologies can be used to build general solutions to common problems and to reduce the operational cost of the experiment computing infrastructure. [ABSTRACT FROM AUTHOR]

Widespread distributed processing of big datasets has been around for more than a decade now thanks to Hadoop, but only recently higher-level abstractions have been proposed for programmers to easily operate on those datasets, e.g. Spark. ROOT has joined that trend with its RDataFrame tool for declarative analysis, which currently supports local multi-threaded parallelisation. However, RDataFrame's programming model is general enough to accommodate multiple implementations or backends: users could write their code once and execute it as-is locally or distributedly, just by selecting the corresponding backend. This abstract introduces PyRDF, a new python library developed on top of RDataFrame to seamlessly switch from local to distributed environments with no changes in the application code. In addition, PyRDF has been integrated with a service for web-based analysis, SWAN, where users can dynamically plug in new resources, as well as write, execute, monitor and debug distributed applications via an intuitive interface. [ABSTRACT FROM AUTHOR]

Software is an essential component of High Energy Physics experiments. Due to the fact that it is upgraded on relatively short timescales, software provides flexibility, but at the same time is susceptible to issues introduced during development process, thus mandating systematic testing. We present recent improvements to LHCbPR, the framework implemented at LHCb to measure physics and computational performance of complete applications. This infrastructure is essential for keeping track of the optimisation activities related to the upgrade of computing systems which is crucial to meet the requirements of the LHCb detector upgrade for the next stage of data taking of the LHC. Latest developments in LHCbPR include application of messaging system to trigger the tests right after the corresponding software version is built within LHCb nightly builds infrastructure. We will also report on the investigation of using big data technologies in LHCbPR. We have found that using tools such as Apache Spark and Hadoop Distributed File System may significantly improve the functionality of the framework, providing an interactive exploration of the test results with efficient data filtering and flexible development of reports. [ABSTRACT FROM AUTHOR]