Search

The token-based certification method is spreading in the distributed computing system of high energy physics. More and more software and middleware are supporting tokens as one of the certification methods. As an example, WLCG has upgraded all the services to support WLCG tokens [1]. In IHEP (Institute of High Energy Physics in China), the Kerberos [2] token has been used as the main certification method in the local cluster. Naturally, it is selected as the certification method in the distributed computing system. In this case, a set of toolkits were developed or introduced to use Kerberos tokens in the distributed computing system, including token producer, token repository, token transfer and token client engine. The token producer is responsible for creating a token and publishing the token file to the token repository. The token repository stores all the latest token files and a refresh service periodically renews the lifetime of those tokens stored in the token repository. The token transfer brings the token file to the worker node. The token client engine initializes the token environment and renews the token's lifetime on the worker node. With these toolkits, the jobs can run in any worker node in any site and use the Kerberos token to access other services, such as EOS [3] and the XRootd [4] proxy service. In IHEP, the Kerberos toolkit has been deployed in the distributed computing system. Currently, three experiments (LHAASO [5], BES [6] and HERD [7]) are using Kerberos tokens to remotely access the data in EOS or Lustre [8]. [ABSTRACT FROM AUTHOR]

China's High-Energy Photon Source (HEPS), the first national highenergy synchrotron radiation light source, is under design and construction. In the future, at the first stage of HEPS, it is predicted that 24PB raw experimental data will be produced per month from 14 beamlines. Faced with such a huge scale of scientific data and diverse data analysis environments in light source disciplines, the HEPS scientific computing platform was designed and implemented based on container mirroring and dynamic orchestration technology to provide HEPS users with a data analysis environment. In this article, a data security access strategy is designed and evaluated for a scientific computing platform to ensure the security and efficiency of data access for users in the entire process of data analysis. First, the general situation of HEPS is introduced. Second, the challenges faced by the HEPS scientific computing system. Third, the architecture and service process of the scientific computing platform are described from the perspective of IT, some key technical implementations will be introduced in detail. Finally, the application effect of data access security policies on computing platforms will be demonstrated. [ABSTRACT FROM AUTHOR]

HTCondor has been widely adopted by HEP clusters to provide high-level scheduling performance. Unlike other schedulers, HTCondor provides loose management of the worker nodes. We developed a maintenance automation tool called "HTCondor MAT" that focuses on dynamic resource management and automatic error handling. A central database records all worker node information, which is sent to the worker node for the startd configuration. If an error happens for the worker node, the node information stored in the database is updated and the worker node is reconfigured with the new node information. The new configuration stops the startd from accepting error-related jobs until the worker node recovers. The MAT has been deployed in the IHEP HTC cluster to provide a central way to manage the worker nodes and remove the impacts of errors on the worker nodes automatically. [ABSTRACT FROM AUTHOR]

This paper introduces a generic and scalable anomaly detection framework. Anomaly detection can improve operation and maintenance efficiency and assure experiments can be carried out effectively. The framework facilitates common tasks such as data sample building, retagging and visualization, deviation measurement and performance measurement for machine learning-based anomaly detection methods. The samples we used are sourced from Ganglia monitoring data. There are several anomaly detection methods to handle spatial and temporal anomalies within the framework. Finally, we show the rudimental application of the framework on Lustre distributed file systems in daily operation and maintenance. [ABSTRACT FROM AUTHOR]

In this paper we will describe the LHAASO distributed computing system based on virtualization and cloud computing technologies. Particularly, we discuss the key points of integrating distributed resources. A solution of integrating cross-domain resources is proposed, which adopt the Openstack+HTCondor to make the distributed resources work as a whole resource pool. A flexible resource scheduling strategy and a job scheduling policy are presented to realize the resource expansion on demand and the efficient job scheduling to remote sites transparently, so as to improve the overall resource utilization. We will also introduce the deployment of the computing system located in Daocheng, the LHAASO observation base using cloud-based architecture, which greatly helps to reduce the operation and maintenance cost as well as to make sure the system availability and stability. Finally, we will show running status of the system. [ABSTRACT FROM AUTHOR]

A typical HEP Computing Center normally runs at least one batch system. As an example, at IHEP (Institute of High Energy Physics, Chinese Academy of Sciences), we've used three batch systems: PBS, HTCondor and SLURM. After running PBS as a local batch system for 10 years, we replaced it by HTCondor (for HTC) and SLURM (for HPC). During that period, problems came up on both user and admin sides. Introduction of the new batch systems implies necessity for users to acquire additional knowledge specific for every batch system, in particular, batch commands. In some cases, users have to use both HTCondor and SLURM in parallel. Furthermore, HTCondor and SLURM provide more functionality, which means more complicated usage mode, compared to the simple PBS commands. On admin side, HTCondor gives more freedom to users, which brings an additional challenge to site administrators. Site administrators have to find the solutions for many problems: preventing users from requesting the resources they are not allowed to use, checking if the required attributes are correct, deciding where requested resources are located (SLURM cluster, the cluster of the virtual machines, the remote sites, etc). To meet the above requirements, HepJob was designed and developed. HepJob provides a set of simple user commands, for example: hep_sub, hep_q, hep_rm, etc. In the submission process, HepJob checks all the attributes and ensures all attributes are correct; assigns proper resources to users (the user and group info is obtained from the management database); routes jobs to the target site; performs other steps as required. Users can start with HepJob very easily and administrators can take the necessary management actions in HepJob. [ABSTRACT FROM AUTHOR]

To improve hardware utilization and save manpower in system maintenance, most of the web services in IHEP have been migrated to a private cloud build upon OpenStack. However, cyber security attacks becomes a serious threats to the cloud progressively. Therefore, a cyber security detection and monitoring system is deployed for this cloud platform. This system collects various security related logs as data sources, and processes them in a framework composed of open source data store, analysis and visualization tools. With this system, security incidents and events can be handled in time and rapid response can be taken to protect cloud platform against cyber security threats. [ABSTRACT FROM AUTHOR]

At IHEP (Institute of High Energy Physics, Chinese Academy of Sciences), computing resources are contributed by different experiments including BES, JUNO, DYW, HXMT, etc. The resources were divided into different partitions to satisfy the dedicated experiment data processing requirements. IHEP had a local Torqu&Maui cluster with 50 queues serving for above 10 experiments. The separated resource partitions leaded to imbalance resource load. In a typical situation, BES resource partition was quite busy without free slot but still with lots of jobs in idle, while JUNO resources are free and wasted seriously. After moving resources from Torque&Maui to HTCondor in 2016, job scheduling efficiency has been improved a lot. In order to balance the imbalance resource load, we designed an efficient sharing strategy to improve the overall resourceutilization. We created an unified pool shared by all experiments. For each experiment, resources are divided into two parts: dedicated resource and sharing resource. The slots in dedicated resource only run jobs from its own experiment, and the slots in sharing resource are shared by jobs from all experiments. Default ratio of dedicated resource to sharing resource is 1:4. To maximize the sharing effectiveness, the ratio is dynamically adjusted between 0:5 and 4:1 based on the number of jobs submitted by each experiment. We have developed a central control system to decide how many resources can be allocated to each experiment group. This system is implemented at two sides: server side and client side. A management database is built at server side, which is storing resource, group and experiment information. Once the sharing ratio needs to be adjusted, resource group will be changed and updated into database. The resource group information is published to the server buffer in real-time. The client periodically pulls resource group information from server buffer via https protocol And resource scheduling configuration at client side is changed based on the resource group information. With this method, share ratio can be modified and deployed dynamically. Sharing strategy is implemented with HTCondor. ClassAd mechanism and accounting-group in HTCondor facilitate to utilizethe sharing strategy at IHEP computing cluster. With the sharing strategy, resource usage has been improved dramatically. [ABSTRACT FROM AUTHOR]