Search

To achieve a tight integration of instrument control and (online) data analysis, the European XFEL decided in 2011 to develop Karabo*, a custom control and data processing system. Karabo provides control via event-driven communication. Signal/slot and request/reply patterns are implemented via a central message broker. Data pipelines for e.g. scientific workflows or detector calibration are implemented as direct TCP/IP connections. The core entities of Karabo are self-describing devices written in C++ or Python. They represent hardware, orchestrate other devices, or provide system services like data logging and configuration storage. To operate Karabo, a Python command line interface and a generic GUI written in PyQt are provided. Control and data widgets compose Karabo scenes that are provided by devices or are manually customized and stored together with device configurations in a central database. Since 2016, Karabo is used to commission and operate the currently three photon beam lines and six scientific instruments at the European XFEL. This contribution summarizes the status of Karabo, highlights achievements and lessons learned, and gives an outlook for future directions., Proceedings of the 17th International Conference on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2019, New York, NY, USA

We describe the data analysis structure that is integrated into the Karabo framework [1] to support scientific experiments and data analysis at European XFEL GmbH. The photon science experiments have a range of data analysis requirements, including online (i.e. near real-time during the actual measurement) and offline data analysis. The Karabo data analysis framework supports execution of automatic data analysis for routine tasks, supports complex experiment protocols including data analysis feedback integration to instrument control, and supports integration of external applications. The online data analysis is carried out using distributed and accelerator hardware (such as GPUs) where required to balance load and achieve near real-time data analysis throughput. Analysis routines provided by Karabo are implemented in C++ and Python, and make use of established scientific libraries. The XFEL control and analysis software team collaborates with users to integrate experiment specific analysis codes, protocols and requirements into this framework, and to make it available for the experiments and subsequent offline data analysis., Proceedings of the 16th Int. Conf. on Accelerator and Large Experimental Control Systems, ICALEPCS2017, Barcelona, Spain

The European XFEL is a 3.4km long X-ray Free Electron Laser in the final construction and commissioning phase in Hamburg. It will produce 27000 bunches per second at 17.5GeV. Early 2015 a first electron beam was produced in the RF-photo-injector and the commissioning of consecutive sections is following during this and next year. The huge number and variety of devices for the accelerator, beam line, experiment, cryogenic and facility systems pose a challenging control task. Multiple systems, including industrial solutions, must be interfaced to each other. The high number of bunches requires a tight time synchronization (down to picoseconds) and high performance data acquisition systems. Fast feedbacks from front-ends, the DAQs and online analysis system with a seamless integration of controls are essential for the accelerator and the initially 6 experimental end stations. It turns out that the European XFEL will be the first installation exceeding 2500 FPGA components in the MicroTCA form factor and will run one of the largest PROFIBUS networks. Many subsystem prototypes are already successfully in operation. An overview and status of the XFEL control system will be given., Proceedings of the 15th Int. Conf. on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2015, Melbourne, Australia

Proceedings of ICALEPCS 2015 15th International Conference on Accelerator and Large Experimental Physics Control Systems, ICALEPCS 2015, Melbourne, Australia, 17 Oct 2015 - 23 Oct 2015 ; MOA3O02(2015)., The European XFEL is a 3.4 km long X-ray Free Electron Laser in the final construction and commissioning phase in Hamburg. It will produce 27000 bunches per second at 17.5 GeV. Early 2015 a first electron beam was produced in the RF-photo-injector and the commissioning of consecutive sections will follow during this and next year. The huge number and variety of devices for the accelerator, beam line, experiment, cryogenic and facility systems pose a challenging control task. Multiple systems, including industrial solutions, must be interfaced to each other. The large number of bunches requires a tight time synchronization (down to picoseconds) and high performance data acquisition systems. Fast feedbacks from front-ends, the DAQs and online analysis system with a seamless integration of controls are essential for the accelerator and the initially 6 experimental end stations. It turns out that the European XFEL will be the first installation exceeding 2500 FPGA components in the MicroTCA form factor and will run one of the largest PROFIBUS networks. Many subsystem prototypes are already successfully in operation. An overview and status of the XFEL control system will begiven.

The European X-ray Free Electron Laser (Altarelli, 2006) is a high-intensity X-ray light source currently being constructed in Hamburg, Germany, that will provide spatially coherent X-rays in the energy range between 0.25 keV and 25 keV. The machine will deliver a unique time structure, consisting of up to 2700 pulses, with a 4.5 MHz repetition rate, 10 times per second at very high photon fluxes up to 1017 photons/s (Tschentscher, 2012). The LPD (Hart, 2012; Koch, 2013), DSSC (Porro, 2010, 2012; Lutz, 2010) and AGIPD (Graafsma, 2009) detectors are being developed to provide Mpixel imaging capabilities at the aforementioned repetition rates for a dynamic range spanning from single photon sensitivity to 104 –105 photons per pixel. The detectors are optimized for specific energy ranges. A direct consequence of the aforementioned detectors’ characteristics is that they generate raw data volumes unprecedented in photon science, ranging up to 1Mpixel x 640 memory cells x 10 pulse/s x 16 bit, i.e. 12.8 Gbyte/s. On-detector vetoing may not necessarily lower these rates much - a memory cell freed by a vetoed pulse may be used by data from one of the remaining 2700 pulses a train consists of. The PC-layer may reduce this data amount by additional software triggering, but this is not guaranteed. Figure 1 gives an overview of the different data products at European XFEL, as well as their flows and involved user roles, under the assumption that processing takes place within XFEL’s Karabo framework (Heisen, 2013). In addition to the high data rates, the Mpixel detectors’ on-sensor memory-cell and multi-gain-stage architectures necessary for the high dynamic range, pose unique challenges in detector-specific data corrections and calibration (Weidenspointner, 2012; Sztuk-Dambietz, 2013a). These challenges are addressed by providing a dedicated and thoroughly characterized set of test stands, which utilize continuous sources (Fe-55, X-ray tubes) as well as a pulsed setup: PulXar (Sztuk-Dambietz, 2013b), which is designed to produce X-ray pulses of 50-150 ns duration, within a 0.6 ms burst followed by a 99.4 ms gap. The radiation it produces thus closely matches the XFEL pulse structure. Additionally, simulation tools are being developed to assist in detector characterization (Bohlen and Joy, 2013).

The European XFEL beam delivery defines a unique time structure that requires acquiring and processing data in short bursts of up to 2700 images every 100 ms. The 2D pixel detectors being developed produce up to 10 GB/s of 1-Mpixel image data. Efficient handling of this huge data volume requires large network bandwidth and computing capabilities. The architecture of the DAQ system is hierarchical and modular. The DAQ network uses 10 GbE switched links to provide large bandwidth data transport between the front-end interfaces (FEI), data handling PC layer servers, and storage and analysis clusters. Front-end interfaces are required to build images acquired during a burst into pulse ordered image trains and forward them to PC layer farm. The PC layer consists of dedicated high-performance computers for raw data monitoring, processing and filtering, and aggregating data files that are then distributed to on-line storage and data analysis clusters. In this contribution we give an overview of the DAQ system architecture, communication protocols, as well as software stack for data acquisition pre-processing, monitoring, storage and analysis.

The expected very high data rates and volumes at the European XFEL demand an efficient concurrent approach of performing experiments. Data analysis must already start whilst data is still being acquired and initial analysis results must immediately be usable to re-adjust the current experiment setup. We have developed a software framework, called Karabo, which allows such a tight integration of these tasks. Karabo is in essence a pluggable, distributed application management system. All Karabo applications (called “Devices”) have a standardized API for self-description/configuration, program-flow organization (state machine), logging and communication. Central services exist for user management, access control, data logging, configuration management etc. The design provides a very scalable but still maintainable system that at the same time can act as a fully-fledged control or a highly parallel distributed scientific workflow system. It allows simple integration and adaption to changing control requirements and the addition of new scientific analysis algorithms, making them automatically and immediately available to experimentalists.

The new European X-ray Free-Electron Laser is the first X-ray free-electron laser capable of delivering X-ray pulses with a megahertz inter-pulse spacing, more than four orders of magnitude higher than previously possible. However, to date, it has been unclear whether it would indeed be possible to measure high-quality diffraction data at megahertz pulse repetition rates. Here, we show that high-quality structures can indeed be obtained using currently available operating conditions at the European XFEL. We present two complete data sets, one from the well-known model system lysozyme and the other from a so far unknown complex of a ��-lactamase from K. pneumoniae involved in antibiotic resistance. This result opens up megahertz serial femtosecond crystallography (SFX) as a tool for reliable structure determination, substrate screening and the efficient measurement of the evolution and dynamics of molecular structures using megahertz repetition rate pulses available at this new class of X-ray laser source.

Nature Communications 9(1), 4025 (2018). doi:doi.org/10.1038/s41467-018-06156-7, The new European X-ray Free-Electron Laser is the first X-ray free-electron laser capable of delivering X-ray pulses with a megahertz inter-pulse spacing, more than four orders of magnitude higher than previously possible. However, to date, it has been unclear whether it would indeed be possible to measure high-quality diffraction data at megahertz pulse repetition rates. Here, we show that high-quality structures can indeed be obtained using currently available operating conditions at the European XFEL. We present two complete data sets, one from the well-known model system lysozyme and the other from a so far unknown complex of a β-lactamase from K. pneumoniae involved in antibiotic resistance. This result opens up megahertz serial femtosecond crystallography (SFX) as a tool for reliable structure determination, substrate screening and the efficient measurement of the evolution and dynamics of molecular structures using megahertz repetition rate pulses available at this new class of X-ray laser source., Published by Nature Publishing Group, London

Nature Communications 9(1), 4025 (2018). doi:10.1038/s41467-018-06156-7, The new European X-ray Free-Electron Laser is the first X-ray free-electron laser capable of delivering X-ray pulses with a megahertz inter-pulse spacing, more than four orders of magnitude higher than previously possible. However, to date, it has been unclear whether it would indeed be possible to measure high-quality diffraction data at megahertz pulse repetition rates. Here, we show that high-quality structures can indeed be obtained using currently available operating conditions at the European XFEL. We present two complete data sets, one from the well-known model system lysozyme and the other from a so far unknown complex of a β-lactamase from K. pneumoniae involved in antibiotic resistance. This result opens up megahertz serial femtosecond crystallography (SFX) as a tool for reli- able structure determination, substrate screening and the efficient measurement of the evolution and dynamics of molecular structures using megahertz repetition rate pulses available at this new class of X-ray laser source., Published by Nature Publishing Group, London