Insite Pipeline - A Pipeline Enabling In-Transit Processing for Arbor, NEST and TVB

This post was handed-in for the HBP Summit 2023. INTRODUCTION/MOTIVATION Simulation of neuronal networks has steadily advanced and now allows for larger and more complex models.However, scaling simulations to such sizes comes with issues and challenges.Especially the amount of data produced, as well as the runtime of the simulation can be demanding.Often, it is not even possible to store all data on disk, and users might have to wait for a long time until they can processthe data.A standard solution in simulation science is to use in-transit approaches [8].In-transit implementations allow users to access data while the simulation is still running and do processing in parallel outside of the simulation.This allows for early insights into the results, early stopping of simulations that are not promising, or even steering of the simulations.Existing in-transit solutions, however, are often complex to integrate into the workflow as they rely on integration into simulators and often use data formats that are complex to handle. This is constraining in the context of multi-disciplinary research conducted in the HBP as such an important feature should be accessible to all users.Especially domain scientists from neuroscience and visualization providers should be able to leverage in-transit processing.To remedy this, we developed Insite [1,2], a pipeline that allows easy in-transit access to simulation data of multiscale simulations conducted with TVB [6], NEST [4], and Arbor[5]. METHODS Insite is designed around providing users with an interface that is easy to integrate into existing workflows and tools.Two achieve this, Insite uses a modular and tiered architecture consisting of simulator modules and a central access node.The simulator modules are provided for TVB, NEST, and Arbor and are responsible for collecting the raw data from the simulation and providing them to the access node.Simulator modules are designed to be as unintrusive as possible and easy for all users to integrate into existing simulation scripts.The access node acts as a single point of contact for users and provides the data of all simulators and simulator instances from a single source.Insite allows access to the data via two paradigms.A push-oriented paradigm via WebSockets where the user gets new data pushed into their application whenever new data is available.Secondly, users can use a pull-based approach based on an HTTP REST API to query data on demand.The data returned in both cases can either be encoded as JSON or flatbuffers.JSON offers a human-readable representation with broad support.Flatbuffer's binary encoding provides a more performant alternative to JSON.By offering these standard protocols and data formats, users can easily use the Insite Pipeline in various programming languages and technologies, as plenty of libraries for this are available.The API provides a variety of parameters that can be used to filter the data making accessing of the data as easy as possible.Thus, making it easy for users to access the data for further processing like analysis or visualization. RESULTS AND DISCUSSION With a focus on ease-of-integration and ease-of-use, Insite is accessible to many developers and users.The Pipeline was successfully integrated into the ViSimpl Visualization Tool [3] and NEST Desktop[7] to add in-transit capabilities,allowing users to get early feedback on their simulations.Due to the design of Insite, both tools could integrate it into their desktop- and web applications.Especially considering the emerging eco-system of web-based solutions in the context of EBRAINS, Insite provides a big advantage over classical in-situ/in-transit approaches. The modular architecture additionally allows extending the access node to provide more data, e.g., metrics or pre-processed data from other sources. Thus, Insite enriches the capabilities of the computational neuroscience community. ACKNOWLEDGEMENTS We would like to thank Simon Oehrl, Jan Müller and Ali C. Demiralp for his contribution to the design and implementation of Insite. This project/research has received funding from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 945539 (Human Brain Project SGA3) and Specific Grant Agreement No. 785907 (Human Brain Project SGA2). REFERENCES [1] Krüger, M.et al.(2022). Insite: A Pipeline Enabling In-Transit Visualization andAnalysis forNeuronal Network Simulations. In: Anzt, H., Bienz, A., Luszczek, P., Baboulin, M. (eds) High Performance Computing. ISC High Performance 2022 International Workshops. ISC High Performance 2022. Lecture Notes in Computer Science, vol 13387. Springer, Cham.https://doi.org/10.1007/978-3-031-23220-6_20 [2] Oehrl, S.et al.(2018). Streaming Live Neuronal Simulation Data into Visualization and Analysis. In: Yokota, R., Weiland, M., Shalf, J., Alam, S. (eds) High Performance Computing. ISC High Performance 2018. Lecture Notes in Computer Science(), vol 11203. Springer, Cham.https://doi.org/10.1007/978-3-030-02465-9_18 [3] Galindo, S. E., Toharia, P., Robles, O. D., & Pastor, L. (2016). ViSimpl: multi-view visual analysis of brain simulation data.Frontiers in Neuroinformatics,10, 44. [4] Marc-Oliver Gewaltig, & Markus Diesmann (2007).NEST (NEural Simulation Tool). Scholarpedia, 2(4), 1430. [5] Nora Abi Akar, John Biddiscombe, Benjamin Cumming, Marko Kabic, Vasileios Karakasis, Wouter Klijn, Anne Küsters, Alexander Peyser, Stuart Yates, Thorsten Hater, Brent Huisman, Espen Hagen, Robin De Schepper, Charl Linssen, Harmen Stoppels, Sebastian Schmitt, Felix Huber, Max Engelen, Fabian Bösch, … Lennart Landsmeer. (2022). Arbor Library v0.8.1 (v0.8.1). Zenodo.https://doi.org/10.5281/zenodo.7473671 [6] P. Sanz Leon, S. A. Knock, M. Woodman, L. Domide, J. Mersmann, A. R. McIntosh, V. Jirsa The Virtual Brain: a simulator of primate brain network dynamics Frontiers in Neuroinformatics 7:10. doi: 10.3389/fninf.2013.00010 [7] Jens Buchertseifer, Sebastian Spreizer, & Benjamin Weyers. (2022). NEST Desktop (v3.1.1). Zenodo.https://doi.org/10.5281/zenodo.6320318 [8]Bennett, J. C., Abbasi, H., Bremer, P. T., Grout, R., Gyulassy, A., Jin, T., ...& Chen, J. (2012, November). Combining in-situ and in-transit processing to enable extreme-scale scientific analysis. InSC'12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis(pp. 1-9). IEEE. DOI:10.1109/SC.2012.31 &nbsp