Leveraging HPC resources with distributed RDataFrame
Abstract: The declarative approach to data analysis provides high-level abstractions for users to operate on their datasets in a much more ergonomic fashion compared to imperative interfaces. ROOT offers such a tool with RDataFrame, which has been tested in production environments and used in real-world analyses with optimal results. Its programming model acts by creating a computation graph with the operations issued by the user and executing it lazily only when the final results are queried. It has always been oriented towards parallelisation, with native support for multi-thread execution on a single machine. Recently, RDataFrame has been extended with a Python layer that is capable of steering and executing the RDataFrame computation graph over a set of distributed resources. In addition, such a layer requires minimal code changes for an RDataFrame application to run distributedly. The new tool effectively allows running a C++ event loop based on RDataFrame while leveraging common industry tools like Dask to schedule the usage of resources. This work presents results and insights gathered through the distributed RDataFrame tool running a physics analysis connecting multiple nodes with a Dask scheduler that requests resources from a Slurm cluster
- Standort
-
Deutsche Nationalbibliothek Frankfurt am Main
- Umfang
-
Online-Ressource
- Sprache
-
Englisch
- Anmerkungen
-
Journal of physics. Conference Series. - 2438, 1 (2023) , 012097, ISSN: 1742-6596
- Ereignis
-
Veröffentlichung
- (wo)
-
Freiburg
- (wer)
-
Universität
- (wann)
-
2024
- Urheber
-
Padulano, Vincenzo Eduardo
Kabadzhov, Ivan Donchev
Saavedra, E. T.
Guiraud, Enrico
- DOI
-
10.1088/1742-6596/2438/1/012097
- URN
-
urn:nbn:de:bsz:25-freidok-2537359
- Rechteinformation
-
Open Access; Der Zugriff auf das Objekt ist unbeschränkt möglich.
- Letzte Aktualisierung
-
25.03.2025, 13:52 MEZ
Datenpartner
Deutsche Nationalbibliothek. Bei Fragen zum Objekt wenden Sie sich bitte an den Datenpartner.
Beteiligte
- Padulano, Vincenzo Eduardo
- Kabadzhov, Ivan Donchev
- Saavedra, E. T.
- Guiraud, Enrico
- Universität
Entstanden
- 2024
Ähnliche Objekte (12)
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Leveraging attestation techniques for trust establishment in distributed systems
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging skills into independent living : distributed cognition and cognitive disability
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
Bringing distributed energy resources to market
Efficiently exploit HPC resources in scientific analysis and visualization with ParaView
Containers in HPC: State of Containers in HPC
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Leveraging attestation techniques for trust establishment in distributed systems
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging skills into independent living : distributed cognition and cognitive disability
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
Bringing distributed energy resources to market
Efficiently exploit HPC resources in scientific analysis and visualization with ParaView
Containers in HPC: State of Containers in HPC
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Leveraging attestation techniques for trust establishment in distributed systems
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging skills into independent living : distributed cognition and cognitive disability
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
Bringing distributed energy resources to market
Efficiently exploit HPC resources in scientific analysis and visualization with ParaView