Leveraging HPC resources with distributed RDataFrame
Abstract: The declarative approach to data analysis provides high-level abstractions for users to operate on their datasets in a much more ergonomic fashion compared to imperative interfaces. ROOT offers such a tool with RDataFrame, which has been tested in production environments and used in real-world analyses with optimal results. Its programming model acts by creating a computation graph with the operations issued by the user and executing it lazily only when the final results are queried. It has always been oriented towards parallelisation, with native support for multi-thread execution on a single machine. Recently, RDataFrame has been extended with a Python layer that is capable of steering and executing the RDataFrame computation graph over a set of distributed resources. In addition, such a layer requires minimal code changes for an RDataFrame application to run distributedly. The new tool effectively allows running a C++ event loop based on RDataFrame while leveraging common industry tools like Dask to schedule the usage of resources. This work presents results and insights gathered through the distributed RDataFrame tool running a physics analysis connecting multiple nodes with a Dask scheduler that requests resources from a Slurm cluster
- Location
-
Deutsche Nationalbibliothek Frankfurt am Main
- Extent
-
Online-Ressource
- Language
-
Englisch
- Notes
-
Journal of physics. Conference Series. - 2438, 1 (2023) , 012097, ISSN: 1742-6596
- Event
-
Veröffentlichung
- (where)
-
Freiburg
- (who)
-
Universität
- (when)
-
2024
- Creator
-
Padulano, Vincenzo Eduardo
Kabadzhov, Ivan Donchev
Saavedra, E. T.
Guiraud, Enrico
- DOI
-
10.1088/1742-6596/2438/1/012097
- URN
-
urn:nbn:de:bsz:25-freidok-2537359
- Rights
-
Open Access; Der Zugriff auf das Objekt ist unbeschränkt möglich.
- Last update
-
15.08.2025, 7:30 AM CEST
Data provider
Deutsche Nationalbibliothek. If you have any questions about the object, please contact the data provider.
Associated
- Padulano, Vincenzo Eduardo
- Kabadzhov, Ivan Donchev
- Saavedra, E. T.
- Guiraud, Enrico
- Universität
Time of origin
- 2024
Other Objects (12)
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging semantic resources in diversified query expansion
Leveraging HPC accelerator architectures with modern techniques — hydrologic modeling on GPUs with ParFlow
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
OpenHPC: Community Building Blocks for HPC Systems
Containers in HPC: State of Containers in HPC
Leveraging attestation techniques for trust establishment in distributed systems
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging semantic resources in diversified query expansion
Leveraging HPC accelerator architectures with modern techniques — hydrologic modeling on GPUs with ParFlow
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
OpenHPC: Community Building Blocks for HPC Systems
Containers in HPC: State of Containers in HPC
Leveraging attestation techniques for trust establishment in distributed systems
Distributed HPC Applications with Unprivileged Containers
Leveraging the benefits of integrating and interacting electric vehicles and distributed energy resources
Resource-Elasticity Support for Distributed Memory HPC Applications
Singularity: Portability of containers across diverse HPC resources with Singularity
ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems
Leveraging semantic resources in diversified query expansion
Leveraging HPC accelerator architectures with modern techniques — hydrologic modeling on GPUs with ParFlow
The Magnificent Modular Mahout: An extensible library for distributed math and HPC
Resource-Aware and Elastic Parallel Software Development for Distributed-Memory HPC Systems
OpenHPC: Community Building Blocks for HPC Systems
Containers in HPC: State of Containers in HPC