On the Optimization of Iterative Programming with Distributed Data Collections - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

On the Optimization of Iterative Programming with Distributed Data Collections

Nils Gesbert
  • Fonction : Auteur
  • PersonId : 958488
Pierre Genevès
Nabil Layaïda

Résumé

Big data programming frameworks are becoming increasingly important for the development of applications, for which performance and scalability are critical. In those complex frameworks, optimizing codeby hand is hard and time-consuming, making automated optimization particularly necessary. In order to automate optimization, a prerequisite isto find suitable abstractions to represent programs; for instance, algebras based on monads or monoids to represent distributed data collections.Currently, however, such algebras do not represent recursive programs in a way which allows analyzing or rewriting them. In this paper, we extend a monoid algebra with a fixpoint operator for representing recursion as a first class citizen and show how it allows new optimizations. Experiments with the Spark platform illustrate performance gains brought by these systematic optimizations.
Fichier principal
Vignette du fichier
paper.pdf (915.14 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02066649 , version 1 (13-03-2019)
hal-02066649 , version 2 (16-10-2020)
hal-02066649 , version 3 (16-10-2020)
hal-02066649 , version 4 (16-10-2020)
hal-02066649 , version 5 (02-03-2021)
hal-02066649 , version 6 (24-05-2022)

Identifiants

  • HAL Id : hal-02066649 , version 4

Citer

Sarah Chlyah, Nils Gesbert, Pierre Genevès, Nabil Layaïda. On the Optimization of Iterative Programming with Distributed Data Collections. 2020. ⟨hal-02066649v4⟩
500 Consultations
572 Téléchargements

Partager

Gmail Facebook X LinkedIn More