Towards Scalable Data Management for Map-Reduce-based Data-Intensive Applications on Cloud and Hybrid Infrastructures

Gabriel Antoniu; Julien Bigot; Christophe Blanchet; Luc Bougé; François Briant; Franck Cappello; Alexandru Costan; Frédéric Desprez; Gilles Fedak; Sylvain Gault; Kate Keahey; Bogdan Nicolae; Christian Pérez; Anthony Simonet; Frédéric Suter; Bing Tang; Raphael Terreux

hal-00684866, version 1

Towards Scalable Data Management for Map-Reduce-based Data-Intensive Applications on Cloud and Hybrid Infrastructures

Gabriel Antoniu (, ) ¹, Julien Bigot ()^d^, 2, Christophe Blanchet ³, Luc Bougé ()^a^, 1, François Briant ⁴, Franck Cappello ^5, 6, 7, Alexandru Costan () ¹, Frédéric Desprez ()^d^, 2, Gilles Fedak ()^d^, 2, Sylvain Gault^d^, 2, Kate Keahey ()^b^, 8, Bogdan Nicolae ()^d^, 5, 6, Christian Pérez ()^d^, 2, Anthony Simonet^d^, 2, Frédéric Suter ()^c^, 9, Bing Tang^d^, 2, Raphael Terreux ³

1st International IBM Cloud Academy Conference - ICA CON 2012 (2012)

Résumé : As Map-Reduce emerges as a leading programming paradigm for data-intensive computing, today's frameworks which support it still have substantial shortcomings that limit its potential scalability. In this paper we discuss several directions where there is room for such progress: they concern storage efficiency under massive data access concurrency, scheduling, volatility and fault-tolerance. We place our discussion in the perspective of the current evolution towards an increasing integration of large-scale distributed platforms (clouds, cloud federations, enterprise desktop grids, etc.). We propose an approach which aims to overcome the current limitations of existing Map-Reduce frameworks, in order to achieve scalable, concurrency-optimized, fault-tolerant Map-Reduce data processing on hybrid infrastructures. This approach will be evaluated with real-life bio-informatics applications on existing Nimbus-powered cloud testbeds interconnected with desktop grids.

a – École normale supérieure de Cachan - ENS Cachan
b – Argonne National Laboratory
c – CNRS
d – INRIA
1 : KerData (INRIA - IRISA)
INRIA – CNRS : UMR6074 – École normale supérieure de Cachan - ENS Cachan – Institut National des Sciences Appliquées (INSA) - Rennes – Université de Rennes 1
2 : AVALON (LIP Lyon / Inria Grenoble Rhône-Alpes)
CNRS : UMR5668 – INRIA – École Normale Supérieure - Lyon – Laboratoire d'informatique du Parallélisme – Université Claude Bernard - Lyon I
3 : Institut de biologie et chimie des protéines [Lyon] (IBCP)
CNRS : UMR5086 – Université Claude Bernard - Lyon I
4 : IBM PSSC Montpellier - Innovation Lab.
IBM PSSC Montpellier
5 : GRAND-LARGE (INRIA Saclay - Ile de France)
INRIA – CNRS : UMR8623 – Université Paris XI - Paris Sud
6 : Joint Laboratory for Petascale Computing [Illinois] (JLPC)
University of Illinois at Urbana-Champaign – INRIA
7 : Laboratoire de Recherche en Informatique (LRI)
CNRS : UMR8623 – Université Paris XI - Paris Sud
8 : Argonne National Laboratory (ANL)
US Department of Energy – University of Chicago
9 : Centre de Calcul de l'inst. national de phy. nucléaire et de phy. des particules (CC IN2P3)
CNRS : USR6402 – IN2P3

hal-00684866, version 1
http://hal.inria.fr/hal-00684866
oai:hal.inria.fr:hal-00684866
Contributeur : Gabriel Antoniu <>
Soumis le : Vendredi 20 Avril 2012, 11:43:30
Dernière modification le : Jeudi 6 Septembre 2012, 16:27:27

Voir la fiche détaillée

Documents associés

PDF :

Exporter

Bibtex EndNote TEI RefWorks

%% hal-00684866, version 1
%% http://hal.inria.fr/hal-00684866
@inproceedings{antoniu:hal-00684866,
    hal_id = {hal-00684866},
    url = {http://hal.inria.fr/hal-00684866},
    title = {{Towards Scalable Data Management for Map-Reduce-based Data-Intensive Applications on Cloud and Hybrid Infrastructures}},
    author = {Antoniu, Gabriel and Bigot, Julien and Blanchet, Christophe and Boug{\'e}, Luc and Briant, Fran{\c c}ois and Cappello, Franck and Costan, Alexandru and Desprez, Fr{\'e}d{\'e}ric and Fedak, Gilles and Gault, Sylvain and Keahey, Kate and Nicolae, Bogdan and P{\'e}rez, Christian and Simonet, Anthony and Suter, Fr{\'e}d{\'e}ric and Tang, Bing and Terreux, Raphael},
    abstract = {{As Map-Reduce emerges as a leading programming paradigm for data-intensive computing, today's frameworks which support it still have substantial shortcomings that limit its potential scalability. In this paper we discuss several directions where there is room for such progress: they concern storage efficiency under massive data access concurrency, scheduling, volatility and fault-tolerance. We place our discussion in the perspective of the current evolution towards an increasing integration of large-scale distributed platforms (clouds, cloud federations, enterprise desktop grids, etc.). We propose an approach which aims to overcome the current limitations of existing Map-Reduce frameworks, in order to achieve scalable, concurrency-optimized, fault-tolerant Map-Reduce data processing on hybrid infrastructures. This approach will be evaluated with real-life bio-informatics applications on existing Nimbus-powered cloud testbeds interconnected with desktop grids.}},
    keywords = {MapReduce, cloud computing, data-intensive computing, hybrid infrastructures, BlobSeer, BitDew, Nimbus, HLCM, Grid'5000},
    language = {Anglais},
    affiliation = {KerData - INRIA - IRISA , AVALON - LIP Lyon / Inria Grenoble Rh{\^o}ne-Alpes , Institut de biologie et chimie des prot{\'e}ines [Lyon] - IBCP , IBM PSSC Montpellier - Innovation Lab. , GRAND-LARGE - INRIA Saclay - Ile de France , Joint Laboratory for Petascale Computing [Illinois] - JLPC , Laboratoire de Recherche en Informatique - LRI , Argonne National Laboratory - ANL , Centre de Calcul de l'inst. national de phy. nucl{\'e}aire et de phy. des particules - CC IN2P3},
    booktitle = {{1st International IBM Cloud Academy Conference - ICA CON 2012}},
    address = {Research Triangle Park, North Carolina, {\'E}tats-Unis},
    audience = {internationale },
    year = {2012},
    pdf = {http://hal.inria.fr/hal-00684866/PDF/ICACON2012-MapReduce.pdf},
}

%F hal-00684866, version 1
%0 Conference Proceedings
%U http://hal.inria.fr/hal-00684866
%2 INFO:INFO_DC
%T Towards Scalable Data Management for Map-Reduce-based Data-Intensive Applications on Cloud and Hybrid Infrastructures
%A Antoniu, Gabriel
%A Bigot, Julien
%A Blanchet, Christophe
%A Bougé, Luc
%A Briant, François
%A Cappello, Franck
%A Costan, Alexandru
%A Desprez, Frédéric
%A Fedak, Gilles
%A Gault, Sylvain
%A Keahey, Kate
%A Nicolae, Bogdan
%A Pérez, Christian
%A Simonet, Anthony
%A Suter, Frédéric
%A Tang, Bing
%A Terreux, Raphael
%+ KerData - INRIA - IRISA , AVALON - LIP Lyon / Inria Grenoble Rhône-Alpes , Institut de biologie et chimie des protéines [Lyon] - IBCP , IBM PSSC Montpellier - Innovation Lab. , GRAND-LARGE - INRIA Saclay - Ile de France , Joint Laboratory for Petascale Computing [Illinois] - JLPC , Laboratoire de Recherche en Informatique - LRI , Argonne National Laboratory - ANL , Centre de Calcul de l'inst. national de phy. nucléaire et de phy. des particules - CC IN2P3
%X As Map-Reduce emerges as a leading programming paradigm for data-intensive computing, today's frameworks which support it still have substantial shortcomings that limit its potential scalability. In this paper we discuss several directions where there is room for such progress: they concern storage efficiency under massive data access concurrency, scheduling, volatility and fault-tolerance. We place our discussion in the perspective of the current evolution towards an increasing integration of large-scale distributed platforms (clouds, cloud federations, enterprise desktop grids, etc.). We propose an approach which aims to overcome the current limitations of existing Map-Reduce frameworks, in order to achieve scalable, concurrency-optimized, fault-tolerant Map-Reduce data processing on hybrid infrastructures. This approach will be evaluated with real-life bio-informatics applications on existing Nimbus-powered cloud testbeds interconnected with desktop grids.
%G Anglais
%8 2012-00-00
%2 internationale
%B 1st International IBM Cloud Academy Conference - ICA CON 2012
%2 Research Triangle Park, North Carolina
%2 États-Unis
%8 2012-04
%K MapReduce, cloud computing, data-intensive computing, hybrid infrastructures, BlobSeer, BitDew, Nimbus, HLCM, Grid'5000
%2 8534

<?xml version="1.0" encoding="UTF-8"?>
<listBibl xmlns="http://www.tei-c.org/ns/1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:mml="http://www.w3.org/1998/Math/MathML">
<listBibl type="inproceedings"><head>Communications avec actes</head>
<biblStruct type="inproceedings" xml:id="hal-00684866"><analytic>
<author><persName><forename>Gabriel</forename><surname>Antoniu</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA - IRISA</orgName>
<orgName type="team">KerData</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR6074</orgName>
<orgName type="institution">École normale supérieure de Cachan - ENS Cachan</orgName>
<orgName type="institution">Institut National des Sciences Appliquées (INSA) - Rennes</orgName>
<orgName type="institution">Université de Rennes 1</orgName>
</affiliation>
</author>
<author><persName><forename>Julien</forename><surname>Bigot</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Christophe</forename><surname>Blanchet</surname></persName>
<affiliation>
<orgName type="laboratory">IBCP</orgName>
<orgName type="team">Institut de biologie et chimie des protéines [Lyon]</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5086</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Luc</forename><surname>Bougé</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA - IRISA</orgName>
<orgName type="team">KerData</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR6074</orgName>
<orgName type="institution">École normale supérieure de Cachan - ENS Cachan</orgName>
<orgName type="institution">Institut National des Sciences Appliquées (INSA) - Rennes</orgName>
<orgName type="institution">Université de Rennes 1</orgName>
</affiliation>
</author>
<author><persName><forename>François</forename><surname>Briant</surname></persName>
<affiliation>
<orgName type="team">IBM PSSC Montpellier - Innovation Lab.</orgName>
<country>FR</country>
<orgName type="institution">IBM PSSC Montpellier</orgName>
</affiliation>
</author>
<author><persName><forename>Franck</forename><surname>Cappello</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA Saclay - Ile de France</orgName>
<orgName type="team">GRAND-LARGE</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR8623</orgName>
<orgName type="institution">Université Paris XI - Paris Sud</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">JLPC</orgName>
<orgName type="team">Joint Laboratory for Petascale Computing [Illinois]</orgName>
<country>US</country>
<orgName type="institution">University of Illinois at Urbana-Champaign</orgName>
<orgName type="institution">INRIA</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">LRI</orgName>
<orgName type="team">Laboratoire de Recherche en Informatique</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR8623</orgName>
<orgName type="institution">Université Paris XI - Paris Sud</orgName>
</affiliation>
</author>
<author><persName><forename>Alexandru</forename><surname>Costan</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA - IRISA</orgName>
<orgName type="team">KerData</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR6074</orgName>
<orgName type="institution">École normale supérieure de Cachan - ENS Cachan</orgName>
<orgName type="institution">Institut National des Sciences Appliquées (INSA) - Rennes</orgName>
<orgName type="institution">Université de Rennes 1</orgName>
</affiliation>
</author>
<author><persName><forename>Frédéric</forename><surname>Desprez</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Gilles</forename><surname>Fedak</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Sylvain</forename><surname>Gault</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Kate</forename><surname>Keahey</surname></persName>
<affiliation>
<orgName type="laboratory">ANL</orgName>
<orgName type="team">Argonne National Laboratory</orgName>
<country>US</country>
<orgName type="institution">US Department of Energy</orgName>
<orgName type="institution">University of Chicago</orgName>
</affiliation>
</author>
<author><persName><forename>Bogdan</forename><surname>Nicolae</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA Saclay - Ile de France</orgName>
<orgName type="team">GRAND-LARGE</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR8623</orgName>
<orgName type="institution">Université Paris XI - Paris Sud</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">JLPC</orgName>
<orgName type="team">Joint Laboratory for Petascale Computing [Illinois]</orgName>
<country>US</country>
<orgName type="institution">University of Illinois at Urbana-Champaign</orgName>
<orgName type="institution">INRIA</orgName>
</affiliation>
</author>
<author><persName><forename>Christian</forename><surname>Pérez</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Anthony</forename><surname>Simonet</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Frédéric</forename><surname>Suter</surname></persName>
<affiliation>
<orgName type="laboratory">CC IN2P3</orgName>
<orgName type="team">Centre de Calcul de l'inst. national de phy. nucléaire et de phy. des particules</orgName>
<country>FR</country>
<orgName type="institution">CNRS : USR6402</orgName>
<orgName type="institution">IN2P3</orgName>
</affiliation>
</author>
<author><persName><forename>Bing</forename><surname>Tang</surname></persName>
<affiliation>
<orgName type="laboratory">LIP Lyon / Inria Grenoble Rhône-Alpes</orgName>
<orgName type="team">AVALON</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5668</orgName>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">École Normale Supérieure - Lyon</orgName>
<orgName type="institution">Laboratoire d'informatique du Parallélisme</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<author><persName><forename>Raphael</forename><surname>Terreux</surname></persName>
<affiliation>
<orgName type="laboratory">IBCP</orgName>
<orgName type="team">Institut de biologie et chimie des protéines [Lyon]</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR5086</orgName>
<orgName type="institution">Université Claude Bernard - Lyon I</orgName>
</affiliation>
</author>
<title type="main" level="a">Towards Scalable Data Management for Map-Reduce-based Data-Intensive Applications on Cloud and Hybrid Infrastructures</title></analytic><monogr><meeting>1st International IBM Cloud Academy Conference - ICA CON 2012</meeting><imprint><pubPlace><settlement>Research Triangle Park, North Carolina</settlement><country>États-Unis</country></pubPlace><biblScope type="publicationDate">2012-00-00</biblScope></imprint></monogr><idno type="HALid">http://hal.inria.fr/hal-00684866</idno><idno type="HALFile">http://hal.inria.fr/hal-00684866/PDF/ICACON2012-MapReduce.pdf</idno></biblStruct>
</listBibl>
</listBibl>