Allocation équitable de tâches pour l'analyse de données massives : MapReduce et système multi-agent

Abstract : Many companies are using MapReduce applications to process very large amounts of data. In order to optimize the task allocation, several systems collect data from previous runs and predict the performance doing job profiling. However they are not effective during the learning phase, or when a new kind of job or data set appears. In this paper, we present an adaptive multiagent system for large data sets analysis with MapReduce. We do not preprocess data but we adopt a dynamic approach, where the reducer agents interact during the job. In order to decrease the workload of the most loaded reducer - and so the running time - we propose a task re-allocation based on negotiation. We prove that the negotiation process terminates and leads to a better task allocation. Our experimentations over real-world data confirm the added-value of negotiation.
Liste complète des métadonnées

https://hal.inria.fr/hal-01570176
Contributor : Cristal Equipe Smac <>
Submitted on : Friday, July 28, 2017 - 3:53:07 PM
Last modification on : Monday, April 15, 2019 - 3:33:40 PM

Identifiers

  • HAL Id : hal-01570176, version 1

Citation

Quentin Baert, Anne-Cécile Caron, Jean-Christophe Routier, Maxime Morge. Allocation équitable de tâches pour l'analyse de données massives : MapReduce et système multi-agent. Revue des Sciences et Technologies de l'Information - Série RIA : Revue d'Intelligence Artificielle, Lavoisier, 2017, 31 (4), pp.401-426. ⟨hal-01570176⟩

Share

Metrics

Record views

205