Towards Engineering a Web-Scale Multimedia Service: A Case Study Using Spark

Abstract : Computing power has now become abundant with multi-core machines, grids and clouds, but it remains a challenge to harness the available power and move towards gracefully handling web-scale datasets. Several researchers have used automatically distributed computing frameworks, notably Hadoop and Spark, for processing multimedia material, but mostly using small collections on small clusters. In this paper , we describe the engineering process for a prototype near-web-scale multimedia service using the Spark framework running on the AWS cloud service. We present experimental results using up to 43 billion SIFT descriptors from the public YFCC 100M collection, making this the largest high-dimensional feature collection reported in the literature. The design of the prototype and performance results demonstrate both the flexibility and scalability of the Spark framework for implementing multimedia services.
Type de document :
Rapport
[Research Report] Inria Rennes Bretagne Atlantique; Reykjavik University; UC Berkeley. 2016
Liste complète des métadonnées


https://hal.inria.fr/hal-01416089
Contributeur : Laurent Amsaleg <>
Soumis le : mercredi 14 décembre 2016 - 08:12:57
Dernière modification le : vendredi 17 février 2017 - 16:10:23
Document(s) archivé(s) le : mercredi 15 mars 2017 - 13:48:11

Fichier

scalinggrace.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01416089, version 1

Citation

Gylfi Guðmundsson, Laurent Amsaleg, Björn Thor Jónsson, Michael Franklin. Towards Engineering a Web-Scale Multimedia Service: A Case Study Using Spark. [Research Report] Inria Rennes Bretagne Atlantique; Reykjavik University; UC Berkeley. 2016. <hal-01416089>

Partager

Métriques

Consultations de
la notice

2044

Téléchargements du document

5287