Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Cost and Quality Assurance in Crowdsourcing Workflows

Loïc Hélouët 1 Zoltan Miklos 2 Rituraj Singh 2
1 SUMO - SUpervision of large MOdular and distributed systems
Inria Rennes – Bretagne Atlantique , IRISA-D4 - LANGAGE ET GÉNIE LOGICIEL
Abstract : Crowdsourcing platforms provide tools to replicate and distribute micro tasks (simple, independent work units) to the crowd and assemble results. However, real-life problems are often complex: they require to collect, organize, process or transform data, guarantee the quality of results, and meet budget constraints. Further, specifications and realization of intricate jobs on crowdsourcing platforms are still in their infancy. In this work, we combine workflows with crowdsourcing to enable efficient execution of complex tasks. Workflows provide ways to organize a complex task in phases and guide the overall realization. The challenge is to interface workflows and crowd-sourcing systems efficiently to achieve good accuracy of results at a reasonable cost. Standard "static" allocation of work in crowd-sourcing affects a fixed number of workers per micro-task to realize and aggregates the results. We propose synchronous and asynchronous dynamic worker allocation techniques on top of workflows, where decisions to replicate tasks, worker allocation and progress of the workflow execution depend on inferred tasks difficulty, worker expertise, confidence in answers and remaining budget. We evaluate the performance of this framework on a benchmark and show that the proposed approaches outperform static allocation in terms of cost and accuracy.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [34 references]  Display  Hide  Download

https://hal.inria.fr/hal-02964736
Contributor : Loic Helouet <>
Submitted on : Monday, October 12, 2020 - 4:28:01 PM
Last modification on : Monday, October 19, 2020 - 10:54:11 AM

File

edbt-workflow.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02964736, version 1

Citation

Loïc Hélouët, Zoltan Miklos, Rituraj Singh. Cost and Quality Assurance in Crowdsourcing Workflows. 2020. ⟨hal-02964736⟩

Share

Metrics

Record views

11

Files downloads

33