Skip to Main content Skip to Navigation

Using hierarchical skills for optimized task selection in crowdsourcing

Abstract : A large number of commercial and academic participative applications rely on a crowd to acquire, disambiguate and clean data. These participative applications are widely known as crowdsourcing platforms where amateur enthusiasts are involved in real scientific or commercial projects. Requesters are outsourcing tasks by posting them on online commercial crowdsourcing platforms such as Amazon MTurk or Crowdflower. There, online participants select and perform these tasks, called microtasks, accepting a micropayment in return. These platforms face challenges such as reassuring the quality of the acquired answers, assisting participants to find relevant and interesting tasks, leveraging expert skills among the crowd, meeting tasks' deadlines and satisfying participants that will happily perform more tasks. However, related work mainly focuses on modeling skills as keywords to improve quality, in this work we formalize skills with the use a hierarchical structure, a taxonomy, that can inherently provide with a natural way to substitute tasks with similar skills. It also takes advantage of the whole crowd workforce. With extensive synthetic and real datasets, we show that there is a significant improvement in quality when someone considers a hierarchical structure of skills instead of pure keywords. On the other hand, we extend our work to study the impact of a participant’s choice given a list of tasks. While our previous solution focused on improving an overall one-to-one matching for tasks and participants we examine how participants can choose from a ranked list of tasks. Selecting from an enormous list of tasks can be challenging and time consuming and has been proved to affect the quality of answers to crowdsourcing platforms. Existing related work concerning crowdsourcing does not use either a taxonomy or ranking methods, that exist in other similar domains, to assist participants. We propose a new model that takes advantage of the diversity of the parcipant's skills and proposes him a smart list of tasks, taking into account their deadlines as well. To the best of our knowledge, we are the first to combine the deadlines of tasks into an urgency metric with the task proposition for knowledge-intensive crowdsourcing. Our extensive synthetic and real experimentation show that we can meet deadlines, get high quality answers, keep the interest of participants while giving them a choice of well selected tasks.
Keywords : crowdsourcing
Document type :
Complete list of metadata

Cited literature [136 references]  Display  Hide  Download
Contributor : David Gross-Amblard Connect in order to contact the contributor
Submitted on : Friday, March 6, 2020 - 5:03:09 PM
Last modification on : Wednesday, November 3, 2021 - 8:14:23 AM
Long-term archiving on: : Sunday, June 7, 2020 - 3:38:06 PM


Files produced by the author(s)


  • HAL Id : tel-02501308, version 1


Panagiotis Mavridis. Using hierarchical skills for optimized task selection in crowdsourcing. Computer science. Université rennes1, 2017. English. ⟨tel-02501308⟩



Record views


Files downloads