R. Khare, B. M. Good, and R. Leaman, Crowdsourcing in biomedicine: challenges and opportunities, Briefings in Bioinformatics, vol.17, issue.1, 2015.
DOI : 10.1093/bib/bbv021

R. Leaman, B. M. Good, and A. I. Su, CROWDSOURCING AND MINING CROWD DATA, Biocomputing 2015, pp.267-269, 2015.
DOI : 10.1142/9789814644730_0026

D. Geiger, S. Seedorf, and T. Schulze, Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes Labeling images with a computer game, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.319-326, 2004.

M. Candito and D. Seddah, Le corpus Sequoia : annotation syntaxique et exploitation pour l'adaptation d'analyseur par pont lexical, Actes de la conférence conjointe JEP-TALN-RECITAL 2012, pp.321-334, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00698938

N. Gupta, D. Martin, and B. V. Hanrahan, Turk-Life in India, Proceedings of the 18th International Conference on Supporting Group Work, GROUP '14, pp.1-11, 2014.
DOI : 10.1145/2660398.2660403

M. Lafourcade, Making people play for Lexical Acquisition, Proc of the 7th Symposium on Natural Language Processing, 2007.
URL : https://hal.archives-ouvertes.fr/lirmm-00200883

J. Chamberlain, M. Poesio, and U. Kruschwitz, Phrase Detectives: A Webbased Collaborative Annotation Game, Proc. of the international Conference on Semantic Systems (I-Semantics '08), 2008.

F. Khatib, S. Cooper, and M. D. Tyka, Algorithm discovery by protein folding game players, Proceedings of the National Academy of Sciences, vol.108, issue.47, pp.18949-18953, 2011.
DOI : 10.1073/pnas.1115898108

K. Fort, B. Guillaume, and H. Chastant, Creating Zombilingo, a Game With A Purpose for dependency syntax annotation, Gamification for Information Retrieval (GamifIR'14) Workshop. 11. Fort, K. (2016) Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects, 2014.
URL : https://hal.archives-ouvertes.fr/hal-00969157

J. Chamberlain, K. Fort, and U. Kruschwitz, Using Games to Create Language Resources: Successes and Limitations of the Approach The People's Web Meets NLP, pp.3-44, 2013.

K. Fort, G. Adda, and K. B. Cohen, Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics, pp.413-420, 2011.

A. Shahri, M. Hosseini, and K. Phalp, Towards a Code of Ethics for Gamification at Enterprise The Practice of Enterprise Modeling: 7th IFIP WG 8.1 Working Conference, Proceedings, pp.235-245, 2014.

J. D. Burger, E. Doughty, and R. Khare, Hybrid curation of genemutation relations combining automated extraction and crowdsourcing. Database : the journal of biological databases and curation, p.16, 2014.

R. Khare, J. D. Burger, and J. S. Aberdeen, Scaling drug indication curation through crowdsourcing, Database, vol.2015, issue.0, 2015.
DOI : 10.1093/database/bav016

T. S. Li, A. Bravo, and L. Furlong, A crowdsourcing workflow for extracting chemical-induced disease relations from free text. Database : the journal of biological databases and curation Microtask crowdsourcing for disease mention annotation in PubMed abstracts, Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing, pp.282-293, 2015.

H. Zhai, T. Lingren, and L. Deleger, Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language Processing, Journal of Medical Internet Research, vol.15, issue.4, p.73, 2013.
DOI : 10.2196/jmir.2426

E. B. Camon, D. G. Barrell, and E. C. Dimmer, An evaluation of GO annotation retrieval for BioCreAtIvE and GOA Text mining and manual curation of chemical-gene-disease networks for the comparative toxicogenomics database (CTD) A CTD-Pfizer collaboration: manual curation of 88,000 scientific articles text mined for drug-disease and drug-phenotype interactions, S17. 21. Wiegers, Database : the journal of biological databases and curation, pp.326-348, 2005.

W. J. Wilbur, A comparison of group and individual performance among subject experts and untrained workers at the document retrieval task, Journal of the American Society for Information Science, vol.30, issue.6, pp.517-529, 1998.
DOI : 10.1002/(SICI)1097-4571(19980501)49:6<517::AID-ASI4>3.0.CO;2-T

A. L. Tarca, M. Lauria, and M. Unger, Strengths and limitations of microarray-based phenotype prediction: lessons learned from the IMPROVER Diagnostic Signature Challenge, Bioinformatics, vol.29, issue.22, pp.2892-2899, 2013.
DOI : 10.1093/bioinformatics/btt492

E. Bilal, T. Sakellaropoulos, and I. N. Melas, A crowd-sourcing approach for the construction of species-specific cell signaling networks, Bioinformatics, vol.31, issue.4, pp.484-491, 2015.
DOI : 10.1093/bioinformatics/btu659

?. Oprea, T. I. Bologa, C. G. Boyer, S. Curpan, R. F. Glen et al., A crowdsourcing evaluation of the NIH chemical probes, Nature Chemical Biology, vol.93, issue.7, pp.441-447, 2009.
DOI : 10.1038/nchembio0709-441

?. Maclean, D. Yoshida, K. Edwards, A. Crossman, L. Clavijo et al., Crowdsourcing genomic analyses of ash and ash dieback ??? power to the people, GigaScience, vol.2, issue.1, 2013.
DOI : 10.1186/2047-217X-2-2

?. Ekins, S. Williams, and A. J. , Reaching Out to Collaborators: Crowdsourcing for Pharmaceutical Research, Pharmaceutical Research, vol.6, issue.Suppl 5, pp.393-395, 2010.
DOI : 10.1007/s11095-010-0059-0

?. Plenge, R. M. Greenberg, J. D. Mangravite, L. M. Derry, J. M. Stahl et al., & International Rheumatoid Arthritis Consortium Crowdsourcing genetic prediction of clinical utility in the Rheumatoid Arthritis Responder Challenge, Nature genetics, issue.5, pp.45-468, 2013.

?. Lakhani, K. R. Boudreau, K. J. Loh, P. R. Backstrom, L. Baldwin et al., Prize-based contests can provide solutions to computational biology problems, Nature Biotechnology, vol.85, issue.2, pp.31-108, 2013.
DOI : 10.1073/pnas.0403723101

?. Good, B. M. Su, and A. I. , Crowdsourcing for bioinformatics, Bioinformatics, vol.29, issue.16, p.333, 2013.
DOI : 10.1093/bioinformatics/btt333

?. Garneau, N. L. Nuessle, T. M. Sloan, M. M. Santorico, S. A. Coughlin et al., Crowdsourcing taste research: genetic and phenotypic predictors of bitter taste perception as a model, Frontiers in integrative neuroscience, 2014.
DOI : 10.1016/S0031-9384(03)00057-X

?. Ledford, H. Barral, A. M. Makhluf, H. Soneral, P. Gasper et al., Molecular biology gets wikified Small World Initiative: crowdsourcing research of new antibiotics to enhance undergraduate biology teaching (618.41), Nature Online The FASEB Journal, vol.23, issue.281, pp.618-659, 2008.

?. Parslow and G. R. , Commentary: Crowdsourcing, foldit, and scientific discovery games, Biochemistry and Molecular Biology Education, vol.41, issue.2, pp.116-117, 2013.
DOI : 10.1002/bmb.20686

?. Prill, R. J. Saez-rodriguez, J. Alexopoulos, L. G. Sorger, P. K. Stolovitzky et al., Crowdsourcing Network Inference: The DREAM Predictive Signaling Network Challenge, Science Signaling, vol.4, issue.189, p.7, 2011.
DOI : 10.1126/scisignal.2002212

?. Krantz, M. S. Berg, and J. S. , Crowdsourcing to define the clinical actionability of incidental findings of genetic testing, NC Med J, vol.74, issue.6, pp.501-502, 2013.

?. Torr-brown and S. , Crowdsourcing for Science and Medicine: Progress and Challenges, The Journal of OncoPathology, vol.1, issue.2, pp.75-81, 2013.

?. Ranard, B. L. Ha, Y. P. Meisel, Z. F. Asch, D. A. Hill et al., Crowdsourcing???Harnessing the Masses to Advance Health and Medicine, a Systematic Review, Journal of General Internal Medicine, vol.14, issue.2, pp.187-203, 2014.
DOI : 10.1007/s11606-013-2536-8

?. Swan and M. , Health 2050: The Realization of Personalized Medicine through Crowdsourcing, the Quantified Self, and the Participatory Biocitizen, Journal of Personalized Medicine, vol.2, issue.4, pp.93-118, 2012.
DOI : 10.3390/jpm2030093

?. Khare, R. Burger, J. D. Aberdeen, J. S. Tresner-kirsch, D. W. Corrales et al., Scaling drug indication curation through crowdsourcing, Database, vol.2015, issue.0, p.16, 2015.
DOI : 10.1093/database/bav016

. Some, H. Zhai, T. Lingren, L. Deleger, Q. Li et al., Cheap, fast, and good enough for the non-biomedical domain but is it usable for clinical natural language processing? Evaluating crowdsourcing for clinical trial announcement named entity annotations, 2012 IEEE Second International Conference on Healthcare Informatics, Imaging and Systems Biology, p.106, 2012.

?. Mortensen, J. M. Musen, M. A. Noy, and N. F. , Crowdsourcing the verification of relationships in biomedical ontologies American Medical Informatics Association, AMIA Annual Symposium Proceedings, p.1020, 2013.

?. Zhai, H. Lingren, T. Deleger, L. Li, Q. Kaiser et al., Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language Processing, Journal of Medical Internet Research, vol.15, issue.4, 2013.
DOI : 10.2196/jmir.2426

?. Burger, J. D. Doughty, E. Khare, R. Wei, C. H. Mishra et al., Hybrid curation of gene-mutation relations combining automated extraction and crowdsourcing, Database, vol.2014, issue.0, p.94, 2014.
DOI : 10.1093/database/bau094

?. Munro?, R. Gunasekara?, L. Nevins?, S. Polepeddi?, L. Rosen? et al., Tracking epidemics with natural language processing and crowdsourcing, 2012.

?. Thessen, A. E. Cui, H. Mozzherin, and D. , Applications of Natural Language Processing in Biodiversity Science, Advances in Bioinformatics, vol.34, issue.1, 2012.
DOI : 10.1016/j.tree.2007.03.013

?. Saunders, D. R. Bex, P. J. Woods, and R. L. , Crowdsourcing a Normative Natural Language Dataset: A Comparison of Amazon Mechanical Turk and In-Lab Data Collection, Journal of Medical Internet Research, vol.15, issue.5, 2013.
DOI : 10.2196/jmir.2620

K. Fort, G. Adda, and K. B. Cohen, Some history of crowdsourcing in natural language processing Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, pp.413-420, 2011.

?. Fort, K. Adda, G. Sagot, B. Mariani, J. Couillault et al., Crowdsourcing for Language Resource Development: Criticisms About Amazon Mechanical Turk Overpowering Use, Human Language Technology Challenges for Computer Science and Linguistics, pp.303-314, 2014.
DOI : 10.1007/978-3-319-14120-6_25
URL : https://hal.archives-ouvertes.fr/hal-01053047

?. Chamberlain, J. Fort, K. Kruschwitz, U. Lafourcade, M. Poesio et al., Using Games to Create Language Resources: Successes and Limitations of the Approach, The People's Web Meets NLP, pp.3-44, 2013.
DOI : 10.1007/978-3-642-35085-6_1
URL : https://hal.archives-ouvertes.fr/lirmm-00831442

?. Sabou, M. Bontcheva, K. Derczynski, L. Scharl, and A. , Corpus annotation through crowdsourcing: Towards best practice guidelines, Proceedings of LREC, 2014.

?. Snow, R. O-'connor, B. Jurafsky, D. Ng, and A. Y. , Cheap and fast---but is it good? evaluating non-expert annotations for natural language tasks, Proceedings of the conference on empirical methods in natural language processing, pp.254-263, 2008.

?. Sabou, M. Bontcheva, K. Scharl, and A. , Crowdsourcing research opportunities, Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies, i-KNOW '12, p.17, 2012.
DOI : 10.1145/2362456.2362479

?. Munro, R. Bethard, S. Kuperman, V. Lai, V. T. Melnick et al., Crowdsourcing and language studies: the new generation of linguistic data, Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pp.122-130, 2010.

?. Finin, T. Murnane, W. Karandikar, A. Keller, N. Martineau et al., Annotating named entities in Twitter data with crowdsourcing, Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pp.80-88, 2010.

?. Le, J. Edmonds, A. Hester, V. Biewald, and L. , Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution, SIGIR 2010 workshop on crowdsourcing for search evaluation, pp.21-26, 2010.

?. Callison-burch, C. Dredze, and M. , Creating speech and language data with Amazon's Mechanical Turk, Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pp.1-12, 2010.

?. Wang, W. Y. Bohus, D. Kamar, E. Horvitz, and E. , Crowdsourcing the acquisition of natural language corpora: Methods and observations, 2012 IEEE Spoken Language Technology Workshop (SLT), pp.2012-73, 2012.
DOI : 10.1109/SLT.2012.6424200

?. Raykar, V. C. Yu, and S. , Ranking annotators for crowdsourced labeling tasks, Advances in neural information processing systems, pp.1809-1817, 2011.