Abstract
Crowdsourcing, in particular microtasking is now a powerful concept used by employers in order to obtain answers on tasks hardly handled by automated computation. These answers are provided by human employees and then combined to get a final answer. Nevertheless, the quality of participants in microtasking platforms is often heterogeneous which makes results imperfect and thus not fully reliable. To tackle this problem, we propose a new approach of label aggregation based on gold standards under the belief function theory. This latter provides several tools able to represent and even combine imperfect information. Experiments conducted on both simulated and real world datasets show that our approach improves results quality even with a high ratio of bad workers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Howe, J.: The rise of crowdsourcing. Wired Magaz. 14(6), 1–4 (2006)
Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. Annals Math. Stat. 38, 325–339 (1967)
Shafer, G.: A Mathematical Theory of Evidence, vol. 1. Princeton University Press, Princeton (1976)
Jousselme, A.-L., Grenier, D., Bossé, É.: A new distance between two bodies of evidence. Inf. Fusion 2, 91–101 (2001)
Lefèvre, E., Elouedi, Z.: How to preserve the confict as an alarm in the combination of belief functions? Decis. Support Syst. 56, 326–333 (2013)
Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990)
Lee, K., Caverlee, J., Webb, S.: The social honeypot project: protecting online communities from spammers. In: International World Wide Web Conference, pp. 1139–1140 (2010)
Abassi, L., Boukhris, I.: Crowd label aggregation under a belief function framework. In: Lehner, F., Fteimi, N. (eds.) KSEM 2016. LNCS, vol. 9983, pp. 185–196. Springer, Cham (2016). doi:10.1007/978-3-319-47650-6_15
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (2010)
Khattak, F.K., Salleb, A.: Quality control of crowd labeling through expert evaluation. In: The Neural Information Processing Systems 2nd Workshop on Computational Social Science and the Wisdom of Crowds, pp. 27–29 (2011)
Smets, P., Mamdani, A., Dubois, D., Prade, H.: Non Standard Logics for Automated Reasoning, pp. 253–286. Academic Press, London (1988)
Trabelsi, A., Elouedi, Z., Lefèvre, E.: Belief function combination: comparative study within the classifier fusion framework. In: Gaber, T., Hassanien, A.E., El-Bendary, N., Dey, N. (eds.) AISI 2015. AISC, vol. 407, pp. 425–435. Springer, Cham (2016). doi:10.1007/978-3-319-26690-9_38
Karger, D., Oh, S., Shah, D.: Iterative learning for reliable crowdsourcing systems. In: Neural Information Processing Systems, pp. 1953–1961 (2011)
Ben Rjab, A., Kharoune, M., Miklos, Z., Martin, A.: Characterization of experts in crowdsourcing platforms. In: Vejnarová, J., Kratochvíl, V. (eds.) BELIEF 2016. LNCS (LNAI), vol. 9861, pp. 97–104. Springer, Cham (2016). doi:10.1007/978-3-319-45559-4_10
Welinder, P., Branson, S., Perona, P., Belongie, S.: The multidimensional wisdom of crowds. In: Neural Information Processing Systems, pp. 2424–2432 (2010)
Frank, A.: UCI machine learning repository (1987). http://archive.ics.uci.edu/ml
Whitehill, J., Wu, T., Bergsma, J., Movellan, R.J., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Neural Information Processing Systems, pp. 2035–2043 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Abassi, L., Boukhris, I. (2017). A Gold Standards-Based Crowd Label Aggregation Within the Belief Function Theory. In: Benferhat, S., Tabia, K., Ali, M. (eds) Advances in Artificial Intelligence: From Theory to Practice. IEA/AIE 2017. Lecture Notes in Computer Science(), vol 10351. Springer, Cham. https://doi.org/10.1007/978-3-319-60045-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-60045-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60044-4
Online ISBN: 978-3-319-60045-1
eBook Packages: Computer ScienceComputer Science (R0)