[1]LI Yinan,WANG Shitong.A feature augmentation method for enhancing the labeling quality of crowdsourcing data[J].CAAI Transactions on Intelligent Systems,2020,15(2):227-234.[doi:10.11992/tis.201810014]
Copy

A feature augmentation method for enhancing the labeling quality of crowdsourcing data

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 15 Number of periods: 2020 2 Page number: 227-234 Column: 学术论文—知识工程 Public date: 2020-03-05
Title:
A feature augmentation method for enhancing the labeling quality of crowdsourcing data
Author(s):
LI Yinan;  WANG Shitong
School of Digital Media, Jiangnan University, Wuxi 214122, China
Keywords:
crowdsourcing; labeling quality; feature augmentation; expert labeling; noise identification; noise correction; noise probability; upper limit of noise number
CLC:
TP181
DOI:
10.11992/tis.201810014
Abstract:
Crowdsourcing is a new method of collecting the labels of data. Although it is economical, crowdsourcing faces an unavoidable problem, i.e., the quality of the labels cannot be guaranteed. In particular, when the quality of labeling work is low because of the existence of objective causes, the result of crowdsourcing will be unreliable. In this study, a feature augmentation method for enhancing the labeling quality of crowdsourcing data is proposed. In the proposed method, first, a small amount of expert data is labeled by several people with professional knowledge. Then, the crowdsourcing data are used to create the classifiers and predict the expert data. The resultant predicted labels are used to augment the expert data. Then, the augmented expert data are used to create the classifiers, predict the original data, and calculate the probability of noise for each instance and the upper limit of noise number to filter out the high-quality dataset from potentially noisy labels. Similarly, the filtered high-quality dataset is utilized to further correct the noisy labels using the proposed feature augmentation method. The experiments conducted on eight UCI datasets show that the proposed feature augmentation method has achieved encouraging results when the number of repeated labels is comparatively small or the quality of labeling is comparatively low.
References:
[1] ZHOU Zhihua. A brief introduction to weakly supervised learning[J]. National science review, 2018, 5(1): 44-53.
[2] HU Huiqi, ZHENG Yudian, BAO Zhifeng, et al. Crowdsourced POI labelling: location-aware result inference and task assignment[C]//Proceedings of 2016 IEEE 32nd International Conference on Data Engineering. Helsinki, Finland, 2016: 61-72.
[3] RODRIGUES F, PEREIRA F C, RIBEIRO B. Gaussian process classification and active learning with multiple annotators[C]//Proceedings of the 31st International Conference on International Conference on Machine Learning. Beijing, China, 2014: II-433-II-441.
[4] ZHANG Jing, SHENG V S, LI Tao, et al. Improving crowdsourced label quality using noise correction[J]. IEEE transactions on neural networks and learning systems, 2018, 29(5): 1675-1688.
[5] IPEIROTIS P G, PROVOST F, SHENG V S, et al. Repeated labeling using multiple noisy labelers[J]. Data mining and knowledge discovery, 2014, 28(2): 402-441.
[6] WHITEHILL J, RUVOLO P, WU Tingfan, et al. Whose vote should count more: optimal integration of labels from labelers of unknown expertise[C]//Proceedings of the 22nd International Conference on Neural Information Processing Systems. Vancouver, British Columbia, Canada, 2009: 2035-2043.
[7] RAYKAR V C, YU Shisheng, ZHAO L H, et al. Learning from crowds[J]. Journal of machine learning research, 2010, 11: 1297-1322.
[8] DEMARTINI G, DIFALLAH D E, CUDRé-MAUROUX P. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking[C]//Proceedings of the 21st International Conference on World Wide Web. Lyon, France, 2012: 469-478.
[9] MUHAMMADI J, RABIEE H R, HOSSEINI A. A unified statistical framework for crowd labeling[J]. Knowledge and information systems, 2015, 45(2): 271-294.
[10] FRENAY B, VERLEYSEN M. Classification in the presence of label noise: a survey[J]. IEEE transactions on neural networks and learning systems, 2014, 25(5): 845-869.
[11] GAMBERGER D, LAVRA? N, D?EROSKI S. Noise elimination in inductive concept learning: a case study in medical diagnosis[C]//Proceedings of the 7th International Workshop on Algorithmic Learning Theory. Sydney, Australia, 1996: 199-212.
[12] SUN Jiangwen, ZHAO Fengying, WANG Chongjun, et al. Identifying and correcting mislabeled training instances[C]//Proceedings of Future Generation Communication and Networking. Jeju, South Korea, 2007: 244-250.
[13] BRODLEY C E, FRIEDL M A. Identifying mislabeled training data[J]. Journal of artificial intelligence research, 1999, 11(1): 131-167.
[14] ZHOU Ta, ISHIBUCHI H, WANG Shitong. Stacked-structure-based hierarchical Takagi-Sugeno-Kang fuzzy classification through feature augmentation[J]. IEEE transactions on emerging topics in computational intelligence, 2017, 1(6): 421-436.
[15] ZHOU Zhihua. Ensemble methods: foundations and algorithms[M]. Boca Raton: Taylor & Francis, 2012.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems