[1]李易南,王士同.面向众包数据的特征扩维标签质量提高方法[J].智能系统学报,2020,15(2):227-234.[doi:10.11992/tis.201810014]
LI Yinan,WANG Shitong.A feature augmentation method for enhancing the labeling quality of crowdsourcing data[J].CAAI Transactions on Intelligent Systems,2020,15(2):227-234.[doi:10.11992/tis.201810014]
点击复制
LI Yinan,WANG Shitong.A feature augmentation method for enhancing the labeling quality of crowdsourcing data[J].CAAI Transactions on Intelligent Systems,2020,15(2):227-234.[doi:10.11992/tis.201810014]
面向众包数据的特征扩维标签质量提高方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
15
期数:
2020年第2期
页码:
227-234
栏目:
学术论文—知识工程
出版日期:
2020-03-05
- Title:
- A feature augmentation method for enhancing the labeling quality of crowdsourcing data
- Keywords:
- crowdsourcing; labeling quality; feature augmentation; expert labeling; noise identification; noise correction; noise probability; upper limit of noise number
- 分类号:
- TP181
- 摘要:
- 众包是一个新兴的收集数据集标签的方法。虽然它经济实惠，但面临着数据标签质量无法保证的问题。尤其是当客观原因存在使得众包工作者工作质量较差时，所得的标签会更加不可靠。因此提出一个名为基于特征扩维提高众包质量的方法(FA-method)，其基本思想是，首先由专家标注少部分标签，再利用众包者标注的数据集训练模型，对专家集进行预测，所得结果作为专家数据集新的特征，并利用扩维后的专家集训练模型进行预测，计算每个实例为噪声的可能性以及噪声数量上限来过滤出潜在含噪声标签的数据集，类似地，对过滤后的高质量集再次使用扩维的方法进一步校正噪声。在8个UCI数据集上进行验证的结果表明，和现有的结合噪声识别和校正的众包标签方法相比，所提方法能够在重复标签数量较少或标注质量较低时均取得很好的效果。
- Abstract:
- Crowdsourcing is a new method of collecting the labels of data. Although it is economical, crowdsourcing faces an unavoidable problem, i.e., the quality of the labels cannot be guaranteed. In particular, when the quality of labeling work is low because of the existence of objective causes, the result of crowdsourcing will be unreliable. In this study, a feature augmentation method for enhancing the labeling quality of crowdsourcing data is proposed. In the proposed method, first, a small amount of expert data is labeled by several people with professional knowledge. Then, the crowdsourcing data are used to create the classifiers and predict the expert data. The resultant predicted labels are used to augment the expert data. Then, the augmented expert data are used to create the classifiers, predict the original data, and calculate the probability of noise for each instance and the upper limit of noise number to filter out the high-quality dataset from potentially noisy labels. Similarly, the filtered high-quality dataset is utilized to further correct the noisy labels using the proposed feature augmentation method. The experiments conducted on eight UCI datasets show that the proposed feature augmentation method has achieved encouraging results when the number of repeated labels is comparatively small or the quality of labeling is comparatively low.
- 参考文献/References:
-
[1] ZHOU Zhihua. A brief introduction to weakly supervised learning[J]. National science review, 2018, 5(1): 44-53.
[2] HU Huiqi, ZHENG Yudian, BAO Zhifeng, et al. Crowdsourced POI labelling: location-aware result inference and task assignment[C]//Proceedings of 2016 IEEE 32nd International Conference on Data Engineering. Helsinki, Finland, 2016: 61-72.
[3] RODRIGUES F, PEREIRA F C, RIBEIRO B. Gaussian process classification and active learning with multiple annotators[C]//Proceedings of the 31st International Conference on International Conference on Machine Learning. Beijing, China, 2014: II-433-II-441.
[4] ZHANG Jing, SHENG V S, LI Tao, et al. Improving crowdsourced label quality using noise correction[J]. IEEE transactions on neural networks and learning systems, 2018, 29(5): 1675-1688.
[5] IPEIROTIS P G, PROVOST F, SHENG V S, et al. Repeated labeling using multiple noisy labelers[J]. Data mining and knowledge discovery, 2014, 28(2): 402-441.
[6] WHITEHILL J, RUVOLO P, WU Tingfan, et al. Whose vote should count more: optimal integration of labels from labelers of unknown expertise[C]//Proceedings of the 22nd International Conference on Neural Information Processing Systems. Vancouver, British Columbia, Canada, 2009: 2035-2043.
[7] RAYKAR V C, YU Shisheng, ZHAO L H, et al. Learning from crowds[J]. Journal of machine learning research, 2010, 11: 1297-1322.
[8] DEMARTINI G, DIFALLAH D E, CUDRé-MAUROUX P. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking[C]//Proceedings of the 21st International Conference on World Wide Web. Lyon, France, 2012: 469-478.
[9] MUHAMMADI J, RABIEE H R, HOSSEINI A. A unified statistical framework for crowd labeling[J]. Knowledge and information systems, 2015, 45(2): 271-294.
[10] FRENAY B, VERLEYSEN M. Classification in the presence of label noise: a survey[J]. IEEE transactions on neural networks and learning systems, 2014, 25(5): 845-869.
[11] GAMBERGER D, LAVRA? N, D?EROSKI S. Noise elimination in inductive concept learning: a case study in medical diagnosis[C]//Proceedings of the 7th International Workshop on Algorithmic Learning Theory. Sydney, Australia, 1996: 199-212.
[12] SUN Jiangwen, ZHAO Fengying, WANG Chongjun, et al. Identifying and correcting mislabeled training instances[C]//Proceedings of Future Generation Communication and Networking. Jeju, South Korea, 2007: 244-250.
[13] BRODLEY C E, FRIEDL M A. Identifying mislabeled training data[J]. Journal of artificial intelligence research, 1999, 11(1): 131-167.
[14] ZHOU Ta, ISHIBUCHI H, WANG Shitong. Stacked-structure-based hierarchical Takagi-Sugeno-Kang fuzzy classification through feature augmentation[J]. IEEE transactions on emerging topics in computational intelligence, 2017, 1(6): 421-436.
[15] ZHOU Zhihua. Ensemble methods: foundations and algorithms[M]. Boca Raton: Taylor & Francis, 2012.
备注/Memo
收稿日期:2018-10-15。
基金项目:国家自然科学基金项目(61272210)
作者简介:李易南，硕士研究生，主要研究方向为人工智能与模式识别;王士同，教授，博士生导师，主要研究方向为人工智能与模式识别。发表学术论文近百篇
通讯作者:李易南.E-mail:1920898036@qq.com
更新日期/Last Update:
1900-01-01