[1]WEN Youfu,JIA Caiyan,CHEN Zhineng.A multi-modal fusion approach for measuring web video relatedness[J].CAAI Transactions on Intelligent Systems,2016,11(3):359-365.[doi:10.11992/tis.201603040]
Copy
A multi-modal fusion approach for measuring web video relatedness
CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume:
11
Number of periods:
2016 3
Page number:
359-365
Column:
学术论文—机器感知与模式识别
Public date:
2016-06-25
- Title:
- A multi-modal fusion approach for measuring web video relatedness
- Keywords:
- web video; large-scale video; social feature; human-video interactions; multi-source heterogeneous information; social features; multi-modal fusion; relatedness measurement; video retrieval
- CLC:
- TP393
- DOI:
- 10.11992/tis.201603040
- Abstract:
- With the advances in internet and multimedia technologies, the number of web videos on social video platforms rapidly grows. Therefore, tasks such as large-scale video retrieval, classification, and annotation become issues that need to be urgently addressed. Web video relatedness serves as a basic and common infrastructure for these issues. This paper investigates the measurement of web video relatedness from a multi-modal fusion perspective. It proposes to measure web video relatedness based on multi-source heterogeneous information. The multi-modal fusion simultaneously leverages videos’ visual content, title, and tag text as well as social features contributed by human-video interactions (i.e., the upload time, channel, and author of a video). Consequently, a novel multi-modal fusion approach is proposed for computing web video relatedness, which serves to give a ranking criterion and is applied to the task of large-scale video retrieval. Experimental results using YouTube videos show that the proposed text, visual, and users’ social feature multi-modal fusion approach performs best in comparison tests with three alternate approaches; i.e., those approaches that compute web video relatedness based just on text features, just on visual features, or jointly on text and visual features.
- References:
-
[1] ZHU Weiyu, TOKLU C, LIOU S P. Automatic news video segmentation and categorization based on closed-captioned text[C]//Proceedings of IEEE International Conference on Multimedia and Expo. Tokyo, Japan, 2001: 829-832.
[2] BREZEALE D, COOK D J. Using closed captions and visual features to classify movies by genre[C]//Poster Session of the Seventh International Workshop on Multimedia Data Mining. Philadelphia, Pennsylvania, USA, 2006.
[3] SCHMIEDEKE S, KELM P, SIKORA T. TUB @ MediaEval 2011 genre tagging task: prediction using bag-of-(visual)-words approaches[C]//Working Notes Proceedings of the MediaEval 2011 Workshop. Pisa, Italy, 2011: 1-2.
[4] LAW-TO J, CHEN Li, JOLY A, et al. Video copy detection: a comparative study[C]//Proceedings of the 6th ACM International Conference on Image and Video Retrieval. New York, NY, USA, 2007: 371-378.
[5] WU Xiao, HAUPTMANN A G, NGO C W. Practical elimination of near-duplicates from web video search[C]//Proceedings of the 15th ACM International Conference on Multimedia. New York, NY, USA, 2007: 218-227.
[6] SONG Jingkuan, YANG Yi, HUANG Zi, et al. Multiple feature hashing for real-time large scale near-duplicate video retrieval[C]//Proceedings of the 19th ACM International Conference on Multimedia. New York, NY, USA, 2011: 423-432.
[7] PERRONNIN F, DANCE C. Fisher kernels on visual vocabularies for image categorization[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN, USA, 2007: 1-8.
[8] JéGOU H, DOUZE M, SCHMID C, et al. Aggregating local descriptors into a compact image representation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Francisco, CA, USA, 2010: 3304-3311.
[9] TAN H K, NGO C W, HONG R, et al. Scalable detection of partial near-duplicate videos by visual-temporal consistency[C]//Proceedings of the 17th ACM International Conference on Multimedia. New York, NY, USA, 2009: 145-154.
[10] FENG Bailan, CAO Juan, CHEN Zhineng, et al. Multi-modal query expansion for web video search[C]//Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA, 2010: 721-722.
[11] BREZEALE D, COOK D J. Automatic video classification: a survey of the literature[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2008, 38(3): 416-430.
[12] YANG Linjun, LIU Jiemin, YANG Xiaokang, et al. Multi-modality web video categorization[C]//Proceedings of the International Workshop on Workshop on Multimedia Information Retrieval. New York, NY, USA, 2007: 265-274.
[13] WU Xiao, ZHAO Wanlei, NGO C W. Towards google challenge: combining contextual and social information for web video categorization[C]//Proceedings of the 17th ACM International Conference on Multimedia. New York, NY, USA, 2009: 1109-1110.
[14] DAVIDSON J, LIEBALD B, LIU J, et al. The YouTube video recommendation system[C]//Proceedings of the 4th ACM Conference on Recommender Systems. New York, NY, USA, 2010: 293-296.
[15] ZHAO Wanlei, WU Xiao, NGO C W. On the annotation of web videos by efficient near-duplicate search[J]. IEEE Transactions on Multimedia, 2010, 12(5): 448-461.
[16] TAN H K, NGO C W, CHUA T S. Efficient mining of multiple partial near-duplicate alignments by temporal network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2010, 20(11): 1486-1498.
[17] CAO J, ZHANG Y D, SONG Y D, et al. MCG-WEBV: a benchmark dataset for web video analysis[R]. Technical Report, Beijing, China: Institute of Computing Technology, 2009: 324-334.
[18] JIANG Yugang, JIANG Yudong, WANG Jiajun. VCDB: a large-scale database for partial copy detection in videos[M]//FLEET D, PAJDLA T, SCHIELE B, et al. Computer Vision-ECCV 2014. Zurich, Switzerland: Springer, 2014: 357-371.
- Similar References:
Memo
-
Last Update:
1900-01-01