Technical Program

Paper Detail

Paper:	SLP-P7.6
Session:	Audio-visual and Multimodal Processing
Time:	Wednesday, May 17, 10:00 - 12:00
Presentation:	Poster
Topic:	Speech and Spoken Language Processing: Multi-modal/multimedia processing (such as audio/visual, etc)
Title:	MULTIMODAL SPEAKER IDENTIFICATION USING CANONICAL CORRELATION ANALYSIS
Authors:	Mehmet Emre Sargin, Engin Erzin, Yucel Yemez, A. Murat Tekalp, Koç University, Turkey
Abstract:	In this work, we explore the use of canonical correlation analysis to improve the performance of multimodal recognition systems that involve multiple modalities that are correlated with each other. More specifically, we consider the audiovisual speaker identification problem, where speech and lip texture (or intensity) modalities are fused in an open-set identification framework. Our motivation is based on the following observation. The late integration strategy, which is also referred to as decision or opinion fusion, is effective especially in case the contributing modalities are uncorrelated and thus the resulting partial decisions are statistically independent. Early integration techniques on the other hand can be favored only if a couple of modalities are highly correlated. However, coupled modalities such as audio and lip texture also consist of some components that are mutually independent. Thus we first perform a cross-correlation analysis on the audio and lip modalities so as to extract the correlated part of the information, and then employ an optimal combination of early and late integration techniques to fuse the extracted features. The results of the experiments testing the performance of the proposed system are also provided.