Summary 

Spatio-temporal patterns occur frequently in video, correlating with salient semantic events in specific domains. In this project, we are studying the problem of automatically discovering patterns without supervision, and this involves learning multi-level temporal statistical models such as Hierarchical HMM incorporating feature selection, and model adaptation techniques. Such unsupervised capability is powerful and complementary with approaches in supervised learning.

We have applied the mining techniques to baseball and soccer videos. Results show automatic discovery of interesting, although unsurprising, patterns corresponding to plays and breaks in the game. When evaluated with manual labels, the accuracy is very encouraging and comparable to those by supervised approaches. We are currently extending the mining techniques and feature selection methods to other domains such as surveillance, news, and consumer.

The diagram on the right shows the general multi-level structure of HHMM. This model retains Markovian structures at each level while exercising bottom-up transition control.

The diagram below shows the expressiveness of HHMM with correspondence to multi-level semantic states in the video.

People

Lexing Xie

Prof. Shih-Fu Chang

in collaboration with

Dr. Ajay Divakaran and Dr. Huifang Sun at Mitsubishi Electric Research Labs (MERL)

Publication

L. Xie, S.-F. Chang, A. Divakaran and H. Sun, Unsupervised Mining of Statistical Temporal Structures in Video, Book Chapter in Video Mining, A. Rosenfeld, D. Doremann and D. Dementhon Eds, Kluwer Academic Publishers, June 2003 . (PS.GZ/PDF)

L. Xie, S.-F. Chang, A. Divakaran and H. Sun, Unsupervised Discovery of Multilevel Statistical Video Structures Using Hierarchical Hidden Markov Models, Intern. Conf. on Multimedia and Exhibition (ICME), July 2003, Baltimore, MD, USA. (PS.GZ/PDF)

L. Xie, S.-F. Chang, A. Divakaran and H. Sun, Feature Selection for Unsupervised Discovery of Statistical Temporal Structures in Video, Intern. Conf. on Image Processing ICIP 2003, September 2003, Barcelona, Spain. (PS.GZ/PDF)

L. Xie, S.-F. Chang, A. Divakaran and H. Sun, Learning Hierarchical Hidden Markov Models for Video Structure Discovery, Advent Tech. Report, Dec. 2002 . (PS.GZ/PDF)

L. Xie, S.-F. Chang, A. Divakaran and H. Sun, Structure Analysis of Soccer Video with Hidden Markov Models, Proc. Interational Conference on Acoustic, Speech and Signal Processing, (ICASSP-2002), Orlando, FL, USA, May 13-17, 2002. (PS.GZ/PDF)

L. Xie, P. Xu, S.-F. Chang, A. Divakaran and H. Sun, Structure Analysis of Soccer Video with Domain Knowledge and Hidden Markov Models, Pattern Recognition Letters, to appear. (PS.GZ/PDF)

For problems or questions regarding this web site contact The Web Master.
Last updated: December 17, 2003