Summary

Visual text is ubiquitous in videos. Detection and recognition of visual text can provide many important semantic features for video analysis. Visual text can be divided into two categories, scene text and overlay text. This project aims to accurately detect and recognize the overlay text in video frames using compressed domain features, statistical fusing of contextual knowledge, and machine learning methods. 

Text Detecting

For overlay text detection, we use texture and motion features extracted from DCT coefficients and motion vectors in MPEG video stream. Multiple hypothetical images are then generated by using color space quantization. Finally, layout analysis is carried out to eliminate false alarms. The system has shown superior performance when testing in TREC 2002 movie videos and broadcast news videos. The system is shown below.

Text Recognition

For text recognition, we developed a multimodal method fusing multiple information sources and language models to enhance text recognition performance. Currently, we fuse the information from closed caption streams and external linguistic corpus. The closed caption model is realized by a novel time-dependent probability model which takes into account the time distance between videotext word and closed caption word. The experiments showed increased performance by fusing multi-modal features.

We have applied the multi-modal statistical fusion technique to detection of caption text in low-resolution news video. The method achieves a significant performance gain, with word recognition rate of 76.8% and character recognition rate of 86.7%. By feeding back the recognition results to the detection module, in conjunction with temporal consistency checking, the proposed methods greatly improve the detection precision (from 70% to 92%) while maintaining a high recall rate.

People

Dongqing Zhang

Prof. Shih-Fu Chang 

Publication

D. Zhang, S.-F. Chang, A Bayesian Framework for Fusing Multiple Word Knowledge Models in Videotext Recognition, Proceeding of the Computer Vision and Pattern Recognition 2003, Madison, Wisconsin, June 16-22, 2003.
(PS.GZ/PDF)

DongQing Zhang, Belle L. Tseng, Ching-Yung Lin and Shih-Fu Chang, "Accurate Overlay Text Extraction for Digital Video Analysis", Proceeding of IEEE International Conference on Information Technology: Research and Education (ITRE 2003).

Dongqing Zhang, and Shih-Fu Chang, “Event Detection in Baseball Video Using Superimposed Caption Recognition”,  ACM Multimedia 2002, Juan Les Pins, France, December 1-6, 2002. (ACM MM 2002).

Dongqing Zhang, and Shih-Fu Chang, “General and Domain-specific Techniques for Detecting and Recognizing Superimposed Text in Video”, Proceeding of International Conference on Image Processing, Rochester, New York, USA. (ICIP 2002). 

Links

DVMM Project: Sports Highlight Sumamrization Using Caption Text Recognition

Caption Text Detection in Mpeg Videos

For problems or questions regarding this web site contact The Web Master.
Last updated: June 12, 2002.