Summary 

This project has reviewed research on the understanding and construction of intelligent entities and systems, respectively, in the fields of psychology, artificial intelligence, semiotics and computer vision. Based on these findings, we have defined the novel MediaNet framework that uses multimedia to represent semantic and perceptual knowledge about the world.

In addition, we have proposed automatic techniques for discovering perceptual and semantic knowledge from annotated image collection by integrating both the processing of images and text. Perceptual knowledge is constructed by clustering the images based on both visual and text feature descriptors, and by discovering statistical and similarity relationships between the clusters. Semantic knowledge is constructed by disambiguating the senses of words in annotations using WordNet and the images clusters, and by finding semantic relations between the detected senses in WordNet.

We have also developed novel automatic techniques for interrelating, summarizing and evaluating the discovered multimedia knowledge. Interdependence among discovered concepts are used to construct Bayesian networks for probabilistic inferencing; knowledge is summarized by grouping similar concepts together; and the conciseness, completeness and consistency of knowledge is evaluated using notions from graph and information theory.

The automatically discovered knowledge also provides a systematic and efficient way in browsing and navigating through the concepts and concepts present in the image collection.

People

In collaboration with Dr. Alex Loui of Eastman Kodak

Contact: Ana B. Benitez, Prof. Shih-Fu Chang

Publications

A. B. Benitez and S.-F. Chang, Organization and Browsing of Annotated Images Using Multiresolution Knowledge Networks, Columbia University ADVENT Technical Report #007, 2002.

A. B. Benitez and S.-F. Chang, Multimedia Knowledge Integration, Summarization and Evaluation, Proceedings of the 2002 International Workshop on Multimedia Data Mining in conjuction with the International Conference on Knowledge Discovery & Data Mining (MDM/KDD-2002), Edmonton, Alberta, Canada, July 23-26, 2002; also Columbia University ADVENT Technical Report #003, 2002. 
(Paper: PDF)

A. B. Benitez and S.-F. Chang, Semantic Knowledge Construction from Annotated Image Collections, Proceedings of the 2002 International Conference On Multimedia & Expo (ICME-2002), Lausanne, Switzerland, Aug 26-29, 2002; also Columbia University ADVENT Technical Report #002, 2002. 
(Paper: PDF)

A. B. Benitez and S.-F. Chang, Perceptual Knowledge Construction from Annotated Image Collections, Proceedings of the 2002 International Conference On Multimedia & Expo (ICME-2002), Lausanne, Switzerland, Aug 26-29, 2002; also Columbia University ADVENT Technical Report #001, 2002. 
(Paper: PDF)

A. B. Benitez, S.-F. Chang, and J. R. Smith, IMKA: A Multimedia Organization System Combining Perceptual and Semantic Knowledge, Proceeding of the 2001 ACM International Conference on Multimedia (ACM MM-2001), Canada, Ottawa, Sep 30-Oct 5, 2001. 
(Paper: PDF)

A. B. Benitez and J. R. Smith, New Frontiers for Intelligent Content-Based Retrieval, Proceedings of the 2001 SPIE Conference on Storage and Retrieval for Media Databases (IS&T/SPIE-2001), Vol. 4315, San Jose, CA, Jan 24-26, 2001. 
(Paper: PDF)

A. B. Benitez, J. R. Smith, and S.-F. Chang, MediaNet: A Multimedia Information Network for Knowledge Representation, Proceedings of the 2000 SPIE Conference on Internet Multimedia Management Systems (IS&T/SPIE-2000), Vol. 4210, Boston, MA, Nov 6-8, 2000. 
(Paper: PDF)

For problems or questions regarding this web site contact The Web Master.
Last updated: July 9, 2003.