MPEG-7 Standardization Project at Columbia

Project's Home Page | Current Research Areas > Standards, Testbed, and Evaluation >

Summary

The MPEG-7 standard, also known as "Multimedia Content Description Interface," provides a standardized set of technologies for describing different aspects of multimedia such as the structure and the semantics of the content. The standard addresses a broad spectrum of multimedia applications and requirements by providing a metadata system for describing the features of multimedia content. MPEG-7 will improve current multimedia applications and enable new exciting ones; some examples are distributed processing, content exchange, personalized representation, and efficient storage/retrieval of multimedia content.

The following are specified in this standard:

Description Schemes (DS) describe entities or relationships pertaining to multimedia content. Description Schemes specify the structure and semantics of their components, which may be Description Schemes, Descriptors, or datatypes.
Descriptors (D) describe features, attributes, or groups of attributes of multimedia content.
Datatypes are the basic reusable datatypes employed by Description Schemes and Descriptors
Systems tools support delivery of descriptions, multiplexing of descriptions with multimedia content, synchronization, file format, and so forth.

This standard is subdivided into eight parts:

Part 1 - Systems: specifies the tools for preparing descriptions for efficient transport and storage, compressing descriptions, and allowing synchronization between content and descriptions.
Part 2 - Description definition language: specifies the language for defining the standard set of description tools (DSs, Ds, and datatypes) and for defining new description tools.
Part 3 - Visual: specifies the description tools pertaining to visual content.
Part 4 - Audio: specifies the description tools pertaining to audio content.
Part 5 - Multimedia description schemes: specifies the generic description tools pertaining to multimedia including audio and visual content.
Part 6 - Reference software: provides a software implementation of the standard.
Part 7 - Conformance testing: specifies the guidelines and procedures for testing conformance of implementations of the standard.
Part 8 - Extraction and use of MPEG-7 descriptions: provides guidelines and examples of the extraction and use of descriptions.

Our group at Columbia University has greatly contributed to the the Multimedia Description Schemes (MDS) part and the Reference Software part of the MPEG-7 standard. In particular, we have actively contributed to the development and specification of MPEG-7 MDS tools for content management, description and organization. First, we proposed self-describing schemes and object-based multimedia description schemes to MPEG-7 for describing images, videos, multimedia, home media and collections of multimedia. The main components of these description tools were objects (e.g., image regions, video segments, semantic objects, etc), and hierarchies and graphs relating the objects. We validated the proposed description schemes for MPEG-7 with numerous core experiments. We participated in the definition, development and edition of several MPEG-7 description tools, in particular, tools for describing the semantics, one multimedia document, a collection of multimedia documents, and the content of multimedia, respectively.

People

Student Researchers: Ana B. Benitez, Alex Jaimes, Seungyup Paek

Faculty: Prof. Shih-Fu Chang

Contact: Ana B. Benitez, Prof. Shih-Fu Chang

Demos

Some of our demos related to MPEG-7 follow.

Two Image Search Engines were developed to show the usefulness of the Multi-Level Indexing Pyramid, which classifies visual attributed into 10 levels. This was one of the results of the CE on the Multi-Level Indexing Pyramid.

The Image Collection Browsing And Searching System is based on our Collection Structure Description Scheme (previously known as Multimedia Collection (Archive) Description Scheme). This was one of the results of our CE on Collection Structure DS.

The Video Retrieval Engine using the Syntactic and the Semantic DS in the Generic DS. The demo was one of the results on our VE on Syntactic/Semantic DSs.

The Home Media Browsing System is based on our Home Media Description Scheme.

The Metasearch Engine for Visual Content makes use of our Collection (Archive) Description Scheme and Code Downloading capabilities.

Publications

Some of our research papers related to MPEG-7 follow.

A. B. Benitez, H. Rising, C. Jörgensen, R. Leonardi, A. Bugatti, K. Hasida, R. Mehrotra, A. Murat Tekalp, A. Ekin, T. Walker, Semantics of Multimedia in MPEG-7, Proceedings of 2002 IEEE Conference on Image Processing (ICIP-2002), Rochester, New York, USA, Sep. 22-25, 2002.
(Paper: PDF)

A. B. Benitez, J. M. Martinez, H. Rising and P. Salembier, Description of a Single Multimedia Document, Book chapter in "Introduction to MPEG 7: Multimedia Content Description Language" edited by B. S. Manjunath, Phillipe Salembier and Thomas Sikora, Wiley, 2002.
(Paper: PDF)

J. R. Smith and A. B. Benitez, Content Organization, Book chapter in "Introduction to MPEG 7: Multimedia Content Description Language" edited by B. S. Manjunath, Phillipe Salembier and Thomas Sikora, Wiley, 2002.
(Paper: PDF)

A. B. Benitez, D. Zhong, S.-F. Chang, and J. R. Smith, MPEG-7 MDS Content Description Tools and Applications, Proceedings of the International Conference on Computer Analysis of Images and Patterns (CAIP-2001), Warsaw, Poland, Sep 5-7, 2001.
(Paper: PDF)

A. B. Benitez, S.-F. Chang, Validation Experiments on Structural, Conceptual, Collection, and Access Description Schemes for MPEG-7, Digest of the IEEE 2000 International Conference on Consumer Electronics (ICCE-2000), Invited Paper on Special Session on MPEG-7, Los Angeles, CA, June 13-15, 2000.
(Paper: PDF)

A. B. Benitez, S. Paek, S.-F. Chang, Object-Based Multimedia Description Schemes and Applications for MPEG-7, To appear in the Image Communications Journal, Invited Paper on a Special Issue on MPEG-7, 2000.
(Paper: PDF)

S. Paek, A. B. Benitez, and S.-F. Chang, Self-Describing Schemes for Interoperable MPEG-7 Multimedia Content Descriptions, Symposium on Electronic Imaging: Visual Communications and Image Processing, IST/SPIE'99, San Jose, CA, Jan 1999.
(Paper: PDF)

Some of our contributions to MPEG-7 follow.

A. Benitez, H. Rising, C. Joergensen, R. Leonardi, A. Bugatti, K. Hasida, A. Vasile, W. Bender, "Report of CE on Semantic DS", ISO/IEC JTC1/SC29/WG11 MPEG01/M7009, Singapore, Singapore, March 2001.

A. Benitez, H. Rising, A. Bugatti, C. Joergensen, R. Leonardi, K. Hasida, "Report of CE on Semantic DS", ISO/IEC JTC1/SC29/WG11 MPEG00/M6498, La Baule, France, Oct. 2000.

A. Benitez, S.-F. Chang, "Segment Relations for MPEG-7", ISO/IEC JTC1/SC29/WG11 MPEG00/M6497, La Baule, France, Oct. 2000.

A. Benitez, A. Jaimes, C. Joergensen, S.-F. Chang, "Report of CE on Multilevel Indexing Pyramid", ISO/IEC JTC1/SC29/WG11 MPEG00/M6495, La Baule, France, Oct. 2000.

A. Bugatti, A. Benitez, R. Mehrotra, K. Hasida, H. Rising, C. Joergensen, R. Leonardi, E. Hartley, M. Tekalp, "Report of CE on Semantic DS", ISO/IEC JTC1/SC29/WG11 MPEG00/M6355, Beijing, China, July 2000.

J. R. Smith, B. Lugeon, A. Benitez, "Report on Core Experiment on Package DS", ISO/IEC JTC1/SC29/WG11 MPEG00/M6300, Beijing, China, July 2000.

M. Shibata, A. Tam, C. Leung, K. Hasida, A. Benitez, A. Jaimes"Report of CE on Structured Textual Description", ISO/IEC JTC1/SC29/WG11 MPEG00/M6240, Beijing, China, July 2000.

T. Walker, H. Rising, A. Benitez, K. Hasida, A. Divakaran, "Report on the CE on Entity Relationship Graph DSs", ISO/IEC JTC1/SC29/WG11 MPEG00/M6236, Beijing, China, July 2000.

A. Jaimes, A. B. Benitez, S.-F. Chang, "Multiple Level Classification of Audio Descriptors", ISO/IEC JTC1/SC29/WG11 MPEG00/M6114, Geneva, Switzerland, May/June 2000.

A. Jaimes, C. Jorgensen, A. B. Benitez, S.-F. Chang, "Report of CE on Multi-Level Indexing Pyramid", ISO/IEC JTC1/SC29/WG11 MPEG00/M6113, Geneva, Switzerland, May/June 2000.

P. Kuhn, T. Suzuki, A. Vetro, J. R. Smith, A. B. Benitez, C. Christopoulos, "Report on the Transcoding Hint DS Core Experiment", ISO/IEC JTC1/SC29/WG11 MPEG00/M6002, Geneva, Switzerland, May/June 2000.

A. Benitez, J. R. Smith, J. Chicharo, A. Perkis, S. Sull, C. Christopoulos, T. Suzuki"Report on Core Experiment on Media Transcoding Hint DS", ISO/IEC JTC1/SC29/WG11 MPEG00/M5803, Noordwijkerhout, NL, March 2000.

A. B. Benitez, J. R. Smith, H. Rising, P. Porskamp, D. Velthausz, "Report on Core Experiment on Collection Structure DS", ISO/IEC JTC1/SC29/WG11 MPEG00/M5802, Noordwijkerhout, NL, March 2000.

A. Jaimes, C. Jorgensen, A. B. Benitez, S.-F. Chang, "Experiments for Multiple Level Classification of Visual Descriptors", ISO/IEC JTC1/SC29/WG11 MPEG99/M5593, Maui, Hawaii, USA, Dec. 1999.

A. B. Benitez, D. Zhong, S.-F. Chang, C. Saraceno, M. Kangas, B. J. Brown, "Report on the Validation Experiment on the Syntactic DS, the Semantic DS, and the Syntactic/Semantic Link DS", ISO/IEC JTC1/SC29/WG11 MPEG99/M5432, Maui, Hawaii, USA, Dec. 1999.

A. Divakaran, A. B. Benitez, "Properties of Relations in Entity-Relation Graphs", ISO/IEC JTC1/SC29/WG11 MPEG99/M5376, Maui, Hawaii, USA, Dec. 1999.

A. Jaimes, C. Jorgensen, A. B. Benitez, S.-F. Chang, "Multiple Level Classification of Visual Descriptors in the Generic AV DS", ISO/IEC JTC1/SC29/WG11 MPEG99/M5251, Melbourne, Australia, Oct. 1999.

C. Saraceno, A. Lindsay, A. B. Benitez, S.-F. Chang, "Integration of Semantics in the Syntactic DS", ISO/IEC JTC1/SC29/WG11 MPEG99/M5227, Melbourne, Australia, Oct. 1999.

A. B. Benitez, S.-F. Chang, J. R. Smith, C.-S. Li, C. Saraceno, "Multimedia Archive Description Scheme", ISO/IEC JTC1/SC29/WG11 MPEG99/M5191, Melbourne, Australia, Oct. 1999.

A. B. Benitez, D. Zhong, S.-F. Chang, C. Saraceno, M. Kangas, B. J. Brown, "MPEG-7 DS Validation Experiment on the Syntactic DS, the Semantic DS, and the Syntactic/Semantic Link DSs", ISO/IEC JTC1/SC29/WG11 MPEG99/M5134, Melbourne, Australia, Oct. 1999.

A. B. Benitez, S. Paek, S.-F. Chang, J. R. Smith, C.-S. Li, A. Puri, Q. Huang, "Review of the Generic AV DS", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4918, Vancouver, Canada, July 1999.

J. R. Smith, P. Brigger, C.-S. Li, A. B. Benitez, S.-F. Chang, "Conceptual Modeling of MPEG-7 Description Schemes", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4775, Vancouver, Canada, July 1999.

A. B. Benitez, A. Jaimes, S. Paek, S.-F. Chang, "Fundamental Entity-Relation Models for a Multimedia Archive DS", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4755, Vancouver, Canada, July 1999.

A. B. Benitez, A. Jaimes, S.-F. Chang, J. R. Smith, C.-S. Li, "Fundamental Entity-Relation Models for the Generic Audio Visual DS", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4754, Vancouver, Canada, July 1999.

J. R. Smith, C.-S. Li, A. Puri, "Model Description Scheme for MPEG-7", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4750, Vancouver, Canada, July 1999.

J. R. Smith, C.-S. Li, R. Mohan, A. Puri, C. Christopoulos, A. B. Benitez, P. Bocheck, S.-F. Chang, T. Ebrahim, V.V. Vinod, "MPEG-7 Content Description for Universal Multimedia Access", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4749, Vancouver, Canada, July 1999.

J. M. Martinez, A. B. Benitez, "A proposal for the Media and Meta information DSs from the Generic AV DS", Contribution to ISO/IEC JTC1/SC29/WG11 MPEG99/M4700, Vancouver, Canada, July 1999.

A. B. Benitez, S. Paek, S.-F. Chang, and C.-S. Li, "Proposal for MPEG-7 Archive Description Scheme", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P482, Lancaster, U.K., Feb 1999.

S. Paek, A. B. Benitez, S.-F. Chang, A. Eleftheriadis, A. Puri, Q. Huang , C.-S. Li, J. R. Smith, and L. D. Bergman, "Proposal for MPEG-7 Video Description Scheme", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P481, Lancaster, U.K., Feb 1999.

S. Paek, A. B. Benitez, S.-F. Chang, C.-S. Li, J. R. Smith, L. D. Bergman, A. Puri, C. Swain, and J. Ostermann, "Proposal for MPEG-7 Image Description Scheme", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P480, Lancaster, U.K., Feb 1999.

A. B. Benitez, S. Paek, S.-F. Chang, C. Judice, and A. Puri, "Proposal for MPEG-7 Home Media Description Scheme", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P479, Lancaster, U.K., Feb 1999.

Q. Huang, A. Puri, A. B. Benitez, S. Paek, and S.-F. Chang, "Proposal for MPEG-7 Integration Description Scheme for Multimedia Content", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P477, Lancaster, U.K., Feb 1999.

Q. Huang, A. Puri, A. B. Benitez, S. Paek, and S.-F. Chang, "Proposal for MPEG-7 Integration Description Scheme for Multimedia Content", Proposal to ISO/IEC JTC1/SC29/WG11 MPEG99/P477, Lancaster, U.K., Feb 1999.

For problems or questions regarding this web site contact The Web Master.
Last updated: July 9, 2003.