LSCOM Lexicon Definitions and Annotations

DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia

Quick Guide to LSCOM Data Sets

  1. LSCOM: a collection of annotations for 449 concepts.

    Download the LSCOM annotation data. (103 MB file. Expands to 2.54 GB on disk.)

    LSCOM Citation: LSCOM Lexicon Definitions and Annotations Version 1.0, DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report #217-2006-3, March 2006. [pdf]

    Lexicon information. (Detailed list of annotated concepts.)


  2. LSCOM-Lite: an overlapping precursor to LSCOM with annotations for 39 concepts.

    Download the LSCOM-lite annotation data. (16 MB file.)

    LSCOM-Lite Citation: M. R. Naphade, L. Kennedy, J. R. Kender, S.-F. Chang, J. R. Smith, P. Over, and A. Hauptmann, “A Light Scale Concept Ontology for Multimedia Understanding for TRECVID 2005,” IBM Research Technical Report, 2005. [pdf]


  3. LSCOM Revised Event/Activity Annotations: video-based re-labeling of 24 LSCOM concepts.

    Download the LSCOM Revised Event/Activity annotations. (236 KB file.)

    LSCOM Revised Event/Activity Annotations Citation: Lyndon Kennedy, Revision of LSCOM Event/Activity Annotations, DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report #221-2006-7, December 2006. [pdf]

Summary

The DTO sponsored LSCOM workshop has developed an expanded multimedia concept lexicon on the order of 1000. Concepts related to events, objects, locations, people, and programs have been selected following a multi-step process involving input solicitation, expert critiquing, comparison with related ontologies, and performance evaluation. Participants of the process include representatives from intelligence community users, ontology specialists, and multimedia analytics researchers. In addition, each concept has been qualitatively assessed according to some criteria, such as utility (usefulness), observability (by humans), and feasibility (by automatic detection). An annotation process was completed in late 2005 by student annotators at Columbia University and CMU, over the entire development set of TRECVID 2005 videos. Human subjects judge the presence or absence of each concept in the key frame of each subshot, resulting in a total of 61901 labels for each concept.

The first version of the LSCOM annotations [3] consist of keyframe-based labels for 449 visual concepts, out of the 834 initial selected concepts, over the entire TRECVID 2005 development set (61901 subshots).

The LSCOM-Lite annotations [1] include 39 high-level features (concepts), which are interim results from the effort in developing a Large-Scale Concept Ontology for Multimedia (LSCOM). Most of the concepts in LSCOM-Lite overlap with the concepts in LSCOM; however, some concepts in LSCOM-Lite are not in LSCOM. The concepts were selected based on semi-automatic mapping of 26377 noun search terms from BBC query logs in late 1998 to WordNet senses, division of semantic concept space into a small number of orthogonal dimensions, and evaluation of 2003 and 2004 TRECVID search topics. The dimensions consist of program category, setting/scene/site, people, object, activity, event, and graphics. A collaborative effort among participants in the TRECVID 2005 benchmark was completed in the summer of 2005 to produce annotations of the 39 concepts over the entire development set of TRECVID 2005 videos. Human subjects judge the presence or absence of each concept in the key frame of each subshot, resulting in a total of 61901 labels for each concept. Ten of the LSCOM-Lite concepts have been chosen for evaluation in the TRECVID 2005 high-level feature detection task and 20 LSCOM-lite concepts were evaluated at TRECVID 2006 [2].

The Revised Event/Activity annotations [4] were conducted on 24 concepts, which contained a temporal component. These concepts were originally annotated in the LSCOM v1.0 release using single keyframes for each shot. Since some concepts require motion, this approach gives unreliable results, so this subset of concepts was re-annotated by having human subjects watch the actual video clips, instead of just viewing single keyframes.

This page shows the listing and definitions for the 856 concepts included in LSCOM Lexicon Version 1.

Annotation Data

Download the LSCOM annotation data
(103 MB file. Expands to 2.54 GB on disk.)

Download the LSCOM-lite annotation data
(16 MB file.)

Download the LSCOM Revised Event/Activity annotations
(236 KB file.)

Concept List

Click here for information on the lexicon, including a list of concepts and there annotations.

Annotation Leads

Workshop Co-PIs

Acknowledgements

The LSCOM project has been sponsored by the Disruptive Technology Office (DTO). This material is based upon work funded in whole by the U.S. Government. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Government.

References

[1] M. R. Naphade, L. Kennedy, J. R. Kender, S.-F. Chang, J. R. Smith, P. Over, and A. Hauptmann, "A Light Scale Concept Ontology for Multimedia Understanding for TRECVID 2005, " IBM Research Technical Report, 2005.

[2] NIST TREC Video Retrieval Evaluation. http://www-nlpir.nist.gov/projects/trecvid/

[3] LSCOM Lexicon Definitions and Annotations Version 1.0, DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report #217-2006-3 , March 2006.

[4] Lyndon Kennedy, Revision of LSCOM Event/Activity Annotations, DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report #221-2006-7 , December 2006.

[5] M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, J. Curtis, "Large-Scale Concept Ontology for Multimedia, " IEEE Multimedia Magazine, 13(3), 2006.