In this paper we investigate a new content-based image query system that enables querying by image regions and spatial layout. VisualSEEk is a hybrid system in that it integrates feature-based image indexing with spatial query methods. The integration relies on the representation of color regions by color sets. For one, color sets provide for a convenient system of automated region extraction through color set back-projection. Second, the color sets are easily indexed for retrieval of similar color sets. As a result, unconstrained images are decomposed into near-symbolic images which lend to efficient spatial query.
We present the strategies for computing queries that specify the colors, sizes and arbitrary spatial layouts of color regions, which include both absolute and relative spatial locations. We describe the extraction process and present the compact color set representation of regional color features. We also address several special case spatial queries involving adjacency, overlap and encapsulation of regions. Finally, we evaluate the VisualSEEk color/spatial queries and demonstrate the improved image query capabilities over non-spatial content-based approaches.