next up previous
Next: Unique Features of VisualSEEk Up: INTRODUCTION Previous: Spatial Image Query

Joint Content-based/Spatial Image Query

In this work, we propose a new system that provides for both feature comparison and spatial query for unconstrained color images. To illustrate, as depicted in Figure 1(b) and (c), each image is decomposed into regions which have feature properties, such as color, and spatial properties such as size, location and relationships to other regions. In this way, images are compared by comparing their regions. Furthermore, the system gives the user control in selecting the regions, see Figure 1(a), and parameters that are most important in determining similarity in a given query. As such, the system accommodates partial matching of images as determined by the user.

The most powerful type of image search system allows users to flexibly query for images by specifying both visual features and spatial properties of the desired images. Several recent content-based image query systems do not provide for both types of querying. The QBIC system [3] provides querying of whole images and manually extracted regions by color, texture and shape but not spatial relationships. The Virage system [10] allows querying of only image global features such as color, composition, texture and structure. Several new techniques have been proposed for injecting small amounts of spatial information into the image feature sets.

Related Work

A recent approach by Stricker and Dimai [2] divides each image into five fuzzy regions which contribute to the color moment representation of the image's color. The authors obtain compact feature sets and also allow the user to assign weights to the five spatial regions. In this way, the technique provides for querying by the five absolute region locations. However, it is not possible to query for images by specifying either arbitrary regions or the spatial relationships of regions.

Pass, Zabih and Miller [11] devised a technique which splits a global image histogram into coherent and scattered components. The measure of color coherence identifies the existence of connected colored regions. Although the technique improves on color histogram indexing, it does not support querying by the spatial locations of the color regions. Jacobs, Finkelstein and Salesin [12] devised an image match criteria and system which uses spatial information and visual features represented by dominant wavelet coefficients. Their system allows the user to sketch example images and provides for improved matching over image distance norms. However, their technique provides for little flexibility in specifying approximate and relative spatial information.

Our Approach

In order to fully integrate content-based and spatial image query capabilities we need to devise an image similarity function which contains both color feature and spatial components. We first note that perceived image similarity consists of both intrinsic and derived parameters. For example, the intrinsic part of a match refers to the similarity between query and target colors and/or region sizes and spatial locations. The derived part refers to the inferences that can be made from the intrinsic parameters, such as relative spatial locations and the overall assessment of image matches consisting of multiple regions.

  figure50
Figure 2:   Image query process. Indexing of intrinsic region features: color sets, spatial locations, sizes and spatial extents represented by minimum bounding rectangles. Computation of image matches and evaluation of spatial relationships.

 

figure54


Figure 3:   VisualSEEk system overview.

The joint color/spatial images query strategy is summarized in Figure 2. In order to quickly process queries, we design the representations for the intrinsic parameters such as region color, spatial location and size to require minimal computation in matching. For example, color matching is achieved efficiently through color sets. The intrinsic parameters are indexed directly to allow for maximal efficiency in queries. This query process identifies candidate regions which are combined to determine image matches.

In this way, a query specified by the user is translated directly into pruning operations on intrinsic parameters. The derived parameters, such as region relative locations and special spatial relations are resolved only in the final stage of the query. This is because these evaluations have the highest complexity. The pruning performed by the queries on the intrinsic parameters reduce the number of candidates images that need to be evaluated at the final stages.


next up previous
Next: Unique Features of VisualSEEk Up: INTRODUCTION Previous: Spatial Image Query

John Smith
Wed Sep 18 11:16:33 EDT 1996