Color sets provide a compact alternative to color histograms for representing color information. Their utilization stems from the conjecture that salient regions have not more than a few, equally prominent colors. The following paragraphs define color sets and explain their relationship to color histograms.
A key issue in defining the color representation is the choice of color space and how it is partitioned. Three dimensions of color can be defined and measured. For example, each image point can be represented as a 3-D vector 
in the RGB color space. The transformation 
and quantization 
of the RGB color space reorganizes and groups the vectors 
. Perceptually distinct colors correspond to the sets of vectors that are mapped to different indices m as diagrammed in Figure 4. Color sets are defined as follows: let 
be the M dimensional binary space such that each axis in corresponds to one unique index value m. A color set is a binary vector in which corresponds to a selection of colors 
.
Figure 4: Generation of the binary color space from color channels, color space transformation , quantization , histograms h[m], and color sets c[m].
For example, let transform RGB to HSV and let where M=8 quantize the HSV color space to 2 hues, 2 saturations and 2 values. The quantizer assigns a unique index m to each quantized HSV color. Then, 
is the eight dimensional binary space whereby each element in corresponds to one of the quantized HSV colors. A color set 
contains a selection from the eight colors. If the color set corresponds to a unit length binary vector, then one color is selected. If a color set has more than one non-zero value, then several colors are selected. For example, the color set 
corresponds to the selection of three colors, m=0, m=3 and m=5, from the quantized HSV color space.
For implementation in VisualSEEk, we choose to transform RGB to the HSV color space because HSV provides a breakdown of color into its most natural components: hue, saturation and intensity. We choose , as illustrated in Figure 5, to quantize HSV into M = 166 colors [13].
Figure 5: Transformation from RGB to HSV and quantization gives 18 hues, 3 saturations, 3 values and 4 grays = 166 colors.
We use a color set back-projection technique in order to extract color regions. We briefly describe the technique here and note that a more detailed presentation appears in [13]. The back-projection process requires several stages: color set selection, back-projection onto the image, thresholding and labeling. Candidate color sets are selected first with one color, then with two colors, etc., until the salient regions extracted.
The back-projection of a color set is accomplished as follows: given image I[x,y] and color set , let k be the index of the color at image point I[x,y], then generate image B[x,y] by
That is, B[x,y] depicts the back-projection of color set . In order to accommodate color similarity in the back-projection process, a correlated back-projection image can be generated by
where 
measures the similarity of colors j and k, which will be discussed in the next section. After back-projecting the model color set, image B[x,y] is filtered and analyzed to reveal spatially localized color regions. The process and back-projection results are illustrated for an example image in Figure 6.
Figure 6: Example back-projection using model color set at top left to extract color region from a target image.
Information about the regions such as the color set used for back-projection, the spatial location and size are added to the REGION relation (see Table 1) and are subsequently used for queries as explained in the next sections. While color set selection and back-projection provides one automated technique for extracting salient color regions from the images, other methods can easily be incorporated into our system. For example, manually extracted regions and their features can also be added to the REGION relation.
