In this project, we collaborated
with colleagues in Teachers College to develop an intuitive real-time
image search system that supports non-textual query cues, such as rough
line sketchs and compositions of color regions.
On the technical side, we are
interested in new algorithms for extracting invariant edge features and
arbitrarily-shaped region features, and search methods for retrieving
images that satisfy the visual attributes in user's mental pictures. On
the pedagogical side, we are interested in developing new curricula that
allow young users (e.g., K-12 students) to easily access art images with
intuitive expression in terms of basic visual composition, without formal
knowledge of art history.
A sample sketch query and search
results are shown below.
The search component supports
queries in the form of rough line-sketches that outline the basic form
and composition of an image. This permits the user to quickly and easily
transform a mental picture of an image into a query. Characterizing archived
images with signatures that represents their strongest edges, and comparing
these to a dynamically generated signature of the user's sketch achieves
this search. The edge signature is generated over multiple scales to account
for the variation in detail of the user's line-sketch. Edge-coherence,
a measure of the perceptual strength of an edge, based on its continuity,
is used in generating the signature. Our search technique is fast enough
to allow the system to respond with results at the completion of each
stroke of a query sketch. The algorithm is robust and invariant to scale,
rotation and translation. We also show how this edge-based search can
be merged seamlessly with the more traditional color-region and shape-based
searches to produce a search interface that is expressive, intuitive,
simple, and works in the absence of sample images.
Kumar and Prof. Shih-Fu
Offline demo available.
Rajendran Kumar, S.-F. Chang,
Retrieval with Sketches and Compositions, IEEE International Conference
on Multimedia and Expo (ICME), New York, July 2000.
or questions regarding this web site contact The
June 12, 2002.