Jump to : Download | Abstract | Contact | BibTex reference | EndNote reference |


Lyndon Kennedy, Shih-Fu Chang, Igor Kozintsev. To Search or To Label?: Predicting the Performance of Search-Based Automatic Image Classifiers. In Multimedia Information Retrieval Workshop (MIR), Santa Barbara, CA, USA, 2006.

Download [help]

Download paper: Adobe portable document (pdf)

Copyright notice:This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


In this work we explore the trade-offs in acquiring training data for image classification models through automated web search as opposed to human annotation. Automated web search comes at no cost in human labor, but sometimes leads to decreased classification performance, while human annotations come at great expense in human labor but result in better performance. The primary contribution of this work is a system for predicting which visual concepts will show the greatest increase in performance from investing human effort in obtaining annotations. We propose to build this system as an estimation of the absolute gain in average precision (AP) experienced from using human annotations instead of web search. To estimate the AP gain, we rely on statistical classifiers built on top of a number of quality prediction features. We employ a feature selection algorithm to compare the quality of each of the predictors and find that cross-domain image similarity and cross-domain model generalization metrics are strong predictors, while concept frequency and within-domain model quality are weak predictors. In a test application, we find that the prediction scheme can result in a savings in annotation effort of up to 75\%, while only incurring marginal damage (10\% relative decrease in mean average precision) to the overall performance of the concept models


Lyndon Kennedy
Shih-Fu Chang

BibTex Reference

   Author = {Kennedy, Lyndon and Chang, Shih-Fu and Kozintsev, Igor},
   Title = {To Search or To Label?: Predicting the Performance of Search-Based Automatic Image Classifiers},
   BookTitle = {Multimedia Information Retrieval Workshop (MIR)},
   Address = {Santa Barbara, CA, USA},
   Year = {2006}

EndNote Reference [help]

Get EndNote Reference (.ref)


For problems or questions regarding this web site contact The Web Master.

This document was translated automatically from BibTEX by bib2html (Copyright 2003 © Eric Marchand, INRIA, Vista Project).