Brain Machine Interface for Image Search

Back to Project List



Human visual perception is able to recognize a wide range of targets under challenging conditions, but has limited throughput. Machine vision and automatic content analytics can process images at a high speed, but suffers from inadequate recognition accuracy for general target classes. In this project, we are developing a new paradigm to explore and combine the strengths of both systems. A single trial EEG-based brain machine interface (BCI) subsystem is used to detect objects of interest of arbitrary classes from an initial subset of images. The EEG detection outcomes are used as input to a graph-based semi-supervised learning subsystem to identify, refine, and propagate the labels to retrieve relevant images from a much larger pool. The combined strategy is unique in its generality, robustness, and high throughput. It has great potential for advancing the state of the art in media retrieval applications. We have evaluated and demonstrated significant performance gains of the proposed system with multiple and diverse image classes over several data sets, including those from Internet (Caltech 101) and remote sensing images.



  • Brain State Decoding for Rapid Image Retrieval NSF Hybrid Neuro-Computer Vision Systems Workshop, April 19-20, 2010. Shih-Fu Chang [slide]


Jun Wang and Shih-Fu Chang.
(in collaboration with Prof. Paul Sajda and Eric Pohlmeyer of Biomedical Engineering Department)  


  1. Jun Wang, Eric Pohlmeyer, Barbara Hanna, Yu-Gang Jiang, Paul Sajda, Shih-Fu Chang. Brain State Decoding for Rapid Image Retrieval. In Proceeding of the ACM international conference on Multimedia (ACM MM), October 2009. [pdf]
  2. Paul Sajda, Eric Pohlmeyer, Jun Wang, Lucas C. Parra, Christoforou Christoforou, Jacek Dmochowski, Barbara Hanna, Claus Bahlmann, Maneesh K. Singh, Shih-Fu Chang. In a Blink of an Eye and a Switch of a transistor: Cortically Coupled Computer Vision. Proceedings of the IEEE, 98(3):462-478, 2010. [pdf]
  3. Eric Pohlmeyer, David Jangraw, Jun Wang, Shih-Fu Chang, Paul Sajda. Combining Computer and Human Vision into a BCI: Can the Whole Be Greater Than the Sum of Its Parts?. In The 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Buenos Aires, Argentina, August 2010. [pdf]
  4. Eric Pohlmeyer, Jun Wang, David Jangraw, Bin Lou, Shih-Fu Chang, Paul Sajda. Closing the loop in cortically-coupled computer vision: a brain-computer interface for searching image databases. Journal of Neural Engineering, 8, 2011. [pdf]