The analysis of emotion, affect and sentiment from visual content has become an exciting area in the multimedia community allowing to build new applications for brand monitoring, advertising, and opinion mining. There exists no corpora for sentiment analysis on visual content, and therefore limits the progress in this critical area. To stimulate innovative research on this challenging issue, we constructed a new benchmark and database (you can browse the database at VSO Browsing Interface). This database contains a Visual Sentiment Ontology (VSO) consisting of 3244 adjective noun pairs (ANP), SentiBank a set of 1200 trained visual concept detectors providing a mid-level representation of sentiment, associated training images acquired from Flickr, and a benchmark containing 603 photo tweets covering a diverse set of 21 topics.

This website provides the above mentioned material for download and is structured as the following:

Benchmark and Dataset Citation
Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel and Shih-Fu Chang. "Large-scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs," ACM Multimedia Conference, Barcelona, Oct 2013.

Update: Thanks for checking out VSO, you may also be interested in our new improved and multilingual version MVSO

For each ANP a concept detector has been trained using 80% of its Flickr sample images. The remaining 20% have been used for detector testing. Results can be seen in the Figures on the right and the bottom. 1553 detectors were evaluated and finally 1200 detectors (all have non-zero AP@20 and most have F-score > 0.6) were retained in the current release of SentiBank.

Concept detector training was performed
according the following setup:

Visual Features
The following visual features have been used:
1. RGB Color Histogram (3x256 dim.)
2. GIST descriptor [1] (512 dim.)
3. Local Binary Pattern [2] (52 dim.)
4. Bag-of-Words (1,000 dictionary
    2-layer spatial pyramid, max pooling)
5. Attributes [3] (2,000 dim.)
*The feature extraction code is presented in SentiBank v1.02 and higher.

Concept Detectors
Detectors have been trained using LibSVM [4]:
1. Linear SVM classifier
2. Optimization Measure:
    - AP@20

System Requirements
Matlab + C hybrid coding is used to implement
SentiBank in our current version. Windows platform
is recommended to run SentiBank. It is possible
(and validated) that our codes can be also run in
Linux platform with Matlab. However, you might
need to set up the -mex configuration to choose
appropriate C compliers.For a user guide on mex
configuration, see here.

Files for Download
1. binary code of SentiBank detectors
   (latest version: 1.1)
2. source code of SentiBank detectors
   (latest version: 1.1)
3. DeepSentiBank detectors
   See paper here
4. MVSO detectors (New!)
   See paper here

Fig.1 Comparison of ANP detectors using different features. Performance computed by avg. over 5 runs of the reduced testsets (SentiBank v1.02). Fig.2 AP@20 of ANP detectors using different fusion approaches. Performance computed by avg. over 5 runs of the reduced testsets (SentiBank v1.02). Fig.1 Comparison of ANP detectors using different features. Performance computed by avg. over 5 runs of the reduced testsets (SentiBank v1.1). Fig.2 AP@20 of ANP detectors using different fusion approaches. Performance computed by avg. over 5 runs of the reduced testsets (SentiBank v1.1). For Early Fusion, the average AP@20 of all ANPs increased from 47% (v1.02) to 63% (v1.1).

By downloading the SentiBank Code, you agree that 1) the use of the data is restricted to research or education purpose only, 2) all copyright and license restrictions associated with the dataset/code will be followed, and 3) the authors of the above paper and their affiliated organizations make no warranties regarding the database or software, including but not limited to non-infringement.

 

SentiBank's detector performance according to the above described implementation.
For more details of our baseline classifier design, please refer to the paper.

 

[1] A. Olivai and A. Torralba Modeling the Shape of the Scene: a Holistic Representation of the Spatial Envelope. IJCV, 2001 - online resource
[2] Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence. 24.7: 971-987, 2002 - online resource
[3] Felix, X. Yu, Liangliang Cao, Rogerio S. Feris, John R. Smith, and Shih-Fu Chang. Designing Category-level Attributes for Discriminative Visual Recognition. CVPR, 2013. - online resource
[4] LIBSVM: a Library for Support Vector Machines, CC Chang, CJ Lin - ACM Transactions on Intelligent Systems and Technology (TIST), 2011 - online resource