##
**Summary**

We propose a rank minimization method
to fuse the predicted confidence scores of multiple models, each of which is obtained
based on a certain kind of feature. Specifically, we convert each confidence score
vector obtained from one model into a pairwise relationship matrix, in which each
entry characterizes the comparative relationship of scores of two test samples.
Our hypothesis is that the relative score relations are consistent among component
models up to certain sparse deviations, despite the large variations that may exist
in the absolute values of the raw scores. Then we formulate the score fusion problem
as seeking a shared rank-2 pairwise relationship matrix based on which each original
score matrix from individual model can be decomposed into the common rank-2 matrix
and sparse deviation errors. A robust score vector is then extracted to fit the
recovered low rank score relation matrix. We formulate the problem as a nuclear
norm and l1 norm optimization objective function and employ the Augmented Lagrange
Multiplier (ALM) method for the optimization. Our method is isotonic (i.e., scale
invariant) to the numeric scales of the scores originated from different models.
We experimentally show that the proposed method achieves significant performance
gains on various tasks including object categorization and video event detection.

##
**Motivation**

**Late Fusion**: Combine the prediction scores of
multiple models.

**Issues**: (1) Scales of scores from the individual models may vary
a lot
(2) Scores from each model may contain noise and outliers

##
**Approach**

**Observation**:

(1) Preserve rank order relationship
among scores instead of absolute values;

(2) If we have a real-value matrix
such that
, we can find a rank-2 factorization of
such that
.

Figure 1. An illustration of our
proposed method

**Steps**:

(1) Convert each confidence score
vector into a pairwise rank relationship matrix to address the scale variance issue;

(2) Seek a shared rank-2 pairwise
matrix based on which each score matrix can be decomposed into the consistent rank-2
matrix and sparse errors;

(3) A robust score vector is extracted
to fit the recovered low rank score rank relation matrix.

##
**Problem Formulation**

Suppose we have a set of
pairwise comparative relationship matrices
, . . . ,
, where each
is constructed from the score vector
of the
th model.

Our robust late fusion is formulated
as follows:

##
**Experiments and
Results**

Experiments confirm that the proposed
method can robustly extract a rank-2 skew-symmetric matrix and sparse errors. Robust
late fusion achieves 5.5%, 6.6%, and 10.4% improvement in Oxford Flower 17, CCV,
and TRECVID.

*Oxford Flower 17*

Table 1. MAP comparison on Oxford
Flower 17 dataset, 5.5% gain over the best baseline

Figure 2. Visualization of the low
rank and sparse matrices obtained by our RLF method from seven different confidence
score vectors of Oxford Flower 17 dataset, each of which is generated by training
a binary classifier based on one feature. To ease visualization, we sample a 30×30 sub-matrix from each 340×340 matrix. Blue cells denote
the values above 0, purple cells denote the values below 0, and white cells denote
0 values. The obtained matrix
is skew-symmetric. This figure is best viewed in color.

*Columbia Consumer Video (CCV)
Dataset*

Figure 3. AP comparison of different
methods on CCV dataset, 6.6% gain over the best baseline

Figure 4. MAP comparison at variant
depths on CCV dataset

*TRECVID MED 2011*

Figure 5. AP comparison on TRECVID MED 2011,10.4% gain over the best baseline
Figure 6. MAP comparison at variant depths on TRECVID MED 2011

##
**People**

Guangnan Ye,
Dong Liu,
I-Hong Jhuo, Shih-Fu Chang

##
**Publications**

Guangnan Ye, Dong Liu, I-Hong Jhuo,
Shih-Fu Chang. Robust **Late Fusion with Rank Minimization**. In *IEEE International
Conference on Computer Vision and Pattern Recognition (CVPR)*, 2012. [pdf]
[poster] [supplementary
material]