Project Overview

Model 4

Trends towards More Visual Communication

As visual content such as images and video become pervasive on the Web and various forms of social media, there is growing interest in understanding how visual content influences outcomes of social communication online. Visual content is generally considered important in attracting user interest and eliciting responses in todays social media platforms. Particularly, in order to make messages viral, content conveying strong emotions is often used.


AICER is an automatic tool that can recognize visual attributes appearing in the picture content, predict the likely viewer responses, and suggest plausible comments to assist users in social communication.

A Novel Mid-Level Concept-Based Representation

Our approach emphasizes a mid-level concept representation, in which intended affects of the image publisher is characterized by a large pool of visual concepts (termed PACs) detected from image content directly instead of textual metadata, evoked viewer affects are represented by concepts (termed VACs) mined from online comments, and statistical methods are used to model the correlations among these two types of concepts. We demonstrate the utilities of such approaches by developing an end-to-end Assistive Comment Robot application, which further includes components for multi-sentence comment generation, interactive interfaces, and relevance feedback functions. Through user studies, we showed machine suggested comments were accepted by users for online posting in 90% of completed user sessions, while very favorable results were also observed in various dimensions (plausibility, preference, and realism) when assessing the quality of the generated image comments.

System Design

Given a new image without any textual keywords or descriptions, concept classifiers like SentiBank are used to detect PACs and generate a concept score vector, whose elements represent the confidence in detecting corresponding individual concepts (such as “misty woods” or “cute dog”). The detected PAC score vector is then fed to the statistical correlation model to predict the likely VACs that may be evoked on the viewer part. The detected PACs and VACs are then used jointly to select most suitable comments from the pre-synthesized database according to several systematic criteria such as plausibility, relevance, and diversity. The selected comments are then suggested to the user who can further edit the comment before posting.

Chrome Extension tool for FaceBook

AICER can be used as a Chrome extension tool supporting seamless use on Facebook website. While browsing friends’ pictures, user can ask for suggestions of creative comments by simply clicking on an embedded function button. User has full control in indicating preferred comments and editing the suggested comments before posting. The Chrome extension tool will be available soon.

- Official Release in Chrome Apps Store

- Developing Version [Package][Installation Guideline]

- User Manual

AICER Web Demo

Try out our web demo! Just upload your favorite image and see what AICER will suggest for comments.