[Talk Summary 11] Explain and answer: Intelligent systems which can communicate about what they see

Dr. Marcus Rohrbach from University of California, Berkeley made a talk "Explain and answer: Intelligent systems which can communicate about what they see" on Friday, 2016//12/02. In the talk, Marcus presented the models which can answer questions but at the same time are modular and expose their semantic reasoning structure, and showed how to generate explanations given only image captions as training data.

 To begin the talk, Marcus showed the motivation of how to make the computer able to talk to about the visual world. He introduced two components in a successful communication: (1) the ability to answer natural language questions about the visual world, and (2) the ability of the system to explain in natural language, allowing a human to trust and understand it. to deal with tasks such as visual question answering, he emphasized that it is important to integrate the representation of textual and visual information together.

 Marcus described the whole process of the system by dividing it into several steps. The first step is tell what a picture is about, describing novel captions by capturing the objects in the picture that has no a prior description. The second step relates to natural language processing; it analyzes questions to understand what we want to ask. After that the system will locate the asked object in the picture to give an answer. 

 For example, there are a basket, a plate, a fork, a carrot on the plate, and an orange in the basket in a picture. The question is that "What is the fruit on the plate?". The system first locates the fruit on the plate and then recognizes what kind of fruit there.

 For the explanation task, Marcus presented an architecture for visual ground. His grounding approach is the fully supervised version of GroundeR. Additionally, he applied deep fine-grained classifier for visual information. Based on these features, the system is able to give more descriptive explanation for its answer. 

Two answers and explanations from the system:
(1) what is the bird doing?
      Walking.
      Because they are on the ground
(2) what is the bird?
      This is a Western Grebe because this bird has a long white neck, pointy yellow beak and red eye. 

12/02/2016
100 Porter Hall
Carnegie Mellon University  

 

Comments

Post a Comment

Popular posts from this blog

Personalized Access to Scientific Publications: from Recommendation to Explanation

Dario De Nart, Felice Ferrara, and Carlo Tasso ( UMAP-2013 ) http://link.springer.com/chapter/10.1007/978-3-642-38844-6_26 In the paper “Personalized Access to Scientific Publications: from Recommendation to Explanation”, Dario De Nart et al. present a Recommender and Explanation System (RES) that is able to recommend scientific publication and show explanation to users.  In order to do recommendation, Dario De Nart et al., firstly, use the The Dikpe Keyphrase extraction algorithm introduced by Pudota (2010) to extract keyphrases from papers. Each keyphrase has a weight called keyphraseness that reveals the several lexical and statistical indicators exploited in the extraction process. Higher is the keyphraseness, more relevant is the KP. For each paper in the collection, they represent it as a conceptual graph. Each node in the graph is a term broken down from a KP. Two nodes are connected when they appear in the same KP. For instance, the KP “document retrieval” is split into tw
Read more

[Talk Summary 2] Dynamic Information Retrieval Model

Image
In this seminar, prof essor Grace Hui Yang presents a dynamic information retrieval model which helps users explore the information space in order to find out which documents are relevant and which aren't, satisfying their information need. The dynamic IR task aims to find relevant documents for a session of multiple queries. It happens when information needs are complex, vague, evolving, often containing multiple subtopics. A dynamic system is one which changes or adapts over time, based on a sequence of events. While static IR does not learn directly from users and the parameters are updated periodically, dynamic IR is developed from interactive IR which exploits users' feedback to give recommendation or improve the search result or optimize ranking, which is called dynamic ranking principle. But interactive IR just treat interaction independently and response to immediate feedback. Dynamic IR tries to optimize over all interaction, uses it for long term gain and models
Read more

Personalized Social Search Based on the User’s Social Network

David Carmel et al. - IBM Research Lab in Haifa, Israel ( CIKM-2009 ) http://dl.acm.org/citation.cfm?doid=1645953.1646109 Personalizing the search process considers the searcher’s personal attributes & preferences so that the system is able to, for example, do personalized query expansion, re-rank the result list or filter presented information. However, few studies consider the preferences of the user’s related people in social network which can be utilized to enrich the user’s preferences. In the paper “Personalized Social Search Based on the User’s Social Network”, Carmel et al. presents a study that leverages both aforementioned information resources to improve social search, which is the search process over “social” data (e.g., documents or people) gathered from Web 2.0 applications. Particularly, they plan to re-rank search results by considering the relationships with other users in the searcher’s social network (SN). Firstly, Carmel et al. build user profiles from t
Read more