Answer Quality Characteristics and Prediction on an Academic Q&A Site: A Case Study on ResearchGate

Lei Li, Daqing He, Wei Jeng,Spencer Goodwin, Chengzhi Zhang (IW3C2-2015)
http://dl.acm.org/citation.cfm?id=2742129
The paper “Answer Quality Characteristics and Prediction on an Academic Q&A Site: A Case Study on ResearchGate” presents a study on ResearchGate, an online social Q&A site. Social Q&A sites are platforms, which allow users (1) interact with others through asking questions and providing answers, and (2) evaluate others’ contributions. Therefore, the quality of user-generated content in such sites are very important; especially, in academic Q&A sites, some questions do not have widely accepted right answers; scholars engage in the discussions to explore knowledge. The answer quality usually is judged in a peer-based fashion; scholars recommend the answer to the question.  Through the study, Lie et al. show the relationship between answer characteristics and answer quality based on peer judgments on ResearchGate, and use useful characteristics as features to predict the answer quality.
Lie et al. present a framework for their study. First, they collect data from 107 Q&A threads in the three disciplines (i.e., LIS, History of Art, Astrophysics), including 1128 posts. After, they generate answer characteristics and answer quality score based on peer judgments (i.e., answer vote information). The answer characteristics include:
  • Answer web-capture features: the responder’s participatory scores (named RG score on the site), responder’s publication impacts (called impact points), responder’ institution participatory scores and publication impacts, answer length, and response time
  • Answer human-coded features: social elements, consensus building (i.e., agreement), and providing factual information, resources, opinion, personal experience or reference to other researchers.
The quality label for each question is based on the number of votes. They categorize all the questions into three groups: 505 low quality answers (0 upvote), 438 medium quality answers (1 or 2 upvotes), and 78 high quality answers (> 3 upvotes).
After constructing the final answer data set, they first analyze the relationship between answer characteristics and answer quality. They found out that the most important features are longer answers, answers which were provided earlier, responders with higher RG score, and researchers from an institution with a higher RG score. Other interesting findings are social element which decreases the quality of an answer, and scholars’ impact points which do not contribute much to the quality of an answer. Based on this analysis, Lei et al. build several classification models to predict the answer quality, which are logistic regression, Naïve Bayes, and SVM. For SVM, they compare the experimental results of 4 kernel functions to choose the best one. They use the metrics AUC, precision, recall and F1 to evaluate the models, and run 10-fold cross-validation. The results show that:
  • SVM classification algorithm with RBF kernel function performs best.
  • Contrastive analysis of three prediction models: the optimized SVM algorithm has a better performance compared to Naïve Bayes or logistic regression.
They also analyze the importance of human-coded features and web-captured features in the SVM model. Based on the results, they claim that web-captured features bring about better performance. Moreover, combining both kinds of features, the model performs best.
In this paper, Lei et al. present an analysis about the relationships between answer characteristics and answer quality. They found out some factors highly contributing to the quality of an answer such as responders with higher RG score, long answer, etc. Based on human-coded features and web-captured features presented in the paper, they are able to build prediction models for predicting answer quality. These results can help to design better academic Q&A sites such as ResearchGate. However, because some characteristics in this study are coded by human, the dataset used for building and testing prediction model is limited. Those features could be captured by machine. Therefore, the authors may improve the study by replacing human-coded features with machine-coded features. Additionally, the reputation of scholars who vote the answer should be a strong indicator for labelling the quality of the answer.

Comments

Post a Comment

Popular posts from this blog

Influences on Query Reformalution in Collaborative Web Search

Zhen Yue, Shuguang Han, Daqing He, and Jiepu Jiang ( Computer-2014 ) http://ieeexplore.ieee.org/abstract/document/6766167/ In the paper “Influences on Query Reformation in Collaborative Web Search”, Yue et al. investigate a study to analyze the influences of collaboration between members in a group for information seeking. They are motivated to study (1) the influence of collaboration on search query formulation and reformulation; (2) similarities and differences exist in terms of query reformulation for collaborative searches compared with individual searches; and (3) the nature of the query task influence query reformulation in collaborative information seeking (CIS). In order to run the study, Yue et al. used CollabSearch, a Web search system developed by their group for collaborative users. The system includes three frames: a topic statement page which shows the description about the search task a group want to conduct; a web search page which displays Google search results...
Read more

Personalized Access to Scientific Publications: from Recommendation to Explanation

Dario De Nart, Felice Ferrara, and Carlo Tasso ( UMAP-2013 ) http://link.springer.com/chapter/10.1007/978-3-642-38844-6_26 In the paper “Personalized Access to Scientific Publications: from Recommendation to Explanation”, Dario De Nart et al. present a Recommender and Explanation System (RES) that is able to recommend scientific publication and show explanation to users.  In order to do recommendation, Dario De Nart et al., firstly, use the The Dikpe Keyphrase extraction algorithm introduced by Pudota (2010) to extract keyphrases from papers. Each keyphrase has a weight called keyphraseness that reveals the several lexical and statistical indicators exploited in the extraction process. Higher is the keyphraseness, more relevant is the KP. For each paper in the collection, they represent it as a conceptual graph. Each node in the graph is a term broken down from a KP. Two nodes are connected when they appear in the same KP. For instance, the KP “document retrieval” is split int...
Read more

SoRec: Social Recommendation Using Probabilistic Matrix Factorization

Hao Ma, Haixuan Yang, Michael R. Lyu, Irwin King ( CIKM-08 ) http://dl.acm.org/citation.cfm?id=1458205 In this paper, Hao Ma et al. present an approach that help to deal with very large datasets and users who have made few ratings in recommender systems. Traditional methods (e.g., Pearson Correlation Coefficient and Cosine) assume that two users have rated at least some items in common so that they can compute the similarity of these users. Thus, memory-based and model-based collaborative filtering algorithms fail to find similarities of users who have never rated any items. In daily life, people usually ask their family or friends for recommendations of new movies, songs, books, restaurants or places to visit. Therefore, the authors propose a factor analysis approach based on probabilistic matrix factorization to solve the data sparsity and poor prediction accuracy problems by employing both users’ social network information and rating records. Different with traditional persp...
Read more