[Talk Summary 12] A/B Testing at Scale

Dr. Pavel Dmitriev, a Principal Data Scientist, from Microsoft's Analysis and Experimentation team had a talk about "A/B Testing at Scale" on Thursday, 2016/12/08. The talk was about the introduction of a controlled experiment, four real experiments that Microsoft had been running, and 5 challenges about testing at scale.

Dr. Pavel started the talk with a brief introduction of controlled experiments, aka A/B tests. A/B testing is a method of comparing two versions of a webpage or app against each other to determine which one performs better. A/B testing is also used to evaluate a new feature of an application. If the feature has an effect on users, the result will show the significant difference (p<0.05); the lack of different is called null hypothesis.

With the evolving product development process, Dr. Pavel presented the motivation for A/B testing. In classical software development, a product is usually designed, developed, tested and then released. However, in customer-driven development, the process is from "build" to "measure" to learn (continuous deployment cycles), because we are poor at assessing the value of ideas. There experiment and get the data can help us to evaluate the value of ideas. To demonstrate four real experiments that Microsoft had been running, he showed the experiments and asked the attendances to choose which design between A and B will win. By doing that, he made some statistics on how different between the two groups to show whether the two designs are significant or not.

Finally, Dr. Pavel claimed that while the theory of experimentation is well established, scaling experimentation to millions of users, devices, platforms, websites, apps, social networks, etc. presents new challenges of A/B testing:
  • Challenge 1: trustworthiness
  • Challenge 2: protecting the users
  • Challenge 3: the OEC (Overall Evaluation Criterion)
  • Challenge 4: violations of classical assumptions of a controlled experiment
  • Challenge 5: analysis of results
    • NHST = Null Hypothesis Testing
    • Heterogeneous treatment effect
12/08/2016
Information Sciences Building, 3rd Floor
University of Pittsburgh


Comments

Post a Comment

Popular posts from this blog

Personalized Access to Scientific Publications: from Recommendation to Explanation

Dario De Nart, Felice Ferrara, and Carlo Tasso ( UMAP-2013 ) http://link.springer.com/chapter/10.1007/978-3-642-38844-6_26 In the paper “Personalized Access to Scientific Publications: from Recommendation to Explanation”, Dario De Nart et al. present a Recommender and Explanation System (RES) that is able to recommend scientific publication and show explanation to users.  In order to do recommendation, Dario De Nart et al., firstly, use the The Dikpe Keyphrase extraction algorithm introduced by Pudota (2010) to extract keyphrases from papers. Each keyphrase has a weight called keyphraseness that reveals the several lexical and statistical indicators exploited in the extraction process. Higher is the keyphraseness, more relevant is the KP. For each paper in the collection, they represent it as a conceptual graph. Each node in the graph is a term broken down from a KP. Two nodes are connected when they appear in the same KP. For instance, the KP “document retrieval” is split into tw
Read more

FolkTrails: Interpreting Navigation Behavior in a Social Tagging System

Thomas Niebler, Martin Becker, Daniel Zoller, Stephan Doerfel, & Andreas Hotho ( CIKM-2016 ) http://dl.acm.org/citation.cfm?doid=2983323.2983686 In the paper “FolkTrails: Interpreting Navigation Behavior in a Social Tagging System”, Niebler et al. present an investigate navigation trails in the popular scholarly social tagging system BibSonomy. They tried to understand users’ dynamic browsing behaviors on this kind of systems, how individual users behave differently on navigation, and whether the semantic nature of the underlying folksonomy can help to explain navigation. Firstly, Niebler et al. study different hypotheses about the navigational user behavior in Bibsonomy, a social bookmarking system. They test these hypotheses by using HypTrails, an approach utilizing first-order Markov chain models and Bayesian inference for expressing and comparing hypotheses about human trails. These hypotheses include: Uniform hypothesis: serves as a baseline, users randomly pick a l
Read more

Personalized Social Search Based on the User’s Social Network

David Carmel et al. - IBM Research Lab in Haifa, Israel ( CIKM-2009 ) http://dl.acm.org/citation.cfm?doid=1645953.1646109 Personalizing the search process considers the searcher’s personal attributes & preferences so that the system is able to, for example, do personalized query expansion, re-rank the result list or filter presented information. However, few studies consider the preferences of the user’s related people in social network which can be utilized to enrich the user’s preferences. In the paper “Personalized Social Search Based on the User’s Social Network”, Carmel et al. presents a study that leverages both aforementioned information resources to improve social search, which is the search process over “social” data (e.g., documents or people) gathered from Web 2.0 applications. Particularly, they plan to re-rank search results by considering the relationships with other users in the searcher’s social network (SN). Firstly, Carmel et al. build user profiles from t
Read more