Search

Digital advertising platforms have emerged as a widely utilized data source in consumer research; yet, the interpretation of such data remains a source of confusion for many researchers. This article aims to address this issue by offering a comprehensive and accessible review of four prominent data collection methods proposed in the marketing literature: "informal studies," "multiple-ad studies without holdout," "single-ad studies with holdout," and "multiple-ad studies with holdout." By outlining the strengths and limitations of each method, we aim to enhance understanding regarding the inferences that can and cannot be drawn from the collected data. Furthermore, we present seven recommendations to effectively leverage these tools for programmatic consumer research. These recommendations provide guidance on how to use these tools to obtain causal and non-causal evidence for the effects of marketing interventions, and the associated psychological processes, in a digital environment regulated by targeting algorithms. We also give recommendations for how to describe the testing tools and the data they generate and urge platforms to be more transparent on how these tools work. [ABSTRACT FROM AUTHOR]

Online reviews play a crucial role in influencing seller–customer dynamics. This research evaluates the credibility and consistency of reviews based on volume, length, and content to understand the impacts of incentives on customer review behaviors, how to improve review quality, and decision-making in purchases. The data analysis reveals major factors such as costs, support, usability, and product features that may influence the impact. The analysis also highlights the indirect impact of company size, the direct impact of user experience, and the varying impacts of changing conditions over the years on the volume of incentive reviews. This study uses methodologies such as Sentence-BERT (SBERT), TF-IDF, spectral clustering, t-SNE, A/B testing, hypothesis testing, and bootstrap distribution to investigate how semantic variances in reviews could be used for personalized shopping experiences. It reveals that incentive reviews have minimal to no impact on purchasing decisions, which is consistent with the credibility and consistency analysis in terms of volume, length, and content. The negligible impact of incentive reviews on purchase decisions underscores the importance of authentic online feedback. This research clarifies how review characteristics sway consumer choices and provides strategic insights for businesses to enhance their review mechanisms and customer engagement. [ABSTRACT FROM AUTHOR]

In today's fast-paced lifestyle, streaming movies and series on platforms like Netflix is a valued recreational activity. However, users often spend considerable time searching for the right content and receive irrelevant recommendations, particularly when facing the "cold start problem" for new users. This challenge arises from existing recommender systems relying on factors like casting, title, and genre, using term frequency-inverse document frequency (TF-IDF) for vectorization, which prioritizes word frequency over semantic meaning. To address this, an innovative recommender system considering not only casting, title, and genre but also the short description of movies or shows is proposed in this study. Leveraging Word2Vec embedding for semantic relationships, this system offers recommendations aligning better with user preferences. Evaluation metrics including precision, mean average precision (MAP), discounted cumulative gain (DCG), and ideal cumulative gain (IDCG) demonstrate the system's effectiveness, achieving a normalized DCG (NDCG)@10 of 0.956. A/B testing shows an improved click-through rate (CTR) of recommendations, showcasing enhanced streaming experience. [ABSTRACT FROM AUTHOR]

Background: Increasing vaccination coverage was key to curbing the COVID-19 pandemic globally. However, lack of trust in the vaccine and fear of side effects in regions like the Caribbean resulted in a low uptake despite enough vaccine supply. Methods: We conducted two correlational analyses and one experiment between five sequential behaviorally informed Facebook campaigns, social media performance outcomes, and district-level vaccination data. First, we ran multivariate linear regression models to estimate the mean differences between the campaigns in (i) social media performance ("Clicks" and "Engagement") and (ii) COVID-19 vaccination uptake at the district level. "Clicks" were measured by the number of people who clicked on the respective Facebook advert and visited the official vaccination site. "Engagements" were the number of people interacting with the advert through likes and emojis. Second, we took advantage of the experimental design during one of the campaigns to analyze the differential effect of messages conveying information about the number of people reporting vaccination side effects using words ("Few"/ "Majority) and numbers ("3 out of 100 ") on social media performance. Results: The correlational analysis showed that the number of "Clicks" and "Engagement" was similar among campaigns, except for the campaign focusing on vaccines' effectiveness, which had 14.65 less clicks and 19.52 less engagements per advert (including controls and district-fixed effects) compared to the base "It's safe" campaign. Vaccination rates were highest at times coinciding with campaigns focusing on vaccination safety and effectiveness. Our experimental results showed that informational messages related to side effects that were framed using words ("Majority did not report discomfort"/ "Few persons reported discomfort") were better at generating "Clicks" compared to those using numbers ("3 out of 100 reported discomforts"). Conclusions: Facebook adverts highlighting vaccine safety had a similar level of social media performance as other campaigns, except for adverts focusing on vaccine efficacy, which performed worse. Communicating side-effect information with words instead of numbers can expand social media interest in low-uptake regions like the Caribbean. Our results serve as preliminary evidence for public health officials to encourage vaccine uptake in high-hesitancy contexts. [ABSTRACT FROM AUTHOR]

Many modern tech companies, such as Google, Uber, and Didi, use online experiments (also known as A/B testing) to evaluate new policies against existing ones. While most studies concentrate on average treatment effects, situations with skewed and heavy-tailed outcome distributions may benefit from alternative criteria, such as quantiles. However, assessing dynamic quantile treatment effects (QTE) remains a challenge, particularly when dealing with data from ride-sourcing platforms that involve sequential decision-making across time and space. In this article, we establish a formal framework to calculate QTE conditional on characteristics independent of the treatment. Under specific model assumptions, we demonstrate that the dynamic conditional QTE (CQTE) equals the sum of individual CQTEs across time, even though the conditional quantile of cumulative rewards may not necessarily equate to the sum of conditional quantiles of individual rewards. This crucial insight significantly streamlines the estimation and inference processes for our target causal estimand. We then introduce two varying coefficient decision process (VCDP) models and devise an innovative method to test the dynamic CQTE. Moreover, we expand our approach to accommodate data from spatiotemporal dependent experiments and examine both conditional quantile direct and indirect effects. To showcase the practical utility of our method, we apply it to three real-world datasets from a ride-sourcing platform. Theoretical findings and comprehensive simulation studies further substantiate our proposal. for this article are available online Code implementing the proposed method is also available at: . [ABSTRACT FROM AUTHOR]

This research focuses on developing LegitCheck's website interface by applying the Parallax Scrolling Effect technique, which is considered as a potential solution to support the startup's promotional strategy. LegitCheck, as a startup that needs an effective way to introduce its services and business concept to consumers and brands, requires a website that can serve as an attractive promotional medium as well as a comprehensive information center. The purpose of this research is to examine the effectiveness of the Parallax Scrolling Effect technique compared to a website without the effect in increasing the effectiveness of the LegitCheck website as a promotional tool. The research method used includes the design and implementation of the Parallax Scrolling Effect technique on the LegitCheck website, followed by evaluation using the A/B testing method. The results showed that the implementation of the Parallax Scrolling Effect technique provided significant benefits, including improved information accuracy, service quality, user satisfaction, and company net profit. Although there are constraints related to page loading speed, the majority of respondents prefer websites with Parallax Scrolling Effect because it provides a more interactive and engaging experience. The implication of this study is that the use of Parallax Scrolling Effect can be an effective strategy for startups such as LegitCheck to increase interaction with users, strengthen brand image, and increase the effectiveness of the website as a promotional tool, although it is necessary to pay attention to technical aspects such as page loading speed to minimize potential disruptions to the user experience. [ABSTRACT FROM AUTHOR]

Firms use A/B testing, a methodology to compare two or more versions of ideas to see which one performs better, to decide on features of their digital products and services. Although A/B testing brings benefits, some A/B testing can lead to digital exploitation (i.e., appropriating resources from users to increase firms' performance). We explicate why A/B testing can lead to digital exploitation, suggest mitigation options by establishing Institutional Review Boards, explicitly seeking users' consent, and implementing incentive schemes, and conclude with research directions on this topic. [ABSTRACT FROM AUTHOR]

Abstract Background Increasing vaccination coverage was key to curbing the COVID-19 pandemic globally. However, lack of trust in the vaccine and fear of side effects in regions like the Caribbean resulted in a low uptake despite enough vaccine supply. Methods We conducted two correlational analyses and one experiment between five sequential behaviorally informed Facebook campaigns, social media performance outcomes, and district-level vaccination data. First, we ran multivariate linear regression models to estimate the mean differences between the campaigns in (i) social media performance (“Clicks” and “Engagement”) and (ii) COVID-19 vaccination uptake at the district level. “Clicks” were measured by the number of people who clicked on the respective Facebook advert and visited the official vaccination site. “Engagements” were the number of people interacting with the advert through likes and emojis. Second, we took advantage of the experimental design during one of the campaigns to analyze the differential effect of messages conveying information about the number of people reporting vaccination side effects using words (“Few”/ “Majority) and numbers (“3 out of 100 “) on social media performance. Results The correlational analysis showed that the number of “Clicks” and “Engagement” was similar among campaigns, except for the campaign focusing on vaccines’ effectiveness, which had 14.65 less clicks and 19.52 less engagements per advert (including controls and district-fixed effects) compared to the base “It’s safe” campaign. Vaccination rates were highest at times coinciding with campaigns focusing on vaccination safety and effectiveness. Our experimental results showed that informational messages related to side effects that were framed using words (“Majority did not report discomfort”/ “Few persons reported discomfort”) were better at generating “Clicks” compared to those using numbers (“3 out of 100 reported discomforts”). Conclusions Facebook adverts highlighting vaccine safety had a similar level of social media performance as other campaigns, except for adverts focusing on vaccine efficacy, which performed worse. Communicating side-effect information with words instead of numbers can expand social media interest in low-uptake regions like the Caribbean. Our results serve as preliminary evidence for public health officials to encourage vaccine uptake in high-hesitancy contexts.

In today’s dynamic fintech environment, user requirements are constantly changing, and digital services need to be developed and adapted to meet their expectations and needs. UI/UX research technologies and methods provide rich opportunities to study user preferences and identify problems and shortcomings in existing digital services. The main aspects of improving the user experience of the mobile bank were reviewed. Recommendations were made for the further development of the IT service. The purpose of this article is to examine the main aspects of improving the user experience and to formulate recommendations for the development of IT services in the banking sector. Research methodology – at the first stage, an analysis of user behavior in fintech was carried out: a study of user expectations and preferences in the field of digital financial services. Then user experience was analyzed and UI/UX studies were conducted for the ‘Notification Center’ of the bank’s mobile application. The final stage of the study was to evaluate the effectiveness of the redesign project of the Notification Centre section of the bank's mobile application and to formulate recommendations for the further development of the mobile application.The research conducted has shown the practical importance of improving the user experience of the bank’s mobile application, which consists of improving usability, increasing customer satisfaction and improving the overall impression of banking services. This leads to increased conversion, customer retention and increased loyalty to the bank.Improving the user experience of the bank’s mobile application makes it possible to increase the efficiency of the bank’s employees, reduce the burden on the customer support department and reduce the number of errors when performing routine operations or actions. This not only optimises the bank’s internal processes, but also helps to improve the overall quality of service, which has a particular impact on the bank’s reputation, competitiveness and financial performance.

A method is proposed to compute more precise standard errors for marketing A/B tests. Incrementality experiments compare customers exposed to a marketing action designed to increase sales with those randomly assigned to a control group. These experiments suffer from noisy responses, which make precise estimation of the average treatment effect (ATE) and marketing return difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment—customers who would buy regardless of ad exposure, those who would buy only if exposed to ads, and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Analytical results and simulations show that the method decreases the sampling variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model used to estimate the strata-level effects is well identified. Applying the procedure to five catalog experiments shows a reduction of 30%–60% in the variance of the overall ATE. This leads to a substantial decrease in decision errors when the estimator is used to determine whether ads should be continued or discontinued. History: Olivier Toubia served as the senior editor. Funding: This work was supported by Adobe Systems [Adobe Digital Experience Research Award]. R. Berman was supported by the United States-Israel Binational Science Foundation [Grant 2020022] and the Wharton Dean's Research Fund, and E. M. Feit was supported by the Dean's Fellowship from LeBow College of Business. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mksc.2022.0297. [ABSTRACT FROM AUTHOR]

The aim of this article is to establish a causal link between the policies implemented by technology companies and the outcomes they yield within intricate temporal and/or spatial dependent experiments. We propose a novel temporal/spatio-temporal Varying Coefficient Decision Process model, capable of effectively capturing the evolving treatment effects in situations characterized by temporal and/or spatial dependence. Our methodology encompasses the decomposition of the average treatment effect into the direct effect (DE) and the indirect effect (IE). We subsequently devise comprehensive procedures for estimating and making inferences about both DE and IE. Additionally, we provide a rigorous analysis of the statistical properties of these procedures, such as asymptotic power. To substantiate the effectiveness of our approach, we carry out extensive simulations and real data analyses. [ABSTRACT FROM AUTHOR]

Copyright of Transfer: Zeitschrift für Kommunikation & Markenmanagement is the property of Deutsche Werbewissenschaftliche Gesellschaft and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This paper proposes a platform trial for conducting A/B tests with multiple arms and interim monitoring to investigate the impact of several factors on the expected sample size and probability of early stopping. We examined the performance of three stopping boundaries: O’Brien Fleming (OBF) stopping for either futility or difference (both), Pocock stopping for futility only, and fixed sample size design. We simulated twelve scenarios of different orders of arms based on various effect sizes, as well as considered 1 or 3 interim looks. The overall findings are summarizing in a flowchart to provide intuitive guidance for the design of the platform based on the simulation. We found that it is better to use OBF stopping for both if there is any effective variant and the trial is sufficiently powered to detect the expected effect size. If the study is underpowered to detect a difference, we recommend fixed sample size design to gather as much information as possible, however if the expected sample size is important to minimize, we recommend using Pocock boundaries with futility monitoring. Our results aimed at helping high-tech companies conduct their own studies without requiring extensive knowledge of clinical trial design and statistical methodology.

The rise of internet-based services and products in the late 1990s brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking.com, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this article we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry. [ABSTRACT FROM AUTHOR]

The metaphor of technical debt (TD) has generated a conceptual framework on factors that weaken the quality of software and accumulate a repair cost. However, user-related aspects like user experience (UX) receive little consideration among TD types, for reasons like the substantial focus on code TD, some dynamics inherent to agile processes, and an apparent lack of cumulative cost over time. This article has two main goals: first, to present evidence of the existence of UXDebt as a type of TD, with a cumulative cost for the development team as well as stakeholders; second, to propose a definition and characterization of UXDebt that may serve as a frame for further research on methods and tools for continuous management within agile processes. For the first goal, we have compiled evidence on the current state of UXDebt from three sources: a literature review, a survey among software engineering professionals in agile teams, and the analysis of UX issues in GitHub. All sources have evidenced some form of UXDebt; surveyed practitioners have recognized its poor management with a cost for the entire team that accumulates over time. Moreover, issue tracking systems allow to visualize and measure a technical form of UXDebt. For the second goal, we have defined a conceptual model that characterizes UXDebt in terms of both technical and non-technical aspects. On the technical side, we propose the notion of UX smells which allows us to discuss concrete management activities. [ABSTRACT FROM AUTHOR]

Randomized controlled trials (RCTs) admit unconfounded design-based inference – randomization largely justifies the assumptions underlying statistical effect estimates – but often have limited sample sizes. However, researchers may have access to big observational data on covariates and outcomes from RCT nonparticipants. For example, data from A/B tests conducted within an educational technology platform exist alongside historical observational data drawn from student logs. We outline a design-based approach to using such observational data for variance reduction in RCTs. First, we use the observational data to train a machine learning algorithm predicting potential outcomes using covariates and then use that algorithm to generate predictions for RCT participants. Then, we use those predictions, perhaps alongside other covariates, to adjust causal effect estimates with a flexible, design-based covariate-adjustment routine. In this way, there is no danger of biases from the observational data leaking into the experimental estimates, which are guaranteed to be exactly unbiased regardless of whether the machine learning models are “correct” in any sense or whether the observational samples closely resemble RCT samples. We demonstrate the method in analyzing 33 randomized A/B tests and show that it decreases standard errors relative to other estimators, sometimes substantially.

Even though online learning is an affordable way to gain new knowledge and skills, and millions of people use different platforms, there is a serious problem with students dropping out, as well as with their completion of assignments during training. One effective way to increase student engagement is to use gamification in teaching. In this study, we study the relationship between the introduction of such a gamification element as virtual currency with the results of learning English on the Skyeng platform. Using data from 2019 to 2020, we analyzed the relationship between the indicated intervention and the percentage of completion of homework, as well as the average homework score of platform users. The results of the experiment conducted in the A/B format showed that the introduction of virtual currency had a significant effect on the educational results of students. Thus, students from the experimental group showed an increase in involvement in doing homework on the platform compared to the control group. Moreover, various types of rewards work in general to increase student engagement in homework, including those that are not directly related to learning. At the same time, students who did not use the game currency available to them showed similar or even lower results than participants in the experiment from the control group. It can be assumed that in this case there is an effect of student motivation as a hidden factor, which explains the greater interest in the functionality and capabilities of the platform itself, and in the educational process as a whole.

Copyright of Techno.com is the property of Universitas Dian Nuswantoro, Fakultas Ilmu Komputer and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Network information has become a common feature of many modern experiments. From vaccine efficacy studies to marketing for product adoption, stakeholders aim to estimate global treatment effects — what happens if everyone in a network is treated versus if no one is treated. Because individual outcomes are potentially influenced by the treatments or behaviors of others in the network, experimental designs must condition on the underlying network. Social networks frequently exhibit homophilous community structure, meaning that individuals within observed or latent communities are more similar to each. This observation motivates the development of community aware experimental design. This design recognizes that information between individuals likely flows along within community edges rather than across community edges. We demonstrate that this design reduces the bias of a simple difference in means estimator, even when the community structure of the graph needs to be estimated. Further, we show that as the community detection problem gets more difficult or if the community structure does not affect the causal question, the proposed design maintains its performance. [ABSTRACT FROM AUTHOR]

A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. Major challenges arise in online experiments of two-sided marketplace platforms (e.g., Uber) where there is only one unit that receives a sequence of treatments over time. In those experiments, the treatment at a given time impacts current outcome as well as future outcomes. The aim of this article is to introduce a reinforcement learning framework for carrying A/B testing in these experiments, while characterizing the long-term treatment effects. Our proposed testing procedure allows for sequential monitoring and online updating. It is generally applicable to a variety of treatment designs in different industries. In addition, we systematically investigate the theoretical properties (e.g., size and power) of our testing procedure. Finally, we apply our framework to both simulated data and a real-world data example obtained from a technological company to illustrate its advantage over the current practice. A Python implementation of our test is available at . for this article are available online. [ABSTRACT FROM AUTHOR]

After a rich history in medicine, randomized control trials (RCTs), both simple and complex, are in increasing use in other areas, such as web-based A/B testing and planning and design of decisions. A main objective of RCTs is to be able to measure parameters, and contrasts in particular, while guarding against biases from hidden confounders. After careful definitions of classical entities such as contrasts, an algebraic method based on circuits is introduced which gives a wide choice of randomization schemes. [ABSTRACT FROM AUTHOR]

Purpose – The aim of this paper is to demonstrate the application of A/B testing for measuring the effectiveness of online sales in order to determine which changes to a website’s user interface have the greatest effect on the improvement of key performance indicators (KPIs) of online sales. Design/Methodology/Approach – A total of five A/B tests were conducted, four of which concerned the manipulation of user interface elements whereas one examined the difference in KPIs depending on the quality of the search engine used. Testing was conducted from January to July 2021 on a sample of a minimum of 7,000 visitors of the website of a company operating on the Croatian market. Findings and Implications – The conducted tests show that sometimes, as can be seen from the results of the first A/B test, the existing version of the website should be kept. However, as shown by the second, fourth, and fifth A/B test, changes to a website’s user interface can be of significant help in improving KPIs. The third A/B test highlighted the need for multiple tests of the same user interface element in order to achieve the full potential of an online business. Limitations – The general limitations of A/B testing, such as its focus on short-term goals, the lack of insights into real consumer behavior, and the use of CMS systems all affect the results of conducted research. Originality – As the analysis of existing research papers established the absence of a unified division of KPIs, their summarization and use for the purposes of the present research is the main contribution of this study.

The use of error spending functions and stopping rules has become a powerful tool for conducting interim analyses. The implementation of an interim analysis is broadly desired not only in traditional clinical trials but also in A/B tests. Although many papers have summarized error spending approaches, limited work has been done in the context of large-scale data that assists in finding the "optimal" boundary. In this paper, we summarized fifteen boundaries that consist of five error spending functions that allow early termination for futility, difference, or both, as well as a fixed sample size design without interim monitoring. The simulation is based on a practical A/B testing problem comparing two independent proportions. We examine sample sizes across a range of values from 500 to 250,000 per arm to reflect different settings where A/B testing may be utilized. The choices of optimal boundaries are summarized using a proposed loss function that incorporates different weights for the expected sample size under a null experiment with no difference between variants, the expected sample size under an experiment with a difference in the variants, and the maximum sample size needed if the A/B test did not stop early at an interim analysis. The results are presented for simulation settings based on adequately powered, under-powered, and over-powered designs with recommendations for selecting the "optimal" design in each setting. [ABSTRACT FROM AUTHOR]

With the development of society, users have increasing requirements for the high-quality experience of products. The pursuit of a high profit conversion rate also gradually puts forward higher requirements for product details in the competition. Product providers need to iterate products fast and with a high quality to enhance user viscosity and activity to improve the profit conversion rate efficiently. A/B testing is a technical method to conduct experiments on target users who use different iterative strategies, and observe which strategy is better through log embedding and statistical analysis. Usually, different businesses of the same company are supported by different business systems, and the A/B tests of different business systems need to be operated in a unified manner. At present, most A/B testing systems cannot provide services for more than one business system at the same time, and there are problems such as high concurrency, scalability, reusability, and flexibility. In this regard, this paper proposes an idea of dynamic strategy distribution, based on which a configuration-driven traffic-multiplexing A/B testing model is constructed and implemented systematically. The model solves the high-concurrency problem when requesting experimental strategies by setting message middleware and strategy cache modules, making the system more lightweight, flexible, and efficient to meet the A/B testing requirements for multiple business systems. [ABSTRACT FROM AUTHOR]

An industry-academic collaboration was established to evaluate the choice of statistical test and study design for A/B testing in larger-scale industry experiments. Specifically, the standard approach at the industry partner was to apply a t-test for all outcomes, both continuous and binary, and to apply naïve interim monitoring strategies that had not evaluated the potential implications on operating characteristics such as power and type I error rates. Although many papers have summarized the robustness of the t-test, its performance for the A/B testing context of large-scale proportion data, with or without interim analyses, is needed. Investigating the effect of interim analyses on the robustness of the t-test is important, because interim analyses rely on a fraction of the total sample size and one should ensure that desired properties are maintained when a t-test is implemented not just at the end of the study, but for making interim decisions. Through simulation studies, the performance of the t-test, Chi-squared test, and Chi-squared test with Yate's correction when applied to binary outcomes data is evaluated. Further, interim monitoring through a naïve approach with no correction for multiple testing versus the O'Brien-Fleming boundary are considered in designs that allow early termination for futility, difference, or both. Results indicate that the t-test achieves similar power and type I error rates for binary outcomes data with the large sample sizes used in industrial A/B tests with and without interim monitoring, and naïve interim monitoring without corrections leads to poorly performing studies.

Many organizations have embraced formal experimentation, i.e., A/B testing, to improve the performance of their products and services. Experimentation, some have argued, should democratize innovation inside organizations by creating a platform to test new ideas, regardless of origin. In this article, we argue that experimentation's promise hinges on having the proper organizational decision-making process that encourages innovation while mitigating the risk of unanticipated failures. We study this question by developing a model of experimentation inside organizations, where decisions to implement are either centralized or decentralized—a tension identified by practitioners and scholars alike. Organizations with centralized mechanisms that rely too much on the input of other teams benefit least from experimentation, as do ones with completely decentralized ones. In contrast, organizations with mostly decentralized decisions, with a single authority that sets consistent thresholds for implementation, achieve growth but with less downside risk. Thus, without considering the organizational decision-making structure, the benefits of experimentation may be limited. [ABSTRACT FROM AUTHOR]

When estimating a global average treatment effect (GATE) under network interference, units can have widely different relationships to the treatment depending on a combination of the structure of their network neighborhood, the structure of the interference mechanism, and how the treatment was distributed in their neighborhood. In this work, we introduce a sequential procedure to generate and select graph- and treatment-based covariates for GATE estimation under regression adjustment. We show that it is possible to simultaneously achieve low bias and considerably reduce variance with such a procedure. To tackle inferential complications caused by our feature generation and selection process, we introduce a way to construct confidence intervals based on a block bootstrap. We illustrate that our selection procedure and subsequent estimator can achieve good performance in terms of root-mean-square error in several semi-synthetic experiments with Bernoulli designs, comparing favorably to an oracle estimator that takes advantage of regression adjustments for the known underlying interference structure. We apply our method to a real-world experimental dataset with strong evidence of interference and demonstrate that it can estimate the GATE reasonably well without knowing the interference process a priori.

Abstract Background In recent years, research on online learning platforms has exploded in quantity. More and more researchers are using these platforms to conduct A/B tests on the impact of different designs, and multiple scientific communities have emerged around studying the big data becoming available from these platforms. However, it is not yet fully understood how each type of research influences future scientific discourse within the broader field. To address this gap, this paper presents the first scientometric study on how researchers build on the contributions of these two types of online learning platform research (particularly in STEM education). We selected a pair of papers (one using A/B testing, the other conducting learning analytics (LA), on platform data of an online STEM education platform), published in the same year, by the same research group, at the same conference. We then analyzed each of the papers that cited these two papers, coding from the paper text (with inter-rater reliability checks) the reason for each citation made. Results After statistically comparing the frequency of each category of citation between papers, we found that the A/B test paper was self-cited more and that citing papers built on its work directly more frequently, whereas the LA paper was more often cited without discussion. Conclusions Hence, the A/B test paper appeared to have had a larger impact on future work than the learning analytics (LA) paper, even though the LA paper had a higher count of total citations with a lower degree of self-citation. This paper also established a novel method for understanding how different types of research make different contributions in learning analytics, and the broader online learning research space of STEM education.

Online experiments are the gold standard for evaluating impact on user experience and accelerating innovation in software. However, since experiments are typically limited in duration, observed treatment effects are not always stable, sometimes revealing increasing or decreasing patterns over time. There are multiple causes for a treatment effect to change over time. In this article, we focus on a particular cause, user-learning, which is primarily associated with novelty or primacy. Novelty describes the desire to use new technology that tends to diminish over time. Primacy describes the growing engagement with technology as a result of adoption of the innovation. Estimating user-learning is critical because it holds experimentation responsible for trustworthiness, empowers organizations to make better decisions by providing a long-term view of expected impact, and prevents user dissatisfaction. In this article, we propose an observational approach, based on difference-in-differences technique to estimate user-learning at scale. We use this approach to test and estimate user-learning in many experiments at Microsoft. We compare our approach with the existing experimental method to show its benefits in terms of ease of use and higher statistical power, and to discuss its limitation in presence of other forms of treatment interaction with time. [ABSTRACT FROM AUTHOR]

We investigate what fraction of all significant results in website A/B testing is actually null effects (i.e., the false discovery rate (FDR)). Our data consist of 4,964 effects from 2,766 experiments conducted on a commercial A/B testing platform. Using three different methods, we find that the FDR ranges between 28% and 37% for tests conducted at 10% significance and between 18% and 25% for tests at 5% significance (two sided). These high FDRs stem mostly from the high fraction of true null effects, about 70%, rather than from low power. Using our estimates, we also assess the potential of various A/B test designs to reduce the FDR. The two main implications are that decision makers should expect one in five interventions achieving significance at 5% confidence to be ineffective when deployed in the field and that analysts should consider using two-stage designs with multiple variations rather than basic A/B tests. This paper was accepted by Eric Anderson, marketing. [ABSTRACT FROM AUTHOR]