Search

As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how do practitioners resolve these disagreements. To this end, we first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and eight different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on misleading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various state-of-the-art methods.

The Department of Integrative Biology and Physiology at the University of Minnesota is committed to supporting outstanding physiology majors interested in research and gaining research experience. We have created a program, UPRIME (Undergraduate Physiology Research In Medicine and Education), which has: 1) Foster young and aspiring undergraduates interested in physiology and biomedical research, and 2) Create unfettered opportunities for students that are both underrepresented in research and have strong interests in physiology research and education. UPRIME (Undergraduate Physiology Research In Medicine and Education) is a 10-week summer program hosted by the Department of Integrative Biology and Physiology at the University of Minnesota and sponsored by an American Heart Association award. Students (“UPRIME Scholars”) were either juniors or seniors that had just started working in physiology labs. They were selected on the basis of: 1) Scientific merit of their projects, 2) If they were underrepresented in research (as defined by NIH and/or especially in Minnesota), and 3) If they were from low socio-economic status. To meet our goals, UPRIME Scholars met once a week to discuss research and scientific methods. As the UPRIME Scholars were relatively inexperienced in research, we designed our meetings to be safe and open spaces for encouraging discussion, especially about their research projects. Discussion topics included techniques (e.g. qPCR, histology, and cell culture), experimental design (e.g. animal models, use of controls), and developing a hypothesis. In addition, we had UPRIME Scholars attend weekly “Graduate Student Colloquia” where our graduate students give formal presentations and chalk talks about their research. Here, UPRIME Scholars were tasked to identify and discuss the hypotheses tested, animal or experimental models used, and interpretation of results. In our first two years, UPRIME Scholars reported very positive experiences in their summer research and in the usefulness of the weekly meetings. They were able to better understand why they were doing their projects, how their projects were an important part of their mentor’s research, and how to troubleshoot the techniques they used. Taken together, the UPRIME program has been successful in the fostering of undergraduates in physiology research. The UPRIME program serves as our basis for future programs for undergraduates interested in research. American Heart Association This is the full abstract presented at the American Physiology Summit 2023 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

A challenge in the undergraduate physiology landscape is that some students who are underrepresented in the biomedical sciences are not aware of or are unable to take advantage of laboratory research opportunities. This is unfortunate because lab research can provide students with unique skill sets that are beneficial for future success, whether the students remain in research or move on to other careers. These skills include opportunities for critical thinking, improvement of oral and written communication skills, development of teamwork and leadership skills, learning about ethics in biomedical decision making and enhancement of digital literacy among others. As a program that sponsors an undergraduate Human Physiology major that has been successful in attracting underrepresented students (>40% students of color and >60% female) for a major Land-grant university, we challenged ourselves to consider how we might help our students make the most of their physiological interests and potentially improve the demographics of representation in our discipline. Starting in the summer of 2020, a group of faculty began a series of meetings to define an undergraduate program that would attract underrepresented students to lab-based internships, minimize the need for outside work by providing stipends and nurture their academic progress through informal student-faculty interactions. Though delayed by the Covid-19 pandemic, this framework has been used to initiate two new outreach programs. In the summer of 2021, we used this framework to initiate an American Heart Association sponsored research internship opportunity for 3rd and 4th year students. This program which we have named UPRIME is the focus of another abstract at this meeting. The INPUT (INtegrative biology and Physiology Undergraduate Training) program was started in fall of 2022. The program is internally supported and targets 1st and 2nd year undergraduates who represent populations that have been underrepresented in biomedical sciences. These students will be provided laboratory research experiences coupled with opportunities for educational support. A stipend of $3000 per student per semester is offered with the prospect of renewal dependent on a post-semester review. A cohort of six students were selected from a campus wide solicitation for applications into the inaugural INPUT class. These students will begin their projects in January 2023. We look forward to contributing to the scientific growth and development of these students. Perceptions of lab-based research and academic progress will be tracked along with career aspirations as the program progresses. Department of Integrative Biology and Physiology, University of Minnesota Medical School This is the full abstract presented at the American Physiology Summit 2023 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

Food waste and insecurity are two societal challenges that coexist in many parts of the world. A prominent force to combat these issues, food rescue platforms match food donations to organizations that serve underprivileged communities, and then rely on external volunteers to transport the food. Previous work has developed machine learning models for food rescue volunteer engagement. However, having long worked with domain practitioners to deploy AI tools to help with food rescues, we understand that there are four main pain points that keep such a machine learning model from being actually useful in practice: small data, data collected only under the default intervention, unmodeled objectives due to communication gap, and unforeseen consequences of the intervention. In this paper, we introduce bandit data-driven optimization which not only helps address these pain points in food rescue, but also is applicable to other nonprofit domains that share similar challenges. Bandit data-driven optimization combines the advantages of online bandit learning and offline predictive analytics in an integrated framework. We propose PROOF, a novel algorithm for this framework and formally prove that it has no-regret. We show that PROOF performs better than existing baseline on food rescue volunteer recommendation.

We consider a ubiquitous scenario in the Internet economy when individual decision-makers (henceforth, agents) both produce and consume information as they make strategic choices in an uncertain environment. This creates a three-way tradeoff between exploration (trying out insufficiently explored alternatives to help others in the future), exploitation (making optimal decisions given the information discovered by other agents), and incentives of the agents (who are myopically interested in exploitation, while preferring the others to explore). We posit a principal who controls the flow of information from agents that came before, and strives to coordinate the agents towards a socially optimal balance between exploration and exploitation, not using any monetary transfers. The goal is to design a recommendation policy for the principal which respects agents' incentives and minimizes a suitable notion of regret. We extend prior work in this direction to allow the agents to interact with one another in a shared environment: at each time step, multiple agents arrive to play a Bayesian game, receive recommendations, choose their actions, receive their payoffs, and then leave the game forever. The agents now face two sources of uncertainty: the actions of the other agents and the parameters of the uncertain game environment. Our main contribution is to show that the principal can achieve constant regret when the utilities are deterministic (where the constant depends on the prior distribution, but not on the time horizon), and logarithmic regret when the utilities are stochastic. As a key technical tool, we introduce the concept of explorable actions, the actions which some incentive-compatible policy can recommend with non-zero probability. We show how the principal can identify (and explore) all explorable actions, and use the revealed information to perform optimally., Comment: All revisions focused on presentation; all results (except Appendix C) have been present since the initial version

Child welfare agencies across the United States are turning to data-driven predictive technologies (commonly called predictive analytics) which use government administrative data to assist workers' decision-making. While some prior work has explored impacted stakeholders' concerns with current uses of data-driven predictive risk models (PRMs), less work has asked stakeholders whether such tools ought to be used in the first place. In this work, we conducted a set of seven design workshops with 35 stakeholders who have been impacted by the child welfare system or who work in it to understand their beliefs and concerns around PRMs, and to engage them in imagining new uses of data and technologies in the child welfare system. We found that participants worried current PRMs perpetuate or exacerbate existing problems in child welfare. Participants suggested new ways to use data and data-driven tools to better support impacted communities and suggested paths to mitigate possible harms of these tools. Participants also suggested low-tech or no-tech alternatives to PRMs to address problems in child welfare. Our study sheds light on how researchers and designers can work in solidarity with impacted communities, possibly to circumvent or oppose child welfare agencies., 10 pages in main body; 4 pages in appendix; Published in the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT'22)

We study the problem of a seller dynamically pricing d distinct types of indivisible goods, when faced with the online arrival of unit-demand buyers drawn independently from an unknown distribution. The goods are not in limited supply, but can only be produced at a limited rate and are costly to produce. The seller observes only the bundle of goods purchased at each day, but nothing else about the buyer’s valuation function. Our main result is a dynamic pricing algorithm for optimizing welfare (including the seller’s cost of production) that runs in time and a number of rounds that are polynomial in d and the approximation parameter. We are able to do this despite the fact that (i) the price-response function is not continuous, and even its fractional relaxation is a non-concave function of the prices, and (ii) the welfare is not observable to the seller. We derive this result as an application of a general technique for optimizing welfare over divisible goods, which is of independent interest. When buyers have strongly concave, Hölder continuous valuation functions over d divisible goods, we give a general polynomial time dynamic pricing technique. We are able to apply this technique to the setting of unit-demand buyers despite the fact that in that setting the goods are not divisible, and the natural fractional relaxation of a unit-demand valuation is not strongly concave. To apply our general technique, we introduce a novel price randomization procedure that has the effect of implicitly inducing buyers to “regularize” their valuations with a strongly concave function. Finally, we also extend our results to a limited-supply setting in which the supply of each good cannot be replenished.

AI-based decision support tools (ADS) are increasingly used to augment human decision-making in high-stakes, social contexts. As public sector agencies begin to adopt ADS, it is critical that we understand workers' experiences with these systems in practice. In this paper, we present findings from a series of interviews and contextual inquiries at a child welfare agency, to understand how they currently make AI-assisted child maltreatment screening decisions. Overall, we observe how workers' reliance upon the ADS is guided by (1) their knowledge of rich, contextual information beyond what the AI model captures, (2) their beliefs about the ADS's capabilities and limitations relative to their own, (3) organizational pressures and incentives around the use of the ADS, and (4) awareness of misalignments between algorithmic predictions and their own decision-making objectives. Drawing upon these findings, we discuss design implications towards supporting more effective human-AI decision-making., Comment: 2022 Conference on Human Factors in Computing Systems

OBJECTIVES: To evaluate the impact of a dietary supplement within an elderly population who reside in a Community Living Center (CLC; VA Nursing Home), on the number of respiratory tract infections (RTIs) during a 90-day study period. METHODS: This is a randomized, double-blind, placebo-controlled study to evaluate the impact of bovine Lactoferrin (bLf) on RTIs in an elderly nursing home population in the US. Subjects will be residents of the CLC and screened within 21 days prior to starting study supplementation. Eligible participants were at least 55 years of age, able to eat and drink and expected to reside at the CLC for the duration of the study. Consent was obtained from the study participant or their legally recognized representative decision maker. Participants will be excluded if: receiving tube feeds or specialized diets for eating disorders; immunocompromised; have a life expectancy of less than six months; or allergic to study products. RESULTS: Subjects will be randomized in a ratio of 1:1 to either investigational (600 mg of bLf) or (placebo arms. All participants will take study supplement by mouth daily for 90 days. Subjects will be assessed daily for RTI symptoms. Blood and saliva will be collected at 45 and 90 days. Ad hoc assessments and a nasal sample will take place if a subject develops a protocol-defined RTI. CONCLUSIONS: The primary outcome will be number of RTIs over the 90-day study period. Secondary outcomes include severity and duration of RTI symptoms, symptoms associated with RTIs, number of RTI complications, and nasopharyngeal swab at onset of RTI, Other secondary outcomes include the following, all measured at baseline, day 45 and day 90: quality of life by questionnaire, weight, saliva markers, laboratory testing and immunological markers. An exploratory endpoint is vaccine specific inflammatory panel (influenza and Sars-Cov2) measured upon vaccination during supplement period. RTI number, severity, duration and complications and medically confirmed adverse events will be compared between the placebo and investigational groups. FUNDING SOURCES: RB/Mead Johnson & Company.

Recent work in fair machine learning has proposed dozens of technical definitions of algorithmic fairness and methods for enforcing these definitions. However, we still lack an understanding of how to develop machine learning systems with fairness criteria that reflect relevant stakeholders' nuanced viewpoints in real-world contexts. To address this gap, we propose a framework for eliciting stakeholders' subjective fairness notions. Combining a user interface that allows stakeholders to examine the data and the algorithm's predictions with an interview protocol to probe stakeholders' thoughts while they are interacting with the interface, we can identify stakeholders' fairness beliefs and principles. We conduct a user study to evaluate our framework in the setting of a child maltreatment predictive system. Our evaluations show that the framework allows stakeholders to comprehensively convey their fairness viewpoints. We also discuss how our results can inform the design of predictive systems.

This article belongs to the Special Issue Breastmilk as a Model: Efforts to Improve Infant Formulae for Term Infants., Research on milk fat globule membrane (MFGM) is gaining traction. The interest is two-fold; on the one hand, it is a unique trilayer structure with specific secretory function. On the other hand, it is the basis for ingredients with the presence of phospho- and sphingolipids and glycoproteins, which are being used as food ingredients with valuable functionality, in particular, for use as a supplement in infant nutrition. This last application is at the center of this Review, which aims to contribute to understanding MFGM’s function in the proper development of immunity, cognition, and intestinal trophism, in addition to other potential effects such as prevention of diseases including cardiovascular disease, impaired bone turnover and inflammation, skin conditions, and infections as well as age-associated cognitive decline and muscle loss. The phospholipid composition of MFGM from bovine milk is quite like human milk and, although there are some differences due to dairy processing, these do not result in a chemical change. The MFGM ingredients, as used to improve the formulation in different clinical studies, have indeed increased the presence of phospholipids, sphingolipids, glycolipids, and glycoproteins with the resulting benefits of different outcomes (especially immune and cognitive outcomes) with no reported adverse effects. Nevertheless, the precise mechanism(s) of action of MFGM remain to be elucidated and further basic investigation is warranted., This study was carried out within the framework of the AGL2017-87884 Project (MINECO/AEI/FEDER, UE). J.F. gratefully acknowledges support from the Spanish Fulbright Program for his stay as a visiting scholar and the hospitality of Dr. Jimenez-Flores at FST (OSU, OH).

OBJECTIVES: To evaluate growth through 24 months of age in infants receiving added bovine milk fat globule membrane (MFGM) in infant formula through 12 months of age. Concentration of MFGM from bovine milk fractions and incorporation in infant formula may better approximate the composition of complex milk lipids in human milk. METHODS: In the double-blind, randomized, controlled Chilean Infant Nutrition Trial (ChiNuT; NCT0262613), term infants whose mothers chose to initiate exclusive infant formula feeding before 4 months of age were randomized to receive: a standard cow's milk-based infant formula (SF, n = 174) or a similar formula with added whey protein-lipid concentrate (5 g/L; source of bovine MFGM) (bMFGM, n = 176). A reference group of infants exclusively receiving human milk (HM, n = 236) was also recruited. Growth through 24 months of age was the primary outcome. Length-for-age (LAZ), weight-for-age (WAZ) and body mass index (BMI)-for-age (BAZ) growth z-scores were analyzed by mixed-effects multiple linear regression models adjusted by sex, age (days), and maternal pregestational BMI (kg/m2). RESULTS: No significant group differences in sex, gestational age at birth, birthweight, maternal age and maternal education were detected, with the exception of maternal pregestational BMI (mean(SD)) (HM: 27.0(5.2) lower vs SF: 28.6(6.2) or bMFGM: 28.9(6.1); P = 0.002). Groups were similar at baseline (weight, length, WAZ, BAZ) with the exception of LAZ (lower in the bMFGM compared to HM group; P

Recently, there have been increasing calls for computer science curricula to complement existing technical training with topics related to Fairness, Accountability, Transparency, and Ethics. In this paper, we present Value Card, an educational toolkit to inform students and practitioners of the social impacts of different machine learning models via deliberation. This paper presents an early use of our approach in a college-level computer science course. Through an in-class activity, we report empirical data for the initial effectiveness of our approach. Our results suggest that the use of the Value Cards toolkit can improve students' understanding of both the technical definitions and trade-offs of performance metrics and apply them in real-world contexts, help them recognize the significance of considering diverse social values in the development of deployment of algorithmic systems, and enable them to communicate, negotiate and synthesize the perspectives of diverse stakeholders. Our study also demonstrates a number of caveats we need to consider when using the different variants of the Value Cards toolkit. Finally, we discuss the challenges as well as future applications of our approach., Updating authors' names

Online learning algorithms, widely used to power search and content optimization on the web, must balance exploration and exploitation, potentially sacrificing the experience of current users in order to gain information that will lead to better decisions in the future. While necessary in the worst case, explicit exploration has a number of disadvantages compared to the greedy algorithm that always "exploits" by choosing an action that currently looks optimal. We ask under what conditions inherent diversity in the data makes explicit exploration unnecessary. We build on a recent line of work on the smoothed analysis of the greedy algorithm in the linear contextual bandits model. We improve on prior results to show that a greedy approach almost matches the best possible Bayesian regret rate of any other algorithm on the same problem instance whenever the diversity conditions hold, and that this regret is at most $\tilde O(T^{1/3})$., Comment: Results in this paper, without any proofs, have been announced in an extended abstract (Raghavan et al., 2018a), and fleshed out in the technical report (Raghavan et al., 2018b [arXiv:1806.00543]). This manuscript covers a subset of results from Raghavan et al. (2018a,b), focusing on the greedy algorithm, and is streamlined accordingly

Artificial intelligence algorithms have been used to enhance a wide variety of products and services, including assisting human decision making in high-stakes contexts. However, these algorithms are complex and have trade-offs, notably between prediction accuracy and fairness to population subgroups. This makes it hard for designers to understand algorithms and design products or services in a way that respects users' goals, values, and needs. We proposed a method to help designers and users explore algorithms, visualize their trade-offs, and select algorithms with trade-offs consistent with their goals and needs. We evaluated our method on the problem of predicting criminal defendants' likelihood to re-offend through (i) a large-scale Amazon Mechanical Turk experiment, and (ii) in-depth interviews with domain experts. Our evaluations show that our method can help designers and users of these systems better understand and navigate algorithmic trade-offs. This paper contributes a new way of providing designers the ability to understand and control the outcomes of algorithmic systems they are creating., Paper appeared at Proceedings of The 2020 ACM conference on Designing Interactive Systems (DIS'20)

Background: Data sharing accelerates scientific progress but sharing individual-level data while preserving patient privacy presents a barrier. Methods and Results: Using pairs of deep neural networks, we generated simulated, synthetic participants that closely resemble participants of the SPRINT trial (Systolic Blood Pressure Trial). We showed that such paired networks can be trained with differential privacy, a formal privacy framework that limits the likelihood that queries of the synthetic participants’ data could identify a real a participant in the trial. Machine learning predictors built on the synthetic population generalize to the original data set. This finding suggests that the synthetic data can be shared with others, enabling them to perform hypothesis-generating analyses as though they had the original trial data. Conclusions: Deep neural networks that generate synthetic participants facilitate secondary analyses and reproducible investigation of clinical data sets by enhancing data sharing while preserving participant privacy.

We empirically study the interplay between exploration and competition. Systems that learn from interactions with users often engage in exploration: making potentially suboptimal decisions in order to acquire new information for future decisions. However, when multiple systems are competing for the same market of users, exploration may hurt a system's reputation in the near term, with adverse competitive effects. In particular, a system may enter a "death spiral", when the short-term reputation cost decreases the number of users for the system to learn from, which degrades its performance relative to competition and further decreases its market share. We ask whether better exploration algorithms are incentivized under competition. We run extensive numerical experiments in a stylized duopoly model in which two firms deploy multi-armed bandit algorithms and compete for myopic users. We find that duopoly and monopoly tend to favor a primitive "greedy algorithm" that does not explore and leads to low consumer welfare, whereas a temporary monopoly (a duopoly with an early entrant) may incentivize better bandit algorithms and lead to higher consumer welfare. Our findings shed light on the first-mover advantage in the digital economy by exploring the role that data can play as a barrier to entry in online markets., This is a preprint of an article accepted for EC 2019

It is common in recommendation systems that users both consume and produce information as they make strategic choices under uncertainty. While a social planner would balance "exploration" and "exploitation" using a multi-armed bandit algorithm, users' incentives may tilt this balance in favor of exploitation. We consider Bayesian Exploration: a simple model in which the recommendation system (the "principal") controls the information flow to the users (the "agents") and strives to incentivize exploration via information asymmetry. A single round of this model is a version of a well-known "Bayesian Persuasion game" from [Kamenica and Gentzkow]. We allow heterogeneous users, relaxing a major assumption from prior work that users have the same preferences from one time step to another. The goal is now to learn the best personalized recommendations. One particular challenge is that it may be impossible to incentivize some of the user types to take some of the actions, no matter what the principal does or how much time she has. We consider several versions of the model, depending on whether and when the user types are reported to the principal, and design a near-optimal "recommendation policy" for each version. We also investigate how the model choice and the diversity of user types impact the set of actions that can possibly be "explored" by each type.

We provide a differentially private algorithm for hypothesis selection. Given samples from an unknown probability distribution $P$ and a set of $m$ probability distributions $\mathcal{H}$, the goal is to output, in a $\varepsilon$-differentially private manner, a distribution from $\mathcal{H}$ whose total variation distance to $P$ is comparable to that of the best such distribution (which we denote by $\alpha$). The sample complexity of our basic algorithm is $O\left(\frac{\log m}{\alpha^2} + \frac{\log m}{\alpha \varepsilon}\right)$, representing a minimal cost for privacy when compared to the non-private algorithm. We also can handle infinite hypothesis classes $\mathcal{H}$ by relaxing to $(\varepsilon,\delta)$-differential privacy. We apply our hypothesis selection algorithm to give learning algorithms for a number of natural distribution classes, including Gaussians, product distributions, sums of independent random variables, piecewise polynomials, and mixture classes. Our hypothesis selection procedure allows us to generically convert a cover for a class to a learning algorithm, complementing known learning lower bounds which are in terms of the size of the packing number of the class. As the covering and packing numbers are often closely related, for constant $\alpha$, our algorithms achieve the optimal sample complexity for many classes of interest. Finally, we describe an application to private distribution-free PAC learning., Comment: Appeared in NeurIPS 2019. Final version to appear in IEEE Transactions on Information Theory

OBJECTIVES: Bovine milk fat globule membrane (bMFGM) added in routine infant formula supports normal growth and safety through 24 mo of age in term infants. The impact on micronutrients and glucose-related biomarkers is assessed here. METHODS: In this double-blind, randomized, controlled trial, formula-fed infants were enrolled (

We develop theory for using heuristics to solve computationally hard problems in differential privacy. Heuristic approaches have enjoyed tremendous success in machine learning, for which performance can be empirically evaluated. However, privacy guarantees cannot be evaluated empirically, and must be proven --- without making heuristic assumptions. We show that learning problems over broad classes of functions --- those that have polynomially sized universal identification sets --- can be solved privately and efficiently, assuming the existence of a non-private oracle for solving the same problem. Our first algorithm yields a privacy guarantee that is contingent on the correctness of the oracle. We then give a reduction which applies to a class of heuristics which we call certifiable, which allows us to convert oracle-dependent privacy guarantees to worst-case privacy guarantee that hold even when the heuristic standing in for the oracle might fail in adversarial ways. Finally, we consider classes of functions for which both they and their dual classes have small universal identification sets. This includes most classes of simple boolean functions studied in the PAC learning literature, including conjunctions, disjunctions, parities, and discrete halfspaces. We show that there is an efficient algorithm for privately constructing synthetic data for any such class, given a non-private learning oracle. This in particular gives the first oracle-efficient algorithm for privately generating synthetic data for contingency tables. The most intriguing question left open by our work is whether or not every problem that can be solved differentially privately can be privately solved with an oracle-efficient algorithm. While we do not resolve this, we give a barrier result that suggests that any generic oracle-efficient reduction must fall outside of a natural class of algorithms (which includes the algorithms given in this paper).

We study the design of rating systems that incentivize (more) efficient social learning among self-interested agents. Agents arrive sequentially and are presented with a set of possible actions, each of which yields a positive reward with an unknown probability. A disclosure policy sends messages about the rewards of previously-chosen actions to arriving agents. These messages can alter agents' incentives towards exploration, taking potentially sub-optimal actions for the sake of learning more about their rewards. Prior work achieves much progress with disclosure policies that merely recommend an action to each user, without any other supporting information, and sometimes recommend exploratory actions. All this work relies heavily on standard, yet very strong rationality assumptions. However, these assumptions are quite problematic in the context of the motivating applications: recommendation systems such as Yelp, Amazon, or Netflix, and macthing markets such as AirBnB. It is very unclear whether users would know and understand a complicated disclosure policy announced by the principal, let alone trust the principal to faithfully implement it. (The principal may deviate from the announced policy either intentionally, or due to insufficient information about the users, or because of bugs in implementation.) Even if the users understand the policy and trust that it was implemented as claimed, they might not react to it rationally, particularly given the lack of supporting information and the possibility of being singled out for exploration. For example, users may find such disclosure policies unacceptable and leave the system. We study a particular class of disclosure policies that use messages, called unbiased subhistories, consisting of the actions and rewards from a subsequence of past agents. Each subsequence is chosen ahead of time, according to a predetermined partial order on the rounds. We posit a flexible model of frequentist agent response, which we argue is plausible for this class of "order-based" disclosure policies. We measure the performance of a policy by its regret, i.e., the difference in expected total reward between the best action and the policy. A disclosure policy that reveals full history in each round risks inducing herding behavior among the agents, and typically has regret linear in the time horizon T. Our main result is an order-based disclosure policy that obtains regret ~O (√T). This regret is known to be optimal in the worst case over reward distributions, even absent incentives. We also exhibit simpler order-based policies with higher, but still sublinear, regret. These policies can be interpreted as dividing a sublinear number of agents into constant-sized focus groups, whose histories are then revealed to future agents. Helping market participants find whatever they are looking for, and coordinating their search and exploration behavior in a globally optimal way, is an essential part of market design. This paper continues the line of work on "incentivized exploration": essentially, exploration-exploitation learning in the presence of self-interested users whose incentives are skewed in favor of exploitation. Conceptually, we study the interplay of information design, social learning, and multi-armed bandit algorithms. To the best of our knowledge, this is the first paper in the literature on incentivized exploration (and possibly in the broader literature on "learning and incentives") which attempts to mitigate the limitations of standard economic assumptions. Full version: https://arxiv.org/abs/1811.06026.

We investigate the problem of equilibrium computation for "large" $n$-player games. Large games have a Lipschitz-type property that no single player's utility is greatly affected by any other individual player's actions. In this paper, we mostly focus on the case where any change of strategy by a player causes other players' payoffs to change by at most $\frac{1}{n}$. We study algorithms having query access to the game's payoff function, aiming to find $\epsilon$-Nash equilibria. We seek algorithms that obtain $\epsilon$ as small as possible, in time polynomial in $n$. Our main result is a randomised algorithm that achieves $\epsilon$ approaching $\frac{1}{8}$ for 2-strategy games in a {\em completely uncoupled} setting, where each player observes her own payoff to a query, and adjusts her behaviour independently of other players' payoffs/actions. $O(\log n)$ rounds/queries are required. We also show how to obtain a slight improvement over $\frac{1}{8}$, by introducing a small amount of communication between the players. Finally, we give extension of our results to large games with more than two strategies per player, and alternative largeness parameters.

Kearns, Neel, Roth, and Wu [ICML 2018] recently proposed a notion of rich subgroup fairness intended to bridge the gap between statistical and individual notions of fairness. Rich subgroup fairness picks a statistical fairness constraint (say, equalizing false positive rates across protected groups), but then asks that this constraint hold over an exponentially or infinitely large collection of subgroups defined by a class of functions with bounded VC dimension. They give an algorithm guaranteed to learn subject to this constraint, under the condition that it has access to oracles for perfectly learning absent a fairness constraint. In this paper, we undertake an extensive empirical evaluation of the algorithm of Kearns et al. On four real datasets for which fairness is a concern, we investigate the basic convergence of the algorithm when instantiated with fast heuristics in place of learning oracles, measure the tradeoffs between fairness and accuracy, and compare this approach with the recent algorithm of Agarwal, Beygelzeimer, Dudik, Langford, and Wallach [ICML 2018], which implements weaker and more traditional marginal fairness constraints defined by individual protected attributes. We find that in general, the Kearns et al. algorithm converges quickly, large gains in fairness can be obtained with mild costs to accuracy, and that optimizing accuracy subject only to marginal fairness leads to classifiers with substantial subgroup unfairness. We also provide a number of analyses and visualizations of the dynamics and behavior of the Kearns et al. algorithm. Overall we find this algorithm to be effective on real data, and rich subgroup fairness to be a viable notion in practice.

We study an online linear classification problem in which the data is generated by strategic agents who manipulate their features in an effort to change the classification outcome. In rounds, the learner deploys a classifier, then an adversarially chosen agent arrives and possibly manipulates her features to optimally respond to the learner's choice of classifier. The learner has no knowledge of the agents' utility functions or "real" features, which may vary widely across agents. Instead, the learner is only able to observe their "revealed preferences", i.e., the manipulated feature vectors they provide. For a broad family of agent cost functions, we give a computationally efficient learning algorithm that is able to obtain diminishing "Stackelberg regret" --- a form of policy regret that guarantees that the learner is realizing loss nearly as small as that of the best classifier in hindsight, even allowing for the fact that agents would have best-responded differently to the optimal classifier.

The 2013 arrival of SportVU player tracking data in all NBA arenas introduced an overwhelming amount of on-court information - information which the league is still learning how to maximize for insights into player performance and basketball strategy. Knowing where the ball and every player on the court are at all times throughout the course of the game produces almost endless possibilities, and it can be difficult figuring out where to begin. This article serves as a step-by-step guide for how to turn a data feed of one million rows of SportVU data from one NBA game into visualizable components you can use to model any player's movement. We detail some utility functions that are helpful for manipulating SportVU data before applying it to the task of visualizing player offensive movement. We conclude with visualizations of the resulting output for one NBA game, as well as what the results look like aggregated across an entire season for three NBA stars with very different offensive tendencies.

We consider settings in which we wish to incentivize myopic agents (such as Airbnb landlords, who may emphasize short-term profits and property safety) to treat arriving clients fairly, in order to prevent overall discrimination against individuals or groups. We model such settings in both classical and contextual bandit models in which the myopic agents maximize rewards according to current empirical averages, but are also amenable to exogenous payments that may cause them to alter their choices. Our notion of fairness asks that more qualified individuals are never (probabilistically) preferred over less qualified ones [Joseph et al]. We investigate whether it is possible to design inexpensive {subsidy} or payment schemes for a principal to motivate myopic agents to play fairly in all or almost all rounds. When the principal has full information about the state of the myopic agents, we show it is possible to induce fair play on every round with a subsidy scheme of total cost $o(T)$ (for the classic setting with $k$ arms, $\tilde{O}(\sqrt{k^3T})$, and for the $d$-dimensional linear contextual setting $\tilde{O}(d\sqrt{k^3 T})$). If the principal has much more limited information (as might often be the case for an external regulator or watchdog), and only observes the number of rounds in which members from each of the $k$ groups were selected, but not the empirical estimates maintained by the myopic agent, the design of such a scheme becomes more complex. We show both positive and negative results in the classic and linear bandit settings by upper and lower bounding the cost of fair subsidy schemes.

The foundation of endovascular procedures by nephrologists was laid in the private practice arena. Because of political issues such as training, credentialing, space and equipment expenses, and co-management concerns surrounding the performance of dialysis-access procedures, the majority of these programs provided care in an outpatient vascular access center. On the basis of the improvement of patient care demonstrated by these centers, several nephrology programs at academic medical centers have also embraced this approach. In addition to providing interventional care on an outpatient basis, academic medical centers have taken a step further to expand collaboration with other specialties with similar expertise (such as with interventional radiologists and cardiologists) to enhance patient care and research. The enthusiastic initiative, cooperative, and mutually collaborative efforts used by academic medical centers have resulted in the successful establishment of interventional nephrology programs. This article describes various models of interventional nephrology programs at academic medical centers across the United States.

In this paper we present an extremely general method for approximately solving a large family of convex programs where the solution can be divided between different agents, subject to joint differential privacy. This class includes multi-commodity flow problems, general allocation problems, and multi-dimensional knapsack problems, among other examples. The accuracy of our algorithm depends on the \emph{number} of constraints that bind between individuals, but crucially, is \emph{nearly independent} of the number of primal variables and hence the number of agents who make up the problem. As the number of agents in a problem grows, the error we introduce often becomes negligible. We also consider the setting where agents are strategic and have preferences over their part of the solution. For any convex program in this class that maximizes \emph{social welfare}, there is a generic reduction that makes the corresponding optimization \emph{approximately dominant strategy truthful} by charging agents prices for resources as a function of the approximately optimal dual variables, which are themselves computed under differential privacy. Our results substantially expand the class of problems that are known to be solvable under both privacy and incentive constraints.

We revisit a classic coordination problem from the perspective of mechanism design: how can we coordinate a social welfare maximizing flow in a network congestion game with selfish players? The classical approach, which computes tolls as a function of known demands, fails when the demands are unknown to the mechanism designer, and naively eliciting them does not necessarily yield a truthful mechanism. Instead, we introduce a weak mediator that can provide suggested routes to players and set tolls as a function of reported demands. However, players can choose to ignore or misreport their type to this mediator. Using techniques from differential privacy, we show how to design a weak mediator such that it is an asymptotic ex-post Nash equilibrium for all players to truthfully report their types to the mediator and faithfully follow its suggestion, and that when they do, they end up playing a nearly optimal flow. Notably, our solution works in settings of incomplete information even in the absence of a prior distribution on player types. Along the way, we develop new techniques for privately solving convex programs which may be of independent interest., Version with latencies not normalized

A Stackelberg game is played between a leader and a follower. The leader first chooses an action, and then the follower plays his best response, and the goal of the leader is to pick the action that will maximize his payoff given the follower's best response. Stackelberg games capture, for example, the following interaction between a retailer and a buyer. The retailer chooses the prices of the goods he produces, and then the buyer chooses to buy a utility-maximizing bundle of goods. The goal of the retailer here is to set prices to maximize his profit---his revenue minus the production cost of the purchased bundle. It is quite natural that the retailer in this example would not know the buyer's utility function. However, he does have access to revealed preference feedback---he can set prices, and then observe the purchased bundle and his own profit. We give algorithms for efficiently solving, in terms of both computational and query complexity, a broad class of Stackelberg games in which the follower's utility function is unknown, using only "revealed preference" access to it. This class includes the profit maximization problem, as well as the optimal tolling problem in nonatomic congestion games, when the latency functions are unknown. Surprisingly, we are able to solve these problems even though the corresponding maximization problems are not concave in the leader's actions.

We initiate the study of a quantity that we call coordination complexity. In a distributed optimization problem, the information defining a problem instance is distributed among $n$ parties, who need to each choose an action, which jointly will form a solution to the optimization problem. The coordination complexity represents the minimal amount of information that a centralized coordinator, who has full knowledge of the problem instance, needs to broadcast in order to coordinate the $n$ parties to play a nearly optimal solution. We show that upper bounds on the coordination complexity of a problem imply the existence of good jointly differentially private algorithms for solving that problem, which in turn are known to upper bound the price of anarchy in certain games with dynamically changing populations. We show several results. We fully characterize the coordination complexity for the problem of computing a many-to-one matching in a bipartite graph by giving almost matching lower and upper bounds.Our upper bound in fact extends much more generally, to the problem of solving a linearly separable convex program. We also give a different upper bound technique, which we use to bound the coordination complexity of coordinating a Nash equilibrium in a routing game, and of computing a stable matching.

We study a very general class of games -- multi-dimensional aggregative games -- which in particular generalize both anonymous games and weighted congestion games. For any such game that is also large, we solve the equilibrium selection problem in a strong sense. In particular, we give an efficient weak mediator: a mechanism which has only the power to listen to reported types and provide non-binding suggested actions, such that a it is an asymptotic Nash equilibrium for every player to truthfully report their type to the mediator, and then follow its suggested action; and b that when players do so, they end up coordinating on a particular asymptotic pure strategy Nash equilibrium of the induced complete information game. In fact, truthful reporting is an ex-post Nash equilibrium of the mediated game, so our solution applies even in settings of incomplete information, and even when player types are arbitrary or worst-case i.e. not drawn from a common prior. We achieve this by giving an efficient differentially private algorithm for computing a Nash equilibrium in such games. The rates of convergence to equilibrium in all of our results are inverse polynomial in the number of players n. We also apply our main results to a multi-dimensional market game. Our results can be viewed as giving, for a rich class of games, a more robust version of the Revelation Principle, in that we work with weaker informational assumptions no common prior, yet provide a stronger solution concept ex-post Nash versus Bayes Nash equilibrium. In comparison to previous work, our main conceptual contribution is showing that weak mediators are a game theoretic object that exist in a wide variety of games --- previously, they were only known to exist in traffic routing games. We also give the first weak mediator that can implement an equilibrium optimizing a linear objective function, rather than implementing a possibly worst-case Nash equilibrium.

We consider a private variant of the classical allocation problem: given k goods and n agents with individual, private valuation functions over bundles of goods, how can we partition the goods amongst the agents to maximize social welfare? An important special case is when each agent desires at most one good, and specifies her (private) value for each good: in this case, the problem is exactly the maximum-weight matching problem in a bipartite graph. Private matching and allocation problems have not been considered in the differential privacy literature, and for good reason: they are plainly impossible to solve under differential privacy. Informally, the allocation must match agents to their preferred goods in order to maximize social welfare, but this preference is exactly what agents wish to hide. Therefore, we consider the problem under the relaxed constraint of joint differential privacy: for any agent i, no coalition of agents excluding i should be able to learn about the valuation function of agent i. In this setting, the full allocation is no longer published---instead, each agent is told what good to get. We first show that with a small number of identical copies of each good, it is possible to efficiently and accurately solve the maximum weight matching problem while guaranteeing joint differential privacy. We then consider the more general allocation problem, when bidder valuations satisfy the gross substitutes condition. Finally, we prove that the allocation problem cannot be solved to non-trivial accuracy under joint differential privacy without requiring multiple copies of each type of good., Comment: Journal version published in SIAM Journal on Computation; an extended abstract appeared in STOC 2014

International audience; We present a practical, differentially private algorithm for answering a large number of queries on high dimensional datasets. Like all algorithms for this task, ours necessarily has worst-case complexity exponential in the dimension of the data. However, our algorithm packages the computationally hard step into a concisely defined integer program, which can be solved non-privately using standard solvers. We prove accuracy and privacy theorems for our algorithm, and then demonstrate experimentally that our algorithm performs well in practice. For example , our algorithm can efficiently and accurately answer millions of queries on the Netflix dataset, which has over 17,000 attributes; this is an improvement on the state of the art by multiple orders of magnitude.