Search

The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures are paramount to to ensure non-dominant values, knowledge and material culture are also reflected in AI models and the media they generate, we argue that dominant structures of community participation in AI development and evaluation are not explicit enough about the benefits and harms that members of socially marginalized groups may experience as a result of their participation. Without explicit interrogation of these benefits by AI developers, as a community we may remain blind to the immensity of systemic change that is needed as well. To support this provocation, we present a speculative case study, developed from our own collective experiences as AI researchers. We use this speculative context to itemize the barriers that need to be overcome in order for the proposed benefits to marginalized communities to be realized, and harms mitigated., Comment: 3 pages, 1 figure. Published as a Short Paper in the NeurIPS 2024 Workshop on Evaluating Evaluations: Examining Best Practices for Measuring Broader Impacts of Generative AI

Platform-based laborers face unprecedented challenges and working conditions that result from algorithmic opacity, insufficient data transparency, and unclear policies and regulations. The CSCW and HCI communities increasingly turn to worker data collectives as a means to advance related policy and regulation, hold platforms accountable for data transparency and disclosure, and empower the collective worker voice. However, fundamental questions remain for designing, governing and sustaining such data infrastructures. In this workshop, we leverage frameworks such as data feminism to design sustainable and power-aware data collectives that tackle challenges present in various types of online labor platforms (e.g., ridesharing, freelancing, crowdwork, carework). While data collectives aim to support worker collectives and complement relevant policy initiatives, the goal of this workshop is to encourage their designers to consider topics of governance, privacy, trust, and transparency. In this one-day session, we convene research and advocacy community members to reflect on critical platform work issues (e.g., worker surveillance, discrimination, wage theft, insufficient platform accountability) as well as to collaborate on codesigning data collectives that ethically and equitably address these concerns by supporting working collectivism and informing policy development.

Rideshare platforms exert significant control over workers through algorithmic systems that can result in financial, emotional, and physical harm. What steps can platforms, designers, and practitioners take to mitigate these negative impacts and meet worker needs? In this paper, through a novel mixed methods study combining a LLM-based analysis of over 1 million comments posted to online platform worker communities with semi-structured interviews of workers, we thickly characterize transparency-related harms, mitigation strategies, and worker needs while validating and contextualizing our findings within the broader worker community. Our findings expose a transparency gap between existing platform designs and the information drivers need, particularly concerning promotions, fares, routes, and task allocation. Our analysis suggests that rideshare workers need key pieces of information, which we refer to as indicators, to make informed work decisions. These indicators include details about rides, driver statistics, algorithmic implementation details, and platform policy information. We argue that instead of relying on platforms to include such information in their designs, new regulations that require platforms to publish public transparency reports may be a more effective solution to improve worker well-being. We offer recommendations for implementing such a policy.

Online discussion forums provide crucial data to understand the concerns of a wide range of real-world communities. However, the typical qualitative and quantitative methods used to analyze those data, such as thematic analysis and topic modeling, are infeasible to scale or require significant human effort to translate outputs to human readable forms. This study introduces QuaLLM, a novel LLM-based framework to analyze and extract quantitative insights from text data on online forums. The framework consists of a novel prompting methodology and evaluation strategy. We applied this framework to analyze over one million comments from two Reddit's rideshare worker communities, marking the largest study of its type. We uncover significant worker concerns regarding AI and algorithmic platform decisions, responding to regulatory calls about worker insights. In short, our work sets a new precedent for AI-assisted quantitative data analysis to surface concerns from online forums., Comment: Accepted to CHI LLM as Research Tools Workshop (2024)

This paper examines the sociotechnical infrastructure of an "indie" food delivery platform. The platform, Nosh, provides an alternative to mainstream services, such as Doordash and Uber Eats, in several communities in the Western United States. We interviewed 28 stakeholders including restauranteurs, couriers, consumers, and platform administrators. Drawing on infrastructure literature, we learned that the platform is a patchwork of disparate technical systems held together by human intervention. Participants join this platform because they receive greater agency, financial security, and local support. We identify human intervention's key role in making food delivery platform users feel respected. This study provides insights into the affordances, limitations, and possibilities of food delivery platforms designed to prioritize local contexts over transnational scales., Comment: Accepted at CHI 2023

Accessibility efforts, how we can make the world usable and useful to as many people as possible, have explicitly focused on how we can support and allow for the autonomy and independence of people with disabilities, neurotypes, chronic conditions, and older adults. Despite these efforts, not all technology is designed or implemented to support everyone's needs. Recently, a community-organized push by creators and general users of TikTok urged the platform to add accessibility features, such as closed captioning to user-generated content, allowing more people to use the platform with greater ease. Our work focuses on an understudied population -- people with ADHD and those who experience similar challenges -- exploring the creative practices people from this community engage in, focusing on the kinds of accessibility they create through their creative work. Through an interview study exploring the experiences of creatives on TikTok, we find that creatives engage in critical infrastructuring -- a process of bottom-up (re)design -- to make the platform more accessible despite the challenges the platform presents to them as creators. We present these critical infrastructuring practices through the themes of: creating and augmenting video editing infrastructures and creating and augmenting video captioning infrastructures. We reflect on the introduction of a top-down infrastructure - the implementation of an auto-captioning feature - shifts the critical infrastructure practices of content creators. Through their infrastructuring, creatives revised sociotechnical capabilities of TikTok to support their own needs as well as the broader needs of the TikTok community. We discuss how the routine of infrastructuring accessibility is actually best conceptualized as incidental care work. We further highlight how accessibility is an evolving sociotechnical construct, and forward the concept of contextual accessibility., Comment: To be published in: Proc. ACM Hum.-Comput. Interact. CSCW '23