Search

The rise of social media has changed the nature of the fashion industry. Influence is no longer concentrated in the hands of an elite few: social networks have distributed power across a broader set of tastemakers. To understand this new landscape of influence, we created FITNet --- a network of the top 10k influencers of the larger Twitter fashion graph. To construct FITNet, we trained a content-based classifier to identify fashion-relevant Twitter accounts. Leveraging this classifier, we estimated the size of Twitter's fashion subgraph, snowball sampled more than 300k fashion-related accounts based on following relationships, and identified the top 10k influencers in the resulting subgraph. We use FITNet to perform a large-scale analysis of fashion influencers, and demonstrate how the network facilitates discovery, surfacing influencers relevant to specific fashion topics that may be of interest to brands, retailers, and media companies.

In this article, we give Wedderburn principal decomposition for locally trivial Jordan algebra bundles using Levi decomposition of Lie algebra bundles.

Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., “send money to Joe”), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., “Joe”) to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.

Measuring similarity between two images often requires performing complex reasoning along different axes (e.g., color, texture, or shape). Insights into what might be important for measuring similarity can can be provided by annotated attributes, but prior work tends to view these annotations as complete, resulting in them using a simplistic approach of predicting attributes on single images, which are, in turn, used to measure similarity. However, it is impractical for a dataset to fully annotate every attribute that may be important. Thus, only representing images based on these incomplete annotations may miss out on key information. To address this issue, we propose the Pairwise Attribute-informed similarity Network (PAN), which breaks similarity learning into capturing similarity conditions and relevance scores from a joint representation of two images. This enables our model to identify that two images contain the same attribute, but can have it deemed irrelevant (e.g., due to fine-grained differences between them) and ignored for measuring similarity between the two images. Notably, while prior methods of using attribute annotations are often unable to outperform prior art, PAN obtains a 4-9% improvement on compatibility prediction between clothing items on Polyvore Outfits, a 5% gain on few shot classification of images using Caltech-UCSD Birds (CUB), and over 1% boost to Recall@1 on In-Shop Clothes Retrieval. Implementation available at https://github.com/samarth4149/PAN, Accepted to ICCV 2021

The Rico dataset, containing design data from more than 9.7 k Android apps spanning \(27\) categories, was released in 2017. It exposes visual, textual, structural, and interactive design properties of more than 72 k unique UI screens. Over the years since its release, the original paper has been cited nearly 100 times according to Google Scholar and the dataset has been used as the basis for numerous research projects. In this chapter, we describe the creation of Rico using a system that combined crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. We then describe two projects that we conducted using the dataset: the training of an autoencoder to identify similarity between UI designs, and an exploration of the use of Google’s Material Design within the dataset using machine learned models. We conclude with an overview of other work that has used Rico to understand our mobile UI world and build data-driven models that assist users, designers, and developers.

Artificial intelligence (AI) and Human Computer Interaction (HCI) share common roots and early work on conversational agents has laid the foundation for both fields. However, in subsequent decades the initial tight connection between the fields has become less pronounced. The recent rise of deep learning has revolutionized AI and has led to a raft of practical methods and tools that significantly impact areas outside of core-AI. In particular, modern AI techniques now power new ways for machines and humans to interact. Thus it is timely to investigate how modern AI can propel HCI research in new ways and how HCI research can help direct AI developments. This workshop offers a forum for researchers to discuss new opportunities that lie in bringing modern AI methods into HCI research, identifying important problems to investigate, showcasing computational and scientific methods that can be applied, and sharing datasets and tools that are already available or proposing those that should be further developed. The topics we are interested in including deep learning methods for understanding and modeling human behaviors and enabling new interaction modalities, hybrid intelligence that combine human and machine intelligence to solve difficult tasks, and tools and methods for interaction data curation and large-scale data-driven design. At the core of these topics, we want to start the conversation on how data-driven and data-centric approaches of modern AI can impact HCI.

Sorting items by user rating is a fundamental interaction pattern of the modern Web, used to rank products (Amazon), posts (Reddit), businesses (Yelp), movies (YouTube), and more. To implement this pattern, designers must take in a distribution of ratings for each item and define a sensible total ordering over them. This is a challenging problem, since each distribution is drawn from a distinct sample population, rendering the most straightforward method of sorting --- comparing averages --- unreliable when the samples are small or of different sizes. Several statistical orderings for binary ratings have been proposed in the literature (e.g., based on the Wilson score, or Laplace smoothing), each attempting to account for the uncertainty introduced by sampling. In this paper, we study this uncertainty through the lens of human perception, and ask "How do people sort by ratings?" In an online study, we collected 48,000 item-ranking pairs from 4,000 crowd workers along with 4,800 rationales, and analyzed the results to understand how users make decisions when comparing rated items. Our results shed light on the cognitive models users employ to choose between rating distributions, which sorts of comparisons are most contentious, and how the presentation of rating information affects users' preferences.

Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

Advances in computer vision and machine learning are changing the way people dress and buy clothes. Given the vast space of fashion problems, where can data-driven technologies provide the most value? To understand consumer pain points and opportunities for technological interventions, this paper presents the results from two independent need-finding studies that explore the gold-standard of personalized shopping: interacting with a personal stylist. Through interviews with five personal stylists, we study the range of problems they address and their in-person processes for working with clients. In a separate study, we investigate how styling experiences map to online settings by building and releasing a chatbot that connects users to one-on-one sessions with a stylist, acquiring more than 70 organic users in three weeks. These conversations reveal that in-person and online styling sessions share similar goals, but online sessions often involve smaller problems that can be resolved more quickly. Based on these explorations, we propose future highly personalized, online interactions that address consumer trust and uncertainty, and discuss opportunities for automation.

Data-driven models help mobile app designers understand best practices and trends, and can be used to make predictions about design performance and support the creation of adaptive UIs. This paper presents Rico, the largest repository of mobile app designs to date, created to support five classes of data-driven applications: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. To create Rico, we built a system that combines crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. The Rico dataset contains design data from more than 9.7k Android apps spanning 27 categories. It exposes visual, textual, structural, and interactive design properties of more than 72k unique UI screens. To demonstrate the kinds of applications that Rico enables, we present results from training an autoencoder for UI layout similarity, which supports query- by-example search over UIs.

To evaluate the performance of mobile app designs, designers and researchers employ techniques such as A/B, usability, and analytics-driven testing. While these are all useful strategies for evaluating known designs, comparing many divergent solutions to identify the most performant remains a costly and difficult problem. This paper introduces a design performance testing approach that leverages existing app implementations and crowd workers to enable comparative testing at scale. This approach is manifest in ZIPT, a zero-integration performance testing platform that allows designers to collect detailed design and interaction data over any Android app -- including apps they do not own and did not build. Designers can deploy scripted tests via ZIPT to collect aggregate user performance metrics (e.g., completion rate, time on task) and qualitative feedback over third-party apps. Through case studies, we demonstrate that designers can use ZIPT's aggregate data and visualizations to understand the relative performance of interaction patterns found in the wild, and identify usability issues in existing Android apps.