Search

This paper reports on a preliminary comparative study of Alexa, Siri, and Google Assistant voice assistants (VA) that explores the origins of answers provided on each platform in an attempt to determine the extent that these origins influence responses. Questions were selected from Text Recognition (TREC) 2017 Live Question Answering (QA) Track Data, a collection of pre-assessed questions that are part of the National Institute of Standards and Technology (NIST) TREC QA track. Responses were collected as voice memos and screen captures, then analyzed to determine the origins of each answer or set of answers provided. Results indicate that the origins of answers are different search engines, and that algorithm-centered processes in each voice assistant result in vast differences in answers to questions. Because the online search results provided as answers by voice assistants are influenced by content and structured data, technical communicators and UX practitioners can help ensure that voice assistants are able to provide accurate, complete, and ethical responses to users’ questions.

As the heart of a search engine, the ranking system plays a crucial role in satisfying users' information demands. More recently, neural rankers fine-tuned from pre-trained language models (PLMs) establish state-of-the-art ranking effectiveness. However, it is nontrivial to directly apply these PLM-based rankers to the large-scale web search system due to the following challenging issues: (1) the prohibitively expensive computations of massive neural PLMs, especially for long texts in the web document, prohibit their deployments in an online ranking system that demands extremely low latency; (2) the discrepancy between existing ranking-agnostic pre-training objectives and the ad-hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online ranking system; (3) a real-world search engine typically involves a committee of ranking components, and thus the compatibility of the individually fine-tuned ranking model is critical for a cooperative ranking system. In this work, we contribute a series of successfully applied techniques in tackling these exposed issues when deploying the state-of-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first articulate a novel practice to cost-efficiently summarize the web document and contextualize the resultant summary content with the query using a cheap yet powerful Pyramid-ERNIE architecture. Then we endow an innovative paradigm to finely exploit the large-scale noisy and biased post-click behavioral data for relevance-oriented pre-training. We also propose a human-anchored fine-tuning strategy tailored for the online ranking system, aiming to stabilize the ranking signals across various online components. Extensive offline and online experimental results show that the proposed techniques significantly boost the search engine's performance.

Designers of online search and recommendation services often need to develop metrics to assess system performance. This tutorial focuses on mixed methods approaches to developing user-focused evaluation metrics. This starts with choosing how data is logged or how to interpret current logged data, with a discussion of how qualitative insights and design decisions can restrict or enable certain types of logging. When we create a metric from that logged data, there are underlying assumptions about how users interact with the system and evaluate those interactions. We will cover what these assumptions look like for some traditional system evaluation metrics and highlight quantitative and qualitative methods that investigate and adapt these assumptions to be more explicit and expressive of genuine user behavior. We discuss the role that mixed methods teams can play at each stage of metric development, starting with data collection, designing both online and offline metrics, and supervising metric selection for decision making. We describe case studies and examples of these methods applied in the context of evaluating personalized search and recommendation systems. Finally, we close with practical advice for applied quantitative researchers who may be in the early stages of planning collaborations with qualitative researchers for mixed methods metrics development.

Online search latency is a major bottleneck in deploying large-scale pre-trained language models, e.g. BERT, in retrieval applications. Inspired by the recent advances in transformer-based document expansion technique, we propose to trade offline relevance weighting for online retrieval efficiency by utilizing the powerful BERT ranker to weight the neighbour documents collected by generated pseudo-queries for each document. In the online retrieval stage, the traditional query-document matching is reduced to the much less expensive query to pseudo-query matching, and a document rank list is quickly recalled according to the pre-computed neighbour documents. Extensive experiments on the standard MS MARCO dataset with both passage and document ranking tasks demonstrate promising results of our method in terms of both online efficiency and effectiveness.

We take the development of retail formats under COVID-19 as an example, using online search data to establish a retail format index to reflect the impact of online retail formats and trends of the impact under COVID-19. Therefore, a time series model of the retail format index was established to achieve an ideal forecast effect, and a short-term dynamic forecast of the impact of COVID-19 offline retail formats was achieved. Based on the special nature of the new online business format, we start from online fresh food, online education, and online office, we use online search data to construct a retail format index. It also has a certain significance to analyze the retail format data statistics, predict trends, and study the impact of COVID-19 on the retail format.

Nowadays, with the development of social media search system When people want to make financial decisions, they tend to use online search engine like Google to find information. According to Herbert Simon [1], people search only for events related to themselves, so an increase in search volume can reflect the fact that the event has an impact on more and more people. I find that I can learn about the orientation of the public and their views on the same thing through Google trend, thus I can infer the impact on the stock market in advance. In this paper, I investigate to see how google trends can help the financial practitioner predict the changes in stock market.

Searching through large structured data is a crucial task in enterprise as well as online search. Existing keyword-based search methods were mostly designed for web document search, and support a limited variety of query types suitable for structured data. Their main underlying drawback is the lack of semantic understanding of the data and users' natural language queries, resulting in a fundamental disconnect. In this paper, we bridge that gap through effective semantic type discovery and indexing of structured data. We demonstrate S3D (Semantic Search over Structured Data), a novel system that supports queries such as finding related tables, rows or columns, amongst others, on a large scale of structured data from multiple heterogeneous sources.

One major discussion point during the COVID-19 crisis is the use of digital contact tracing applications to reduce the spreading of the Corona virus among the population. Whereas this is important for public health from a pandemic point of view, this raised many discussions about privacy and other data mining concerns in the population.In this study we monitored the search behavior of the Swiss online population from mid of May until mid of June to measure the public interest in digital contact tracing and corresponding applications. We identified the most popular keywords and we followed the online channels that were primarily used to search for such topics. Such insights are crucial to enhance the visibility and acceptance of digital contact tracing applications.

The use of the Internet for learning provides a unique and growing opportunity to revisit the task of quantifying how much people have learned about a given subject in different regions around the world. Google alone receives over 5 billion searches a day and its publicly available data provides insight into learning process that is otherwise unobservable on a global scale. In this paper we, introduce the Computer Science Literacy-Proxy Index via Search (CSLI-s), a measure that utilizes online search data to make an educated guess around trends in computer science education. This measure uses a statistical signal processing technique to compose search volumes from a spectrum of topics into a coherent score. We intentionally explore and mitigate the biases of search data and, in the process, develop CSLI-s scores that correlate with traditional, more expensive metrics of learning. We then use search-trend data to measure patterns in subject literacy across countries and over time. To the best of our knowledge, this is the first measure of learning via Internet search-trends. The Internet is becoming a standard tool for learners and, as such, we anticipate search-trend data will have growing relevance to the learning science community.

When searching the internet e.g. for a person, solution to a problem, or some topic of interest, the wanted outcome is usually specific answers. The result quality for this kind of search is reasonably precise, most of the time we get the answers we need. However, searching a second or third time with the same query, the outcome seems to be minor variations on the same results.So what if the search for information is of a different nature, more like exploring. A typical case would be when a person has a hobby, and time after time wants to search for information about it. Very soon all the quickly accessed information has already been seen, and is not that interesting in the context of new information.This paper presents an approach to Incremental Information Retrieval, where each repeated search with a given query, will provide the user with previously obscured (i.e. unseen) results. We have implemented a prototype system, called IIR, where we demonstrate and test our approach. The system targets situations where users have a continuous information need, that cannot be satisfied through a single search on the Internet, but where the user may want to see new results on the same subject over a period of days, months, or even years. A detailed description of the IIR system and results of our tests are presented.

The majority of US Internet users have searched the internet for health-related information. When people conduct these health searches, searching for information about medical treatments is among the more common reasons. While being a convenient and fast method to collect information, when used by people for health search, search engines can be biased toward results saying treatments are helpful, regardless of the truth. The presence of incorrect information in search results may potentially cause harm, especially if people believe what they read without further research or professional medical advice. In this paper, we aim to better understand the decision making process of determining the efficacy of medical treatments using search result pages. We conducted a think-aloud study in order to gain insights on strategies people use during online search for health related topics. We found that, even when participants are careful and focused on the task, biased search engine results can significantly influence people to make decisions consistent with the bias. The chief reason biased search engines results were able to influence participants is that participants often considered what the majority of the search results stated as part of their decision-making. We also found that participants looked for indications of authoritativeness and quality when evaluating online content. While rank bias and a bias towards wanting treatments to be helpful has been found in prior studies, our participants did not reveal these biases as part of their spoken thoughts. Our results imply that more attention should be paid to search engines' biases given people's bias towards accepting the most common answer in the results as the correct answer. When search results are biased toward incorrect results for health-related searches, dire consequences may be the result.

Online platforms such as LinkedIn or specialized platforms such as Glassdoor are widely used by job seekers before applying for the job. These web platforms have rating and reviews about employer and jobs. Hence a job seeker do online search for the employer, before applying for the job. They try to find if the employer and job is good for them or not, what are the pros and cons of working there etc. Therefore, these reviews and ratings have an impact on job seekers decision as it portrays the pros and cons of working in a particular firm. Hence, the main objective of this study is main objective of this study is to find how the job seekers search for online employer reviews and the impact of these reviews on employer attractiveness and job pursuit intention. The other objective is to find the most crucial job factors that are given priority by the employee. For this, the study is proposed to be conducted in two stages, first, collecting data from the website Glassdoor, having 600000 companies' reviews. In the second stage, conducting an experimental study to examine the influence of job attributes (high vs. low) and employer rating (high vs. low) on job choice and employer attractiveness.

Graph-based approximate nearest neighbor search has attracted more and more attentions due to its online search advantages. Numbers of methods studying the enhancement of speed and recall have been put forward. However, few of them focus on the efficiency and scale of offline graph-construction. For a deployed visual search system with several billions of online images in total, building a billion-scale offline graph in hours is essential, which is almost unachievable by most existing methods. In this paper, we propose a novel algorithm called Binary Distributed Graph to solve this problem. Specifically, we combine binary codes with graph structure to speedup online and offline procedures, and achieve comparable performance with the ones in real-value based scenarios by recalling more binary candidates. Furthermore, the graph-construction is optimized to completely distributed implementation, which significantly accelerates the offline process and gets rid of the limitation of memory and disk within a single machine. Experimental comparisons on Alibaba Commodity Data Set (more than three billion images) show that the proposed method outperforms the state-of-the-art with respect to the online/offline trade-off., This paper has been accepted by CIKM2019. Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019

We analyze the consumer-age-specific patterns of restaurant preferences in commercial areas of Seoul, through the mining of place recommendation results from the Naver Place online service. We calculate indices for 188 distinct areas of Seoul measuring the heterogeneity of taste across age groups, and the dominance of any one age group over the general options presented to the public. Our results suggest that both high-traffic and rapidly changing commercial areas present diverse options appealing to all age groups, and that this diversity is primarily driven by the tastes of younger age groups. Recognizing these patterns may help stakeholders predict gentrification and proactively shape neighborhood transformation from business turnover. This study contributes to the broader literature on applying online behavioral data to study urban economic activity.

Exploring Hidden Points of Interest (H-POIs), which are rarely referred in online search and recommendation systems due to insufficient check-in records, benefits business and individuals. In this work, we investigate how to eliminate the hidden feature of H-POIs by enhancing conventional crowdsourced ranking aggregation framework with heterogeneous (i.e., H-POI and Popular Point of Interest (P-POI)) pairwise tasks. We propose a two-phase solution focusing on both effectiveness and efficiency. In offline phase, we substantially narrow down the search space by retrieving a set of geo-textual valid heterogeneous pairs as the initial candidates and develop two practical data-driven strategies to compute worker qualities. In the online phase, we minimize the cost of assessment by introducing an active learning algorithm to jointly select pairs and workers with worker quality, uncertainty of P-POI rankings and uncertainty of the model taken into account. In addition, a (Minimum Spanning) Tree-constrained Skip search strategy is proposed for the purpose of reducing search time cost. Empirical experiments based on real POI datasets verify that the ranking accuracy of H-POIs can be greatly improved with small number of query iterations.

We develop a model of consumer search with spatial learning in which sampling the payoff of one product causes consumers to update their beliefs about the payoffs of other products that are nearby in attribute space. Spatial learning gives rise to path dependence, as each new search decision depends on past experiences through the updating process. We present evidence of spatial learning in data that records online search for digital cameras. Consumers’ search paths tend to converge to the chosen product in attribute space, and consumers take larger steps away from rarely purchased products. We estimate the structural parameters of the model and show that these patterns can be rationalized by our model, but not by a model without spatial learning. Eliminating spatial learning reduces consumer welfare by 12%: cross-product inferences allow consumers to locate better products in a shorter time. Spatial learning has important implications for the power of search intermediaries. We use simulations to show that consumer-optimal product recommendations are that are most informative about other products.

Airbnb is a two-sided rental marketplace offering a variety of unique and more traditional accommodation options. Similar to other online marketplaces we invest in optimizing the content surfaced on the search UI and ranking relevance to improve the guest online search experience. The unique Airbnb inventory, however, surfaces some major data challenges. Given the high stakes of booking less traditional accommodations, users can spend many days to weeks searching and scanning the description page of many accommodation ”listings” before making a decision to book. Moreover, much of the information about a listing is unstructured and can only be found by the user after they go through the details on the listing page. As a result, we have found traditional search metrics do not work well in the context of our platform. Basic metrics of single user actions, such as click-through-rates, number of listings viewed, or dwell time, are not consistently directionally correlated with our downstream business metrics. To address these issues we leverage machine learning to isolate signals of intent from rich behavioral data. These signals have key applications including analytical insights, ranking modeling inputs, and experimentation velocity. In this paper, we describe the development of a model-based user intent metric, ”intentful listing view”, which combines the signals of a variety of user micro-actions on the listing description page. We demonstrate this learned metric is directionally correlated with downstream conversion metrics and sensitive across a variety of historical search experiments.

A considerable body of research has demonstrated that online search data can be used to complement current syndromic surveillance systems. The vast majority of previous work proposes solutions that are based on supervised learning paradigms, in which historical disease rates are required for training a model. However, for many geographical regions this information is either sparse or not available due to a poor health infrastructure. It is these regions that have the most to benefit from inferring population health statistics from online user search activity. To address this issue, we propose a statistical framework in which we first learn a supervised model for a region with adequate historical disease rates, and then transfer it to a target region, where no syndromic surveillance data exists. This transfer learning solution consists of three steps: (i) learn a regularized regression model for a source country, (ii) map the source queries to target ones using semantic and temporal similarity metrics, and (iii) re-adjust the weights of the target queries. It is evaluated on the task of estimating influenza-like illness (ILI) rates. We learn a source model for the United States, and subsequently transfer it to three other countries, namely France, Spain and Australia. Overall, the transferred (unsupervised) models achieve strong performance in terms of Pearson correlation with the ground truth (> .92 on average), and their mean absolute error does not deviate greatly from a fully supervised baseline.

Search engine users always endeavor to formulate proper search queries during online search. To help users accurately express their information need during search, search engines are equipped with query suggestions to refine users' follow-up search queries. The success of a query suggestion system counts on whether we can understand and model user search intent accurately. In this work, we propose Click Feedback-Aware Network (CFAN) to provide feedback-aware query suggestions. In addition to modeling sequential search queries issued by a user, CFAN also considers user clicks on previous suggested queries as the user feedback. These clicked suggestions, together with the issued search query sequence, jointly capture the underlying search intent of users. In addition, we explicitly focus on improving the robustness of the query suggestion system through adversarial training. Adversarial examples are introduced into the training of the query suggestion system, which not only improves the robustness of system to nuisance perturbations, but also enhances the generalization performance for original training data. Extensive experiments are conducted on a recent real search engine log. The experimental results demonstrate that the proposed method, CFAN, outperforms competitive baseline methods across various situations on the task of query suggestion.

Background: Blog articles have potential value as a source of practitioner generated evidence in grey literature reviews: they could complement already accepted sources (e.g. interviews and focus groups). To be valuable to research, blog articles need to be relevant, rigorous and evidence-based.Objective: This paper focuses on the rigour of blog articles. We develop, evaluate and partially validate a set of reasoning markers that can be used to search for rigorous blog articles. We then demonstrate how these markers can be used in the online search of grey literature for software testing.Method: We identify discourse markers from literature and then select those that are explicit indicators of reasoning. We evaluate the set against a corpus of persuasive essays, and validate false negatives to refine our set further. We then demonstrate the use of the set in a search of grey literature on software testing.Results: The set of markers is reasonably successful at detecting reasoning within the corpus, achieving a precision of 89.6% in the first pass and 91.1% after refining our set. However, recall is low due to specifically focusing only on explicit indicators of reasoning (31.3% in the first pass and 38.7% in the second). Our overall F1-Score is 46.4% in the first pass and 54.3% in the second. This is acceptable for the time being as the current focus is on the quality of results retrieved from a keyword-based search. Improving the recall of results is left for future research.Conclusion: Our work provides a set of discourse markers that can be used to indicate the presence of reasoning in a blog article, and therefore provides an indication of the more rigorous blog article content. We intend to extend the work through considering the presence of evidence, and improving the relevance of blog articles found.

In 2014-2015, the U.S. Federal Trade Commission (FTC) commissioned a study to assess consumers' ability to recognize ads and other paid content in online search results and news/article feeds. The co-authors designed the study, oversaw its execution, and analyzed the results, with support from FTC staff. The goals of the research were to assess the effectiveness of methods that online services use to label ads, and to see if alternative methods of labeling ads could improve consumers' ability to recognize them. In a controlled experiment, 48 consumers interacted with both desktop and mobile Web pages that were captured from search and online magazine websites. In half of the conditions, the Web pages were modified based on established Web design guidelines to improve the clarity of ad labeling. The participants' behavior, comments, and eye movements were recorded. Initial findings of this experiment are: (a) consumers cannot always distinguish ads, paid content, and paid search results from unpaid content, and (b) improving the salience and placement of labels based on established UI design guidelines can improve consumers' ability to recognize ads, paid content, and paid search results. We conclude with implications of the results and areas for future research.

Interactive systems, such as online search interfaces, require appropriate input if they are to produce accurate information. Without this, they can be inaccurate if the user is uncertain about the keywords. Current systems do not have the means to detect uncertainty, which may lead to a negative user experience. We explore physiological and behavioral measurements as tools to implicitly detect users' uncertainty, and provide a method to integrate input variability in interactive systems. We conducted a laboratory study where participants answered questions of varying difficulty, recording physiological data via a key logger, an eye tracker, and a heart rate sensor. Our results show that participants spent significantly more time on difficult questions and looked longer at their answers before submitting them. Based on our results, we provide initial insights on how data from physiological sensors and logged user behavior can be utilized to enrich interactive systems and evaluate a user's uncertainty level.

The myriad geotagged posts in the social media constitute a vibrant information source that can be used to support geosocial search, that is, a search for geographic locations based on user activities in online social networks and microblogging platforms. Unlike a traditional geographic search, the results of a geosocial search are not restricted to predefined entities, and may reflect events, sentiments, and other matters that are expressed in the social media. A search for "jogging", for instance, will indicate popular jogging places. A search for "4-th of July Fireworks" would point out places where people watch the spectacle and tweet about it. Yet, geosocial search is different from ordinary Web search because there is no natural partition of the space into documents. There is a need to find new ways to effectively rank, filter, and present results. In this paper, we introduce a novel two-step search process of first, quickly finding relevant areas by using an arbitrarily indexed partition of the space, and second, applying clustering to the geotagged posts in the discovered areas, to present more accurate results. We propose and compare four different ranking measures for evaluating the relevance of an area to a given query. Our experiments, over a dataset of more than 40 million geotagged posts, illustrate the effectiveness of geosocial search, e.g., for finding events, or in a search based on a sentiment, in comparison to ordinary geographic search. Online search is supported by a partition-aware inverted index. Using the index, results are retrieved in a fraction of a second over millions of posts, even on a single standard machine.

Rhetorical agency in online search is not a straightforward proposition, emanating from either users or algorithms. Rather, agency emerges from assemblages that form and disassemble throughout the search session. Specific entities that coalesce into such assemblages include users, environments, technologies, networks, hardware, software, corporations, ideologies, and data structures. This paper proposes Levi Bryant's [11] onto-cartography as a framework for depicting emergent assemblage agency across four aspects of online search: 1) search and its results page; 2) the content of search results; 3) the searcher's research literacy; and 4) the ambient environment of the search session. The paper suggests methods to be used for studying emergent agency and encourages technical communicators to engage in studies that identify and trace rhetorical agency, especially in post-human assemblages of human and nonhuman actants.

The unprecedented increase in online search user diversity across the globe has led to new challenges for search engine providers. In particular, among these challenges is the need to better support individuals who are proficient in multiple languages. To investigate this particular user characteristic, this paper presents an analysis of multilingual search user behaviors through a series of large-scale studies using crowdsourcing. Results show that multilingual users make significant use of each of their languages when searching, and that there are significant differences in behaviors between querying and result selection. In addition, results show that language use strongly depends on a number of task factors and individual user characteristics. These results are discussed in terms of building novel adaptive multilingual search solutions that better support and adapt to users who have multiple language abilities.

This study used wheel activity as an indicator of users' browsing strategy, and explored the effects of various contextual factors on users' browsing patterns. Users' wheel activities were extracted from a search log in a user experiment, in which forty participants with different backgrounds conducted an online search in various contexts. To statistically test the potential effects of contexts on browsing strategy, we calculated three types of scroll-based variables to capture different types of browsing behaviors, and analyzed the effects of contextual factors via OLS regression analysis. Our results revealed that information understanding type tasks might lead to more proportion of revisit browsing range, while higher pressure level and higher pre-familiarity might lead to larger one-time browsing range; however, time constraint did not influence their browsing strategies. Our study also provided methodological implication for future interactive information retrieval studies by testifying and highlighting the usefulness of users' wheel activities in depicting online browsing patterns.

Online search evaluation, and A/B testing in particular, is an irreplaceable tool for modern search engines. Typically, online experiments last for several days or weeks and require a considerable portion of the search traffic. This restricts their usefulness and applicability. To alleviate the need for large sample sizes in A/B experiments, several approaches were proposed. Primarily, these approaches are based on increasing the sensitivity (informally, the ability to detect changes with less observations) of the evaluation metrics. Such sensitivity improvements are achieved by applying variance reduction methods, e.g. stratification and control covariates. However, the ability to learn sensitive metric combinations that (a) agree with the ground-truth metric, and (b) are more sensitive, was not explored in the A/B testing scenario. In this work, we aim to close this gap. We formulate the problem of finding a sensitive metric combination as a data-driven machine learning problem and propose two intuitive optimization approaches to address it. Next, we perform an extensive experimental study of our proposed approaches. In our experiments, we use a dataset of 118 A/B tests performed by Yandex and study eight state-of-the-art ground-truth user engagement metrics, including Sessions per User and Absence Time. Our results suggest that a considerable sensitivity improvements over the ground-truth metrics can be achieved by using our proposed approaches.

Increasingly central to online search and information discovery are methods from Natural Language Processing (NLP). Research in the field, however, is progressing at what feels a frenetic pace, making it a daunting task to keep up with the latest results. This talk will describe some of the tasks and sub-areas of NLP that have experienced fundamental advances in the state of the art over past few years, focusing on what these advances mean for understanding and extracting information from online text.

This paper analyzes the current active Medical Tourism websites and discusses the availability of various features among these websites. The analysis is based on the collected data about the features' importance and users' online search patterns from people have experiences in Medical Tourism and medical information search on the Internet. This study focuses on the Medical Tourism market in Thailand due to its popularity and high ranking among other international Medical Tourism destinations. A Medical Tourism aggregative system is proposed by using one of the Data Mining tools - Association Rule.

Design plays an important role in adoption of apps. App design, however, is a complex process with multiple design activities. To enable data-driven app design applications, we present interaction mining -- capturing both static (UI layouts, visual details) and dynamic (user flows, motion details) components of an app's design. We present ERICA, a system that takes a scalable, human-computer approach to interaction mining existing Android apps without the need to modify them in any way. As users interact with apps through ERICA, it detects UI changes, seamlessly records multiple data-streams in the background, and unifies them into a user interaction trace. Using ERICA we collected interaction traces from over a thousand popular Android apps. Leveraging this trace data, we built machine learning classifiers to detect elements and layouts indicative of 23 common user flows. User flows are an important component of UX design and consists of a sequence of UI states that represent semantically meaningful tasks such as searching or composing. With these classifiers, we identified and indexed more than 3000 flow examples, and released the largest online search engine of user flows in Android apps.

Software APIs often contain too many methods and parameters for developers to memorize or navigate effectively. Instead, developers resort to finding answers through online search engines and systems such as Stack Overflow. However, the process of finding and integrating a working solution is often very time-consuming. Though code search engines have increased in quality, there remain significant language- and workflow-gaps in meeting end-user needs. Novice and intermediate programmers often lack the language to query, and the expertise in transferring found code to their task. To address this problem, we present CodeMend, a system to support finding and integration of code. CodeMend leverages a neural embedding model to jointly model natural language and code as mined from large Web and code datasets. We also demonstrate a novel, mixed-initiative, interface to support query and integration steps. Through CodeMend, end-users describe their goal in natural language. The system makes salient the relevant API functions, the lines in the end-user's program that should be changed, as well as proposing the actual change. We demonstrate the utility and accuracy of CodeMend through lab and simulation studies.

To combat the ease of online shopping in pajamas, offline mall owners focus increasingly on driving satisfaction and improving retention by identifying customers' preferences. However, most of these studies are based on customers' offline consuming history only. Benefiting from the internet, we can also get customers' online behaviors, such as the search logs, web browsing logs, online shopping logs, and so on. Might these seemingly irrelevant information from two different modalities (i.e. online and offline) be somehow interrelated? How can we make use of the online behaviors and offline actions jointly to promote recommendation for offline retailing? In this study, we formulate this task as a cross-modality recommendation problem, and present its solution via a proposed probabilistic graphical model, called Online-to-Offline Topic Modeling (O2OTM). Specifically, this method explicitly models the relationships between online and offline topics so that the likelihood of both online and offline behaviors is maximized. Then, the recommendation is made only based on the pairs of online and offline topics, denoted by (t,l), with high values of lift, such that the existence of the online topic $t$ greatly increases the response on the corresponding offline topic $l$ compared with the average response for the population without the online topic t. Furthermore, we evaluate this solution in both live and retrospect experiments. The real-world deployment of this model for the anniversary promotion campaign of a famous shopping mall in Beijing shows that our approach increases the occurred customer purchases per promotion message by 29.75\% compared with the baseline. Also, our model finds some interesting interpretable relationships between the online search topics and offline brand topics.

While a major share of prior work have considered search sessions as the focal unit of analysis for seeking behavioral insights, search tasks are emerging as a competing perspective in this space. In the current work, we quantify user search task behavior for both single- as well as multi-task search sessions and relate it to tasks and topics. Specifically, we analyze user-disposition, topic and user-interest level heterogeneities that are prevalent in search task behavior. Our results show that while search multi-tasking is a common phenomenon among the search engine users, the extent and choice of multi-tasking topics vary significantly across users. We find that not only do users have varying propensities to multi-task, they also search for distinct topics across single-task and multi-task sessions. To our knowledge, this is among the first studies to fully characterize online search tasks with a focus on user- and topic-level differences that are observable from search sessions.

In the age of the internet, the ability to fully control online information about oneself is no longer possible; once information is online, it is easily disseminated and quickly becomes part of the internet archive. In this paper, we explore data from a set of in-person questionnaires and contextual interviews with both searchers and searched individuals and an additional data set from an online survey of internet users to identify strategies for vetting, managing, and interpreting content online. We see a desire for agency, a contextualization of information based on social context and temporal context, and a desire to constrain the exploration of online content to a relevant subset of information when searching others, i.e. a desire for selectiveness. We offer an evocative design sketch to highlight these issues and foster a discussion of emerging socio-cultural norms around online search behavior.

The US Digital Millennium Copyright Act (DMCA) of 1998 [1] adopted a notice-and-take-down procedure to help tackle alleged online infringements through online service providers' actions. The European Directive 2000/31/EC (e-Commerce Directive) [2] introduced similar liability exemptions, but did not specify any take-down procedure. Many intermediary (host, and online search engine) service providers even in Europe have followed this notice-and-take-down procedure to enable copyright owners to issue notices to take down allegedly infringing Web resources. However, the accuracy of take-down is not known, and notice receivers do not reveal clear information about how they check the legitimacy of these requests, about whether and how they check the lawfulness of allegedly infringing content, or what criteria they use for these actions. In this paper, we use Google's Transparency Report as the benchmark to investigate the information content of take-down notices and the accuracy of the resulting take-downs of allegedly infringing Web resources. The analysis of copyright infringement is limited to the five scenarios most frequently encountered in our study of Web resources. Based on our investigation, we propose a Content-Linking-Context (CLC) model of the criteria to be considered by intermediary service providers to achieve more accurate take-down.

Online form-filling and transactions are extremely common, both for industry and government; and it is important to provide a satisfying user experience during these tasks if customers or citizens are to continue using online channels. However, reliable measures of experience in these cases are limited. Other areas of information interaction, e.g., online search, news, and shopping, are increasingly exploring and attempting to measure the concept of user engagement (UE). In this study, we ask whether UE is an appropriate outcome for the utilitarian activities of online form-filling and transactions.We describe work in progress which measures UE using the User Engagement Scale (UES) with utilitarian tasks, and which looks for behaviours which correlate with the UES. Early results suggest that, first, the UES can be adapted to such situations; and second, that readily observable user behaviours including time on site, mouse movements, and keypresses correlate with UES sub-scales and can, to some extent, predict users' responses.

While search expertise has long been considered an important variable in the study of online information search, the creation of a robust measure to characterize it has eluded researchers. Instead, those wishing to measure search expertise often include a single item that asks people to indicate how often they search or for how long they have been searching. In this paper, we report initial results of a four-phase study aimed at developing a measure of search expertise. Interviews and focus groups were conducted with nine professional searchers and an inventory of items related to online search expertise was created. While participants identified a number of usual items for determining search expertise, such as prior search experience, domain knowledge and resource knowledge, they also identified a large number of items related to personality characteristics such as persistence, flexibility, curiosity, adaptability and humility. These results suggest that measuring personality characteristics in addition to experience and knowledge might help determine who will be a more successful searcher. In today's search environments, where many people describe themselves as experts, representing these types of characteristics in a measure of search expertise might allow researchers and practitioners to observe more variation in populations of interest, design more effective instruction and make better predictions about search outcomes.

Multi-tasking within a single online search sessions is an increasingly popular phenomenon. In this work, we quantify multi-tasking behavior of web search users. Using insights from large-scale search logs, we seek to characterize user groups and search sessions with a focus on multi-task sessions. Our findings show that dual-task sessions are more prevalent than single-task sessions in online search, and that over 50\% of search sessions have more than 2 tasks. Further, we provide a method to categorize users into focused, multi-taskers or supertaskers depending on their level of task-multiplicity and show that the search effort expended by these users varies across the groups. The findings from this analysis provide useful insights about task-multiplicity in an online search environment and hold potential value for search engines that wish to personalize and support search experiences of users based on their task behavior.

During online search, users frequently modify the existing query to get better results. For voice search users, the system error is a second reason of query reformulation. In keyboard system, query reformulation usually involves partial modification which is not well supported by the current voice search systems. In this proposal, I aim to develop a set of voice commands which will improve the user experience in query reformulation during voice search. The study will include four phases. Firstly, a survey will be used to collect the user's perception of the current voice search system. Secondly, an experiment will be performed to find out how the users would like to communicate with the system. Thirdly, a set of voice commands will be developed based on the findings of the second phase. Finally, another experiment will be used to test the effects of the voice commands.

The cyber space is exploding in rapidity that nobody has ever imagined, it becomes very essential to search the web of cyber space efficiently and effectively. One solution came up as an resolution to this problem is search engines. By now a lot of commercial business search engines have been put on the market. The web has grown up a lot in terms of search engines. However these search engines sometimes respond with awkward and bulky results, this was unbearable for domain specific experts. Numerous domain specific search engines are being developed using a dedicate hardware and a commercial software. These engines are widely used for variety of purposes and topic specific searches. A domain specific search engine is different from a general web search engine; the domain specific search focuses on a specific segment of online content. They are also called specialty or topical search engines. The domain specific content area may be based on different topics i.e. topicality based, type of media, or type of content. General verticals include shopping, the automotives, legal information, medical, literature, and travel. In this paper we will be studying the changing nature of web search in context to domain specific search[1].

Online Search (OLS) is a key component of many popular Internet services. Datacenters running OLS consume significant amounts of energy. However, reducing their energy is challenging due to their tight response time requirements. A key aspect of OLS is that each user query goes to all or many of the nodes in the cluster, so that the overall time budget is dictated by the tail of the replies' latency distribution; replies see latency variations both in the network and compute. Previous work proposes to achieve load-proportional energy by slowing down the computation at lower datacenter loads based directly on response times (i.e., at lower loads, the proposal exploits the average slack in the time budget provisioned for the peak load). In contrast, we propose TimeTrader to reduce energy by exploiting the latency slack in the sub-critical replies which arrive before the deadline (e.g., 80% of replies are 3–4× faster than the tail). This slack is present at all loads and subsumes the previous work's load-related slack. While the previous work shifts the leaves' response time distribution to consume the slack at lower loads, TimeTrader reshapes the distribution at all loads by slowing down individual sub-critical nodes without increasing missed deadlines. TimeTrader exploits slack in both the network and compute budgets. Further, TimeTrader leverages Earliest Deadline First scheduling to largely decouple critical requests from the queuing delays of sub-critical requests which can then be slowed down without hurting critical requests. A combination of real-system measurements and at-scale simulations shows that without adding to missed deadlines, TimeTrader saves 15% and 40% energy at 90% and 30% loading, respectively, in a datacenter with 512 nodes, whereas previous work saves 0% and 30%. Further, as a proof-of-concept, we build a small-scale real implementation to evaluate TimeTrader and show 10–30% energy savings.

Top search ad placement is the coin of today's Internet services realm. An entire industry of search engine marketing companies have emerged to help advertisers optimize their ad campaigns to deliver high returns on investment, peddling a plethora of advertising strategies. Yet, very little is publicly known about the effectiveness of online search advertising, especially when trying to compare the various campaign strategies used by advertisers.This paper presents the first large-scale measurement of the effectiveness-measured in terms of incremental conversion gains'of online search ads. We develop a simple metric called net acquisition benefit (NAB) that admits comparisons between the efficacy of different ad campaign strategies without access to advertisers' private financial information. We study three common campaign strategies used by advertisers on a large search ad network: cannibalization, poaching, and ad extensions. Considering data from a month in the last two years, we employ NAB to identify cases where these campaign strategies are justified. Advertisers and ad agencies can replicate our methodology to apply it to other strategies of interest.

Query understanding is a challenging but beneficial task. In this paper, we propose a context-aware method to use the encyclopedic knowledge to aid in query understanding. Given a query, we first use a dictionary constructed from the encyclopedic knowledge bases to detect the possible entities and their associated categories. Then, we use a topic based ethod to derive semantic information from the query. By comparing the topical similarity between various candidate phrases, we get the most likely entities and their related categories. Experimental results show that our method has achieved a great improvement over previous approaches and the efficiency is acceptable for online search.

Modern Internet companies improve evaluation criteria of their data-driven decision-making that is based on online controlled experiments (also known as A/B tests). The amplitude metrics of user engagement are known to be well sensitive to service changes, but they could not be used to determine, whether the treatment effect is positive or negative. We propose to overcome this sign-agnostic issue by paying attention to the phase of the corresponding DFT sine wave. We refine the amplitude metrics of the first frequency by the phase ones and formalize our intuition in several novel overall evaluation criteria. These criteria are then verified over A/B experiments on real users of Yandex. We find that our approach holds the sensitivity level of the amplitudes and makes their changes sign-aware w.r.t. the treatment effect.

Yandex is one of the largest Internet companies in Europe, operating Russia's most popular search engine, generating 58.6\% of all search traffic in Russia (as of April 2015). As all modern search engines, Yandex increasingly relies on online evaluation methods such as A/B tests and interleaving. These online evaluation methods test various changes in the search engine by analyzing the changes in the character of its interactions with its users. There are several grand challenges in online evaluation, including the choice of an appropriate online metric and the need to deal the limited number of user interactions available for a search engine for experimentation. In my talk, I will overview our latest research on improving the sensitivity of well-known online metrics, on discovery of more sensitive and robust online metrics, on scheduling and early stopping of online experiments.

Faceted browsing has become ubiquitous with modern digital libraries and online search engines, yet the process is still difficult to abstractly model in a manner that supports the development of interoperable and reusable interfaces. Existing efforts in facet modeling are based upon set theory, formal concept analysis, and light-weight ontologies, but in many regards, they are implementations of faceted browsing rather than a specification of the basic, underlying structures and interactions. We propose category theory as a theoretical foundation for faceted browsing and demonstrate how the interactive process can be mathematically abstracted in a way that naturally supports interoperability and reuse.

Online search and item recommendation systems are often based on being able to correctly label items with topical keywords. Typically, topical labelers analyze the main text associated with the item, but social media posts are often multimedia in nature and contain contents beyond the main text. Topic labeling for social media posts is therefore an important open problem for supporting effective social media search and recommendation. In this work, we present a novel solution to this problem for Google+ posts, in which we integrated a number of different entity extractors and annotators, each responsible for a part of the post (e.g. text body, embedded picture, video, or web link). To account for the varying quality of different annotator outputs, we first utilized crowdsourcing to measure the accuracy of individual entity annotators, and then used supervised machine learning to combine different entity annotators based on their relative accuracy. Evaluating using a ground truth data set, we found that our approach substantially outperforms topic labels obtained from the main text, as well as naive combinations of the individual annotators. By accurately applying topic labels according to their relevance to social media posts, the results enables better search and item recommendation.

Online evaluation allows information retrieval systems to be assessed based on how real users respond to search results presented. Compared with traditional offline evaluation based on manual relevance assessments, online evaluation is particularly attractive in settings where reliable assessments are difficult or too expensive to obtain. However, the successful use of online evaluation requires the right metrics to be used, as real user behaviour is often difficult to interpret. I will present interleaving, a sensitive online evaluation approach that creates paired comparisons for every user query, and compare it with alternative A/B online evaluation approaches. I will also show how interleaving can be parameterized to create a family of evaluation metrics that can be chosen to best match the goals of an evaluation.

Probabilistic topic models, such as PLSA and LDA, are gaining popularity in many fields due to their high-quality results. Unfortunately, existing topic models suffer from two drawbacks: (1) model complexity and (2) disjoint topic groups. That is, when a topic model involves multiple entities (such as authors, papers, conferences, and institutions) and they are connected through multiple relationships, the model becomes too difficult to analyze and often leads to in-tractable solutions. Also, different entity types are classified into disjoint topic groups that are not directly comparable, so it is difficult to see whether heterogeneous entities (such as authors and conferences) are on the same topic or not (e.g., are Rakesh Agrawal and KDD related to the same topic?). In this paper, we propose a novel universal topic framework (UniZ) that addresses these two drawbacks using "prior topic incorporation." Since our framework enables representation of heterogeneous entities in a single universal topic space, all entities can be directly compared within the same topic space. In addition, UniZ breaks complex models into much smaller units, learns the topic group of each entity from the smaller units, and then propagates the learned topics to others. This way, it leverages all the available signals without introducing significant computational complexity, enabling a richer representation of entities and highly accurate results. In a widely-used DBLP dataset prediction problem, our approach achieves the best prediction performance over many state-of-the-art methods. We also demonstrate practical potential of our approach with search logs from a commercial search engine.

The purpose of this study is to investigate the extent to which two theories, Information Scent and Need for Cognition, explain people's search behaviors when interacting with search engine results pages (SERPs). Information Scent, the perception of the value of information sources, was manipulated by varying the number and distribution of relevant results on the first SERP. Need for Cognition (NFC), a personality trait that measures the extent to which a person enjoys cognitively effortful activities, was measured by a standardized scale. A laboratory experiment was conducted with forty-eight participants, who completed six open-ended search tasks. Results showed that while interacting with SERPs containing more relevant documents, participants examined more documents and clicked deeper in the search result list. When interacting with SERPs that contained the same number of relevant results distributed across different ranks, participants were more likely to abandon their queries when relevant documents appeared later on the SERP. With respect to NFC, participants with higher NFC paginated less frequently and paid less attention to results at lower ranks than those with lower NFC. The interaction between NFC and the number of relevant results on the SERP affected the time spent on searching and a participant's likelihood to reformulate, paginate and stop. Our findings suggest evaluating system effectiveness based on the first page of results, even for tasks that require the user to view multiple documents, and varying interface features based on NFC.