Search

Spatial crowdsourcing (SC) allows requesters to crowdsource tasks to workers based on location proximity. To preserve privacy, the location should not be disclosed to untrustworthy entities (even the SC platform). Previous solutions to preserve workers' location privacy require an online trusty third party (TTP), which is not practical in reality. In this paper, we design a framework that allows the SC platform to assign tasks to nearest workers in an online manner without knowing their actual locations. We propose an encryption algorithm to encrypt the locations of tasks and workers, and design an indexing method that assigns tasks to workers without losing too much privacy. We prove that there exists a trade-off between efficiency and security theoretically, which can be controlled based on user preference. We verify our method on real-world datasets and experimental results show that our method is efficient, effective and practical.

The use of non-probability Internet panels and crowdsource websites is increasing in gambling research. These paid online sampling methods offer a convenient and inexpensive recruitment strategy. T...

Smartphones have gained immense popularity among people and they incorporate several sophisticated capabilities like sensors that broaden the domain of usability. Consequently, smartphone-based crowdsourcing methods show importance as an efficient data acquisition strategy. GPS receiver and sensors like accelerometer and gyroscope can be used to gather data which helps identify the travel mode of the smartphone user. Numerous research projects have been conducted regarding the area of concern and this study uses GPS and Accelerometer data collected from smartphones using crowdsourced methods to develop a robust public transport mode detection approach using machine learning algorithms.

Scientists are increasingly using volunteer efforts of citizen scientists to classify images captured by motion‐activated trail cameras. The rising popularity of citizen science reflects its potential to engage the public in conservation science and accelerate processing of the large volume of images generated by trail cameras. While image classification accuracy by citizen scientists can vary across species, the influence of other factors on accuracy is poorly understood. Inaccuracy diminishes the value of citizen science derived data and prompts the need for specific best‐practice protocols to decrease error. We compare the accuracy between three programs that use crowdsourced citizen scientists to process images online: Snapshot Serengeti, Wildwatch Kenya, and AmazonCam Tambopata. We hypothesized that habitat type and camera settings would influence accuracy. To evaluate these factors, each photograph was circulated to multiple volunteers. All volunteer classifications were aggregated to a single best answer for each photograph using a plurality algorithm. Subsequently, a subset of these images underwent expert review and were compared to the citizen scientist results. Classification errors were categorized by the nature of the error (e.g., false species or false empty), and reason for the false classification (e.g., misidentification). Our results show that Snapshot Serengeti had the highest accuracy (97.9%), followed by AmazonCam Tambopata (93.5%), then Wildwatch Kenya (83.4%). Error type was influenced by habitat, with false empty images more prevalent in open‐grassy habitat (27%) compared to woodlands (10%). For medium to large animal surveys across all habitat types, our results suggest that to significantly improve accuracy in crowdsourced projects, researchers should use a trail camera set up protocol with a burst of three consecutive photographs, a short field of view, and determine camera sensitivity settings based on in situ testing. Accuracy level comparisons such as this study can improve reliability of future citizen science projects, and subsequently encourage the increased use of such data., We show that the accuracy levels of trail‐camera image classification by citizen scientists are affected by habitat type and trail‐camera set up. By comparing the accuracy results from three camera trap citizen science projects, we found that setting trail cameras to capture 3 images per burst, testing the appropriate camera sensitivity, and a shorter field of view resulting from dense vegetation may significantly improve citizen scientist image classification accuracy when compared to classifications by experts.

The proliferation of smart home devices has created new opportunities for empirical research in ubiquitous computing, ranging from security and privacy to personal health. Yet, data from smart home deployments are hard to come by, and existing empirical studies of smart home devices typically involve only a small number of devices in lab settings. To contribute to data-driven smart home research, we crowdsource the largest known dataset of labeled network traffic from smart home devices from within real-world home networks. To do so, we developed and released IoT Inspector, an open-source tool that allows users to observe the traffic from smart home devices on their own home networks. Between April 10, 2019 and January 21, 2020, 5,404 users have installed IoT Inspector, allowing us to collect labeled network traffic from 54,094 smart home devices. At the time of publication, IoT Inspector is still gaining users and collecting data from more devices. We demonstrate how this data enables new research into smart homes through two case studies focused on security and privacy. First, we find that many device vendors, including Amazon and Google, use outdated TLS versions and send unencrypted traffic, sometimes to advertising and tracking services. Second, we discover that smart TVs from at least 10 vendors communicated with advertising and tracking services. Finally, we find widespread cross-border communications, sometimes unencrypted, between devices and Internet services that are located in countries with potentially poor privacy practices. To facilitate future reproducible research in smart homes, we will release the IoT Inspector data to the public.

This article presents a two-step methodology to annotate temporally anchored spatial knowledge on top of OntoNotes. We first generate potential knowledge using semantic roles or syntactic dependencies and then crowdsource annotations to validate the potential knowledge. The resulting annotations indicate how long entities are or are not located somewhere and temporally anchor this spatial information. We present an in-depth corpus analysis comparing the spatial knowledge generated by manipulating roles or dependencies. Experiments show that working with syntactic dependencies instead of semantic roles allows us to generate more potential entity-related spatial knowledge and obtain better results in a realistic scenario, that is, with predicted linguistic information.

Location-based services (LBSs) typically crowdsource geo-tagged data from mobile users. Collecting more data will generally improve the utility for LBS providers; however, it also leads to more privacy exposure of users’ mobility patterns. Although the tension between data utility and user privacy has been recognized, there lacks a solution that determines how much data to collect—in both spatial and temporal domains—is the “best” for both mobile users and the service provider. This article proposes a strategy toward making an optimal tradeoff such that a user submits data only if her mobility privacy will not be compromised and the data utility of the LBS provider will be sufficiently improved. To this end, we first define and formulate a concept called privacy exposure , which incorporates both the spatial distribution and the temporal transition of a user’s activity points . Second, we define and quantify data utility in terms of spatial repetitions and temporal closeness among data based on an economic principle. Then, we propose a PRivacy-preserving and UTility-Enhancing Crowdsourcing (PRUTEC) algorithm to determine, on behalf of each mobile user, whether a newly sensed piece of data should be submitted to the LBS provider. Our simulation demonstrates that PRUTEC improves the data utility of the service provider with a much less amount of data to collect and reduces privacy exposure for mobile users while collecting useful data continuously.

Flood is one of the most disruptive natural hazards, responsible for loss of lives and damage to properties. A number of cities are subject to monsoons influences and hence face the disaster almost every year. Early notification of flood incident could benefit the authorities and public to devise both short and long terms preventive measures, to prepare evacuation and rescue mission, and to relieve the flood victims. Geographical locations of affected areas and respective severities, for instances, are among the key determinants in most flood administration. Thus far, an effective means of anticipating flood in advance remains lacking. Existing tools were typically based on manually input and prepared data. The processes were tedious and thus prohibitive for real-time and early forecasts. Furthermore, these tools did not fully exploit more comprehensive information available in current big data platforms. Therefore, this paper proposes a novel flood forecasting system based on fusing meteorological, hydrological, geospatial, and crowdsource big data in an adaptive machine learning framework. Data intelligence was driven by state-of-the-art learning strategies. Subjective and objective evaluations indicated that the developed system was able to forecast flood incidents, happening in specific areas and time frames. It was also later revealed by benchmarking experiments that the system configured with an MLP ANN gave the most effective prediction, with correct percentage, Kappa, MAE and RMSE of 97.93, 0.89, 0.01 and 0.10, respectively.

Making the knowledge contained in scientific papers machine-readable and formally computable would allow researchers to take full advantage of this information by enabling integration with other knowledge sources to support data analysis and interpretation. Here we describe Biofactoid, a web-based platform that allows scientists to specify networks of interactions between genes, their products, and chemical compounds, and then translates this information into a representation suitable for computational analysis, search and discovery. We also report the results of a pilot study to encourage the wide adoption of Biofactoid by the scientific community.

Volunteered geographical information (VGI) refers to geographical information that the general public voluntarily collects and shares in the environment instead of for-profit businesses or government entities. Crowdsourcing such information on urgent needs in a disaster can improve the quick emergency responses. This study incorporates the capability of smartphone sensors, GPS, Web 2.0, VGI, and server-based technologies to design and develop a system for collecting target hazard information from volunteers. One of the most important contributions in designing this system is considering the improvement of the positional accuracy of the target buildings based on the position of the mobile device. Several approaches have been recommended for this purpose. The solutions include the use of online map services, geocoding services, and trigonometric methods based on the measurements of sensors such as camera, accelerometer, and magnetic field embedded in a smart mobile phone. The accuracy assessment showed that the trigonometric method by the means of embedded sensors would yield the best result. However, geocoding is more economical in terms of time than other methods. Potentially, the evaluation of the mobile application provided by a group of volunteers showed the overwhelming preference of crowdsource mapping over current telephone communication systems in disaster management.

Mobile ads are plagued with fraudulent clicks which is a major challenge for the advertising community. Although popular ad networks use many techniques to detect click fraud, they do not protect the client from possible collusion between publishers and ad networks. In addition, ad networks are not able to monitor the user’s activity for click fraud detection once they are redirected to the advertising site after clicking the ad. We propose a new crowdsource-based system called Click Fraud Crowdsourcing (CFC) that collaborates with both advertisers and ad networks in order to protect both parties from any possible click fraudulent acts. The system benefits from both a global view, where it gathers multiple ad requests corresponding to different ad network-publisher-advertiser combinations, and a local view, where it is able to track the users’ engagement in each advertising website. The results demonstrated that our approach offers a lower false positive rate (0.1) when detecting click fraud as opposed to proposed solutions in the literature, while maintaining a high true positive rate (0.9). Furthermore, we propose a new mobile ad charging model that benefits from our system to charge advertisers based on the duration spent in the advertiser’s website.

Crowdsourcing has become a low-cost and scalable alternative to gather relevance assessments through crowdsourcing platforms. However, when gathering subjective human judgments, it can be challenging to enforce data quality due to the lack of vision about how judges make a decision. It is important to determine the attributes that could affect the effectiveness of crowsourced-judgments in an information retrieval systems evaluation. The purpose of the experiment that is discussed in this paper is to investigate if logical reasoning ability of the crowd workers is related to the quality of the relevant judgments produced through the crowdsource process. The study also evaluates the effect of cognitive characteristics on the quality of relevance judgment compared to the gold standard dataset. Through this experiment, a comparison study is done between the quality of the judgments obtained through the crowdsourcing process and the original baseline judgments generated by the hired experts by TREC. In the study, the systems performances were measured using both of these sets of relevance judgments to see its correlation. The experimentation reveals that quality of relevance judgments is highly correlated with the logical reasoning ability of individuals. The judgment difficulty level reported by the crowdsource workers and the confidence level claimed by the workers showed a significant correlation with the quality of the judgments. Unexpectedly though, self-reported knowledge about a given topic and demographics data have no correlation with the quality of judgments produced through crowdsourcing

Video summarization is one of the key techniques to access and manage a huge chunk of videos. Video summarization is used to extract the effective contents of a video sequence to generate a concise representation of its content. The involvement of crowdsourcing models in recent years is effectively used by researchers in E-commerce domain to increase the quality of the data contents. In this paper, we present a video summarization framework based on users emotion while they watch videos by analyzing cerebral activities through Electroencephalogram (EEG) signals. Three emotions, namely happy, sad and neutral have been extracted from the EEG signals. Video frames have been synchronized with EEG signals and tagged with various emotions. Finally, a crowdsourcing model has been used for effective summarization of the videos. The qualitative assessment of video summarization has been conducted with the help of user ratings using online Google Forms application. EEG signals of 28 users have been recorded while video streams of different emotions. An average accuracy of 83.93% has been recorded in emotion classification using crowdsourcing. Output summarized videos include dynamic video skims and the corresponding audio stream for better understanding.

The email arrived midmorning on a Friday. “Thank you for your participation in our crowd reviewing (CR) program!” it read. Cheery and succinct, the message notified more than 80 expert reviewers that a manuscript was waiting for them, ready to be critiqued and considered for publication in the journal Synlett. A link within the email led the potential reviewers to a website called Filestage. There, they could click on any portion of the manuscript to pop up a box to compose comments. Half an hour after the message hit reviewers’ in-boxes, pale green circles dotted the screen where eager readers left suggestions and edits. Over the next four days, more than a dozen of the original 80 invitees anonymously debated the new chemical synthesis presented in the manuscript. They commented on the reaction’s novelty, questioned the molecular mechanisms proposed by the authors, suggested control experiments, pointed out spectra of samples

Various technologies have been utilized today for recognizing client or user in the indoor areas. These technologies incorporate RSSI, Bluetooth Low Energy Beacons, Ultrasound waves, Vision-based advances, for example, fixed camera recordings QR codes, remote gadgets, etc. RSSI fingerprinting technique requires more effort and it is also expensive to be used for indoor localization frameworks working in real-time. In this research, indoor localization based on images is investigated as an option in contrast to other indoor positioning techniques using these days. Image-based indoor positioning is more affordable than RSSI based technologies being utilized. A mobile phone camera is utilized to take the pictures of area inside the building to find the user inside the building. Sensor data from various sensors isn’t required or no extra framework is required to find the client in the building utilizing indoor positioning based on an image. Microsoft Azure Custom Vision Services are utilized to locate the client; MS Azure classifies the pictures in one of the labels made. Strategy’s attainability is demonstrated by various investigations and accomplished accuracy and review is recorded above 90%. The average precision of the trained model is recorded above 95%.

Many mountain lions are killed in the state of California every year from roadkill. To reduce these numbers, it is important that a system be built to track where these mountain lions have been around. One such system could be built using the platform-as-a-service, Firebase. Firebase is a platform service that collects and manages data that comes in through a mobile application. For the development of cross-platform mobile applications, Flutter is used as a toolkit for developers for both iOS and Android. This entire system, Cat Tracks is proposed as a crowdsource platform to track wildlife, with the current focus on California mountain lions. By building such a system, researchers could use the data to save the lives of many mountain lions.

We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.

Early warning of a potential pandemic with res- piratory symptoms is crucial for global health management. It enables timely intervention to reduce the likelihood of uncon- trollable massive virus spread. In this research, we propose to leverage the ubiquitous wearable devices to develop a wearable crowdsource system to monitor respiratory symptoms such as cough and fever. Wearable devices nowadays can directly and non-intrusively measure people’s vital signs in real-time with a variety of sensors embedded. We collect the data from wearable devices and develop machine learning algorithms to analyze the data for respiratory symptom monitoring and early warning. In particular, we focus on cough detection through multi-source data fusion (e.g., accelerometer amplitude and microphone audio). Preliminary results show that our algorithms result in higher detection accuracy and less false positive with the least use of computing resources. This research potentially transforms the way the pandemic early warning is implemented and the way we respond to public health crises in the years to come. Nanyang Technological University National Research Foundation (NRF) Accepted version This research is funded by National Research Foundation (NRF) via the Green Buildings Innovation Cluster (Grant NO.: NRF2015ENC_GBICRD001-012), administered by Building and Construction Authority (BCA) Singapore. In addition, this research is sponsored by National Research Foundation (NRF) via the Behavioural Studies in Energy, Water, Waste and Transportation Sectors (Grant NO.: BSEWWT2017_2_06), administered by National University of Singapore (NUS). Moreover, this research is funded by Nanyang Technological University (NTU) via the Data Science & Artificial Intelligence Research Centre @ NTU (Grant NO.: DSAIR@NTU).

Data visualizations have been increasingly used in oral presentations to communicate data patterns to the general public. Clear verbal introductions of visualizations to explain how to interpret the visually encoded information are essential to convey the takeaways and avoid misunderstandings. We contribute a series of studies to investigate how to effectively introduce visualizations to the audience with varying degrees of visualization literacy. We begin with understanding how people are introducing visualizations. We crowdsource 110 introductions of visualizations and categorize them based on their content and structures. From these crowdsourced introductions, we identify different introduction strategies and generate a set of introductions for evaluation. We conduct experiments to systematically compare the effectiveness of different introduction strategies across four visualizations with 1,080 participants. We find that introductions explaining visual encodings with concrete examples are the most effective. Our study provides both qualitative and quantitative insights into how to construct effective verbal introductions of visualizations in presentations, inspiring further research in data storytelling., Comment: 12 pages, 5 figures, accepted to IEEE Transaction on Visualization and Graphics

During times of crisis, information access is crucial. Given the opaque processes behind modern search engines, it is important to understand the extent to which the “picture” of the Covid-19 pandemic accessed by users differs. We explore variations in what users “see” concerning the pandemic through Google image search, using a two-step approach. First, we crowdsource a search task to users in four regions of Europe, asking them to help us create a photo documentary of Covid-19 by providing image search queries. Analysing the queries, we find five common themes describing information needs. Next, we study three sources of variation – users’ information needs, their geo-locations and query languages – and analyse their influences on the similarity of results. We find that users see the pandemic differently depending on where they live, as evidenced by the 46% similarity across results. When users expressed a given query in different languages, there was no overlap for most of the results. Our analysis suggests that localisation plays a major role in the (dis)similarity of results, and provides evidence of the diverse “picture” of the pandemic seen through Google., This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy.

International audience; This paper investigates a general framework for synchronous educational language games that simultaneously allows researchers to crowdsource learner answers in a controlled environment. Our prototype Substituto allows teachers and students to interact in real-time while undergoing language learning exercises; ensuring that the learner’s progress is not hurt by the introduction of crowdsourcing elements. We evaluate Substituto with a small-scale user study that focuses on training the use of English verb-particle constructions (VPCs), such as break down or take over, and test their use with second language learners of English of different proficiency levels over five pilot sessions. With the study we aim to ensure that our prototypical implementation behaves as expected and to identify any major design flaws that should be addressed. The preliminary results we achieved in order to evaluate the educational value, the user experience and the crowdsourcing capacity of XYZ-Bot confirm that it has the potential to become a valuable asset for language learning, a pleasant learning instrument and a crowdsourcing tool for collecting linguistic knowledge.

Most of the automated approaches to support software engineering tasks for mobile applications, use source code as entry, which due to privacy factors imposes hard constraints on the implementation of those approaches by third-party services. However, the continuous pressure from market is leading practitioners to crowdsource/outsource software engineering tasks to third-parties that provide on-the-cloud infrastructures. Therefore, approaches that work at closed-source level (i.e., do not require source code) are desirable to enable automated outsourced software engineering tasks by third-party services. Specifically, in this paper we focus on testing since is a task that has broadly accepted approaches at source-code level, but new efforts are required to enable source-codeless approaches. We analyzed the current challenges and shortfalls of the currently proposed approaches that works at source code and APK level, and present our research plan oriented to reduce the existing gap for enabling testing techniques at APK level.

The Drug-Gene Interaction Database (DGIdb, www.dgidb.org) is a web resource that provides information on drug-gene interactions and druggable genes from various sources including publications, databases, and other web-based sources in one resource. These drug, gene, and interaction claims are normalized and grouped to identify aliases, merge concepts, and reduce redundancy. The information contained in this resource is available to users through a straightforward search interface, an application programming interface (API), and TSV data downloads. DGIdb 4.0 is the latest major update of this database. Seven new sources have been added, bringing the total number of sources included to 41. Of the previously aggregated sources, 15 have been updated. DGIdb 4.0 also includes improvements to the process of drug normalization and grouping of imported sources. Other notable updates include further development of automatic jobs for routine data updates, more sophisticated query scores for interaction search results, extensive manual curation of interaction source link outs, and the inclusion of interaction directionality. A major focus of this update was integration with crowd-sourced efforts, including leveraging the curation activities of Drug Target Commons, using Wikidata to facilitate term normalization, and integrating into NDEx for producing network representations.

It is always a tourism challenge – and aspiration – to discover scenery routes and tourism attractions in unfamiliar areas. Tourism information is getting more extensive, comprehensive and complex, so first-time tourists have to manage and mine large volumes of data to better plan their trip. Nowadays, geotagged photos are uploaded by users to social media photo-sharing online websites, which become more popular and commonly used by travelers to share their tourism experiences. Handling, mining and interpreting these user-generated ‘digital footprints’ can be used to reconstruct travel trajectories of users to recover their activity and knowledge. In this research, we showcase Flickr geotagged crowdsource photo database as a source for mining users’ trajectories to effectively compute walking tourism routes. Our methodology mines tourism context by conceptualizing a set of adaptive spatiotemporal descriptors to identify photographers that show tourism activity of first-time visitors. By implementing spatial clustering, we find popular locations that are traversed by these tourism-oriented photographers’ trajectories. To analyze our approach, we develop a greedy route computation algorithm that seeks the most popular traversed locations between origin and destination points defined by the user. Results for two cities are presented, proving the robust mining and retrieving of valuable tourism context and information from social media photos. We evaluate and validate our results by comparing the computed walking routes to recognized tourism information. The computed walking routes are scenery and pass through the main popular tourism sights and landmarks in the city, including additional attractive places that are frequently visited by tourism-photographers.

We present a methodology that uses crowdsourced detections as an initial location to obtain fast and reliable hypocenter parameters for felt earthquakes using arrival-time data from the GEOFON Program. We derive selection criteria for issuing an alert message using a 3-year-long training set from the trial runs at the European-Mediterranean Seismological Centre (EMSC) to identify accurate event locations at a high confidence level. Since an event may have several crowdsourced detections, we also develop a methodology dealing with multiple triggers. We validate the selection criteria using real-time processing of recent data and demonstrate that 95% of the selected events are within 50 km distance from the traditional seismic location published by the EMSC. Since CsLoc remains essentially a seismic location algorithm, the selection criteria measure the quality of the seismological network coverage used in the location, not the method itself. We show that our methodology provides accurate locations much faster than those published by conventional seismic methods. On average, the EMSC CsLoc service can provide rapid and accurate locations within a minute after the occurrence of a felt earthquake, thus it can provide timely and accurate information on a felt earthquake to the civil protection services and the general public.

The SARS-CoV-2 pandemic has rapidly saturated healthcare resources across the globe and has led to a restricted screening process, hindering efforts at comprehensive case detection. This has not only facilitated community spread but has also resulted in an underestimation of the true incidence of disease, a statistic which is useful for policy making aimed at controlling the current pandemic and in preparing for future outbreaks. In this perspective, we present a crowdsourced platform developed by us for the true estimation of all SARS-CoV-2 infections in the community, through active self-reporting and layering other authentic datasets. The granularity of data captured by this system could prove to be useful in assisting governments to identify SARS-CoV-2 hotspots in the community facilitating lifting of restrictions in a controlled fashion.

Growing attention on the benefits of non-motorized travel has increased the demand for accurate and timely pedestrian and bicycle travel data. Advancements in technologies and the proliferation of smartphones have created new data sources that can help eliminate limitations related to small sample size and infrequent updates due to limited resources. This study reviews the emerging data sources and their current use, focusing on non-motorized travel monitoring. In this study, the emerging data are categorized into mode-unspecified and mode-specified data based on whether the mode used can be detected with no or little effort. While mode-unspecified data are collected without sorting out non-motorized travelers, mode-specified data at least know who (which mode) is being monitored. So far, commercial vendors provide a vast volume of mode-unspecified data, but their products have been mainly used for motorized trips or are in initial stages of development. Meanwhile, readily available data sources and their applications are more concentrated on mode-specified data, which have enabled varying non-motorized travel studies—including travel pattern identification, route-choice modeling, crash/air pollution exposure estimation, and new facility provision evaluation—but are mostly focused on bicycling. Despite the potential of emerging data, their use also has several challenges, such as limited mode inference, sample bias, and lack of detailed trip/traveler information due to privacy issues. More efforts are needed, such as improving data accuracy and developing robust data fusion techniques, to be able to fully utilize the emerging data sources.

Research involving Virtual Reality (VR) headsets is becoming more and more popular. However, scaling VR experiments is challenging as researchers are often limited to using one or a small number of headsets for in-lab studies. One general way to scale experiments is through crowdsourcing so as to have access to a large pool of diverse participants with relatively little expense of time and money. Unfortunately, there is no easy way to crowdsource VR experiments. We demonstrate that it is possible to implement and run crowdsourced VR experiments using a pre-existing massively multiplayer online VR social platform---VRChat. Our small (n=10) demonstration experiment required participants to navigate a maze in VR. Participants searched for two targets then returned to the exit while we captured completion time and position over time. While there are some limitations with using VRChat, overall we have demonstrated a promising approach for running crowdsourced VR experiments.

Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.

The Waymo Open Dataset has been released recently, providing a platform to crowdsource some fundamental challenges for automated vehicles (AVs), such as 3D detection and tracking. While the dataset provides a large amount of high-quality and multi-source driving information, people in academia are more interested in the underlying driving policy programmed in Waymo self-driving cars, which is inaccessible due to AV manufacturers&rsquo, proprietary protection. Accordingly, academic researchers have to make various assumptions to implement AV components in their models or simulations, which may not represent the realistic interactions in real-world traffic. Thus, this paper introduces an approach to learn a long short-term memory (LSTM)-based model for imitating the behavior of Waymo&rsquo, s self-driving model. The proposed model has been evaluated based on Mean Absolute Error (MAE). The experimental results show that our model outperforms several baseline models in driving action prediction. In addition, a visualization tool is presented for verifying the performance of the model.

The research proposes the neural network methods to include a textual dependency tree structure in classification tasks of Russian texts. Author profiling task of gender identification was chosen to test the models, and two corpora used in experiments: based on a crowdsource, and in-person polling. The first approach is based on a long short-term memory (LSTM) layers, and developed graph embedding algorithm. The second one is based on a graph convolution network and LSTM. Two syntactic parsers were used to obtain dependency trees from the texts. Input data was represented in different forms: morphological binary vectors, FastText vectors, and their combination. The developed models result was compared to the state-of-the-art, that is neural network model based on a convolutional and LSTM layers. Finally, we demonstrate that including textual dependency tree structure to input feature space improves f1-score of gender classification task on 4% for the RusPersonality dataset, and 7% for the crowdsource dataset in average. The developed models resulting f1-score is 84% and 83%, respectively.

Purpose – to explore a crowdsourcing data-driven approach to construct crowdknowledge databases for innovation through supporting creative idea generation. In the approach, social media will be used as platforms to crowdsource knowledge for producing the databases. Findings. Creativity is an essential element of innovation, but producing creative ideas is often challenging in design. Many computational tools have become available recently to support designers in producing creative ideas that are new to individuals. As a standard feature, most of the tools rely on the databases employed, such as ConceptNet and the US Patent Database. This study highlighted that the limitations of these databases have constrained the capabilities of the tools and, thereby, new computational databases supporting the generation of new ideas to a crowd or even history are needed. Crowdsourcing outsources tasks conventionally performed in-house to a crowd and uses external knowledge to solve problems and democratize innovation. Social media is often employed in crowdsourcing for a crowd to create and share knowledge. Originality/value/scientific novelty of the research. This paper proposes a novel approach employing social media to crowdsource knowledge from a crowd for constructing crowd knowledge databases. Practical importance of the research. The crowd knowledge database is expected to be used by the current computational tools to support designers producing highly creative ideas that are new to the crowd, in new product design, and ultimately to innovation. Research limitations/Future research. In this study to provide insights and potential directions for future research are discussed that challenges of employing described approach. Paper type – theoretical.

Social media has been used extensively for information spreading during disasters. The problem occurs nowadays is overloaded of information from social media that is spread by people and confused the other people. Current function in Twitter allows people to ‘favorite’ tweets, perform re-tweet instantly, or added their own opinion on the tweet by quote and re-tweet. It is allowed the user to report tweets, sensitive or harmful tweets, and non-interested tweet, but there are no features to identify the credibility and accuracy of those tweets. Therefore, we proposed a technical solution where a prototype was developed in a Twitter-like environment by adding crowdsource rating features. The purpose of this research is to investigate the influence of crowd-sourced rating on tweets for decision making for the credibility of information spreading. A pilot study was conducted to a small real sample group of 31 respondents to know the respondent’s feedback on the prototype. The pilot study shows that most of the respondents were agree with the capabilities of crowdsourcing rating feature in identifying the accuracy and credibility of that information. The prototype design and the questionnaires have to be modified again, but the rating features was still the same. This prototype then redistributed to 139 respondents, and the respondents need to answer the control scenario part, prototype part, and the questionnaires. Control scenario design was similar like a prototype but without adding the crowdsource rating features. The questionnaires have divided into four parts, which is demographic information, social media usage, evaluation of the prototype, and evaluation on the usability of the prototype. Overall, the result from the questionnaires answer was a positive result where most of the respondents were agreed that crowdsourced rating feature was useful and helped them to identify and determine the accuracy and credibility of the information and help to prevent from spreading the misinformation in social media.

A collaborative health care plan using a multi-agent system assists adult individuals to live an independent healthy life by analyzing their routine life activities. Robust recognition of activities provides services such as health monitoring and fitness assessment. In this paper, we propose Collaborative Health Care Plan a novel system, to improve the independent living of an individual using a smartphone sensor, machine learning algorithm, multiple agents i.e. doctor, gym trainer, guardian, and intelligent ranker agent. The novelty of the devised approach is that it shares the daily life assessment of activities among care provides which in return provides a care plan or recommendation to ensure good health of the individual. We use a machine learning algorithm to recognize the adult individual's daily life physical activities. This system provides recommendations by care providers as well as suggest an optimal care plan decided by an intelligent autonomous agent using crowdsource data.

We describe Quanti.us , a crowd-based image-annotation platform that provides an accurate alternative to computational algorithms for difficult image-analysis problems. We used Quanti.us for a variety of medium-throughput image-analysis tasks and achieved 10–50× savings in analysis time compared with that required for the same task by a single expert annotator. We show equivalent deep learning performance for Quanti.us-derived and expert-derived annotations, which should allow scalable integration with tailored machine learning algorithms. Annotated image data are required for image analysis, to test analytical methods, and to train learning algorithms. This paper describes and characterizes a tool that allows researchers to crowdsource image-annotation tasks.

The authors examine an increasingly popular open innovation practice, “design crowdsourcing,” wherein firms seek external inputs in the form of functional design solutions for new product development from the “crowd.” They investigate conditions under which managers crowdsource design and determine whether such decisions subsequently boost product sales. The empirical analysis is guided by qualitative insights gathered from executive interviews. The authors use a novel data set from a pioneering crowdsourcing firm and find that three concept design characteristics—perceived usability, reliability, and technical complexity—are associated with the decision to crowdsource design. They use an instrumental variable method accounting for the endogenous nature of crowdsourcing decisions to understand when such a decision affects downstream sales. The authors find that design crowdsourcing is positively related to unit sales and that this effect is moderated by the idea quality of the initial product concept. Using a change-score analysis of consumer ratings, they find that design crowdsourcing enhances perceived reliability and usability. They discuss the strategic implications of involving the crowd, beyond ideation, in helping transform ideas into effective products.

As the crowdsourcing strategy becomes better known, the managerial decisions necessary to establish it as a viable business process are becoming increasingly important. However, a divide and conquer approach, currently dominant in the field, leads to scattered decision support for the crowdsourcing processes. We propose an ontology-based decision tool that supports the whole business process crowdsourcing. The advantage of the ontology approach is that it collects and consolidates knowledge from the existing literature to provide a solid knowledge base for the tool construction. Operationalising the ontology, the tool helps make the decision to crowdsource or not, and choose appropriate design alternatives for the crowdsourcing process. We evaluated the tool through a controlled experiment with 190 participants. The obtained results show that the tool is useful by significantly increasing: (1) the performance in making the decision to crowdsource or not, and (2) the design of crowdsourcing processes.

SPLASSH (Student Programs Like Aquatic Science Sampling Headquarters, https://splassh.org) is a collaborative web-based application that crowdsources environmental data in real time. Originally launched in 2014, SPLASSH beta version 1.0 was designed to showcase water projects conducted by students. Through its development, it has broadened its reach from students to educators (formal and informal), researchers, resource managers, science professionals, and the public (citizen scientists). SPLASSH's beta version 2.0 (Patent Pending) has an innovative, customizable environmental tracker and project management capabilities that foster community building through collaboration. SPLASSH offers the public an opportunity to contribute more than just data to an existing project. It encourages citizens to play a lead role by initiating their own projects, truly validating and broadening the definition of citizen science. Learning and project outcomes will be measured for their impact and effectiveness.

When companies crowdsource users’ ideas, some ask their users for any ideas they may have while others present specific problems they need help in solving. This paper explores whether different approaches elicit different numbers of ideas shared by users. Two experiments in different organizations were conducted to determine which types of simple verbal stimuli are most effective for improving the users’ ideation process. It was shown that simple but specific problem statements presented by a company resulted in the highest number of relevant ideas being given by its users. Companies who look for inspiration and insight into users’ needs are advised to make specific ideation challenges so that users produce and share a larger number of ideas.

Cell-site simulators, also known as IMSI-catchers and stingrays, are used around the world by governments and criminals to track and eavesdrop on cell phones. Despite extensive public debate surrounding their use, few hard facts about them are available. For example, the richest sources of information on U.S. government cell-site simulator usage are from anonymous leaks, public records requests, and court proceedings. This lack of concrete information and the difficulty of independently obtaining such information hampers the public discussion. To address this deficiency, we build, deploy, and evaluate SeaGlass, a city-wide cellsite simulator detection network. SeaGlass consists of sensors that measure and upload data on the cellular environment to find the signatures of portable cell-site simulators. SeaGlass sensors are designed to be robust, low-maintenance, and deployable in vehicles for long durations. The data they generate is used to learn a city’s network properties to find anomalies consistent with cell-site simulators. We installed SeaGlass sensors into 15 ridesharing vehicles across two cities, collecting two months of data in each city. Using this data, we evaluate the system and show how SeaGlass can be used to detect signatures of portable cell-site simulators. Finally, we evaluate our signature detection methods and discuss anomalies discovered in the data.

Data collection in consumer research has progressively moved away from traditional samples (e.g., university undergraduates) and toward Internet samples. In the last complete volume of the Journal of Consumer Research (June 2015–April 2016), 43% of behavioral studies were conducted on the crowdsourcing website Amazon Mechanical Turk (MTurk). The option to crowdsource empirical investigations has great efficiency benefits for both individual researchers and the field, but it also poses new challenges and questions for how research should be designed, conducted, analyzed, and evaluated. We assess the evidence on the reliability of crowdsourced populations and the conditions under which crowdsourcing is a valid strategy for data collection. Based on this evidence, we propose specific guidelines for researchers to conduct high-quality research via crowdsourcing. We hope this tutorial will strengthen the community’s scrutiny on data collection practices and move the field toward better and more valid crowdsourcing of consumer research.

Growing demand for software has attracted the attention of the software development industry. Crowdsourced software development has provided a new method for the software industry to produce quality software based on an open-call format. Selecting an appropriate task to develop (developer-end) or evaluate (platform-end) is one of the primary problems in this type of open-call format. Receiving or assigning an improper task to an improper crowdsource (CS) developer does not only decrease the quality of the software deliverables, but also causes overburden on both the platform and the developers. To solve this problem, sorting the tasks based on the developers' human characteristics may increase task relevancy for developers, which can accelerate efficiency and lessen complexity. Thus, this paper has conducted an empirical experiment to measure the influence of personality on task selection based on the important characteristics of a task: money, time, and type. A total of 83 students from the University of Sindh voluntarily participated in four different short-duration rounds of task development using the developed CS platform. The personality types of the participants were measured based on the Myers-Briggs type indicator. In addition, a complex network technique called weighted degree centrality was applied to identify the most suitable personality for task sorting based on money or complexity attractions (i.e., time or type). Based on the results, it can be observed that personality has a significant relationship with task selection. For instance, developers with intuitive (N) and feeling (F) personality traits are primarily focused on the time duration of a project.

Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday activities often requires large amounts of training datasets, precisely labeled with the start and end times of the activities of interest. Acquiring annotated data is challenging and time-consuming. Applied games, such as human computation games (HCGs) have been used to annotate images, sounds, and videos to support advances in machine learning using the collective effort of “non-expert game players.” However, their potential to annotate accelerometer data has not been formally explored. In this paper, we present two proof-of-concept, web-based HCGs aimed at enabling game players to annotate accelerometer data. Using results from pilot studies with Amazon Mechanical Turk players, we discuss key challenges, opportunities, and, more generally, the potential of using applied videogames for annotating raw accelerometer data to support activity recognition research.

Machine learning to infer suicide risk and urgency is applied to a dataset of Reddit users in which the risk and urgency labels were derived from crowdsource consensus. We present the results of machine learning models based on transfer learning from document embeddings trained on large external corpora, and find that they have very high F1 scores (.83 -. 92) in distinguishing which users are labeled as being most at risk of committing suicide. We further show that the document embedding approach outperforms a method based on word importance, where important words were identified by domain experts. Finally, we find, using a Latent Dirichlet Allocation (LDA) topic model, that users labeled at-risk for suicide post about different topics to the rest of Reddit than non-suicidal users.

In recent years, the crowdsourced data analytics have gained an unprecedented interest and adoption from the research communities. In data crowdsourcing, large scientific problems can be solved by utilizing collective human intelligence, where various researchers or groups work on solving various sub-problems. However, the main challenges that encompasses a crowdsourcing system are data access, management, and analysis along with user privacy and trust preservation. In this paper, we present a decentralized data access and control framework to handle data crowdsourcing using blockchain technology. Leveraging on blockchain, our permissioned system enables confidentiality, accountability and traceability of the Crowdsourced data while maintaining control of it via Proof of Stake consensus algorithm. The proposed framework will be employed to crowdsource operational data to the National Energy Research Scientific Computing Center (NERSC) users at Lawrence Berkeley National Laboratory.

Crowdsource testing is an evolving trend in software testing, which incorporate benefits of crowd sourcing in software testing. Testers used to perform testing at various places and they are given the tasks involving different types of techniques through open call format. It means software is tested under multiple testers using different platforms, which makes it more reliable and error free. Appropriate task selection in a crowdsourced testing from a large number of tasks required different types of testing that is a complex job for the testers in crowd testing environment. In addition, platform has to involve in an arduous work to evaluate a lot of testing tasks submitted by testers involved in a crowd sourcing environment. Assigning task require testing technique that is improper for a crowdsourced tester does not only reduce the software product's quality, but also overload the platform which assign task to the testers. The core purpose of this paper is to make crowdsourced testing more efficient by finding what personality types are suitable for which type of testing techniques used in crowd sourcing environment. Moreover, this research has conducted a practical experiment to realize which personality type is more suitable for which type of testing in a crowd sourced environment. In total 92 students from Mehran university of engineering and technology has participated in this research. The Myers-Briggs type indicator was used to measure the personality types of students.

Mapathons and hackathons are short-lived events with different purposes. A mapathon is a collaborative effort for collecting geographic data in unmapped areas, while hackathons are focused on application development. Mapathon outputs need to be high quality to be reusable, but often, when applications are later built on top of map data, there is a mismatch between data collected and application requirements. We conducted an international collaboration project aiming to address this situation by creating a circular process where geographic information is collected in a mapathon and later used in a hackathon. Based on user feedback this cycle can be repeated so the collected data and developed applications can be improved. In this event report, we describe the two mapathon-hackathon cycles that were part of our pilot to validate that process. We present their outcomes and some lessons learned. We focused on the so-called "blue economy" (i.e. the sustainable use of marine and ocean resources for economic growth and improved livelihoods in coastal areas.) as the target domain for this pilot. Data for carefully selected areas of the South African coast was collected through in mapathons and later use in hackathons. The mapathons were held in South Africa, and the hackathons took place in Brazil.

In this chapter, we investigate the feasibility of large-scale haptic evaluation on online platforms such as Amazon Mechanical Turk. We introduce proxy modalities as a way to crowdsource vibrotactile sensations by reliably communicating high-level features through a crowd-accessible channel. We investigate two proxy modalities to represent a high-fidelity tactor: a new vibrotactile visualization, and low-fidelity vibratory translations playable on commodity smartphones. We translated 10 high-fidelity vibrations into both modalities, and in two user studies found that both proxy modalities can communicate affective features, and are consistent when deployed remotely over Mechanical Turk. We analyze fit of features to modalities, and suggest future improvements.