Search

Spatial crowdsourcing (SC) allows requesters to crowdsource tasks to workers based on location proximity. To preserve privacy, the location should not be disclosed to untrustworthy entities (even the SC platform). Previous solutions to preserve workers' location privacy require an online trusty third party (TTP), which is not practical in reality. In this paper, we design a framework that allows the SC platform to assign tasks to nearest workers in an online manner without knowing their actual locations. We propose an encryption algorithm to encrypt the locations of tasks and workers, and design an indexing method that assigns tasks to workers without losing too much privacy. We prove that there exists a trade-off between efficiency and security theoretically, which can be controlled based on user preference. We verify our method on real-world datasets and experimental results show that our method is efficient, effective and practical.

Heritage at risk is a terminology used to describe the sites that are highly at risk of being lost as a result of intentional demolition, deterioration, negligence or subject to improper preservation or mistreatment. Iraq is one of the countries that suffered in the last decade from intentional demolition of highly valuable heritage sites and objects. As Iraq gradually recovering from wars and violence with limited resources and budgets, historical and heritage places are still at risk because of neglect, community ignorance, insufficient planning, and military actions. Therefore, in this paper, we propose the idea of using crowdsource drone images and videos which are captured by amateurs for the documentation of heritage sites. Those crowdsource images represent a great source of data that does not require significant financial and hard labor resources. It should be noted that there is no guarantee to have the captured data being sufficient for the 3D documentation and therefore it is proposed to integrate, when possible, multiple captured crowdsource data to ensure complete documentation. In this paper, three Iraqi historical sites are 3D reconstructed using crowdsource drone videos, namely: Rabban Hormizd Monastery (AD 640), Taq Kasra (AD 242 to 272), and the Great Mosque of Samarra (AD 849–851). The experiments showed a successful 3D modeling of the three mentioned heritage objects using the crowdsource drone video images despite being captured for non-3D purposes which require high expertise and planning. With the absence of highly accurate reference data, the overall relative accuracy of the object’s dimensions is found to be less than 1 m.

The use of non-probability Internet panels and crowdsource websites is increasing in gambling research. These paid online sampling methods offer a convenient and inexpensive recruitment strategy. T...

This study uses open-source intelligence to analyse the illicit excavation and illicit trafficking of archaeological goods (and forgeries) across the Balkan-Eastern Mediterranean region(s) of Bosnia and Herzegovina, Croatia, Kosovo, Montenegro, North Macedonia, Serbia and Slovenia. It draws on texts and images that have been published by hundreds of artefact-hunters across tens of online communities and other online platforms. These include online forums; social networks, such as Facebook and Instagram; social media, such as Pinterest and YouTube; generic trading platforms, such as eBay, Etsy and olx.ba; and specialist trading platforms, such as VCoins. It shows how artefact-hunters target sites, features and objects; reveal the objects that are collectible and/or marketable; acquire equipment; form patron-client relationships, peer-to-peer partnerships and other cooperative groups; engage in transnational activity; crowdsource techniques for smuggling; crowdsource ways to avoid being caught or punished; and respond to policing. Often, they give identifying details or leave an electronic paper trail that enables their identification. Such information also reveals the destructiveness of processes of extraction and consumption; the economics of the low-end market in cultural goods from poor countries; the gender dimension in cultural property crime and cyber-enabled crime; and the interaction between political allegiance and criminal activity. Thereby, this study shows how netnography and social network analysis can support intelligence-led policing.

Smartphones have gained immense popularity among people and they incorporate several sophisticated capabilities like sensors that broaden the domain of usability. Consequently, smartphone-based crowdsourcing methods show importance as an efficient data acquisition strategy. GPS receiver and sensors like accelerometer and gyroscope can be used to gather data which helps identify the travel mode of the smartphone user. Numerous research projects have been conducted regarding the area of concern and this study uses GPS and Accelerometer data collected from smartphones using crowdsourced methods to develop a robust public transport mode detection approach using machine learning algorithms.

To help fighting the many adversarial effects that the COVID-19 pandemic brought, many initiatives supported by public, private and non-governmental organizations are crowdsourcing the development of solutions through hackathons and innovation contests. This article discusses the models and common aspects of two successful initiatives led by governmental institutions that used a crowdsourcing approach to foster solutions aiming at pandemic-related issues: the EUVsVirus (Europe) and the MPLabs COVID-19 Challenge (Brazil).

The Singapore myResponder is a novel smartphone application developed by the Singapore Civil Defence Force (SCDF) that notifies volunteer first responders of a suspected out-of-hospital cardiac arrest (OHCA) case and locations of Automated External Defibrillators (AED) in the vicinity so that they can assist with resuscitation. We aimed to examine the performance of this application, challenges encountered, and future directions.

Psoriasis is a skin condition that affects over 100 million people worldwide, while multiple myeloma (MM) accounts for 10% of all hematologic malignancies in the US. There has been limited research on the intersection of psoriasis and MM, and clinicians often face difficult decisions in treating patients diagnosed with both conditions. For instance, the management of psoriasis with systemic immunotherapies in MM patients can be challenging because of concern about immunosuppression and possible worsening of MM. Online crowdsourcing platforms have recently become innovative tools that can actively empower patients in scientific research by enabling the contribution of health data. One such platform, HealthTree®, helps MM patients find optimal myeloma treatments and has registered > 6000 patients, many of whom have uploaded medical records and genetic profiles. By taking advantage of patient health data available on HealthTree, researchers can gain a greater understanding of the clinical characteristics and treatment responses of patients diagnosed with psoriasis and MM. In this case series, we first report a psoriasis and MM patient treated with the IL-17 inhibitor ixekizumab who demonstrated a temporary, 2-month improvement in MM biomarkers (M-protein, kappa, and kappa:lambda ratio). We then report on the clinical characteristics of 18 patients with verified profiles on HealthTree indicating concurrent psoriasis and MM conditions. We surveyed gender, age, psoriasis type, psoriasis treatment history, myeloma type, myeloma genetic features, and myeloma association with bone damage, hypercalcemia, or osteopenia. Four patients were treated with systemic immunomodulators for psoriasis, with responses suggesting that these therapies did not worsen MM progression. Our results validate crowdsourcing as a way to assess patient demographics and treatment responses for use in dermatology research. We examine the demographics of patients diagnosed with psoriasis and MM and investigate the use of systemic immunomodulators for treatment of psoriasis in MM patients. Electronic supplementary material The online version of this article (10.1007/s13555-020-00416-5) contains supplementary material, which is available to authorized users.

The proliferation of smart home devices has created new opportunities for empirical research in ubiquitous computing, ranging from security and privacy to personal health. Yet, data from smart home deployments are hard to come by, and existing empirical studies of smart home devices typically involve only a small number of devices in lab settings. To contribute to data-driven smart home research, we crowdsource the largest known dataset of labeled network traffic from smart home devices from within real-world home networks. To do so, we developed and released IoT Inspector, an open-source tool that allows users to observe the traffic from smart home devices on their own home networks. Between April 10, 2019 and January 21, 2020, 5,404 users have installed IoT Inspector, allowing us to collect labeled network traffic from 54,094 smart home devices. At the time of publication, IoT Inspector is still gaining users and collecting data from more devices. We demonstrate how this data enables new research into smart homes through two case studies focused on security and privacy. First, we find that many device vendors, including Amazon and Google, use outdated TLS versions and send unencrypted traffic, sometimes to advertising and tracking services. Second, we discover that smart TVs from at least 10 vendors communicated with advertising and tracking services. Finally, we find widespread cross-border communications, sometimes unencrypted, between devices and Internet services that are located in countries with potentially poor privacy practices. To facilitate future reproducible research in smart homes, we will release the IoT Inspector data to the public.

This article presents a two-step methodology to annotate temporally anchored spatial knowledge on top of OntoNotes. We first generate potential knowledge using semantic roles or syntactic dependencies and then crowdsource annotations to validate the potential knowledge. The resulting annotations indicate how long entities are or are not located somewhere and temporally anchor this spatial information. We present an in-depth corpus analysis comparing the spatial knowledge generated by manipulating roles or dependencies. Experiments show that working with syntactic dependencies instead of semantic roles allows us to generate more potential entity-related spatial knowledge and obtain better results in a realistic scenario, that is, with predicted linguistic information.

Location-based services (LBSs) typically crowdsource geo-tagged data from mobile users. Collecting more data will generally improve the utility for LBS providers; however, it also leads to more privacy exposure of users’ mobility patterns. Although the tension between data utility and user privacy has been recognized, there lacks a solution that determines how much data to collect—in both spatial and temporal domains—is the “best” for both mobile users and the service provider. This article proposes a strategy toward making an optimal tradeoff such that a user submits data only if her mobility privacy will not be compromised and the data utility of the LBS provider will be sufficiently improved. To this end, we first define and formulate a concept called privacy exposure , which incorporates both the spatial distribution and the temporal transition of a user’s activity points . Second, we define and quantify data utility in terms of spatial repetitions and temporal closeness among data based on an economic principle. Then, we propose a PRivacy-preserving and UTility-Enhancing Crowdsourcing (PRUTEC) algorithm to determine, on behalf of each mobile user, whether a newly sensed piece of data should be submitted to the LBS provider. Our simulation demonstrates that PRUTEC improves the data utility of the service provider with a much less amount of data to collect and reduces privacy exposure for mobile users while collecting useful data continuously.

Flood is one of the most disruptive natural hazards, responsible for loss of lives and damage to properties. A number of cities are subject to monsoons influences and hence face the disaster almost every year. Early notification of flood incident could benefit the authorities and public to devise both short and long terms preventive measures, to prepare evacuation and rescue mission, and to relieve the flood victims. Geographical locations of affected areas and respective severities, for instances, are among the key determinants in most flood administration. Thus far, an effective means of anticipating flood in advance remains lacking. Existing tools were typically based on manually input and prepared data. The processes were tedious and thus prohibitive for real-time and early forecasts. Furthermore, these tools did not fully exploit more comprehensive information available in current big data platforms. Therefore, this paper proposes a novel flood forecasting system based on fusing meteorological, hydrological, geospatial, and crowdsource big data in an adaptive machine learning framework. Data intelligence was driven by state-of-the-art learning strategies. Subjective and objective evaluations indicated that the developed system was able to forecast flood incidents, happening in specific areas and time frames. It was also later revealed by benchmarking experiments that the system configured with an MLP ANN gave the most effective prediction, with correct percentage, Kappa, MAE and RMSE of 97.93, 0.89, 0.01 and 0.10, respectively.

A pilot programme run by the United States Patent and Trademark Office sought to crowdsource prior art search to help patent examiners examine patent applications in emerging technology areas. By c...

A fundamental problem in rare-disease diagnostics is the lack of consensus as to which genes have sufficient evidence to attribute causation. To address this issue, we have created PanelApp ( https://panelapp.genomicsengland.co.uk ), a publicly available knowledge base of curated virtual gene panels.

Researchers have increasingly turned to Amazon Mechanical Turk (AMT) to crowdsource speech data, predominantly in English. Although AMT and similar platforms are well positioned to enhance the state of the art in L2 research, it is unclear if crowdsourced L2 speech ratings are reliable, particularly in languages other than English. The present study describes the development and deployment of an AMT task to crowdsource comprehensibility, fluency, and accentedness ratings for L2 Spanish speech samples. Fifty-four AMT workers who were native Spanish speakers from 11 countries participated in the ratings. Intraclass correlation coefficients were used to estimate group-level interrater reliability, and Rasch analyses were undertaken to examine individual differences in rater severity and fit. Excellent reliability was observed for the comprehensibility and fluency ratings, but indices were slightly lower for accentedness, leading to recommendations to improve the task for future data collection.

Reports on the IEEE Empower a Billion program, a global competition to crowdsource the development of scalable solutions to energy access, which can rapidly improve the lifestyle and livelihood of the 1.1 billion people who live off grid and the 3 billion people who live with severe energy poverty.

Volunteered geographical information (VGI) refers to geographical information that the general public voluntarily collects and shares in the environment instead of for-profit businesses or government entities. Crowdsourcing such information on urgent needs in a disaster can improve the quick emergency responses. This study incorporates the capability of smartphone sensors, GPS, Web 2.0, VGI, and server-based technologies to design and develop a system for collecting target hazard information from volunteers. One of the most important contributions in designing this system is considering the improvement of the positional accuracy of the target buildings based on the position of the mobile device. Several approaches have been recommended for this purpose. The solutions include the use of online map services, geocoding services, and trigonometric methods based on the measurements of sensors such as camera, accelerometer, and magnetic field embedded in a smart mobile phone. The accuracy assessment showed that the trigonometric method by the means of embedded sensors would yield the best result. However, geocoding is more economical in terms of time than other methods. Potentially, the evaluation of the mobile application provided by a group of volunteers showed the overwhelming preference of crowdsource mapping over current telephone communication systems in disaster management.

Mobile ads are plagued with fraudulent clicks which is a major challenge for the advertising community. Although popular ad networks use many techniques to detect click fraud, they do not protect the client from possible collusion between publishers and ad networks. In addition, ad networks are not able to monitor the user’s activity for click fraud detection once they are redirected to the advertising site after clicking the ad. We propose a new crowdsource-based system called Click Fraud Crowdsourcing (CFC) that collaborates with both advertisers and ad networks in order to protect both parties from any possible click fraudulent acts. The system benefits from both a global view, where it gathers multiple ad requests corresponding to different ad network-publisher-advertiser combinations, and a local view, where it is able to track the users’ engagement in each advertising website. The results demonstrated that our approach offers a lower false positive rate (0.1) when detecting click fraud as opposed to proposed solutions in the literature, while maintaining a high true positive rate (0.9). Furthermore, we propose a new mobile ad charging model that benefits from our system to charge advertisers based on the duration spent in the advertiser’s website.

Objectives: Crowdsourcing works through an institution outsourcing a function normally performed by an employee or group of individuals. Within a crowdsource users, known as the crowd, form a community who voluntarily undertake a task which involves the pooling of knowledge resources. A literature review was undertaken to identify how the tool is being used in health professions education, and potential for use in radiography education. Key findings: 17 papers were returned. Literature identified was assessed against an established crowdsourcing definition. Reviewing these yielded four themes for discussion: student selection procedures, lesson planning, teaching materials and assessment. Conclusion: Crowdsourcing is associated with innovative activities through collective solution seeking via a large network of users. It is increasingly being adopted in healthcare training and maybe transferable to educational activities within the field of radiography education.

Conspiracies about vaccination are prevalent. We assessed how the health information sources people rely upon and their political ideologies are associated with acceptance of vaccine conspiracies.Online survey (N = 599) on Amazon's Mechanical Turk crowdsource platform. Hypotheses were tested via structural equation modeling.Acceptance of vaccine conspiracy beliefs was associated positively with greater reliance on social media for health information (coef. = 0.42, p .001), inversely related to use of medical websites (coef. = -0.21, p .001), and not significantly related to use of providers for health information (coef. = -0.13, p = .061). In addition, liberal political orientation was negatively associated with acceptance of vaccine conspiracies (coef. = -0.29, p .001).An understanding of vaccine conspiracy acceptance requires a consideration of people's health information sources. The greater susceptibility of political conservatives to conspiracy beliefs extends to the topic of vaccination.

When organizations crowdsource ideas, they select only a small share of the ideas that contributors submit for implementation. If a contributor submits an idea to an organization for the first time (i.e., newcomer), and the organization does not select the idea, this may negatively affect the newcomer’s relationship with the organization and willingness to submit ideas to the organization in the future. We suggest that organizations can increase newcomers’ willingness to submit further ideas by providing a thus far understudied form of feedback: rejections. Though counterintuitive, we suggest that rejections encourage newcomers to bond with an organization. Rejections signal contributors that an organization is interested in both receiving their ideas and developing relationships with them. To test our theory, we examine the crowdsourcing of 70,159 organizations that receive ideas from 1,336,154 contributors. Using text analysis, we examine differences in how rejections are written to disentangle the mech...

This paper considers an alternative way of gathering and evaluating information on corporate social responsibility (CSR) and the actions taken towards sustainable development. Social accounting the...

Advancements in smartphone technologies and the use of specialized health care applications offer an exciting new era to promote melanoma awareness to the public and improve education and prevention strategies. These applications also afford an opportunity to power meaningful research aimed at improving image diagnostics and early melanoma detection. Here, we summarize our experience associated with developing and managing the implementation of MoleMapper™, a research-based application that not only provides an efficient way for users to digitally track images of moles and facilitate skin self-examinations but also provides a platform to crowdsource research participants and the curation of mole images in efforts to advance melanoma research. Obtaining electronic consent, safeguarding participant data, and employing a framework to ensure collection of meaningful data represent a few of the inherent difficulties associated with orchestrating such a wide-scale research enterprise. In this review, we discuss strategies to overcome these and other challenges leading to the implementation of MoleMapper™.

Firms increasingly use innovation tournaments to crowdsource innovation ideas from customers. This article uncovers antecedents and consequences of customers’ participation intensity over the course of a tournament. More specifically, the authors theorize on the effects that the type and timing of moderating feedback have on tournament participants’ participation intensity, as well as the effect of the latter on idea quality. Through two longitudinal experiments using a commercial innovation tournament platform, the authors show that moderating feedback stimulates ideators’ participation intensity. They find that negative feedback increases participation intensity, as compared to no feedback and positive feedback. Moreover, negative feedback, either provided in isolation or together with positive feedback, is more effective during the early stages than in the later stages of a tournament. Using a large-scale managerial survey, the authors show that higher participation intensity leads to higher idea quality and better business performance. The effect of participation intensity on idea quality is stronger than the effect of number of ideas and as strong as the effect of number of participants on idea quality.

This study aims to (i) identify patterns of sociophonetic variation in Taiwan Mandarin, and (ii) evaluate smartphone technologies as a tool for crowdsourcing sociophonetic data. Specifically, this study examines both phonological variables found in prior literature to be highly salient (deretroflexion, labiovelar glide deletion), and variables that are less likely to index social properties (merging of final /n, ŋ/, changes to Tones 2 and 3). Unlike past studies which have primarily relied on smaller sample sizes, I utilize a smartphone application to crowdsource audio recordings across Taiwan; subsequent Rbrul analysis of 292 recordings revealed robust patterns of sociolinguistic variation. Deretroflexion correlates strongly with gender and age, while glide deletion correlates with gender. Nasal final merging and tonal change exhibit less socio-indexical variation, but provide evidence of potential change in progress. These findings suggest that smartphone-based crowdsourcing can complement traditional sociolinguistic fieldwork, and reveal new knowledge about large-scale variation.

Crowdsourcing has become a low-cost and scalable alternative to gather relevance assessments through crowdsourcing platforms. However, when gathering subjective human judgments, it can be challenging to enforce data quality due to the lack of vision about how judges make a decision. It is important to determine the attributes that could affect the effectiveness of crowsourced-judgments in an information retrieval systems evaluation. The purpose of the experiment that is discussed in this paper is to investigate if logical reasoning ability of the crowd workers is related to the quality of the relevant judgments produced through the crowdsource process. The study also evaluates the effect of cognitive characteristics on the quality of relevance judgment compared to the gold standard dataset. Through this experiment, a comparison study is done between the quality of the judgments obtained through the crowdsourcing process and the original baseline judgments generated by the hired experts by TREC. In the study, the systems performances were measured using both of these sets of relevance judgments to see its correlation. The experimentation reveals that quality of relevance judgments is highly correlated with the logical reasoning ability of individuals. The judgment difficulty level reported by the crowdsource workers and the confidence level claimed by the workers showed a significant correlation with the quality of the judgments. Unexpectedly though, self-reported knowledge about a given topic and demographics data have no correlation with the quality of judgments produced through crowdsourcing

Video summarization is one of the key techniques to access and manage a huge chunk of videos. Video summarization is used to extract the effective contents of a video sequence to generate a concise representation of its content. The involvement of crowdsourcing models in recent years is effectively used by researchers in E-commerce domain to increase the quality of the data contents. In this paper, we present a video summarization framework based on users emotion while they watch videos by analyzing cerebral activities through Electroencephalogram (EEG) signals. Three emotions, namely happy, sad and neutral have been extracted from the EEG signals. Video frames have been synchronized with EEG signals and tagged with various emotions. Finally, a crowdsourcing model has been used for effective summarization of the videos. The qualitative assessment of video summarization has been conducted with the help of user ratings using online Google Forms application. EEG signals of 28 users have been recorded while video streams of different emotions. An average accuracy of 83.93% has been recorded in emotion classification using crowdsourcing. Output summarized videos include dynamic video skims and the corresponding audio stream for better understanding.

The email arrived midmorning on a Friday. “Thank you for your participation in our crowd reviewing (CR) program!” it read. Cheery and succinct, the message notified more than 80 expert reviewers that a manuscript was waiting for them, ready to be critiqued and considered for publication in the journal Synlett. A link within the email led the potential reviewers to a website called Filestage. There, they could click on any portion of the manuscript to pop up a box to compose comments. Half an hour after the message hit reviewers’ in-boxes, pale green circles dotted the screen where eager readers left suggestions and edits. Over the next four days, more than a dozen of the original 80 invitees anonymously debated the new chemical synthesis presented in the manuscript. They commented on the reaction’s novelty, questioned the molecular mechanisms proposed by the authors, suggested control experiments, pointed out spectra of samples

Various technologies have been utilized today for recognizing client or user in the indoor areas. These technologies incorporate RSSI, Bluetooth Low Energy Beacons, Ultrasound waves, Vision-based advances, for example, fixed camera recordings QR codes, remote gadgets, etc. RSSI fingerprinting technique requires more effort and it is also expensive to be used for indoor localization frameworks working in real-time. In this research, indoor localization based on images is investigated as an option in contrast to other indoor positioning techniques using these days. Image-based indoor positioning is more affordable than RSSI based technologies being utilized. A mobile phone camera is utilized to take the pictures of area inside the building to find the user inside the building. Sensor data from various sensors isn’t required or no extra framework is required to find the client in the building utilizing indoor positioning based on an image. Microsoft Azure Custom Vision Services are utilized to locate the client; MS Azure classifies the pictures in one of the labels made. Strategy’s attainability is demonstrated by various investigations and accomplished accuracy and review is recorded above 90%. The average precision of the trained model is recorded above 95%.

Many mountain lions are killed in the state of California every year from roadkill. To reduce these numbers, it is important that a system be built to track where these mountain lions have been around. One such system could be built using the platform-as-a-service, Firebase. Firebase is a platform service that collects and manages data that comes in through a mobile application. For the development of cross-platform mobile applications, Flutter is used as a toolkit for developers for both iOS and Android. This entire system, Cat Tracks is proposed as a crowdsource platform to track wildlife, with the current focus on California mountain lions. By building such a system, researchers could use the data to save the lives of many mountain lions.

We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.

Early warning of a potential pandemic with res- piratory symptoms is crucial for global health management. It enables timely intervention to reduce the likelihood of uncon- trollable massive virus spread. In this research, we propose to leverage the ubiquitous wearable devices to develop a wearable crowdsource system to monitor respiratory symptoms such as cough and fever. Wearable devices nowadays can directly and non-intrusively measure people’s vital signs in real-time with a variety of sensors embedded. We collect the data from wearable devices and develop machine learning algorithms to analyze the data for respiratory symptom monitoring and early warning. In particular, we focus on cough detection through multi-source data fusion (e.g., accelerometer amplitude and microphone audio). Preliminary results show that our algorithms result in higher detection accuracy and less false positive with the least use of computing resources. This research potentially transforms the way the pandemic early warning is implemented and the way we respond to public health crises in the years to come. Nanyang Technological University National Research Foundation (NRF) Accepted version This research is funded by National Research Foundation (NRF) via the Green Buildings Innovation Cluster (Grant NO.: NRF2015ENC_GBICRD001-012), administered by Building and Construction Authority (BCA) Singapore. In addition, this research is sponsored by National Research Foundation (NRF) via the Behavioural Studies in Energy, Water, Waste and Transportation Sectors (Grant NO.: BSEWWT2017_2_06), administered by National University of Singapore (NUS). Moreover, this research is funded by Nanyang Technological University (NTU) via the Data Science & Artificial Intelligence Research Centre @ NTU (Grant NO.: DSAIR@NTU).

Data visualizations have been increasingly used in oral presentations to communicate data patterns to the general public. Clear verbal introductions of visualizations to explain how to interpret the visually encoded information are essential to convey the takeaways and avoid misunderstandings. We contribute a series of studies to investigate how to effectively introduce visualizations to the audience with varying degrees of visualization literacy. We begin with understanding how people are introducing visualizations. We crowdsource 110 introductions of visualizations and categorize them based on their content and structures. From these crowdsourced introductions, we identify different introduction strategies and generate a set of introductions for evaluation. We conduct experiments to systematically compare the effectiveness of different introduction strategies across four visualizations with 1,080 participants. We find that introductions explaining visual encodings with concrete examples are the most effective. Our study provides both qualitative and quantitative insights into how to construct effective verbal introductions of visualizations in presentations, inspiring further research in data storytelling., Comment: 12 pages, 5 figures, accepted to IEEE Transaction on Visualization and Graphics

Background The traditional model of promotion and tenure in the health professions relies heavily on formal scholarship through teaching, research, and service. Institutions consider how much weight to give activities in each of these areas and determine a threshold for advancement. With the emergence of social media, scholars can engage wider audiences in creative ways and have a broader impact. Conventional metrics like the h-index do not account for social media impact. Social media engagement is poorly represented in most curricula vitae (CV) and therefore is undervalued in promotion and tenure reviews. Objective The objective was to develop crowdsourced guidelines for documenting social media scholarship. These guidelines aimed to provide a structure for documenting a scholar’s general impact on social media, as well as methods of documenting individual social media contributions exemplifying innovation, education, mentorship, advocacy, and dissemination. Methods To create unifying guidelines, we created a crowdsourced process that capitalized on the strengths of social media and generated a case example of successful use of the medium for academic collaboration. The primary author created a draft of the guidelines and then sought input from users on Twitter via a publicly accessible Google Document. There was no limitation on who could provide input and the work was done in a democratic, collaborative fashion. Contributors edited the draft over a period of 1 week (September 12-18, 2020). The primary and secondary authors then revised the draft to make it more concise. The guidelines and manuscript were then distributed to the contributors for edits and adopted by the group. All contributors were given the opportunity to serve as coauthors on the publication and were told upfront that authorship would depend on whether they were able to document the ways in which they met the 4 International Committee of Medical Journal Editors authorship criteria. Results We developed 2 sets of guidelines: Guidelines for Listing All Social Media Scholarship Under Public Scholarship (in Research/Scholarship Section of CV) and Guidelines for Listing Social Media Scholarship Under Research, Teaching, and Service Sections of CV. Institutions can choose which set fits their existing CV format. Conclusions With more uniformity, scholars can better represent the full scope and impact of their work. These guidelines are not intended to dictate how individual institutions should weigh social media contributions within promotion and tenure cases. Instead, by providing an initial set of guidelines, we hope to provide scholars and their institutions with a common format and language to document social media scholarship.

International audience; This paper investigates a general framework for synchronous educational language games that simultaneously allows researchers to crowdsource learner answers in a controlled environment. Our prototype Substituto allows teachers and students to interact in real-time while undergoing language learning exercises; ensuring that the learner’s progress is not hurt by the introduction of crowdsourcing elements. We evaluate Substituto with a small-scale user study that focuses on training the use of English verb-particle constructions (VPCs), such as break down or take over, and test their use with second language learners of English of different proficiency levels over five pilot sessions. With the study we aim to ensure that our prototypical implementation behaves as expected and to identify any major design flaws that should be addressed. The preliminary results we achieved in order to evaluate the educational value, the user experience and the crowdsourcing capacity of XYZ-Bot confirm that it has the potential to become a valuable asset for language learning, a pleasant learning instrument and a crowdsourcing tool for collecting linguistic knowledge.

Most of the automated approaches to support software engineering tasks for mobile applications, use source code as entry, which due to privacy factors imposes hard constraints on the implementation of those approaches by third-party services. However, the continuous pressure from market is leading practitioners to crowdsource/outsource software engineering tasks to third-parties that provide on-the-cloud infrastructures. Therefore, approaches that work at closed-source level (i.e., do not require source code) are desirable to enable automated outsourced software engineering tasks by third-party services. Specifically, in this paper we focus on testing since is a task that has broadly accepted approaches at source-code level, but new efforts are required to enable source-codeless approaches. We analyzed the current challenges and shortfalls of the currently proposed approaches that works at source code and APK level, and present our research plan oriented to reduce the existing gap for enabling testing techniques at APK level.

The paper presents a cross-European survey on teachers and crowdsourcing. The survey examines how familiar language teachers are with the concept of crowdsourcing and addresses their attitude towards including crowdsourcing into language teaching activities. The survey was administrated via an online questionnaire and collected volunteers’ data on: (a) teachers’ experience with organizing crowdsourcing activities for students/pupils, (b) the development of crowdsourced resources and materials as well as (c) teachers’ motivation for participating in or employing crowdsourcing activities. The questionnaire was disseminated in over 30 European countries. The final sample comprises 1129 language teachers aged 20 to 65, mostly working at institutions of tertiary education. The data indicates that many participants are not familiar with the concept of crowdsourcing resulting in a low rate of crowdsourcing activities in the classroom. However, a high percentage of responding teachers is potentially willing to crowdsource teaching materials for the language(s) they teach. They are particularly willing to collaborate with other teachers in the creation of interactive digital learning materials, and to select, edit, and share language examples for exercises or tests. Since the inclusion of crowdsourcing activities in language teaching is still in its initial stage, steps for further research are highlighted. U radu je predstavljeno europsko istraživanje o nastavnicima i gomilizaciji (engl. crowdsourcing). U istraživanju je ispitano koliko su nastavnici jezika upoznati s konceptom gomilizacije te su ispitani njihovi stavovi o uključivanju gomilizacije u aktivnosti poučavanja jezika. Istraživanje je provedeno s pomoću mrežnog upitnika, a prikupljani su podatci o: (a) iskustvu nastavnika u organizaciji gomilizacijskih aktivnosti s učenicima/studentima, (b) razvoju gomilizacijskih resursa i materijala, kao i (c) motivaciji nastavnika za sudjelovanjem u gomilizaciji i upotrebi njezinih aktivnosti. Upitnik je distribuiran u više od 30 europskih zemalja. Uzorak obuhvaća 1129 nastavnika jezika u dobi od 20 do 65 godina, koji rade uglavnom u institucijama tercijarnog obrazovanja. Podatci upućuju na to da ispitanici nisu upoznati s konceptom gomilizacije, zbog čega je niska stopa gomilizacijskih aktivnosti u učionici. Međutim, velik je postotak nastavnika koji su voljni sudjelovati u gomilizaciji kako bi se razvili obrazovni sadržaji za jezike koje poučavaju. Posebno su voljni surađivati s drugim nastavnicima u stvaranju interaktivnih digitalnih obrazovnih sadržaja te odabirati, uređivati i dijeliti jezične primjere za vježbe i testove. Budući da je uključivanje gomilizacijskih aktivnosti u poučavanje jezika još u povojima, istaknuti su koraci za daljnja istraživanja.

The SARS-CoV-2 pandemic has rapidly saturated healthcare resources across the globe and has led to a restricted screening process, hindering efforts at comprehensive case detection. This has not only facilitated community spread but has also resulted in an underestimation of the true incidence of disease, a statistic which is useful for policy making aimed at controlling the current pandemic and in preparing for future outbreaks. In this perspective, we present a crowdsourced platform developed by us for the true estimation of all SARS-CoV-2 infections in the community, through active self-reporting and layering other authentic datasets. The granularity of data captured by this system could prove to be useful in assisting governments to identify SARS-CoV-2 hotspots in the community facilitating lifting of restrictions in a controlled fashion.

Growing attention on the benefits of non-motorized travel has increased the demand for accurate and timely pedestrian and bicycle travel data. Advancements in technologies and the proliferation of smartphones have created new data sources that can help eliminate limitations related to small sample size and infrequent updates due to limited resources. This study reviews the emerging data sources and their current use, focusing on non-motorized travel monitoring. In this study, the emerging data are categorized into mode-unspecified and mode-specified data based on whether the mode used can be detected with no or little effort. While mode-unspecified data are collected without sorting out non-motorized travelers, mode-specified data at least know who (which mode) is being monitored. So far, commercial vendors provide a vast volume of mode-unspecified data, but their products have been mainly used for motorized trips or are in initial stages of development. Meanwhile, readily available data sources and their applications are more concentrated on mode-specified data, which have enabled varying non-motorized travel studies—including travel pattern identification, route-choice modeling, crash/air pollution exposure estimation, and new facility provision evaluation—but are mostly focused on bicycling. Despite the potential of emerging data, their use also has several challenges, such as limited mode inference, sample bias, and lack of detailed trip/traveler information due to privacy issues. More efforts are needed, such as improving data accuracy and developing robust data fusion techniques, to be able to fully utilize the emerging data sources.

Research involving Virtual Reality (VR) headsets is becoming more and more popular. However, scaling VR experiments is challenging as researchers are often limited to using one or a small number of headsets for in-lab studies. One general way to scale experiments is through crowdsourcing so as to have access to a large pool of diverse participants with relatively little expense of time and money. Unfortunately, there is no easy way to crowdsource VR experiments. We demonstrate that it is possible to implement and run crowdsourced VR experiments using a pre-existing massively multiplayer online VR social platform---VRChat. Our small (n=10) demonstration experiment required participants to navigate a maze in VR. Participants searched for two targets then returned to the exit while we captured completion time and position over time. While there are some limitations with using VRChat, overall we have demonstrated a promising approach for running crowdsourced VR experiments.

Introduction Stakeholder participation in healthcare curriculum design is an important aspect of higher education with stakeholders including students, staff members, clinical partners, healthcare organisations, patients and members of the public. Significantly, student co-creation, of the curriculum, has become increasingly important. Yet there is limited research which addresses how to engage this group in design processes. Methods This paper represents the first phase of a three stage action research spiral whereby the authors evaluated the use of a novel tool for curriculum design processes, anonymised crowdsourcing. This initial phase was open to all students enrolled on an undergraduate diagnostic radiography programme in the UK. To confirm the reliability of the crowdsource design an established eight point crowdsourcing verification tool was applied. Results Twenty-three unique ideas were generated by participants, 40 comments made and 173 votes cast. Inductive analysis of the comments generated five themes. These included: the role of technology enhanced learning; simulation activities; patient focused curriculum; mental wealth (resilience) authentic assessment approaches. An evaluation of those who had and had not engaged highlighted areas of improvement for the administration of the second and third iterations which will include a wider pool of participants. Conclusion This study from a single programme offers lessons for others wishing to adopt and develop this approach elsewhere. Implications for practice Several ideas elicited by the crowdsource have been considered by the curriculum design team and will be implemented in the 2020 curriculum thus demonstrating the impact on local education practice of this research approach.

Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.

The Waymo Open Dataset has been released recently, providing a platform to crowdsource some fundamental challenges for automated vehicles (AVs), such as 3D detection and tracking. While the dataset provides a large amount of high-quality and multi-source driving information, people in academia are more interested in the underlying driving policy programmed in Waymo self-driving cars, which is inaccessible due to AV manufacturers&rsquo, proprietary protection. Accordingly, academic researchers have to make various assumptions to implement AV components in their models or simulations, which may not represent the realistic interactions in real-world traffic. Thus, this paper introduces an approach to learn a long short-term memory (LSTM)-based model for imitating the behavior of Waymo&rsquo, s self-driving model. The proposed model has been evaluated based on Mean Absolute Error (MAE). The experimental results show that our model outperforms several baseline models in driving action prediction. In addition, a visualization tool is presented for verifying the performance of the model.

We study initial coin offerings (ICOs) to understand how an unregulated market overcomes information frictions and conflicts of interest. Listing platforms both independently assess an offering and crowdsource information from "expert" reviewers. These experts provide more balanced textual reviews as they gain experience and receive positive feedback from the community, consistent with a reputation effect. We find that proceeds are higher when reviews are more positive even after controlling for both the reviewer's and platform's numerical rating. Finally, experts with greater potential conflicts of interest are more positive than other reviewers, but investors identify these conflicts and discount their reviews.

The research proposes the neural network methods to include a textual dependency tree structure in classification tasks of Russian texts. Author profiling task of gender identification was chosen to test the models, and two corpora used in experiments: based on a crowdsource, and in-person polling. The first approach is based on a long short-term memory (LSTM) layers, and developed graph embedding algorithm. The second one is based on a graph convolution network and LSTM. Two syntactic parsers were used to obtain dependency trees from the texts. Input data was represented in different forms: morphological binary vectors, FastText vectors, and their combination. The developed models result was compared to the state-of-the-art, that is neural network model based on a convolutional and LSTM layers. Finally, we demonstrate that including textual dependency tree structure to input feature space improves f1-score of gender classification task on 4% for the RusPersonality dataset, and 7% for the crowdsource dataset in average. The developed models resulting f1-score is 84% and 83%, respectively.

Purpose – to explore a crowdsourcing data-driven approach to construct crowdknowledge databases for innovation through supporting creative idea generation. In the approach, social media will be used as platforms to crowdsource knowledge for producing the databases. Findings. Creativity is an essential element of innovation, but producing creative ideas is often challenging in design. Many computational tools have become available recently to support designers in producing creative ideas that are new to individuals. As a standard feature, most of the tools rely on the databases employed, such as ConceptNet and the US Patent Database. This study highlighted that the limitations of these databases have constrained the capabilities of the tools and, thereby, new computational databases supporting the generation of new ideas to a crowd or even history are needed. Crowdsourcing outsources tasks conventionally performed in-house to a crowd and uses external knowledge to solve problems and democratize innovation. Social media is often employed in crowdsourcing for a crowd to create and share knowledge. Originality/value/scientific novelty of the research. This paper proposes a novel approach employing social media to crowdsource knowledge from a crowd for constructing crowd knowledge databases. Practical importance of the research. The crowd knowledge database is expected to be used by the current computational tools to support designers producing highly creative ideas that are new to the crowd, in new product design, and ultimately to innovation. Research limitations/Future research. In this study to provide insights and potential directions for future research are discussed that challenges of employing described approach. Paper type – theoretical.

Social media has been used extensively for information spreading during disasters. The problem occurs nowadays is overloaded of information from social media that is spread by people and confused the other people. Current function in Twitter allows people to ‘favorite’ tweets, perform re-tweet instantly, or added their own opinion on the tweet by quote and re-tweet. It is allowed the user to report tweets, sensitive or harmful tweets, and non-interested tweet, but there are no features to identify the credibility and accuracy of those tweets. Therefore, we proposed a technical solution where a prototype was developed in a Twitter-like environment by adding crowdsource rating features. The purpose of this research is to investigate the influence of crowd-sourced rating on tweets for decision making for the credibility of information spreading. A pilot study was conducted to a small real sample group of 31 respondents to know the respondent’s feedback on the prototype. The pilot study shows that most of the respondents were agree with the capabilities of crowdsourcing rating feature in identifying the accuracy and credibility of that information. The prototype design and the questionnaires have to be modified again, but the rating features was still the same. This prototype then redistributed to 139 respondents, and the respondents need to answer the control scenario part, prototype part, and the questionnaires. Control scenario design was similar like a prototype but without adding the crowdsource rating features. The questionnaires have divided into four parts, which is demographic information, social media usage, evaluation of the prototype, and evaluation on the usability of the prototype. Overall, the result from the questionnaires answer was a positive result where most of the respondents were agreed that crowdsourced rating feature was useful and helped them to identify and determine the accuracy and credibility of the information and help to prevent from spreading the misinformation in social media.