Search

As more educators integrate their curricula with online learning, it is easier to crowdsource content from them. Crowdsourced tutoring has been proven to reliably increase students' next problem correctness. In this work, we confirmed the findings of a previous study in this area, with stronger confidence margins than previously, and revealed that only a portion of crowdsourced content creators had a reliable benefit to students. Furthermore, this work provides a method to rank content creators relative to each other, which was used to determine which content creators were most effective overall, and which content creators were most effective for specific groups of students. When exploring data from TeacherASSIST, a feature within the ASSISTments learning platform that crowdsources tutoring from teachers, we found that while overall this program provides a benefit to students, some teachers created more effective content than others. Despite this finding, we did not find evidence that the effectiveness of content reliably varied by student knowledge-level, suggesting that the content is unlikely suitable for personalizing instruction based on student knowledge alone. These findings are promising for the future of crowdsourced tutoring as they help provide a foundation for assessing the quality of crowdsourced content and investigating content for opportunities to personalize students' education.

Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday activities often requires large amounts of training datasets, precisely labeled with the start and end times of the activities of interest. Acquiring annotated data is challenging and time-consuming. Applied games, such as human computation games (HCGs) have been used to annotate images, sounds, and videos to support advances in machine learning using the collective effort of “non-expert game players.” However, their potential to annotate accelerometer data has not been formally explored. In this paper, we present two proof-of-concept, web-based HCGs aimed at enabling game players to annotate accelerometer data. Using results from pilot studies with Amazon Mechanical Turk players, we discuss key challenges, opportunities, and, more generally, the potential of using applied videogames for annotating raw accelerometer data to support activity recognition research.

Crowdsource testing is an evolving trend in software testing, which incorporate benefits of crowd sourcing in software testing. Testers used to perform testing at various places and they are given the tasks involving different types of techniques through open call format. It means software is tested under multiple testers using different platforms, which makes it more reliable and error free. Appropriate task selection in a crowdsourced testing from a large number of tasks required different types of testing that is a complex job for the testers in crowd testing environment. In addition, platform has to involve in an arduous work to evaluate a lot of testing tasks submitted by testers involved in a crowd sourcing environment. Assigning task require testing technique that is improper for a crowdsourced tester does not only reduce the software product's quality, but also overload the platform which assign task to the testers. The core purpose of this paper is to make crowdsourced testing more efficient by finding what personality types are suitable for which type of testing techniques used in crowd sourcing environment. Moreover, this research has conducted a practical experiment to realize which personality type is more suitable for which type of testing in a crowd sourced environment. In total 92 students from Mehran university of engineering and technology has participated in this research. The Myers-Briggs type indicator was used to measure the personality types of students.

Mapathons and hackathons are short-lived events with different purposes. A mapathon is a collaborative effort for collecting geographic data in unmapped areas, while hackathons are focused on application development. Mapathon outputs need to be high quality to be reusable, but often, when applications are later built on top of map data, there is a mismatch between data collected and application requirements. We conducted an international collaboration project aiming to address this situation by creating a circular process where geographic information is collected in a mapathon and later used in a hackathon. Based on user feedback this cycle can be repeated so the collected data and developed applications can be improved. In this event report, we describe the two mapathon-hackathon cycles that were part of our pilot to validate that process. We present their outcomes and some lessons learned. We focused on the so-called "blue economy" (i.e. the sustainable use of marine and ocean resources for economic growth and improved livelihoods in coastal areas.) as the target domain for this pilot. Data for carefully selected areas of the South African coast was collected through in mapathons and later use in hackathons. The mapathons were held in South Africa, and the hackathons took place in Brazil.

Whenever a car owner drives to a destination, the first thing to consider is parking. Due to the lack of parking facilities, car owners are forced to park along the secondary or sometimes main roads to go about their business. With smart phones increasingly being integrated into our lives, it is not a new idea to utilize phones as a tool to help in solving societal problems such as the lack of available parking space in the metro. The project aims to solve this problem by developing a mobile application to create collaborative online community marketplace with sharing economy concept by bringing together two crowds: people who are looking for parking slots and business-minded persons who wants to make profit out of his/her unutilized parking lot/space. Alongside, ParkPal also offers the ability to monitor the density of public commercial spaces through the implicit collection of data contributed by ParkPal users generated from Activity Recognition API and Geofencing API.

Recent research has been devoted to designing mobile applications that encourage users to complete microtasks in everyday context, known as "mobile crowdsourcing". In this case study, we present our ongoing effort of a publicly-available mobile application, Crowdsource, that has over 540,000 global users from 200 countries or regions. Over 15 million sessions have been performed since the initial launch in August 2016. To better support users' motivations and goals, we conducted a survey with our active users and validate their feedback with a set of usability studies. Our findings suggest design considerations for crowdsourcing microtasks with mobile users at the global scale.

We present C-Saw, a system that measures Internet censorship by offering data-driven censorship circumvention to users. The adaptive circumvention capability of C-Saw incentivizes users to opt-in by offering small page load times (PLTs). As users crowdsource, the measurement data gets richer, offering greater insights into censorship mechanisms over a wider region, and in turn leading to even better circumvention capabilities. C-Saw incorporates user consent in its design by measuring only those URLs that a user actually visits. Using a cross-platform implementation of C-Saw, we show that it is effective at collecting and disseminating censorship measurements, selecting circumvention approaches, and optimizing user experience. C-Saw improves the average PLT by up to 48% and 63% over Lantern and Tor, respectively. We demonstrate the feasibility of a large-scale deployment of C-Saw with a pilot study.

This 1 paper presents, 'Interact', an interactive backchannel system providing an array of features to both the students and instructors. It allows the students to post comments, ask questions, and rate their understanding of various concepts (pre-populated by the instructor), taught in a lecture session. Additionally, the system employs text analytics using Latent Dirichlet allocation (LDA), to extract topics from the comments, and aggregate sentiments for each topic using AFINN-165 wordlist. The instructors can view: (1) top-ranked 'm' topics discussed in 'n' lectures; (2) sentiment trends for 'm' topics, and 'n' lectures, including positive and negative words; (3) popular questions asked in 'n' lectures. A user-study was conducted with 40 students and 10 academics were interviewed to understand their perception towards the system. Additionally, a crowdsource study examined the effectiveness of aggregation techniques. We believe that Interact will help to improve teaching and develop effective learning, with the aid of automatic processing, summarization and visualization of student feedback obtained through comments, questions and rating concepts.

Relevance is the strength of the relationship between a user's perceived information need and an information object. Systems designed to help users identify relevant information can often rely on high quality labelled datasets. However, the subjective and personal nature of relevance means that establishing ground truth labels is difficult. In this work, we conduct a user study on text documents to crowdsource relevance assessments against four topics. Workers' self-reported measures and task completion speed are used to calculate a range of ground truth measures against which classification performance can be assessed. Our results indicate that average subjective relevance and confidence-weighted measures are on par with the annotations from an expert panel. Further work is planned to expand these findings.

Mobile app reviews are important as a crowdsource to improve the quality of these softwares. App stores, which have app reviews, provide a wealth of information derived from users. These information help developers to fix bugs and implement new features desired by users. Despite the reviews usefulness, one of the challenges of application developers is the huge number of reviews published daily, making manual analysis laborious. Hence, the delay in satisfying users may influence the loss of customers. Current researches into this topic have adopted a supervised approach to classify the reviews of the users. In this paper, we used an unsupervised approach to categorize the reviews aiming to generate a summary of the main bugs and new features pointed by users, assisting the application developers to improve the quality their apps. We evaluated the approach empirically and obtained promising results to generate user reviews summaries.

We created the "freedge" smartfridge network to level the playing field for the 1 in 10 Americans that live with "food insecurity", the condition of not knowing where a next meal will come from at least once per week. To understand more about how to alleviate hunger, we spoke with food assistance organizations and conducted a cultural probe with the general public. Our research suggested our design must (1) focus on both the short- and long-term problems (2) be close to people's homes (3) support ad-hoc use (4) enable low-obligation volunteering. Our prototype "freedge" is a device for neighborhoods to collect and distribute food donations. It has built-in cameras and locking compartments, with a companion app that allows users to browse and reserve food remotely as well as to crowdsource quality assurance.

While most existing research knowledge is distributed in the form of papers, there has been a shift towards learning and consuming information through video. Limited time and resources, however, prevent individual researchers from making their work available in this format. Crowdsourcing this task is a promising alternative, but it requires solving complex coordination and collaborative video production problems. In this paper, we propose an end-to-end solution to crowdsource the creation of research videos. To assist coordination, we designed a structured workflow that enables efficient delegation of tasks, while also providing a collaborative learning environment to the crowd for motivation. To facilitate video production, we developed an online system through which groups can make micro audio recordings that are automatically stitched together to create a complete talk. We tested this approach with a group of volunteers recruited from 24 countries through an open call. This distributed crowd produced five video talks based on the best paper winners and nominees from WWW 2016. Evaluations from the authors of the papers and outside reviewers rated the talks "very good" (giving a median score of 4 out of 5). We further applied this method to translate these talks and produce 11 additional videos in Spanish, Romanian, and Catalan. These results suggest that our crowdsourcing approach has the potential to significantly increase learning and the accessibility of the scientific knowledge.

Crowdsource management imposes many challenges including varying degree of demands for food and daily routine consumables. The explicit problem for estimating increase or decrease in demand arises when crowd moves dynamically to different locations with respect to time. This may result in wastage or shortage of many resources and loss for various stakeholders. To overcome the issue of demand estimation more precisely food demand and efficient supply, this paper proposes a conceptual framework to sense the data of crowd such as number of humans and update the suppliers timely. IoT and WSN techniques are used to count the number of humans in crowd at some particular area with respect to time for food demand estimation.

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: the classifier (i.e., the machine) selects the highest quality examples for training the crowdsource workers (i.e., the humans). Then, the latter annotate the lower quality examples such that the classifier can be re-trained with more accurate examples. This process can be iterated several times. We tested our approach on two different tasks, Relation Extraction and Community Question Answering, which are also in two different languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. At the same time, our method approach the performance than state-of-the-art methods using expensive Gold Standard for training workers

This paper describes the outcomes of a preliminary study into the design of a mobile app to crowdsource information related to “risk”. For the purpose of this study the notion of risk is defined broadly; however, we predominantly focus on the personal, subjective perception of risk. The study involved building a prototypical mobile app to crowdsource risk and exploring the use of the app as part of an expert workshop. Outcomes show challenges and opportunities with regards to the categorisation of results, the motivation of users, and interaction design of the prototype. The study provides value by giving an initial insight into this design space.

After a disaster, authorities need to efficiently collect and analyze data from the disaster area in order to increase their situational awareness and make informed decisions. The conventional data acquisition methods such as dispatching inspection teams are often time-consuming. With the widespread availability of mobile devices, crowdsourcing has become an effective alternative means for data acquisition. However, the large amount of crowdsourced data is often overwhelming and requires triage on the collected data. In this paper, we introduce a framework to crowdsource post-disaster data and a new prioritization strategy based on the expected value of the information contained in the collected data (entropy) and their significance. We propose a multi-objective problem to analyze a portion of the collected data such that the entropy retrieved from the disaster area and the significance of analyzed data are maximized. We solve this problem using Pareto optimization that strikes a balance between both objectives. We evaluate our framework by applying it on bridges inspection after the 2001 Nisqually earthquake as a case study. We also investigate the feasibility of sending the crowdsourced data to the crowd for reviewing. The results demonstrate the effectiveness and feasibility of the proposed framework.

Participatory sensing systems use people and their smartphones as a sensing infrastructure, and getting people to make contributions remains a critical challenge. Little work details how system designers should combine different interactions to increase coverage of service location. Tiramisu, a participatory sensing system, invites transit riders to crowdsource real-time arrival information by sharing location traces when they commute. We extended this system with a new feature that allows riders at stops to "spot" buses passing by. To better understand the impact of this new feature, we conducted an observational log analysis, examining changes in coverage and user behavior before and after the new feature. Following the addition of the spotting feature, participants' contributions increased coverage (the number of trips with real-time data) by 98%, and they used the app more than twice as much. The addition of the spotting feature was also followed by a significant increase of trace contributions.

The widespread day-to-day carrying of powerful smartphones gives opportunities for crowd-sourcing information about the users' activities to gain insight into patterns of use of a large population in cities. Here we report the design and initial investigations into a crowdsourcing approach for sudden decelerations to identify dangerous road sections. Sudden brakes and near misses are much more common than police reportable accidents but under exploited and have the potential for more responsive reaction than waiting for accidents. We also discuss different multimodal feedback conditions to warn drivers approaching a dangerous zone. We believe this crowdsourcing approach gives cost and coverage benefits over infrastructural smart-city approaches but that users need incentivized for use.

A life without food - a leitmotiv of science fiction dystopias is becoming a voluntary choice for thousands of people consuming the powdered food replacement soylent. The DIY soylent powders are designed by a group of nutrition hobbyists who distrust existing healthy eating standards and crowdsource experimental soylent recipes in their online user forums. This DIY food-tech rebellion offers an opportunity to look at the issues around present food standards and policies from the HCI perspective. This paper reports findings from in-depth interviews with 21 soylent dieters, with the aim of identifying the risks and opportunities of the expert-amateur DIY food practices.

Large annotated images from the Web and crowdsource, together with powerful machine learning tools, play a crucial role in rapid progress in semantic recognition of image data in recent years. However, as inferred in the saying "A Picture is Worth More Than 1,000 Words," there is much richer information than just semantic labels associated with images from the Web resources and Crowdsource Fora. Such additional information covers the rich unexploited aspects, such as visual aesthetics, emotions, sentiments, user intention, and knowledge structure. Discovering such novel dimensions of image descriptions beyond semantics will have huge impact for exciting emerging applications such as personalized search and content recommendation. But it requires rigorous research in concept definition, task formulation, data crawling, and evaluation mechanisms. In this talk, I will address these issues by sharing our experiences [1-4] in discovering beyond-semantic visual descriptions related to visual sentiment, video upload intent classification, cultural influence on visual sentiment, and finally a wiki-style video event ontology.

Active transportation is an important contributor to physical activity. Understanding active transportation trends and transportation mode share is important to public health research and city planners. Objective measurement of active transportation can be costly and time-consuming, and existing camera-based algorithms, while developing, are functionally limited to specific settings and distances. In this study, 28,992 publicly available webcam images from two intersections in Washington, D.C., were used to establish trends in active transportation. Amazon Mechanical Turk workers were found to be reliable identifiers of pedestrian and vehicular activity, data validated against trained research assistant image annotation. Webcam and crowdsource annotation provides a cost-effective alternative to traditional objective measures of active transportation and mode share through the use of publicly available wireless webcams. Additional research is needed to expand the utility and external validity of publicly available imaged-based active transportation methodology and image annotation.

Social help systems are changing the role of professional technical communicators from directly writing documentation to the support of amateurs who write documentation. This paper looks at one genre of social documentation---the crowdsourced documentation wiki---and explores how author support guides on these wikis are constructed. Ultimately, the paper argues that author support guides can be reconceptualized as documentation, and consequently that documentation design theories can be used to improve the construction of these guides. The paper concludes by showing how minimalist documentation design theory can improve the way we support amateur authors on these wikis.

We propose an approach to automatically select a set of representative images to generate a concise visual summary of a real-world event from the Tumblr microblogging platform. Central to our approach is a unified graph model with heterogeneous nodes and edges to capture the interrelationship among various entities (e.g., users, posts, images, and tags) in online social media. With the graph representation, we then cast the summarization problem as a graph-based ranking problem by identifying the most representative images regarding to an event. The intuition behind our work is that not only can we crowdsource social media users as sensors to capture and share data, but we can also use them as filters to identify the most useful information through analyzing their interaction in the microblogging network. In addition, we propose a greedy algorithm to encourage diversity among top ranked results for the generation of temporal highlights of targeted events. Our approach is flexible to support different query tasks and is adaptable to additional graph entities and relationships.

Most would agree that modern app-markets have been flooded with applications that not only threaten the security of the OS superficially, but also in their majority, trample on user's privacy through the exposure of sensitive information not necessarily needed for their operation. In this context, the current work revolves around 3 key questions: Is there a way for the end-user to easily track - the many times - hidden privacy leaks occurring due to the way mobile apps operate? Can crowdsourcing provide the end-user with a quantitative assessment per app in terms of privacy exposure level? And if yes, in which way a cloud-based crowdsourcing mechanism can detect and alert for changes in the apps' behavior? Motivated by the aforementioned questions, we design a cloud-based system that operates under a crowdsourcing logic, with the aim to provide i) a real-time privacy-flow tracking service, ii) a collaborative infrastructure for exchanging information related to apps' privacy exposure level, and iii) potentially a behavior-driven detection mechanism in an effort to take advantage of the crowdsourcing data to its maximum efficasy.

This paper describes creation of custom OCR profiles for improved recognition of text from historical documents. Presented workflow is based on tools developed by Poznan Supercomputing and Networking Center in the framework of SYNAT project. OCR customization consists of three steps. The first two include preparation of training material in a dedicated web application called Cutouts and processing of the training material using command-line tool called a page-generator. The results of the latter step are passed into Tesseract OCR engine. In the final step Tesseract creates a recognition profile which might be used for an OCR in Virtual Transcription Laboratory (VTL). The described tools allow to crowdsource the most tedious parts of the mentioned process: training material preparation and post OCR correction.

Creating good quality image databases for affective computing systems is key to most computer vision research, but is unfortunately costly and time-consuming. This paper describes BeFaced, a tile matching casual tablet game that enables massive crowdsourcing of facial expressions to advance facial expression analysis. BeFaced uses state-of-the-art facial expression tracking technology with dynamic difficulty adjustment to keep the player engaged and hence obtain a large and varied face dataset. CHI attendees will be able to experience a novel game interface that uses the iPad's front camera to track and capture facial expressions as the primary player input, and also investigate how the game design in general enables massive crowdsourcing in an extensible manner.

Participatory sensing systems (PSS) require frequent injection of information that has a short shelf-life. The use of crowds to gather information for PSS is therefore particularly challenging. In this study, we explore the impact of two policies on user contributions. A quid-pro-quo policy exchanges contributions from users for access to critical information in the system. A request policy simply reminds the user that information is needed to make the system function well. Prior research has shown that request for help in crowdsourced system is an effective mechanism to increase contributions. During a large-scale experimental study within a publicly deployed, crowdsourced, transit information system, we analyzed metrics associated with frequency of contribution and commitment to long-term use over a 10-month period. Our results confirmed that quid-pro-quo led to more contribution, but at a cost of faster departure from the study. When a participant was simply requested to contribute, but could still access community-generated data if they ignored a request, was largely ineffective and was statistically similar to the control condition where no request for contribution occurred. Thus crowdsource system designers should consider imposing quid-pro-quo type policies for PSS that concentrate on fewer users, but makes them more productive.

In this paper, we describe our newest project endeavor in which we conceptualize crowdsourcing technology development for and with rural communities in Namibia. The project is based on design work which was carried out over a longer period of time with a single rural pilot community in Namibia and its transferability of technology and concepts into other rural communities. In an attempt to overcome expensive technology adaptations we explore the possibility of having rural communities' crowdsource their defined and specified technology needs. We describe the concept and our current implementation with a first user evaluation in two rural communities. We also discuss the next phase of the project.

Spatial crowdsourcing has gained emerging interest from both research communities and industries. Most of current spatial crowdsourcing frameworks assume independent and atomic tasks. However, there could be some cases that one needs to crowdsource a spatial complex task which consists of some spatial sub-tasks (i.e., tasks related to a specific location). The spatial complex task's assignment requires assignments of all of its sub-tasks. The currently available frameworks are inapplicable to such kind of tasks. In this paper, we introduce a novel approach to crowdsource spatial complex tasks. We first formally define the Maximum Complex Task Assignment (MCTA) problem and propose alternative solutions. Subsequently, we perform various experiments using both real and synthetic datasets to investigate and verify the usability of our proposed approach.

Creative individuals increasingly rely on online crowdfunding platforms to crowdsource funding for new ventures. For novice crowdfunding project creators, however, there are few resources to turn to for assistance in the planning of crowdfunding projects. We are building a tool for novice project creators to get feedback on their project designs. One component of this tool is a comparison to existing projects. As such, we have applied a variety of machine learning classifiers to learn the concept of a successful online crowdfunding project at the time of project launch. Currently our classifier can predict with roughly 68% accuracy, whether a project will be successful or not. The classification results will eventually power a prediction segment of the proposed feedback tool. Future work involves turning the results of the machine learning algorithms into human-readable content and integrating this content into the feedback tool.

The paper introduces a novel approach for crowd curation of ambient music played in venues using physical Near Field Communication (NFC) enabled check-ins. The CheckinDJ jukebox system uses NFC enabled objects that are linked to individuals social network accounts. CheckinDJ can thus be regarded as a crowdsourced DJ that curates an evolving music playlist based on the musical preferences of those present at a location at any given time. In this paper we present the design and operation of the currently deployed system along with highlights of the promising potential of this novel concept.

In this demo, we present CrowdTerrier, an infrastructure extension to the open source Terrier IR platform that enables the semi-automatic generation of relevance assessments for a variety of document ranking tasks using crowdsourcing. The aim of CrowdTerrier is to reduce the time and expertise required to effectively Crowdsource relevance assessments by abstracting away from the complexities of the crowdsourcing process. It achieves this by automating the assessment process as much as possible, via a close integration of the IR system that ranks the documents (Terrier) and the crowdsourcing marketplace that is used to assess those documents (Amazon's Mechanical Turk).

Online labor markets, such as Amazon's Mechanical Turk, have been used to crowdsource simple, short tasks like image labeling and transcription. However, expert knowledge is often lacking in such markets, making it impossible to complete certain classes of tasks. In this work we introduce an alternative mechanism for crowdsourcing tasks that require specialized knowledge or skill: communitysourcing --- the use of physical kiosks to elicit work from specific populations. We investigate the potential of communitysourcing by designing, implementing and evaluating Umati: the communitysourcing vending machine. Umati allows users to earn credits by performing tasks using a touchscreen attached to the machine. Physical rewards (in this case, snacks) are dispensed through traditional vending mechanics. We evaluated whether communitysourcing can accomplish expert work by using Umati to grade Computer Science exams. We placed Umati in a university Computer Science building, targeting students with grading tasks for snacks. Over one week, 328 unique users (302 of whom were students) completed 7771 tasks (7240 by students). 80% of users had never participated in a crowdsourcing market before. We found that Umati was able to grade exams with 2% higher accuracy (at the same price) or at 33% lower cost (at equivalent accuracy) than traditional single-expert grading. Mechanical Turk workers had no success grading the same exams. These results indicate that communitysourcing can successfully elicit high-quality expert work from specific communities.

The wealth of freely available, structured information on the Web is constantly growing. Driving domains are public data from and about governments and administrations, scientific data, and data about media, such as articles, books and albums. In addition, general-purpose datasets, such as DB-pedia and Freebase from the linked open data community, serve as a focal point for many data sets. Thus, it is possible to query or integrate data from multiple sources and create new, integrated data sets with added value.

Thousands of websites have been created to crowdsource tasks. In this paper, systems that crowdsource human-based computations are organized into four distinct classes using two factors: the users' motivation for completing the task (direct or indirect) and whether task completion is competitive. These classes are described and compared. Considerations and selection criteria for systems designers are presented. This investigation also identified several opportunities for further research. For example, existing systems might benefit from the integration of methods for transforming complex tasks into many simple tasks.