Search

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Online testing for behavioral research has become an increasingly used tool. Although more researchers have been using online data collection methods, few studies have assessed the replicability of findings for speech intelligibility tasks. Here we assess intelligibility in quiet and two noise-added conditions for several different accents of English (Midland American, Standard Southern British, Scottish, German-accented, Mandarin-accented, Japanese-accented, and Hindi-English bilingual). Participants were tested in person at a museum-based laboratory and online. Results showed little to no difference between the two settings for the easier noise condition and in quiet, but large performance differences in the most difficult noise condition with an advantage for the participants tested online. Technology-based variables did not appear to drive the setting effect, but experimenter presence may have influenced response strategy for the in-person group and differences in demographics could have provided advantages for the online group. Additional research should continue to investigate how setting, demographic factors, experimenter presence, and motivational factors interact to determine performance in speech perception experiments. [ABSTRACT FROM AUTHOR]

Data collection using Internet-based samples has become increasingly popular in many social science disciplines, including advertising. This research examines whether one popular Internet data source, Amazon's Mechanical Turk (MTurk), is an appropriate substitute for other popular samples utilized in advertising research. Specifically, a five-sample between-subjects experiment was conducted to help researchers who utilize MTurk in advertising experiments understand the strengths and weaknesses of MTurk relative to student samples and professional panels. In comparisons across five samples, results show that the MTurk data outperformed panel data procured from two separate professional marketing research companies across various measures of data quality. The MTurk data were also compared to two different student samples, and results show the data were at least comparable in quality. While researchers may consider MTurk samples as a viable alternative to student samples when testing theory-driven outcomes, precautions should be taken to ensure the quality of data regardless of the source. Best practices for ensuring data quality are offered for advertising researchers who utilize MTurk for data collection. [ABSTRACT FROM PUBLISHER]

While race is often discussed in relation to police, there is still little empirical evidence on the role of race in shaping citizens' evaluations of government performance. Using an experimental design, this study examines how different levels of police performance affect perceptions of police trustworthiness and to which degree observed effects vary by individual race and across communities with different racial makeups. Specifically, we study the effect of different levels of performance on perceived trustworthiness in two communities—one predominantly African American and one predominantly White—and replicate across two samples: a sample consisting of primarily White participants and another consisting of only African American participants. Results indicate that the interaction between the racial composition of a community and individual race plays a critical role in shaping the effects of performance among White participants, who appeared more sensitive to community context than African American participants. Implications for research and practice are discussed. [ABSTRACT FROM AUTHOR]

Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)—that is, objective questions about key elements of the experiment—can identify individual‐level attentiveness to experimental information and, as a consequence, better enable researchers to diagnose experimental findings. We then find, through four replication studies, little evidence that FMC placement affects treatment effects, and that placing FMCs immediately post‐outcome does not attenuate FMC passage rates. Additionally, FMC and IMC passage rates are only weakly related, suggesting that each technique identifies different sets of attentive subjects. Thus, unlike other methods, FMCs can confirm attentiveness to experimental protocols. [ABSTRACT FROM AUTHOR]

This study examined the current trajectory of arts participation and engagement through the lens of socioeconomic status. The study was structured to explore and describe any potential correlation between education, familial support, and introduction to the arts in youth, with current adult experience and patronage. The multiple choice survey posed questions that underscore the possible relationship between youth experience and adult outcomes in terms of participation and engagement with the arts. The survey found that the level of activity during adolescence had a closer relationship to adult participation in the arts than did overall exposure. Even though exposure is seen as a more consumptive aspect of arts participation, it was less of a predictor of adult experience than early engagement on an active level. The study also found that synthesis of two or more factors including exposure, engagement, education, familial support, and acculturation may be required to effect change in socioeconomic status in adulthood. [ABSTRACT FROM AUTHOR]

Social meaning-based approaches to linguistic variation treat variation as a semiotic system, in which sociolinguistic signs—indexical links between linguistic forms and social meanings—serve as interactional resources that individuals use to project personae. This article explores the perceptual nature of the links between social personae and linguistic forms, examining how information about a speaker's persona can influence a listener's linguistic perceptions of a continuous phonetic feature. Using a phoneme categorization task, this study examines associations between gradient phonetic manifestations on a continuum from /æ/ to /ɑ/ and three social personae. Findings illustrate that the social persona made relevant for a listener influences the ways in which points on this phonetic continuum are categorized phonemically as either trap or lot. Overall, this shows that the social constructs of personae influence phonetically detailed perceptions of linguistic material. (Sociolinguistic perception, personae, indexicality, sociophonetics, sociolinguistic signs)* [ABSTRACT FROM AUTHOR]

Advertisers need a better understanding of online video sharing to design strategies for creating viral advertising. Previous studies have focused on users' deeper psychological motivations for sharing online ads. This research focuses on implications of psychological motivations for content strategy. Employing an online study with millennials, it identifies significant triggers for eliciting virality for video ads (i.e., appear ahead of trends, stimulate discussion, elicit strong reactions, make a statement of uniqueness, and share positive emotions etc.). Managerial implications for applying these triggers to the content of video ads to elicit higher levels of sharing are included. [ABSTRACT FROM AUTHOR]

Most research on evaluations of women candidates considers single elections in isolation. Using two Dynamic Process Tracing experiments, this article examines whether voters alter their evaluations of women candidates, as well as their willingness to learn about and vote for them, based on the presence of other women running simultaneously in concurrent contests. We find a consistent pattern in which female candidates are not adversely affected when they are the only woman on a voter’s ballot, but they are disadvantaged when other women appear on the same party’s ballot in other races. This effect is more prominent for women in lower offices: women running for the House of Representatives are more disadvantaged than women running for higher offices are. [ABSTRACT FROM AUTHOR]

Crowdsourcing is an increasingly popular method for researchers in the social and behavioral sciences, including experimental philosophy, to recruit survey respondents. Crowdsourcing platforms, such as Amazon’s Mechanical Turk (MTurk), have been seen as a way to produce high quality survey data both quickly and cheaply. However, in the last few years, a number of authors have claimed that the low pay rates on MTurk are morally unacceptable. In this paper, I explore some of the methodological implications for online experimental philosophy research if, in fact, typical pay practices on MTurk are morally impermissible. I argue that the most straightforward solution to this apparent moral problem—paying survey respondents more and relying only on “high reputation” respondents—will likely increase the number of subjects who have previous experience with survey materials and thus are “non-naïve” with respect to those materials. I then discuss some likely effects that this increase in experimental non-naivete will have on some aspects of the “negative” program in experimental philosophy, focusing in particular on recent debates about philosophical expertise. [ABSTRACT FROM AUTHOR]

Word intuition is speakers' intuitive knowledge on wordhood. Collective word intuition is the word intuition of the whole language community. Given this definition, the optimal word segmentation result in Chinese NLP should reflect collective word intuition. It is also believed that an ideal definition of Chinese word should accord with the collective word intuition of Chinese speakers. To test the validity and feasibility of modeling collective word intuition, it is important to know to what extent Chinese speakers agree with each other on what is a word. In this study, we measured word intuition agreement using Mechanical Turk-based Chinese word segmentation experiment. Three metrics were used: proportionate agreement, Cohen's kappa, and Fleiss' kappa. The results show that Chinese speakers agree with each other almost perfectly on what is a word. And we found no evidence to support an effect of semantic transparency on word intuition agreement. Such high word intuition agreement among Chinese speakers supports the psychological reality of Chinese word and also suggests that that it is quite feasible to formulate a definition of Chinese word by modeling the collective word intuition of Chinese speakers. [ABSTRACT FROM AUTHOR]

Objective Existing research on support for anti-poverty programs largely focuses on broad categories of welfare. In this article, we examine variations in support across different public assistance programs. Methods We use an experimental survey design to examine whether support for public assistance is dependent on the type of aid offered. Results We find that programs that offer benefits in-kind are more popular than cash transfers. Moreover, food stamps and child-care subsidies enjoy more support than housing assistance. Open-ended survey responses show that when evaluating anti-poverty programs, respondents adopt one of two perspectives: (1) cash assistance is problematic but other forms of assistance are acceptable or (2) any assistance is problematic. Conclusion By too narrowly focusing on welfare, social scientists run the risk of developing theories and explanations that may not apply to the much larger part of the safety net that is delivered in-kind. [ABSTRACT FROM AUTHOR]

There has been a continuous increase in the design and application of computer games for purposes other than entertainment in recent years. Serious games--games that motivate behavior and retain attention in serious contexts--can change the attitudes, behaviors, and habits of players. These games for change have been shown to motivate behavior change, persuade people, and promote learning using various persuasive strategies. However, persuasive strategies that motivate one player may demotivate another. In this article, we show the importance of tailoring games for change in the context of a game designed to improve healthy eating habits. We tailored a custom-designed game by adapting only the persuasive strategies employed; the game mechanics themselves did not vary. Tailoring the game design to players' personality type improved the effectiveness of the games in promoting positive attitudes, intention to change behavior, and self-efficacy. Furthermore, we show that the benefits of tailoring the game intervention are not explained by the improved player experience, but directly by the choice of persuasive strategy employed. Designers and researchers of games for change can use our results to improve the efficacy of their game-based interventions. [ABSTRACT FROM AUTHOR]

Crowdsourcing has proliferated across disciplines and professional fields. Implementers in the public sector face practical challenges, however, in the execution of crowdsourcing. This review synthesizes prior crowdsourcing research and practices from a variety of disciplines and focuses to identify lessons for meeting the practical challenges of crowdsourcing in the public sector. It identifies three distinct categories of crowdsourcing: organizations, products and services, and holistic systems. Lessons about the fundamental logic of process design-alignment, motivation, and evaluation-identified across the three categories are discussed. Conclusions drawn from past studies and the resulting evidence can help public managers better design and implement crowdsourcing in the public sector. [ABSTRACT FROM AUTHOR]

To answer questions about the origins and outcomes of collective action, political scientists increasingly turn to datasets with social network information culled from online sources. However, a fundamental question of external validity remains untested: are the relationships measured between a person and her online peers informative of the kind of offline, “real-world” relationships to which network theories typically speak? This article offers the first direct comparison of the nature and consequences of online and offline social ties, using data collected via a novel network elicitation technique in an experimental setting. We document strong, robust similarity between online and offline relationships. This parity is not driven by shared identity of online and offline ties, but a shared nature of relationships in both domains. Our results affirm that online social tie data offer great promise for testing long-standing theories in the social sciences about the role of social networks. [ABSTRACT FROM PUBLISHER]

Despite its widespread use in studies of race and ethnic politics, there exists a long-standing debate about whether racial resentment primarily measures antiblack prejudice or ideological conservatism. In this paper, we attempt to resolve this debate by examining racial resentment's role in shaping white opinion on a "racialized" policy issue that involves no federal action and no government redistribution of resources: "pay for play" in college athletics. Using cross-sectional and experimental data from the 2014 Cooperative Congressional Election Study and Amazon's Mechanical Turk, we find evidence not only that racial resentment items tap racial predispositions but also that whites rely on these predispositions when forming and expressing their views on paying college athletes. More specifically, we demonstrate that racially resentful whites who were subtly primed to think about African Americans are more likely to express opposition to paying college athletes when compared with similarly resentful whites who were primed to think about whites. Because free-market conservatism, resistance to changes in the status quo, opposition to expanding federal power, and reluctance to endorse government redistributive policies cannot possibly explain these results, we conclude that racial resentment is a valid measure of antiblack prejudice. [ABSTRACT FROM AUTHOR]

Guided by the belief that anonymity inevitably breeds the kind of uncivil discourse that hurts their readers, many news organizations have chosen to ban anonymous comments sections on their websites in recent years. Unfortunately, little empirical research has been conducted to assess whether exposure to anonymous comments actually does influence people's attitudes. In this paper, we address this oversight by asking: do anonymous comments posted on a newspaper website shape how internet users feel about the media? Using an online experiment to systematically manipulate exposure to anonymous comments attached to a “hard news” report, we find strong evidence that exposure to non-attributed posts—regardless of their tone—leads internet users to feel more negatively towards specific news organizations and the media in general. [ABSTRACT FROM AUTHOR]

As part of daily interactions humor is an integral element of work relationships with consequences for well-being or turnover intentions. However, its adequate assessment in the work context has yet to be improved. While the Humor Styles Questionnaire (HSQ; Martin et al. 2003. Individual differences in uses of humor and their relation to psychological well-being: Development of the Humor Styles Questionnaire. Journal of Research in Personality 37. 48-75) covers adaptive and maladaptive functions of humor styles, it assesses humor in general life. With the adaption of the HSQ, that is, the short (12-item) and work-related version of the HSQ (swHSQ), we provide an efficient tool for the assessment of individual humor styles in the work context. We validated and tested the swHSQ in two Austrian-German and one U.S. sample and analyzed its structure and impact by means of confirmatory factor analyses as well as regression analyses. The swHSQ seems valid and reliable (α between 0.62 and 0.86) in a German and an English version, thus representing an efficient self-report tool for field research. Evidence for construct validity is provided with regard to humor styles' associations with optimism, occupational self-efficacy, irritation, and job satisfaction. While the positive (affiliative, self-enhancing) humor styles showed expected relationships with the outcomes, patterns of relationships of the negative styles suggest intercultural differences in workplace humor. Thus, self-defeating but not aggressive humor seems impactful in the Austrian-German sample, while the contrary was found in the U.S. sample. [ABSTRACT FROM AUTHOR]

The choice-overload hypothesis states that increasing the number of alternatives reduces people's motivation to choose. Possible adverse effects of choice overload in liberalized public service markets have been discussed repeatedly, however, an empirical evaluation of whether this holds true is missing. In this study, we extend and test the theory of choice overload. By means of a randomized survey experiment, we investigate whether or not increasing the number of providers of public services in the US electricity sector has detrimental effects on peoples' motivation to switch their provider after a service failure. We randomly varied the number of service providers in a hypothetical service failure scenario. Results show that increasing provider choice reduces people's likelihood of stating that they would switch away from a poor performing provider by 10% points. These findings also hold when replicating the experiment with an independent online sample. Thus our results indicate that increasing provider choice in public service markets reduces peoples' motivation to switch away from poor performing public services. In turn, this may lead to a situation where citizens become locked-in to a suboptimal provider simply due to an overload of choices. The theoretical and practical implications of our findings are discussed. [ABSTRACT FROM AUTHOR]

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers. [ABSTRACT FROM AUTHOR]

This manuscript presents a study on how leader motivating language and follower self-leadership act to influence a follower's feelings of psychological safety. This study found that both constructs significantly influenced psychological safety in samples from India and the USA. Additionally, this study found that this influence occurred through the mediating processes of trust in leadership, leader inclusiveness, and role clarity. These mediators fully explained motivating language's relationship with psychological safety, but only partially explained self-leadership's relationship. Differences existed in the model between samples, but self-leadership showed an overall consistency between the samples for most relationships. Follow-up analysis indicated that self-leadership without leader communication support lead to a weak or non-existent relationship between self-leadership and psychological safety, but a positive and relatively strong relationship in the presence of motivating language. [ABSTRACT FROM AUTHOR]