Search

The aggregation of knowledge embedded in large language models (LLMs) holds the promise of new solutions to problems of observability and measurement in the social sciences. We examine this potential in a challenging setting: measuring latent ideology -- crucial for better understanding core political functions such as democratic representation. We scale pairwise liberal-conservative comparisons between members of the 116th U.S. Senate using prompts made to ChatGPT. Our measure strongly correlates with widely used liberal-conservative scales such as DW-NOMINATE. Our scale also has interpretative advantages, such as not placing senators who vote against their party for ideologically extreme reasons towards the middle. Our measure is more strongly associated with political activists' perceptions of senators than other measures, consistent with LLMs synthesizing vast amounts of politically relevant data from internet/book corpora rather than memorizing existing measures. LLMs will likely open new avenues for measuring latent constructs utilizing modeled information from massive text corpora., Comment: 18 pages, 4 figures; V2: fixed graphical error on Figure 2; V3: reorganized sections, updated prose

The prevalence of so-called fake news - false or misleading content purposefully produced to resemble news stories - and its potential to influence the public debate, affect citizens’ attitudes, and even distort electoral results has become a widespread concern in our society (Lazer et al., 2018; Tucker et al., 2018). Across distinct scholarly and popular accounts, the role of social media in daily life has often been raised as an environment facilitating the spread of online fake news and accentuating the harmful effects of misinformation on the quality of democratic regimes. Despite great academic interest in the relationship between social media and disinformation, a large part of the scholarship on the effects of social media comes from advanced democracies. Empirical evidence discussing the impact of social media on exposure and beliefs on misinformation in democracies in the Global South is still somewhat sparse (Tucker et al., 2018; Barberá, 2020; Mitchelstein and Boczkowski, 2021). In these young democracies, political institutions are arguably more unstable, identities are commonly more fragmented, and the interference of non-political actors in elections is more recurrent. Therefore, the disruptive role of social media on the quality of democratic regimes may arguably be even more harmful compared to advanced democracies. For years WhatsApp has been the primary social media application in many Global South countries, with several anecdotal and scholarly works suggesting that WhatsApp is the primary channel through which citizens are exposed to misinformation online in the Global South (Machado et al., 2019; Resende et al., 2019; Garimella and Tyson, 2018; Garimella and Eckles, 2020; Batista Pereira et al., ND). The platforms and dynamics differ from those studies in advanced democracies, where Twitter and particularly Facebook have traditionally received more attention for facilitating the spread of online misinformation. To fill these gaps in the literature on social media and misinformation, we propose to conduct a field experiment deployed during the Presidential election in Brazil focusing on reducing users’ exposure to misinformation shared on WhatsApp. Our field experiment will randomly assign users to a Media- Constrained WhatsApp, in which we ask them not to access any media, image, video, or audio received during the weeks up to the forthcoming Presidential election in Brazil on October 3, 2022. After users participate in our Media-Constrained WhatsApp Deactivation for three weeks, we will ask participants to respond a final survey to measure the intervention effects on their ability to identify misinformation and political polarization. The Media-Constrained WhatsApp intervention is motivated by two primary reasons. First, previous literature has suggested that video and images are the primary source of mis- information spreading on WhatsApp in the Global South (Resende et al., 2019; Freitas Melo et al., 2019; Machado et al., 2019; Garimella and Tyson, 2018; Garimella and Eckles, 2020). Second, because of user’s reliance on WhatsApp in daily life, a complete deactivation is unlikely to be feasible from an external validity perspective. Our study makes three primary contributions. We contribute to the growing experimental litera- ture on the political effects of social media usage. So far, no studies have been conducted to identify a causal connection between WhatsApp usage and attitudinal and behavioral outcomes, particularly users’ ability to determine the veracity of online news and the decision to share false information. To date, we are only aware of prior studies identifying the causal effects of a randomized deactivation of Facebook accounts (Hanley et al., 2019; Vanman et al., 2018; Hall et al., 2021; Allcott et al., 2020; Asimovic et al., 2021). Our study is the first to provide a causal link between messaging apps like WhatsApp and users’ behaviors and attitudes. Messaging applications, like WhatsApp, differ from other social media platforms in a few key dimensions. For the specific case of WhatsApp, any communication within the app is end-to-end encrypted, which means that content is effectively unregulated and unmoderated by the platform. In addition, WhatsApp provides several features for viral spreading, such as public and private groups, automatic forwarding, and broadcasting of content to many contacts with just a few taps. Finally, content is not curated by any algorithmic ranking as in other social media applications; all messages are sorted chronologically. These particular affordances justify the purpose of this study, particularly considering that some of these features make the messaging apps a fertile ground for the circulation of misinformation and other harmful campaigns. Second, in contrast to previous studies that use a full deactivation to focus more on outgroup ethnic and partisan hostility and personal well-being, our study focuses primarily on reducing users’ exposure to media content, which the previous literature has suggested is the model category for the spread of misinformation in the Global South. Therefore, our work provide as more robust assessment for the causal effects of social media usage on users’ believes and exposure misinformation (Allcott et al., 2020; Asimovic et al., 2021). Although we also pre-register tests of hypotheses regarding polarization, due to the evidence of media as the main format for the circulation of misinformation on WhatsApp, our study primary focus deals with measuring the effects of the WhatsApp Media Deactivation on misinformation. Our belief is that the subset of people willing to deactivate WhatsApp entirely (a platform used for a great deal of both personal and professional communication, as well as financial transactions such as bill payments) would be a very distinct subsection of the population from whom we would be hesitant to generalize outcomes. Third, social media messaging apps have become the primary means through which online misinformation circulates in major markets in the Global South. We are the first to assess the causal effects of social media usage in the context of elections outside of the United States, providing new insights into studies of social media and politics and bringing greater external validity to related research deployed in other contexts.

Scholarship has identified key determinants of people’s belief in misinformation, but our knowledge predominantly comes from English-language misinformation in the United States. However, in the global media environment, multilingual citizens often consume media in more than one language. To this end, we ask how the language in which misinformation is consumed affects belief in misinformation in multilingual media environments? We suggest that the extent to which people believe misinformation will depend on the language in which the misinformation is consumed. We hypothesize that language passes on credibility cues influenced by both domestic status of the language they prefer to speak and international conditions. These cues affect how bilinguals evaluate misinformation in their less preferred language. In a ten-week survey experiment with bilingual adults in Ukraine, we plan to measure if subjects who are asked to evaluate misinformation in their less-preferred language are less likely to believe it.

Understanding the relationship between digital media literacy and susceptibility to dubious sources of onlinecontent is crucial for designing interventions to counter the spread of misinformation (Guess et al. 2020).Unlike some competencies, these skills can potentially be taught, raising the possibility of scalable solutionsto our informational malaise. This project sets out to understand the lasting impact of such initiativesonline. For this project, we test the efficacy of the web extension NewsGuard on shifting online consumptionfrom unreliable sources known for publishing misleading or false content to credible sources. We test if this will also reduce the level of belief in misinformation and increase trust in credible sources.We test if this intervention has larger effects on sub-populations most vulnerable to fake news and if there are downstream effects on other outcomes (affective polarization, political cynicism, etc.) that correlate with viewing, believing, and sharing news from unreliable sources. The following pre-analysis plan was filed before the researchers gained access to the survey or tracking data.

This individual study is a part of a larger study that is motivated by two very closely related research questions. First, recent literature on fake news identifies some types of individuals most likely to spread fake news across social media platforms. However, previous research has much less to say about who possesses the ability to discern fake from genuine news and, therefore, who is most vulnerable to believing fake news to be true, or, conversely, believing true news to be false. While we seek to test these hypotheses in order to advance scientific understanding of the determinants of susceptibility to believing fake news to be true (or true news to be false), we are also motivated by two pragmatic questions. First, is it possible to ”crowd-source” fact checking in a way that allows us to come reasonably close to matching the performance of professional fact checkers? Second, what is the trade-off between making our crowd representative of the general population – which may be desirable for normative or political reasons – and the average level of the crowd’s accuracy? In answering this second question, we draw upon the empirical findings from our first set of individual-level hypotheses that are laid out in a previous pre-registration (https://osf.io/dp8ze/?view only=4c3c24c0cb9f4625ba2574c7bd5aac9f). In this study we will test a series of hypotheses about the individual-level characteristics associated with correctly discerning fake news from true news. We will compare how individuals evaluate fake news in a normal setting relative to a crisis, such as Covid-19. Previous pilot studies have found that individuals are less likely to match the assessment of professional fact checkers when the partisan slant of a fake news article matches the respondent’s own partisanship than when the partisan slant of a fake news article does not match the individual’s own partisanship. We would like to understand if individuals are more or less susceptible to fake news about a large crisis even (Covid-19). Specifically we would like to test what effect partisanship has on believing fake news stories during a crisis. In the current study, we plan to run a 4-week survey in which individuals evaluate fake and true news stories within 72 hours of their publication online. For the purpose of this study, we will ascertain whether each article is true, false/misleading, or whether the veracity of the story is unclear by having a team of professional fact-checkers label the story along these lines. Respondents in our survey will be required to complete the survey within 24 hours of when the survey is open for responses. Therefore, respondents will evaluate these articles within the first 72 hours after they are published online. Each week the survey will be run for two days (due to difficulties with recruitment on Fridays and necessary fact-checks before the articles go out, the survey will only be open on Tuesdays and Thursdays). Each day the survey runs, a new group of 5 popular articles will be selected for evaluation. These articles are determined to be related to coronavirus if they meet this definition: ”Articles that are directly related to the coronavirus, the causes of coronavirus, the medical and public health consequences of coronavirus, the direct economic and social consequences from coronavirus, and policies related to the pandemic (e.g., lockdowns or “openings”).” To track the information individuals engage with if they choose to search for evidence to evaluate the news article, we run google searches using the headline of the article hourly and scrape the top eleven search results during the 24-hour period in which the survey is open. We collect the URL, headline shown in each google search result, text below the headline shown in each google search result, and the full text from each article that is in the top eleven search results.

Scholarship has identified key determinants of people’s belief in misinformation, but our knowledge predominantly comes from English-language misinformation in the United States. However, in the global media environment, multilingual citizens often consume media in more than one language. To this end, we ask how the language in which misinformation is consumed affects belief in misinformation in multilingual media environments? We suggest that the extent to which people believe misinformation will depend on the language in which the misinformation is consumed. We hypothesize that language passes on credibility cues influenced by both domestic status of the language they prefer to speak and international conditions. These cues affect how bilinguals evaluate misinformation in their less preferred language. In a ten-week survey experiment with bilingual adults in Ukraine, we plan to measure if subjects who are asked to evaluate misinformation in their less-preferred language are less likely to believe it.