Search

The present study introduces the concept of validating data quality indicators through re-classification. We use the term re-classification to mean the evaluation of how well an indicator detects the quality of different versions of a survey question for which the quality is known a priori. We illustrate its application with two examples. In both, we make use of 12 questions from prior experiments that manipulated text features of questions to create 'low' and 'high' quality versions of each question. In the first example, we coded each question version in SQP 2.1 to obtain indicators of validity, reliability, and quality. We compared these indicators between the two versions of each question to assess whether the SQP outcomes were sensitive to text features. In the second example, we used a pretest survey to obtain three indicators of survey quality: response latencies, item nonresponse, and consistency over time. Again, we compared these indicators between question versions to assess whether the indicators were sensitive to text features. We give recommendations for applying re-classification and an outlook for future research opportunities. [ABSTRACT FROM AUTHOR]

Field-testing is an essential yet often resource-intensive step in the development of high-quality educational assessments. I introduce an innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root mean squared error = 0.18). The study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach did not achieve the level of accuracy obtainable with human examinee response data. If further refined, potential resource savings in transitioning from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low-quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1 [ABSTRACT FROM AUTHOR]

Pediatric intelligence tests, such as the Wechsler Intelligence Scale for Children, are commonly used diagnostic tools used in the process of diagnosing learning and behavior disabilities. Decisions concerning treatment are made based on the results of these tests and they are frequently used in educational and vocational contexts for important decisions that impact persons' academic or professional lives. Research has however shown that important errors may occur despite the application of validation processes and adherence to quality criteria for psychometric tests. At the same time this evidence seems not to be pervasively acknowledged in psychological practice and research. In this article, I will showcase research that places attention on sources of measurement error in pediatric intelligence testing, discuss a process-performance approach to measurement in intelligence testing, and propose the "pretest methods," methods stemming from the field of survey methodology commonly used in questionnaire construction, as a method to help address the problem of sources of measurement error in pediatric intelligence testing and improve the development of these intelligence tests. [ABSTRACT FROM AUTHOR]

Background: Much social marketing research is done on-line recruiting participants through Amazon Mechanical Turk, vetted panel vendors, social media, or community sources. When compensation is offered, care must be taken to distinguish genuine respondents from those with ulterior motives. Focus of the Article: We present a case study based on unanticipated empirical observations made while evaluating perceived effectiveness (PE) ratings of anti-tobacco public service announcements (PSAs) using facial expression (FE) analysis (pretesting). Importance to the Social Marketing Field: This study alerts social marketers to the risk and impact of disinterest or fraud in compensated on-line surveys. We introduce FE analysis to detect and remove bad data, improving the rigor and validity of on-line data collection. We also compare community (free) and vetted panel (fee added) recruitment in terms of usable samples. Methods: We recruited respondents through (Community) sources and through a well-known (Panel) vendor. Respondents completed a one-time, random block design Qualtrics® survey that collected PE ratings and recorded FE in response to PSAs. We used the AFFDEX ® feature of iMotions ® to calculate respondent attention and expressions; we also visually inspected respondent video records. Based on this quan/qual analysis, we divided 501 respondents (1503 observations) into three groups: (1) Those demonstrably watching PSAs before rating them (Valid), (2) those who were inattentive but completed the rating tasks (Disinterested), and (3) those employing various techniques to game the system (Deceitful). We used one-way analysis of variance (ANOVA) of attention (head positioning), engagement (all facial expressions), and specific facial expressions (FE) to test the likelihood a respondent fell into one of the three behavior groups. Results: PE ratings: The Community pool (N = 92) was infiltrated by Deceitful actors (58%), but the remaining 42% was "attentive" (i.e., no disinterest). The Panel pool (N = 409) included 11% deceitful and 2% disinterested respondents. Over half of the PSAs change rank order when deceitful responses are included in the Community sample. The smaller proportion of Deceitful and Disinterested (D&D) respondents in the Panel affected 2 (out of 12) videos. In both samples, the effect was to lower the PE ranking of more diverse and "locally made" PSAs. D&D responses clustered tightly to the mean values, believed to be an artefact of "professional" test taking behavior. FE analysis: The combined Valid sample was more attentive (87.2% of the time) compared to Disinterested (51%) or Deceitful (41%) (ANOVA F = 195.6, p <.001). Models using "engagement" and specific Fes ("cheek raise and smirk") distinguished Valid from D&D responses. Recommendations: False PE pretesting scores waste social marketing budgets and could have disastrous results. Risk can be reduced by using vetted panels with a trade-off that community sources may produce more authentically interested respondents. Ways to make surveys more tamper-evident, with and without webcam recording, are provided as well as procedures to clean data. Check data before compensating respondents! Limitations: This was an accidental finding in a parent study. The study required computers which potentially biased the pool of survey respondents. The community pool is smaller than the panel group, limiting statistical power. [ABSTRACT FROM AUTHOR]

In this article, we consider inference in the linear instrumentalvariables models with one or more endogenous variables and potentially weak instruments. I developed a command, twostepweakiv, to implement the twostep identification-robust confidence sets proposed by Andrews (2018, Review of Economics and Statistics 100: 337-348) based on Wald tests and linear combination tests (Andrews, 2016, Econometrica 84: 2155-2182). Unlike popular procedures based on first-stage F statistics (Stock and Yogo, 2005, Testing for weak instruments in linear IV regression, in Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg), the two-step identificationrobust confidence sets control coverage distortion without assuming the data are homoskedastic. I demonstrate the use of twostepweakiv with an example of analyzing the effect of wages on married female labor supply. For inference on subsets of parameters, twostepweakiv also implements the refined projection method (Chaudhuri and Zivot, 2011, Journal of Econometrics 164: 239-251). I illustrate that this method is more powerful than the conventional projection method using Monte Carlo simulations. [ABSTRACT FROM AUTHOR]

In questionnaire pretesting, supplementing cognitive interviewing with eye tracking is a promising new method that provides additional insights into respondents’ cognitive processes while answering survey questions. When incorporating eye tracking into cognitive interviewing, two retrospective probing techniques seem to be particularly useful. In the first technique—retrospective probing—participants complete an online questionnaire, while cognitive interviewers monitor participants’ eye movements in an adjacent room and note down any peculiarities in their reading patterns. Afterward, the interviewers ask targeted probing questions about these peculiarities in a subsequent cognitive interview. In the second technique—gaze video cued retrospective probing—respondents are additionally shown a video of their eye movements during the cognitive interview. This video stimulus is supposed to serve as a visual cue that may better enable respondents to remember their thoughts while answering the questions. We examine whether one of the two techniques is more effective when it comes to identifying problematic survey questions. In a lab experiment, participants’ eye movements (n = 42) were tracked while they completed six questions of an online questionnaire. Simultaneously, their reading patterns were monitored by an interviewer for evidence of response problems. After completion of the online survey, a cognitive interview was conducted. In the retrospective probing condition, probing questions were asked if peculiar reading patterns were observed during the eye-tracking session (e.g., rereadings of specific words or text passages). In the other condition, participants were shown a video of their recorded eye movements, in addition to receiving probing questions about the questions displayed. Results show that both techniques did not differ in terms of the total number of problems identified. However, gaze video cued retrospective probing identified fewer unique problems and fewer types of problems than pure retrospective probing. [ABSTRACT FROM AUTHOR]

The study, reported in this article, shows that the connection between the interviewer’s and/or interviewee’s behaviors, which deviate from the ideal course of standardization, and the interpretative distortions due to badly formulated questions is weak. The behaviorist theoretical thesis, which supports the methodological structure of the verbal interaction coding, has to be revised. To diagnose the questions which make the interpretative problems more frequent in interview, social researcher cannot limit himself/herself to detecting and encoding verbal behaviors. His/her goal is to evaluate the suitability of the interviewer’s and/or interviewee’s behaviors with respect to the meanings mutually exchanged during the interview. This is the way to follow for modifying the verbal interaction coding. [ABSTRACT FROM AUTHOR]

Qualitative methods are becoming widely used and increasingly accepted in biomedical research involving teams formed by experts from developing and developed practice environments. Resources are rare in offering guidance on how to surmount challenges of team integration and resolution of complicated logistical issues in a global setting. In this article we present a critical reflection of lessons learned and necessary steps taken to achieve methodological coherence and international team synergy. A series of 10 pretest interviews were conducted to assess instrumentation rigor and formulate measures to address any limitations or threats to bias and management procedures before carrying out the formal phase of qualitative research, contributing to an evidence-based stroke-preventive care clinical trial study. The experience of pretesting notably helped to identify obstacles and thus increase the methodological and social reliability central to conducting credible qualitative research, while also ensuring both personal and professional fulfillment of our team members. [ABSTRACT FROM AUTHOR]

Text message services that provide sexual health information are increasingly popular, but often they are not promoted. This article describes the development of a social marketing campaign promoting a state-based sexual health text message service that allows teens to text a sexual health question directly to a trained health educator. Preproduction and production research using in-depth interviews and focus groups (n = 35) conducted in the southeastern United States provided information on perceived benefits and barriers and teens’ perceptions. Teens were interested in the service and wanted to see relevant settings and relatable teens in advertisements. Teens said that the promotional messages needed to include clear instructions on how to use such a service, as teens were only somewhat familiar with the process of using such a system. Implications for promotion of similar services are discussed. [ABSTRACT FROM AUTHOR]

Expert reviews are frequently used as a questionnaire evaluation method but have received little empirical attention. Questions from two surveys are evaluated by six expert reviewers using a standardized evaluation form. Each of the questions has validation data available from records. Large inconsistencies in ratings across the six experts are found. Despite the lack of reliability, the average expert ratings successfully identify questions that had higher item nonresponse rates and higher levels of inaccurate reporting. This article provides empirical evidence that experts are able to discern questions that manifest data quality problems, even if individual experts vary in what they rate as being problematic. Compared to a publicly available computerized question evaluation tool, ratings by the human experts positively predict questions with data quality problems, whereas the computerized tool varies in success in identifying these questions. These results indicate that expert reviews have value in identifying question problems that result in lower survey data quality. [ABSTRACT FROM PUBLISHER]

This article explores the use of expert respondent debriefings to evaluate the quality of survey data. In this case study, subject matter experts observed 169 interviews and conducted qualitative respondent debriefings on select cases in a field test of a census coverage survey. By comparing "true" residence status for each person as determined by the debriefing against the residence status obtained by the questionnaire alone, the authors determined whether the questionnaire was collecting accurate information. For the 473 people for whom survey data was available, the questionnaire failed only five times to assign the correct residence status code in a way that would have detrimentally affected coverage estimates. The respondent debriefing technique helped pinpoint specific problems in the questionnaire, as well as confirm that the questionnaire performed adequately in most cases. This article describes the expert respondent debriefing methodology that was used in a face-to-face interview setting and discusses how it could be adapted to telephone interviewing. [ABSTRACT FROM AUTHOR]

Behavior coding is a widely used method for pretesting survey questionnaires. It is based on the idea that certain types of interviewer-respondent interactions indicate question flaws if the number of such interactions exceeds some specified percentage of tile pretest interviews. In this article, we examine statistical approaches for assessing tile reliability of different observed percentages of problematic interactions. [ABSTRACT FROM AUTHOR]

A brochure that had been revised on the basis of feedback from readers using the plus- minus evaluation method was evaluated again using the same method. This article compares the results of these two successive evaluation studies to examine the dynamics of evaluating and revising using a troubleshooting method based on verbal self-reports. The findings show that the plus-minus method does not necessarily lead to a decrease in the number of problems readers find in a revised document. But the types of problems readers find are significantly different. For example, after the brochure was revised, it had fewer clarity and structural problems, and readers could focus more on credibility issues. [ABSTRACT FROM AUTHOR]