Search

Regression discontinuity (RD) designs have gained significant popularity as a quasi-experimental device for evaluating education programs and policies. In this paper, we present a comprehensive review of RD designs, focusing on the continuity-based framework, the most widely adopted RD framework. We first review the fundamental aspects of RD designs, drawing on potential outcomes and causal graphs. We then discuss the validity threats in RD designs, including manipulation, discreteness of the running variable, statistical power, and generalizability. Additionally, we provide an overview of the existing extensions to RD designs. To exemplify the application of RD methods, we analyze the effect of New Jersey's pre-kindergarten program on children's vocabulary test scores, using an educational dataset. Finally, we offer practical guidelines in the conclusion to promote the appropriate use of RD methods in educational research.

Growth in the online survey market may be increasing response burden and possibly jeopardizing higher response rates. This meta-analysis evaluated survey trends over one decade (2011-2020) to determine: (1) changes in survey publication rates over time, (2) changes in response rates over time, (3) typical response rates within health sciences education research, (4) the factors influencing survey completion levels, and (5) common gaps in survey methods and outcomes reporting. Study I estimated survey publication trends between 2011 and 2020 using articles published in the top three health sciences education research journals. Study II searched the anatomical sciences education literature across six databases and extracted study/survey features and survey response rates. Time plots and a proportional meta-analysis were performed. Per 2926 research articles, the annual estimated proportion of studies with survey methodologies has remained constant, with no linear trend (p > 0.050) over time (Study I). Study II reported a pooled absolute response rate of 67% (95% CI = 63.9-69.0) across 360 studies ("k"), totaling 115,526 distributed surveys. Despite response rate oscillations over time, no significant linear trend (p = 0.995) was detected. Neither survey length, incentives, sponsorship, nor population type affected absolute response rates (p [greater than or equal to] 0.070). Only 35% (120 of 339) of studies utilizing a Likert scale reported evidence of survey validity. Survey response rates and the prevalence of studies with survey methodologies have remained stable with no linear trends over time. We recommend researchers strive for a typical absolute response rate of 67% or higher and clearly document evidence of survey validity for empirical studies.

Research on anxiety and motivation in second language (L2) learning is proliferating. However, these two individual differences are commonly measured by self-report questionnaires, the psychometric properties of which have been questioned. Building on previous works on study quality, this methodological synthesis systematically describes and evaluates the state of study and scale quality in L2 anxiety and motivation research. A total of 104 peer-reviewed articles (113 independent samples) that used 340 L2 anxiety and motivation scales were coded for 84 features targeting: study, survey, and scale designs; participant demographics; scale validity and reliability; and reporting practices associated with transparency. The results show a number of strengths but also reveal multiple areas in need of methodological enhancement. The article concludes by highlighting areas that merit further investigation in the domain of L2 survey research and making recommendations for future studies.

The Research Methods Skills Assessment (RMSA) was created to measure psychology majors' statistics knowledge and skills. The American Psychological Association's Guidelines for the Undergraduate Major in Psychology (APA, 2007, 2013) served as a framework for development. Results from a Rasch analysis with data from n = 330 undergraduates showed good fit for 16 of the 21 items created. Validity analysis showed that using the RMSA to measure students' statistics knowledge was an improvement over the assessment recommended by the APA Guidelines and was less time consuming. Together the findings provide a preliminary test that can be further developed to provide a tool for instructors and departments to use when assessing psychology students' statistics knowledge and skills. Future research should expand the RMSA to include items that measure knowledge with confidence intervals as the proposed items were removed due to poor fit. Future studies should also aim to replicate the current findings with additional samples of psychology students.

Structural equation modeling (SEM) is one of the most popular multivariate statistical techniques in Turkish educational research. This study elaborates the SEM procedures employed by 75 educational research articles which were published from 2010 to 2015 in Turkey. After documenting and coding 75 academic papers, categorical frequencies and percentages were calculated in the content analysis. Methodological issues related to use of SEM were identified. The results showed that usage and reporting SEM in Turkish educational research articles were mostly problematic. Content analysis revealed that educational researchers tend to avoid assumption testing of SEM and reporting estimation methods. General guidelines and suggestions on the use of SEM were provided.

Comparative accuracy studies evaluate the relative performance of two or more diagnostic tests. As any other form of research, such studies should be reported in an informative manner, to allow replication and to be useful for decision-making. In this study we aimed to assess whether and how components of test comparisons were reported in comparative accuracy studies. We evaluated 100 comparative accuracy studies, published in 2015, 2016 or 2017, randomly sampled from 238 comparative accuracy systematic reviews. We extracted information on 20 reporting items, pertaining to the identification of the test comparison, its validity, and the actual results of the comparison. About a third of the studies (n = 36) did not report the comparison as a study objective or hypothesis. Although most studies (n = 86) reported how participants had been allocated to index tests, we could often not evaluate whether test interpreters had been blinded to the results of other index tests (n = 40; among 59 applicable studies), nor could we identify the sequence of index tests (n = 52; among 90 applicable studies) or the methods for comparing measures of accuracy (n = 59). Two-by-four table data (revealing the agreement between index tests) were only reported by 9 of 90 paired comparative studies. More than half of the studies (n = 64) did not provide measures of statistical imprecision for comparative accuracy. Our findings suggest that components of test comparisons are frequently missing or incompletely described in comparative accuracy studies included in systematic reviews. Explicit guidance for reporting comparative accuracy studies may facilitate the production of full and informative study reports.

The main aim of this study is to determine whether a significant difference exits among data collection with printed materials and with smart phones. The research was conducted with 282 teacher candidates who take pedagogical formation training. Three data collection tools were used throughout the study. As the results of the current research, no significant differences were obtained among attitude scores towards teaching profession with regard to common effect of application method and faculty variables. However, significant differences in participant opinions were determined among different application methods (smartphone and printed) of Metacognitive Thinking Skills and Information Literacy Scales. In addition, no significant difference was found among opinions towards Metacognitive Thinking Skills and Information Literacy Scales with regard to common effect of application method and faculty variables. Based on these research findings, it is not recommended to distribute survey items containing technical information or some complex expressions via smartphones. Relations among the questionnaire content and the department of education confirm this situation.

One of the major successes of advanced quantitative methods has been its seeming ability to provide unbiased determinations of which education practices are effective for education in general and for improving the educational achievement and opportunity of the neediest students. The power of this methodology as applied in the top education research journals has led to periodic implementation of federal and state "effective practices policies". In such policies the government or its proxy determines which programs are effective and then requires or encourages schools to spend its funds exclusively on those proven programs. For example, the federal Investing in Innovation (i3) initiative requires those applying for its largest dissemination grants to have had their intervention validated as effective by the What Works Clearinghouse (WWC). Some are now even advocating that all expenditures of federal funds by school districts be restricted to purchasing programs that research has proven to be effective, i.e., that they work. While this seems like rational policy on the surface, between 1998 and 2002 I produced a series of articles that showed that the most research validated program to develop the reading skills of students born into poverty was not actually effective in practice. Was this dichotomy between published research proving something to be effective and what was happening in the real world an anomaly or a more widespread problem? This article (a) analyzes in easy-to-understand language the validity of the gold standard scientific methodology used by the top research journals and WWC to determine whether practices are effective, and (b) examines the history of "effective practices policies" and their actual effectiveness. I conclude that the increasingly sophisticated methods used to assess the effectiveness of practices (a) are flawed and exaggerate actual effectiveness, and (b) do not provide the type of information practitioners need. As a result, research on effective practices tends to mislead rather than inform practice and are a major reason why efforts to reform high-poverty schools have had limited success. I therefore conclude that "effective practices policies" should not be implemented. I then suggest ideas for reforming the scientific process used to assess the effectiveness of education interventions.

Studies based on longitudinal data focus on the change and development of the situation being investigated and allow for examining cases regarding education, individual development, cultural change, and socioeconomic improvement in time. However, as these studies require taking repeated measures in different time periods, they may include various error sources that are exclusive of the investigated variable and that change over time. In Addition, the period of observation is critical in these studies, and observant (variable) attrition or the inability to obtain measurements for various reasons (death, moving away, resignation etc.) may be observed. This study presents the coefficient of individual agreement (CIA) developed for studies in which more than one rater are needed to provide repeated scoring and examines the applications of this agreement coefficient in relation to different sample size, missing value and number of repeated measures. In this context, trial scores were generated based on different sample sizes, missing value ratios, and the number of repeated measures provided by two raters, and the expected agreement between raters was simulated as very high. Related data were replicated 1000 times. Data analysis included the mean and standard deviation of agreement ratios calculated for each combination. According to the results, independently from sample size and number of retests, the CIA takes much lower values than the expected value when random missing value rates reaches 20%. Similar conclusions can be drawn for the existence of monotone missing values except where the sample size is 30 and the number of retests are two and three. It was observed that the CIA was higher than the expected value for the data consisting of two repeated evaluations of the raters with 5% or 10% missing values when the sample size was 30. In general, the agreement coefficient did not exceed the expected level in the situation of three and four replications of measurements. As a result, it can be concluded that an increase in the replication number of measurement decreased the obtained agreement values. The obtained data showed that missing value ratio caused differentiations in the CIA. As a result, increases in missing value ratio cause inconsistencies in CIA. Therefore, it is suggested that data obtained from studies be interpreted considering missing value ratio.

The crisis caused by the COVID-19 virus has far-reaching effects in the field of education, as schools were closed in March 2020 in many countries around the world. In this article, we present and discuss the School Barometer, a fast survey (in terms of reaction time, time to answer and dissemination time) that was conducted in Germany, Austria and Switzerland during the early weeks of the school lockdown to assess and evaluate the current school situation caused by COVID-19. Later, the School Barometer was extended to an international survey, and some countries conducted the survey in their own languages. In Germany, Austria and Switzerland, 7116 persons participated in the German language version: 2222 parents, 2152 students, 1949 school staff, 655 school leaders, 58 school authority and 80 members of the school support system. The aim was to gather, analyse and present data in an exploratory way to inform policy, practice and further research. In this article, we present some exemplary first results and possible implications for policy, practice and research. Furthermore, we reflect on the strengths and limitations of the School Barometer and fast surveys as well as the methodological options for data collection and analysis when using a short monitoring survey approach. Specifically, we discuss the methodological challenges associated with survey data of this kind, including challenges related to hypothesis testing, the testing of causal effects and approaches to ensure reliability and validity. By doing this, we reflect on issues of assessment, evaluation and accountability in times of crisis.

Borrowing from a clinical psychology observational methodology, thin-slice observations were used to assess autism characteristics in toddlers. Thin-slices are short observations taken from a longer behavior stream which are assigned ratings by multiple raters using a 5-point scale. The raters' observations are averaged together to assign a "thin-slice" value for each observation. In this study, a total of 60 toddlers were selected from a video archive: 20 children with typical development, 20 children with developmental language disorder, and 20 children with autism. In the first part of this study, 20 raters observed small play segments between toddlers and an assessor. Raters assigned scores to each of the 60 toddlers on items related to autism symptomatology. Item analysis and generalizability and decision studies were conducted to determine the factor structure and optimal number of raters to achieve a stable estimate of autism characteristics. In the second part of the study, generalizability and decision studies were conducted to determine the most efficient and optimal combination of raters and naturalistic contexts. This pilot study provides recommendations for optimizing the utility of thin-slice observations for measuring autism symptomatology in young children.

The criterion-related validity of divergent thinking (DT) tests has been questioned because of relatively low correlations between DT test scores and creative performance. A possible inhibiting factor in previous correlations--heterogeneity of the sample--has been ignored. If groups with different DT profiles are involved in this line of research, low correlations would not necessarily be indicative of actual relationships, and criterion-related validity would appear to be lower than it is. Thus, this study identified groups with their own unique DT profiles and examined their creative performance. To this end, elementary children's (N = 325) responses in 3 DT tasks were analyzed using latent profile analysis. Results suggested 3 groups with unique DT profiles: One group scored moderate on fluency and low on originality; another group scored moderate on fluency and high on originality; the third group scored uniformly low across fluency, originality, and flexibility. Surprisingly, only the low-originality group excelled in creative achievement, but the hypothesis of this investigation was supported by the fact that only the high-originality group significantly excelled in ideation. These results confirm that low correlations between DT indices and creative performance do not necessarily indicate a lack of the criterion-related validity of DT tests. Differences between the creative achievement and the ideation criterion measures are detailed, and idiosyncratic properties in DT profiles and their corresponding creative strengths are discussed as educational implications.

This systematic review provides a summary of studies on teaching, social, and cognitive presences in the community of inquiry (CoI) model using the CoI instrument in a higher education setting since its development in 2008 by Arbaugh et al. A total of 24 peer-reviewed studies on the CoI instrument from 2008-2017 were selected and analyzed explore the types of research on the instrument in higher education settings have been published, ways the instrument has been used for learning and teaching, and the implications the instrument poses for online instructors and instructional designers. It is organized into 4 themes: testing the instrument for validity and reliability, measuring CoI presence in different environments, examining causal relationships among the elements, and exploring potential revisions to the model.

Developmental assessment scales are important tools for determining developmental delays and planning preventive interventions. One broad assessment scale used to evaluate child development is the Battelle Developmental Inventories (BDIs). The BDI-2 has a standardized version in English with good psychometric properties and a translated version in Spanish; however, despite widespread clinical use, there has been limited early childhood development research focused on this assessment tool in the past 10 years. The purpose of this systematic review was to evaluate the literature about the BDIs and their screening test and analyze their use for child assessment. Seven databases were used to retrieve articles in English, Portuguese, and Spanish. Overall, 34 articles were evaluated for general features, salient findings, and key methodological issues, such as target population, objectives, research design, and main results. Authors' considerations about the psychometric properties and utility of the BDIs were also examined. Results indicated that 41.17% of studies with the BDIs were from the United States, and the most common target population was children with autism spectrum disorders. Generally, the articles highlighted the broad applications of this measure and robust psychometric properties cited in the BDIs examiner's manual as reasons for their use. This review suggests that it is important to conduct independent analyses of the psychometric properties of the BDIs as well as validation studies to ensure appropriate applications of the BDI, including for use with non-American populations.

The current set of three studies further evaluates the validity and application of the Psychological Research Inventory of Concepts (PRIC). In Study 1, we administered the PRIC to a sample of introductory psychology students and online (Mechanical Turk) participants along with measures assessing theoretically related concepts. We found evidence of concurrent validity by demonstrating that PRIC scores relate to education, cognitive effort, and general psychological knowledge. In Study 2, we demonstrated that advanced psychology majors score higher on the PRIC than college graduates who did not major in psychology, suggesting that the PRIC assesses domain-specific knowledge and may be useful in assessment of the undergraduate psychology major. Finally, in Study 3, we demonstrated that scores on the PRIC increase from the start to the end of a research methods course. Together, these studies provide further evidence that the PRIC may be a useful index of learning research methods and statistical knowledge important for the undergraduate psychology degree.

The report of the national response to intervention (RTI) evaluation study, conducted during 2011-2012, was released in November 2015. Anyone who has read the lengthy report can attest to its complexity and the design used in the study. Both these factors can influence the interpretation of the results from this evaluation. In this commentary, we (a) explain what the national RTI evaluation examined and highlight the strengths and weaknesses of the design, (b) clarify the results of the evaluation and highlight some key implementation issues, (c) describe how rigorous efficacy trials on reading interventions can supplement several issues left unanswered by the national evaluation, and (d) discuss implications for future research and practice based on the findings of the national evaluation and reading intervention research.

This article offers a rationale for the contributions of qualitative research to evidence-based practice in special education. In it, I make the argument that qualitative research encompasses the ability to study significant problems of practice, engage with practitioners in the conduct of research studies, learn and change processes during a study, and provide expansive data sets that help clarify both independent and dependent variables. Qualitative methodologies can shape and advance important questions of educational practice and policy. The concern with the degree to which qualitative research can contribute to the research base may be a reflection of the degree to which our field as a whole adequately prepares its researchers to understand and engage in high-quality qualitative research that adheres to benchmarks for internal and external validity from inception through dissemination of results.

Fuzzy-set qualitative comparative analysis (fsQCA) has become one of the most prominent methods in the social sciences for capturing causal complexity, especially for scholars with small- and medium-"N" data sets. This research note explores two key assumptions in fsQCA's methodology for testing for necessary and sufficient conditions--the cumulation assumption and the triangular data assumption--and argues that, in combination, they produce a form of aggregation bias that has not been recognized in the fsQCA literature. It also offers a straightforward test to help researchers answer the question of whether their findings are plausibly the result of aggregation bias.

The 1961 Copyright Office study on renewals, authored by Barbara Ringer, has cast an outsized influence on discussions of the U.S. 1923-1963 public domain. As more concrete data emerge from initiatives such as the large-scale determination process in the Copyright Review Management System (CRMS) project, questions are raised about the reliability or meaning of the Ringer data. A closer examination of both the Ringer study and CRMS data demonstrates fundamental misunderstandings and misrepresentations of the Ringer data, as well as possible methodological issues. Estimates of the size of the corpus of public domain books published in the United States from 1923 through 1963 have been inflated by problematic assumptions, and we should be able to correct mistaken conclusions with reasonable effort.

Although peer nomination measures have been used by researchers for nearly a century, common methodological practices and rules of thumb (e.g., which variables to measure; use of limited vs. unlimited nomination methods) have continued to develop in recent decades. At the same time, other key aspects of the basic nomination procedure (e.g., whether nonparticipants should be included as nominees, the consequences of pairing code numbers with names on rosters) are underdiscussed and understudied. Beyond providing a general introduction to peer nomination methods and their utility, the current article discusses the advantages and disadvantages of various methodological choices facing researchers who wish to use peer nomination methods, in addition to other considerations that researchers must make in collecting peer nomination data (e.g., establishing reliability and validity, maximizing participation rates, computerized assessments). This article provides recommendations for researchers based on empirical findings (where possible) and the typical practices used in the recent published literature.

Cognitive interviewing (CI) is a method to identify sources of confusion in assessment items and to assess validity evidence on the basis of content and response processes. We introduce readers to CI and describe a process for conducting such interviews and analyzing the results. Recommendations for best practice are provided.

In this commentary, we offer a set of visual tools that can assist education researchers, especially those in the field of mathematics, in developing cohesiveness from a mixed methods perspective, commencing at a study's research questions and literature review, through its data collection and analysis, and finally to its results. This expounds upon the ideas and methodological insights relating to research connections and structure presented by other scholars (e.g., Creswell, 2009; Heid & Blume, 2011; Mertens, 2010; Thanheiser, Ellis, & Herbel-Eisenmann, 2012). As faculty who work with graduate students in education, we have found that these visual tools help students elicit and maintain salient connections throughout the research process.

Using as a starting point in the recent work of Mountford-Zimdars et al., the authors analyse attitudes towards expanding higher education (HE) opportunities in the UK. The authors propose that the approach of Mountford-Zimdars et al. is flawed not only in its adoption of a multivariate logistic regression but also in its interpretation of results. The authors make a number of adaptations, chief among them being the use of an ordered probit approach and the addition of a time dimension to test for changes in attitudes between 2000 and 2010. The authors find that attitudes towards HE expansion have intensified during the decade 2000-2010, but the authors uncover no evidence that this is due to graduates wanting to "pull up the ladder", as suggested by Mountford-Zimdars et al. The authors argue that evidence of a widespread desire to reduce access to HE can most likely be explained by social congestion theory, internal institutional disaffection and rising tuition fees.

When classifying the evidence base of practices, special education scholars typically appraise study quality to identify and exclude from consideration in their reviews unacceptable-quality studies that are likely biased and might bias review findings if included. However, study quality appraisals used in the process of identifying evidence-based practices for students with learning and other disabilities have not been empirically validated (e.g., studies classified as unacceptable quality shown to have different, and presumably more biased, effects than high-quality studies). Using Gersten et al.'s (2005) approach for appraising the quality of group experimental studies in special education, we examined whether (a) studies classified as unacceptable quality and high quality had meaningfully different effects and (b) unacceptable-quality studies were more likely to have outlying effects than high-quality studies among 36 group experimental studies that investigated the effectiveness of instructional practices for students with learning disabilities. Our preliminary analyses found that the effects of unacceptable-quality studies were not meaningfully different from the effects of high-quality studies. We discuss implications of these findings and call for more research to be conducted in this area.

Critically examining common statistical approaches and their strengths and weaknesses is an important step in advancing recreation and leisure sciences. To continue this critical examination and to inform methodological decision making, this study compared three approaches to determine how alternative approaches may result in contradictory conclusions in the interpretation of the psychometric properties of a scale and in response to a given research question. To this end, this study explored what factors best predicted parental endorsement of competition climbing in a sample of 184 parents of youth competition climbers. The study findings suggest that the three distinct approaches provided meaningfully different conclusions regarding the adapted psychometric properties of the questionnaire, but offered no meaningful differences in the primary finding of the study: Parent-coach relationship quality is the best predictor of parental endorsement of competition climbing. The results suggest that deeper examination of self-report questionnaire data may advance our understanding of complex recreation and leisure constructs beyond what can be understood with less advanced analytic techniques.

We explore the conditions under which short, comparative interrupted time-series (CITS) designs represent valid alternatives to randomized experiments in educational evaluations. To do so, we conduct three within-study comparisons, each of which uses a unique data set to test the validity of the CITS design by comparing its causal estimates to those from a randomized controlled trial (RCT) that shares the same treatment group. The degree of correspondence between RCT and CITS estimates depends on the observed pretest time trend differences and how they are modeled. Where the trend differences are clear and can be easily modeled, no bias results; where the trend differences are more volatile and cannot be easily modeled, the degree of correspondence is more mixed, and the best results come from matching comparison units on both pretest and demographic covariates.

In the current article, we consider the influential position of the Programme for International Student Assessment (PISA) and discuss several methodological areas that demonstrate the need for caution when using and interpreting PISA results. We motivate our argument by briefly describing the program's increased influence in educational policy over time. Subsequently, we describe the methodological areas of interest, including sampling participants, the achievement estimation model, and measuring trends. We also offer our perspectives on how the Organisation for Economic Co-operation and Development might productively and more clearly communicate PISA's limitations.

Most discipline-based education researchers (DBERs) were formally trained in the methods of scientific disciplines such as biology, chemistry, and physics, rather than social science disciplines such as psychology and education. As a result, DBERs may have never taken specific courses in the social science research methodology--either quantitative or qualitative--on which their scholarship often relies so heavily. One particular aspect of (quantitative) social science research that differs markedly from disciplines such as biology and chemistry is the instrumentation used to quantify phenomena. In response, this "Research Methods" essay offers a contemporary social science perspective on test validity and the validation process. The instructional piece explores the concepts of test validity, the validation process, validity evidence, and key threats to validity. The essay also includes an in-depth example of a validity argument and validation approach for a test of student argument analysis. In addition to DBERs, this essay should benefit practitioners (e.g., lab directors, faculty members) in the development, evaluation, and/or selection of instruments for their work assessing students or evaluating pedagogical innovations.

The results of large-scale international assessments such as Programme for International Student Assessment (PISA) have attracted a considerable attention worldwide and are often used by policy-makers to support educational policies. To ensure that the published results represent the actual population, these surveys go through a thorough scrutiny to ensure their validity. One important issue that may affect the results is the occurrence of different participation rates across groups with heterogeneous average student scores. In this study, we illustrate how problems of representativeness of the samples may be corrected by the use of post-stratified weights. We use the case of Portugal, a country where we uncover a meaningful difference between the actual population and the PISA samples in terms of the distribution of students per grade and track of studies. We show that after post-stratification, the temporal evolution of the PISA scores is quite different from the one reported.

In this article, I provide answers to the questions posed by Will Shadish about the history and development of the Schmidt-Hunter methods of meta-analysis. In the 1970s, I headed a research program on personnel selection at the US Office of Personnel Management (OPM). After our research showed that validity studies have low statistical power, OPM felt a need for a better way to demonstrate test validity, especially in light of court cases challenging selection methods. In response, we created our method of meta-analysis (initially called validity generalization). Results showed that most of the variability of validity estimates from study to study was because of sampling error and other research artifacts such as variations in range restriction and measurement error. Corrections for these artifacts in our research and in replications by others showed that the predictive validity of most tests was high and generalizable. This conclusion challenged long-standing beliefs and so provoked resistance, which over time was overcome. The 1982 book that we published extending these methods to research areas beyond personnel selection was positively received and was followed by expanded books in 1990, 2004, and 2014. Today, these methods are being applied in a wide variety of areas.

Evaluation is a key component to learning about the effectiveness of a program. This article provides descriptive statistics of the newly developed National 4-H Common Measures (science, healthy living, citizenship, and youth development) based on data from 721 California 4-H youth. The measures were evaluated for their reliability and validity of individual items and overall measures using exploratory factor analysis. The measures overall appear to assess what they are intended to assess, but there are several methodological issues, such as cross-loading items and low variance. Recommendations for scale refinement and modifications are made.

Black and Hispanic students are undeniably underidentified as gifted and underrepresented in gifted education. The underrepresentation of the two largest groups of "minority" students is long-standing, dating several decades, and is a serious area of contention. Most debates focus on the efficacy of traditional intelligence tests with verbal, quantitative, and nonverbal scales compared to intelligence tests that are nonverbal when identifying underserved gifted students. Test developers; researchers; federal, state, and local government officials; policymakers; administrators; and educators have debated different solutions to the problem of underrepresentation of minorities in gifted educational programs for decades. Controversies surrounding how to equitably identify these gifted students abound, and arguments are quite polemic and entrenched; nonetheless, in many instances gifted Hispanic and Black students are often disproportionally denied access to gifted education because of the methods and instruments used. In this article, we review a study by Giessman, Gambrell, and Stebbins regarding one Nonverbal Test of General Ability (NNAT2), which has been widely used for identification of gifted non-White students. We address concerns about conclusions raised by Giessman and coauthors and present cautions about the problems involved in reporting archival data.

Replications and robustness checks are key elements of the scientific method and a staple in many disciplines. However, leading journals in developmental psychology rarely include explicit replications of prior research conducted by different investigators, and few require authors to establish in their articles or online appendices that their key results are robust across estimation methods, data sets, and demographic subgroups. This article makes the case for prioritizing both explicit replications and, especially, within-study robustness checks in developmental psychology. It provides evidence on variation in effect sizes in developmental studies and documents strikingly different replication and robustness-checking practices in a sample of journals in developmental psychology and a sister behavioral science--applied economics. Our goal is not to show that any one behavioral science has a monopoly on best practices, but rather to show how journals from a related discipline address vital concerns of replication and generalizability shared by all social and behavioral sciences. We provide recommendations for promoting graduate training in replication and robustness-checking methods and for editorial policies that encourage these practices. Although some of our recommendations may shift the form and substance of developmental research articles, we argue that they would generate considerable scientific benefits for the field.

Daniel Koretz is the Henry Lee Shattuck Professor of Education at the Harvard Graduate School of Education. His research focuses on educational assessment and policy, particularly the effects of high-stakes testing on educational practice and the validity of score gains. He is the author of "Measuring Up: What Educational Testing Really Tells Us" (Harvard University Press, 2008). Koretz comments here that he finds it distressing after generations of work and countless articles and chapters, that those in the field still cannot agree on what validity means. As someone who has worked for three decades to persuade people of the need to devote more focus to the evaluation of impact, Koretz argues he would find it helpful if more colleagues would use simple language that people outside of the field understand when making the argument for "impact" in order to make the term "validity" useful.

Kathleen Slaney, associate professor in the History, Quantitative and Theoretical Psychology stream in the Department of Psychology at Simon Fraser University, comments on three issues she considers central to a fruitful discussion of how "validity" should be used in the context of testing.

Background: Since the development of the Death Literacy Index (DLI) in 2019 in Australia, subsequent internationally validated versions have prompted rewording and refinement of the original survey questions. Use of the DLI in the community has also resulted in requests for a short format. Objectives: To examine and report on the psychometric properties of a revised version of the DLI-R and develop a short format DLI-9. Design: A cross-sectional national survey was conducted for the validation of the revised DLI. Methods: The DLI items were revised by the research team using the international literature. DLI data were collected from a representative online non-probability panel of 1202 Australian adults, based on age, gender, and geographical location. Confirmatory factor analysis (CFA) was conducted to ensure the revised version (DLI-R) was consistent with the original. To develop a short format version of the DLI (DLI-9), items were first removed based on face validity, followed by an exploratory factor analysis (EFA) and CFA. The internal reliability of the DLI-R and the DLI-9 was assessed using Cronbach's alpha. The intraclass correlation coefficient was calculated to examine the inter-rater reliability between the DLI-R and DLI-9. Results: Twenty-four questions in the DLI were reworded for clarity. A CFA on the 29 items of this modified version of the DLI indicated a good model fit (Tucker-Lewis Index (TLI): 0.93; Comparative Fit Index (CFI): 0.93; root mean square of approximation (RMSEA): 0.06; standardized root mean residual (SRMR): 0.06), with six latent variables and an underlying latent variable "death literacy." For the DLI-9, an EFA identified a nine-item, two-factor structure model (DLI-9). A subsequent CFA in a separate sample demonstrated a good model fit for the DLI-9 (TLI: 0.92; CFI: 0.94; RMSEA: 0.089; SRMR: 0.07). Excellent inter-rater reliability (0.98) was observed between DLI-9 and DLI-R. Cronbach's alpha coefficients for DLI-R scales and subscales and the DLI-9 all exceeded 0.8, indicating high internal consistency. Conclusion: The DLI-R and the DLI-9 were found to have acceptable psychometric properties. The development of a shorter version of the DLI provides a valid measure of overall death literacy. Plain language summary: Progressing the Death Literacy Index: the development of a revised version (DLI-R) and a short format (DLI-9) The Death Literacy Index (DLI) is used to measure the knowledge and skills people about death and dying. Multiple countries have made their own versions, which helped improve the original questions. Community members also asked for a shorter version of the DLI. The research team updated the DLI questions using information from other studies around the world. Data was collected from 1202 Australian adults. A method called Confirmatory Factor Analysis (CFA) was used to check if the new version (DLI-R) was similar to the original. To make a shorter version, some questions were removed first, then Exploratory Factor Analysis (EFA) and CFA were used. Reliability of both versions was tested using a test called Cronbach's alpha, and the short version was compared to the DLI-R to see if results were similar. Twenty-four questions were reworded to make them clearer. The revised 29-question version (DLI-R) worked well and showed a good fit with six main ideas under the main topic of "death literacy." Nine questions were chosen for the shorter version (DLI-9) and were found to be closely related to the longer version. Both versions showed high reliability, meaning the questions consistently measured what they were supposed to. Both the DLI-R and the DLI-9 worked well and were found to be reliable survey tools. [ABSTRACT FROM AUTHOR]

Background: Vitamin B12 deficiency is widely prevalent in all age groups which is of major concern. However, there is no valid Food Frequency Questionnaire (FFQ) for dietary vitamin B12 estimation. Hence, we aimed to develop and validate an FFQ for the estimation of dietary intake of Vitamin B12. Materials and Methods: Commonly consumed B12-rich food items were selected from literature and filtered using a market survey. For concordant and discriminant validation, B12 and homocysteine levels were estimated. To establish convergent validity, the Cobalamin Intake in North Indians by Food Frequency Questionnaire (COIN-FFQ) and 72-hour dietary recall (72HrDR) were both administered to the same subjects. The COIN-FFQ was readministered after initial administration for test-retest reliability. Internal consistency of the FFQ was then tested using Cronbach's alpha. Results: We enrolled 115 adults with a mean age and weight of 31.9 ± 8.7 years and 66.0 ± 11.8 kg, respectively. In total, 19.1% were vegetarian. The dietary B12 using COIN-FFQ (n = 60; mean = 4.3 ± 1.8 µg/d) was significantly correlated (r = 0.255; P = 0.049) with serum levels (mean = 120.1 ± 62.6 pmol/L) establishing concordant validity. A significant difference was noted between the dietary, serum B12, and homocysteine levels of vegetarians versus nonvegetarians establishing discriminant validity (mean diff 1.4 (0.5-2.4), P = 0.004; Z-statistic -2.182, P value 0.029, and Z-statistic -2.438; P value 0.015), respectively. FFQ was strongly correlated with 72HrDR and test--retest FFQ (n = 27; r = 0.814, P < 0.001 and r = 0.869, P < 0.001, respectively) establishing convergent validity and test--retest reliability. The internal consistency with Cronbach's alpha was in the acceptable range, 0.631 (n = 115). Conclusion: The newly developed COIN-FFQ is valid and reliable in estimating dietary B12 intake. [ABSTRACT FROM AUTHOR]

Purpose: To develop an index to assess capacity to work in relation to common mental disorders (CMDs) in the general working population and field test its psychometric properties. Methods: Content analysis of three qualitative studies on people (n = 49) with their own experiences of working with CMD guided the items selected for the index. Face and content validity and test-retest reliability were performed. The index was field tested in two versions with 26 and 17 items, respectively, among health care professionals regarding internal reliability, component structure and concurrent validity. Results: The final version of the Capacity to Work Index (C2WI; 17 items) was normally distributed in the field test with high internal reliability (Cronbach's alpha, 0.84). Missing responses were randomly distributed and nonspecific. Principal component analysis showed one clear component with negatively framed items. Concurrent validity showed high correlation with the WHO-5 Well-Being Scale (Pearson's r, 0.68), but lower correlation for the general health question (r, − 0.44), one item of the Work Ability Index (r, − 0.33), and the Stress of Conscience constructs (r, 0.44). Conclusion: The C2WI showed promising psychometric qualities. Low and negative correlation with the item from Work Ability Index suggests that the C2WI measures additional dimensions, but further testing in larger and more diverse samples is required. [ABSTRACT FROM AUTHOR]

Purpose: Despite the importance of measuring delirium severity in patients with Alzheimer's disease and related dementias (ADRD), no validated instrument currently exists. Our goal was to generate items for use in such an instrument. Method: An interdisciplinary expert panel developed items to assess seven domains and 21 subdomains of delirium severity. Nursing experts provided input on items. Results: Experts achieved consensus on 21 items, including best respondents for each item, and appropriate look-back periods. Nursing experts emphasized the need for educating nursing staff and obtaining information from caregivers. Conclusion: Careful, nuanced identification of distinguishing features is key for generating items for measuring delirium severity in persons with ADRD. Once developed, engaging nurses will be essential to facilitate adoption and relevance of the tool. [Journal of Gerontological Nursing, 50(9), 24–30.] [ABSTRACT FROM AUTHOR]

Background: Management of type 2 diabetes (T2D) as a chronic disease requires treatment adherence such as controlling the blood glucose level and adopting a healthy lifestyle. The present study aimed to develop and psychometrically evaluate a questionnaire based on the Pender's Health Promotion Model (HPM) to measure treatment adherence and the associated factors among T2D patients. Methods: The present study was conducted in qualitative and the quantitative phases between March 2022 and March 2023. The participants were T2D patients visiting Shahid Mohammadi hospital Diabetes Clinic in Bandar Abbas in the south of Iran. The first draft of items was extracted from the qualitative phase. The present study used interviews with T2D patients, item construction, validity and reliability evaluation of the instrument, and the relevant statistical analyses. It emphasized the significance of content, face, and construct validity, along with reliability testing using Cronbach's alpha and test-retest method. Data were analyzed using SPSS software, V16 and AMOS, V23. Results: A 97-item questionnaire was developed through the qualitative phase and, after content validity, it was reduced to 86 items. Five items were removed in face validation, and after the test-retest method, 79 items were retained. The confirmatory factors analysis confirmed a 65-item model with appropriate fitness of data. Cronbach's alpha coefficient showed an acceptable reliability of the diabetes treatment adherence questionnaire (α = 0.92). Conclusion: The questionnaire developed based on the HPM model provides a standard and comprehensive measurement of the degree of adherence to treatment and the associated factors among Iranian T2D patients. This is especially valuable in the Iranian healthcare context, where effective management of chronic diseases such as diabetes is of a top priority. Questionnaires can help identify barriers and facilitators of treatment adherence to inform systematic and goal-oriented interventions. The proposed questionnaire had good psychometric properties, and can be used as a valid and practical instrument to measure the factors related to treatment adherence behaviors. [ABSTRACT FROM AUTHOR]

Introduction: An increasing population and a shortage of identified potential organ donors are causing the waiting list for organ transplants to grow continuously. Donation after circulatory death (DCD) is a method aimed at meeting the demand for transplantable organs. However, it presents new challenges in nursing care, and there is a lack of studies investigating nurses' attitudes and knowledge of DCD. Objective: The objective of this study was to determine and describe intensive care nurses' (ICNs') knowledge, attitudes, and views on DCD before a national implementation in Sweden. Method: This study utilized a cross-sectional mixed-method design. A convenience sampling method was employed, targeting ICNs working in four intensive care units in Sweden. A study-specific tool comprising fixed and free-text questions was developed. Fifty-one ICNs participated. Data were analyzed descriptively, and correlation analysis was performed using Spearman's correlation. Free-text answers were qualitatively assessed and analyzed. An integrated analysis was conducted to synthesize the quantitative and qualitative findings. Results: Fifty-three percent reported limited knowledge about DCD. Nurses with previous education on DCD had significantly higher knowledge (r =.380, p =.006), were more engaged with the public debate on organ donation (r =.423, p =.002), and considered the ethical aspects of DCD more thoroughly (r =.386, p =.022). The qualitative analysis identified four key categories: The importance of the team, the need for ethical discussions, increased knowledge of DCD, and unanswered questions and unmet needs. The integrated analysis underscored the need for targeted education, clear guidelines, and ongoing ethical discussions to prepare ICU nurses for DCD. Conclusion: Nurses educated in organ donor care had better knowledge and a more positive attitude toward DCD implementation. The study also highlights the importance of interprofessional teamwork in DCD implementation. The findings suggest that education on DCD could improve the identification and implementation of DCD donors, addressing the global shortage of transplantable organs. [ABSTRACT FROM AUTHOR]

Background: While numeric scales to represent pain intensity have been well validated, individuals use various conceptualizations when assigning a number to pain intensity, referred to as pain rating schema. The 18-item Pain Schema Inventory (PSI-18) quantifies pain rating schema by asking for numeric values for multiple mild, moderate or severe pain conditions. This study aimed to assess the validity and reliability of a shortened form of the PSI, using only 6 items (PSI-6). Methods: A secondary analysis was performed on two existing datasets. The first (n = 641) involved a community-based population that completed the PSI-18. The second (n = 182) included participants with chronic pain who completed the PSI-6 twice, one week apart. We assessed face validity, convergent validity, offset biases, test-retest reliability, and internal consistency of the PSI-6 compared to the PSI-18. Results: Both the PSI-18 and PSI-6 demonstrated excellent face validity. The PSI-6 demonstrated excellent convergent validity relative to the PSI-18, with correlations from r = 0.88 to 0.92. Bland-Altman plots revealed offset biases near zero (< 0.22 on 0-10 scale) across all categories of mild, moderate, severe and average pain. Internal consistency was excellent, with Cronbach's Alpha = 0.91 and 0.80, for PSI-18 and PSI-6 respectively. Test-retest reliability of the PSI-6 was high with correlations from r = 0.70-0.76. Conclusion: The PSI-6 is a valid and reliable tool to assess pain rating schema with reduced subject burden, to better interpret individuals' pain ratings and adjust for inter-individual variability. [ABSTRACT FROM AUTHOR]

Background: Promoting patient participation stands as a global priority in nursing care. Currently, there is a lack of a standardized tool to assess the culture of patient participation from the perspective of nurses in China. Aims: To translate and examine the validity and reliability of the Patient Participation Culture Tool for healthcare workers (PaCT-HCW) on general hospital wards in Chinese nursing context. Methods: A cross-sectional research study was conducted among 812 nurses. Brislin's recommendations were adhered to during the translation of the scale. Validity was assessed using construct validity, content validity, and face validity. Split-half reliability, test–retest reliability, and internal consistency reliability were used to evaluate dependability. The study was guided and reported following the STROBE checklist and recommendations for reporting the results of studies of instrument and scale development and testing. Results: The Chinese version of PaCT-HCW (the PaCT-HCW-C) exhibits good face validity and content validity. A rigorous exploratory factor analyse verified a six-factor (competence, support, perceived lack of time, information sharing and dialogue, response to questions and acceptance of a new role) scale structure with a cumulative variance contribution of the factors with 44 items of 68.840%. With a Cronbach's α coefficient of 0.962, split-half reliability of 0.866, and intraclass correlation coefficients of 0.989, the instrument demonstrates great reliability. Confirmatory factor analysis results validated the consistency of the six factors with the structure of the PaCT-HCW-C scale. Conclusions: The 44-item PaCT-HCW-C is a valid and reliable instrument with satisfactory psychometric properties. It could serve as a tool for assessing the effectiveness of international programs aimed at fostering patient participation from the perspective of nurses, while also providing insights from China's practical experiences. [ABSTRACT FROM AUTHOR]

Lifestyle medicine (LM) should be incorporated as part of routine clinical work and medical education programs. Objective: To develop and test the validity and reliability of a questionnaire that measures the level of knowledge, attitude, and practice (KAP) of LM domains among medical trainees through practicing physicians. Methods: The KAP questionnaire sections covered the nine domains of LM. The validation process included face and content validity. A total of 151 individuals from the medical field residing in Saudi Arabia were recruited through a convenient sampling technique to participate in the study. Item response theory (IRT) was applied to validate the knowledge domain, while exploratory factor analysis (EFA) was used to assess attitude and practice. Cronbach's alpha was performed to test the reliability of the three sections. Results: The questionnaire contained 37 items of knowledge, 45 attitudes, and 28 practice items. According to the IRT analysis, 27 items of knowledge were within the acceptable range of difficulty and discrimination. The EFA analysis resulted in 6 factors, including all the items in the attitude domain, and 4 factors, for a total of 27 items in the practice domain, with satisfactory factor loading (>0.4). The Cronbach's alpha for the three domains was very high (≥0.88). Conclusions: The KAP questionnaire for LM is valid and reliable across a spectrum, from medical trainees to practicing physicians. This tool could serve as an instrument to evaluate and develop adequate educational programs for medical doctors. [ABSTRACT FROM AUTHOR]

Background: Nurse interns' capacity for innovative behavior is a key component of healthcare settings because of increasing demands and complexities. Thus, it is important to find strategies that promote their innovative behavior. The development of psychological capital (PsyCap) improves positive behaviors and attitudes, such as engagement, motivation, and satisfaction, in the work environment. Therefore, this study aimed to assess the effect of a PsyCap educational program on nurse interns' innovative behavior. Methods: A quasi-experimental design was used in this study. This study was executed at Fayoum University hospitals. The study participants comprised all the available nurse interns (n = 223) registered in the internship year (2022–2023) in the aforementioned settings. The data were gathered using three instruments, namely, the PsyCap Knowledge Questionnaire, the PsyCap Questionnaire, and the Innovative Behavior Inventory. Results: The nurse interns' mean scores regarding total knowledge about PsyCap, total perception of PsyCap, and total perception of innovative behavior significantly improved through the posttest phase (41.27 ± 9.31, 92.22 ± 6.26, 91.31 ± 9.06, respectively) and the follow-up phase (37.83 ± 8.83, 89.96 ± 6.31, 88.89 ± 8.33) in comparison with the pretest phase (14.39 ± 5.83, 69.04 ± 8.13, 60.55 ± 7.15). Conclusion: The PsyCap educational program was effective and beneficial for improving the nurse interns' perceptions of innovative behavior. Therefore, PsyCap interventions should be implemented in hospitals through professional development programs and orientation programs. [ABSTRACT FROM AUTHOR]

Background Objectives Methods Results and Conclusions In recent years, eye tracking has become a prominent method for learning research as it is assumed to indicate (meta)cognitive processes. However, there is little empirical evidence for hypothesized relations between eye tracking indicators and specific (meta)cognitive processes so that construct validity of used metrics can be questioned.The main goal was to provide validity hints in order to create an empirical basis for interpreting specific eye tracking indicators in terms of respective (meta)cognitive processes of multimedia learning.N = 60 students learned with multimedia instructional material. Referring to a process model of multimedia learning, correlations between theoretically deduced eye tracking indicators with verbalized (meta)cognitive processes of multimedia learning captured by think‐aloud protocols were examined. In addition, the sensitivity of both process measures of (meta)cognitive processes was regarded considering the well‐investigated seductive details effect of an established multimedia instruction in a two‐group design. Finally, serial mediations were calculated in order to investigate whether both process measures complement one another in a joint explanation of the seductive details effect.Eye tracking indicators and verbalized (meta)cognitive processes did only partly correspond as it was shown by correlation and serial mediation analyses. However, both measures were sensitive to indicate the seductive details effect. Thus, even though the study provided insights in how validation could be possible, further systematic research will be needed for validating eye tracking indicators of specific (meta)cognitive processes in multimedia learning. [ABSTRACT FROM AUTHOR]