Search

Regression discontinuity (RD) designs have gained significant popularity as a quasi-experimental device for evaluating education programs and policies. In this paper, we present a comprehensive review of RD designs, focusing on the continuity-based framework, the most widely adopted RD framework. We first review the fundamental aspects of RD designs, drawing on potential outcomes and causal graphs. We then discuss the validity threats in RD designs, including manipulation, discreteness of the running variable, statistical power, and generalizability. Additionally, we provide an overview of the existing extensions to RD designs. To exemplify the application of RD methods, we analyze the effect of New Jersey's pre-kindergarten program on children's vocabulary test scores, using an educational dataset. Finally, we offer practical guidelines in the conclusion to promote the appropriate use of RD methods in educational research.

Growth in the online survey market may be increasing response burden and possibly jeopardizing higher response rates. This meta-analysis evaluated survey trends over one decade (2011-2020) to determine: (1) changes in survey publication rates over time, (2) changes in response rates over time, (3) typical response rates within health sciences education research, (4) the factors influencing survey completion levels, and (5) common gaps in survey methods and outcomes reporting. Study I estimated survey publication trends between 2011 and 2020 using articles published in the top three health sciences education research journals. Study II searched the anatomical sciences education literature across six databases and extracted study/survey features and survey response rates. Time plots and a proportional meta-analysis were performed. Per 2926 research articles, the annual estimated proportion of studies with survey methodologies has remained constant, with no linear trend (p > 0.050) over time (Study I). Study II reported a pooled absolute response rate of 67% (95% CI = 63.9-69.0) across 360 studies ("k"), totaling 115,526 distributed surveys. Despite response rate oscillations over time, no significant linear trend (p = 0.995) was detected. Neither survey length, incentives, sponsorship, nor population type affected absolute response rates (p [greater than or equal to] 0.070). Only 35% (120 of 339) of studies utilizing a Likert scale reported evidence of survey validity. Survey response rates and the prevalence of studies with survey methodologies have remained stable with no linear trends over time. We recommend researchers strive for a typical absolute response rate of 67% or higher and clearly document evidence of survey validity for empirical studies.

Research on anxiety and motivation in second language (L2) learning is proliferating. However, these two individual differences are commonly measured by self-report questionnaires, the psychometric properties of which have been questioned. Building on previous works on study quality, this methodological synthesis systematically describes and evaluates the state of study and scale quality in L2 anxiety and motivation research. A total of 104 peer-reviewed articles (113 independent samples) that used 340 L2 anxiety and motivation scales were coded for 84 features targeting: study, survey, and scale designs; participant demographics; scale validity and reliability; and reporting practices associated with transparency. The results show a number of strengths but also reveal multiple areas in need of methodological enhancement. The article concludes by highlighting areas that merit further investigation in the domain of L2 survey research and making recommendations for future studies.

The Research Methods Skills Assessment (RMSA) was created to measure psychology majors' statistics knowledge and skills. The American Psychological Association's Guidelines for the Undergraduate Major in Psychology (APA, 2007, 2013) served as a framework for development. Results from a Rasch analysis with data from n = 330 undergraduates showed good fit for 16 of the 21 items created. Validity analysis showed that using the RMSA to measure students' statistics knowledge was an improvement over the assessment recommended by the APA Guidelines and was less time consuming. Together the findings provide a preliminary test that can be further developed to provide a tool for instructors and departments to use when assessing psychology students' statistics knowledge and skills. Future research should expand the RMSA to include items that measure knowledge with confidence intervals as the proposed items were removed due to poor fit. Future studies should also aim to replicate the current findings with additional samples of psychology students.

Structural equation modeling (SEM) is one of the most popular multivariate statistical techniques in Turkish educational research. This study elaborates the SEM procedures employed by 75 educational research articles which were published from 2010 to 2015 in Turkey. After documenting and coding 75 academic papers, categorical frequencies and percentages were calculated in the content analysis. Methodological issues related to use of SEM were identified. The results showed that usage and reporting SEM in Turkish educational research articles were mostly problematic. Content analysis revealed that educational researchers tend to avoid assumption testing of SEM and reporting estimation methods. General guidelines and suggestions on the use of SEM were provided.

Comparative accuracy studies evaluate the relative performance of two or more diagnostic tests. As any other form of research, such studies should be reported in an informative manner, to allow replication and to be useful for decision-making. In this study we aimed to assess whether and how components of test comparisons were reported in comparative accuracy studies. We evaluated 100 comparative accuracy studies, published in 2015, 2016 or 2017, randomly sampled from 238 comparative accuracy systematic reviews. We extracted information on 20 reporting items, pertaining to the identification of the test comparison, its validity, and the actual results of the comparison. About a third of the studies (n = 36) did not report the comparison as a study objective or hypothesis. Although most studies (n = 86) reported how participants had been allocated to index tests, we could often not evaluate whether test interpreters had been blinded to the results of other index tests (n = 40; among 59 applicable studies), nor could we identify the sequence of index tests (n = 52; among 90 applicable studies) or the methods for comparing measures of accuracy (n = 59). Two-by-four table data (revealing the agreement between index tests) were only reported by 9 of 90 paired comparative studies. More than half of the studies (n = 64) did not provide measures of statistical imprecision for comparative accuracy. Our findings suggest that components of test comparisons are frequently missing or incompletely described in comparative accuracy studies included in systematic reviews. Explicit guidance for reporting comparative accuracy studies may facilitate the production of full and informative study reports.

The main aim of this study is to determine whether a significant difference exits among data collection with printed materials and with smart phones. The research was conducted with 282 teacher candidates who take pedagogical formation training. Three data collection tools were used throughout the study. As the results of the current research, no significant differences were obtained among attitude scores towards teaching profession with regard to common effect of application method and faculty variables. However, significant differences in participant opinions were determined among different application methods (smartphone and printed) of Metacognitive Thinking Skills and Information Literacy Scales. In addition, no significant difference was found among opinions towards Metacognitive Thinking Skills and Information Literacy Scales with regard to common effect of application method and faculty variables. Based on these research findings, it is not recommended to distribute survey items containing technical information or some complex expressions via smartphones. Relations among the questionnaire content and the department of education confirm this situation.

One of the major successes of advanced quantitative methods has been its seeming ability to provide unbiased determinations of which education practices are effective for education in general and for improving the educational achievement and opportunity of the neediest students. The power of this methodology as applied in the top education research journals has led to periodic implementation of federal and state "effective practices policies". In such policies the government or its proxy determines which programs are effective and then requires or encourages schools to spend its funds exclusively on those proven programs. For example, the federal Investing in Innovation (i3) initiative requires those applying for its largest dissemination grants to have had their intervention validated as effective by the What Works Clearinghouse (WWC). Some are now even advocating that all expenditures of federal funds by school districts be restricted to purchasing programs that research has proven to be effective, i.e., that they work. While this seems like rational policy on the surface, between 1998 and 2002 I produced a series of articles that showed that the most research validated program to develop the reading skills of students born into poverty was not actually effective in practice. Was this dichotomy between published research proving something to be effective and what was happening in the real world an anomaly or a more widespread problem? This article (a) analyzes in easy-to-understand language the validity of the gold standard scientific methodology used by the top research journals and WWC to determine whether practices are effective, and (b) examines the history of "effective practices policies" and their actual effectiveness. I conclude that the increasingly sophisticated methods used to assess the effectiveness of practices (a) are flawed and exaggerate actual effectiveness, and (b) do not provide the type of information practitioners need. As a result, research on effective practices tends to mislead rather than inform practice and are a major reason why efforts to reform high-poverty schools have had limited success. I therefore conclude that "effective practices policies" should not be implemented. I then suggest ideas for reforming the scientific process used to assess the effectiveness of education interventions.

Studies based on longitudinal data focus on the change and development of the situation being investigated and allow for examining cases regarding education, individual development, cultural change, and socioeconomic improvement in time. However, as these studies require taking repeated measures in different time periods, they may include various error sources that are exclusive of the investigated variable and that change over time. In Addition, the period of observation is critical in these studies, and observant (variable) attrition or the inability to obtain measurements for various reasons (death, moving away, resignation etc.) may be observed. This study presents the coefficient of individual agreement (CIA) developed for studies in which more than one rater are needed to provide repeated scoring and examines the applications of this agreement coefficient in relation to different sample size, missing value and number of repeated measures. In this context, trial scores were generated based on different sample sizes, missing value ratios, and the number of repeated measures provided by two raters, and the expected agreement between raters was simulated as very high. Related data were replicated 1000 times. Data analysis included the mean and standard deviation of agreement ratios calculated for each combination. According to the results, independently from sample size and number of retests, the CIA takes much lower values than the expected value when random missing value rates reaches 20%. Similar conclusions can be drawn for the existence of monotone missing values except where the sample size is 30 and the number of retests are two and three. It was observed that the CIA was higher than the expected value for the data consisting of two repeated evaluations of the raters with 5% or 10% missing values when the sample size was 30. In general, the agreement coefficient did not exceed the expected level in the situation of three and four replications of measurements. As a result, it can be concluded that an increase in the replication number of measurement decreased the obtained agreement values. The obtained data showed that missing value ratio caused differentiations in the CIA. As a result, increases in missing value ratio cause inconsistencies in CIA. Therefore, it is suggested that data obtained from studies be interpreted considering missing value ratio.

The crisis caused by the COVID-19 virus has far-reaching effects in the field of education, as schools were closed in March 2020 in many countries around the world. In this article, we present and discuss the School Barometer, a fast survey (in terms of reaction time, time to answer and dissemination time) that was conducted in Germany, Austria and Switzerland during the early weeks of the school lockdown to assess and evaluate the current school situation caused by COVID-19. Later, the School Barometer was extended to an international survey, and some countries conducted the survey in their own languages. In Germany, Austria and Switzerland, 7116 persons participated in the German language version: 2222 parents, 2152 students, 1949 school staff, 655 school leaders, 58 school authority and 80 members of the school support system. The aim was to gather, analyse and present data in an exploratory way to inform policy, practice and further research. In this article, we present some exemplary first results and possible implications for policy, practice and research. Furthermore, we reflect on the strengths and limitations of the School Barometer and fast surveys as well as the methodological options for data collection and analysis when using a short monitoring survey approach. Specifically, we discuss the methodological challenges associated with survey data of this kind, including challenges related to hypothesis testing, the testing of causal effects and approaches to ensure reliability and validity. By doing this, we reflect on issues of assessment, evaluation and accountability in times of crisis.

Borrowing from a clinical psychology observational methodology, thin-slice observations were used to assess autism characteristics in toddlers. Thin-slices are short observations taken from a longer behavior stream which are assigned ratings by multiple raters using a 5-point scale. The raters' observations are averaged together to assign a "thin-slice" value for each observation. In this study, a total of 60 toddlers were selected from a video archive: 20 children with typical development, 20 children with developmental language disorder, and 20 children with autism. In the first part of this study, 20 raters observed small play segments between toddlers and an assessor. Raters assigned scores to each of the 60 toddlers on items related to autism symptomatology. Item analysis and generalizability and decision studies were conducted to determine the factor structure and optimal number of raters to achieve a stable estimate of autism characteristics. In the second part of the study, generalizability and decision studies were conducted to determine the most efficient and optimal combination of raters and naturalistic contexts. This pilot study provides recommendations for optimizing the utility of thin-slice observations for measuring autism symptomatology in young children.

In this paper, the writer discusses the philosophical underpinnings of the two dominant research methods in social sciences; quantitative and qualitative paradigms. The natures of two paradigms are quite different so this leads many researchers to discuss these issues in a comparative way. This paper tackles the knowledge and understanding of quantitative and qualitative paradigms, their nature, advantages, disadvantages, assumptions, criticisms, and the possibility of linking two paradigms. Issues of validity, reliability, objectivity, and generalisation are also investigated from two distinctive perspectives of paradigms. Finally, the conclusion and suggestions are given at the end of the paper. The novice researchers who are in dilemmas about making decisions about their research designs would potentially find the discussions useful.

The recent "Nationwide academic achievement and study situation survey" was clearly influenced by the idea of "authentic assessment", an educational assessment perspective focused on "quality" and "engagement". However, when "performance assessment", the assessment method corresponding to this focus, is adopted in academic achievement surveys, it runs the risk of turning into a rigid hollow structure. In this paper I will reflect on the ideal application of performance assessment in academic achievement surveys, and will investigate the concepts of "consequential validity", "equity", and "moderation" in regard to their potential to further develop the discourse. (Contains 1 note.)

The criterion-related validity of divergent thinking (DT) tests has been questioned because of relatively low correlations between DT test scores and creative performance. A possible inhibiting factor in previous correlations--heterogeneity of the sample--has been ignored. If groups with different DT profiles are involved in this line of research, low correlations would not necessarily be indicative of actual relationships, and criterion-related validity would appear to be lower than it is. Thus, this study identified groups with their own unique DT profiles and examined their creative performance. To this end, elementary children's (N = 325) responses in 3 DT tasks were analyzed using latent profile analysis. Results suggested 3 groups with unique DT profiles: One group scored moderate on fluency and low on originality; another group scored moderate on fluency and high on originality; the third group scored uniformly low across fluency, originality, and flexibility. Surprisingly, only the low-originality group excelled in creative achievement, but the hypothesis of this investigation was supported by the fact that only the high-originality group significantly excelled in ideation. These results confirm that low correlations between DT indices and creative performance do not necessarily indicate a lack of the criterion-related validity of DT tests. Differences between the creative achievement and the ideation criterion measures are detailed, and idiosyncratic properties in DT profiles and their corresponding creative strengths are discussed as educational implications.

This systematic review provides a summary of studies on teaching, social, and cognitive presences in the community of inquiry (CoI) model using the CoI instrument in a higher education setting since its development in 2008 by Arbaugh et al. A total of 24 peer-reviewed studies on the CoI instrument from 2008-2017 were selected and analyzed explore the types of research on the instrument in higher education settings have been published, ways the instrument has been used for learning and teaching, and the implications the instrument poses for online instructors and instructional designers. It is organized into 4 themes: testing the instrument for validity and reliability, measuring CoI presence in different environments, examining causal relationships among the elements, and exploring potential revisions to the model.

Developmental assessment scales are important tools for determining developmental delays and planning preventive interventions. One broad assessment scale used to evaluate child development is the Battelle Developmental Inventories (BDIs). The BDI-2 has a standardized version in English with good psychometric properties and a translated version in Spanish; however, despite widespread clinical use, there has been limited early childhood development research focused on this assessment tool in the past 10 years. The purpose of this systematic review was to evaluate the literature about the BDIs and their screening test and analyze their use for child assessment. Seven databases were used to retrieve articles in English, Portuguese, and Spanish. Overall, 34 articles were evaluated for general features, salient findings, and key methodological issues, such as target population, objectives, research design, and main results. Authors' considerations about the psychometric properties and utility of the BDIs were also examined. Results indicated that 41.17% of studies with the BDIs were from the United States, and the most common target population was children with autism spectrum disorders. Generally, the articles highlighted the broad applications of this measure and robust psychometric properties cited in the BDIs examiner's manual as reasons for their use. This review suggests that it is important to conduct independent analyses of the psychometric properties of the BDIs as well as validation studies to ensure appropriate applications of the BDI, including for use with non-American populations.

The computer-supported Project Work classroom learning environment discussed in this paper represents a paradigm shift from teacher-centered to student-centered teaching and learning in Singapore schools. Besides the face-to-face weekly lessons in existing Project Work classrooms, the students engaged in computer-supported online forum discussions. Two hundred and sixty students and 26 teachers from seven high schools participated in this study. Their perceptions of this new learning environment were assessed using a modified version of the Web-based Learning Environment Instrument (WEBLEI). In addition, the instrument was also validated. The WEBLEI exhibited satisfactory internal consistency reliability and discriminant validity. Comparing the actual and preferred perceptions of the students and teachers revealed that there were differences between the actual and preferred perceptions for both students and teachers. With these in mind, the implications for using technology to support the face-to-face teaching and learning in Project Work classrooms are discussed in this paper. (Contains 2 tables and 1 figure.)

The current set of three studies further evaluates the validity and application of the Psychological Research Inventory of Concepts (PRIC). In Study 1, we administered the PRIC to a sample of introductory psychology students and online (Mechanical Turk) participants along with measures assessing theoretically related concepts. We found evidence of concurrent validity by demonstrating that PRIC scores relate to education, cognitive effort, and general psychological knowledge. In Study 2, we demonstrated that advanced psychology majors score higher on the PRIC than college graduates who did not major in psychology, suggesting that the PRIC assesses domain-specific knowledge and may be useful in assessment of the undergraduate psychology major. Finally, in Study 3, we demonstrated that scores on the PRIC increase from the start to the end of a research methods course. Together, these studies provide further evidence that the PRIC may be a useful index of learning research methods and statistical knowledge important for the undergraduate psychology degree.

The report of the national response to intervention (RTI) evaluation study, conducted during 2011-2012, was released in November 2015. Anyone who has read the lengthy report can attest to its complexity and the design used in the study. Both these factors can influence the interpretation of the results from this evaluation. In this commentary, we (a) explain what the national RTI evaluation examined and highlight the strengths and weaknesses of the design, (b) clarify the results of the evaluation and highlight some key implementation issues, (c) describe how rigorous efficacy trials on reading interventions can supplement several issues left unanswered by the national evaluation, and (d) discuss implications for future research and practice based on the findings of the national evaluation and reading intervention research.

This article offers a rationale for the contributions of qualitative research to evidence-based practice in special education. In it, I make the argument that qualitative research encompasses the ability to study significant problems of practice, engage with practitioners in the conduct of research studies, learn and change processes during a study, and provide expansive data sets that help clarify both independent and dependent variables. Qualitative methodologies can shape and advance important questions of educational practice and policy. The concern with the degree to which qualitative research can contribute to the research base may be a reflection of the degree to which our field as a whole adequately prepares its researchers to understand and engage in high-quality qualitative research that adheres to benchmarks for internal and external validity from inception through dissemination of results.

Fuzzy-set qualitative comparative analysis (fsQCA) has become one of the most prominent methods in the social sciences for capturing causal complexity, especially for scholars with small- and medium-"N" data sets. This research note explores two key assumptions in fsQCA's methodology for testing for necessary and sufficient conditions--the cumulation assumption and the triangular data assumption--and argues that, in combination, they produce a form of aggregation bias that has not been recognized in the fsQCA literature. It also offers a straightforward test to help researchers answer the question of whether their findings are plausibly the result of aggregation bias.

The 1961 Copyright Office study on renewals, authored by Barbara Ringer, has cast an outsized influence on discussions of the U.S. 1923-1963 public domain. As more concrete data emerge from initiatives such as the large-scale determination process in the Copyright Review Management System (CRMS) project, questions are raised about the reliability or meaning of the Ringer data. A closer examination of both the Ringer study and CRMS data demonstrates fundamental misunderstandings and misrepresentations of the Ringer data, as well as possible methodological issues. Estimates of the size of the corpus of public domain books published in the United States from 1923 through 1963 have been inflated by problematic assumptions, and we should be able to correct mistaken conclusions with reasonable effort.

Although peer nomination measures have been used by researchers for nearly a century, common methodological practices and rules of thumb (e.g., which variables to measure; use of limited vs. unlimited nomination methods) have continued to develop in recent decades. At the same time, other key aspects of the basic nomination procedure (e.g., whether nonparticipants should be included as nominees, the consequences of pairing code numbers with names on rosters) are underdiscussed and understudied. Beyond providing a general introduction to peer nomination methods and their utility, the current article discusses the advantages and disadvantages of various methodological choices facing researchers who wish to use peer nomination methods, in addition to other considerations that researchers must make in collecting peer nomination data (e.g., establishing reliability and validity, maximizing participation rates, computerized assessments). This article provides recommendations for researchers based on empirical findings (where possible) and the typical practices used in the recent published literature.

Cognitive interviewing (CI) is a method to identify sources of confusion in assessment items and to assess validity evidence on the basis of content and response processes. We introduce readers to CI and describe a process for conducting such interviews and analyzing the results. Recommendations for best practice are provided.

In this commentary, we offer a set of visual tools that can assist education researchers, especially those in the field of mathematics, in developing cohesiveness from a mixed methods perspective, commencing at a study's research questions and literature review, through its data collection and analysis, and finally to its results. This expounds upon the ideas and methodological insights relating to research connections and structure presented by other scholars (e.g., Creswell, 2009; Heid & Blume, 2011; Mertens, 2010; Thanheiser, Ellis, & Herbel-Eisenmann, 2012). As faculty who work with graduate students in education, we have found that these visual tools help students elicit and maintain salient connections throughout the research process.

Using as a starting point in the recent work of Mountford-Zimdars et al., the authors analyse attitudes towards expanding higher education (HE) opportunities in the UK. The authors propose that the approach of Mountford-Zimdars et al. is flawed not only in its adoption of a multivariate logistic regression but also in its interpretation of results. The authors make a number of adaptations, chief among them being the use of an ordered probit approach and the addition of a time dimension to test for changes in attitudes between 2000 and 2010. The authors find that attitudes towards HE expansion have intensified during the decade 2000-2010, but the authors uncover no evidence that this is due to graduates wanting to "pull up the ladder", as suggested by Mountford-Zimdars et al. The authors argue that evidence of a widespread desire to reduce access to HE can most likely be explained by social congestion theory, internal institutional disaffection and rising tuition fees.

When classifying the evidence base of practices, special education scholars typically appraise study quality to identify and exclude from consideration in their reviews unacceptable-quality studies that are likely biased and might bias review findings if included. However, study quality appraisals used in the process of identifying evidence-based practices for students with learning and other disabilities have not been empirically validated (e.g., studies classified as unacceptable quality shown to have different, and presumably more biased, effects than high-quality studies). Using Gersten et al.'s (2005) approach for appraising the quality of group experimental studies in special education, we examined whether (a) studies classified as unacceptable quality and high quality had meaningfully different effects and (b) unacceptable-quality studies were more likely to have outlying effects than high-quality studies among 36 group experimental studies that investigated the effectiveness of instructional practices for students with learning disabilities. Our preliminary analyses found that the effects of unacceptable-quality studies were not meaningfully different from the effects of high-quality studies. We discuss implications of these findings and call for more research to be conducted in this area.

Critically examining common statistical approaches and their strengths and weaknesses is an important step in advancing recreation and leisure sciences. To continue this critical examination and to inform methodological decision making, this study compared three approaches to determine how alternative approaches may result in contradictory conclusions in the interpretation of the psychometric properties of a scale and in response to a given research question. To this end, this study explored what factors best predicted parental endorsement of competition climbing in a sample of 184 parents of youth competition climbers. The study findings suggest that the three distinct approaches provided meaningfully different conclusions regarding the adapted psychometric properties of the questionnaire, but offered no meaningful differences in the primary finding of the study: Parent-coach relationship quality is the best predictor of parental endorsement of competition climbing. The results suggest that deeper examination of self-report questionnaire data may advance our understanding of complex recreation and leisure constructs beyond what can be understood with less advanced analytic techniques.