Search

There exist passionate debates about the best way to teach children to read. Since the Bush administration, school districts receiving federal funds have been required to have research evidence justifying their methods of teaching reading, and in recent years the need for evidence-based practices have intensified with the "Science of Reading" movement. Last year the country's largest school system changed from balanced reading to three other programs for teaching children to read. In this study we examined the empirical evidence regarding the effectiveness of the three newly adopted reading programs for the country's largest school system. Our study consisted of website review of the research evidence provided by the three publishing companies regarding the effectiveness of their reading programs. Borrowing from studies of program effectiveness, we applied three criteria for analyzing the quality of research used by the publishers to document their program effectiveness. These criteria were the following: (1) Comparative clinical trials; (2) replication of results and (3) publication of evidence in professional journals. The results of our analyses indicated that only one of the selected reading programs was supported with empirical evidence meeting our criteria regarding program effectiveness. We argue that teachers, not programs, are what best affects children's learning to read. Overly simplified and politicized methods of teaching reading lacking empirical evidence of effectiveness are unlikely to accomplish this important goal.

Higher education professionals often are tasked with providing evidence to stakeholders that programs, services, and practices implemented on their campuses contribute to student success. Furthermore, in the absence of a solid base of evidence related to effective practices, higher education researchers and practitioners are left questioning what works--and what does not--on college campuses. The goal of this essay is to advance a conversation about evidence within the higher education literature through an illustration of a nonrandom research study designed to evaluate the impact of a national interfaith organization on institutions' efforts to promote interfaith learning and development. We briefly provide an overview of the relationship between evidence and evaluation efforts, then move to illustrating one project as a vehicle for understanding how evidence emerging from a nonrandom evaluation design can be done effectively in a national study related to one dimension of student success, and finally, offer support for the essay's central thesis--"that evaluation designs can present reliable evidence of impact in the absence of randomization."

High quality student outcome measures provide the foundation for effective learning assessment. These outcomes can be organized into different categories, such as skills or habits of mind, and evidence types, such as direct or indirect. Prevailing models of assessment, however, focus only on the distinction between direct and indirect evidence without incorporating the outcome categories. Such a binary model of evidence can be limiting, privileges measures of skills and knowledge over habits of mind and values, and risks excluding validated psychometric tools as a source of measurement. An argument is made for a new 2x2 outcome matrix which incorporates outcome category and evidence type as a possible alternative to the dominant direct vs. indirect dichotomy.

Scientific argumentation skills are a key component of science learning practices needed by students in the 21st century. Where the essence of scientific argumentation is to support the argument with evidence and reasoning and then refute the claims and evidence of the opponent's argument. Supporting arguments with evidence and reasoning can be facilitated through inquiry activities. Meanwhile, refuting claims and evidence from opposing arguments can be facilitated through debate activities. Inquiry and debate can be a solution to improving students' scientific argumentation skills. This study aims to prove the truth of this statement through a systematic literature review by applying the PRISMA 2020 criteria which are expanded by bibliometric analysis with the help of the VOSviewer software. The results of the study show that research related to inquiry, debate, and their influence on students' scientific argumentation skills has a pretty good trend. In Indonesia the research climate is still quite good, research topics are still in great demand, and it is very possible to carry out research innovations on related topics. Another finding is that the integration of inquiry and debate in science learning can be a potential strategy to improve students' scientific argumentation skills.

A recent issue of the Australian Journal of Language and Literacy included an article reporting on a systematic narrative review of the research literature that indicated that there was insufficient evidence to conclude whether genre theory and systemic functional linguistics either 'worked' or 'did not work'. The criteria used to evaluate these studies excluded any study that did not conform to the 'gold standard' associated with experimental research such as randomised controlled trials. In response to this provocative finding, a group of SFL researchers decided to examine just what counts as evidence of quality literacy research these days. In this paper, we question the overreliance on experimental research at the expense of other methods. We illustrate this with a sample of notable studies that do not meet experimental criteria, but which nevertheless have made a significant contribution to school literacy outcomes in Australia and elsewhere.

Increasing concerns about the trustworthiness of research have prompted calls to scrutinise studies' Individual Participant Data (IPD), but guidance on how to do this was lacking. To address this, we developed the IPD Integrity Tool to screen randomised controlled trials (RCTs) for integrity issues. Development of the tool involved a literature review, consultation with an expert advisory group, piloting on two IPD meta-analyses (including 73 trials with IPD), preliminary validation on 13 datasets with and without known integrity issues, and evaluation to inform iterative refinements. The IPD Integrity Tool comprises 31 items (13 study-level, 18 IPD-specific). IPD-specific items are automated where possible, and are grouped into eight domains, including unusual data patterns, baseline characteristics, correlations, date violations, patterns of allocation, internal and external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to decision rules, and justification for each rating is recorded. Overall, the tool guides decision-making by determining whether a trial has no concerns, some concerns requiring further information, or major concerns warranting exclusion from evidence synthesis or publication. In our preliminary validation checks, the tool accurately identified all five studies with known integrity issues. The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The tool may be applied by evidence synthesists, editors and others to determine whether an RCT should be considered sufficiently trustworthy to contribute to the evidence base that informs policy and practice.

Systematic reviews of single-case experimental research (SCER) in special education often use the What Works Clearinghouse (WWC) Standards to assess the methodological rigor of studies within a given literature base. While significant changes were made between the two most recent versions of the WWC standards, no research to date has evaluated the extent to which these standards would result in different evaluation outcomes. To examine potential differences, we applied version 4.1 and 5.0 of the standards to a sample database of SCER addressing the challenging behavior of students with autism in general education settings. Systematic search, screening, and review procedures resulted in a total of 20 articles included for this review. Findings indicated a 27% change in overall ratings across the studies, with version 5.0 leading to lower ratings than version 4.1, due to factors such as a lack of assessment of procedural fidelity, therapeutic baseline trends, and insufficient data points in the first baseline phase. We discuss implications for research based on our findings and recommend future directions for SCER in the field of special education.

Systematic reviews and meta-analyses aggregate research findings across studies and populations, making them a valuable form of research evidence. Over the past decade, studies in medical education using these methods have increased by 630%. However, many manuscripts are not publication-ready due to inadequate planning and insufficient analyses. These guidelines aim to improve the clarity and comprehensiveness of reporting methodologies and outcomes, ensuring high quality and comparability. They align with existing standards like PRISMA, providing examples and best practices. Adhering to these guidelines is crucial for publication consideration in "Anatomical Sciences Education."

Developing students' source-based argument writing skills is a vital educational goal for the 21st-century information society. Consequently, researchers and educators continually seek ways to understand and improve students' capacities for advancing arguments and synthesizing multiple documents, texts, or sources in a range of subject areas in secondary schools. This study examined differences between middle and high school students' argument essays (N = 207) in multiple dimensions of source-based argument writing in history, the dimensions writing in history, and the relations of identified dimensions to overall writing quality. Using multivariate analysis of covariance, middle and high school students' writing significantly varied in areas of writing related to language use, the presentation of ideas, and evidence use. Their writing varied less so for skills related to historical thinking, indicating a lack of development in these skills across secondary school. Findings from confirmatory factor analysis and structural equation modeling showed a bifactor model with a general factor and 4 specific factors--Presentation of Ideas, Evidence Use, Language Use, and Historical Thinking--best represented writing in this genre, with the general factor strongly predicting holistic writing scores. Implications for both research and educational practice are discussed, including the importance of attending to developmental variation in discrete writing skills.

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

The present study contributes to the development of effective measures to evaluate classroom climate, especially in elementary education where these remain limited. In addition, it addresses a typical "flaw" of several studies by approaching classroom climate as a group-level construct rather than an individual characteristic. Following a multilevel approach, we attempted to establish validity evidence for an adapted version of the Classroom Behavioral Climate (CBC) questionnaire and examine its equivalence across two contexts. The participants included 2963 students from 176 fourth and fifth grade classes. Multilevel Confirmatory Factor Analyses resulted in a shorter version of the CBC with three correlated factors in both countries. Two-level models performed better than the single-level models. Configural, metric, and partial scalar invariance was also achieved. The study highlights the need for researchers and practitioners to use appropriate methods and instruments to evaluate the classroom environment based on their research aims and intended use.

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing--remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Quinacridone red and violet are visually different colors, an observation confirmed by their visible reflectance spectra and CIE L*a*b* coordinates. However, their IR spectra are extremely similar. Though chemically identical, the two quinacridones are polymorphs. In this experiment, designed for and tested by nonscience majors, the pigments are extracted from oil paints with hexane in a short procedure involving sonification and centrifugation. ATR-FTIR spectra of the recovered brilliantly colored powders reveal the presence of a calcium carbonate filler in one of the two paints. Factoring out the carbonate bands, the spectra are nearly identical. Students must reconcile the evidence of color and the evidence of the spectra.

This paper explored the link between information literacy (IL) and other factors that enable or inhibit the utilisation of research evidence in policymaking in Zimbabwe. The study assumes that if policymakers possess appropriate IL skills to access, assess, synthesise, and apply research evidence, they will naturally use the evidence to inform their policy decisions. Face-to-face interviews with 26 policymakers -- technocrats selected from the Parliament of Zimbabwe and two ministries, Industry and Commerce, and Youth, Sport, and Recreation -- produced evidence to inform the findings and conclusions. Data synthesis using thematic content analysis confirmed the findings. The results show that while IL skills are critical in enabling policymakers' use of research evidence, multiple other factors also influence the use of research evidence in policymaking due to the complexity of the process. The political and socioeconomic context plays a profound role because of the intricate and nonlinear nature of the policymaking process. Therefore, enhancing evidence use in policymaking revolves around strengthening IL skills at the individual level, including institutional and the broader policy ecosystem, by acknowledging and leveraging personal and institutional relationships. This insight illuminates the need to reorient IL programmes to link them to these other factors.

Qualitative comparative analysis (QCA) is a hybrid method designed to bridge the gap between qualitative and quantitative research in a case-sensitive approach that considers each case holistically as a complex configuration of conditions and outcomes. QCA allows for multiple conjunctural causation, implying that it is often a combination of conditions that produces an outcome, that multiple pathways may lead to the same outcome, and that in different contexts, the same condition may have a different impact on the outcome. This approach to complexity allows QCA to provide a practical understanding for complex, real-world situations, and the context of implementing interventions. There are guides for conducting QCA in primary research and quantitative systematic reviews yet, to our knowledge, no guidance for conducting QCA in systematic mixed studies reviews (SMSRs). Thus, the specific objectives of this paper are to (1) describe a step-by-step approach for novice researchers for using QCA to integrate qualitative and quantitative evidence, including guidance on how to use software; (2) highlight specific challenges; (3) propose potential solutions from a worked example; and (4) provide recommendations for reporting.

This scoping review aims to identify and systematically review published mapping reviews to assess their commonality and heterogeneity and determine whether additional efforts should be made to standardise methodology and reporting. The following databases were searched; Ovid MEDLINE, Embase, CINAHL, PsycINFO, Campbell collaboration database, Social Science Abstracts, Library and Information Science Abstracts (LISA). Following a pilot-test on a random sample of 20 citations included within title and abstracts, two team members independently completed all screening. Ten articles were piloted at full-text screening, and then each citation was reviewed independently by two team members. Discrepancies at both stages were resolved through discussion. Following a pilot-test on a random sample of five relevant full-text articles, one team member abstracted all the relevant data. Uncertainties in the data abstraction were resolved by another team member. A total of 335 articles were eligible for this scoping review and subsequently included. There was an increasing growth in the number of published mapping reviews over the years from 5 in 2010 to 73 in 2021. Moreover, there was a significant variability in reporting the included mapping reviews including their research question, priori protocol, methodology, data synthesis and reporting. This work has further highlighted the gaps in evidence synthesis methodologies. Further guidance developed by evidence synthesis organisations, such as JBI and Campbell, has the potential to clarify challenges experienced by researchers, given the magnitude of mapping reviews published every year.

Prediction-based accounts of language acquisition have the potential to explain several different effects in child language acquisition and adult language processing. However, evidence regarding the developmental predictions of such accounts is mixed. Here, we consider several predictions of these accounts in two large-scale developmental studies of syntactic priming of the English dative alternation. Study 1 was a cross-sectional study (N = 140) of children aged 3-9 years, in which we found strong evidence of abstract priming and the lexical boost, but little evidence that either effect was moderated by age. We found weak evidence for a prime surprisal effect; however, exploratory analyses revealed a protracted developmental trajectory for verb-structure biases, providing an explanation as for why prime surprisal effects are more elusive in developmental populations. In a longitudinal study (N = 102) of children in tightly controlled age bands at 42, 48, and 54 months, we found priming effects emerged on trials with verb overlap early but did not observe clear evidence of priming on trials without verb overlap until 54 months. There was no evidence of a prime surprisal effect at any time point and none of the effects were moderated by age. The results relating to the emergence of the abstract priming and lexical boost effects are consistent with prediction-based models, while the absence of age-related effects appears to reflect the structure-specific challenges the dative presents to English-acquiring children. Overall, our complex pattern of findings demonstrates the value of developmental data sets in testing psycholinguistic theory.

This study provides a comprehensive overview of methodological aspects when using heart rate variability (HRV) measures in educational research. Following PRISMA 2020 guidelines, we searched four databases for relevant studies published until March 5, 2024. From the 48 studies reviewed, we extracted data across three analytical categories: (1) area of study interest and participant populations, (2) data collection and analysis methodologies, and (3) the concurrent and predictive validity of HRV measurement for educational research. Study quality was evaluated using QualSyst assessment criteria. Most studies measured stress and enlisted undergraduate students as participants. Data were predominately collected using wearable devices, measuring HRV for durations of less than 30 min, and in varied contexts, including during exams, while learning, and in experiments. The parameters analyzed varied within both time and frequency domains. HRV data had a moderate level of concurrent validity as a measure of stress in an educational context. The concurrent validity of HRV data for measuring attention remains uncertain with insufficient evidence. Limited correlations appeared between stress and performance. The findings, potentials, and limitations of HRV measures are discussed, and synthesized recommendations for educational research using HRV data are provided.

McKinney (2018) has argued that for the scholarship of teaching and learning (SoTL) to advance within a discipline, the integration of SoTL must be closely examined and opportunities for growth in SoTL must be recognized and discussed. To that end, this paper reflects on the degree to which SoTL is integrated into communication sciences and disorders (CSD) by examining a variety of topics: perspectives and theories historically valued by our discipline, existing supports for SoTL at various levels (i.e., individual teacher-scholars, departments, institutions, and the CSD discipline as a whole), the application of SoTL findings in teaching and learning. Four specific recommendations are made because of this examination and reflection.

Research has previously examined the role of the epistemological background of students in accepting or rejecting the theory of evolution. In this study, the participants were 11 undergraduate biology major students in a private university in Beirut, Lebanon. We collected data through semi-structured interviews, and we extracted the patterns regarding how they perceived tentativeness, what constituted acceptable evidence for them to accept the theory of evolution and the role of religion in their lives. Our findings indicated that, unlike their professor, students uncertain about the theory or rejecting it did so due to the tentative nature of science. They also rejected historical evidence as valid evidence in science, struggled to form a cooperative relationship between evolution and religion and refused common descent because they considered humans as superior organisms. We discuss the implications of those results for curriculum and instruction.

In current educational research communications, especially in English and increasingly also in German publications, the term "evidence" refers to international homogenizing gold standards and is often linked to European evidence policies, large scale assessments and justified and proven knowledge. Against the background of international communication and the related circulation of terms and concepts, this paper analyses German and English OECD publications that recommend the development of educational research in 1970s and 90s. In England, the OECD and thus the external perception of their own education system hardly played a significant role due to years of awareness of the need for reform. Rather, Hargreaves's lecture (1996) about the disappointing effects of educational research when compared with the achievements of evidence-based medicine was decisive for the evidence movement. In addition, non-university institutions have gained legitimacy through acting as "evidence" providers for the school system. Based on analyses of German educational research literature, we show that "evidence" appeared in neurological, medical, technological, and economic texts in the 90s and early 2000s. Usage of "evidence" increased after the PISA shock in 2000 and is now linked to the expression of disciplinary development into empirical educational research. However, based on relatively stable patterns of communication and interpretation in two academic cultures, it is shown that the connectivity of an epistemological term like "evidence" does not necessarily reduce misunderstanding.

This article provides a conceptual framework for considering both the theoretical and methodological factors that underpin the successful validation of a competency framework. Drawing on educational assessment literature, this article argues that a valid competency framework relates to an interpretive judgement of the credibility of the claims made. To establish a credible approach to competency framework validation, there is a requirement to align the purposes of the competency framework, the claims developers make concerning the uses of the framework, and evidence collection methods to substantiate or challenge these claims. This article concludes with a template of questions for competency framework developers to consider in determining the range of potential claims to be made concerning their framework, and in understanding competency framework users and contexts.

Background: Local governments increasingly rely on sales taxes to raise revenue, often justifying the need for a local sales tax increase with a specific programmatic goal, such as better education or transportation. In Washington State, the legislature explained that a local sales tax increase was necessary to support criminal justice because criminal justice requires more police, courts, and jails. Objective: Informed by decades of literature questioning the efficacy of fighting crime through police, courts, and jails, the objective of this study seeks to use empirical evidence to explain whether the social fabric offers indicators that can better define criminal justice and thus inform local tax policy so that local sales tax revenue may be used to prevent crime rather than fight it. Method: The study compiled a 29-year history of 36 social variables across all 39 Washington counties to determine whether crime is predictable and what is most predictive of crime. Pearson coefficients of determination were calculated to identify cross-sectional associations between social variables and crime variables. An ARIMAX predictive model was then constructed to test the predictive power of the multivariate time series. Results: The study finds that crime is predictable and social observations specific to how a child grows up are consistently predictive explanatory variables of crime. There is reason to believe that the most effective action state and local governments can take to promote criminal justice and prevent crime is to leverage their taxing power to ensure that every child (a) has access to food and basic necessities, (b) is raised in a safe and stable home, and (c) graduates from college.

This paper describes a student-centered approach to teaching and learning physics in a high school classroom. The teacher designs and uses a 5E learning cycle to teach series and parallel electric circuits. This article provides an example of the implementation of 5E model for a physics lesson to address three-dimensional learning-science and engineering practices, disciplinary core ideas, and crosscutting concepts. In this lesson, students could make predictions, exchange ideas, and collect and analyze data to construct evidence-based explanations about electric circuits, Ohm's law. This type of lesson design is effective to enhance students' engagement in scientific practices and develop conceptual understanding.

Given that standards-based education has been commonplace since the early 1980's, most practicing education professionals cannot remember a time where standards did not exist. Standards have historically served as a mechanism for accountability and academic achievement. In physical education, while not required in initial educational reforms, the development of standards was seen as a way to combat the marginalization of our subject. Despite this, we lack evidence demonstrating the impact of standards on physical education programming and student learning. Further, we rarely take the time to consider why standards are commonplace in education, and how they are developed. This paper brings to life the historical timeline of relevant educational reforms, offers a critical perspective of standards in physical education, and proposes several questions in order to challenge the status quo and encourage productive discourse around standards-based physical education.

This study reviewed thirty-seven articles on intercultural learning supported by technology. The results are reported in terms of strength of evidence and relationship among research variables. The results indicated the following strength of evidence: (1) moderate evidence showed higher frequency of the technology usage in higher education or above; strong evidence revealed that technology was applied more frequently in studies with large sample than in those with small sample; (2) moderate evidence demonstrated most of the main findings were obtained from studies related to higher education or above; (3) moderate evidence illustrated that the main finding was intercultural learning outcome and it was frequently obtained using various measuring instruments; (4) there was moderate evidence that mixed research was prevalent in the reviewed studies, and most of the main findings were obtained through mixed research; (5) strong evidence showed that computer vision technologies had the most significant impacts on participants' intercultural learning. In addition, the following five relationships were identified: participants and technology, participants and main findings, instruments and main findings, types and main findings, and impacts of technology on intercultural learning. Taken together, the implication of this research provides reliable evidence for driving innovation in technology-assisted cross-cultural learning studies.