Search

There is a norm in psychology to use causally ambiguous statistical language, rather than straightforward causal language, when describing methods and results of nonexperimental studies. However, causally ambiguous language may inhibit a critical examination of the studys causal assumptions and lead to a greater acceptance of policy recommendations that rely on causal interpretations of nonexperimental findings. In a preregistered experiment, 142 psychology faculty, postdocs, and doctoral students (54% female), ages 22-67 (M = 33.20, SD = 8.96), rated the design and analysis from hypothetical studies with causally ambiguous statistical language as of higher quality (by .34-.80 SD) and as similarly or more supportive (by .16-.27 SD) of policy recommendations than studies described in straightforward causal language. Thus, using statistical rather than causal language to describe nonexperimental findings did not decrease, and may have increased, perceived support for implicitly causal conclusions.

The flipped classroom (FC) is becoming an increasingly popular teaching method in mathematics education. However, few studies have rigorously evaluated its effectiveness, and less so in countries where students' level of mathematics achievement is low. In this study, we evaluated the impact of an FC intervention in Saudi Arabia, the country with the lowest level of maths achievement in the last iteration of the Trends International Mathematics and Science Study TIMSS . A total of 281 secondary school students received eight weeks of algebra training using either FC or traditional instruction, and were tested on algebra problems taken from past national standardised tests, as well as on a measure of self-efficacy. Students who received the intervention showed higher self-efficacy but no significant difference in maths achievement was observed. Students' and teachers' perceptions of the intervention were positive.

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (p < [alpha]) We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

Research findings regarding the effects of educational interventions--typically reported in units of standard deviations (e.g., Cohen's d)--are often translated into more intuitive metrics before being communicated to teachers. However, there is no consensus about the most suitable metric, and no study has systematically examined how teachers respond to the different options. We conducted two preregistered studies addressing this issue. We found that teachers have strong preferences concerning effect size metrics in terms of informativeness, understandability, and helpfulness. These preferences challenge current research reporting recommendations. Most importantly, we found that different metrics induce different perceptions of an intervention's effectiveness--a situation that could cause teachers to have unrealistic expectations about what a given intervention may achieve. Implications for how educational effects should be communicated are discussed.

In two studies (N's = 55 and 54), we examined a basic form of conceptual understanding of rational number arithmetic, the direction of effect of decimal arithmetic operations, at a level of detail useful for informing instruction. Middle school students were presented tasks examining knowledge of the direction of effects (e.g., "True or false: 0.77 * 0.63 > 0.77), number line estimation of decimal magnitudes, and knowledge of decimal arithmetic procedures. Their confidence in their direction of effect judgments was also assessed. We found (1) most students incorrectly predicted the direction of effect of multiplication and division with decimals below one; (2) this pattern held for students who accurately estimated the magnitudes of individual decimals and correctly executed decimal arithmetic operations; (3) explanations of direction of effect judgments that cited both the arithmetic operation and the numbers' magnitudes were strongly associated with accurate judgments; and (4) judgments were more accurate when multiplication problems involved a whole number and a decimal below one than with two decimals below one. Implications of the findings for instruction are discussed. [At time of submission to ERIC this article was in press with the "Journal of Educational Psychology."]

Fraction and decimal arithmetic are crucial for later mathematics achievement and for ability to succeed in many professions. Unfortunately, these capabilities pose large difficulties for many children and adults, and students' proficiency in them has shown little sign of improvement over the past three decades. To summarize what is known about fraction and decimal arithmetic and to stimulate greater amounts of research in the area, we devoted this review to analyzing why learning fraction and decimal arithmetic is so difficult. We identify and discuss two types of difficulties: (1) Difficulties that are inherent to fraction and decimal arithmetic, and (2) Culturally contingent sources that could be reduced by improved instruction and prior knowledge of learners. We conclude the review by discussing commonalities among three interventions that have helped children overcome the difficulties of mastering fraction and decimal arithmetic. [This article was published in the "Developmental Review," v38 p201-221 Dec 2015.]

Understanding of numerical development is growing rapidly, but the volume and diversity of findings can make it difficult to perceive any coherence in the process. The integrative theory of numerical development posits that a coherent theme is present, however--progressive broadening of the set of numbers whose magnitudes can be accurately represented--and that this theme unifies numerical development from infancy to adulthood. From this perspective, development of numerical representations involves four major acquisitions: 1) increasingly precise representations of magnitudes of numbers expressed nonsymbolically, 2) linking nonsymbolic to symbolic numerical representations, 3) extending understanding to increasingly large whole numbers, and 4) extending understanding to all rational numbers. Thus, the mental number line expands rightward to encompass larger whole numbers, leftward to encompass negatives, and interstitially to include fractions and decimals. [This article was published in "Child Development Perspectives," v8 n3 p144-150 Sep 2014.]

There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K-12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.

Creativity is widely considered a skill essential to succeeding in the modern world. Numerous creativity training programs have been developed, and several meta-analyses have attempted to summarize the effectiveness of these programs and identify the features influencing their impact. Unfortunately, previous meta-analyses share a number of limitations, most notably overlooking the potentially strong impact of publication bias and the influence of study quality on effect sizes. We undertook a meta-analysis of 169 creativity training studies across 5 decades (844 effect sizes, the largest meta-analysis of creativity training to date), including a substantial number of unpublished studies (48 studies; 262 effect sizes). We employed a range of statistical methods to detect and adjust for publication bias and evaluated the robustness of the evidence in the field. In line with previous meta-analyses, we found a moderate training effect (0.53 SDs; unadjusted for publication bias). Critically, we observed converging evidence consistent with strong publication bias. All adjustment methods considerably lowered our original estimate (adjusted estimates ranged from 0.29 to 0.32 SDs). This severe bias casts doubt on the representativeness of the published literature in the field and on the conclusions of previous meta-analyses. Our analysis also revealed a high prevalence of methodological shortcomings in creativity training studies (likely to have inflated our average effect), and little signs of methodological improvement over time—a situation that limits the usefulness of this body of work. We conclude by presenting implications and recommendations for researchers and practitioners, and we propose an agenda for future research. Public Significance Statement: Creativity is considered an essential skill in many contexts, leading to the development of numerous training programs aimed at improving this skill. We examined 5 decades of creativity training studies (169 studies). Using a range of meta-analytic techniques, we found converging evidence consistent with publication bias. Moreover, our analysis revealed a large number of studies with methodological shortcomings. Both of these findings suggest that creativity training may be less effective than previously thought. In view of these findings, we proposed a set of recommendations to assist researchers and practitioners in interpreting findings from the creativity training literature, and we outlined directions for future research to increase the informativeness of creativity training studies. [ABSTRACT FROM AUTHOR]

In this response, we first show that Simpson's proposed analysis answers a different and less interesting question than ours. We then justify the choice of prior for our Bayes factors calculations, but we also demonstrate that the substantive conclusions of our article are not substantially affected by varying this choice.

In 2 studies (Ns = 55 and 54), the authors examined a basic form of conceptual understanding of rational number arithmetic, the direction of effect of decimal arithmetic operations, at a level of detail useful for informing instruction. Middle school students were presented tasks examining knowledge of the direction of effects (e.g., "True or false: 0.77 * 0.63 > 0.77"), knowledge of decimal magnitudes, and knowledge of decimal arithmetic procedures. Their confidence in their direction of effect judgments was also assessed. The authors found (a) most students incorrectly predicted the direction of effect of multiplication and division with decimals below 1; (b) this pattern held for students who accurately compared the magnitudes of individual decimals and correctly executed decimal arithmetic operations; (c) explanations of direction of effect judgments that cited both the arithmetic operation and the numbers' magnitudes were strongly associated with accurate judgments; and (d) judgments were more accurate when multiplication problems involved a whole number and a decimal below 1 than with 2 decimals below 1. Implications of the findings for instruction are discussed. [For the Grantee Submission, see ED569126.]

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( p < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. [ABSTRACT FROM AUTHOR]

Understanding an arithmetic operation implies, at minimum, knowing the direction of effects that the operation produces. However, many children and adults, even those who execute arithmetic procedures correctly, may lack this knowledge on some operations and types of numbers. To test this hypothesis, we presented preservice teachers (Study 1), middle school students (Study 2), and math and science majors at a selective university (Study 3) with a novel direction of effects task with fractions. On this task, participants were asked to predict without calculating whether the answer to an inequality would be larger or smaller than the larger fraction in the problem (e.g., "True or false: 31/56 * 17/42 > 31/56"). Both preservice teachers and middle school students correctly answered less often than chance on problems involving multiplication and division of fractions below 1, though they were consistently correct on all other types of problems. In contrast, the math and science students from the selective university were consistently correct on all items. Interestingly, the weak understanding of multiplication and division of fractions below 1 was present even among middle school students and preservice teachers who correctly executed the fraction arithmetic procedures and had highly accurate knowledge of the magnitudes of individual fractions, which ruled out several otherwise plausible interpretations of the findings. Theoretical and educational implications of the findings are discussed.

Abstract reasoning is critical for science and mathematics, but is very difficult. In 3 studies, the hypothesis that alternatives generation required for conditional reasoning with false premises facilitates abstract reasoning is examined. Study 1 (n = 372) found that reasoning with false premises improved abstract reasoning in 12- to 15-year-olds. Study 2 (n = 366) found a positive effect of simply generating alternatives, but only in 19-year-olds. Study 3 (n = 92) found that 9- to 11-year-olds were able to respond logically with false premises, whereas no such ability was observed in 6- to 7-year-olds. Reasoning with false premises was found to improve reasoning with semiabstract premises in the older children. These results support the idea that alternatives generation with false premises facilitates abstract reasoning.

The flipped classroom (FC) is becoming an increasingly popular teaching method in mathematics education. However, few studies have rigorously evaluated its effectiveness, and less so in countries where students' level of mathematics achievement is low. In this study, we evaluated the impact of an FC intervention in Saudi Arabia, the country with the lowest level of maths achievement in the last iteration of the Trends International Mathematics and Science Study TIMSS. A total of 281 secondary school students received eight weeks of algebra training using either FC or traditional instruction, and were tested on algebra problems taken from past national standardised tests, as well as on a measure of self‐efficacy. Students who received the intervention showed higher self‐efficacy but no significant difference in maths achievement was observed. Students' and teachers' perceptions of the intervention were positive. Practitioner notes: What is already known about this topic The flipped classroom is an increasingly popular teaching method in mathematics education.There is evidence that the method is beneficial for students.However, there have been very few rigorous evaluation studies of flipped classrooms and most were conducted in the US, Taiwan and China.Whether the beneficial impact of flipped classrooms can be generalised to other countries, particularly countries with low proficiency in mathematics, remains unclear. What this paper adds We evaluated the impact of the flipped classroom in Saudi Arabia, a country with low proficiency in mathematics and limited technological resources.We used a rigorous design (with a control group) and educationally relevant outcome measures.We found that a flipped classroom can have a positive impact on students' self‐efficacy, and that the method was perceived positively by both teachers and students.Based on interview data, we also documented the perceptions and concerns of the participating teachers. Implications for practice and/or policy Flipped classrooms showed promising results in a context which differs markedly from previous evaluations, suggesting that the method can be beneficial in a range of contexts.This positive impact was observed despite the intervention being relatively short (six weeks) and implemented by teachers previously unfamiliar with the flipped classroom method. [ABSTRACT FROM AUTHOR]