Search

Abstract. A significant problem that assessments of scientific reasoning face at the level of higher education is the question of domain generality, that is, whether a test will produce biased results for students from different domains. This study applied three recently developed methods of analyzing differential item functioning (DIF) to evaluate the domain generality assumption of a common scientific reasoning test. Additionally, we evaluated the usefulness of these new, tree- and lasso-based, methods to analyze DIF and compared them with methods based on classical test theory. We gave the scientific reasoning test to 507 university students majoring in physics, biology, or medicine. All three DIF analysis methods indicated a domain bias present in about one-third of the items, mostly benefiting biology students. We did not find this bias by using methods based on classical test theory. Those methods indicated instead that all items were easier for physics students compared to biology students. Thus, the tree- and lasso-based methods provide a clear added value to test evaluation. Taken together, our analyses indicate that the scientific reasoning test is neither entirely domain-general, nor entirely domain-specific. We advise against using it in high-stakes situations involving domain comparisons.

Investigation into the structure of Hare's Psychopathy Checklist-Revised (PCL-R) has consistently shown it to be multidimensional. The implications of multidimensionality for the scale are nontrivial. In the first place, because the construct of psychopathy has traditionally been considered to be dimensional and unitary, results seeming to show that the PCL-R is multidimensional could be taken as calling into question the construct validity of the scale as a measure of psychopathy. In the second, unidimensionality is the sole psychometric justification for employing a composite of test items to scale individuals. Herein, we argue that the purported multidimensionality might, in actuality, be an artifact of the grounding of dimensionality assessment on species of dimensionality- those specific to principal component and linear factor analysis- which are incongruous with the scale. A reconsideration of the scale's theoretical structure singles out the unidimensional monotone latent variable (UMLV) brand of unidimensionality as the one appropriate to the role of grounding dimensionality assessments of the PCL-R. Empirical support for this position is provided by a small-scale dimensionality analysis informed by the UMLV sense of unidimensionality.

To provide researchers with a means of assessing the fit of the structural component of structural equation models, structural fit indices- modifications of the composite fit indices, RMSEA, SRMR, and CFI- have recently been developed. We investigated the performance of four of these structural fit indices- RMSEA-P, RMSEA_S, SRMR_S, and CFI_S-, when paired with widely accepted cutoff values, in the service of detecting structural misspecification. In particular, by way of simulation study, for each of seven fit indices- 3 composite and 4 structural-, and the traditional chi-square test of perfect composite fit, we estimated the following rates: a) Type I error rate (i.e., the probability of (incorrect) rejection of a correctly specified structural component), under each of four degrees of misspecification in the measurement component; and b) Power (i.e., the probability of (correct) rejection of an incorrectly specified structural model), under each condition formed of the pairing of one of three degrees of structural misspecification with one of four degrees of measurement component misspecification. In addition to sample size, the impacts of two model features, incidental to model misspecification- number of manifest variables per latent variable and magnitude of factor loading- were investigated. The results suggested that, although the structural fit indices performed relatively better than the composite fit indices, none of the goodness-of-fit index with a fixed cutoff value pairings was capable of delivering an entirely satisfactory Type I error rate/Power balance, [RMSEA_S,.05] failing entirely in this regard. Of the remaining pairings; a) RMSEA-P and CFI_S suffered from a severely inflated Type I error rate; b) despite the fact that they were designed to pick up on structural features of candidate models, all pairings- and especially, RMSEA-P and CFI_S- manifested sensitivities to model features, incidental to structural misspecification; and c) although, in the main, behaving in a sensible fashion, SRMR_S was only sensitive to structural misspecification when it occurred at a relatively high degree.

Although phonological awareness (PA) and rapid automatized naming (RAN) are confirmed as early predictors of reading in a large number of orthographies, it is as yet unclear whether the predictive patterns are universal or language specific. This was examined in a longitudinal study across Grades 1 and 2 with 1,120 children acquiring one of five alphabetic orthographies with different degrees of orthographic complexity (English, French, German, Dutch, and Greek). Path analyses revealed that a universal model could not be confirmed. When we specified the best-fitting model separately for each language, RAN was a consistent predictor of reading fluency in all orthographies, whereas the association between PA and reading was complex and mostly interactive. We conclude that RAN taps into a language-universal cognitive mechanism that is involved in reading alphabetic orthographies (independent of complexity), whereas the PA–reading relationship depends on many factors like task characteristics, developmental status, and orthographic complexity.

Zusammenfassung. Groß angelegte Replikationsprojekte der letzten Jahre legen ein aus unserer Sicht beunruhigendes Ausmaß an nicht-replizierbaren Befunden in der wissenschaftlichen Literatur nahe, sowohl in der Psychologie als auch in anderen Disziplinen. Basierend auf einer Analyse einiger Ursachen dieser Situation argumentieren wir, dass der Wandel hin zu einer offenen Wissenschaft („Open Science“) eine Konsequenz aus der Glaubwürdigkeitskrise sein muss. Wir plädieren für konkrete und machbare Änderungen in den Arbeitseinheiten und Instituten vor Ort, und zeigen exemplarisch, welche Schritte am Department Psychologie der Ludwig-Maximilians-Universität München umgesetzt wurden. Diese Schritte betreffen Anreizstrukturen, die Forschungskultur, die Lehre und die Verzahnung mit der Ethikkommission. Sie haben das Ziel, eine reproduzierbarere und glaubwürdigere Forschung zu unterstützen, ohne unnötige bürokratische Belastungen zu erzeugen.

Education systems increasingly emphasize the importance of scientific reasoning skills such as generating hypotheses and evaluating evidence. Despite this importance, we do not know which tests of ...

Lehrpersonen werden in ihrem Berufsalltag mit einer Vielzahl an Aufgabenstellungen konfrontiert, die die Erstellung von Diagnosen und diagnostische Kenntnisse erfordern. Im Fokus der Diagnostik durch Lehrpersonen stehen Lernvoraussetzungen, Lernprozesse und Lernergebnisse von Schulerinnen und Schulern (Schrader 2013; Kap. 25), die mit unterschiedlichen Verfahren gemessen werden. Neben Verfahren zur Lernergebnismessung kann auch der Einsatz offentlich verfugbarer oder selbst konstruierter Fragebogen fur die Erfassung schulbezogener Einstellungen (z. B. gegenuber dem unterrichteten Fach oder den eingesetzten Unterrichtsmethoden) relevant sein. Welche Informationen mittels diagnostischer Methoden gesammelt werden sollen, richtet sich nach den konkreten Fragestellungen. Die Beantwortung dieser Fragestellungen ist oftmals mit wichtigen padagogischen Entscheidungen verbunden (z. B. Empfehlung fur eine bestimmte Schulart). Die Erfassung nicht geeigneter Informationen und/oder methodisch schlecht erfasste Informationen erschweren padagogische Entscheidungen (van Ophuysen & Behrmann 2015). Um fundierte padagogische Entscheidungen mit Hilfe einer sachgemasen Diagnostik treffen zu konnen, benotigen Lehrpersonen bestimmte Fahigkeiten und Fertigkeiten. Die Kompetenz von Lehrpersonen zur erfolgreichen Bewaltigung diagnostischer Aufgabenstellungen wird als diagnostische Kompetenz bezeichnet. Sie schliest die Qualitat der erbrachten Diagnosestellungen mit ein (Schrader 2011). Helmke (2009, S. 122) nennt einige wichtige Voraussetzungen fur die Diagnosekompetenz von Lehrpersonen. Hierzu gehoren u. a. Kenntnisse uber grundlegende Begriffe der padagogischen Diagnose, Gutekriterien diagnostischer Leistungen, ausgewahlte Diagnoseinstrumente, Tests und Fragebogenverfahren. Weitere entscheidende Merkmale der Diagnosekompetenz von Lehrpersonen sind nach Helmke (2009, S. 122) Fahigkeiten zur geeigneten Auswahl von Diagnoseverfahren fur einen gegebenen Anlass, die Bildung einer zusammenfassenden Beurteilung auf Basis erhaltener Informationen, die Ableitung von Hypothesen und Prognosen uber die zukunftige Entwicklung aus einer Diagnose und die verstandliche und konstruktive Vermittlung von Diagnoseergebnissen an Schulerinnen, Schuler und Eltern.

Factorial validity is often assessed using confirmatory factor analysis. Model fit is commonly evaluated using the cutoff values for the fit indices proposed by Hu and Bentler (1999) . There is a body of research showing that those cutoff values cannot be generalized. Model fit does not only depend on the severity of misspecification, but also on nuisance parameters, which are independent of the misspecification. Using a simulation study, we demonstrate their influence on measures of model fit. We specified a severe misspecification, omitting a second factor, which signifies factorial invalidity. Measures of model fit showed only small misfit because nuisance parameters, magnitude of factor loadings and a balanced/imbalanced number of indicators per factor, also influenced the degree of misfit. Drawing from our results, we discuss challenges in the assessment of factorial validity.

Background: As a modular inventory, the Heidelberger Drogenbogen (HDB) assesses substance-specific knowledge regarding the illicit psychoactive substances cocaine, amphetamines, cannabis, ecstasy, and hallucinogens. This diagnostic tool allows for assessing the level of substance-specific knowledge in most frameworks of prevention.Objectives: This study outlines basic psychometric parameters of the HDB. Different groups of drug users (clinical versus non-clinical, users versus non-users) are compared concerning their knowledge about psychoactive substances.Methods: A community sample (recruited at schools, universities and in subcultural contexts [n = 4212]) was compared to a clinical sample (recruited in in- and out-patient drug abuse treatment departments [n = 582]).Results: Cronbach’s α coefficients (0.61 ≤ α ≤ 0.87) and test–retest correlations (0.81 ≤ rp ≤ 0.87) indicate that the scales are reliable measures of substance-specific knowledge about drugs. Drug users generally have a higher level...

G-factor models such as the bifactor model and the hierarchical G-factor model are increasingly applied in psychology. Many applications of these models have produced anomalous and unexpected results that are often not in line with the theoretical assumptions on which these applications are based. Examples of such anomalous results are vanishing specific factors and irregular loading patterns. In this article, the authors show that from the perspective of stochastic measurement theory anomalous results have to be expected when G-factor models are applied to a single-level (rather than a 2-level) sampling process. The authors argue that the application of the bifactor model and related models require a 2-level sampling process that is usually not present in empirical studies. We demonstrate how alternative models with a G-factor and specific factors can be derived that are more well-defined for the actual single-level sampling design that underlies most empirical studies. It is shown in detail how 2 alternative models, the bifactor-(S - 1) model and the bifactor-(S·I - 1) model, can be defined. The properties of these models are described and illustrated with an empirical example. Finally, further alternatives for analyzing multidimensional models are discussed. (PsycINFO Database Record

Short scales have become widely used in settings in which participant time is limited and when assessment would otherwise be impossible. Based on simulated data on the population level we investigate whether scale shortening affects the desired invariance of criterion-related validities as well as differences between estimated expected values of populations. We conclude that, under a unidimensional model, decreasing the number of items does neither affect criterion validity nor differences between expected values between two populations. It is, however, discussed, that possible problems of scale shortening can occur on the construct level and, practically more important, on the individual score level.

Suboptimal vitamin B status might affect cognitive performance in early childhood. We tested the hypothesis that short-term supplementation with folic acid and selected B vitamins improves cognitive function in healthy children in a population with relatively low folate status. We screened 1,002 kindergarten children for suboptimal folate status by assessing the total urinary para-aminobenzoylglutamate excretion. Two hundred and fifty low ranking subjects were recruited into a double blind, randomized, controlled trial to receive daily a sachet containing 220 μg folic acid, 1.1 mg vitamin B2, 0.73 mg B6, 1.2 μg B12 and 130 mg calcium, or calcium only for 3 months. Primary outcomes were changes in verbal IQ, short-term memory and processing speed between baseline and study end. Secondary outcomes were urinary markers of folate and vitamin B12 status, acetyl-para-aminobenzoylglutamate and methylmalonic acid, respectively, and, in a subgroup of 120 participants, blood folate and plasma homocysteine. Pre- and post-intervention cognitive measurements were completed by 115 children in the intervention and 122 in the control group. Compared to control, median blood folate increased by about 50 % (P for difference, P

Findings of studies on the unique effects of reasoning and working memory regarding complex problem solving are inconsistent. To find out if these inconsistencies are due to a lack of symmetry between the studies, we reconsidered the findings of three published studies on this issue, which resulted in conflicting conclusions regarding the inter-relations between reasoning, working memory, and complex problem solving. This was achieved by analysing so far unpublished problem solving data from the study of Bühner, Krumm, Ziegler, and Plücken (2006) (N= 124). One of the three published studies indicated unique effects of working memory and reasoning on complex problem solving using aggregated scores, a second study found no unique contribution of working memory using only figural scores, and a third study reported a unique influence only for reasoning using only numerical scores. Our data featured an evaluation of differences across content facets and levels of aggregation of the working memory scores. Path models showed that the results of the first study could not be replicated using content aggregated scores; the results of the second study could be replicated if only figural scores were used, and the results of the third study could be obtained by using only numerical scores. For verbal content, none of the published results could be replicated. This leads to the assumption that not only symmetry is an issue when correlating non-symmetrical data, but that content also has to be taken into account when comparing different studies on the same topic.

Many studies are concerned with the bivariate relationships between Openness, fluid intelligence (Gf), and crystallized intelligence (Gc). Results suggest an influence of Gf and Openness on Gc. However, the overlap between Gf and Openness is rarely controlled for. Moreover, interaction effects or longitudinal influences are also often neglected. The present two studies aimed to elucidate exactly these interactions and longitudinal influences. Besides a main effect of Gf on Gc, Study 1 (N = 180) revealed an interaction effect between Openness and Gf. Study 2 utilized longitudinal data (N = 172) and identified an effect of Openness on the development of Gf. Gf and Openness predicted Gc 6 years later. A model integrating the results and providing a theoretical framework and outlook is proposed.

This simulation study investigated the sensitivity of commonly used cutoff values for global-model-fit indexes, with regard to different degrees of violations of the assumption of uncorrelated errors in confirmatory factor analysis. It is shown that the global-model-fit indexes fell short in identifying weak to strong model misspecifications under both different degrees of correlated error terms, and various simulation conditions. On the basis of an example misspecification search, it is argued that global model testing must be supplemented by this procedure. Implications for the use of structural equation modeling are discussed.

The Statistics Anxiety Rating Scale (STARS) was adapted into German to examine its psychometric properties (n = 400). Two validation studies (n = 66, n = 96) were conducted to examine its criterion-related validity. The psychometric properties of the questionnaire were very similar to those previously reported for the original English version in various countries and other language versions. Confirmatory factor analysis indicated 2 second-order factors: One was more closely related to anxiety and the other was more closely related to negative attitudes toward statistics. Predictive validity of the STARS was shown both in an experimental exam-like situation in the laboratory and during a real examination situation. Taken together, the findings indicate that statistics anxiety as assessed by the STARS is a useful construct that is more than just an expression of a more general disposition to anxiety.

Researchers applying the Rasch model are frequently confronted with a problem, namely, that its core requirement—non-intersecting item response functions (i.e., uniform item discrimination paramete...

The ETS has recently released new estimates of validities of the GRE for predicting cumulative graduate GPA. They average in the middle thirties – twice as high as those previously reported by a number of independent investigators.It is shown in the first part of this paper that this unexpected finding can be traced to a flawed methodology that tends to inflate multiple correlation estimates, especially those of populations values near zero.Secondly, the issue of upward corrections of validity estimates for restriction of range is taken up. It is shown that they depend on assumptions that are rarely met by the data.Finally, it is argued more generally that conventional test theory, which is couched in terms of cor- relations and variances, is not only unnecessarily abstract but, more importantly, incomplete, since the practical utility of a test does not only depend on its validity, but also on base-rates and admission quotas. A more direct and conclusive method for gauging the utility of a test involves misclassification rates, and entirely dispenses with questionable assumptions and post-hoc "corrections".On applying this approach to the GRE, it emerges (1) that the GRE discriminates against ethnic and economic minorities, and (2) that it often produces more erroneous decisions than a purely random admissions policy would.

The Heidelberger Drogenbogen (HDB) is a German language assessment of substance-specific knowledge and consumption patterns of the illicit psychoactive substances cannabis, MDMA (ecstasy), amphetamines, cocaine, and hallucinogens. The behavior modules for each of these five drugs/drug groups allow for a diagnostic evaluation of the extent of harmful consumption behavior. Each of the five modules represents a single standardized test.This paper outlines several statistical parameters, Cronbach's alpha, retest reliabilities, as well as numerous validity and cut-off-criteria of the behavioral modules.Participants (N = 4,794) were recruited at schools, universities, in subcultural contexts, and in institutions of substance abuse treatment.Internal consistencies range from = .68 to .79 while test-retest reliabilities between .87 and .94 were found. The behavior modules of the HDB can discriminate between populations with and without clinical levels of substance use. Furthermore, this measure has incremental validity and higher diagnostic accuracy over competing measures.The behavior modules of the HDB are reliable and valid measures of substance use and misuse.

Publication bias remains a controversial issue in psychological science. The tendency of psychological science to avoid publishing null results produces a situation that limits the replicability assumption of science, as replication cannot be meaningful without the potential acknowledgment of failed replications. We argue that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact.

The history of the past four decades of the theory and application of additive conjoint measurement (ACM) is characterized by vivid developments of its theoretical foundation (cf. Luce and Tukey, 1964; Krantz et al., 1971, 2006; Narens, 1974), industrious developments of statistical and computational implementations (cf. Karabatsos and Ullrich, 2002; Karabatsos and Sheu, 2004; Karabatsos, 2005; Myung et al., 2005) and heated debates about its applicability and significance in psychology (cf. Michell, 1997, 2009; Borsboom and Mellenbergh, 2004; Barrett, 2008; Borsboom and Scholten, 2008; Kyngdon, 2008a; Trendler, 2009). What started as a promising foundation to solve the everlasting debate about the quantitative nature of psychological attributes (Ferguson et al., 1939) ended in perseverative debates with very little transfer to mainstream psychological science still being dominated by structural equation modeling (SEM) and item response theory (IRT). After reading the aforementioned articles, and comparing their implications with the day-to-day business of mainstream psychological science, even an unbiased reader would certainly agree with Cliff (1992) that ACM was a “… revolution that never happened” (p. 186). It is not the aim of this article, to discredit the efforts of mathematical psychology and proponents of ACM in particular. I just want to address the naive but relevant question why ACM as a stringent way to formalize and to test the requirements of quantitative measurement in psychology has not been embraced by mainstream psychology as a means to an end to test what they always claim: that most of the attributes (e.g., intelligence and personality factors) are quantitative. An attribute possessing a quantitative structure is required to satisfy the three conditions of ordinality (transitivity, antisymmetry, and strong connexity) and the six conditions of additivity (associativity, commutativity, monotonicity, solvability, positivity, and the Archimedean condition; cf. Michell, 1990, p. 52f.). Most of these conditions are testable hypotheses but I have never seen any empirical test in psychological articles before data were analyzed with SEM or IRT models, which already assume the quantitative structure of the attributes under consideration as argued below. Somewhere during my psychology studies at the university I learned that psychology is an empirical science and that there is therefore no room for claims that should just be believed. However, given the assumed but almost never tested quantitative nature of most of the psychological attributes as reflected in factor analysis, SEM and IRT models, I must have missed or misunderstood something.

Fit indices are widely used in order to test the model fit for structural equation models. In a highly influential study, Hu and Bentler (1999) showed that certain cutoff values for these indices could be derived, which, over time, has led to the reification of these suggested thresholds as "golden rules" for establishing the fit or other aspects of structural equation models. The current study shows how differences in unique variances influence the value of the global chi-square model test and the most commonly used fit indices: Root-mean-square error of approximation, standardized root-mean-square residual, and the comparative fit index. Using data simulation, the authors illustrate how the value of the chi-square test, the root-mean-square error of approximation, and the standardized root-mean-square residual are decreased when unique variances are increased although model misspecification is present. For a broader understanding of the phenomenon, the authors used different sample sizes, number of observed variables per factor, and types of misspecification. A theoretical explanation is provided, and implications for the application of structural equation modeling are discussed.

Research on bystander intervention has produced a great number of studies showing that the presence of other people in a critical situation reduces the likelihood that an individual will help. As the last systematic review of bystander research was published in 1981 and was not a quantitative meta-analysis in the modern sense, the present meta-analysis updates the knowledge about the bystander effect and its potential moderators. The present work (a) integrates the bystander literature from the 1960s to 2010, (b) provides statistical tests of potential moderators, and (c) presents new theoretical and empirical perspectives on the novel finding of non-negative bystander effects in certain dangerous emergencies as well as situations where bystanders are a source of physical support for the potentially intervening individual. In a fixed effects model, data from over 7,700 participants and 105 independent effect sizes revealed an overall effect size of g = -0.35. The bystander effect was attenuated when situations were perceived as dangerous (compared with non-dangerous), perpetrators were present (compared with non-present), and the costs of intervention were physical (compared with non-physical). This pattern of findings is consistent with the arousal-cost-reward model, which proposes that dangerous emergencies are recognized faster and more clearly as real emergencies, thereby inducing higher levels of arousal and hence more helping. We also identified situations where bystanders provide welcome physical support for the potentially intervening individual and thus reduce the bystander effect, such as when the bystanders were exclusively male, when they were naive rather than passive confederates or only virtually present persons, and when the bystanders were not strangers.

Objectives: This study investigated the psychological as well as neuroendocrine stress response across one week before an important sport competition, introducing the cortisol awakening response (CAR) to sport psychological research. Methods: On three days in the week before the German Nationals, martial artists (N = 17) reported their competitive state anxiety and collected five samples of salivary cortisol during the first hour after awakening. Results: Hierarchic-linear models and multiple regressions were conducted. Despite a significant rise in "somatic anxiety" (p < .05), the increment of CAR across the week remained non-significant. A moderator function of competitive anxiety on the released amount of cortisol in the morning was not found significant. Results did not show any significant regression of changes in the neuroendocrine response on changes in state anxiety. Conclusion: Non- significant increments of CAR with a closer proximity to the competition may be interpreted as a possible habituation of basal hypothalamus-pituitary-adrenal activity. Moreover, athletes appear to have a lower CAR than found in norm studies, which points to further investigation of interindividual and situational effects on the temporal pattern of the neuroendocrine response to sport competitions.