Search

This paper studied the usefulness of differential item functioning (DIF) methodology for examining potential mode effects. Although the goal was not to validate the comparability of the assessments per se, it is of interest to speculate why some formats could give rise to differential performance. Data were obtained from two instruments on which validation studies were conducted to determine if scores were comparable across modes of administration. The first instrument was a norm-referenced aptitude test called InView, administered to students in grades 4 to 9 in computer and paper modes for this study. Additional data were obtained from a Test of Adult Basic Education (TABE), also norm-referenced. Test takers were asked about their levels of computer experience and preferences. Two DIF statistics, the Linn-Harnisch procedure (1981) and the nonparametric Standardized Mean Difference were used to assess DIF. Results show that some levels of InView had substantial numbers of items that were flagged for the online comparisons with the standardization study, while others had relatively few. The TABE had two flagged items, and differences between computer-based and standardization groups were throughout the ability range with more apparent differences in the lower portion of the distribution. In general, students were neutral to mode or preferred computer-administered tests. Findings show that DIF methodology presents a well-studied method for examining group differences at the item level that can be used to examine mode of administration differences at the item level. Appendixes contain the student surveys and derivations of the DIF methods. (Contains 10 figures, 4 tables, and 10 references.) (SLD)

Predictive analytics is widely used in various domains, including education, to inform decision-making and improve outcomes. However, many predictive models are proprietary and inaccessible for evaluation or modification by researchers and practitioners, limiting their accountability and ethical design. Moreover, predictive models are often opaque and incomprehensible to the officials who use them, reducing their trust and utility. Furthermore, predictive models may introduce or exacerbate bias and inequity, as they have done in many sectors of society. Therefore, there is a need for transparent, interpretable, and fair predictive models that can be easily adopted and adapted by different stakeholders. In this paper, we propose a fair predictive model based on multivariate adaptive regression splines (MARS) that incorporates fairness measures in the learning process. MARS is a non-parametric regression model that performs feature selection, handles non-linear relationships, generates interpretable decision rules, and derives optimal splitting criteria on the variables. Specifically, we integrate fairness into the knot optimization algorithm and provide theoretical and empirical evidence of how it results in a fair knot placement. We apply our "fair"MARS model to real-world data and demonstrate its effectiveness in terms of accuracy and equity. Our paper contributes to the advancement of responsible and ethical predictive analytics for social good. [This paper was presented at an Association for the Advancement of Artificial Intelligence conference.]

Location tracking devices are becoming increasingly popular in practice to study movement of customers or track inventory. However, using location tracking devices in education contexts is quite novel. In this paper, we present a robust Bayesian nonparametric mixture model that clusters location data. We successfully apply this model on location data retrieved from two different events using Quuppa, a location tracking system, to identify statistically significant clusters. Implications of this model include identifying "unexpected" clusters during interactive learning activities such as groups of students isolated from other students. As a consequence, further research in behavioral analysis can be done to analyze why such behavior is exhibited.

Many researchers recognize that factor analysis can be conducted on both correlation matrices and variance-covariance matrices. Although most researchers extract factors from non-distribution free or parametric methods, researchers can also extract factors from distribution free or non-parametric methods. The nature of the data dictates the method selected. The purpose of this paper is to differentiate between the questions asked by Pearson product-moment correlations and Spearman's rho coefficients. Additionally, the paper compares distribution free and non-distribution free methods for extracting factors using correlational and covariance matrices, and describes the advantages of each method. (Contains 9 tables and 4 figures.)

This paper advocates the use of nonparametric statistics. First, the consequence of using parametric inferential techniques under nonnormality is described. Second, the advantages of using nonparametric techniques are presented. The third purpose is to demonstrate empirically how infrequently nonparametric statistics appear in studies, even those published in the most reputable journals. Fourth, a typology of nonparametric statistics is presented for all univariate general linear model analyses. The nonparametric statistics that are available in the most commonly used statistical software are delineated, and finally, nonparametric effect size indices are outlined. (Contains 1 table and 52 references.) (Author/SLD)

Nonparametric procedures are often more powerful than classical tests for real world data, which are rarely normally distributed. However, there are difficulties in using these tests. Computational formulas are scattered throughout the literature, and there is a lack of availability of tables of critical values. This paper brings together the computational formulas for 20 commonly used nonparametric tests that have large-sample approximations for the critical value. Because there is no generally agreed upon lower limit for the sample size, Monte Carlo methods have been used to determine the smallest sample size that can be used with the large-sample approximations. The statistics reviewed include single-population tests, comparisons of two populations, comparisons of several populations, and tests of association. (Contains 4 tables and 59 references.) (Author/SLD)

Item response theory (IRT) has been adapted as the theoretical foundation of computerized adaptive testing (CAT) for several decades. In applying IRT to CAT, there are certain considerations that are essential, and yet tend to be neglected. These essential issues are addressed in this paper, and then several ways of eliminating noise and bias in estimating the individual parameter, theta, of person "a" are proposed and discussed, so that accuracy and efficiency in ability estimation can be increased. The content validity of the ability dimension is emphasized, and the idea of core test items is proposed. Devices are suggested to eliminate noise from multiple-choice items by using the nonparametric estimation of operating characteristics effectively in pilot studies. The use of the normal ogive model is suggested instead of the three-parameter logistic model. It is further suggested that several graded response items be used at the beginning of the CAT to avoid the influence of bias and lack of information inherent in dichotomous response items. The Weighted Likelihood Estimate of T. Warm (1989) and its expanded form for general discrete responses are discussed as an effective method of eliminating bias in ability estimation, and the usefulness of Warm's weight function as a prior is discussed. Use of the modified test information function is suggested for the same purpose. (Contains 10 figures and 18 references.) (SLD)

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all new and currently considered computer-based tests. In addition to developing new models, researchers will need to give some attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This paper introduces an adaptive testing algorithm that balances maximum differentiation among test takers with stable estimation at each stage of testing, and compares this algorithm with a traditional one using IRT and maximum information. The adaptive testing algorithm introduced is based on the classification and regression tree approach described in L. Breiman, J. Friedman, R. Olshen, and C. Stone (1984) and J. Chambers and T. Hastie (1992). Simulation results from the regression tree approach were compared with simulation results from three parameter logistic model IRT. Simulation results show that the nonparametric tree-based approach to adaptive testing may be superior to conventional IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. It clearly outperformed one-dimensional IRT when the pool was strongly two-dimensional. A technical appendix describes the algorithm. (Contains three figures and six references.) (SLD)

This study examines learners' perceptions of the helpfulness of various multimedia components embedded in digital flashcards for explicit, informal foreign language vocabulary learning. Advanced learners of English (N = 59) studied 48 new words using digital flashcards on smartphones. After ten days, the learners completed perception surveys. The goal of this study was to investigate which flashcard components were perceived as most helpful in learning vocabulary. The results of nonparametric statistical tests revealed that students perceive Foreign Language (FL) definitions to be significantly less helpful compared to other flashcard components. Moreover, when given a choice between having access to translation or pronunciation, translation was perceived as significantly more helpful in vocabulary learning. These findings have practical implications for the development of multimedia digital flashcards in mobile-assisted language learning applications. [For the complete volume, see ED578177.]

Despite positive evidence demonstrating the pedagogical benefits of Text-To-Speech (TTS) synthesisers for second/foreign language learning (Liakin, Cardoso, & Liakina, 2017), there is a need for up-to-date formal evaluations, specifically regarding its potential to promote learning. This study evaluates the voice quality of a TTS system in comparison with a human voice, and examines its pedagogical potential for use in an English as a Foreign Language (EFL) setting in terms of speech quality, ability to be understood by L2 users, and potential to focus on a specific language form. EFL learners in Brazil completed four tasks to evaluate the quality of TTS-generated texts. Results suggest that the TTS voice performed equally as well as the human voice in almost every assessment measure, demonstrating a high level of intelligibility and the ability to provide learners with opportunities to notice linguistic forms. [For the complete volume, see ED578177.]

We explore how crowdworkers can be trained to tackle complex crowdsourcing tasks. We are particularly interested in training novice workers to perform well on solving tasks in situations where the space of strategies is large and workers need to discover and try different strategies to be successful. In a first experiment, we perform a comparison of five different training strategies. For complex web search challenges, we show that providing expert examples is an effective form of training, surpassing other forms of training in nearly all measures of interest. However, such training relies on access to domain expertise, which may be expensive or lacking. Therefore, in a second experiment we study the feasibility of training workers in the absence of domain expertise. We show that having workers validate the work of their peer workers can be even more effective than having them review expert examples if we only present solutions filtered by a threshold length. The results suggest that crowdsourced solutions of peer workers may be harnessed in an automated training pipeline.

This paper illustrates the use of the McNemar Test, using a hypothetical problem. The McNemar Test is a nonparametric statistical test that is a type of chi square test using dependent, rather than independent, samples to assess before-after designs in which each subject is used as his or her own control. Results of the McNemar test make it possible to determine whether there is a significant difference between the pretest and posttest scores of students on the dependent variable. A significant difference usually implies that an intervention or treatment has had an effect. In the hypothetical example, an elementary school counselor is asked to conduct group guidance activities for all 278 students at an elementary school because teachers suspect some students are cheating. The counselor administers the same questionnaire about attitudes toward cheating before and after a counseling program. The example shows that the McNemar test is relatively easy to calculate and interpret. One limitation is that of sample size: the McNemar test is designed for use with large samples. (SLD)

Bootstrap analysis, both for nonparametric statistical inference and for describing sample results stability and replicability, has been gaining prominence among quantitative researchers in educational and psychological research. Procedurally, however, it is often quite a challenge for quantitative researchers to implement bootstrap analysis in their research because bootstrap analysis is typically not an automated program option in statistical software programs. This paper uses a few heuristic analytical examples to show how bootstrap analysis can be accomplished through the use of some commonly available statistical software programs. Until bootstrap analysis becomes an automated program option in standard statistical software programs (e.g., the Statistical Package for the Social Sciences or the Statistical Analysis System), quantitative researchers may have to make do with these or other creative approaches to accomplish bootstrap analysis in their research. (Contains 4 tables, 10 figures, and 37 references.) (Author/SLD)

Despite the understandings that constructivist and multimedia didactics, as well as curricular theory and multiple intelligences theory, have been providing for years, what happens in the classroom and in the teaching process is still mostly teacher-centred. The didactic and methodological scenarios that prevail in our classes are more suitable to times long ago than to students belonging to the net-generation. The aim of this study is to examine the assessments of changes that happen in school and in classes (or those that could happen) under the influence of new didactic understandings and digital education technology. A specifically constructed questionnaire was used on a sample of primary education teachers (N = 213), which is representative of Central Croatia, since the data were collected through individual surveys in schools located in the City of Zagreb and in eight counties. The data were collected by fourth-year students of teacher education studies from Zagreb, Petrinja, and Cakovec at the beginning of 2015. The questionnaire contained 25 statements that the respondents had to assess on a Likert-type scale. The results show that teachers consider that curricular changes are needed and that classes still lack a pedagogy of success for all. However, less criticism is directed towards the present (inadequate) teaching environment.

Research studies using booklet classification were implemented by the American College Testing Program to investigate the linkage between the National Assessment of Educational Progress (NAEP) Achievement Levels Descriptions and the cutpoints set to represent student performance with respect to the achievement levels. This paper describes the process and reports the results of the booklet classification study (BCS) implemented for the science achievement levels. It explores the possibility of using booklet classification as a way to set achievement levels by investigating methodologies for computing achievement level cutpoints using booklet classification data. These methodologies were applied to BCS data for science in this study and had been applied to geography and U.S. history. The BCS for science achievement levels involved grades 4 and 8, with 13 panelists for each grade level. Eighteen booklets were selected from NAEP forms, and 22 from other sources. The BCS for science, geography, and U.S. history have all resulted in panelists' classifying student performance at a lower level than plausible values scores indicate. These results indicate that cutpoints computed from booklet classification data would be higher than cutpoints based on the item-by-item rating methods that were used operationally. Procedures using the proportional odds model and nonparametric discriminant analysis were developed as a way to compute Achievement Level cutpoints using booklet classification data. Further refinements to these procedures, especially the nonparametric discriminant analysis, are needed before they could be used operationally to set cutpoints. (Contains 15 tables, 3 figures, and 13 references.) (SLD)

The L statistic of E. B. Page (1963) tests the agreement of a single group of judges with an a priori ordering of alternative treatments. This paper extends the two group test of D. W. Leitner and C. M. Dayton (1976), an extension of the L test, to analyze difference in consensus between two unequally sized groups of judges. Exact critical values are tabulated for small numbers of treatments and judges, and a unit normal approximation is developed for larger samples. A brief computation example is also provided. Analysis of the two-tailed unit normal approximation shows that disparity between sample sizes often leads to underestimating the probability of a Type I error. However, as the number of treatments increases, the fit becomes better. This test, in comparison to two parametric competitors in a 2 by 4 mixed design, made fewer Type I errors when data were sampled from the normal, uniform, and exponential distribution, but it was also shown to be generally more conservative. The unit normal approximation is not severely affected by unequally sized groups having heterogeneous variances. The proposed test shows adequate power in conditions that do not favor parametric tests (i.e., interval level, normally distributed variables). Four tables present analysis results, and four figures illustrate the discussion. (SLD)