Search

In order to integrate technology into the language curriculum, it is essential not only to evaluate a program within a second language acquisition (SLA) framework but also to seek the students' attitudes towards the use of computer-assisted language learning (CALL) materials. Guided by Chapelle's (2001) framework for CALL evaluation, this paper reports on a 2-year empirical study of intermediate level learners' perceptions of the use of a web-based multimedia program with authentic video clips and its effectiveness as a language-learning tool. Students believed that the learned culture and vocabulary enhanced their listening skills. A pretest/posttest design assessed the long-term gains in listening and speaking performances. Results indicated a significant gain in listening and speaking. The paper concludes by highlighting what lessons to consider when integrating CALL into language courses. (Contains 7 tables, 1 figure and 2 notes.)

Multi-User Virtual Environments (MUVEs) are the subject of increasing interest for educators and trainers. This article reports on a longitudinal project that seeks to establish a virtual agile software development workshop hosted in the Open Wonderland MUVE, designed to help learners to understand the basic principles of some core agile software development techniques. The authors take a design-based research approach that allows for cycles of learning and reflection that has enabled following an iterative path of development through two major iterations. The authors trace the research path followed from the real world workshop to its virtual incarnation, describing the design philosophy and the constructed virtual artefacts. The authors conclude by reflecting on the insights into learner perceptions and practical implementations gained from building and evaluating the virtual workshop.

We report on the origins, development and refinement of an online inventory to help cooperating teachers focus on selected dimensions of their practice. The Mentoring Profile Inventory (MPI) helps quantify important features of both the motivating and challenging aspects of mentoring student teachers and provides results to respondents in a graphic, easy-to-understand and immediate feedback report (14 sub-scales and 3 summary charts). Psychometric properties of the MPI are shown to be robust. Results can be used individually or collectively to facilitate cooperating teacher professional development by providing the opportunity for dialog around a set of common issues. (Contains 2 figures and 7 tables.)

This study analyzed data from four evaluation designs incorporating the retrospective pretest (i.e., thentest), analyzing the effects of self-report pretesting and post-program survey format on a set of self-report measures. Validity of self-report data was assessed by comparing the criterion validity of then ratings to established benchmarks and by including a control condition. This study found that designs incorporating separate posttest and thentest surveys yielded the most comparable levels of criterion validity for then ratings and the least biased measures of program effectiveness. Conversely, designs that incorporated a single post-program survey, with adjacent post and then items, yielded the least comparable levels of criterion validity for then ratings and the most biased measures of program effectiveness. This study also found a slight interaction between the effects of self-report pretesting and post-program survey format. Implications for program evaluation are discussed. (Contains 3 figures and 4 tables.)

This article presents a method to evaluate pretest effects on posttest scores in the absence of an un-pretested control group using published results of pretesting effects due to Willson and Putnam. Confidence intervals around the expected theoretical gain due to pretesting are computed, and observed gains or differential gains are compared with the original reported results. Examples from the "American Educational Research Journal" and the "Journal of Educational Psychology" are presented that illustrate both support for the original interpretation and evidence for reinterpretation of the results as statistically significant or nonsignificant opposite to the original conclusion. Implications for using the results of meta-analyses in substantive research are discussed. (Contains 4 tables and 2 figures.)

In this qualitative pilot study, the authors report on curriculum field trials within a personal learning environment (PLE) designed by a collaboration of academic researchers and nonprofit volunteers working together in the virtual world of Second Life. The purpose of the PLE is to provide learners less likely to have access to educational opportunities with a means to create a "new life" in the real world, through a basic web-based curriculum and an advanced Second Life curriculum. Field trials of the Second Life curriculum were held with youth from underserved populations (n=6) to identify participant characteristics that facilitate success with the curriculum. Performance on instructional outcomes was examined in addition to a participatory action research methodology (PAR) that was employed with participants as co-researchers. To protect identities, the authors use a case study approach to track one composite participant/co-researcher through the curriculum.

We discuss collecting data simultaneously from many different instruments, including both qualitative and quantitative sources. A list of instruments used is provided, and data collection methods are described. Many practical logistical concerns are discussed, warning of potential pitfalls to be avoided. This article is intended to be a practical guide for other research groups wishing to use data from multiple sources, especially if they have not collected qualitative data before. (Contains 2 tables and 1 figure.)

Background: All national curriculum tests in England are pre-tested as part of the development process. Differences in pupil performance between pre-test and live test are consistently found. This difference has been termed the pre-test effect. Understanding the pre-test effect is essential in the test development and selection processes and in the setting of cut scores for the national curriculum tests in England. This study looked at the national science tests for 11-year-olds in England (termed key stage 2). Purpose: The aim of this study was to investigate whether the pre-test effect differs when the item data is divided in three ways: by pupil level, item level and item type. Sample: Second pre-test data was captured from Year 6 (aged 11) pupils (2006: n = 1010 for test A, n = 1002 for test B; 2007: n = 895 for test A, n = 889 for test B) from a representative population of state-funded schools in England as defined by school type, size of year group, region and achievement. Live test data was captured as part of the Monitoring Curriculum and Assessment Project where over 1000 nationally representative schools in England taking part in a study for the Centre for Formative Assessment Studies (CFAS) sent in the live test results from a sample of Year 6 pupils (2006: n = 2732 for both tests A and B; 2007: n = 2156 for both tests A and B). Design and methods: The study looked at item level data from pre-test 2 and the live test for each of the years 2006 and 2007 for the key stage 2 science tests and compared the performance of the items by grouping the data in the three specific ways: by pupil level, item level and item type. Average facilities were calculated for the various subsets of items and pupils, and the differences between the groupings were compared using either t-tests or analysis of variance (ANOVA). Results: Higher achieving pupils show a significantly greater pre-test effect than lower achieving ones, while the lowest achieving pupils do not show a significant pre-test effect at all. Items that are of the greatest difficulty show a greater pre-test effect than items that are easier. Items that are open response and require a short answer show a greater pre-test effect than open response items requiring a long answer. These latter item types in turn show a greater pre-test effect than closed response items. Conclusions: The study questions whether knowledge of test characteristics such as item level, item type and knowledge about the sample (i.e. levels of pupils) should affect the pre-test effect used in level setting. Item levels and pupil abilities are already accounted for in current pre-test effect predictions by means of test and sample specifications. Proportions of items of different formats are not specified, however. As the study showed a different pre-test effect for different item types, it is suggested there is consideration of the make-up of the tests with regard to item types during the level setting process. (Contains 7 figures, 2 tables and 3 notes.)

This paper provides an example of the application of the cognitive interview, a qualitative tool for pre-testing a survey instrument to check its cognitive validity, that is, whether the items mean to respondents what they mean to the item designers. The instrument is the survey used in the final phase of the International Study of Principal Preparation (ISPP http://www.ucalgary.ca/~cwebber/ISPP/index.htm). This study involving researchers in 13 countries investigates those aspects of principals' work perceived by them to be most challenging in their early years in the position and the extent to which principals believe they were prepared for these challenges. Both probing and think aloud approaches to the cognitive interview were used and revealed a small number of items requiring amendment.

Distance education has dramatically increased the educational opportunities for underserved populations. This is evidenced by the fairly recent proliferation and success of online universities, and it is happening in spite of criticism of e-learning and online degree programs among academia and the popular press. As a result, many institutions, including traditional and respected universities across the country--who were once among the harshest critics--are now offering distance education courses. However, not all students who wish to enroll in online courses are ready or prepared to be successful completing online coursework. Research suggests that at least half of all students enrolled in online courses are not course completers. Obviously, increasing the percentage of program completers in the distance learning environment is essential. This article details a proven and comprehensive system to achieve this objective, using the authors' experiences at Purdue University Calumet in Hammond, Indiana, as a framework. (Contains 1 note.)

As part of an NSF funded project we developed new course materials for a general introductory statistics course designed to engage students in statistical discovery. The materials were designed to actively involve students in the design and implementation of data collection and the analysis and interpretation of the resulting data. Our overall goal was to have students begin to think like statisticians, to construct ways of thinking about data collection and analysis, to solve problems using data in context. During their development, the materials and related activities were field tested in a small special section of an introductory statistics course for two semesters. This field testing was a "proof of concept," that is that the materials could work in the laboratory setting and that the materials showed promise for improving students' learning. As a first step in evaluating these materials, students who enrolled in regular sections of the introductory course were used as a comparison group. In this paper, the development and use of the course materials will be discussed briefly. The strategy for evaluating the materials while they were being developed and analysis of students' performance on common assessment questions and the course project will be presented. In addition, the relationship between student attitudes toward statistics and students' performance will be examined. (Contains 17 tables.)

Spontaneous transfer of learning is often difficult to elicit. This finding may be widespread partly because pretests proactively interfere with transfer. To test this hypothesis, 7-year-olds' transfer was examined across 2 numerical tasks (number line estimation and categorization) in which similar representational changes have been observed. As predicted, children given feedback on numerical estimates learned to use a linear representation of numerical quantity instead of a logarithmic one, but providing practice on a categorization pretest led children to continue using a logarithmic representation on the same task, which they otherwise abandoned with surprising frequency. These findings imply unsupervised practice of inappropriate representations impedes transfer, and studies of learning can greatly underestimate children's potential for transfer if pretest effects are uncontrolled.

In the past decades, English learning has received lots of attention all over the world, especially for those who are not native English speakers. Various English learning and testing systems have been developed on the Internet. Nevertheless, most existing English testing systems represent the learning status of a student by assigning that student with a score or grade. This approach makes the student aware of his/her learning status through the score or grade, but the student might be unable to improve his/her learning status without further guidance. In this paper, an intelligent English tense learning and diagnostic system is proposed, which is able to identify student learning problems on English verb tenses and to provide personalized learning suggestions in accordance with each student's learning portfolio. Experimental results on hundreds of college students have depicted the superiority of the novel approach.

This article presents an approach and tools that can help learners appropriate a Web-based learning curriculum and become active participants in their learning. The approach is based on a detailed modeling of the curriculum and intends to equip the learners with different computer-based tools facilitating a multiple point of view perception of the curriculum, while promoting self evaluation and self regulation of the learners' curriculum performance. The proposed architecture is generic and can be used in the context of an already existing Web-based learning system. We define what we call "appropriation," describe our approach, present different tools that have been implemented, and present the findings from the first experiments.

We investigated the use of quizzes administered prior to lecture (i.e., prelecture quizzes) and compared them to no-quiz control groups. In previous research, the success of administering quizzes after covering a topic (i.e., postlecture quizzes) was contingent on the quizzes and the subsequent exams being of similar level and content. However, our study revealed that such contingencies were not true of prelecture quizzes. Students who received either matching or fill-in-the-blank prelecture quizzes, as compared to no quiz, performed better on both multiple-choice and essay exam questions. They also rated the lectures as a better preparation for exams and as more clear and organized. Finally, the quizzed students asked more high-level questions.

Cognitive pretesting (CP) is an interview methodology for pretesting the validity of items during the development of self-report instruments. This article reports on the development and evaluation of a systematic method to rate self-report item validity performance utilizing CP interview text data. Five raters were trained in the application of that system, resulting in acceptable to substantial levels of interrater agreement. Results from this study suggest that excellent reliability can be achieved in the analysis of CP data. Guidelines for systematically rating the qualitative data collected using CP methods are provided. Future research should focus on empirical demonstrations of how such rating procedures can lead to improvements in the validity of self-report instruments. (Contains 3 tables and 1 figure.)

Proper design analysis is indispensable to assure quality and reduce emergent costs due to faulty software. Teaching proper design verification skills early during pedagogical development is crucial, as such analysis is the only tractable way of resolving software problems early when they are easy to fix. The premise of the presented strategy is based on the observation that fundamental component of any engineering curriculum is the use of formal and sound techniques that facilitate analysis of artifacts produced by students. Yet, fundamental roadblocks exist in bringing the state of the art in design analysis to the classroom due to the steep learning curve and quagmire of theoretical details involved in formal methods. This paper suggests a strategy and tool support that promotes the attainment of design evaluation skills. We also discuss how selective and pragmatic application of formal methods can be used in software design education.

The purpose of the present study was to examine a common practice in some areas of program evaluation, the retrospective pretest, and to present recommendations regarding its use. The authors review literature to emphasize first, that bias is likely in both prospective and retrospective ratings, and second, that under some circumstances, retrospective pretests may introduce greater bias than traditional pretests. The authors examined data from 100 participants who attended a family program at 15 sites. Results supported the hypothesis that items on which parents were asked to endorse socially desirable parenting behaviors resulted in greater discrepancies between prospective and retrospective ratings. Effect sizes calculated from prospective ratings were comparable with published effect sizes from research trials of the same program. The authors conclude that replacing traditional with retrospective pretests does not eliminate bias. They recommend traditional pretests for examination of program effects and retrospective pretests for examination of subjective experiences of program-related change.

In the context of the importance of valid self-report measures to research and evidence-based practice in social work, an argument-based approach to validity is presented and the concept of developmental validity is introduced. Cognitive development theories are applied to the self-report process of children, and cognitive pretesting is reviewed as a methodology to advance the validity of self-report instruments for children. An application of cognitive pretesting is presented in the development of the Elementary School Success Profile. Two phases of cognitive pretesting were completed to gather data about how children read, interpret, and answer self-report items. Cognitive pretesting procedures identified validity problems with numerous items leading to modifications. Cognitive pretesting framed by an argument-based approach to validity holds significant potential to improve the developmental validity of child self-report instruments. (Contains 1 figure and 1 table.)

A new procedure, CATSIB, for assessing differential item functioning (DIF) on computerized adaptive tests (CATs) is proposed. CATSIB, a modified SIBTEST procedure, matches test takers on estimated ability and controls for impact-induced Type 1 error inflation by employing a CAT version of the IBTEST "regression correction." The performance of CATSIB in terms of detection of DIF in pretest items was evaluated in a simulation study. Simulated test takers were adaptively administered 25 operational items from a pool of 1,000 and were linearly administered 16 pretest items that were evaluated for DIF. Sample size varied from 250 to 500 in each group. Simulated impact levels ranged from a 0- to 1-standard-deviation difference in mean ability levels. The results showed that CATSIB with the regression correction displayed good control over Type 1 error, whereas CATSIB without the regression correction displayed impact-induced Type 1 error inflation. With 500 test takers in each group, power rates were exceptionally high (84% to 99%) for values of DIF at the boundary between moderate and large DIF. For smaller samples of 250 test takers in each group, the corresponding power rates ranged from 47% to 95%. In addition, in all cases, CATSIB was very accurate in estimating the true values of DIF, displaying at most only minor estimation bias.