Search

Analyzing heterogeneous treatment effects (HTEs) plays a crucial role in understanding the impacts of educational interventions. A standard practice for HTE analysis is to examine interactions between treatment status and preintervention participant characteristics, such as pretest scores, to identify how different groups respond to treatment. This study demonstrates that the identical patterns of HTE on test score outcomes can emerge either from variation in treatment effects due to a preintervention participant characteristic or from correlations between treatment effects and item easiness parameters. We demonstrate analytically and through simulation that these two scenarios cannot be distinguished if analysis is based on summary scores alone. We then describe a novel approach that identifies the relevant data-generating process by leveraging item-level data. We apply our approach to a randomized trial of a reading intervention in second grade and show that any apparent HTE by pretest ability is driven by the correlation between treatment effect size and item easiness. Our results highlight the potential of employing measurement principles in causal analysis, beyond their common use in test construction.

The field of geometrical optics features a broad knowledge base concerning conceptual learning difficulties and student conceptions. However, in contrast to other areas of physics, no widely used standard test instrument exists to assess middle school students' conceptual knowledge of geometrical optics. This is even more true for other areas of the electromagnetic spectrum. For this reason, a concept test was developed for an intervention study to collect preconceptions about visible light as well as preconceptions about infrared radiation in the subject area of reflection and transmission. A pilot study with 108 middle school students preceded final revisions and implementation in the main study of 820 learners aged 12-16. The measurement model is presented for the developed concept test with the subscales 'preconceptions and knowledge about reflection and transmission in visible optics', 'preconceptions and knowledge about reflection and transmission in infrared optics' and 'applied knowledge on reflection and transmission in visible and infrared optics'. The results show that the concept test validly captures the development of preconceptions and knowledge regarding specific aspects of reflection and transmission of the part of the electromagnetic spectrum from 3 µm to 380 nm.

What follows is a practical guide for establishing the validity of a survey for research purposes. The motivation for providing this guide is our observation that researchers, not necessarily being survey researchers per se, but wanting to use a survey method, lack a concise resource on validity. There is far more to know about surveys and survey construction than what this guide provides; and this guide should only be used as a starting point. However, for the needs of many researchers, this guide provides sufficient, basic information on survey validity. The guide, furthermore, includes references to important handbooks for researchers needing further information.

This study discusses the need, development, and validation of the Innovation Test Instrument (ITI). This article outlines how the researchers identified the content domain of the assessment and created test items. Then, it describes initial validation testing of the instrument. The findings suggest that the ITI is a good first step in creating an innovation assessment because it is more inclusive of both divergent and convergent thinking. In comparison, past innovation assessments have only assessed either divergence or convergence. The ITI still needs further validation and improvement to make strong claims about its ability to determine the effectiveness of an innovation course.

We revisit the merits of the retrospective pretest-posttest (RPP) design for repeated-measures research. The underutilized RPP method asks respondents to rate survey items twice during the same posttest measurement occasion from two specific frames of reference: "now" and "then." Individuals first report their current attitudes or beliefs following a given intervention, and next they are prompted to think back to a specific time prior to the given intervention and rate the item again retrospectively. The design addresses many of the validity concerns that plague the traditional pretest-posttest design. Particularly when measuring noncognitive constructs, the RPP design allows participants to gauge the degree of change that they experience with greater awareness and precision than a traditional approach. We review the undesirable features of traditional designs and highlight the benefits of the retrospective approach. We offer examples from two recent, original studies and conclude with the recommendation that the RPP design be employed more broadly. We also conclude with a discussion of important directions for future examination of this design.

This study aims to develop Taiwan undergraduates' volunteer service motivation scale. To begin with, item pool was proposed on the basis of literature. After discussing with three Taiwan undergraduates, item pool, exploratory factor analysis (EFA) (N = 150) was proceeded, where three tests were conducted EFA: 1. Item analysis: comparisons of extreme groups and correlation analysis were employed. The former's critical ratios are all more than 3, and reach significant standard, showing the 20 items have discriminating power. The latter shows that the r value of all items and the total score are all more than .60, and reach significant standard (p<0.001), showing the items can measure the same concept. 2. Factor analysis: primary axis factoring and the promax method of oblique rotation were adopted. In the first EFA, the 5 items concerning interpersonal relationship were distributed to other factors; and, due to incompatibility with other factors, so this dimension of Interpersonal Relationship was deleted. The 2nd EFA's results are consistent with the original design, which is categorized into Self-Growth, Serving Others, and Social Identification, and has appropriate total variation and total reliability. In order to verify the stability of the EFA factors, another batch of samples (N = 452) were investigated through confirmatory factor analysis (CFA). In accordance with the CFA's results, the initial model's test are acceptable without offending estimations, the basic fit of the initial model is in good fitness (including individual reliability, composite reliability, the average variance extracted, construct discrimination). To conclude, these 15 items 3 factors of Taiwan Undergraduates' Volunteer Service Motivation Scale has acceptable reliability and validity; therefore, it can serve as the scale to investigate Taiwan undergraduates' volunteer service motivation for proceeding the relative studies.

Maintaining score interchangeability and scale consistency is crucial for any testing programs that administer multiple forms across years. The use of a multiple linking design, which involves equating a new form to multiple old forms and averaging the conversions, has been proposed to control scale drift. However, the use of multiple linking often conflicts with the need for minimizing old item/form exposure and the need for pretesting. This study tried to find a balance point where the needs for equating, item/form exposure and controlling, and pretesting can be satisfied. Three equating scenarios were examined using real data: equating to one old form, equating to two old forms, or equating to three old forms. The finding is that the equating based on one old form produced persistent score drift and also showed increased variability in score means and standard deviations over time. In contrast, equating back to two or three old forms produced much more stable conversions and had less variation. Overall, equating based on multiple linking designs shows the promise of producing more consistent results and preventing scale drift. We recommend that testing programs and practitioners consider the use of multiple linking whenever possible.

The original SOCO Scale was reduced to 10 items by Thomas, Soutar, and Ryan (2001). The author conducted a pretest and a posttest in his Personal Selling class during the Fall 2009 semester. Significant differences by gender, student sales experience and family member in the sales field were identified. The author once again pretested the shortened scale in his Spring 2010 Personal Selling class; however, spousal health problems derailed the planned posttest. Significant differences by gender and student sales experience were found. Paired t-test findings and comparison of the two data sets data are also addressed in this paper.

As specialized instructional support personnel begin learning and using motivational interviewing (MI) techniques in school-based settings, there is growing need for context-specific measures to assess initial MI skill development. In this article, we describe the iterative development and preliminary evaluation of two measures of MI skill adapted from the substance abuse field for use in school-based settings. We developed the Video Assessment of Simulated Encounters for School-Based Applications and the Written Assessment of Simulated Encounters for School-Based Applications to evaluate the initial MI skill development of school-based personnel participating in a multi-component, MI training program. Preliminary psychometric evidence supports continued development and refinement of the measures. [This article was published in "Advances in School Mental Health Promotion" (EJ1088365).]

This paper addresses the use of computer-based testing in distance education, based on the experience of Universitas Terbuka (UT), Indonesia. Computer-based testing has been developed at UT for reasons of meeting the specific needs of distance students as the following: (1) students' inability to sit for the scheduled test; (2) conflicting test schedules; and (3) students' flexibility to take examination to improve their grades. In 2004, UT initiated a pilot project in the development of system and program for computer-based testing method. Then in 2005 and 2006 tryouts in the use of computer-based testing methods were conducted in 7 Regional Offices that were considered as having sufficient supporting resources. The results of the tryouts revealed that students were enthusiastic in taking computer-based tests and they expected that the test method would be provided by UT as alternative to the traditional paper and pencil test method. UT then implemented computer-based testing method in 6 and 12 Regional Offices in 2007 and 2008 respectively. The computer-based testing was administered in the city of the designated Regional Office and was supervised by the Regional Office staff. The development of the computer-based testing was initiated with conducting tests using computers in networked configuration. The system has been continually improved, and it currently uses devices linked to the internet or the World Wide Web. The construction of the test involves the generation and selection of the test items from the item bank collection of the UT Examination Center. Thus the combination of the selected items compromises the test specification. Currently UT has offered 250 courses involving the use of computer-based testing. Students expect that more courses are offered with computer-based testing in Regional Offices within easy access by students.

This paper aims to extend understanding of the business and societal impact of academic research. From a business school perspective, it has taken stock of the role of academic research and relevance in business and society. The proposed conceptual framework highlights the forces influencing the pursuit of academic rigour and relevance in scholarly outputs. A theoretical model for measuring the societal impact of academic journal articles--the Academic Rigour and Relevance Index (AR2I)--was developed. This index comprises six key parameters, which are assessed by three stakeholder groups connected with academic research into business issues, these groups being: business practitioners, society and academics. The behaviour of the AR2I model was evaluated using the Monte Carlo simulation model. Taking into account the relationships between the standard deviations and the differences of classification between articles with different levels of rigour and relevance, it is demonstrated that the AR2I model is an effective tool.

Cognitive interviewing is a method to develop culturally appropriate survey questions and scale items. We conducted two rounds of cognitive interviews with 24 pregnant women in Ethiopia and Kenya to assess the appropriateness, acceptability, and comprehension of general and micronutrient supplement adherence-specific social support scales. We stopped the first round of cognitive interviews after receiving negative feedback from interviewers and participants about their distressing and uncomfortable experiences with cognitive probes and challenges related to cultural perspectives on social support. Through an iterative process, we made substantial changes to the cognitive interview guides and items from both social support scales. In the second round, the revised cognitive interviewing process substantially improved interviewer and participant experiences and increased comprehension and appropriateness of both social support scales. This study confirms the importance of cultural adaptation of the cognitive interviewing process as well as social support scales.

The primary measure used to determine relative effectiveness of in-class activities has been student performance on pre/posttests. However, in today's active-learning classrooms, learning is a social activity, requiring students to interact and learn from their peers. To develop effective active-learning exercises that engage students, it is important to gain a more holistic view of the student experience in an active-learning classroom. We have taken a mixed-methods approach to iteratively develop and validate a 16-item survey to measure multiple facets of the student experience during active-learning exercises. The instrument, which we call Assessing Student Perspective of Engagement in Class Tool (ASPECT), was administered to a large introductory biology class, and student responses were subjected to exploratory factor analysis. The 16 items loaded onto three factors that cumulatively explained 52% of the variation in student response: (1) value of activity, (2) personal effort, and (3) instructor contribution. ASPECT provides a rapid, easily administered means to measure student perception of engagement in an active-learning classroom. Gaining a better understanding of students' level of engagement will help inform instructor best practices and provide an additional measure for comprehensively assessing the impact of different active-learning strategies.

The development of questionnaires, surveys and psychometric scales is an iterative research process that includes a number of carefully planned stages. Pretesting is a method of checking that questions work as intended and are understood by those individuals who are likely to respond to them. However, detailed reports of appropriate methods to undertake pretesting are currently underrepresented within the literature. This study presents a detailed protocol of a cognitive interview pretesting approach that informed the development of the Exercise Referral Quality of Life Scale -- a measure of life-quality designed specifically for structured clinical exercise settings. This documented approach to pretesting proved a vital stage in the scale development process, without which the item problems detected would have carried forward into the statistical analyses. The current protocol intends to contribute to reducing the current shortfall in pretesting guidance for practitioners and researchers.

Few studies have explored the subjective mental health of adolescents with intellectual disabilities, while proxy ratings indicate an overrepresentation of mental health problems. The present study reports on the design and an initial empirical evaluation of the Well-being in Special Education Questionnaire (WellSEQ). Questions, response scales, and an application for tablet PCs were developed in cooperation with students and teachers in special education schools. One hundred and thirteen students (age 12-16) and their parents and teachers participated. Positive results in terms of test-retest reliability, internal consistency of scales, and response rates were obtained. Level of reading appeared to affect the students' understanding of items. Teachers' and parents' ratings on the WellSEQ correlated well with established measures. Correlations between proxy ratings and students' reports varied. The questionnaire and technology of WellSEQ may enable students with IDD to participate independently in research with good completion rates and reliable responding.

Problem solving is a complex process valuable in everyday life and crucial for learning in the STEM fields. To support the development of problem-solving skills it is important for researchers and curriculum developers to have practical tools that can measure the difference between novice and expert problem-solving performance in authentic classroom work. It is also useful if such tools can be employed by instructors to guide their pedagogy. We describe the design, development, and testing of a simple rubric to assess written solutions to problems given in undergraduate introductory physics courses. In particular, we present evidence for the validity, reliability, and utility of the instrument. The rubric identifies five general problem-solving processes and defines the criteria to attain a score in each: organizing problem information into a Useful Description, selecting appropriate principles (Physics Approach), applying those principles to the specific conditions in the problem (Specific Application of Physics), using Mathematical Procedures appropriately, and displaying evidence of an organized reasoning pattern (Logical Progression).

The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students' inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.

Are South Africans financially literate, and how can this be measured? Until 2009 there was no South African financial literacy measure and, therefore, the aim was to develop a South African measurement instrument that is scientific, socially acceptable, valid and reliable. To achieve this aim a contextual and conceptual analysis of financial literacy that indicated the importance of financial literacy, the scope and impact of financial literacy education, and uncovered an acceptable financial literacy definition and its constituent concepts, was applied. A rigorous five-step process was then followed in developing a questionnaire that measures financial literacy knowledge, behaviour and attitude. This draft questionnaire was applied at the South African Military Academy (SAMA) to firstly determine and improve its validity and reliability, and secondly to measure the financial literacy levels of school leavers. Experts and users found this measurement instrument to be valid, and internal consistency levels of above 0.7 registered its reliability. On average the first-year SAMA students achieved scores of 55.55%, 69.85%, and 77.11% for financial literacy knowledge, behaviour and attitude. As a result it is postulated that there is now a scientific and socially relevant, valid and reliable South African financial literacy measurement instrument available.

The purpose of this study was to develop a scoring rubric to assess students' manipulation skills and identify students' learning difficulties in conducting organic chemistry experiments. In constructing the scoring rubric, we first analyzed the skills needed in the experiment, then divided the skills into subskills, and finally constructed a rubric for each subskill. We applied this developed instrument to on-site evaluation of 55 second-year students by eight teaching assistants (TAs). The data were analyzed using one-way analysis of variance, which compared the students' mean score on each skill at the 0.05 significance level. Results showed that the skill of gravity filtration and suction filtration needed to be improved, as the students had the lowest mean scores in these two skills. On-site observations further indicated that only 5% of the students correctly performed the rinsing technique for gravity filtration. We also found that TAs became more aware of students' experimental skills after using the rubric for assessment. By using a scoring rubric as an assessment tool, students' learning difficulties could be easily identified and TAs could improve their teaching effectiveness by checking student performance and modifying their instructional strategies accordingly. This study provides a valuable and easy-to-apply quantitative assessment for use in the organic chemistry laboratory.

With the increasing number of preschool-age children of Latino heritage entering U.S. schools comes a growing need to accurately determine children's individual needs and identify potential disabilities, beginning with the screening process. Unfortunately, teachers face many challenges when screening English language learners. Often, parents have important information that can contribute to teachers' understanding of children's development, but there are limited tools available to collect information from parents. The Formulario Familiar Bilingue de Información Formulario y Observación/Family Bilingual Information & Observation (BIO) Questionnaire is a new parent report tool that can be used to collect information from parents of Latino heritage concerning their child's language development history, current language usage, and exposure to Spanish and English. A multiphase research study was conducted to collect formative data and to establish construct validity for the questionnaire. Results from a focus group, pilot study, and face validity review provided numerous suggestions to improve the tool, and established that the BIO is a viable means to collect parent perspectives on their own children's language development.

We report on the origins, development and refinement of an online inventory to help cooperating teachers focus on selected dimensions of their practice. The Mentoring Profile Inventory (MPI) helps quantify important features of both the motivating and challenging aspects of mentoring student teachers and provides results to respondents in a graphic, easy-to-understand and immediate feedback report (14 sub-scales and 3 summary charts). Psychometric properties of the MPI are shown to be robust. Results can be used individually or collectively to facilitate cooperating teacher professional development by providing the opportunity for dialog around a set of common issues. (Contains 2 figures and 7 tables.)

Background: All national curriculum tests in England are pre-tested as part of the development process. Differences in pupil performance between pre-test and live test are consistently found. This difference has been termed the pre-test effect. Understanding the pre-test effect is essential in the test development and selection processes and in the setting of cut scores for the national curriculum tests in England. This study looked at the national science tests for 11-year-olds in England (termed key stage 2). Purpose: The aim of this study was to investigate whether the pre-test effect differs when the item data is divided in three ways: by pupil level, item level and item type. Sample: Second pre-test data was captured from Year 6 (aged 11) pupils (2006: n = 1010 for test A, n = 1002 for test B; 2007: n = 895 for test A, n = 889 for test B) from a representative population of state-funded schools in England as defined by school type, size of year group, region and achievement. Live test data was captured as part of the Monitoring Curriculum and Assessment Project where over 1000 nationally representative schools in England taking part in a study for the Centre for Formative Assessment Studies (CFAS) sent in the live test results from a sample of Year 6 pupils (2006: n = 2732 for both tests A and B; 2007: n = 2156 for both tests A and B). Design and methods: The study looked at item level data from pre-test 2 and the live test for each of the years 2006 and 2007 for the key stage 2 science tests and compared the performance of the items by grouping the data in the three specific ways: by pupil level, item level and item type. Average facilities were calculated for the various subsets of items and pupils, and the differences between the groupings were compared using either t-tests or analysis of variance (ANOVA). Results: Higher achieving pupils show a significantly greater pre-test effect than lower achieving ones, while the lowest achieving pupils do not show a significant pre-test effect at all. Items that are of the greatest difficulty show a greater pre-test effect than items that are easier. Items that are open response and require a short answer show a greater pre-test effect than open response items requiring a long answer. These latter item types in turn show a greater pre-test effect than closed response items. Conclusions: The study questions whether knowledge of test characteristics such as item level, item type and knowledge about the sample (i.e. levels of pupils) should affect the pre-test effect used in level setting. Item levels and pupil abilities are already accounted for in current pre-test effect predictions by means of test and sample specifications. Proportions of items of different formats are not specified, however. As the study showed a different pre-test effect for different item types, it is suggested there is consideration of the make-up of the tests with regard to item types during the level setting process. (Contains 7 figures, 2 tables and 3 notes.)