30,245 results
Search Results
2. New Visions in Action: National Assessment Summit Papers
- Author
-
National K-12 Foreign Language Resource Center and Rosenbusch, Marcia Harmon
- Abstract
The National Assessment Summit was designed to bring together users and producers of assessments and engage them in a discussion of unmet assessment needs and untapped assessment capacities. The meeting was truly a summit in that national foreign language organizations and associations were asked to nominate representatives to bring their organization's perspective to the forum and also to take back to their organization ideas for action. Specifically, the National Assessment Summit aimed to identify a set of priorities for the next two years (2005-07). At a follow-up National Assessment Summit in 2007 in Portland, Oregon, under the sponsorship of the Center for Applied Second Language Studies (CASLS) at the University of Oregon and the Center for Advanced Research in Language Acquisition (CARLA) at the University of Minnesota, these organizations will gather to report on progress made and to consider an agenda for the following two years (2007-09). The National Assessment Summit identified seven priorities for the coming two years. Each priority is addressed in a section of this series of papers and is authored by one of the participants. These priorities are: (1) Articulation; (2) Assessment Literacy; (3) Suite of Assessments; (4) Test Database; (5) Models and Anchors; (6) Research; and (7) Advocacy. Taken as a whole, these papers describe state-of-the-art language assessment and articulate a two-year agenda for improving assessment and educational practice. This publication includes: (1) Introduction (Carl Falsgraf); (2) Achieving Articulation through Assessment (Jacque Bott Van Houten); (3) Assessment Literacy (Peggy Boyles); (4) Suite of Assessments (Ann Tollefson); (5) Foreign Language Assessment Database (Margaret E. Malone and David W. MacGregor); (6) Models and Anchors (Elvira Swender); (7) Assessments Research (Ursula Lentz); (8) Promoting Language Advocacy Through Assessment (Martha G. Abbott); and (9) Conclusion (Paul Sandrock). Individual papers contain references and footnotes.
- Published
- 2006
3. An Analysis of Self-Reported Graduates. Technical Paper
- Author
-
National Centre for Vocational Education Research and Braysher, Ben
- Abstract
The annual Student Outcomes Survey collects information on the outcomes of two groups of students--those that have completed a qualification (graduates) and those that have completed only part of a course and then left the vocational education and training (VET) system (module completers). At the time of selecting the survey sample, insufficient information is available to identify "actual" module completers. Instead, a sample of potential module completers is drawn that includes students still in training and students who self-report that they have completed the qualification; that is, self-reported graduates. For many years these students have been counted as graduates in survey outputs. This paper examines whether: (1) self-reported graduates were eligible for the qualifications they claimed (in fact, around two-thirds were not eligible); and (2) it is possible to predict a self-reported graduate's eligibility for the claimed qualification using their personal and training characteristics. The report recommends that self-reported graduates should be assigned to the graduate or module completer category using a predictive model. This model incorporates information on the type of training provider, field of education, level of qualification, whether an apprentice or trainee, enrolment type, and method of answering the survey. The current practice of treating self-reported graduates biases the survey results. The predictive model will be used for 2012 Student Outcomes Survey reporting. The methodology will change estimates from previous surveys substantially, so these will be back cast to 2005 using the method described. Appended are: (1) Characteristics of self-reported graduates; (2) Variables fitted and their status in the final logistic model; (3) Regression results; (4) Comparison of self-reported graduates' eligibility for the claimed qualification; and (5) Comparison of survey outputs and modelled estimates. (Contains 16 tables and 1 footnote.)
- Published
- 2012
4. Mapping Adult Literacy Performance. Background Paper
- Author
-
National Centre for Vocational Education Research, Circelli, Michelle, Curtis, David, and Perkins, Kate
- Abstract
Language, literacy and numeracy are necessary for greater workforce participation, productivity and social inclusion. Being able to measure the level of proficiency in these skills, and any changes in the level of skills, is important for getting a sense of how well language, literacy and numeracy programs are working. Two measurement tools used in Australia are the Adult Literacy and Life Skills (ALLS) survey and the Australian Core Skills Framework (ACSF). This paper outlines a National Centre for Vocational Education Research study which will investigate whether it is possible to map the performance levels of the ALLS to the performance levels of the ACSF in order to gain more frequent information than is currently available about the literacy and numeracy progression of adult learners against national goals. Appended are: (1) About the ALLS survey and the ACSF; and (2) Delphi method and descriptions of ALLS items. (Contains 1 table and 9 footnotes.)
- Published
- 2011
5. Respecting and Fulfilling the Right of Post-Primary Pupils to Consent to Participate in Trials and Evaluative Research: A Discussion Paper
- Author
-
Maguire, Lisa K., Byrne, Bronagh, and Kehoe, Susan
- Abstract
This paper provides an introduction to issues surrounding the participation rights of young people in research and the implications of their growing involvement in research as well as providing a discourse on the ethical implications related to consent. The unique contribution of this paper is that it considers children's rights in respect to the increasing opportunities for young people to take part in evaluation research. The aim of this paper, therefore, is to acknowledge the growing involvement for young people in research and the implications of ensuring that their rights of participation are respected. Secondly, we will consider the children's rights legislation and our obligations as researchers to implement this. Finally, we will explore consent as an issue in its own right as well as the practicalities of accessing participants. This paper will postulate that any research about young people should involve and prioritize at all stages of the research process; including participation in decision-making. We conclude by identifying five key principles, which we believe can help to facilitate the fulfilment of post-primary pupils' ability to consent to participate in trials and evaluative research.
- Published
- 2018
- Full Text
- View/download PDF
6. Empirical Issues in the Design of Group-Randomized Studies to Measure the Effects of Interventions for Children. MDRC Working Papers on Research Methodology
- Author
-
MDRC, Bloom, Howard, Zhu, Pei, and Jacob, Robin
- Abstract
This paper provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children. To do so, the paper: (1) provides new empirical information about the values of parameters that influence the precision of impact estimates (intra-class correlations and R-squares); (2) examines the implications of planning group-randomized studies for three-level hierarchical situations, using empirical information obtained by estimating two-level hierarchical models (which under many conditions appears to not be problematic); and (3) assesses the implications of the uncertainty that exists when the design of group-randomized studies is based on estimates of intra-class correlations. Data for the paper come from two studies: the Chicago Literacy Initiative: Making Better Early Readers study (CLIMBERs) and the School Break-fast Pilot Project (SBPP). The analysis sample from CLIMBERs comprised 430 4-year old children from 47 preschool classrooms in 23 Chicago public schools. The analysis sample from the SBPP study comprised 1,151 third-graders from 233 classrooms in 111 schools in six school districts. Four appendixes are included: (1) Description of Outcome Measures; (2) Definition of the Multiplier M; (3) Complete Set of Results for Three-Level vs. Two-Level Model Comparisons: Nonstandardized Unconditional Variance Components; and (4) Proofs of the Relationship between Three-Level Models and Two-Level Models in Terms of Precision. (Contains 20 footnotes, 1 figure and 14 tables.) [This document is also sponsored by Judith Gueron Fund for methodological Innovation in Social Policy Research at MDRC, and the Jerry Lee foundation.]
- Published
- 2008
7. Designing and Testing Evaluation Tools to Inform Foundation Education Reforms and Build Grantee Capacity. WCER Working Paper No. 2010-1
- Author
-
Wisconsin Center for Education Research, Kimball, Steven M., Lander, Rachel, and Thorn, Christopher A.
- Abstract
Beginning in 2002, The Chicago Community Trust embarked on an ambitious grant- making strategy to improve education outcomes primarily in the city of Chicago and Cook County, Illinois. Known as The Education Initiative, this effort focused on three priority grant areas: literacy, professional development, and alternative models of schools. In June, 2005 (the 3rd year of the initiative), the Trust awarded a contract to researchers at the University of Wisconsin-Madison, Wisconsin Center for Education Research (WCER) to assess the impact of the Education Initiative. During the 3rd year of the evaluation, the program officers and evaluators were especially interested in pilot testing evaluation guidelines that were developed as part of the overall evaluation of the Education Initiative. These guidelines and related evaluation reporting tools were intended to help grantees tailor projects to the foundations priorities, and help the foundation assess the impact of their diverse grant activities. The authors report on the design and pilot testing of the evaluation guidelines and reporting tools developed to support the education program of the Trust. The next section describes the impetus for developing these tools and the evaluation findings that informed their design. In the fourth section, the authors briefly describe the pilot study design and the participants. They also report findings from the pilot study. In the final section, they describe implications for use of the guidelines and tools by the Trust and other foundations. Appendices include: (1) Draft CCT Evaluation Guidelines; (2) Draft Evaluation Plan Template; and (3) Logic Model Example Adapted from BSF Grantee Model. (Contains 1 table, 7 footnotes, and 2 figures.) [This paper was written with Katie Herrem.]
- Published
- 2010
8. Institutional versus Academic Discipline Measures of Student Experience: A Matter of Relative Validity. A Student Experience in the Research University (SERU) Project Research Paper. Research & Occasional Paper Series: CSHE.8.07
- Author
-
University of California, Berkeley, Center for Studies in Higher Education and Chatman, Steve
- Abstract
The University of California's census survey of undergraduates, UCUES [University of California Undergraduate Experience Survey], presents an opportunity to measure both disciplinary and institutional differences in students' academic experience. Results from nearly 60,000 responses (38% response rate) from the 2006 administration found greater variance among majors within an institution than between equivalent majors across institutions. Cluster analysis techniques were employed to establish disciplinary patterns, with traditional distinctions between hard and soft sciences generally supported. Reporting practices called into question range from institutional comparisons that ignore academic program mix and discipline to campus performance comparisons that do not recognize pedagogical differences by academic major. More specifically, these results suggest that calls for comparable institutional performance measures, as proposed by the Spellings Commission, must take into consideration disciplinary differences in instruction. (Contains 1 table and 9 figures.) [A version of this paper was presented at the SERU Project Symposium, "Assessing the Undergraduate Experience in the Postmodern University" (Berkeley, CA, April 25, 2007).]
- Published
- 2007
9. Evaluating and Interpreting Research Syntheses in Adult Learning and Literacy. NCSALL Occasional Paper
- Author
-
National Center for the Study of Adult Learning and Literacy, Boston, MA., Cooper, Harris, Cooper, Harris, and National Center for the Study of Adult Learning and Literacy, Boston, MA.
- Abstract
This occasional paper introduces the methods of research synthesis and meta-analysis to researchers and consumers of research in the field of adult learning and literacy. To begin, the first section of the paper defines key terms and offers a brief history of how the methodologies developed. The second section provides a conceptualization of research synthesis that views it no differently from other research endeavors in the social sciences. Then, the tasks of research synthesis are presented in more detail within the context of a hypothetical example drawn from the literature on adult learning and literacy. (Contains 1 figure and 5 tables.)
- Published
- 2007
10. Civil Rights Activists in the Information Age: The Development of Math Literacy Workers. CIRCLE Working Paper 50
- Author
-
Watts, Roderick J. and Guessous, Omar
- Abstract
Roderick Watts and Omar Guessous of Georgia State University conducted exploratory research to investigate the link between math and civic engagement. Their research is based on an evaluation of the Young People's Project (YPP), a national program that recruits, trains, and deploys high school and college Math Literacy Workers for mentoring middle and elementary school students. The purpose of this study is to examine whether the Young Peoples Project, as a community-based math and social justice organization for youth, contributes to the math, academic, sociopolitical, ethnoracial identity, and positive youth development of participants. Five specific research questions addressed by this study were: (RQ #1) Program effect: Do YPP participants show more favorable outcomes than the two comparison groups on outcome indicators of interest? (RQ #2) Dosage effect: Do veterans show more favorable outcomes than novices on the outcome indicators of interest? (RQ #3) Program x dosage interaction: Is the dosage effect stronger among YPP participants than among the School Controls? In other words, is there an interaction effect between program membership and level of seniority? (RQ #4) Math--SPD link: Is there a positive relationship between math confidence and indicators of sociopolitical development? (RQ #5) Math--SPD x Program: Is this link moderated by group membership, such that YPP participants demonstrate a stronger positive relationship between math confidence and indicators of sociopolitical development than the School Controls (SC)? Participant in this cross-sectional study were Chicago-area residents in grades 8 and above who were administered a paper-and-pencil survey in a group format at the site of each YPP and After School Matters (ASM) program. YPP participants were encouraged to recruit a peer who did not participate in YPP to serve as matched comparisons. Additionally, ASM participants from three Chicago public high schools were recruited for participation in this study, to serve as an additional comparison group (N=73). There were a total of 196 participants. Each variable of interest was assessed using one or more self-report survey instruments that were answered on a numerical Likert scale. These included measures of social analysis, sense of urgency, societal involvement, cultural and ethnoracial identity. academic and math self-concept, and positive youth development. The five research questions were examined using Univariate Analysis of Covariance and Hierarchical Multiple Regression. Among the outcomes was a favorable link between YPP participation and aspirations for mathematics education, and another was evidence of sociopolitical development. The third was an exploration of other youth development outcomes of interest to the program and the researchers. Sense of Agency was positively related to math self confidence and negatively related to math self-doubt. Similarly, commitment to societal involvement activities was negatively related to math self-doubt. And in the only instance were the ethnoracial variables came into play, cultural pride reinforcement was found to positively correlate to math self confidence. The relationship between sociopolitical development and math confidence was consistently stronger among YPP participants compared to School Controls. This suggests that YPP contributed to a link between math knowledge or reasoning and to both sociopolitical agency and commitment; that is, YPP experiences are associated with the idea that "math is power." (Contains 14 tables and 9 figures.) [This Working Paper was produced by CIRCLE (The Center for Information and Research on Civic Learning and Engagement). For CIRCLE Working Paper 49, see ED494074.]
- Published
- 2006
11. Integrating Formative and Summative Assessment: Progress toward a Seamless System? OECD Education Working Papers, No. 58
- Author
-
Organisation for Economic Cooperation and Development and Looney, Janet W.
- Abstract
A long-held ambition for many educators and assessment experts has been to integrate summative and formative assessments so that data from external assessments used for system monitoring may also be used to shape teaching and learning in classrooms. In turn, classroom-based assessments may provide valuable data for decision makers at school and system levels. Currently there are important technical barriers to this kind of seamless integration. Nevertheless there are a number of promising developments in the field. Ongoing research and development aims at improving testing and measurement technologies, as well strengthening classroom-based formative assessment practices. Improved integration of formative and summative assessment will require investments in new testing technologies, teacher training and professional development, and further research and development.
- Published
- 2011
- Full Text
- View/download PDF
12. New Estimates of Design Parameters for Clustered Randomization Studies: Findings from North Carolina and Florida. Working Paper 43
- Author
-
Urban Institute, National Center for Analysis of Longitudinal Data in Education Research (CALDER), Xu, Zeyu, and Nichols, Austin
- Abstract
The gold standard in making causal inference on program effects is a randomized trial. Most randomization designs in education randomize classrooms or schools rather than individual students. Such "clustered randomization" designs have one principal drawback: They tend to have limited statistical power or precision. This study aims to provide empirical information needed to design adequately powered studies that randomize schools using data from Florida and North Carolina. The authors assess how different covariates contribute to improving the statistical power of a randomization design and examine differences between math and reading tests; differences between test types (curriculum-referenced tests versus norm-referenced tests); and differences between elementary school and secondary school, to see if the test subject, test type, or grade level makes a large difference in the crucial design parameters. Finally they assess bias in 2-level models that ignore the clustering of students in classrooms. (Contains 5 figures, 46 tables and 2 footnotes.)
- Published
- 2010
13. Policy Effectiveness of Interim Assessments in Providence Public Schools. WCER Working Paper No. 2008-10
- Author
-
Wisconsin Center for Education Research, Clune, William H., and White, Paula A.
- Abstract
Many urban districts have adopted interim assessments in recent years as a supplement to annual testing. Possible purposes for such tests include monitoring and assisting student progress, aligning the curriculum, and practicing for state exams, but little is known about how effectively real systems advance these purposes. The Providence Public School District (PPSD) implemented a system of quarterly assessments at every grade starting in 2004 and discontinued them in 2007. During the implementation period, authors interviewed district officials about what they were trying to accomplish, and teachers and other school personnel about their experiences with the assessments and their use of the results. Evidence of effectiveness as well as limitations were found. Uncertainty about the magnitude of positive impacts raised questions about the net value of the assessments given their considerable costs to districts, teachers, and students. The authors conclude that interim assessments are costly, and costs increase if the desired effects on student achievement require significant investments in professional development of teachers. Sound policy, therefore, should be carefully designed and evaluated. Two appendixes are included: (1) Interview Protocol; and (2) Findings from Interviews and Survey. (Contains 3 tables.)
- Published
- 2008
14. Should the Curriculum Be Set by State Fiat? An Empirical Test Using Economics Courses in High School. Occasional Paper.
- Author
-
Columbia Univ., New York, NY. National Center for the Study of Privatization in Education. and Belfield, Clive R.
- Abstract
This paper estimates the effects of state-imposed curriculum mandates on the test scores of public-school students who took the SAT in 2001. By 1998, 14 states had mandates that high-school students should take an economics course. For these states, the proportion of public-school students taking high-school economics was around twice that of states without mandates. The mandate may be interpreted as a regulation on input use in the education sector, potentially impairing the efficiency of schools. It is hypothesized that where there is a mandate, test scores should be lower. The merit of the mandate on scores can be assessed empirically. Using microlevel data on a large sample of students across the United States, this inquiry tests whether students' academic performances are higher or lower in states where there are mandates. This test offers a direct evaluation of the efficiency of the mandate. The mandate reduces test scores by as much as 0.25 standard deviations for those students who would not otherwise have enrolled. This paper sets out the theoretical explanation from the imposition of a mandate, describes the dataset, sets out the estimation method, and reports the results. (Contains 6 tables and 24 references.) (Author/WFA)
- Published
- 2002
15. University Grade Inflation through Twenty Years: An Analytical and Empirical Investigation. ASHE Annual Meeting Paper.
- Author
-
Wood, Anda L., Ridley, Dennis R., and Summerville, Richard M.
- Abstract
This paper presents a new method to test for grade inflation. Defining grade inflation as the decreasing value of grades in the coin of student achievement, the study avoids the assumption that a rise in mean grade is a necessary condition of grade inflation. The study, which was conducted at a single university, also sought to determine whether grade inflation could be demonstrated if mean grades have leveled off. To measure general academic abilities, Graduate Record Examination (GRE) data was used. Data was grouped into three time spans (1976-80, 1984-88, and 1992-96). To examine the influence of students' majors, data were also separated into two broad groups, "humanistic" and "scientific," because literature had shown roughly that the former revealed more grade inflation than the latter. Findings suggested grade inflation relative to GRE total and verbal scores, but no evidence of grade inflation when GRE quantitative or GRE analytical scores were used, and it is postulated that this is a result of declining academic standards in the sense that verbal reasoning ability associated with grade point average is lower than it used to be. Differences were also found between disciplines, with some evidence of grade inflation found for science majors. (Contains 17 references.) (RH)
- Published
- 1999
16. Interaction as a Construct of Oral Proficiency. Paper I.
- Author
-
Pollard, John D. E.
- Abstract
This paper describes efforts to devise a more effective method of assessing oral proficiency in English using a computer-based testing and computer-based test result management system. This new test would be designed to directly operationalize the second language constructs it measured, and would be developed with reference to the best practices revealed by recent research in the field. Scores are automatically sent to databases at the completion and evaluation of each task. Student performance is then used to drive an adaptive algorithm, determining the difficulty of successive tasks. This computerized testing would aid in assigning skills or constructs to individual tasks, apportion the skills required for each task, and identify the features of interaction elicited in individual tasks. The study results revealed not only which tasks elicited most assessor/interlocutor interaction, but also that a great deal of the interaction took place outside the task boundaries. There has been a substantial move towards acknowledging the importance of interaction in oral proficiency and, therefore, oral proficiency testing featuring discourse and conversation analysis. The research detailed in this paper indicates that greater interaction does seem to be associated with greater oral proficiency, yet there are nagging issues of test validity still to be addressed in future research. (Contains 47 references.) (KFT)
- Published
- 1998
17. Alternative Designs for Evaluating Workplace Literacy Programs. Conference Proceedings and Commissioned Papers at the 'Design Guidance for Evaluating Workplace Literacy Programs' Work Group Conference (Washington, D.C. April 13, 1993).
- Author
-
Research Triangle Inst., Research Triangle Park, NC.
- Abstract
This document contains the five papers presented at a meeting at which key issues in evaluating workplace literacy programs were discussed. In "Key Components of Workplace Liteacy Projects and Definitions of Project 'Modules,'" Judith A. Alamprese describes the context for evaluating the National Extension Program, components of workplace literacy programs, and validation of program components. Design alternatives for a national program evaluation and design options for an effective practices study are explored by Jorie W. Philippi in "Measuring Workplace Literacy Program Effectiveness: Evaluation Purposes and Methodologies." In "Advantages and Disadvantages of Longitudinal Designs for Evaluating Workplace Literacy Programs," Elisabeth Hayes examines the following: the short-term nature of literacy gains, random assignment versus other methods, availability of comparison/control groups, and practical feasibility of alternative designs and procedures. The last two papers, "Articulation and Measurement of Participant Outcomes in Workplace Literacy Programs" (Larry Mikulecky) and "Articulation and Measurement of Program Outcomes" (Anthony R. Sarmiento), focus on the relationships between curriculum/instructional design and various participant/program outcomes and assess the availability and appropriateness of data and various data collection methods. Most papers include substantial bibliographies. Appended are the work group conference agenda, biographies of the presenters, and a list of work group participants. (MN)
- Published
- 1994
18. Estimating the Home Schooled Population. Working Paper.
- Author
-
Office of Educational Research and Improvement (ED), Washington, DC. and Lines, Patricia M.
- Abstract
Curriculum suppliers, state departments of education, and home school leaders, are the sources used to estimate that between 248,500 and 353,500 school-aged children (K-12) were educated at home in the 1990-91 school year. This paper explains how this estimate was derived. It provides information on process, possible bias, alternate survey collection of data, and why the three methods were chosen. In the estimates reported, most adjustments to actual data were based on a survey of a sample of the membership of the Home School Legal Defense Association. In deriving figures for the first method, a survey of 16 suppliers of curricular packages yielded an estimate of between 309,000 and 353,500 K-12 children who are educated at home. For the second method, 37 state departments of education reported 82,061 children for whom parents have filed papers with state or local officials, representing a fraction of the total. For the third method, information from 25 home schooling leaders in 17 states yielded a national estimate of between 248,500 and 304,000. A brief analysis of recent trends with a conclusion is given. Appended are home school data collected by states, information from associations, and a summary of selected research reports from two state departments of education. (RR)
- Published
- 1991
19. Assessment of Learning Outcomes in Higher Education: A Comparative Review of Selected Practices. OECD Education Working Papers, No. 15
- Author
-
Organisation for Economic Cooperation and Development and Nusche, Deborah
- Abstract
Higher education institutions (HEIs) have experienced increasing pressures to provide accountability data and consumer information on the quality of teaching and learning. Existing ratings and rankings of HEIs tend to neglect information on student learning outcomes. Instead, they focus on inputs, activities and research outputs, such as resources used, classes taught, and articles published. Such indicators provide no indication of the degree to which HEIs actually develop the knowledge and skills of their students. In most countries, hardly any comparable information is available on the educational quality of different programmes and institutions. In some countries, approaches to assess higher education learning outcomes have been developed, but little cross-country information is available on the characteristics of the instruments used. This paper provides an overview of experience gained in this domain across OECD and partner countries. Based on illustrative evidence collected for 18 assessment instruments, it examines conceptual, organizational and methodological aspects of existing assessments. It proposes a typology of higher education learning outcomes and reviews the ways in which these have been assessed across countries. Examples are drawn from Australia, Brazil, Mexico, the United Kingdom and the United States. (Contains 2 footnotes and 7 tables.)
- Published
- 2008
- Full Text
- View/download PDF
20. Paper-Based versus Computer-Based Assessment: Key Factors Associated with the Test Mode Effect.
- Author
-
Clariana, Roy and Wallace, Patricia
- Abstract
Describes a study that seeks to confirm several key factors in computer-based versus paper-based assessment. Based on earlier research, the factors considered in this study of undergraduates include content familiarity; computer familiarity; competitiveness; and gender. Reports results of analysis of variance that showed the computer-based test group outperformed the paper-based test group. (Author/LRW)
- Published
- 2002
21. Internet Administration of Paper-and-Pencil Questionnaires Used in Couple Research: Assessing Psychometric Equivalence
- Author
-
Brock, Rebecca L., Barry, Robin A., Lawrence, Erika, Dey, Jodi, and Rolffs, Jaci
- Abstract
This study examined the psychometric equivalence of paper-and-pencil and Internet formats of key questionnaires used in couple research. Self-report questionnaires assessing interpersonal constructs (relationship satisfaction, communication/conflict management, partner support, emotional intimacy) and intrapersonal constructs (individual traits, psychological symptoms, contextual influences) were administered to young adults in committed dating relationships. The same measures were administered twice via paper-and-pencil and/or Internet methods over a 2-week period. Method order was counterbalanced among participants, and temporal stability was controlled. Intrapersonal and interpersonal measures generally remained reliable when administered online and demonstrated quantitative and qualitative equivalence across methods. The implications of online administration of questionnaires are discussed, and specific recommendations are made for researchers who wish to transition to online data collection. (Contains 4 tables.)
- Published
- 2012
- Full Text
- View/download PDF
22. Paper-and-Pencil and Web-Based Testing: The Measurement Invariance of the Big Five Personality Tests in Applied Settings
- Author
-
Vecchione, Michele, Alessandri, Guido, and Barbaranelli, Claudio
- Abstract
This study investigates the measurement equivalence of a five-factor measure of personality across two groups applying for jobs, who completed the same questionnaire using either a paper-and-pencil (n = 429) or a web online answer format (n = 651). The data were collected using the Big Five Questionnaire-2 (BFQ-2; which is a measure of the Five Factor Model) of personality traits. Multiple-group confirmatory factor analysis was used to test for the equivalence of factor covariance and mean structures of the BFQ-2. Findings suggested that the Five Factor Model scales have the same measurement unit and origin across applicants using different administration modes. However, latent means were slightly higher for applicants who responded in a web and unproctored condition than for applicants who completed a paper-and-pencil version of the same test. Practical implications for personality assessment are discussed. (Contains 1 table and 2 notes.)
- Published
- 2012
- Full Text
- View/download PDF
23. Redundancies in 'H' Index Variants and the Proposal of the Number of Top-Cited Papers as an Attractive Indicator
- Author
-
Bornmann, Lutz
- Abstract
Ruscio, Seaman, D'Oriano, Stremlo, and Mahalchik (this issue) evaluate 22 bibliometric indicators, including conventional measures, like the number of publications, the "h" index, and many "h" index variants. To assess the quality of the indicators, their well-justified criteria encompass conceptual, empirical, and practical issues: ease of understanding, accuracy of calculation, effects on incentives, influence of extreme scores, and validity. Although they found that some indicators are more attractive for research evaluation purposes than others, the correlation between most of the indicators is (very) high. A generally high correlation between the indicators points out a redundant contribution each to each other. Calculating a comprehensive metaanalysis, Bornmann and colleagues (Bornmann, Mutz, Hug, & Daniel, 2011) came to a very similar result: Only a few "h" index variants make a nonredundant contribution to the "h" index; most of the variants correlate very highly with the "h" index. They presented the first metaanalysis of studies that computed correlations between the "h" index and variants of the "h" index that have been proposed and discussed in the literature. Thus, the study was not about the added value of the "h" index or "h" index variants to other bibliometric indices, e.g., total number of citations. They tested whether there is a generally high (or low) correlation between the "h" index and the variants. The metaanalysis included a total of 135 different bivariate correlation coefficients between the "h" index and an "h" index variant (such as the "g" index). In total, they analyzed 37 different "h" variants. Their results show, with an overall mean value between 0.8 and 0.9, there is a high correlation between the "h" index and the "h" index variants. According to Navon (2009), "high correlations indicate that despite the differences in how the metrics are calculated, there is too much redundancy in the information they yield." Even if the "actual" correlations might be higher than those claimed in the original papers introducing the new "h"-type indices due to systematic sampling selection effects, a mean correlation coefficient of between 0.8 and 0.9 is still high and might not justify the development of more and more "h" index variants. However, not all "h" index variants have a high correlation with the "h" index. Their results indicate that some "h" index variants have been developed that have a relatively low correlation with the "h" index, and it can be assumed that they can make a nonredundant contribution to the "h" index. These variants are mainly the MII (Sypsa & Hatzakis, 2009) and the "m" index (Bornmann, Mutz, & Daniel, 2008). For evaluative purposes, then, the "h" index could be combined with these "h" index variants to better depict research performance bibliometrically. (Contains 1 table.)
- Published
- 2012
- Full Text
- View/download PDF
24. Rates of Missing Responses in Personal Digital Assistant (PDA) versus Paper Assessments
- Author
-
Palen, Lori-Ann, Graham, John W., Smith, Edward A., Caldwell, Linda L., Mathews, Catherine, and Flisher, Alan J.
- Abstract
This article describes rates of missing item responses in personal digital assistant (PDA) assessments as compared to paper assessments. Data come from the evaluation of a classroom-based leisure, life skills, and sexuality education program delivered to high school students in Cape Town, South Africa. Analyses show that the paper assessments had much higher rates of missingness than PDA assessments. This association is moderated by item order. Certain analyses also suggest that paper assessments have higher rates of missingness for items pertaining to participants' sexual behavior. Implications of these results for evaluation research will be discussed. (Contains 5 tables, 1 figure, and 2 notes.)
- Published
- 2008
- Full Text
- View/download PDF
25. A Comparison of Student Feedback Obtained through Paper-Based and Web-Based Surveys of Faculty Teaching
- Author
-
Ardalan, Ali, Ardalan, Roya, and Coppage, Samuel
- Abstract
Many colleges have either begun or are contemplating using the web-based survey to gather student feedback on faculty teaching. Unlike the paper-based method, the web-based approach gives every student in the class the opportunity to provide feedback. Hence, the populations that participate in the web-based and paper-based methods may be quite different, and so may be the feedback. This paper compares the results of student feedback gathered through paper-based and web-based approaches. The results provide information to faculty and administrators on any differences they may expect as they make the transition from a paper-based to a web-based survey of faculty teaching.
- Published
- 2007
- Full Text
- View/download PDF
26. A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost
- Author
-
Greenlaw, Corey and Brown-Welty, Sharon
- Abstract
Web-based surveys have become more prevalent in areas such as evaluation, research, and marketing research to name a few. The proliferation of these online surveys raises the question, how do response rates compare with traditional surveys and at what cost? This research explored response rates and costs for Web-based surveys, paper surveys, and mixed-mode surveys. The participants included evaluators from the American Evaluation Association (AEA). Results included that mixed-mode, while more expensive, had higher response rates. (Contains 3 tables and 1 figure.)
- Published
- 2009
- Full Text
- View/download PDF
27. Technological Alternatives to Paper-Based Components of Team-Based Learning
- Author
-
Robinson, Daniel H. and Walker, Joshua D.
- Abstract
The authors have been using components of team-based learning (TBL) in two undergraduate courses at the University of Texas for several years: an educational psychology survey course--Cognition, Human Learning and Motivation--and Introduction to Statistics. In this chapter, they describe how they used technology in classes of fifty to seventy students to improve the implementation of three key TBL activities: readiness assurance tests, reporting complex team assignments, and providing feedback on peer evaluations.
- Published
- 2008
- Full Text
- View/download PDF
28. Attitudes of Business Faculty towards Two Methods of Collecting Teaching Evaluations: Paper vs. Online.
- Author
-
Dommeyer, Curt J., Baum, Paul, and Chapman, Kenneth S.
- Abstract
A survey of business faculty showed that they preferred the traditional paper method of teacher evaluation to an online version, mainly because they believed it would produce a higher and more accurate response than the online method. Found no characteristics that correlated to attitudes. (EV)
- Published
- 2002
29. Evaluating the Benefits of Lifelong Learning: A Framework. The Wider Benefits of Learning Papers.
- Author
-
London Univ. (England). Centre for Research on the Wider Benefits of Learning., Plewis, Ian, Preston, John, Plewis, Ian, Preston, John, and London Univ. (England). Centre for Research on the Wider Benefits of Learning.
- Abstract
This document presents the key principles underpinning effective evaluations as applied to policy initiatives in the field of lifelong learning (LL) and the various methods available to implement them. The following are among the topics discussed in Chapters 1-6: (1) the distinction between monitoring, modeling, and evaluation and considerations in classifying interventions; (2) types and styles of evaluation in LL (evaluating implementation; evaluating impact, linking implementation to impact; the systems approach to evaluating LL); (3) outcomes from LL interventions (immediate outcomes and measuring the impact of LL policies); (4) designs for impact evaluation (defining the target population; selecting controls; sampling); (5) estimating impact using statistical modeling (fundamental issues; embedding control into the research design, creating control in the analysis; modeling outcomes at different levels); and (6) cost-benefit analysis (CBA) (benefits of learning; ascribing monetary values to the benefits of education; costs of learning; methods of calculation; CBA of the voucher scheme; uses and abuses of CBA). Chapter 7 summarizes the preceding chapters and offers 11 recommendations. Fourteen tables/figures are included. The bibliography lists 80 references. The following items are appended: an annotated list of 13 data sources for monitoring LL; a systems representation of a LL program; and the Stufflebeam scheme of evaluation comparisons. (Contains 80 references and 14 tables/figures.) (MN)
- Published
- 2001
30. The Relative Effectiveness of Private and Public Schools: Evidence from Two Developing Countries. Living Standards Measurement Study Working Paper No. 60.
- Author
-
World Bank, Washington, DC., Jimenez, Emmanuel, and Cox, Donald
- Abstract
Fiscal constraints have limited education budgets in developing countries. Increases necessary to expand access to highly subsidized and publicly provided education have not been forthcoming. Another concern is that education is being inadequately and inefficiently provided by the public sector. This study is an inquiry into the nature of the achievement differential between public and private schools in Colombia and Tanzania. It questions the extent to which the differential in academic achievement can be attributed to student characteristics rather than school inputs, attempts to estimate the quality differences between the public and private secondary schools in these two countries, as measured by student performance on the achievement tests (ACATESTs). Estimated sample-selection effects suggest that Colombian students sort themselves by type of institution, public or private; but Tanzanian students are selected by a hierarchical mechanism, with the lowest-scoring students entering private institutions. These effects are consistent with the different institutional frameworks for educational choice in these two countries. For each country, however, private schools offer an achievement advantage. By standardizing for differences in student and school attributes, private school students have higher achievement test scores. (Author/NL)
- Published
- 1989
31. State Welfare Waiver Evaluations: Will They Increase Our Understanding of the Impact of Welfare Reform on Children? Working Paper and Supplement.
- Author
-
National Center for Children in Poverty, New York, NY., Collins, Ann, Aber, J. Lawrence, and Bernard, Stanley N.
- Abstract
As of June 1995, the federal government had granted to 28 states waivers from current federal laws and regulations, allowing states to test a wide range of welfare reform approaches. This study, conducted by the National Center for Children in Poverty, examined the evaluation plans of 21 waiver experiments in 17 states, focusing on the impact of welfare reform on family income, parenting stress behaviors, and children's access to services. For each impact study, recipients of Aid to Families with Dependent Children were randomly assigned to either an experimental group participating in the welfare reform approach or a control group. The major findings indicated that: (1) evaluators are collecting a great range of child-related data, but few variables were common among many states, reducing the chances for cross-state learning to inform national policy; (2) some waiver evaluations may lead to new information regarding children's education, particularly for states evaluating "learnfare" approaches; (3) some evaluations track potential harms to children related to family dysfunction, foster care placement, and substance abuse; (4) the waiver evaluations will not be able to demonstrate the ways welfare reform affects children's growth and development because very few variables directly measure such changes; and (5) variables in some evaluation plans related to children's services, but few related to quality services or to outcome measures indicating the effectiveness of those services, especially for child care and child health. (Appendices contain background information on selected states, research questions, and children's well-being variables. Contains 24 references.) (KB)
- Published
- 1996
32. Quality Assurance and Transparency Tools. ENQA Workshop Report 15
- Author
-
European Association for Quality Assurance in Higher Education (ENQA), Costes, Nathalie, Hopbach, Achim, Kekalainen, Helka, van IJperen, Robin, and Walsh, Padraig
- Abstract
The European Association for Quality Assurance in Higher Education (ENQA) realises that there is a steadily growing interest among students and all stakeholders of Higher Education and the public at large in accessing detailed and reliable information on the quality of individual study programmes, faculties and higher education institutions. ENQA is aware that external quality assurance activities in higher education set up by quality assurance agencies are producing only part of the information. Besides quality assurance agencies that enshrine their activities in the ESG, there is a growing number of providers of information about higher education. The proliferation of ranking schemes and other transparency tools is one aspect of this increasing need for information about higher education institutions and their activities. The purpose of this seminar was to discuss expectations from higher education institutions and other stakeholders concerning information on quality of higher education, and the contribution of quality assurance and transparency tools to meet these expectations. The outcome of the discussions is a position paper on information and transparency tools, which can be found in Part 2 of this report. This paper is divided into two parts: (1) Key messages; and (2) ENQA position paper on transparency tools. Programme of the Seminar is appended. (Contains 2 footnotes.)
- Published
- 2010
33. Investigating differences between traditional (paper bag) ordering and online ordering from primary school canteens: a cross-sectional study comparing menu, usage and lunch order characteristics.
- Author
-
Leonard, Alecia, Delaney, Tessa, Seward, Kirsty, Zoetemeyer, Rachel, Lamont, Hannah, Sutherland, Rachel, Reilly, Kathryn, Lecathelinais, Christophe, and Wyse, Rebecca
- Subjects
PAPER bags ,PRIMARY schools ,SCHOOL children ,CROSS-sectional method ,LUNCHEONS ,PACKAGED foods ,RESEARCH ,FERRANS & Powers Quality of Life Index ,RESEARCH methodology ,MEDICAL cooperation ,EVALUATION research ,COMPARATIVE studies ,SCHOOLS ,FOOD service - Abstract
Objective: To assess differences between traditional paper bag ordering and online ordering from primary school canteens in terms of menu, usage and lunch order characteristics.Design: A cross-sectional study.Setting: New South Wales (NSW) primary schools that offered both paper bag and online canteen ordering.Participants: Students (aged 5-12 years) with a lunch order on the day of the observation.Results: Across the six school canteens, 59-90 % of all available items were listed on both the online and paper menus, with no significant differences in the nutritional quality ('Everyday'/'Occasional') or nutritional content (kJ/saturated fat/sugar/sodium) of menu items. In total, 387 student lunch orders were placed, containing 776 menu items. Most orders (68 %) were placed online. There were no significant differences between order modality in the quantity of items ordered or the cost of orders, or the nutritional quality of orders based on the classification system of the NSW Healthy School Canteen Strategy ('Everyday'/'Occasional'). However, nutritional analysis revealed that paper bag orders contained 222 fewer kJ than online orders (P = 0·001), 0·65 g less saturated fat (P = 0·04) and 4·7 g less sugar (P < 0·001).Conclusions: Online canteens are commonly used to order canteen lunches for primary school children. This is the first study to investigate differences between traditional paper bag ordering and online ordering in this setting. Given the rapid increase in the use of online ordering systems in schools and other food settings and their potential to deliver public health nutrition interventions, additional research is warranted to further investigate differences in ordering modalities. [ABSTRACT FROM AUTHOR]- Published
- 2021
- Full Text
- View/download PDF
34. Evaluation Procedures Used To Measure the Efficiency of Higher Education Systems and Institutions. New Papers on Higher Education: Studies and Research.
- Author
-
International Association for Educational Assessment., United Nations Educational, Scientific, and Cultural Organization, Paris (France)., and Dockrell, W. B.
- Abstract
This study of higher education efficiency evaluation consists of five country reports from five different regions of the world: Colombia, Jordan, Malaysia, the Netherlands, and Nigeria. The reports describe the approaches to the evaluation process in each country and in the process highlight common concerns and awareness. The studies were all commissioned by the United Nations Educational, Scientific, and Cultural Organization and provide an overview of the various evaluation procedures, mechanisms and instruments currently in force in the selected countries which are intended to measure the efficiency of systems and institutions of higher education in relation to their specific economic, social and cultural roles. Each report presents one of the studies and proceeds by describing the general evaluation philosophy and methodology, the procedures at the national level and procedures in a representative sample of institutions with a conclusion analyzing the efficiency of the mechanisms at various levels. The book's concluding chapter provides an analysis of all the studies taken together. That chapter points out the striking diversity among higher education systems while arguing that the commonalities are also significant. This analysis touches on national systems, the institutions, and presents seven proposals for a future, more focused study. (JB)
- Published
- 1990
35. Some Epistemological Considerations Concerning Quantitative Analysis
- Author
-
Dobrescu, Emilian
- Abstract
This article presents the author's address at the 2007 "Journal of Applied Quantitative Methods" ("JAQM") prize awarding festivity. The festivity was included in the opening of the 4th International Conference on Applied Statistics, November 22, 2008, Bucharest, Romania. In the address, the author reflects on three theses that question the gnoseological and operational efficiency of quantitative methods in the social domain. The first refers to symbolical analysis, where the connections between different qualitative categories are transposed into equation systems, including different types of inequalities or existence conditions. The second refers to empirical research. The third thesis upon which the author reflects refers to the skepticism concerning the predictive capacity of quantitative analysis. Here, the author cites several causal factors contributing to the recent global crisis in the context of these theories. (Contains 1 note.)
- Published
- 2008
36. What Does the Research Say about Students with Disabilities and Service Learning?
- Author
-
Frankson, Delsue and Nevin, Ann
- Abstract
The purpose of this paper is to summarize the research on service learning pedagogy into the realm of professional practice for students with disabilities. At this point in the development of a research base for service learning, an emerging need for studies is reported. Limited research exists which studies the impact of service learning activities with respect to the retention of students with disabilities, particularly those who are also African American. The authors suggest that one role that university faculty can take is to create, implement, and evaluate service-learning programs in urban communities that incorporate students, adults with disabilities, the local public school k-12, and teacher educators and researchers from nearby colleges and universities. (Contains 2 tables and 2 footnotes.)
- Published
- 2007
37. A Survey on Evaluation Practices in Teaching and Learning Centres
- Author
-
Kolomitro, Klodiana and Anstey, Lauren M.
- Abstract
There is general consensus that teaching and learning centres have a positive impact on promoting and supporting a culture that values teaching and learning, yet there is limited evidence on how centres evaluate their work. For this purpose, a survey was developed and administered to the directors of 88 Canadian centres; 46 of which completed the survey. In this paper, we share centres' approaches to evaluation that emerged from the survey findings and the artifact analysis. By raising awareness of evaluation approaches, we are better positioned to identify further opportunities to develop a more systematic and coherent approach to evaluation.
- Published
- 2017
- Full Text
- View/download PDF
38. Assessment of Early Scientific Research Skills Training for Medical Undergraduates in China
- Author
-
Xueer Liu, Huiyu Chen, Xiaoying Liu, Teng Teng, Xuemei Li, Dan Zhu, and Xinyu Zhou
- Abstract
The medical field is facing a physician-scientist shortage, threatening future medical research and development. Medical institutions can contribute to developing physician-scientists by stimulating students' involvement in research. In this study, a medical undergraduate teaching module to develop research skills and encourage interest in research was held in Chongqing, China. Undergraduate medical students at Chongqing Medical University completed research skills training. Before and after the training, an online, anonymous, self-reported questionnaire was administered. The self-reported questionnaire investigated students' demographic characteristics, students' perception of attitudes toward conducting research, research skills (e.g., identifying and utilizing information, critical appraisal of literature, paper writing, and understanding of research), and feedback on scientific research training. The module was implemented with 25 students from July 2019 to October 2019, and 96.0% (N = 24) of participants responded to the questionnaire both prior to and after the training. In the evaluation of research skills, results showed that use of appropriate tools for research (Z = -3.340, p < 0.01), students' ability to undertake a focused literature search (Z = -3.40, p < 0.01), identifying and utilizing information (Z = -3.34, p < 0.01), and paper-writing skills (Z = -3.49, p < 0.01) were significantly improved after the undergraduates participated in the scientific research training. A qualitative analysis of the feedback showed that students found that the training helped to enhance their knowledge, improve their study scores, and motivate them to conduct research in the future. Early scientific research ability training strengthened the research skills of medical undergraduates and motivated them to pursue research.
- Published
- 2024
- Full Text
- View/download PDF
39. A Scoping Review of Evaluation Use: An Ever-Growing Field
- Author
-
Worthington, Paisley, Cooper, Amanda, Searle, Michelle J., Hughes, Jennifer, and Gokiert, Rebecca J.
- Abstract
Evaluation use is a complex, iterative process involving diverse stakeholders and activities. Factors influencing evaluation use has long been discussed, however, little is known about current conceptualizations evaluation use including what counts as use, what encourages use, and how to measure use. Following two previous scoping reviews, we identify enablers and constraints to evaluation use through a scoping review of literature published since 2006 (n = 47). The aims of this scoping review are two-fold: (1) to characterize recent evaluation use literature to identify factors influencing evaluation use, and (2) to coalesce findings, implications, and areas for future practice and research related to use. Our review generated insights regarding publication trends, terminology, factors influencing evaluation use and informing evaluation practice.
- Published
- 2023
- Full Text
- View/download PDF
40. Class Size and Sorting in Market Equilibrium: Theory and Evidence. NBER Working Paper No. 13303
- Author
-
National Bureau of Economic Research, Cambridge, MA., Urquiola, Miguel, and Verhoogen, Eric
- Abstract
This paper examines how schools choose class size and how households sort in response to those choices. Focusing on the highly liberalized Chilean education market, we develop a model in which schools are heterogeneous in an underlying productivity parameter, class size is a component of school quality, households are heterogeneous in income and hence willingness to pay for school quality, and schools are subject to a class-size cap. The model offers an explanation for two distinct empirical patterns observed among private schools that accept government vouchers: (i) There is an inverted-U relationship between class size and household income in equilibrium, which will tend to bias cross-sectional estimates of the effect of class size on student performance. (ii) Some schools at the class size cap adjust prices (or enrollments) to avoid adding another classroom, which produces stacking at enrollments that are multiples of the class size cap. This generates discontinuities in the relationship between enrollment and household characteristics at those points, violating the assumptions underlying regression-discontinuity (RD) research designs. This result suggests that caution is warranted in applying the RD approach in settings in which parents have substantial school choice and schools are free to set prices and influence their enrollments.
- Published
- 2007
41. The current state of uncertainty reporting in ecosystem studies: a systematic evaluation of peer‐reviewed literature.
- Author
-
Yanai, Ruth D., Mann, Thomas A., Hong, Sunghoon D., Pu, Ge, and Zukswert, Jenna M.
- Subjects
LITERARY criticism ,ECOSYSTEMS ,ERRORS-in-variables models ,SOIL sampling ,EVALUATION research ,UNCERTAINTY ,SAMPLING errors - Abstract
Transparency in reporting is essential to scientific progress. No report should be considered complete without a full account of uncertainties, including those due to natural variation and measurement and model error and those incurred by handling problematic data, such as outliers. We randomly selected 132 papers published in 2019 from a list of 100 scientific journals to characterize the current state of uncertainty reporting in ecosystem studies. Each paper was evaluated for the extent to which it reported measures of uncertainty in any of four topic areas common to ecosystem studies: vegetation, soils, precipitation, and surface water. We found that most papers reported a minority of the uncertainty sources we deemed relevant. Papers on surface water reported the highest fraction of uncertainty sources (averaging 47% ± 5%), followed by soils (45% ± 4%), vegetation (32% ± 4%), and precipitation (21% ± 8%). A greater fraction of relevant uncertainty sources were reported when the topics were the primary focus of the paper (44% ± 3%) than when they were not (32% ± 4%). Sampling error—the uncertainty in replicate measurements—was the source most commonly reported in studies of vegetation (84%), soil mass (56%), and surface water (76% of papers). The source of measurement error most often reported was chemical analysis, with 41% of papers on surface water and 75% of papers on precipitation reporting this source, if applicable. In contrast, only 1 of 12 papers reporting chemistry of vegetation provided information on analytical uncertainty. Fewer papers reported what methods were used for handling missing or unusable data and observations below detection limits, but it was difficult to judge whether these sources were relevant if they were not mentioned. Finally, we found that a minority of the papers made all (21%) or some (an additional 21%) of their data available in online repositories, after correcting for a failure rate of 13% of the links. Clearly, there is room for improving the completeness and transparency of scientific reporting in ecosystem studies. [ABSTRACT FROM AUTHOR]
- Published
- 2021
- Full Text
- View/download PDF
42. Assessing the Impact of Planned Social Change. Occasional Paper Series, #8.
- Author
-
Western Michigan Univ., Kalamazoo. Evaluation Center. and Campbell, Donald T.
- Abstract
Program impact methodology--usually referred to as evaluation research--is described as it is developing in the United States. Several problems face the field of evaluation research. First, those issues grouped as "meta-scientific" include: (1) the distinction between qualitative and quantitative studies; (2) the separation of implementation and evaluation; (3) maximizing replication and criticism; and (4) evaluation research as normal rather than extraordinary science. Other problems are those of statistical issues. These include: (1) all problems related to the interrupted time-series design; (2) regression adjustments as substitutes for randomization; (3) problems with randomized experiments; (4) attrition and differential attrition; and (5) regression-discontinuity design. Political-methodological problems form a third major group of evaluation research problems, which include resistance to evaluation. It is contended that many new methodological problems will be apparent as the field moves from the laboratory into social problem evaluation. Three figures, seven graphs, and a 107-item list of references are provided. (SLD)
- Published
- 1976
43. Digital SAT® Research Summary
- Author
-
College Board
- Abstract
Over the past several years, content experts, psychometricians, and researchers have been hard at work developing, refining, and studying the digital SAT. The work is grounded in foundational best practices and advances in measurement and assessment design, with fairness for students informing all of the work done. This paper shares learnings from key research studies that have informed the design and development of the digital SAT and the current understanding of how well the assessment is working for its intended uses.
- Published
- 2023
44. Evaluating Higher Education. Papers from the International Journal of Institutional Management in Higher Education, Centre for Educational Research and Innovation, Organisation for Economic Co-operation and Development.
- Author
-
Organisation for Economic Cooperation and Development, Paris (France). Centre for Educational Research and Innovation., Kogan, Maurice, Kogan, Maurice, and Organisation for Economic Cooperation and Development, Paris (France). Centre for Educational Research and Innovation.
- Abstract
This book contains 24 essays addressing higher education evaluation and published in the International Journal of Institutional Management in Higher Education between 1979 and 1986. They are as follows: "The Evaluation of Higher Education: An Introductory Note" (Maurice Kogan); "Allocation of Public Funds on Instructional Performance/Quality Indicators" (Grady Bogue); "Northeast Missouri State University's Value Added Assessment Program: A Model for Educational Accountability" (Charles J. McClain et al.); "Assessment of Institutional Effectivesness at the University of Tennessee, Knoxville" (Trudy W. Banta et al.); "A Commentary on the Northeast Missouri and Tennessee Evaluation Models" (Marianne Bauer); "Improvement of Education through Internal Evaluation (AMOS)" (Pieter J. D. Drenth et al.); "A Preliminary Assessment of a New Method of Course Evaluation Based on Directed Small Group Discussions" (Robert W. Talbot and Georges Bordage); "Institutional Self Evaluation in Sweden" (Ann-Marie Furumark); "Assessing Institutional Performance: The Importance of Being Careful," (Leonard C. Romney et al.); "The Second Irony: The System of Institutional Evaluation in the United States" (H. R. Kells); "Institutional Performance Assessment Under Conditions of Changing Needs" (John Sizer); "Le Comite National d'Evaluation. An Innovation in French Higher Education" (Andre Staropoli); "Academic Program Review and the Determination of University Priorities" (Frederick E. Balderston); "Peer Review and Partnership: Patterns of Validation in the Public Sector of Higher Education in the United Kingdom" (John Brennan); "A Ten Year Perspective on Faculty Evaluation" (Richard I. Miller); "Evaluating Basic Units: Seven Fundamental Questions" (Rune Premfors); "Methods for the Evaluation of Research" (Michael Gibbons); "Resource Allocation Based on Evaluation of Research" (Michael Gibbons); "Resource Allocation Based on Evaluation of Research" (Rune Fransson); "Evaluation and Funding of University Research: Where Does the Need Lie?" (Jean Marie Archambault); "Excellence Versus Relevance: The Evaluation of Research" (Mary Henkel); "The Use of Bibliometric Data as Tools for University Research" (H. F. Moed et al.); "A Case Study on the Evaluation of Research at the Technical University of Denmark" (Peter Rasmussen); "The Evaluation of Research: An Economic Process" (Anne M. T. Rouban); and "External Assessment of Dutch Research Programmes" (Jack Spaapen). Contains 207 references and an index. (GLR)
- Published
- 1989
45. Test Score Comparison Tables: How Well are They Serving Test Users?
- Author
-
Ute Knoch and Jason Fan
- Abstract
While several test concordance tables have been published, the research underpinning such tables has rarely been examined in detail. This study aimed to survey the publically available studies or documentation underpinning the test concordance tables of the providers of four major international language tests, all accepted by the Australian Department of Home Affairs for Australian visa purposes. To evaluate the concordance studies, we first identified the good practice principles in concordance research through a review of both the relevant literature and leading professional standards in the field of educational measurement and language assessment. Next, we reviewed the concordance studies against the identified good practice principles. Our findings revealed that the information supplied by test providers varied, with some making the full research papers available, whereas others providing little information about their underpinning research. None of the concordance studies fulfilled all the good practice principles. Based on the findings of this study, we offer recommendations for future concordance research in the field of language testing as well as suggestions for practice.
- Published
- 2024
- Full Text
- View/download PDF
46. An Analysis of the GCE A* Grade
- Author
-
Acquah, Daniel K.
- Abstract
The General Certificate of Education (GCE) A* grade was first awarded to students in 2010. It was introduced to assist higher education institutions in differentiating between the highest performing students and to promote and reward greater stretch and challenge. This paper, based on a synthesis of key policy documents, an analysis of quantitative data and a review of existing research, provides an analysis of the GCE A* grade. The paper begins by introducing the historical and political context that led to the introduction of the A* grade, drawing together policy documents and research carried out by the awarding bodies. The paper then presents an analysis of 2011 A-level data, considering the characteristics of students who achieve the grade and addressing concerns around equality and fairness. A further line of evidence concerns public confidence in the new grade, especially in relation to whether it is fulfilling its purposes. The paper concludes by identifying a number of policy lessons for the future of the GCE A* grade. Forthcoming reforms to the A-level will necessitate a reappraisal of the way A* is calculated: this paper will be a useful source of evidence with which to consider the issues.
- Published
- 2013
- Full Text
- View/download PDF
47. Evaluating Leadership Development: Experiences with the NCSL
- Author
-
Simkins, Tim, Bush, Tony, and Wright, Nigel
- Abstract
A symposium was organized at the BELMAS/SCRELM Research Conference in July 2004 to explore the issues arising from the experiences of those contracted by the National College for School Leadership to evaluate its programmes. This article presents four contributions that drew on a wide range of evaluation experience and a response from a representative of the NCSL to the issues being raised. These are: (1) "Control, partnership and empowerment" by Tony Bush; (2) "Matching methods to purpose" by Tim Simkins; (3) "Making evaluation count" by Nigel Wright and Derek Colquhoun; and (4) "Assessing impact--why evaluation is hard" by Peter Earley. In a response from the NCSL, Colin Conner identified three major themes from the papers. The first theme relates to the kind of knowledge that NCSL commissioned evaluations generate. The second theme relates to the issue of ownership and control of evaluations. A third theme draws together a number of issues, in particular the finance and time scales available for evaluations and the extent to which it is possible to explore issues in depth. A summary of these major themes is presented.
- Published
- 2005
- Full Text
- View/download PDF
48. Introduction: More than Methods--Reflections on Participatory Action Research in Geographic Teaching, Learning and Research
- Author
-
Kindon, Sara and Elwood, Sarah
- Abstract
This paper provides an introduction to this symposium issue on participatory action research in geographic teaching, learning and research. It introduces the themes of the symposium and contributions from the participating authors, and also offers additional discussion of the attendant benefits and challenges of using participatory action research within a pedagogic context. The authors begin by discussing the recent growth of participatory pedagogies in higher education, including service learning, research-based teaching and community-based research. They contend that participatory action research may be incorporated in all of these learner-centred and community-engaged pedagogies with beneficial outcomes for teaching, student learning and academic research; however, they advocate for its incorporation into research-oriented and research-based teaching strategies where possible. They also discuss some of the unique political, ethical and logistical challenges involved, and argue that an engagement with participatory action research in our teaching requires more than attention to methods, but, rather, a rethinking of theory-practice, teacher-student, and university-community relationships. They conclude with a brief discussion of the symposium and the authors' contributions. (Contains 1 table and 3 notes.)
- Published
- 2009
- Full Text
- View/download PDF
49. Peer Review of Submissions to the Annual American Evaluation Association Conference by the Graduate Student & New Evaluators Topical Interest Group
- Author
-
Schroter, Daniela C., Coryn, Chris L. S., and Montrosse, Bianca E.
- Abstract
Peer review is an umbrella term that refers to a class of selection and oversight practices, including the familiar mechanisms of the review of proposals submitted for funding, of manuscripts for scholarly publications, and of personnel qualifications and portfolios for selection and promotion. Peer review has long been a cornerstone of modern scientific method premised on the assumption that those within a discipline are best suited to assess the work of others within that field. As such, it is also frequently employed to evaluate proposals submitted for professional meetings such as the annual conference of the American Evaluation Association (AEA). This paper presents a blind peer review method developed by AEA's Graduate Student & New Evaluators (GS&NE) Topical Interest Group (TIG) in an effort to construct an impartial and reliable process in proposal selection. Implications for conference review processes, AEA, and the field of evaluation in general are discussed. (Contains 2 tables and 1 endnote.)
- Published
- 2008
50. Who Makes the Transition to College? Why We Should Care, What We Know, and What We Need to Do
- Author
-
Louie, Vivian
- Abstract
Background/Context: In the last several decades, a college education, in particular the bachelor's degree, has become the key to higher earnings, and overall, to a middle-class lifestyle in the United States. In an increasingly globalized economy, privileging information and communication technologies, it is more than likely that this emphasis on higher education in the American labor market will continue in the future. While we know much about the increasing link between a college education and social mobility, writ large, we know considerably less about who actually makes the transition to college, how this occurs, and why. Despite the strides made, large numbers of individuals are not making the transition to college. Among those who are, there are important differences in the kinds of postsecondary institutions they are attending, and in completion rates, with attendant implications for social mobility. Purpose/Objective/Research Question/Focus of Study: This article has two purposes: (1) to synthesize the key facets of our knowledge base of how to expand the college pipeline and relatedly, to outline particular areas that have been under-examined across the disciplines; (2) to provide future directions that will allow us to better address this important inquiry along substantive and methodological lines. Research Design: This article draws on disciplinary papers commissioned by the Social Science Research Council's Transitions to College Committee. The discussion of preparation is framed around the contributions from the fields of anthropology, and political science; access vis-a-vis history, sociology, and demography; paying for college vis-a-vis economics; and completion vis-a-vis the field of higher education. Conclusions/Recommendations: The article calls for an overall integrative model, e.g., K-16 and interdisciplinary, improved coverage in large-scale datasets, and a more refined mixed methods approach to attend to notable gaps in our understandings of the transition to college. Across disciplines, a key substantive gap is variation along the lines of race, ethnicity, immigrant status, gender, and age across domains, which needs to be illuminated.
- Published
- 2007
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.