Search

This paper explains the process used to validate the construct validity of the Factors Influencing Pursuit of Higher Education Questionnaire. This questionnaire is a literature-based, researcher-developed instrument which gathers information on the factors thought to affect a person's decision to pursue higher education. The questionnaire includes 10 scales: parental influence, extended family support, peer support, locus of control, relative functionalism, glass ceiling effect, financial aid concerns, influence of mentors, presence of role models, and general preparation for college. The questionnaire was completed by 434 college students enrolled in general studies classes at two southeastern universities in the spring of 1997. Results indicate that the questionnaire has a high degree of internal consistency. Reliability estimates for the 10 scales included in the questionnaire ranged from adequate to excellent. Overall, the questionnaire was determined to have satisfactory construct validity for use in further investigations of the factors which influence individuals to pursue higher education. The questionnaire is appended. (Contains 42 references.) (DB)

This document reports on the content model of the Occupational Information Network (O*NET), which is intended to replace the "Dictionary of Occupational Titles." The following chapters are included: "Introduction" (Michael D. Mumford and Norman G. Peterson); "Content Model" (Michael D. Mumford and Norman G. Peterson); "Skills" (Michael D. Mumford and Norman G. Peterson); "Knowledges" (Edwin A. Fleishman, David P. Costanza, Leon I. Wetrogan, Charles E. Uhlman, Joanne C. Marshall-Mies); "Education" (Lance Anderson); "Generalized Work Activities" (P. Richard Jeanneret and Walter C. Borman); "Work Context" (S. Morton McPhail, Barry R. Blakley, Mark H. Strong, Tonya J. Collings, P. Richard Jeanneret, Laura Galarza); "Organizational Context" (Sharon Arad, Rob Schneider, Mary Ann Hanson); "Training, Experience, and Licensure" (Lance Anderson); "Abilities" (Edwin A. Fleishman, Leon I. Wetrogan, Charles E. Uhlman, Joanne C. Marshall-Mies); "Occupational Values and Interests" (Christopher E. Sager); "Work Styles" (Walter C. Borman, Amy Schwartz McKee, Robert J. Schneider); "Occupational Characteristics (Lloyd Feldman, Frank Bennici, Regina Yudd); "Procedures for Collecting Occupation-Specific Information" (Michael D. Mumford); and "Conclusions" (Michael D. Mumford). The following questionnaires are appended: skills questionnaire; knowledges questionnaire; training, education, licensure, and experience questionnaire; generalized work activities questionnaire; work context questionnaire; organizational context questionnaire; abilities questionnaire; occupational values questionnaire; and work styles questionnaire. Many chapters include substantial bibliographies. (MN)

This paper describes the task analysis of performance-based science tasks that were designed for the 1994 National Assessment of Educational Progress (NAEP) science assessment, now postponed until 1996, and field tested in 1993. A brief description of the science performance tasks is followed by a description of the task analyses performed and a discussion of the various uses of task analysis. In 1993, 17 performance-based tasks were field tested, with 5 for grade 4, 2 for grades 4 through 8, 3 for grade 8 only, 2 for grades 8 through 12, and 5 for grade 12 alone. In level one task analysis, the unit of analysis was the task as a whole. Analysis methods included observations, the think aloud method, questionnaires, and expert opinions. In level two analysis, the cognitive operations and skills of steps in the task were analyzed. As results in these field tests demonstrated, level one analysis results in a mapping of the task's structure from the perspectives of test takers. Level two analysis shows the nature of the skills for completing the task. Together, the levels can ascertain the extent to which tasks meet standards of validity. (Contains 3 tables, 2 figures, and 13 references.) (SLD)

This study attempted to develop a reliable and valid instrument for assessing work environment and continuous quality improvement efforts in the non-academic sectors of colleges and universities particularly those institutions who have adopted Total Quality Management programs. A model of a work environment for continuous quality improvement was developed and an instrument was designed to assess non-academic staff's perceptions of the quality process, practices, and results at the University of Michigan (UM). A literature review identified major organizational and higher education models of quality improvement. Researchers designed a study instrument of 217 items which was evaluated by pilot testing with focus groups of UM staff. The final instrument consisted of 201 items that incorporated 35 "quality culture and climate" dimensions. Items focused on perceptions at the respondents' immediate work "unit" level. All permanent non-academic staff completed the questionnaire in February 1994. Of 10,400 questionnaires distributed 4,900 were completed and returned. Results were subjected to data reduction, index development, and reliability and validity testing from which 27 factors emerged. Reliability for indices ranged from .53 to .96. Extensive appendixes contain details of methodology, factor analyses, indices descriptions, and comparisons of conceptual categories and factor indices of a quality culture and a quality climate. (Contains 44 references.) (JB)

This paper discusses a method of eliciting parental self-reports about child-rearing behavior. The method, called computer-presented social situations (CPSS), uses vignettes that are presented on computers. Several studies examined: (1) what mothers think about the technique; (2) reliability of data collected by the technique; and (3) validity of data compared to actual maternal behavior. In one study, mothers responded through computers, questionnaires, or interviews to 24 questions concerning child misbehavior vignettes. The CPSS was rated more favorably on four factors than were the other methods of eliciting self-reports, but were rated less favorably on expression than was the interview. A second study failed to replicate these results. In a third study, mothers responded twice, before and after a two-week interval, to a 104-question CPSS program. Test-retest results were acceptable and compared favorably with other methods. In a fourth study, mothers were observed on a visit to a supermarket with their children, and responded to a CPSS program about child misbehavior during the visit. The overall agreement between mothers' observed behavior and intentions assessed by means of the computer was 36 percent. A reference list of six items is included. (BC)

The "PISA 2012 Technical Report" describes the methodology underlying the PISA 2012 survey, which tested 15-year-olds' competencies in mathematics, reading and science and, in some countries, problem solving and financial literacy. It examines the design and implementation of the project at a level of detail that allows researchers to understand and replicate the resulting data and analyses. The reader will find a wealth of information on the test and sample design, modes of delivery (paper-based or computer-based), methodologies used to analyse the data, technical features of the project, and quality control mechanisms. Annexes include: (1) Main Survey Item Pool Classification; (2) Contrast Coding Used in Conditioning; (3) Standard Errors of Means, Sample Sizes, School Variance Estimates, and Other Sampling Outcomes; (4) Mapping of ISCED [International Standard Classification of Education] to Years; (5) National Household Possession Items; (6) Technical Standards for PISA 2012; and (7) PISA Consortium, Staff and Consultants. (Individual chapters contain references.)

Quality measures were originally developed for research aimed at describing the settings that children spend time in and identifying the characteristics of these environments that contribute to children's development. They were also developed to guide improvements in practice. Increasingly, however, measures of quality are being used for further purposes. In particular, they are being used to guide components of state policies. For example, many states are developing Quality Initiatives and employing measures originally created for research or for guiding improvement in practice for the new purpose of assigning quality ratings to early care and education settings. States are also using these measures to monitor change in quality over time. This publication is being compiled by Child Trends for the Office of Planning, Research and Evaluation of the Administration for Children and Families, U.S. Department of Health and Human Services, to provide a consistent framework with which to review the existing measures of the quality of early care and education settings. Its aim is to provide uniform information about quality measures. It is hoped that such information will be useful to researchers, practitioners, and in informing the measurement of quality for policy-related purposes. Individual profiles contain figures, references, tables, and footnotes. [This paper was prepared for the Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.]

This document contains four symposium papers on measurement and research tools. "Income Effects of Human Resource Development for Higher Educated Professionals" (Martin Mulder, Bob Witziers) reports on a study of 1,876 higher-educated professionals that found no correlation between participation in human resource development activities and subsequent job level or income. "Do Higher Performing Managers Actually Receive Better Ratings? A Validation of Multirater Assessment Methodology" (Allan H. Church), which is an applied example of a validation process based on a sample of 76 senior-level managers from a global health services organization, confirms the relationship between managerial performance and higher ratings on multirater feedback instrumentation. "Examination of the Tendency of Coefficient Alpha to Support the Choice of Instruments with Response Category Restriction" (Shani D. Carter) demonstrates that interpretation of coefficient alpha and the decision to use an instrument can be improved by considering item construction before deciding to use an instrument. "A Factor Analytic Investigation of the Content of Participant Reactions: Affective and Utility Judgments" (Ronald B. Morgan, Wendy Casper) presents the results of a study suggesting that participant reactions are multidimensional and "utility" judgments represent an underlying factor. (MN)

A study was conducted (1) to analyze the Educational Participation Scale (EPS) factor patterns derived from a sample of senior citizens in order to contribute additional reliability and validity data to the instrument; and (2) to look at reasons that have influenced senior citizens to pursue educational activities, specifically the Elderhostel Program. The sample for the study was 160 Elderhostel participants at the University of New Mexico and the College of Santa Fe during the summer of 1981. Most participants had a college degree or postsecondary education. The Educational Participation Scale was administered to this sample, with a 75 percent return rate. Four scales consisting of 31 items were used to rank participants' reasons for enrolling in terms of escape/stimulation, social welfare, social relationships, and cognitive interest. (Items from earlier versions of the instrument pertaining to professional advancement were deleted.) It was found that the Educational Participation Scale yielded factor patterns similar to earlier studies by Boshier and Riddle, and that this form of the EPS would be a useful instrument for adult educators and program planners in planning programs for adults. The results from the study further indicate that cognitive interest was the most powerful motivator of the population, with social relationships the next most powerful motivator. No differences in motivation were found by age and sex. (A list of recommendations and practical tips for conducting programs for elderly persons is given at the end of this paper.) (KC)

Research was conducted that initially focused on identifying and synthesizing the large number of previously identified vocational special needs teacher competencies into an educational model. The model contained a matrix that organized all these competencies into twelve domains of four performance phases each. The Competency Matrix was the construct upon which a needs assessment instrument was designed. A second project activity focused on developing and pilot testing a needs assessment instrument to measure individual educators' perceived needs for training within each phase of twelve domains and their preferences for the delivery of this training. The instrument was designed to provide data to assist inservice planners/deliverers in decision-making efforts at local, state, and/or national levels. A pilot test of the instrument was conducted in six Minnesota schools to determine the extent to which the instrument could be shown to exhibit face, content, utility, and construct validity. Pilot test results indicated that the instrument could give useful data for making inservice/preservice planning decisions and that population subgroups had differing training needs. This instrument can provide planning information, but may need to be tailored to various populations' levels of awareness and expertise. (Author)

A review of the literature on the validity and reliability of survey data is presented prior to an analysis of the reliability of selected questions in the Second Followup Questionnaire of the National Longitudinal Study of the High School Class of 1972 (NLS). The reliability study includes an evaluation of test-retest reliability as a function of data collection procedures (whether mail-in or personal interview), item characteristics (response format, item content, and item length), respondent characteristics (sex, ethnicity, socioeconomic status, and academic ability), and the interaction of these diverse factors. At issue is not only the quality of NLS questionnaire data, but the consideration of guidelines for analyzing survey data and improving the quality of that data. In lieu of an empirical analysis of validity that might have encroached on the privacy of respondents, the validity check was based on a literature review, focusing on NLS types of items. The validity study also focused on data collection procedures, item characteristics, respondent characteristics, and their interactions. An overview of the purpose of the NLS and the methodologies employed are also included. The Short Form of the Second Follow-up Questionnaire is appended. (Author/EVH)

A study assessed the applicability of the American Institutes for Research (AIR) procedures' Performance-Against-Standard form for use in personnel evaluation by the Ohio Cooperative Extension Service. The study also compared relevant test construction approaches identified in the literature to those used in the AIR study in attempting to confirm and improve the reliability and validity of the performance appraisal form. During the study, supervisors of Ohio's 10 extension areas each appraised the performance of those agents working under them who had been employed for at least 6 months but for less than 3 years. Job analysis and the technique of item selection were used to judge item availability on the form. To measure the reliability of the form, researchers utilized Cronbach's alpha. In addition, multivariate measurement was used to determine the accuracy and relevance of the form. Because evidence indicated that the revised Performance-Against-Standard form does provide an accurate and relevant measure of employee performance, recommendations were made calling for including the form into the appraisal system of the Ohio Cooperative Extension Service. (The six page personnel appraisal form is appended.) (MN)

With increasing reliance being placed on the results of their surveys, the National Center for Education Statistics (NCES) recognized a need for these survey results to be valid and reliable. As part of the work of the Statistical Analysis Group in Education (SAGE), an effort was undertaken to investigate validity and reliability in survey research. This document is the result of that effort. The first section provides an overview of the concepts of reliability and validity. Procedures for measuring sources of error are suggested. Several ways of approaching validity are mentioned including content validity, criterion-related validity, and construct validity. The second section presents some suggestions for approving the reliability and validity of survey data, with a focus on the data collection phase. These suggestions are based upon previous data collection experiences of the staff of SAGE. The final section, encompassing 106 pages, provides an annotated bibliography of selected materials relevant to validity and reliability in survey research. (JAZ)

Explores the work done in revising and field testing the Scientific Attitude Instrument and raises doubts about the instrument's validity. Discusses consideration of approaches to validity, conceptual analysis of science attitude instruments, using other studies to establish validity, and guidelines for valid instruments. Contains 12 references. (Author/JRH)

A metanalysis involving 62 studies compared the response rate to mailed surveys with and without a monetary gratuity. The average response rate increased 19% when a gratuity was enclosed. Other findings that substantiate that the external validity of surveys can be increased by gratuities are discussed. (SLD)

Describes the development of a questionnaire that measures teachers' perceptions of eight aspects of a school environment. Discusses the theory base and predecessors of this instrument, the criteria and process used to develop it, its validation, and its application so far. The 56 questionnaire items are appended. (MCG)

A theory of interaction between college students and college faculty based on similarity in style of intellectual functioning was proposed, instruments to measure student and teacher characteristics were developed, and the relationship between achievement and intellectual style similarity was studied. Studies that focus on the impacts of physical, social, and intellectual environments on the interaction between student and teacher are reviewed. Two scales were developed to measure cognitive style: Student Goal Orientation scales and Staff Role Orientation scales. Pilot testing of the scales and reliability, validity, and other measures of their quality are described. The study was conducted with 287 students and 30 staff of a university physics department. It was found that the greater the degree of similarity in student goal orientation between student and lecturer, the greater was the student's achievement, as measured by that lecturer. Additionally, there was no relationship between similarity in staff role orientation between student and lecturer and that student's achievement, as measured by that lecturer. Implications of the findings for further research and for tertiary teaching are considered. Study instruments and related articles are appended. (SW)

Based on the evaluation of 24 faculty development programs in the United States and other scholarly activities, the benefits of systematic planning and evaluation of faculty development programs are considered. In addition, a set of field tested instruments for planning and evaluation and case studies of institutional programs are presented, and the faculty development movement and the literature on faculty are reviewed. Principles that may be helpful in effective planning of professional growth programs are as follows: decision-making requires reliable information; planning programs for people demands flexibility; good planning entails broad participation; evaluation and feedback are essential elements of the planning process; planning should be comprehensive; and planning must be coordinated, organized, and sustained. A planning and evaluation model is diagrammed that encompasses problem identification and solution, tasks, outcomes, and areas of focus. Attention is directed to the reliability and validity of the six instruments that were employed to assess faculty, administrators, and program goals and practices. Two other instruments, a demographic sheet and goal inventory, were also used, and all the instruments are appended. Information is also presented concerning how faculty evaluated the professional development programs on their campuses and what the evaluation team felt were model programs uncovered in the study. The following model programs are described: The University of Rhode Island, Syracuse University, Gordon College, Earlham College, St. Mary's Junior College, and Illinois State University. (SW)

This project was conducted to refine and expand previous research indicating the need for an instrument counselors can use in working with area secondary schools and potential students to assess their progress in making career and training decisions. Instruments developed during this research process were pilot tested at two area schools and initial results prepared. This continuation project stressed enhancement of initial results and assessment of the validity of research instruments. Participating in the project were over 3,500 students from 8 high schools representative of the region (1 urban, 3 suburban, 2 rural, 1 private, and 1 vocational school). Some of the procedures conducted during the project were the following: (1) research instruments were evaluated and altered to address areas of concern for articulation research; (2) the career planning survey was revised to facilitate administration and student completion; (3) 40 area schools were assisted during the funded period; (4) one high school was enabled to begin conducting longitudinal research on its students; (5) a specific research instrument was developed to be used at the Indiana School for the Deaf; (6) a needs assessment instrument to be used with parents was developed; and (7) a computer-based analysis system was developed. As a result of the project, the research instruments have been refined to the point that they may be used repeatedly with little change. They are expected to yield reliable information in the future. (Survey instruments and results constitute the bulk of this document.) (KC)

Intended as an aid to researchers who must decide which measurement techniques to use in the study of organizational communication, this paper describes and analyzes three of the most widely used data collection instruments: the Communication Audit Survey Questionnaire (CAS), the Communication Satisfaction Questionnaire (CSQ), and the Organizational Communication Questionnaire. The first part of the paper provides a general introduction, while the second part examines the construct of organization-wide communication and offers a comparative analysis of the dimensions covered by the three instruments. The third part of the paper comments on the research applications of each instrument, and the final part offers a detailed description of each, including background, development history, reliability and validity assessment information, and illustrative questions for each instrument dimension. References for the published and unpublished literature explaining, criticizing, or using each instrument are appended. (FL)

A series of studies and analyses to develop brief, reliable measures of intelligence and achievement motivation that would be feasible for use in a cross-sectional sample is summarized. One measure of intelligence meets these criteria. It is a 13-item test of sentence completion skill, which correlates effectively with other measures of intelligence and with status and achievement variables in a meaningful way. A questionnaire measure of achievement motivation also emerges as a potentially useful measure for most groups except black females. It correlates moderately well with both a projective measure of achievement motivation and a behavioral assessment of moderate risk-taking in most groups, except the black female group. Both the intelligence and the motivation measures suggested are independently effective in predicting the education attained by black males and white females regardless of their status backgrounds. Only the intelligence measure is an independent predictor of education attainment in white males and black females, regardless of their status backgrounds. (Author/MV)