Search

Standard-setting research has yielded a rich array of more than 50 standard-setting procedures, but practitioners are likely to be confused about which to use. By synthesizing the accumulated research on standard setting and progress monitoring, this study developed a three-dimensional taxonomy for conceptualizing and operationalizing the various procedures: outcome versus growth assessment, theory-driven versus data-driven approach, and observed scale versus latent scale mapping.An empirical study is reported to illustrate how these various approaches can be implemented to meet the accountability challence in the No Child Left Behind era. Consistency analysis of 12 standard-setting procedures reveals vastly disparate pass/fail decisions among different procedures, even within the same conceptual framework or mapping operation. Particularly disturbing is the finding that the passing rate may jump from as low as 29% to as high as 79%, depending on whether the standard is mapped to the observed-score scale or the latent-score scale. Implications and future directions for policy makers, school officials, and psychometricians are discussed. (Contains 9 tables and 107 references.) (Author/SLD)

The topic of high-stakes testing (HST) is important because HST has direct and indirect effects on career-technical education (CTE) programs and timely because HST increasingly enters public discussion and has produced a large body of research and practice that generalizes to CTE. A review of HST has identified two persisting dilemmas: policy and public expectations of testing exceeding tests' technical capacities and tension between testing to increase fairness and testing to classify. Applicable strategies to provide validity for HST are reliability estimation of scores used to make decisions, expert judgements of item linkage to curricula, studies of the predictive power of HST scores, and studies of consequences. Two opposing perspectives on the accountability-testing theme are that use of HST for accountability is a positive application of data-driven management to education and that the consequences of HST are negative. Descriptions of HST systems in Kentucky, Texas, and Massachusetts indicate different ways to accomplish HST; use of advisory panels to represent stakeholders' viewpoints; and continuous change. Findings of an e-mail survey of state CTE directors suggest ways to expand assessment modalities--computer delivery of assessments and authentic assessments or multimodal assessments that include high- and low-stakes components. Implications are that the CTE community needs awareness of HST; tests should be used responsibly; and a useful database system should be developed. (Contains 76 references and a list of 8 Internet sites.) (YLB)

Career-technical educators face three issues in credentialing through assessment. First, the occupational credentialing domain is large and evolving, and a clear understanding of it is a prerequisite to considering adoption of a credential. Three types of credentialing are registration, certification, and licensure. Credentialing organizations are categorized by their mission (government regulatory board, trade association, vendor-specific, National Skills Standards Board). Oversight organizations are professional organizations that disseminate information and provide voluntary oversight by evaluating credentialing systems. Second, a set of clear, comprehensive standards is needed to define credential quality and credibility. Proposed evaluative criteria/standards to select assessment-credentialing are marketability, recognition, alignment to curriculum, quality of input standards, quality of assessments, and usability for career-technical education (CTE) setting. Third, CTE policymakers and educators need a rational, efficient process to evaluate credential systems and associated assessments against a set of standards. The following nine steps are the process: (1) define purposes and uses of occupational credentialing systems; (2) set evaluation criteria; (3) identify credentialing systems and evaluate preliminary link to programs; (4) conduct initial screening; (5) determine quality of input standards; (6) determine quality of credentialing assessments; (7) conduct final linkage to curriculum; (8) determine marketability and recognition; and (9) develop data collection procedures. (Contains a 53-item bibliography.) (YLB)

The process of assessment in career and technical education (CTE) is changing significantly under the influence of forces such as emphasis on assessment for individual and program accountability; emphasis on the investigation of consequences of assessment; emergence of item response theory, which supports computer adaptive testing; and pressure for authentic assessment by critics of standardized testing. The use of technology for assessment is being increasingly stressed. Potential advantages of computerized assessment include rapid feedback, money savings, enhanced security, more curriculum time, and capability to track process-oriented variables. Potential disadvantages include costs of equipment, personnel, and training; increased marginalization of groups based on ethnic or socioeconomic status; and missed opportunities to implement authentic assessments. Examples of applications of technology to evaluation in CTE include test design, item creation, presentation, item scoring, and location (through use of the Internet). In moving toward computerized assessment, consequences should be considered in line with current evaluation and validation models. Especially for policy makers and administrators, implementation is an important aspect of computerized assessment. Technology can support the broad changes in assessment that are ongoing as a function of internal and external scrutiny, and both authentic as well as traditional assessment can benefit from technology. (Contains 24 references.) (KC)

Describes the development of the Minnesota Multiphasic Personality Inventory (MMPI). Details revisions in the MMPI and then evaluates positive and negative features of these revisions in the light of construct validity and professional practice. Argues that many suggested refinements to the instrument are being actively investigated by its developers. (RJM)