Search

The Problem Solving in Technology-Rich Environments (TRE) study was designed to demonstrate and explore innovative use of computers for developing, administering, scoring, and analyzing the results of National Assessment of Educational Progress (NAEP) assessments. Two scenarios (Search and Simulation) were created for measuring problem solving with technology and administered to nationally representative samples of students. Resultant data were used to describe the measurement characteristics of the scenarios and student performance. The Search scenario required students to locate and synthesize information from a simulated World Wide Web environment. The Simulation scenario required students to experiment to solve problems of increasing complexity. TRE Search consisted of 11 observables and produced a total score and two subscores: scientific inquiry and computer skills. Findings of the Search scenario include: (1) Internal consistency of the three TRE Search scores (total, scientific inquiry, and computer skills) ranged from 0.65 to 0.74; (2) Search scores provided overlapping but not redundant information; (3) Scientific inquiry skill scale score was most related to the relevance of the World Wide Web pages visited or bookmarked, the quality of the constructed response to a question designed to motivate students to search for and synthesize information from the Web, and the degree of use of relevant search terms; (4) The computer skills scale score was related primarily to the use of hyperlinks, the use of the Back button, the number of searches needed to get relevant hits, and the use of bookmarking; and (5) Statistically significant differences in performance were found on one or more TRE Search scales for NAEP reporting groups categorized by race/ethnicity, parents' highest education level, student eligibility for free or reduced-price school lunch, and school location, but not for reporting groups categorized by gender. The TRE Simulation scenario consisted of 28 observables and produced a total score and three subscores: scientific exploration, scientific synthesis, and computer skills. Findings of the Simulation scenario include: (1) Internal consistency of the four scales ranged from 0.73 to 0.89; (2) Simulation scores provided overlapping but not redundant information; (3) Scientific exploration skill scale score was most related to which experiments students chose to solve the Simulation problems; (4) Scientific synthesis scale was primarily related to the degree of correctness and completeness of conclusions drawn for each problem; (5) Performance on the computer skills scale was related mainly to the number of characters in the written responses students gave for each of the Simulation problems; and (6) Statistically significant differences in performance were found on one or more TRE Simulation scales for NAEP reporting groups categorized by race/ethnicity, parents' highest education level, and student eligibility for free or reduced-price school lunch, but not for reporting groups categorized by gender or school location. It is noted that this report presents results that do not reach definitive conclusions at this point in time and techniques and inferences made from the data may be subject to future revision. Appendixes include: (A) Development Committee for the Problem Solving in Technology-Rich Environments (TRE) Study; (B) Sample Selection; (C) Technical Specifications for Participating Schools; (D) Prior Knowledge and Background Questions for Search and Simulation Scenarios; (E) TRE Simulation Glossary, Help and Tutorial Screens; (F) Bayesian Estimation in the Problem Solving in Technology-Rich Environments Study; (G) C-rater Rules for Scoring Students' Search Queries; (H) TRE Search and Simulation Scale Scores and Percentiles by Student Reporting Groups for Scales on Which Statistically Significant Group Differences Were Observed; (I) Summary Statistics for Prior Knowledge Measures and Mean Scale Scores for Background-Question Response Options; (J) Performance on Problem Solving in Technology-Rich Environments (TRE) Observables; and (K) Understanding NAEP Reporting Groups. (Contains 116 figures and 44 tables.) [This report was written in collaboration with: Douglas Forer, Bruce Kaplan, Michael Wagner, and Lou Mang. The NAEP Problem Solving in Technology-Rich Environments (TRE) study was part of the Technology-Based Assessment (TBA) project, a collaborative effort led by the National Center for Education Statistics (NCES) and the National Assessment Governing Board, and carried out by Educational Testing Service (ETS) and Westat. The Problem Solving in TRE study is the last of three field investigations in the NAEP Technology-Based Assessment Project, which explores the use of new technology in administering NAEP. For previous investigations in this series, see ED485780.]

In 2001, the National Assessment of Educational Progress (NAEP) conducted a geography assessment of the nation's fourth-, eighth-, and twelfth-grade students. This report presents the results of that assessment. Results in 2001 are compared to results of 1994's NAEP geography assessment, which was the preceding NAEP geography assessment and the only other geography assessment conducted under the current framework. Students' performance on the assessment is described in terms of average scores on a 0-500 scale and of percentage of students attaining three achievement levels: (1) basic; (2) proficient; and (3) advanced. Average geography scores for fourth and eighth graders were higher in 2001 than in 1994, while the performance of twelfth graders was not significantly different. At both grades 4 and 8, score increases occurred among the lower-performing students. The 2001 assessment showed that 21% of fourth graders, 30% of eighth graders, and 25% of twelfth graders performed at or above the proficient level for their respective grades. These levels are identified as those at which all students should perform. Both grades 4 and 8 showed an increase from 1994 to 2001 in the percentage of students at or above basic. No significant changes occurred in the percentage at or above "Proficient" at any grade. In addition to overall results, the report provides data on the performance of various subgroups of students and information about the contexts for learning by administering questionnaires to assessed students, their teachers, and their school administrators. The report contains six chapters, each of which includes extensive figures and tables. Appended are additional data and an overview of procedures used for the assessment. (BT)

Civics education in the nation's schools informs students about the structures, functions, and processes of government and about the meaningful ways in which citizens can make decisions about public issues and participate in governance. This report, part of the Nation's Report Card program and based on findings from the 1998 National Assessment of Educational Progress (NAEP), provides a view of student achievement in civics over a 10-year period from 1988 to 1998. In 1998, NAEP conducted two national assessments of student civics knowledge in grades 4, 8, and 12. One assessment employed a set of new test specifications, while the other assessment was a special study that repeated a number of multiple-choice test questions used in 1988. The special study results are reported here as a summary of trends in student knowledge and classroom teacher practices over the 10-year period. Results are based on the assessment of a student sample at each grade that is statistically representative of the entire nation. Student performance is described in terms of average percentage correct, rather than the traditional NAEP scaled scores. The report provides results for subgroups of students defined by various background and contextual characteristics. Analysis focuses on differences between 1988 and 1998, rather than differences among groups within each year. To illustrate the civics knowledge that was assessed, numerous samples of test questions are provided. The report also explores trends in the classroom coverage of civics topics from 1988 to 1998, as well as trends in classroom instructional activities over the decade. Four appendices are included: (1) "Overview of Procedures Used in the 1988 and 1998 NAEP Civics Special Study"; (2) "Standard Errors"; (3) "Average Percentage Correct"; and (4) "Members of the NAEP Civics Standing Committee." (Contains approximately 95 tables and figures.) (BT)

The National Assessment of Educational Progress (NAEP) is the nation's only ongoing survey of what students know and can do in various academic subject areas. This report summarizes the results of the 1998 NAEP civics assessment for the nation. The results are based on an assessment of students at grade 4, grade 8, and grade 12. Students' performance is described in terms of their average civics score on a 0 to 300 scale and in terms of the percentage of students attaining each of three achievement levels: (1) basic; (2) proficient; and (3) advanced. The achievement levels are performance standards adopted by the National Assessment Governing Board (NAGB) as part of its statutory responsibilities. In addition to providing average scores and achievement level performance for the nation, this report also provides results for subgroups of students defined by various background and contextual characteristics. Further, the report explores relationships between selected teacher and classroom activities and student performance. The chapter titles include: (1) "Average Scale Score and Achievement Level Results for the Nation"; (2) "Average Civics Scale Score Results for Selected Subgroups"; (3) "Civics Achievement Level Results for Selected Subgroups"; (4) "Contexts for Learning Civics: School/Teacher Policies and Practices"; and (5) "Contexts for Learning Civics: Classroom Practices and Student Variables." The appendices include an overview of procedures used in the assessment, standard errors, and a list of the members of the NAEP civics development and standing committees. (LB)