Search

The National Math + Science Initiative (NMSI) is a nonprofit organization committed to improving educational outcomes that traces its roots back to the early 1990s. NMSI's College Readiness Program (CRP) is a long-standing program with the goal of promoting STEM education in high schools to improve students' preparation for college. The three-year program provides teacher, student, and school supports to promote high school students' success in English, mathematics, and science Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 27 schools in the 2016-2017 school year (Treatment Schools) and in 21 schools in the 2017-2018 school year (Delayed Treatment Schools), collectively identified as Program Schools. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 48 CRP schools and 48 Comparison Schools in 10 states. The evaluation of the CRP consisted of three parts: (1) measuring the program's impact on selected student AP exam outcomes; (2) determining the impact of the program on school perspectives and culture; and (3) assessing of the fidelity of implementation of the CRP at the school level. AP exam data from 48 Treatment Schools, with a total of 8,778 exams in 2018 and 9,378 in 2019, and 48 matched control schools, with 7,505 exams in 2018 and 2019 in Year 3, were analyzed for this study. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools. The analyses revealed that in 2018 the probability of a student taking an AP exam in the Program Schools was, on average, 7% higher than the paired Comparison Schools, and the difference was statistically significant. And in 2019 the effect was even greater with the probability of taking an AP exam being significantly higher in Program Schools (18%) than in Comparison Schools (3%). When looking at the probability of an exam yielding a qualifying score, in 2018, the HGLM analyses found no significant difference between the two groups. In 2019, however, exams taken at the Program Schools had a significantly higher overall probability (2%) of receiving a qualifying score than the Comparison Schools (0%). These analyses compared results to the total school population. In the next analyses, we looked only at those students who took AP exams. In 2018, overall, the fitted probability of achieving a qualifying score among the exams taken was 8% in the Program Schools, compared to 22% in the Comparison Schools. In 2019, however, the difference between the Program Schools (7%) and the Comparison Schools (9%) was not statistically significant. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. In 2018, results indicated that 43 out of 48 schools (90%) achieved 80% or better implementation fidelity, for an average fidelity score of 89%. Four schools achieved a perfect 100% fidelity score. In 2019, 88% of schools achieved 80% or better implementation fidelity. Ten schools achieved a perfect 100% fidelity score. In 2018 in more than 75% of schools, not all teachers fulfilled their requirements for attending the required teacher training sessions, and so this component was not implemented with fidelity. In 2019, this picture improved a little with 15 schools (31%) meeting the 80% threshold. Teacher stipends, administrator bonuses, and student qualifying score awards were paid as expected. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports. Mentoring was chosen, across all years, as the least effective program component. When prompted for the second most effective CRP component, the same number of teachers selected the funding of classroom and lab supplies as did teacher training. [For "2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program," see ED615910.]

Understanding digital supports for early learning is paramount for school readiness and later mathematics learning. We present results from a randomized control trial evaluating a digital app (Measure Up!) and a parent companion app (Super Vision) designed to teach children measurement concepts, a skill that many teachers do not feel comfortable teaching. Ninety-nine 4- and 5-year-old children were randomly assigned to one of three conditions: Measure Up!, Super Vision + Measure Up!, or a control game. Analyses revealed a statistically significant effect of being in the treatment group (Measure Up! or Measure Up! + Super Vision) on children's posttest scores (about two additional questions correct), controlling for the pretest and demographic characteristics (gender, SES). In particular, gains were made for children's understanding of pan balances. There was no significant difference between the two treatment groups. Results suggest that apps can be designed to help children learn important mathematics skills; however, more research needs to be done to understand how parent supports can be better designed. Implications for evaluation and design of game-based learning tools are discussed. [This paper was published in "Computers and Education" v146 2020.]

The National Math + Science Initiative's (NMSI's) College Readiness Program (CRP) is an established program whose goal is to promote science, technology, engineering, and mathematics education in high schools to improve students' readiness for college. The program provides teacher, student, and school supports to promote high school students' success in mathematics, science, and English Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 28 schools in the 2016-2017 school year. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 28 CRP schools and 24 control schools in 10 states. The evaluation of the CRP consisted of two parts: (1) assessment of the program's impact on selected student AP exam outcomes and (2) assessment of the fidelity of implementation of the CRP. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools The descriptive statistics showed that a higher percetange of students in the treatment schools took at least one AP course (30.7%) compared to those in the control schools (26.4%) by approximately 4.3%, however the difference was not statistically significant. In addition, students in the treatment schools were not more likely to achieve a score of 3 or higher, when compared to the delayed treatment schools. We further examined the effectiveness of the CRP using the prior year's school-level performance on the AP exam as a covariate. As with the above findings, the results indicated the probability of a student taking at least one AP course or scoring 3 or higher on at least one AP exam is not statistically different between students in the treatment schools and those in the control treatment schools. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. Overall results, however, indicated that 23 schools out of 28 treatment schools (82.1%) achieved 80% or better implementation fidelity, for an average fidelity score of 89.5%. Seven schools achieved a perfect 100% fidelity score. Looking at the different indicator groups (school, teacher and student), we found that all school support measures across all schools were implemented with fidelity. In over 80% of schools, not all teachers fulfilled their requirements for attending all training sessions, and so this component was not implemented with fidelity. Stipends and teacher awards were paid as expected as were student award payments. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports relating to helping increase student achievement in AP courses. Teacher incentives were chosen as the least important program component relating to increasing student performance by 16% of teachers and student incentives by 12% of teachers. Teachers did, however, view the student incentives as an important program component to encourage enrollment in AP courses. Likewise, students rated the financial incentives on average as somewhat important in encouraging them to participate in AP courses.

This report addresses a renewed interest in individualized instruction, driven in part by advances in technology and assessment as well as a persistent desire to increase the access, efficiency, and cost effectiveness of training and education. Using computer-based instruction we delivered extremely efficient instruction targeted to low knowledge learners in sixth-grade algebra readiness classes and eighth-grade Algebra 1A classes. Our research questions were the following: (1) To what extent can very brief exposure to instruction result in learning? and (2) How effective is the instruction compared to no exposure? We found that computer-based "instructional parcels" can be developed to provide very brief instruction that results in learning of mathematics content. The following are appended: (1) Summary of Design Guidelines Implemented in Instruction; (2) Student Background Detail; (3) Student Background Measures; (4) Sample of Pretest Items and (5) Transfer Test Items.

A large-scale randomized controlled trial tested the effects of researcher-developed learning games on a transfer measure of fractions knowledge. The measure contained items similar to standardized assessments. Thirty treatment and 29 control classrooms (~1500 students, 9 districts, 26 schools) participated in the study. Students in treatment classrooms played fractions games and students in the control classrooms played solving equations games. Multilevel multidimensional item response theory modeling of the outcome measure produced scaled scores that were more sensitive to the instructional treatment than standard measurement approaches. Hierarchical linear modeling of the scaled scores showed that the treatment condition performed significantly higher on the outcome measure than the control condition. The effect (d = 0.58) was medium to large (Cohen, 1992). Two appendices are included: (1) Descriptive Statistics of Pretest and Posttest Scores by Schools and Conditions; and (2) Summary of Efficacy Trial Procedures.

Student mathematical errors are rarely random and often occur because students are applying procedures that they believe to be accurate. Traditional approaches often view such errors as indicators of students' failure to understand the construct in question, but some theorists view errors as opportunities for students to expand their mental model and create a deeper understanding of the construct. This study examines errors in an educational video game that are indicative of two specific misunderstandings students have about fractions (unitizing and partitioning) in order to determine whether the occurrence of those errors makes students more likely to learn from the game or more likely to be confused by the game. Analysis indicates that students who made unitizing errors were more likely to be confused by the game while students who made partitioning errors were more likely to learn from the game.

Though video games are commonly considered to hold great potential as learning environments, their effectiveness as a teaching tool has yet to be determined. One reason for this is that researchers often run into the problem of multicollinearity between prior knowledge, in-game performance, and posttest scores, thereby making the determination of the amount of learning attributable to the game difficult. This study uses tests for mediation effects to determine the true relationship between in-game performance and posttest performance, determining that in this case in-game performance is a perfect mediator of prior knowledge on posttest score. (Contains 1 table, 3 figures and 1 footnote.) [This paper was supported under the Educational Research and Development Centers Program.]

The purpose of this study was to validate inferences about scores of one task designed to measure subject matter knowledge and three tasks designed to measure aspects of pedagogical content knowledge. Evidence for the validity of inferences was based on two expectations. First, if tasks were sensitive to expertise, we would find group differences. Second, tasks that measured similar types of knowledge would correlate strongly, and tasks that measured different types of knowledge would correlate weakly. We recruited and assessed four groups of participants including 46 experienced algebra teachers (2+ years experience), 17 novice algebra teachers (0-2 years experience), 10 teaching experts, and 13 subject matter experts. Results indicate one task differentiated among levels of expertise and measured several aspects of knowledge needed to teach algebra. Results also highlight that future studies should use a combination of tasks to accurately measure different aspects of teacher knowledge. (Contains 7 tables and 4 figures.) [This paper was supported under the Educational Research and Development Centers Program.]

Appropriately designed technology-based learning environments such as video games can be used to give immediate and individualized feedback to students. However, little is known about the design and use of feedback in instructional video games. This study investigated how feedback used in a mathematics video game about fractions impacted student actions in the game. Results indicated the type of feedback did not significantly affect student actions. Process data were also analyzed to identify specific student errors as well as opportunities to provide feedback for future versions of the game. Results of this study suggest that process data are a unique feature of technology-based learning environments that can be used to analyze errors and create targeted feedback for students. (Contains 1 table and 7 figures.)

Commercial video games undergo usability studies to determine the degree to which the player is able to learn, control, and understand the game. Usability studies allow game designers to improve their games before they are released to the public. If usability studies could be expanded to include information about the presentation of the instructional content, they could help improve educational video games. In this study, cluster analysis was used to identify usability information from the log files from an educational video game called "Save Patch". Cluster analysis was able to pinpoint specific levels in the game that could be improved as well as identify specific components of the level design under which certain errors were likely to occur, culminating in specific recommendations to improve the game in ways likely to increase learning. Appended are: (1) Final Attempt Distribution; and (2) First Attempt Distribution. (Contains 2 tables and 5 figures.) [This paper was supported under the Educational Research and Development Centers Program.]

In this primer we briefly describe our perspective and experience in using data logging to support measurement of student learning in a game testbed ("Save Patch") we developed for research purposes. The goal of data logging is to support the derivation of cognitively meaningful measures and affectively meaningful measures from a combination of player behaviors, game events, and game states. Key best practices we have developed are to record data that reflects behavior rather than inferences about the behavior, specify the behavior to log ahead of time, log in-game behaviors that map directly to targeted knowledge, skills, and attitudes, encode sufficient information so that the data elements are unambiguous at the desired grain size, capture context to allowing linking of the data element to an individual's specific game experience (e.g., school, teacher, period, stage, level, game event, game state), and use a structured and delimited record format. The capturing of behavioral events that presumably is a manifestation of cognitive and affective processes allows for the investigation of numerous research questions that connect game play to students' background, strategy use, knowledge, and cognitive processes. Appended are: (1) Brief Summary of Save Patch; and (2) Data Code Definition for Save Patch. (Contains 5 tables and 3 footnotes.) [This paper was supported under the Educational Research and Development Centers Program.]

In this study we compared the effects of two math game designs on math and game performance, using discrete-time survival analysis (DTSA) to model players' risk of not advancing to the next level in the game. 137 students were randomly assigned to two game conditions. The game covered the concept of a unit and the addition of like-sized fractional pieces. The math content in the baseline version of the game focused on procedures and did not elaborate on the math topics. The experimental version of the game provided more conceptual instruction by emphasizing the underlying concepts in fractional addition. Students played the game for 30 minutes. DTSA was used to examine student game performance, and whether and how game performance relates to students' prior math knowledge and game experience. Students who played the experimental version of the game were less likely to fail in the game relative to students who played the baseline version of the game (odds ratio = 0.64). Students with higher prior knowledge of fractions also were less likely to fail in the game (odds ratio = 0.41), and students with more game experience were less likely to fail (odds ration = 0.58). The use of DTSA provided an analytical method to understand game performance and game process data. DTSA enabled examination of the game play progression of students with various characteristics over sequences of game levels. Goals and Knowledge Specifications in "Save Patch" Game Levels are appended. (Contains 5 figures and 6 tables.) [This paper was supported under the Educational Research and Development Centers Program.]

The use of a narrative in educational contexts has been found to increase learners' experience of flow or absorption in a task. This increased experience of flow can in turn result in increased retention and learning outcomes. However, narrative can also be polarizing particularly in the male-dominated realm of video game play due to gender stereotyping or underrepresentation of females in games. Considering studies on narrative, flow, and gender differences in video games, it is expected that masculine and feminine narratives used in an educational game would result in differences in learning for the two genders. This study implemented two narratives into a rational numbers math video game to compare engagement and learning outcomes between genders and in comparison to a non-narrative version of the game. One hundred seventy-seven students enrolled in a remedial math course, pre-algebra, or Algebra 1 participated in the main study; this included 85 males, 80 females, and 12 students who did not report their gender. Results indicate that narrative did increase student experience of flow and positive perceptions of the game, especially when males were matched with the masculine version of the game. Increased learning outcomes took place only when students were placed in the masculine narrative and when males were matched to the masculine narrative. Appended are: (1) Narrative Screens; (2) Rubric for Open-Ended Narrative Question; (3) Background Information Survey; (4) Video Game Experience Survey; and (5) Study Missing Data Analyses. (Contains 16 tables and 5 figures.) [This paper was supported under the Educational Research and Development Centers Program.]

This study is an exploratory study of the relationship between collaboration and mathematics and game outcomes in a video game aimed at teaching concepts related to rational numbers. The sample included 243 middle school students who played the video game either with one partner or individually for 40 minutes. Results suggest that participants with high and low prior knowledge benefit from different conditions during gameplay. Participants with low prior knowledge tend to perform better on math outcomes by working collaboratively and participants with high prior knowledge tend to perform better on math outcomes by working individually. These results are similar to prior findings from classroom research which indicates that collaboration is more effective for low-performing students. These results have implications for designing game environments for low-performing students. Appended are: (1) Math Assessment Items; (2) Background Questions; and (3) Missing Data Analysis. (Contains 3 figures and 10 tables.) [This paper was supported under the Educational Research and Development Centers Program.]

In this study, we compared the effects of simulator-based virtual ultrasound scanning practice to classroom-based hands-on ultrasound scanning practice on participants' knowledge of FAST window quadrants and interpretation, and on participants' performance on live patient FAST exams. Twenty-five novice participants were randomly assigned to the simulation-practice condition and 24 participants to the classroom-practice condition. Participants were mostly medical school students, with some nursing students. Participants who received simulation-based practice scored significantly higher on interpreting static images of FAST windows. On live patient exams where participants scanned the RUQ, LUQ, and suprapubic quadrant of a normal patient and an ascites positive patient (6 quadrants total), there were no statistical differences between the two conditions on scan time, window acquisition, and window interpretation except for the following: the classroom-practice condition had shorter scan time for the LUQ, and a higher number of participants attaining high window quality on the RUQ (normal patient only) and suprapubic quadrant (positive patient only) and correct window interpretation on the LUQ (normal patient only). Performance was statistically similar on all other window quality and window interpretation measures on the other quadrants. Overall, classroom-based practice appeared to promote physical acquisition skills (e.g., acquiring a window) and simulator-based practice appeared to promote window interpretation skills. The simulator used (SonoSimulator[TM]) was well received by participants and appears promising as a training tool not only to increase probe time, but also to increase exposure to FAST windows reflecting various anatomy and disease states. Appendices include: (1) Posttest Form (Knowledge of Ultrasound and Background Survey); (2) Prior Knowledge of Anatomy Items; (3) Participant Feedback Form; (4) Participant Comments; (5) Picture of a Classroom Practice Session; (6) Picture of the Lab Setup; (7) Classroom Practice Observation Form; (8) Performance test Observation Form; (9) Picture of the Simulation Setup; (10) Simulation User Interface Screenshots; (11) Simulation Condition Directions; (12) Ultrasound Simulation Report; and (13) Sonographer Feedback Form. (Contains 11 tables, 3 figures and 3 footnotes.)

The UCLA National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is under contract from the Naval Postgraduate School (NPS) to conduct research on assessment models and tools designed to support Marine Corps rifle marksmanship. In this deliverable, we first review the literature on known-distance rifle marksmanship research in Section I. In Section II we examine USMC combat marksmanship. Because the USMC Combat Marksmanship Program (CMP) (including moving targets) is new, there is no current research that we know of that has specifically examined assessment of CMP elements and we thus identify assessment issues. Section III reviews the use of sensors in the assessment of rifle marksmanship skill, and we provide detailed information on our prior work on developing and validating measures for known-distance rifle marksmanship. Section IV provides an overview of various marksmanship training systems in use by the USMC as of 2006. Based on the available information, we examined each system with respect to capabilities needed to support research on marksmanship. Section V provides a detailed research plan and schedule intended to investigate the assessment issues covered in Sections I to IV. Appendices include: (1) Key Knowledge Components; (2) Key Rifle Marksmanship Facts; (3) Key Rifle Marksmanship Cause-Effect Relations; and (4) Rifle Marksmanship Training Systems (USMC, 2006). (Contains 11 tables, 11 figures and 1 footnote.)

This study provides empirical evidence of a highly specific use of games in education--the assessment of the learner. Linear regressions were used to examine the predictive and convergent validity of a math game as assessment of mathematical understanding. Results indicate that prior knowledge significantly predicts game performance. Results also indicate that game performance significantly predicts posttest scores, even when controlling for prior knowledge. These results provide evidence that game performance taps into mathematical understanding. (Contains 2 figures.)

The goal of this report was to test the use of sensor-based skill measures in evaluating performance differences in rifle marksmanship. Ten shots were collected from 30 novices and 9 experts. Three measures for breath control and one for trigger control were used to predict skill classification. The data were fitted with a logistic regression model using holdout validation to assess the quality of model classifications. Individually, all four measures were significant; when considered together, only three measures were significant predictors for level of expertise (p less than 0.05). Overall percent correct in shot classification for the testing data was 90.0%, with a sensitivity of 67.5%, and 96.0% specificity. (Contains 9 tables and 1 footnote.) [This report was supported by a grant from the Advanced Brain Monitoring, Inc., PR/Award Number 20064169.]

Measures of rifle marksmanship skill and performance were developed using a prototype instrumented laser-based training system. Measures of performance were derived from laser strikes on a video-projected target. Measures of rifle marksmanship skill--breath control, trigger control, and muzzle wobble--were developed from shooters' breathing and trigger squeeze patterns. Existing marksmanship instructional materials and expert shooters' breath and trigger control profiles guided the development of the skill measures. A shooter's breath control was described as where and how long into the respiratory cycle the trigger broke. A shooter's trigger control was described as the duration of the trigger squeeze. A shooter's muzzle was described as the total acceleration during the two seconds prior to the shot. The use of sensor-based measures provides insight into exactly how a shooter is executing two of the three skills considered to be the fundamentals of rifle marksmanship. (Contains 9 figures, 2 tables and 1 footnote.) [This report was supported by a grant from the Advanced Brain Monitoring, Inc., PR/Award Number 20064169.]

In this report, researchers examined whether individualized multimedia-based instruction would influence the development of rifle marksmanship skills in novice shooters with little or no prior rifle marksmanship experience. Forty-eight novice shooters used an M4 rifle training simulator system to shoot at an 8-inch target at a simulated distance of 200 yards. Participants received either (a) no instruction, (b) only an overview of rifle marksmanship, or (c) an overview and instruction targeted at particular skill gaps. Support was found for the idea that multimedia-based instruction can be highly effective for novices, with a large increase in shooting performance observed after 10 to 15 minutes of multimedia instruction. Subsequent individualized instruction using very short multimedia instruction appeared to be effective in shaping participants' skills toward an "ideal" state consistent with shooting doctrine. (Contains 2 figures, 11 tables and 1 footnote.) [This report was supported by a grant from the Advanced Brain Monitoring, Inc., PR/Award Number 20064169.]

In this report, researchers examined rifle marksmanship development within a skill development framework outlined by Chung, Delacruz, de Vries, Bewley, and Baker (2006). Thirty-three novice shooters used an M4 rifle training simulator system to learn to shoot an 8-inch target at a simulated distance of 200 yards. Cognitive, psychomotor, and affective measures were gathered in addition to measures of performance and component skills. Partial support was found for rifle marksmanship skill development following Ackerman's (1988) skill development theory. Support was found for the idea that known distance rifle marksmanship can transition rapidly from a learning phase to a practice phase, and that the cognitive and affective variables have a substantial influence on performance and skill development during the learning phase. (Contains 5 tables.) [This report was supported by a grant from the Advanced Brain Monitoring, Inc., PR/Award Number 20064169.]

In this study we explored the effectiveness of individualizing instruction using a system that diagnosed students' knowledge gaps in pre-algebra and provided individualized instruction and practice. One hundred middle school students participated, and received either no instruction or individualized computer-based instruction. Diagnosis of individual student knowledge gaps was based on a rapid 90-item pretest that mapped to a Bayesian network of presumed knowledge dependencies among various pre-algebra concepts. Given a diagnosis, a student received computer-based direct instruction. Instruction was generally less than 5 minutes followed by practice opportunities and its design was based on empirically supported features. In 4 out of 6 posttest scales, compared to students receiving no instruction, students who were diagnosed as low-knowledge on a concept and received instruction on that concept performed higher on the posttest scale for that concept with effect sizes ranging from .76 to .91. This finding held only for students who were engaged in the task (i.e., compliant as measured by participants' clickstream). This finding provides support for the idea that very brief instruction can be developed for pre-algebra concepts, be matched to an individual, and be effective. Implications are discussed for the development of "dumb" instructional and assessment systems (vs. "smart" systems such as intelligent tutoring systems), empirically guided instructional design, instructional efficiency using technology-enabled techniques, and the importance of gathering process data to measure degree of implementation. (Contains 7 tables. The following are appended: (1) Summary of Design Guidelines Implemented in Instruction; (2) Student Background Detail; (3) Student Background Measures; (4) Sample of Pretest Items; and (5) Transfer Test Items.)

Because of the great promise of distance learning for delivering cost-effective instruction, there is great interest in determining whether or not it actually is effective, and--more interesting--determining what variables of design and implementation make it more or less effective. Unfortunately, much of the research has been based on simple comparisons of distance learning to the "traditional" method of instruction rather than examining the variables influencing the effectiveness of distance learning. In addition to not manipulating or controlling important independent variables, the dependent measures used in such studies are often inappropriate, ranging from the obviously inadequate, e.g., the "smile test," to standardized tests that have known psychometric properties but are not aligned with course objectives, to homegrown measures that appear to be aligned with instructional objectives but are of unknown reliability and validity. We have addressed the problem of limitations in dependent measures with research on measures of student achievement based on families of cognitive demands, and have developed assessment models for these families that can be used to design assessments across a variety of subject matters. We have also developed computer-based assessment tools implementing the models, including tools for data collection, scoring, analysis and reporting, assessment authoring, and knowledge acquisition and representation. With support from the Office of Naval Research we have developed a distance learning testbed to apply these models and tools to distance learning research and evaluation. This paper describes our summer of 2004 testbed implementation and presents three examples of the research conducted in the testbed on methods for assessing human performance via distance learning technologies. (Contains 3 figures.)

A critical first step in developing training systems is gathering quality information about a trainee's competency in a skill or knowledge domain. Such information includes an estimate of what the trainee knows prior to training, how much has been learned from training, how well the trainee may perform in future task situations, and whether to recommend remediation to bolster the trainee's knowledge. This paper describes the design, development, testing, and application of a Web-based tool designed to assess a trainee's understanding of a content domain in a distributed learning environment. The tool, called the CRESST Human Performance Knowledge Mapping Tool (HPKMT), enables trainees to express their understanding of a content area by creating graphical, network representations of concepts and links that define the relationships of concepts. Knowledge mappers have been used for several years, almost always as an aid for organizing information in support of problem solving or in instructional applications. To use knowledge maps as assessments there must be a reliable scoring method and there must be evidence for the validity of scores produced by the method. Further, to be practical in a distributed learning environment, the scoring should be automated. The HPKMT provides automated, reliable, and valid scoring, and its functionality and scoring method have been built from a base of empirical research. We review and evaluate alternative knowledge mapping scoring methods and online mapping systems. We then describe the overall design approach, functionality, scoring method, usability testing, and authoring capabilities of the CRESST HPKMT. The paper ends with descriptions of applications of the HPKMT to military training, limitations of the system, and next steps. (Contains 2 tables and 3 figures.)

In this study we report on a test of a method that uses ontologies to individualize instruction by directly linking assessment results to the delivery of relevant content. Our sample was 2nd Lieutenants undergoing entry-level training on rifle marksmanship. Ontologies are explicit expressions of the concepts in a domain, the links among the concepts, and the governing constraints of these links. We have developed an ontology for the domain of rifle marksmanship. The ontology contains over 160 concepts and over 160 relationships that capture the different types of relations among the concepts (e.g., causal, part-whole, classifying, functional). The content was drawn from Marine field manuals, and interviews with snipers and coaches. Concepts were tagged with instructional content (e.g., definitions, explanations, elaborations, multimedia examples). Relations were tagged with an explanation of why the particular relation holds under particular conditions. Assessment is tied to instruction via influence (Bayesian) networks. Performance on assessment items determines what content is pulled from the ontology for delivery. For example, if a Marine scores poorly on all assessment items related to breathing control, then instructional content tied to the ontology concept "breathing control" (and any linked concepts) could be delivered. Conversely, if a Marine scores low on items that suggest poor knowledge of the shot group associated with poor breathing control, then only a shot group related to breathing might be delivered. Our test of this approach appears feasible and promising. The Bayesian network appeared to be successful in identifying knowledge gaps, and relevant and targeted content was served to Marines. Learning appeared to be occurring at a faster rate over time for Marines who received targeted instruction compared to Marines in a control group. Implications are discussed. (Contains 3 tables and 4 figures.) [The work reported herein was supported under the Office of Naval Research.]

Assessment models support the design of quality performance assessments. Assessment tools are being developed to enable easy and effective application of the models. Based on representations of assessment design knowledge and domain knowledge in ontologies, the tools provide guidance to assessment designers, and through constraint processing check the completeness and accuracy of designs. With the addition of Bayesian networks, the tools can also enable individualized instruction by identifying knowledge gaps and prescribing instruction to fill the gaps. This paper describes the technical approach to developing the tools and discusses applications of ontologies and Bayesian networks for assessment authoring and individualized instruction. (Contains 2 figures and 1 table.) [This report was re-published as a CRESST report with permission from the International Test and Evaluation Association (ITEA) Journal. The original article was published in the ITEA Journal, March/April 2004, 25(1), 47-52.]

Our research question was whether we could develop a feasible technique, using Bayesian networks, to diagnose gaps in student knowledge. Thirty-four college-age participants completed tasks designed to measure conceptual knowledge, procedural knowledge, and problem-solving skills related to circuit analysis. A Bayesian network was used to model the knowledge dependencies among the circuit analysis concepts. Preliminary results suggested that the Bayesian network was generally working as intended. When high- and low-performing groups were formed on the basis of posterior probabilities, significant group differences were found favoring the high-performing group with respect to circuit definitions and circuit analysis problems, for both actual and self-assessments, and higher major GPA. The Bayesian network was able to predict participants' performance on a problem-solving item on average 75% of the time. The findings of this study are promising for our long-term goal of developing scalable and feasible online automated reasoning techniques to diagnose student knowledge gaps. (Contains 12 tables and 2 figures.) [Appended are: (1) Node-Voltage Analysis Problem-Solving Procedure (Kaiser, 2003); and (2) Bayesian Network.]

Collaborative problem-solving skills are considered necessary skills for success in today's world of work and school. Cooperative learning refers to learning environments in which small groups of people work together to achieve a common goal, and problem solving is defined as "cognitive processing directed at achieving a common goal when no solution method is obvious to the problem solver" (R. E. Mayer & M. C. Wittrock, 1996, p. 47). Thus, collaborative problem solving is defined as problem-solving activities that involve interactions among a group of individuals. This paper will address several key issues (e.g., theory and measurement of collaborative problem solving and issues in measuring problem-solving processes). People rely on computerization of the administration, scoring, and reporting of collaborative problem-solving skills, thus potentially increasing reliability and validity. (Contains 1 figure.) [A revised version of this report is published as: O'Neil, H. F., Chuang, S., & Chung, G. K. W. K. (2003). Issues in the computer-based assessment of collaborative problem solving. Assessment in Education, 10, 361-373.]

This study examined a civil engineering capstone course that embedded a sophisticated simulation-based task within instruction. Students (n=28) were required to conduct a hazardous waste site investigation using simulation software designed specifically for the course (Interactive Site Investigation Software) (ISIS). The software simulated physical processes as well as real-world engineering processes. The main research question focused on how ISIS impacted students learning of course-related content, ability to handle complex, open-ended problems, attitudes toward ISIS and the course, and use of teamwork processes. Evidence was also gathered on the validity of knowledge mapping for advanced students. Assessment of student outcomes was based on the use of constructed-response knowledge maps to measure content understanding and surveys to measure student attitudes, use of cognitive processes, and use of teamwork processes. Students reported very positive attitudes toward the ISIS experience, demonstrated significant gains in knowledge across the course of ISIS use, and perceived the ISIS activity as being generally effective in improving their skills in handling complex projects linking theory to real-world applications, and improving their problem solving performance. The assessment of student outcomes was a successful first attempt, but more work is needed to validate the measures with advanced students, particularly when the task is complete and requires interdisciplinary knowledge and teamwork skills. Nines appendixes contain the study instruments. (Contains 18 tables and 39 references.) (Author/SLD)

This paper discusses the use of ontologies and their applications to assessment. An ontology provides a shared and common understanding of a domain that can be communicated among people and computational systems. The ontology captures one or more experts' conceptual representation of a domain expressed in terms of concepts and the relationships among the concepts. The ontology standardized the terms and structures of the domain. The paper presents two examples of the use of ontologies for assessment purposes. The first example describes than application for the domain of rifle marksmanship, and the second example discusses the use of ontologies in the design of authoring systems to link assessment, content, and cognitive demands. The overall purpose for developing the ontology of rifle marksmanship was to capture the knowledge and structure of the domain in a way that allows the exploration of the use of ontologies for assessment and instructional purposes. In the other example, the ontology was developed to use as a domain template against which to score knowledge maps. The technology behind ontologies has matured to the point that computational tools are readily available. An ontology-based authoring system can impose structure on the authoring task and check that user-specified values simultaneously satisfy all constraints among the assessment components. (Contains 28 references.) (SLD)

Knowledge mapping is expected to measure deep conceptual understanding and allow students to characterize relationships among concepts in a domain visually. This research examined the validity of knowledge mapping as an assessment tool in science. The approach to investigating this validity was three-pronged. First, a model was outlined for the creation of knowledge mapping tasks, proposing a standard set of steps and using content area and educational experts to ensure the content validity of the measures. Next, a scoring method was developed to evaluate student performance. This report contains a discussion of the methods reliability and its relation to other possible scoring systems. Finally, the report presents the statistical results based on the participation of 56 fourth and fifth graders, including comparative analyses, the multitrait-multimethod (MTMM) validity analyses involving two traits (students understanding of hearing and of vision), and three different measurement methods (knowledge mapping, essay, and multiple-choice tasks), critical proposition analyses, and analyses of students, propositional elaborations. Results show knowledge maps to be sensitive to students competency level, with mixed MTMM results. The report concludes with a discussion of implications and directions for future work. (Contains 7 tables, 2 figures, and 33 references.) (Author/SLD)

The feasibility and validity of using a computer-based networked collaborative knowledge mapping system to measure teamwork skills was examined. Student groups (10 groups of 3 ninth graders each) were assessed with the system twice in the academic year, once in the fall and once the following spring. The study focused on the nature of the interaction among team members as they jointly constructed a knowledge map. Each student was randomly assigned to a team and communicated (anonymously) with other members by sending predefined messages. Teamwork processes were measured by examining message usage. Each message was categorized as belonging to one of six team processes: (1) adaptability; (2) communication; (3) coordination; (4) decision making; (5) interpersonal; and (6) leadership. Team performance was measured by scoring each team's knowledge map using four expert maps as the criterion. No significant correlation was found between team processes and team outcomes. This unexpected finding may be due in part to a split-attention effect resulting from the design of the user interface. However, student teams were able to construct knowledge maps successfully, suggesting that the general approach to using networked computers to measure group processes remains viable. Two appendixes contain messages grouped by category and message content and the handout describing messages. (Contains 17 tables and 21 references.) (Author/SLD)

This report examines the feasibility of scoring essays using computer-based techniques. Essays have been incorporated into many of the standardized testing programs. Issues of validity and reliability must be addressed to deploy automated approaches to scoring fully. Two approaches that have been used to classify documents, surface- and word-based techniques, are reviewed. The candidate approaches to the automated classification of documents are reviewed, and then how these approaches could be used to achieve the overarching goal of the automated scoring of essays is discussed. The two approaches considered are Project Essay Grade (PEG) (A. Daigon, 1966; E. Page, 1966, 1968, 1994; E. Page and N. Peterson, 1995) and latent semantic analysis (LSA) (P. Foltz, 1996; T. Landauer, D. Laham, B. Rehder, and M. Schreiner, 1997). PEG uses a regression model where the independent variables are surface features of the text (document length, word length, and punctuation) and the dependent variable is the essay score, and LSA is based on a factor-analytic model of word co-occurrences. Following the review of PEG and LSA, additional uses of automated scoring of text-based data are explored. The final section outlines a plan for a feasibility study of the automated processing of text-based data. One assumption of this report and of both scoring approaches is that the human rating is the best estimate of the true essay score, and that there will almost always be a need for some portion of the documents to be scored using multiple trained raters. The potential of the proposed investigation lies more in the potential for practical spin-offs than in any theoretical contribution to writing, education, assessment, or cognition. (Contains 5 figures, 2 tables, and 50 references.) (SLD)

The feasibility of using a computer-based networked collaborative concept mapping system to measure teamwork skills was studied. A concept map is a node-link-node representation of content, where the nodes represent concepts and links represent relationships between connected concepts. Teamwork processes were examined for a group concept mapping activity using networked computers and HyperCard software developed for the study. Two pilot studies investigated the feasibility of the approach, and one of these is reported in this study. Thirty 9th graders worked together in the first pilot study to create collaborate maps on the computer systems. Results of this study and a second confirmatory study resulted in development of a study with 23 groups of 3 students each from middle schools and high schools. Usable data were recorded for 15 groups (45 participants). Results suggest that the real-time measurement of teamwork skills is feasible and that students using the networked computers were able to create a concept map jointly. The number of concepts used by groups ranged from 7 to 18, and the number of links ranged from 8 to 26. It is suggested, however, that the task involved may be more like a small group task than a "team" task if the purpose of a team is defined as performing a task an individual cannot perform alone. The developed software architecture has been designed to be domain independent and should be useful in other computer-based team environments. (Contains 1 figure, 8 tables, and 20 references.) (SLD)