Search

In this article we describe a three-year study that was conducted in three phases to evaluate the feasibility of assessing digitized portfolios of student creative work for high-stakes purposes. The first two phases suggested that creative work could be digitized with adequate fidelity, and that students could submit their own work from schools to an online portfolio system. These portfolios were assessed using online tools to facilitate both an analytical and comparative judgment method of scoring, with the latter appearing to be the more reliable. In the final phase this approach was evaluated to facilitate moderation and professional learning of standards with rural teachers. The study showed that comparative judgment is a viable and reliable method for assessing digitized creative products and coupled with online communication systems provides an effective and efficient approach to scoring, moderation and teacher professional development, particularly in rural locations. [ABSTRACT FROM PUBLISHER]

This paper reports on the outcomes of a three-year study investigating the use of digital technologies to increase the authenticity of high-stakes summative assessment in four Western Australian senior secondary courses. The study involved 82 teachers and 1015 students and a range of digital forms of assessment using computer-based exams, digital portfolios and audiovisual recordings. The results were analysed using a feasibility framework concerning manageability, technical facility, functional operation and pedagogic alignment. By the end of the study, each form of assessment that was implemented was found to be feasible once some obstacles were overcome. Two methods of marking were tried, analytical rubric-based marking and holistic comparative pairs marking, with the latter found to generate more reliable scores. With the increased use of digital technologies in schools and the expectation that children will achieve more complex performances, more use of digital forms of assessment will be required. [ABSTRACT FROM PUBLISHER]

The collection and scoring of student artwork, for high-stakes assessment, across a large jurisdiction such as Western Australia is challenging. An alternative approach would be to submit digital representations of the artworks online for assessing. However, to give a valid and reliable measure the representations would need to be of adequate quality. Further, judgements of artworks are necessarily subjective giving concern about the reliability of scoring. The comparative pairs method of scoring lends itself to addressing this problem and is feasible where the work is in digital form. This paper reports on one component of a three-year study to investigate the representation of student practical work in digital forms for the purpose of summative assessment. The first phase of the project involved the researchers creating digital representations of the artwork submitted at the end of secondary schooling by a sample of students. The second phase involved a sample of students creating digital representations of their own work and submitting them through an online system. The study found this process was feasible, and the results were acceptable, but it lacked support from teachers and students who wanted the original artworks to be assessed. The comparative pairs method of scoring was found to provide reliable scores and be well suited to making judgements around creative work. [ABSTRACT FROM AUTHOR]

This paper presents the findings of the first phase of a three-year study investigating the efficacy of the digitisation of creative practical work as digital portfolios for the purposes of high-stakes summative assessment. At the same time the paired comparisons method of scoring was tried as an alternative to analytical rubric-based marking because we believed that it was likely that a more holistic approach to scoring would be more appropriate. Researchers created digital representations of the practical submissions of 75 Visual Arts and 82 Design students graduating from secondary school in Western Australia. These digital portfolios were scored using the two methods, with the scores compared to those officially awarded to the physical forms. It was concluded that the digital representations of the Visual Arts submissions had adequate fidelity for the purpose of awarding high-stakes scores particularly using the paired comparisons method. However, the Visual Arts teachers and students were opposed to digitisation. For the Design portfolios teachers and students were supportive of digital submission, but the structure of the portfolios reduced the reliability and validity of scores, particularly from analytical marking. [ABSTRACT FROM PUBLISHER]

There is little doubt that the curriculum content and pedagogy in schools is driven by the structure and forms of assessment employed, particularly for summative purposes. When most such assessment was limited to what a student could do with a pen and paper in short 'exams' this pushed the content of the curriculum towards small descriptive chunks, and the pedagogy towards memorisation and replication techniques. Over the past two decades alternative forms of assessment supported by the power of computer systems have been conceived and tried. This paper discusses progress towards various forms of computer-based exams and how these may encourage curriculum and pedagogy suited to 21st Century learning. In particular it draws attention to research I have led to investigate the feasibility of a number of forms of computer-based exams for high-stakes summative assessment in secondary education. The result was the successful development and implementation of audiovisual stimuli and response computer-based exams, and digital production exams resulting in portfolios of evidence. Each implemented form of exam was tested for feasibility to ensure a defensible balance of manageability, reliability and validity. I believe that for most highstakes summative assessment in Australian schools it is time to replace the ancient paper-based technology with computer-based technologies. [ABSTRACT FROM AUTHOR]

In this paper we report on one component of a three-year study into the use of digital technologies for summative performance assessment in senior secondary courses in Western Australia. One of the courses was Italian Studies, which had an oral communication outcome externally assessed with an oral performance for which students travelled to a central location and undertook an interview with two assessors. Apart from the logistical difficulties for both students and the organising body, this method did not leave an enduring record of the process, and raised questions about the reliability of the assessment. Over the three years of this study, we tried several approaches to using digital technology to assess oral performance, including a portfolio of sub-tasks leading up to a video-recorded oral presentation, a computer-based exam, a video recorded interview, and an online exam that included oral audio-recordings. For each of the years online marking tools supported two methods of drawing inferences about student performance from the representations: the more traditional analytical method and the comparative pairs method. Rasch analysis of the results of the two methods showed that both were at an acceptable level of reliability. Overall, students and teachers reported that they liked using audiovisual recordings and online performance tasks for revision but not for summative assessment. The study also demonstrated that the scores from externally marked computer-based oral tasks carried out in class time correlated highly with the scores from traditional face-to-face recorded interviews. Therefore, online assessment of oral performance appears to be an equally effective way to facilitate assessment when compared with traditional methods and offers other affordances, such as convenience and access from a variety of locations, as well as providing an enduring record of student performance. [ABSTRACT FROM PUBLISHER]

There is a critical need for research into the use of digital technologies to support the assessment of performance on complex tasks in schools. This paper reports on a component of a pilot study aimed at investigating the use of digital forms of performance assessment, manageable within schools, with high levels of reliability and capable of being scaled up for statewide implementation. Student practical performances were represented in digital files, placed in an online repository and marked using both a standards-reference and comparative pairs method. It was found that the short performance tasks exam was readily implemented and the work easily marked. The resulting scores were shown to be highly reliable and well correlated when compared with the other case studies and suitable for ranking students. However, the limitations of the tasks, time and scaffolding of the tasks reduced the study's validity in assessing student capability in the course. Overall the results were encouraging and provide a basis for a larger and longer study that has commenced. [ABSTRACT FROM AUTHOR]

Research on Information and Communications Technologies (ICT) use in schools is increasingly finding that the impact on learning outcomes is not directly causal but depends on how the technologies are used within learning environments. This paper presents one key dimension, the 'Learning Environment Attributes' dimension, of a complete literature-based framework — the 'New ICT Supporting Schooling' (NISS) framework — designed to provide systematic rich guidance for teachers and school leaders in deciding how to facilitate ICT use. A substantial trial showed that this theory-building stage has produced an effective rubric-based tool to facilitate the measurement of this dimension. Indications are that it is particularly useful for whole-school analysis. [ABSTRACT FROM AUTHOR]

This article begins by constructing an argument from the literature to support the view that research into the use of computers in classrooms must consider the overall learning environment. This provides the rationale for the development and use of the New Classroom Environment Instrument (NCEI) in research conducted into the use of portable computers in a school. This study particularly focused on the impact of the computers on classroom learning environments. An example is given of the use of the instrument to investigate the use of the portable computers with classes of 13‐year‐old students. The intention of the article is not to present the results of the study but to demonstrate the use of the instrument and encourage others to use such instruments in educational computing research. [ABSTRACT FROM AUTHOR]