Search

BACKGROUND Formal examinations are often used in engineering classes as the tool to evaluate student learning. These exams are often high stakes assessment tasks and provide no opportunity for feed-forward. Despite academic claims that all topics in their subject are requisite material, students are regularly able to pass these assessment tasks with unsatisfactory, and perhaps even no capacity to demonstrate learning in some topics. Furthermore, while undertaking the exam often highlights to students their learning deficiencies, it typically has no impact on their learning as they rarely receive feedback other than a mark or grade and there is no further opportunity to address these learning gaps. This paper reports on the impact of a two-staged examination process on both student learning and assurance of that learning. PURPOSE The aim of the staged examination process was to improve confidence that students had satisfactory knowledge in all requisite subject topics and to test its capacity to be learning-oriented in that it provides improved opportunities for students to learn while simultaneously increasing the level of learning assurance. DESIGN/METHOD The first stage of the process was an exam that covered all requisite subject topics. This exam consisted of multiple choice questions set at or just above the level of threshold learning outcomes. Students were required to score 80% on this exam to qualify to undertake the second part of the assessment process at a later date. Students used IFAT (Immediate Feedback Assessment Technique) cards for this stage to facilitate immediate feedback as to their strengths and weaknesses. The time between exams allowed students to review identified areas of weakness before attempting the second stage of the exam. Note: while not contributing to their final grade students who failed the first exam were also permitted to undertake the second exam as an opportunity to learn and as a means of evaluating the impact of the process. The second exam consisted of open-ended questions requiring students to explain their critical thinking and judgement used to arrive at their answer. Evaluation of the effectiveness of this process was based on a student survey, focus group discussions and an analysis of student examination scripts. RESULTS The threshold learning outcome exam was effective in improving assurance of learning in that students had to demonstrate satisfactory learning across topics to achieve the 80% required to âpassâ the exam. Furthermore, students reported that they used the opportunity between exams to address identified learning gaps, hence demonstrating the learning orientation and feed-forward capacity of the two stage process. However, the fact that two students who did not achieve the threshold level of 80% in the first exam were able to address their learning gaps and pass the second and harder exam suggests that an alternative to the 80% exclusion criteria should be considered. CONCLUSIONS The study demonstrated that a two staged examination process improved confidence in assurance of learning while providing students with an opportunity to first identify and subsequently reduce learning gaps. However, the fact that some students who failed the threshold exam demonstrated significant improvement in their understanding in the second exam suggests that more research is needed to both understand the impact of and improve the benefits from this activity.

It is well recognised that changes in the contemporary world demand that professionals continuously learn. Often this continual professional learning takes the shape of formal courses, seminars and other obviously educational events. The starting point of this paper is an acknowledgement that people also learn in the day-to-day practices that constitute their work. Work can be understood as a bundle of practices that are typically shared by most people employed in that profession. For engineers, and experienced engineers in particular, an example might be attending design review meetings, toolbox talks and or carrying out site walks. In this paper we posit that these practices afford important opportunities for professional learning.

BACKGROUND The benefits to student learning from participation in formative assessments have been reported by many educational researchers and scholars. This literature reports improved engagement when students see formative activities as being highly relevant and valuable for their learning. However, many academics still report that students are reluctant to participate in a learning activity unless it contributes some marks to their final subject grade. PURPOSE The purpose of this study was to investigate the impact on learning of undertaking a series of out-of-class formative learning activities. In addition, for the students that undertook these activities, we are interested in why they chose to participate and what, if any, learning benefits they experienced from participating. In regards to the students that chose not to undertake the activities, we are interested in their reasons for not participating. We are also interested in investigating the impact on students of scaffolding the learning activities used throughout the semester. DESIGN/METHOD For the Autumn 2012 offering of Engineering Mechanics, out-of-class formative learning activities were designed for four topics in the syllabus. These activities required the students to read the relevant section of the textbook and subsequently answer multiple-choice questions provided online including entering comments to explain their chosen answer. While there were no marks allocated to these activities, the instructor did allocate more time in lectures to the material related to questions with the largest variation in responses. Students who answered the online questions could log back on at the end of the submission period to compare their responses to the instructorâs answers and reasoning. At the conclusion of the semester students were asked to complete a survey consisting of both closed and open-ended questions to investigate their perceptions of the effects of the online activities. Descriptive statistics were used to analyse the closed responses to this survey and also to determine if there is any significant correlation between participation in the formative assessment task and subsequent performance in summative assessment tasks in this subject. RESULTS The results show that the majority of students in the class did not undertake all of the activities. The results also indicate that students that participated in 3 or more activities scored, on average, almost one grade increment higher than those that participated in 0 or 1 activity. In addition, increases in the institutional student feedback survey results for the subject indicate an improvement in student perception of the learning experience compared to the previous semester. CONCLUSIONS Students who participated in the formative assessment activities in this subject benefitted in terms of their learning which was suggested in the final summative assessment task. These results and what we learnt about studentsâ motivation to participate will be used in future semesters to demonstrate to students the link between participating in the formative activities and scoring well in the summative tasks. We argue that making this link explicit and scaffolding the learning activities are the keys to improving students motivation to learn.

BACKGROUND Arguably the most important opportunity to acquire the standards and norms of any discipline and develop researchers' judgement is the peer review process â and this is probably particularly true in an emerging discipline such as engineering education. Ironically, research in many disciplines has established that the review process is deeply flawed in conception as well as (often) in operation, with the American Medical Association asserting that if peer review were a drug it would never be allowed on to the market. And yet university ranking systems for published research, on which all of our careers depend, rely on this flawed instrument. With this in mind we have been examining how members of our community (AAEE) give and respond to reviews with a view to making the process more useful. PURPOSE Reviewing is an inexact and subjective process so it would be misguided to think that somehow inter-rater reliability or some notion of objective âtruthâ may be attained. Instead, we ask what reviewers need to do to provide helpful advice that can help shape norms and standards in the field. DESIGN/METHOD In previous work (Willey et. al. 2011; Jolly et.al. 2011) there appeared to be a need for well-expressed criteria that would guide authors on what a publication should contain and guide reviewers in how judgements should be made. With the help of a Delphi panel made up of 12 international researchers in the field a set of criteria were developed. Volunteers were then sought to apply the criteria to sample texts in an online tool (SPARKplus). Individual interviews with some respondents were then used to clarify participantâs understandings and goals. RESULTS The criteria developed by the Delphi panel are those being used for this conference. The members of the panel particularly approved the âcommentsâ accompanying the criteria per se which were intended primarily as guidance to authors about acceptable practice. Anecdotal evidence to date suggests that authors should find the criteria and comments clarify expectations but the matter of standards will remain. The use of the criteria in the second stage and analysis of the discussions in particular will produce information both about present expectations and practices and visions of future growth and improvement. CONCLUSIONS Our analysis of the stage 2 data will aim to describe consensus on research quality and how to use the peer review process to help attain it, in the form of recommendations for future application of the criteria, in journals as well as at conferences. Our international experts from the Delphi panel have expressed an interest in being involved in stage 2 and informed about the outcomes so the potential also exists for this community to develop best practice peer review in engineering education thorough the sharing of their expertise in this way.