Search

E-learning and particularly distance-based learning is becoming an increasingly important mechanism for education. A leading Virtual Learning Environment (VLE) reports a user base of 70 million students and 1.2 million teachers across 7.5 million courses. Whilst e-learning has introduced flexibility and remote/distance-based learning, there are still aspects of course delivery that rely upon traditional approaches. The most significant of these is examinations. The lack of being able to provide invigilation in a remote-mode has restricted the types of assessments, with exams or in-class test assessments proving difficult to validate. Students are still required to attend physical testing centres in order to ensure strict examination conditions are applied. Whilst research has begun to propose solutions in this respect, they fundamentally fail to provide the integrity required. This thesis seeks to research and develop an e-invigilator that will provide continuous and transparent invigilation of the individual undertaking an electronic based exam or test. The analysis of the e-invigilation solutions has shown that the suggested approaches to minimise cheating behaviours during the online test have varied. They have suffered from a wide range of weaknesses and lacked an implementation achieving continuous and transparent authentication with appropriate security restrictions. To this end, the most transparent biometric approaches are identified to be incorporated in an appropriate solution whilst maintaining security beyond the point-of-entry. Given the existing issues of intrusiveness and point-of-entry user authentication, a complete architecture has been developed based upon maintaining student convenience but providing effective identity verification throughout the test, rather than merely at the beginning. It also provides continuous system-level monitoring to prevent cheating, as well as a variety of management-level functionalities for creating and managing assessments including a prioritised and usable interface in order to enable the academics to quickly verify and check cases of possible cheating. The research includes a detailed discussion of the architecture requirements, components, and complete design to be the core of the system which captures, processes, and monitors students in a completely controlled e-test environment. In order to highlight the ease of use and lightweight nature of the system, a prototype was developed. Employing student face recognition as the most transparent multimodal (2D and 3D modes) biometrics, and novel security features through eye tracking, head movements, speech recognition, and multiple faces detection in order to enable a robust and flexible e-invigilation approach. Therefore, an experiment (Experiment 1) has been conducted utilising the developed prototype involving 51 participants. In this experiment, the focus has been mainly upon the usability of the system under normal use. The FRR of those 51 legitimate participants was 0 for every participant in the 2D mode; however, it was 0 for 45 of them and less than 0.096 for the rest 6 in the 3D mode. Consequently, for all the 51 participants of this experiment, on average, the FRR was 0 in 2D facial recognition mode, however, in 3D facial recognition mode, it was 0.048. Furthermore, in order to evaluate the robustness of the approach against targeted misuse 3 participants were tasked with a series of scenarios that map to typical misuse (Experiment 2). The FAR was 0.038 in the 2D mode and 0 in the 3D mode. The results of both experiments support the feasibility, security, and applicability of the suggested system. Finally, a series of scenario-based evaluations, involving the three separate stakeholders namely: Experts, Academics (qualitative-based surveys) and Students (a quantitative-based and qualitative-based survey) have also been utilised to provide a comprehensive evaluation into the effectiveness of the proposed approach. The vast majority of the interview/feedback outcomes can be considered as positive, constructive and valuable. The respondents agree with the idea of continuous and transparent authentication in e-assessments as it is vital for ensuring solid and convenient security beyond the point-of-entry. The outcomes have also supported the feasibility and practicality of the approach, as well as the efficiency of the system management via well-designed and smart interfaces.

The work reported in this dissertation investigates the potential for embedding Computerised Adaptive Testing (CAT) in students' and tutors' educational experiences. It seems that the tailored assessments that CAT can provide would be of real educational value in a range of contemporary Higher Education settings, however the resource requirements of some forms of CAT are prohibitive for making CAT assessments available to students across their studies. A form of CAT that is less resource intensive than other forms, the Flexilevel test, was selected for this programme of research to investigate its effectiveness in real educational contexts and explore possible applications for the approach. Ten empirical studies and a real data simulation study were conducted to test the effectiveness of the approach. It was found to show statistically significant correlations with other forms of assessment - in particular conventional Computer Based Testing (CBT) assessments, which is commonly used in contemporary educational settings. Another strand of work concerned the attitudes of stakeholders to the approach. Part of this work was carried out through the empirical studies, and further studies including interviews were also undertaken to explore the views of academic staff and students to the use of the Flexilevel test. Both groups were positive about the use of the Flexilevel test and this was taken to support the idea that academic staff and students would accept the use of the Flexilevel test in their educational experiences. In terms of both effectiveness and the acceptability of the approach to academic staff and students, the Flexilevel test was found to be a good candidate for embedding CAT in real educational contexts in Higher Education.

Currently, Information and Communication Technology (ICT) is available to almosteveryone. In the last few decades its impact has widely expanded in all fieldsincluding the educational one. There are important attitudinal factors affecting theusage of learning technology successfully. Information system researchers haveidentified the importance of personal factors, such as attitudes, beliefs, culture andbehaviours in technology acceptance (Davis, Bagozzi, & Warshaw, 1992). Previousresearch has analysed these factors in the acceptance of educational technology inhigher education (Liu, Liao, & Pratt, 2009), (Teo, 2009), (Terzis & Economides, 2011),(Cheung & Vogel, 2013). However, the factors playing a role in a mathematicalcontext have not been fully analysed. This research explores the role of several attitudinal factors in the acceptance ofeducational technology for the assessment process in mathematics in a highereducation setting. We examine the effects of formative on-line feedback on theadoption of educational technology by analysing both teachers' and students'opinions regarding enriched formative on-line assessment for mathematics. Ourresults suggest that UK students find on-line feedback is more enjoyable and usefulthan traditional feedback. Attitude and enjoyment are two important factorsinfluencing usage intentions. Results also show that two influential factors to facilitateusing the on-line testing environment are that feeling confident about computers andthe availability of information technology services. In Mexico, findings reveal thatstudents' attitude has the strongest influence on usage intentions which means thattheir feelings and opinions are important. They also think that on-line feedback ismore enjoyable than face-to-face feedback. Students who are proficient in on-linetesting are more likely to find it easy to use. Students also agree that when an on-lineenvironment is easy to use, it is also more likely that they consider it as useful andenjoyable. Taking into account these results can be the best way to design amathematics e-assessment activity for UK and Mexican students. UK instructors agreethat on-line assessments are useful tools to enrich instructional strategies. Teachersin Mexico have a similar opinion. Both results show that they really value providingon-line feedback.

Due to the availability of technology, there has been a shift from traditional assessment methods to e-assessment methods designed to support learning. With this development there is a need to address the suitability and effectiveness of the e-assessment interface. One development in the e-assessment interface has been the use of the multimodal metaphor. Unfortunately, the associated effectiveness of multimodality in terms of usability and its suitability in achieving assessment aims has not been fully addressed. Thus, there is a need to determine the impact of multimodality on the effectiveness of e-assessment and to reveal the benefits, primarily to the user. Moreover, those involved in the development and assessment should be aware of potential impacts and benefits. This thesis investigates the role and effectiveness of multimodal metaphors in e-assessment, specifically; the thesis assesses the effect of multimodal metaphors, alone or in combination, on usability in e-assessment. Usability includes efficiency, effectiveness and user satisfaction. The empirical research described in this study consisted of three experiments of 30 participants each to evaluate the effect of description text, avatars and images individually, avatars, description text and recorded speech in combination with images, and finally, the use of avatars with whole body gestures, earcons and auditory icons. The experimental stages were designed as a progression towards the main focus of the study, which was the effectiveness of full body gesture avatar, considered to be the latest development in multimodal metaphors. The experimentation also assessed the role that an avatar could play as a tutor in e-assessment interfaces. The results proved the positive effectiveness and applicability of metaphors to enhance e-assessment usability. This was achieved through a more effective interaction between the user and the assessment interface. A set of empirically derived guidelines for the design and use of these metaphors to enhance e-assessment is also used in order to generate more usable e-assessment interfaces.

Automatic marking of coursework has many advantages in terms of resource benefits and consistency. Diagrams are quite common in many domains including computer science but marking them automatically is a challenging task. There has been previous research to accomplish this, but results to date have been limited. Much of the meaning of a diagram is contained in the labels and in order to automatically mark the diagrams the labels need to be understood. However the choice of labels used by students in a diagram is largely unrestricted and diversity of labels can be a problem while matching. This thesis has measured the extent of the diagram label matching problem and proposed and evaluated a configurable extensible framework to solve it. A new hybrid syntax matching algorithm has also been proposed and evaluated. This hybrid approach is based on the multiple existing syntax algorithms. Experiments were conducted on a corpus of coursework which was large scale, realistic and representative of UK HEI students. The results show that the diagram label matching is a substantial problem and cannot be easily avoided for the e-assessment of diagrams. The results also show that the hybrid approach was better than the three existing syntax algorithms. The results also show that the framework has been effective but only to limited extent and needs to be further refined for the semantic stage. The framework proposed in this Thesis is configurable and extensible. It can be extended to include other algorithms and set of parameters. The framework uses configuration XML, dynamic loading of classes and two design patterns namely strategy design pattern and facade design pattern. A software prototype implementation of the framework has been developed in order to evaluate it. Finally this thesis also contributes the corpus of coursework and an open source software implementation of the proposed framework. Since the framework is configurable and extensible, its software implementation can be extended and used by the research community.

Academic assessment that is used to assist in producing knowledge having a lasting effect should be the concern of all stakeholders. Subjective assessment contributes to achieving this type of knowledge. In particular for multi-steps questions (MSQ) human assessors play important roles. This evaluation includes assessing solution strategies, working produced from employing those strategies, and the final answer. This assessment can reveal levels of conceptual understanding displayed by students. Subjective assessment gives options to students to express their understanding of the topic. However, when computer-aided assessment (CAA) systems are used, there is much less subjectivity. These systems subdivide MSQ into stub-steps, and students can provide one step’s answer without showing their workings. This technique is similar to assessing objective questions such as multiple-choice questions (MCQ) which do not, generally, examine conceptual understanding. The inability of CAA systems to assess solutions of MSQ is considered a complex problem. To solve a complex problem, accumulating knowledge about the problem, producing solutions and acquiring ability to apply those solutions are required. Design science research (DSR), which is the paradigm to investigate complex problems- in information systems research and design science - has been used in this research. Two hundred and fifty-eight student scripts containing working with solutions of MSQ were analysed using an inductive qualitative content analysis and the quantitative survey approach. The findings from the analyses of scripts were re-analysed iteratively to produce the knowledge that contributed to the understanding of the research problem and producing solutions. This knowledge includes types of solution strategies and student errors that can also be used in designing questions to be used in CAA systems for assessing objective questions. The methodologies that were used in producing this knowledge could also be used in similar disciplines to produce similar output. A significant contribution of this research is to analyse student workings for extracting solution strategies, and this helps understanding ways of solving MSQ and obtaining these strategies. This approach does not appear to have been used elsewhere. The questionnaire that was created for measuring the significance of types of strategies and student errors is an innovative instrument. It has not been used previously and can be adapted for similar studies. The information in the analyses of student workings also provides directions when solutions of MSQ are assessed by humans. Furthermore, all the above information as well as domain knowledge was used to develop DSR constructs and models artefacts. These models represent processes and data in the student workings, which was divided into sub-tasks with each task being represented by one model. The implementation of a model in a DSR prototype artefact accomplishes the assessment of that sub-task. One of these models was implemented, overcoming difficulty in designing user interfaces that can be used without disclosing the domain knowledge students are examined. These user interfaces show an innovative way of extracting conceptual understanding of topics. Furthermore, these models are highly generalisable to a very broad class of problems.

El artículo se enmarca en el campo de la eevaluación en la Universidad. Se pretende identificar cómo se gestiona el proceso de evaluación de los estudiantes y, a la vez, conocer la actitud de los docentes universitarios hacia la incorporación de las tecnologías en el proceso de evaluación de aprendizajes. La metodología descriptiva-correlacional empleada se basa en la aplicación de una encuesta electrónica a una muestra representativa por rama y categoría profesional de docentes en la Universidad de Salamanca. Los resultados obtenidos, tras la realización de los contrastes inferenciales oportunos de “actitud” por rama de conocimiento, categoría profesional y años de docencia, muestran diferencias estadísticamente significativas por rama de conocimiento.