Search

In this work, we study students' learning effectiveness through their use of a homegrown innovative educational technology, Web Programming Grading Assistant (WPGA), which facilitates grading and feedback delivery of paper-based assessments. We designed a lab study and a classroom study from a lower-division blended-instruction computer science course. We evaluated a partial credit assignment algorithm. We tracked and modeled students’ learning behaviors through their use of WPGA. Results showed that students demonstrated an effort and desire to review assessments regardless of if they were graded for academic performance or for attendance. Diligent students achieved higher exam scores on average and were found to review their exams and the correct questions frequently. Additionally, student cohorts exhibited similar initial reviewing patterns, but different in-depth reviewing and reflecting strategies. Ultimately, the work contributes to multidimensional learning analytics aggregation across the physical and cybersphere.

Different sources of data about students, ranging from static demographics to dynamic behavior logs, can be harnessed from a variety sources at Higher Education Institutions. Combining these assembles a rich digital footprint for students, which can enable institutions to better understand student behaviour and to better prepare for guiding students towards reaching their academic potential. This paper presents a new research methodology to automatically detect students ``at-risk'' of failing an assignment in computer programming modules (courses) and to simultaneously support adaptive feedback. By leveraging historical student data, we built predictive models using students' offline (static) information including student characteristics and demographics, and online (dynamic) resources using programming and behaviour activity logs. Predictions are generated weekly during semester. Overall, the predictive and personalised feedback helped to reduce the gap between the lower and higher-performing students. Furthermore, students praised the prediction and the personalised feedback, conveying strong recommendations for future students to use the system. We also found that students who followed their personalised guidance and recommendations performed better in examinations.

In this work, we present an intelligent classroom orchestration technology to capture semantic learning analytics from paper-based programming exams. We design and study an innovative visual analytics system, EduAnalysis, to support programming content semantics extraction and analysis. EduAnalysis indexes each programming exam question to a set of concepts based on the ontology. It utilizes automatic indexing algorithm and interactive visualization interfaces to establish the concepts and questions associations. We collect the indexing ground truths of the targeted set from teachers and experts from the crowd. We found that the system significantly extracted more and diverse concepts from exams and achieved high coherence within exam. We also discovered that indexing effectiveness was especially prevalent for complex content. Overall, the semantic enriching approach for programming problems reveals systematic learning analytics from the paper exams. Provide immediate technology support for classrooms that are instrumenting paper-based exams.Introduce novel method to automatically associate concepts and programming problems.Conduct controlled crowdsourcing experiment to harness educational ground truth.Significant results were found to enrich programming content semantics.Indexing effectiveness was especially prevalent for complex content.

Background/Context A large number of educational resources are now made available on the web to support both regular classroom learning and online learning. The abundance of available content has produced at least two problems: how to help students find the most appropriate resources and how to engage them in using and benefiting from these resources. Personalized and social learning have been suggested as potential ways to address these problems. Our work attempts to integrate these directions of research by combining the ideas of adaptive navigation support and open student modeling with the ideas of social comparison and social visualization. We call our approach Open Social Student Modeling (OSSM). Objective/Research Questions First, we review a sequence of our earlier projects focused on Open Social Student Modeling for one kind of learning content and formulate several key design principles that contribute to the success of OSSM. Second, we present our exploration of OSSM in a more challenging context of modeling student progress for two kinds of learning content in parallel. We aim to answer the following research questions: How do we design OSSM interfaces to support many kinds of learning content in parallel? Will current identified design principles (key features) confirm the power of the learning community through OSSM with multiple learning-resource collections? Will the OSSM visualization provide successful personalized guidance within a richer collection of educational resources? Research Design We designed four classroom studies to assess the value of different options for OSSM visualization of one and multiple kinds of learning content in the context of programming-language learning. We examined the comparative success of different design options to distill successful design patterns and other important lessons for the future developers of OSSM for personalized and social e-learning. Findings/Results The results confirmed the motivational impact of personalized social guidance provided by the OSSM system in the target context. The interface encouraged students to explore more topics and motivated them to work ahead of the course schedule. Both strong and weak students worked with the appropriate levels of questions for their readiness, which yielded consistent performance across different levels of complex problems. Additionally, providing more realistic content collection on the navigation-supported OSSM visualizations resulted in uniform performance for the group. Conclusions/Recommendation A sequence of studies of several OSSM interfaces confirmed that a combination of adaptive navigational support, open student modeling, and social visualization in the form of the OSSM interface can reinforce the navigational and motivational values of these approaches. In several contexts, the OSSM interface demonstrated its ability to offer effective guidance in helping students to locate the most relevant content at the right time while increasing student motivation to work with diverse learning content.

In this work, we propose a new methodology to profile individual students of computer science based on their programming design using a technique called embeddings. We investigate different approaches to analyze user source code submissions in the Python language. We compare the performances of different source code vectorization techniques to predict the correctness of a code submission. In addition, we propose a new mechanism to represent students based on their code submissions for a given set of laboratory tasks on a particular course. This way, we can make deeper recommendations for programming solutions and pathways to support student learning and progression in computer programming modules effectively at a Higher Education Institution. Recent work using Deep Learning tends to work better when more and more data is provided. However, in Learning Analytics, the number of students in a course is an unavoidable limit. Thus we cannot simply generate more data as is done in other domains such as FinTech or Social Network Analysis. Our findings indicate there is a need to learn and develop better mechanisms to extract and learn effective data features from students so as to analyze the students' progression and performance effectively.

There are numerous studies have reported the effectiveness of example-based programming learning. However, less is explored recommending code examples with advanced Machine Learning-based models. In this work, we propose a new method to explore the semantic analytics between programming codes and the annotations. We hypothesize that these semantics analytics will capture mass amount of valuable information that can be used as features to build predictive models. We evaluated the proposed semantic analytics extraction method with multiple deep learning algorithms. Results showed that deep learning models outperformed other models and baseline in most cases. Further analysis indicated that in special cases, the proposed method outperformed deep learning models by restricting false-positive classifications.

This Research Work in Progress paper presents a study on improving student learning performance in a virtual hands-on lab system in cybersecurity education. As the demand for cybersecurity-trained professionals rapidly increasing, virtual hands-on lab systems have been introduced into cybersecurity education as a tool to enhance students’ learning. To improve learning in a virtual hands-on lab system, instructors need to understand: what learning activities are associated with students’ learning performance in this system? What relationship exists between different learning activities? What instructors can do to improve learning outcomes in this system? However, few of these questions has been studied for using virtual hands-on lab in cybersecurity education. In this research, we present our recent findings by identifying that two learning activities are positively associated with students’ learning performance. Notably, the learning activity of reading lab materials (p < 0:01) plays a more significant role in hands-on learning than the learning activity of working on lab tasks (p < 0:05) in cybersecurity education.In addition, a student, who spends longer time on reading lab materials, may work longer time on lab tasks (p < 0:01).

This Research Full Paper presents an end-to-end framework to enhance personalized programming learning at a Higher Education Institution. A bundle of static and dynamic student data features (i.e. demographics, academic history and progression, clickstream or programming submissions) are collected to build predictive models. Artificial Intelligence and machine learning techniques are employed to automatically detect students-in-need and to adaptively provide them feedback in computer programming courses. At the macro-level, we hypothesize the bundled data features capture a meaningful set of predictors for each stage of the semester and each course. At the micro-level, being able to predict student's performances along with his/her progression provides automatic timely feedback. Three computer programming courses' historical student data was used to train the models. Predictions were then generated every week for the new cohorts of incoming students for those modules. A weekly report was sent to the lecturers with a rank of students and their associated probability of failing the next programming exam. The evaluation results show that, at the macro level, the predictions worked well with limited data. On one hand, student characteristics features are more relevant at the beginning of the course; on the other hand, the student's effort in the programming labs increasingly gain importance throughout the semester. In examining the micro-level assumption, we found that students who opted-in to receive personalized feedback indeed leveraged the generated predictions regarding to their predicted performance on their next exam. The personalized feedback includes various programming code recommendations, such as the programming code solutions from top-performers. We found that students corrected their programs after the programming suggestions from top-performers. Moreover, lower-performers improved their performance in the second laboratory examination with respect to students that were suggested programs but did not correct them and higher-performers respectively. In both scenarios, students who learned from the programs suggested and lower-performing students, showed a learning improvement over the two other groups. We should point out lower-performers had more room for improvement as they had lower grades in their first exam. Finally, we collected students' opinions about the feedback on the personalization intervention, to understand how it affects their behavior within the modules and if it encourages them to try new solutions or to revise material. Overall, it was very positive and most students would recommend this system to students attending the same course next year or would like to see this system included in other courses. Also, the last question captured their impression and how they would improve the platform. One student said: "It gave me confidence about the module. It gave me reassurance as to how I was getting on." while another said: "Good service, very helpful and effective way to manage your module". Higher-performers were getting an increasingly similar response were demanding a more personalized notification and some other additional learning resources.

It is a common practice for programmers to leave annotations during program development. Most of the annotated documentations are predominantly being used as the archive of the coding events for limited developers. We hypothesize that these annotations captured mass amount of valuable information which can be utilized to identify similar codes or to examine code quality. However, due to the annotating behaviors vary and the language composition can be complex, this work sets out to investigate a systematic method to examine the annotation semantics and their relations with codes. We designed a semantic parser to extract concepts from codes and the corresponding annotations. Additionally, text mining techniques are applied to summarize linguistic features from the annotations. We then build models to predict concepts in programming code annotations. Results show that the proposed semantic modeling method achieved a higher performance compared to a random guessed baseline.

This paper presents a methodology to analyze large amount of students’ learning states on two math courses offered by Global Fresh- man Academy program at Arizona State University. These two courses utilised ALEKS (Assessment and Learning in Knowledge Spaces) Arti- ficial Intelligence technology to facilitate massive open online learning. We explore social network analysis and unsupervised learning approaches (such as probabilistic graphical models) on these type of Intelligent Tu- toring Systems to examine the potential of the embedding representa- tions on students learning.

The increased volumes of online learning content have produced two problems: how to help students to find the most appropriate resources and how to engage them in using these resources. Personalized and social learning have been suggested as potential ways to address these problems. Our work presented in this paper combines the ideas of personalized and social learning in the context of educational hypermedia. We introduce Progressor, an innovative Web-based tool based on the concepts of social navigation and open student modeling that helps students to find the most relevant resources in a large collection of parameterized self-assessment questions on Java programming. We have evaluated Progressor in a semester-long classroom study, the results of which are presented in this paper. The study confirmed the impact of personalized social navigation support provided by the system in the target context. The interface encouraged students to explore more topics attempting more questions and achieving higher success rates in answering them. A deeper analysis of the social navigation support mechanism revealed that the top students successfully led the way to discovering most relevant resources by creating clear pathways for weaker students.

We survey across generations of work in authorized cheat sheets, and emphasize on extracting generic features for designing one. We construct generic models to address cognitive aspects of learning and focus on the content of knowledge components in the domain of programming language learning. We evaluate the models with real classroom study data and found that the amount of notes on a cheat sheet is associated with gains in students' declarative knowledge, but not procedural knowledge. We also discovered that working in creating new cheat sheets can help students to achieve better exam grades accumulatively beyond and above the effect of their performance in a prior exam. We discuss how these findings can guide the design of technology-based study guides.

We design and study ExamParser, an innovative intelligent semantic automatic indexing method, for orchestrating today's programming classes. ExamParser automatically processes paper-based exams by associating sets of concepts to the exam questions, which provide graders semantic grading guidelines and leave personalized semantic feedback. Results showed that the ExamPraser significantly extract more and diverse concepts from exams. It also achieves high coherence within exam, indicating the automatic concept extraction from exams is promising and could be a potential technological solution to provide personalized feedback for large-size programming classes.

In this paper, we report a study of an innovative mobile application to support grading paper-based programming exams. We call the app – Programming Grading Assistant (PGA). It scans pre-generated QR-codes of paper-based question-and-concepts associations and uses OCR to recognize handwritten answers. PGA provides interfaces for teachers to calibrate recognition results, as well as to adjust partial credit assignment according to conceptual incorrectness of the answers. We evaluate the mobile grading process and the quality of grading results based on the assessed semantic information. The results demonstrate that the mobile grading approach keeps persistent traces of students’ performance, including semantic feedback and ultimately enhances grading consistency.

We designed and studied an innovative semantic visual learning analytics for orchestrating today's programming classes. The visual analytics integrates sources of learning activities by their content semantics. It automatically processs paper-based exams by associating sets of concepts to the exam questions. Results indicated the automatic concept extraction from exams were promising and could be a potential technological solution to address a real world issue. We also discovered that indexing effectiveness was especially prevalent for complex content by covering more comprehensive semantics. Subjective evaluation revealed that the dynamic concept indexing provided teachers with immediate feedback on producing more balanced exams.

Rapid growth of the volume of interactive questions available to the students of modern E-Learning courses placed the problem of personalized guidance on the agenda of E-Learning researchers. Without proper guidance, students frequently select too simple or too complicated problems and ended either bored or discouraged. This paper explores a specific personalized guidance technology known as adaptive navigation support. We developed JavaGuide, a system, which guides students to appropriate questions in a Java programming course, and investigated the effect of personalized guidance a three-semester long classroom study. The results of this study confirm the educational and motivational effects of adaptive navigation support.

One of the original goals of intelligent educational systems is to guide every student to the most appropriate educational content. Exploring both knowledge-based and social guidance approaches in past work, we learned that each of these approaches has weak sides. In this paper we follow the idea of combining social guidance with more traditional knowledge-based guidance to support more optimal content navigation. We proposed a greedy sequencing approach that maximizes student’s level of knowledge and tested it in a classroom. Results indicated that this approach positively impacts students’ navigation.

Over the last 10 years two major research directions explored the benefits of visualizing student learning progress. One stream of research on learning performance visualization attempts to build a visual presentation of students' learning progress, targeting the needs of instructors and academic advisors. The other stream of research on Open Student Modeling (OSM) attempts to visualize the state of individual student's knowledge and present the visualization directly to the student. The results of the studies in that area show that, presenting students with basic representation of their knowledge will result in facilitating their metacognitive activities and promoting self-reflection and awareness. This paper tries to study the impact of a more sophisticated form of performance visualization on students. We believe that our visualization tool can positively influence students by granting them the opportunity to get a view of their performance in the content of the class progress. Moreover, we tried to boost their motivation by building a positive sense of competition using a representation of average class performance. In this paper we present study comparing two groups of students, one using the visualization and another without visualization. The results of the study shows that: 1) the students are likely to use the social visualization tool during the whole semester to monitor their progress in comparison with their peers; 2) the visualization tool encourages students to use the learning materials in a more continuous manner during the whole semester and 3) students will achieve a higher success rate in answering self-assessment quizzes.

In this paper, we present an asynchronous video discussion tool called 'Vialogues' and empirical evidence of system usage for supporting modern social learning environment. Statistical analysis showed that Vialogues not only enriches online learning experiences but also successfully retains users and creates social learning opportunities. Evidence of learning taking place in a new technology, called Vialogues, is reported.

Teacher and students can use WWW as a limitless source of learning material for nearly any subject. Yet, such abundance of content comes with the problem of finding the right piece at the right time. Conventional adaptive educational systems cannot support personalized access to open-corpus learning material as they rely on manually constructed content models. This paper presents an approach to this problem that does not require intervention from a human expert. The approach has been implemented in an adaptive system that recommends students supplementary reading material and adaptively annotates it. The results of the evaluation experiment have demonstrated several significant effects of using the system on students' learning.

In this paper, we present a novel approach to integrate social adaptive navigation support for self-assessment questions with an open student model using QuizMap, a TreeMap-based interface. By exposing student model in contrast to student peers and the whole class, QuizMap attempts to provide social guidance and increase student performance. The paper explains the nature of the QuizMap approach and its implementation in the context of self-assessment questions for Java programming. It also presents the design of a semester-long classroom study that we ran to evaluate QuizMap and reports the evaluation results.

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 21st ACM conference on Hypertext and hypermedia, http://dx.doi.org/10.1145/1810617.1810657, Adaptive link annotation is one of the most popular adaptive educational hypermedia techniques. It has been widely studied and demonstrated its ability to help students to acquire knowledge faster, improve learning outcomes, reduce navigation overhead, increase motivation, and encourage the beneficial non-sequential navigation. However, almost all studies of adaptive link annotation have been performed in the context of dedicated adaptive educational hypermedia systems. The role of this technique in the context of widely popular learning portals has not yet been demonstrated. In this paper, we attempt to fill this gap by investigating the value of adaptive navigation support embedded into the learning portal. We compare the effect of portal-based adaptive navigation support to both the effect of the adaptive navigation support in adaptive educational hypermedia systems and to non-adaptive learning portals., This work is supported by National Science Foundation under Grant IIS-0447083, Spanish Ministry of Science and Education (TIN2007-64718) and the Comunidad Autónoma de Madrid (S2009/TIC-1650)

Problem-tracing questions are popular among teachers of various programming languages. In an assessment mode these questions allows to evaluate student knowledge of language semantics. In a self-assessment mode, they provide an excellent learning tool. A 2004 ITiCSE working group report [4] stressed the importance of this type of questions to build foundation of higher-level knowledge. Yet the use of problem-tracing questions is still limited due to a large authoring overhead. To resolve this bottleneck, we explored the idea of parameterized question generation [2]. We developed QuizPACK [1], a system which can generate parameterized problem-tracing questions for C programming language. We also developed QuizGuide [1], a personalized guidance system for QuizPACK, which models student knowledge and guides students individually to most appropriate questions to try. The results of our studies demonstrated that QuizPACK strongly benefits student knowledge and that QuizGuide personalized guidance technology increased student ability to answer questions correctly and encouraged them to use the system more extensively (which, in turn, positively impacted their knowledge) [1]. However, parameterized questions in area of C programming were not as diverse from the complexity point of view as parameterized questions explored in other areas such as physics [2]. As a result, it was left unclear whether personalized guidance technology can successfully guide students to a broader range of questions from relatively simple to very difficult.The work reported in this poster expands our work on parameterized questions to a more sophisticated domain of object-oriented Java programming, which allowed us to introduce questions of much broader. Capitalizing on our experience with QuizPACK, we developed QuizJET (Java Evaluation Toolkit), which supports authoring, delivery, and evaluation of parameterized questions for Java [3]. We also implemented JavaGuide system (Figure 1), which provides personalized guidance for QuizJET questions. We assessed the impact of adaptive navigation support to student work with questions of different complexity as well as the impact of this technology on weaker and stronger students. The results of two classroom studies indicate that personalized guidance encouraged students to use parameterized questions more extensively and also helped them to access right questions at the right time. Students were 2.5 times more likely to answer a quiz correctly with personalized guidance than without such it. In addition, we found that personalized guidance especially benefited weak students to achieve scores comparable with the scores of strong students on each complexity level of questions.

This paper explores the impact of adaptive navigation support on student work with parameterized questions in the domain of object-oriented programming. In the past, we developed QuizJET system, which is able to generate and assess parameterized Java programming questions. More recently, we developed JavaGuide system, which enhances QuizJET questions with adaptive navigation support. This system introduces QuizJET and JavaGuide and reports the results of classroom studies, which explored the impact of these systems and assessed an added value of adaptive navigation support. The results of the studies indicate that adaptive navigation support encourages students use parameterized questions more extensively. Students are also 2.5 times more likely to answer parameterized questions correctly with adaptive navigation support than without such support. In addition, we found that adaptive navigation support especially benefit weaker students helping to close the gap between strong and weak students.

Each educational resource management website relies on an authoring tool to provide example content. It takes time and experience for authors to create valuable content. Providing support during authoring can affect the quality and quantity of the examples. In this paper we propose a mashup solution to automatically link community wisdom to authors and ease various difficulties in authoring.

This paper analyzes problems encountered by our team while creating an educational digital library of program examples. We present approaches to resolving these problems, and evaluations of the suggested approaches.