Search

To provide practice and assessment of computational thinking, we need specific problems students can solve. There are many such problems, but they are hard to find. Learning environments and assessments often use only specific types of problems and thus do not cover computational thinking in its whole scope. We provide an extensive catalog of well-structured computational thinking problem sets together with a systematic encoding of their features. Based on this encoding, we propose a four-level taxonomy that provides an organization of a wide variety of problems. The catalog, taxonomy, and problem features are useful for content authors, designers of learning environments, and researchers studying computational thinking.

Complexity and difficulty are two closely related but distinct concepts. These concepts are important in the development of intelligent learning systems, e.g., for sequencing items, student modeling, or content management. We show how to use complexity and difficulty measures in the development of learning systems and provide guidance on how to think, reason, and communicate about these notions. To do so, we propose a pragmatic distinction between difficulty and complexity measures. At the same time, we acknowledge the limitations of any simple distinction and discuss several potentially confounding issues: context, biases, and scaffoldings. We also provide an overview of specific measures and their applications in several educational domains and a detailed analysis of measures for problems in introductory programming.

Block-based programming is a popular approach to teaching introductory programming. Block-based programming often works in the context of microworlds, where students solve specific puzzles. It is u...

Adaptive learning systems collect data on student performance and use them to personalize system behavior. Most current personalization techniques focus on the correctness of answers. Although the correctness of answers is the most straightforward source of information about student state, research suggests that additional data are also useful, e.g., response times, hints usage, or specific values of incorrect answers. However, these sources of data are not easy to utilize and are often used in an ad hoc fashion. We propose to use answer classification as an interface between raw data about student performance and algorithms for adaptive behavior. Specifically, we propose a classification of student answers into six categories: three classes of correct answers and three classes of incorrect answers. The proposed classification is broadly applicable and makes the use of additional interaction data much more feasible. We support the proposal by analysis of extensive data from adaptive learning systems.

Student models are typically evaluated through predicting the correctness of the next answer. This approach is insufficient in the problem-solving context, especially for student models that use performance data beyond binary correctness. We propose more comprehensive methods for validating student models and illustrate them in the context of introductory programming. We demonstrate the insufficiency of the next answer correctness prediction task, as it is neither able to reveal low validity of student models that use just binary correctness, nor does it show increased validity of models that use other performance data. The key message is that the prevalent usage of the next answer correctness for validating student models and binary correctness as the only input to the models is not always warranted and limits the progress in learning analytics.

In introductory programming and other problem-solving activities, students can create many variants of a solution. For teachers, content developers, or applications in student modeling, it is useful to find structure in the set of all submitted solutions. We propose a generic, modular algorithm for the construction of interpretable clustering of students’ solutions in problem-solving activities. We describe a specific realization of the algorithm for introductory Python programming and report results of the evaluation on a diverse set of problems.

Teaching assistants (TAs) are heavily used in computer science courses as a way to handle high enrollment and still being able to offer students individual tutoring and detailed assessments. TAs are themselves students who take on this additional role in parallel with their own studies at the same institution. Previous research has shown that being a TA can be challenging but has mainly been conducted on TAs from a single institution or within a single course. This paper offers a multi-institutional, multi-national perspective of challenges that TAs in computer science face. This has been done by conducting a thematic analysis of 180 reflective essays written by TAs from three institutions across Europe. The thematic analysis resulted in five main challenges: becoming a professional TA, student focused challenges, assessment, defining and using best practice, and threats to best practice. In addition, these challenges were all identified within the essays from all three institutions, indicating that the identified challenges are not particularly context-dependent. Based on these findings, we also outline implications for educators involved in TA training and coordinators of computer science courses with TAs.

Additive Factors Model is a widely used student model, which is primarily used for refining knowledge component models (Q-matrices). We explore the robustness and generalizability of the model. We explicitly formulate simplifying assumptions that the model makes and we discuss methods for visualizing learning curves based on the model. We also report on an application of the model to data from a learning system for introductory programming; these experiments illustrate possibly misleading interpretation of model results due to differences in item difficulty. Overall, our results show that greater care has to be taken in the application of the model and in the interpretation of results obtained with the model.

The evaluation of student models involves many methodological decisions, e.g., the choice of performance metric, data filtering, and cross-validation setting. Such issues may seem like technical details, and they do not get much attention in published research. Nevertheless, their impact on experiments can be significant. We report experiments with six models for predicting problem-solving times in four introductory programming exercises. Our focus is not on these models per se but rather on the methodological choices necessary for performing these experiments. The results show, particularly, the importance of the choice of performance metric, including details of its computation and presentation.

Large scale learning systems for introductory programming need to be able to automatically assess the quality of students' performance on programming tasks. This assessment is done using a performance measure, which provides feedback to students and teachers, and an input to the domain, student and tutor models. The choice of a good performance measure is nontrivial, since the performance of students can be measured in many ways, and the design of measure can interact with the adaptive features of a learning system or imperfections in the used domain model. We discuss the important design decisions and illustrate the process of an iterative design and evaluation of a performance measure in a case study.

Quantification of the difficulty of problem solving tasks has many applications in the development of adaptive learning systems, e.g., task sequencing, student modeling, and insight for content authors. There are, however, many potential conceptualizations and measures of problem difficulty and the computation of difficulty measures is influenced by biases in data collection. In this work, we explore difficulty measures for introductory programming tasks. The results provide insight into non-trivial behavior of even simple difficulty measures.

Hour of Code activities became a de facto standard for the first encounter with programming, reaching millions of children every year. These activities are typically not personalized and offer the same sequence of tasks to everybody, which leads to too slow pace for some students, while too fast for others. We aim to improve upon the current state of the art in teaching introductory programming by providing insight into how adaptive learning techniques can make the Hour of Code activities more efficient and engaging.

A personalized learning system needs a large pool of items for learners to solve. When working with a large pool of items, it is useful to measure the similarity of items. We outline a general approach to measuring the similarity of items and discuss specific measures for items used in introductory programming. Evaluation of quality of similarity measures is difficult. To this end, we propose an evaluation approach utilizing three levels of abstraction. We illustrate our approach to measuring similarity and provide evaluation using items from three diverse programming environments.

Block-based environments are today commonly used for introductory programming activities like those that are part of the Hour of Code campaign, which reaches millions of students. These activities typically consist of a static series of problems. Our aim is to make this type of activities more efficient by incorporating adaptive behavior. In this work, we discuss steps towards this goal, specifically a proposal and implementation of a programming game that supports both elementary problems and interesting programming challenges and thus provides an environment for meaningful adaptation. We also discuss methods of adaptivity and the issue of evaluating student performance while solving a problem.