Search

IEA's International Computer and Information Literacy Study (ICILS) 2018 investigated how well students are prepared for study, work, and life in a digital world. ICILS 2018 measured international differences in students' computer and information literacy (CIL): their ability to use computers to investigate, create, participate, and communicate at home, at school, in the workplace, and in the community. Participating countries had an additional option for their students to complete an assessment of computational thinking (CT): their ability to recognize aspects of real-world problems appropriate for computational formulation, and to evaluate and develop algorithmic solutions to those problems, so that the solutions could be operationalized with a computer. This technical report follows the publication of several international and regional reports that presented the results of ICILS 2018. It provides a comprehensive account of the conceptual, methodological, and analytical implementation of the study. It includes detailed information on the development of the data-collection instruments used, including their translation and translation verification, on sampling design and implementation, sampling weights and participation rates, survey operation procedures, quality control of data collection, data management and creation of the international database, scaling procedures, and analysis of ICILS 2018 data. The technical report enables researchers to evaluate published reports and articles based on data from this study and, used in conjunction with the ICILS 2018 User Guide for the International Database, will provide guidance for their own analyses. [Contributors to this report include: John Ainley, Sebastian Meyer, Ralph Carstens, Ekaterina Mikheeva, Alex Daraganov, Lauren Musu, Lauren Musu, Louise Ockwell, David Ebbs, Wolfram Schulz, Julian Fraillon, Sabine Tieck, and Tim Friedman. For "IEA International Computer and Information Literacy Study 2018. User Guide for the International Database," see ED610700.]

The International Association for the Evaluation of Educational Achievement (IEA) has been researching the impact of information and communications technologies (ICT) on educational processes, and factors supporting or impeding the pedagogical use of ICT, since the late-1980s. More recently it has turned its attention to investigating ICT-related educational outcomes. IEA's International Computer and Information Literacy Study (ICILS) was developed in response to the increasing use of ICT in modern society and the need for people to have the capabilities necessary to participate effectively in a digital world. The first cycle of ICILS in 2013 (ICILS 2013) assessed students' computer and information literacy (CIL) which brings together technical competence with computer applications and the capacity to manage information. This places an emphasis on the use of computers as information seeking, management, and communication tools which are key to participation in the digital age. This second cycle of ICILS in 2018 (ICILS 2018) continued to investigate students' CIL but also investigated students' computational thinking (CT). This dimension involves conceptualizing problems (through algorithmic or systems thinking) and operationalizing solutions (creating, implementing, and evaluating computer-based responses to problems). Authentic computer-based assessments administered to students in their eighth year of schooling generated data that provided measures of these two domains of digital competence (CIL and CT). The inclusion of CT as an option in ICILS 2018 reflects recent interest by educators, researchers, and policymakers in the value of CT in schooling. ICILS 2018 studied how these components of digital competence related to each other and to the school and out-of-school contexts that support learning with and about computer technology. This report presents the outcomes at the international level of analyses of data collected in the ICILS main survey in 2018. The report aims to provide an international perspective on the ICILS data relating to CIL collected across countries. Twelve countries participated in ICILS 2018: Chile, Denmark, Finland, France, Germany, Italy, Kazakhstan, the Republic of Korea (hereafter referred to as Korea, for ease of reading), Luxembourg, Portugal, the United States, and Uruguay. The city of Moscow (Russian Federation) and the state of North Rhine-Westphalia (Germany) participated as benchmarking participants. Eight countries completed the CT assessment: Denmark, Finland, France, Germany, Korea, Luxembourg, Portugal, and the United States. The CT option was also completed by the state of North Rhine-Westphalia (Germany) as a benchmarking participant. This report responds to the ICILS research questions and provides observations and interpretations that may stimulate further investigation within and across countries. [This report was published by SpringerOpen. For "ICILS 2013 Technical Report," see ED610534. For the accompanying report, "IEA International Computer and Information Literacy Study 2018 Assessment Framework," see ED610537.]

The International Computer and Information Literacy Study (ICILS) studied the extent to which young people have developed computer and information literacy (CIL) to support their capacity to participate in the digital age. Many countries recognize the importance that education in information and communication technologies (ICT) has for enabling citizens to develop the competencies needed to access information and participate in transactions using ICT (European Commission, 2006). ICILS systematically investigated differences in CIL outcomes across the participating countries and looked at how these countries provide CIL-related education. The study also explored differences within and across 21 countries with respect to the relationship between the outcomes of CIL education and student characteristics and school contexts. ICILS based its investigation on four research questions concerned with: (1) Variations in CIL between and within countries; (2) Aspects of schools, education systems, and teaching associated with student achievement in CIL; (3) The extent to which students' access to, familiarity with, and self-reported proficiency in using computers is associated with student achievement in CIL; and (4) The aspects of students' personal and social backgrounds that are associated with CIL. Following an overview, this report begins with three chapters that cover the instruments that were used in the study. Appreciation of the material in these chapters provides an essential foundation for interpreting the results of the study. Chapter 4 details the development of the questionnaires used in ICILS for gathering data from students, teachers, principals, and school ICT coordinators. The chapter also provides an outline of the development of the national contexts survey, completed by the national research coordinators (NRCs). Chapters 5 through 9 focus on the implementation of the survey in 2013. Chapters 10 through 13 are concerned with data management and analysis. [Contributors to this report include: John Ainley, Ralph Carstens, Diego Cortes, David Ebbs, Julian Fraillon, Tim Friedman, Eveline Gebhardt, Michael Jung, Paulína Koršnáková, Sabine Meinck, and Wolfram Schulz.]

The International Civic and Citizenship Education Study (ICCS) is the largest international study of civic and citizenship education ever conducted. Over 140,000 Grade 8 students, 62,000 teachers, and 5,300 school principals from 38 countries participated in this study. Among these were five from Asia, 26 from Europe, six from Latin America, and one from Australasia. The ICCS assessment addressed students' civic knowledge and understanding, perceptions and attitudes, and engagement and behavior. It also collected information on various aspects of students' home backgrounds. Each chapter, written by one or more members of the country's national research center, begins with a description of the country's general demographic features as well as its political characteristics. Authors then detail the background and structure of the education system, the general approach to civic and citizenship education, how it is included in the national curriculum, which school activities relate to it, and whether this learning area is subject to current reforms and debates. Authors also outline the requirements for educating teachers for teaching in general and for teaching civic and citizenship education in particular. With regard to the latter, consideration is given to the extent to which teachers receive teacher education (both preservice and inservice) that prepares them to teach issues related to this learning area. Each chapter concludes with a description of the national approaches to assessment and quality assurance of educational outcomes on the one hand and of learning topics related to civic and citizenship education on the other. A list of organizations and individuals involved in ICCS is appended. (Individual chapters contain tables, figures, footnotes, and references.) [Additional funding for this document was provided by the Inter-American Development Bank.]

Based on student survey data from five East Asian countries, the paper contains an analysis of attitudes towards the use of personal connections in politics and towards personal morality among politicians. The first part of the analysis describes the extent and variations of these attitudes, which are viewed as of particular relevance within the East Asian context. The second part of the analysis investigates the relationship of these attitudes with student background and school-related variables such as civic knowledge as well as uncovering the extent to which these attitudes are related to indicators of future civic engagement among students. (Contains 1 figure and 7 tables.)

This supplement contains documentation on the explicit and implicit stratification variables included in the International Civic and Citizenship Education Study (ICCS) 2009 data files. The explicit strata are smaller sampling frames, created from the national sampling frames, from which national samples of schools were drawn. The implicit strata are nested within the explicit strata, and were used for sorting the sampling frame prior to the systematic sampling of the schools. For details of the ICCS 2009 sample design and implementation, please refer to the "ICCS 2009 Technical Report" (ED544622). [For the "ICCS 2009 User Guide for the International Database," see ED544615. For Supplement 2 (National Adaptations of International Questionnaires), see ED544616. For Supplement 3 (Variables Derived from the Survey Data), see ED544617. For Supplement 5 (ICCS 2009 Released Items), see ED544619.]

This report is structured so as to provide technical detail about each aspect of International Civic and Citizenship Education Study (ICCS). The overview is followed by a series of chapters that provide detail about different aspects of ICCS. Chapters, 2, 3, 4, and 5 are concerned with the instruments. Chapter 2 provides information about the development and properties of the ICCS cognitive test. Chapter 3 details the development and properties of the ICCS international questionnaires. These include the international student questionnaire, the teacher questionnaire, and the school questionnaire, as well as the national contexts survey. Chapter 4 is concerned with the development of the ICCS regional instruments: the European test and questionnaire, the Latin American test and questionnaire, and the Asian questionnaire. Chapter 5 describes the procedures used to translate and adapt ICCS instruments. Chapters 6 and 7 are concerned with aspects of sampling. Chapter 6 details the sampling design and implementation and Chapter 7 documents the weighting procedures that were used to ensure the results from ICCS represented the defined populations in each country. Chapters 8, 9, and 10 focus on the survey implementation. Chapter 8 details the field operation procedures and the process of preparation of data files. Chapter 9 documents the quality control protocols and procedures that were used in the ICCS data collection. Chapter 10 provides an account of data management in ICCS and the creation of the ICCS database. Chapters 11, 12, and 13 describe the psychometric and statistical analyses used in ICCS. Chapter 11 reports on the scaling procedures for the ICCS cognitive test and how responses to the test items were used to construct scores on the ICCS civic knowledge scale. Chapter 12 describes the methods used to form scales from the ICCS questionnaire items, while Chapter 13 details how the ICCS results were reported and gives an account of the conventions adopted for the construction of tables. The following are appended: (1) Organizations and individuals involved in ICCS; (2) Characteristics of national samples; (3) Descriptions for cognitive test items and allocations to proficiency levels; and (4) Tables with coding information. (Contains 132 tables and 44 figures. Individual chapters contain references and footnotes. The appendices contain additional tables.) [The following people contributed to writing this report: John Ainley, Falk Brese, Julian Fraillon, Tim Friedman, Michael Jung, David Kerr, Bruno Losito, Barbara Malak, Wolfram Schulz, Caroline Vandenplas, Alana Yu, and Olaf Zuehlke. See ED544615, for the ICCS 2009 User Guide for the International Database.]

This supplement contains documentation on all the derived variables contained in the International Civic and Citizenship Education Study (ICCS) 2009 data files that are based on survey variables. These variables were used to report data in the ICCS 2009 international reports, and are made available as part of the ICCS 2009 International Database to be used in secondary analyses. This supplement has seven sections corresponding to each survey instrument from which reporting variables are derived: (1) School Questionnaire; (2) Teacher Questionnaire; (3) Student Achievement Booklets; (4) Student Questionnaire; (5) European Module Questionnaire; (6) Latin American Module Questionnaire; and (7) Asian Module Questionnaire. Each section lists first the indices and then the scales derived from survey variables in the order of the variables that were used to derive the variable as they appear in the instruments. The following information is provided for each derived variable: (1) Variable Name: The name of the derived variable; (2) Description: A description of the variable content; (3) Procedure: A procedural description of how the derived variable was computed; and (4) Source: Source variables used to derive scale or index. [For the "ICCS 2009 Technical Report," see ED544622.]

This document outlines the framework and assessment design for the International Civic and Citizenship Education Study (ICCS) sponsored by the International Association for the Evaluation of Educational Achievement (IEA). Over the past 50 years, IEA has conducted comparative research studies focusing on educational policies, practices, and outcomes in more than 80 countries around the world. The purpose of ICCS is to investigate the ways in which young people in lower secondary schools are prepared to undertake their roles as citizens in a range of countries. The study will report on student achievement on a test of conceptual understandings and competencies in civic and citizenship education and also will collect and analyze data about student dispositions and attitudes relating to civic and citizenship education. Teacher and school questionnaires will gather information about teaching and class-management practices, school governance and climate, and other matters. A national context survey will collect information about civic and citizenship education and its contexts (aspects related to political, cultural, and educational contexts) in each participating country. The study framework also offers "regional modules" that will allow groups of participating countries from the same region to address region-specific issues in civic and citizenship education. The three regional modules established for ICCS relate to Europe, Latin America, and Asia. Appendices include: (1) Institutions and Staff; and (2) Sample Test Items. (Contains 8 footnotes, 3 figures, and 5 tables.) [In co-operation with the IEA Secretariat, the IEA DPC (data processing and research center), and the NRCs (national research coordinators) the following three organizations are responsible for the study's organization: Australian Council for Educational Research (ACER), as lead institution; the National Foundation for Educational Research (NFER) in the United Kingdom; and Laboratorio di Pedagogia sperimentale (LPS) at the Roma Tre University in Italy.]

The purpose of the "International Civic and Citizenship Education Study" (ICCS) is to investigate the ways in which young people are prepared to undertake their roles as citizens in a range of countries. The study will report on student achievement on a test of conceptual understandings and competencies in civic and citizenship education. It will also collect and analyse data about student dispositions and attitudes relating to civic and citizenship education. The study builds on the previous IEA study of civic education (CIVED) undertaken in 1999. A website with information about ICCS can be found at http://iccs.acer.edu.au/. It is recognised that there is substantial diversity in the field of civic and citizenship education within and across countries. Consequently, maximising the involvement of researchers from participating countries in this international comparative study is deemed to be of particular importance for the success of this study. Input from national research centres will be sought throughout the study and the consortium will develop strategies to encourage country contributions to instrument development as well as to the dissemination of results. This document gives an overview of the study and summarise the different aspects of its implementation. It also outlines the structure of the assessment framework and the process of the development and implementation of instruments as well as the benefits for participating countries. (Contains 4 footnotes, 1 figure and 3 tables.) [Paper prepared for the 51st annual conference of the Comparative and International Education Society in Baltimore, 25 February-1 March 2007.]

International studies like PISA use context student or school questionnaires to collect data on student family background, attitudes and learning context. Questionnaire constructs are typically measured using dichotomous or Likert-type items. Scaling of questionnaire items in order to obtain measures of family background, student attitudes or perceptions of learning context requires a thorough cross-country validation of the underlying constructs. Two methodological approaches to construct validation are applied within the context of the OECD PISA study: (1) Confirmatory Factor Analysis (CFA) based on mean and covariance structures can be used to test the measurement equivalence and help to detect possible socio-cultural differences on the constructs; and (2) Item Response Theory (IRT) has been used for scaling items using a logistic function and obtaining measures of the latent construct. This paper describes ways of implementing tests of parameter invariance in CFA and IRT, examines results using these two different approaches to validate questionnaire constructs based on data from the OECD PISA study and discusses their implications for cross-national or cross-cultural research. Appended are: (1) Multiple-group model results; and (2) Country effects on item parameters. (Contains 18 tables and 8 footnotes.)

One of the consistent findings of educational research studies is the effect of the students' family socio-economic background on their learning achievement. Consequently, international comparative studies emphasis the role of socio-economic background for determining learning outcomes. In particular, PISA results have been used to describe how different structures of the educational system can mediate the impact of socio-economic family background on performance with comprehensive system generally providing more equity in educational opportunities. This paper addresses the issue of measuring socio-economic background in the context of the OECD PISA study. It describes the computation of a composite index of "Economic, Social and Cultural Status" derived from occupational status of parents, educational level of parents and home possessions for the first two PISA cycles. It also shows results from the first two PISA surveys regarding the relationship between socio-economic background and student performance, both using single-level and multi-level analysis. Furthermore, it reviews the consistency of student and parent reports on socio-economic based on field trial data from the third PISA cycle collected in 2005. The analysis presented in this paper show that within PISA countries the effects of socio-economic background measures on student performance across the first two PISA cycles remain mostly unchanged. But there is some variation which might partly be due to problem with measurement and researchers as well as policy-makers are strongly advised not to interpret minor changes in the relationship between SES and performance (for example as possible outcomes of recent policy changes). It needs to be recognized that there is a certain amount of measurement error associated with student reports on family background which needs to be taken into account when interpreting findings from PISA. (Contains 13 tables.)

Differences in student knowledge about democracy, institutions, and citizenship and students skills in interpreting political communication were studied through multilevel regression analysis of results from the second International Education Association (IEA) Study. This study provides data on 14-year-old students from 28 countries in Europe, North America, South America, Asia, and the Pacific. Item response theory estimates for Civic Knowledge were used as independent variables in the multilevel regression models presented in this paper. To facilitate the interpretation of the results, countries were grouped tentatively into nine groups that should reflect common characteristics related to geography, history, culture, or educational systems. Looking at the whole international database and estimating the proportions of the overall variance between students and schools, and between countries, it can be shown that approximately 66% of the total variance exists between students, 25% between schools, and only 9% between countries. These results demonstrate that most of the variance is found on the student level. These analyses confirm some earlier findings from single-level analyses of Civic Knowledge. Gender does have an effect in a considerable number of countries when controlling for other factors. Expected Education and Home Literacy are consistent predictors. Classroom Climate appears to have considerable effects in most countries, and reported participation in school council or parliaments does have positive effects on Civic Knowledge in a number of countries. (Contains 3 tables and 14 references.) (SLD)