Search

Calls for increased monitoring and evaluation of education policies and practices have not, so far, included widespread and consistent assessments of the inclusiveness of education settings. Measuring inclusion in education has proven to be a challenging exercise, due not only to the complexity and different uses of the concept, but also to its holistic nature. Indeed, measuring inclusion implies analysing a variety of policy areas within education systems, while also considering the different roles of the system, the school and the classroom. This paper discusses the application of the input-process-outcome model to the measurement of inclusion in education, and key indicators that can be adopted by education systems and schools to this end. It makes considerations relevant to policy makers when designing indicators to measure inclusion, such as the extent of their application, the constraints related to data disaggregation and the relevance of intersectional approaches to inclusion.

This report aims to showcase the value of implementing a Bayesian framework to analyse and report results from international large-scale surveys and provide guidance to users who want to analyse the data using this approach. The motivation for this report stems from the recognition that Bayesian statistical inference is fast becoming a popular methodological framework for the analysis of educational data generally, and large-scale surveys more specifically. The report argues that Bayesian statistical methods can provide a more nuanced analysis of results of policy relevance compared to standard frequentist approaches commonly found in large-scale survey reports. The data utilised for this report comes from the OECD Teaching and Learning International Survey (TALIS). The report provides steps in implementing a Bayesian analysis and proposes a workflow that can be applied not only to TALIS but to large-scale surveys in general. The report closes with discussion of other Bayesian approaches to international large-scale survey data, in particularly for predictive modelling.

The feedback between vocational education and training (VET) and the labour market can provide important input for the review and renewal of qualifications. A feedback loop that is based on learning outcomes helps provide deeper insights into what is required on the labour market, what is offered in training provisions and assessed at the end of a learning programme. The aim of this study is to contribute to strengthening the quality and relevance of qualifications and completing the feedback loop between education and the labour market. It examines methods of collecting data on the match/mismatch between qualifications and labour market requirements, including analysis of how achieved learning outcomes are applied and perceived in the labour market (for example methods of collecting the experience of employers with holders of these qualifications). This report addresses the following two questions: (1) which data already exist in the countries, providing insight into the relevance of qualifications to employees, employers and other labour market stakeholders?; and (2) how can survey methodology be designed to systematically capture the experiences and appreciations of employers as regards the content and profile of qualifications? To what extent, based on limited testing, can scalability of the methodology be achieved?

A number of national and international organizations have compiled lists of grand challenges to unify the efforts of scholars and practitioners in a field. Unified efforts increase the possibility of creating meaningful and lasting progress. In this paper we share ten grand challenges that were identified through an examination of the assessment literature and a national survey. Each of the grand challenges are described. The four challenges which were identified as being of greatest concern to the assessment community are currently being addressed by the Grand Challenges in Assessment Project for possible collective solutions.

This paper describes a method for pooling grouped, ordered-categorical data across multiple waves to improve small-sample heteroskedastic ordered probit (HETOP) estimates of latent distributional parameters. We illustrate the method with aggregate proficiency data reporting the number of students in schools or districts scoring in each of a small number of ordered "proficiency" levels. HETOP models can be used to estimate means and standard deviations of the underlying (latent) test score distributions, but may yield biased or very imprecise estimates when group sample sizes are small. A simulation study demonstrates that pooled HETOP models can reduce the bias and sampling error of standard deviation estimates when group sample sizes are small. An analysis of real test score data suggests the pooled models are likely to improve estimates in applied contexts.

Responsible Operations is intended to help chart library community engagement with data science, machine learning, and artificial intelligence (AI) and was developed in partnership with an advisory group and a landscape group comprised of more than 70 librarians and professionals from universities, libraries, museums, archives, and other organizations. This research agenda presents an interdependent set of technical, organizational, and social challenges to be addressed en route to library operationalization of data science, machine learning, and AI. Challenges are organized across seven areas of investigation: (1) Committing to Responsible Operations; (2) Description and Discovery; (3) Shared Methods and Data; (4) Machine-Actionable Collections; (5) Workforce Development; (6) Data Science Services; (7) Sustaining Interprofessional and Interdisciplinary Collaboration. Organizations can use Responsible Operations to make a case for addressing challenges, and the recommendations provide an excellent starting place for discussion and action.

Education researchers use surveys widely. Yet, critics question respondents' ability to provide high-quality responses. As schools increasingly use student surveys to drive policymaking, respondents' (lack of) motivation to provide quality responses may threaten the wisdom of using surveys for data-based decision-making. To better understand student satisficing (suboptimal responding on surveys) and its impact on data quality, we examined the pervasiveness and impact of this practice on a large-scale social-emotional learning survey administered to 409,721 students in grades 2-12. Findings indicated that despite the prevalence of satisficing in our sample, its impact on data quality appeared more modest than anticipated. We conclude by providing an accessible approach for defining and calculating satisficing for researchers, practitioners, and policymakers working with large-scale datasets.

Recent advances in computational linguistics and the social sciences have created new opportunities for the education research community to analyze relevant large-scale text data. However, the take-up of these advances in education research is still nascent. In this paper, we review the recent automated text methods relevant to educational processes and determinants. We discuss both lexical-based and supervised methods, which expand the scale of text that researchers can analyze, as well as unsupervised methods, which allow researchers to discover new themes in their data. To illustrate these methods, we analyze the text interactions from a field experiment in the discussion forums of online classes. Our application shows that respondents provide less assistance and discuss slightly different topics with the randomized female posters, but respond with similar levels of positive and negative sentiment. These results demonstrate that combining qualitative coding with machine learning techniques can provide for a rich understanding of text-based interactions.

It is timely to take stock of the wellbeing of educational research in Aotearoa New Zealand. The major Education Work Programme 2021 includes a Research, Evaluation and Development Strategy. Te Ara Paerangi--Future Pathways Green Paper has launched a major review of the design of public funding for research institutions and for research taking place in tertiary and other education institutions. This paper canvasses the recent history and current landscape of educational research funding and support in Aotearoa New Zealand. It draws on the author's extensive experience in this field, and the need to test the author's growing unease that educational research in this country is, overall, not in good shape.

Returns to education have received much attention from scholars as well as policy makers and the media. It is a well-established fact that educational attainment improves employability and wages and that university graduates better than their less educated counterparts. However, the educational expansion rises the importance of the horizontal dimension in explaining social stratification. The field of study has proved to be an important factor for graduates' labor market performance in many countries. This paper discusses the differences in the labor market outcomes in early careers of graduates of various disciplines in Poland where the topic has been hotly debated for many years but remained under researched. We use data from the second measurement of the recently established Polish National Graduate Tracking System (ELA). The system uses administrative data exported from the national register of students and graduates as well as monthly records on employment status and earnings fetched from the Social Insurance Institution registers. The data cover entire population of recent higher education graduates in Poland and enables tracking for twenty-one months after graduation. We focus particularly on the fields of Science, Technology, Engineering, and Mathematics which are presumed to lead to the best labor market positions. We demonstrate that, contrary to popular opinion, only some of the STEM disciplines give their graduates an edge on the labor market while the rest struggle to achieve results comparable with those of graduates of Humanities and Social Sciences.

The field of learning analytics holds considerable promise for higher education, with reports of successful uses now emerging in selected institutions. At the same time, critics have expressed concerns regarding privacy, ethics, and intrusions into teachers' pedagogy. Without attentive planning, higher-education professionals applying learning analytics may inadvertently undermine their institutions' core teaching and learning missions. The authors offer a framework for moving forward with learning analytics, organized around three principles: (a) Institutions should take the lead in their conversations with vendors, emphasizing the distinctive values of higher education; (b) Learning analytics data should be balanced with other forms of evidence that analytics cannot capture, especially participant experiences; and (c) Successful implementations will leave room for adaptations by people on the ground--to notice what is working and integrate the tools into their practices. Only by empowering students, faculty, and staff can these tools fulfill their potential in higher education.

Community college transfer students encounter challenges progressing toward a bachelor's degree, leading to widespread transfer credit loss. This in turn may lower students' chances of credential completion and increase the time and costs for students, their families, and taxpayers. In this study we review three definitions of credit transfer inefficiency--"credit transferability," "credit applicability," and "excess credits among completers"--focusing on the last to examine why students who start at a community college and transfer to a four-year institution so often end up with excess credits that do not count toward a bachelor's degree. To shed light on credit transfer inefficiency, we examine the course-taking behaviors of community college transfer students who earn bachelor's degrees with numerous excess credits compared with transfer students who earn bachelor's degrees with few excess credits. We employ data-mining techniques to analyze student transcripts from two state systems, enabling us to examine a large number of variables that could explain the variation in students' excess credits at graduation. These variables include not only student demographics but also the types and timing of courses taken. Overall, we find more excess credits associated with several factors, including taking larger proportions of 100- and 200-level courses and smaller proportions of 300-level courses throughout students' progression toward completion, and taking 100-level courses in any subject--and specifically 100-level math courses--immediately after transferring to a four-year institution. Findings suggest that institutions could help students reduce credit transfer inefficiency by encouraging them to explore and choose a bachelor's degree major early on so they can take the required lower division (100- and 200-level) courses at the community college, thereby enabling them to take mostly upper division 300- and 400-level courses in their desired major field once they transfer to a four-year institution.

We use longitudinal data on all high school students in Washington State, including postsecondary education and workforce outcomes, to investigate predictors of intermediate and postsecondary outcomes for students with disabilities. We pay particular attention to career and technical education (CTE) enrollment and the extent of inclusion in general education classrooms, as prior research suggests these factors may be particularly important in influencing the outcomes of students with disabilities. We estimate models that compare students with other students within the same school district, who are receiving special education services for the same disability, and have similar baseline measures of academic performance and other demographic information. We find generally weak relationships between CTE enrollment in any particular grade and intermediate and postsecondary outcomes for students with disabilities, though we replicate earlier findings that students with disabilities who are enrolled in a "concentration" of CTE courses have higher rates of employment after graduation than students with disabilities who are similar in other observable ways but are enrolled in fewer CTE courses. We also find consistently strong evidence that students with disabilities who spend more time in general education classrooms experience better outcomes--fewer absences, higher academic performance, higher rates of grade progression and on-time graduation, and higher rates of college attendance and employment--than students with disabilities who are similar in other observable ways but spend less time in general education classrooms.

This working paper is the first publication coming out of a project investigating the role of vocational education and training (VET) in developing digital skills in the Australian workforce, using two sectors as case studies--Transport and Logistics, and Public Safety and Correctional Services. The study employs a mixed method approach, combining both qualitative and quantitative analyses. It involves industry training package content analysis, content extraction and analysis from online job vacancy advertisements, and key industry interviews, as well as a quantitative employer survey. In the online job vacancy analysis, a total of 1,708 job advertisements covering 74 occupations/job titles were analysed to explore digital skills requirements. In addition, a detailed content analysis was conducted of 11 training packages, with a specific focus on the qualifications for these occupations. In this analysis, 758 units of competency were analysed to examine how and the extent to which digital skills provision is embedded into qualifications. Findings showed that only a small number mentioned digital skills as a requirement. The training packages for these industries contained significant digital training content but with the majority of these units of competency occurring as electives. These findings raise questions about whether employers are making assumptions about the digital skills of potential employees.

The digital revolution and automation are accelerating changes in the labour market and in workplace skills, changes that are further affected by fluctuations in international and regional economic cycles and employment opportunity. These factors pose a universal policy challenge for all advanced economies and governments. In the workplace, people seek to acquire contemporary and relevant skills to gain employment and retain transferable skills to maintain employment. The central purpose of this paper is to investigate how other nations or regions are dealing with these issues. What approaches are they taking to understanding the mix and dynamics of the skills attained by individuals and, more broadly, the totality of skills that in aggregate constitute a highly capable and adaptable labour force, one that supports firm viability and greater national productivity. This research has examined a range of initiatives and approaches being developed or in use in selected countries, including the United States, Singapore and New Zealand, and agencies/organisations; for example, the European Commission and the Skills for the Information Age Foundation. In doing so, it showcases the good practices used to ensure that occupational-level skills information remains current and widely accessible. [For "Identifying Work Skills: International Case Summaries. Support Document," see ED579875.]

We use longitudinal data from Washington State to provide estimates of the extent to which performance on the edTPA, a performance-based, subject-specific assessment of teacher candidates, is predictive of the likelihood of employment in the teacher workforce and value-added measures of teacher effectiveness. While edTPA scores are highly predictive of employment in the state's public teaching workforce, evidence on the relationship between edTPA scores and teaching effectiveness is more mixed. Specifically, continuous edTPA scores are a significant predictor of student mathematics achievement in some specifications, but when we consider that the edTPA is a binary screen of teaching effectiveness (i.e., pass/fail), we find that passing the edTPA is significantly predictive of teacher effectiveness in reading but not in mathematics. We also find that Hispanic candidates in Washington were more than three times more likely to fail the edTPA after it became consequential in the state than non-Hispanic White candidates. Supplementary tables are appended.

This working paper identifies OECD countries' interests in digital innovation in education by analysing their policy papers on digital education. Many OECD countries have adopted a specific strategy on digital education, or integrated the topic in a generic strategy on digital innovation as such. The ideas that are expressed in the strategies differ greatly; some are work in progress, others contain bold envisions of the future. There is a high awareness among OECD countries of the benefits of digitalisation, and the role of government to support digital innovation in education. This paper covers and documents countries' policy focus before the 2020 coronavirus crisis.

This work asks one simple question: "how reliable is the method used by the National Centre for Vocational Education Research (NCVER) to estimate projected rates of VET program completion?" In other words, how well do early projections align with actual completion rates some years later? Completion rates are simple to calculate with a cohort of students who start together in a very short program with a defined end date. The context in vocational education and training (VET) is, however, far more complex. Program lengths vary and may span several years, students commence at different times and many study part-time. Waiting for all students to complete or "drop out" of their training before calculating an actual completion rate gives a reliable answer, but is somewhat impractical. This paper summarises the key findings from a technical review of the validity of the method long used by NCVER in estimating projected completion rates for government-funded VET programs. This analysis required the interrogation of large longitudinal data sets with tens of millions of enrolments over multiple years. The outcomes are revealing because of ever-high interest in completion rates as measures of the efficiency and effectiveness of the VET sector. Key findings include the following: (1) The method long used by NCVER for estimating VET program completion rates using data from the National VET Provider Collection is shown to be reliable and aligns well with actual rates of completion for historical estimates; (2) Given that it takes a number of years for actual rates of completion to stabilise, the method is well suited for inclusion as part of any method of assessing completion rates, where the projected completion rate method is used to estimate rates for the most recent years and actual rates used for prior years; (3) The technical review has also shown that the current predictive method can be improved by defining a program's commencing year as the year it first appears in the National VET Provider Collection rather than using the commencing flag variable; and (4) It is anticipated that the incorporation of unique student identifiers into any preferred methodology, and its extension to total VET activity, can be phased in from the collection of 2017 training activity. "The Current Method for Projecting Rates of Completion: A Working Example" is appended.

The teaching performance expectations and assignments that preservice art teachers currently address in field experiences and in their coursework are designed to help them meet the expectations of being a quality art educator in the 21st century. These assignments may be very different from the assignments that art educators had in their preservice programs just a decade ago. Using tasks from a current preservice assignment, this White Paper presents information and how-to steps that provide essential understandings and relevancy about gathering and using assessment data and assessment procedures in a manner that is clear and supports practical perspectives for art educators. Assessment White Papers for Art Education is a collection of essays that teach about current and established assessment practices and theories, and how they are relevant to all of the National Art Education Association's membership. This series offers a starting point for NAEA members--across all levels--as well as parents, administrators, and the general public to understand assessments' applications to art education.

Most public school teachers in the United States are enrolled in defined benefit (DB) pension plans. Using administrative micro data from four states, combined with national pension funding data, we show these plans have accumulated substantial unfunded liabilities -- effectively debt -- owing to previous plan operations. On average across 49 state plans, an amount that exceeds 10 percent of current teachers' earnings is being set aside to pay for previously-accrued pension liabilities. To the extent that the costs of the unfunded liabilities drag on teacher compensation, they may exacerbate problems of teacher recruitment and retention. We briefly discuss three policy changes that could end or reduce the accumulation of unfunded liabilities in educator pension plans: (1) transition teachers to defined-contribution retirement plans; (2) transition teachers to cash-balance retirement plans; and (3) tighten the link between funding and benefit formulas within the current defined-benefit structure.

We find an individual's rank within their reference group has effects on later objective outcomes. To evaluate the impact of local rank, we use a large administrative dataset tracking over two million students in England from primary through to secondary school. Academic rank within primary school has sizable, robust and significant effects on later achievement in secondary school, conditional on national test scores. Moreover we find boys gain four times more in later test scores from being top compared to girls. We provide evidence for a mechanism using matched survey data, which shows that rank affects an individual's self-concept. The paper discusses other potential channels but concludes that malleable non-cognitive skills such as confidence and belief in own ability are most likely to generate these results. We put forward a basic model where rank effects costs and effort allocation when faced with multiple tasks. We believe this is the first large-scale study to show large and robust effects of rank position on objective outcomes of that have consequences in the labour market. The following are appended: (1) Model; (2) Peer Effects; and (3) Measurement Error in Test Scores. Five tables are also included in the appendices.