Search

This paper presents a conceptual exploration of how Digital Learning Platforms (DLPs) can be utilized to investigate the impact of language clarity, precision, engagement, and contextual relevance on mathematics learning from word problems. Focusing on three distinct DLPs--ASSISTments/E-TRIALS, MATHia/UpGrade, and Canvas/Terracotta--we propose hypothetical studies aimed at uncovering how nuanced language modifications can enhance mathematical understanding and engagement. While these studies are illustrative in nature, they provide a blueprint for researchers interested in leveraging DLPs for empirical investigation so that future investigators gain a better understanding of the emerging infrastructure for research in DLPs and the opportunities provided by them. In highlighting three distinct implementations of the same core research question, we reveal both commonalities as well as differences in how different educational technologies might build evidence, offering a unique opportunity to advance the field of math education and other education research fields.

In the field of education, we bump constantly into buzzwords, pass unclear phrases in the hall, and greet acronyms as good friends. Yet rarely is it that we unpack or critically examine what is meant when phrases are uttered. What does it actually "mean" to "enhance institutional effectiveness"? To "foster synergy"? To "improve student learning"? In the field of assessment, there is a stated desire for "more use of assessment results," to "close the loop." There is agreement that the purpose and intention for engaging in assessing student learning is to ultimately "improve student learning" but there is no clear framework for what "improving student learning" entails or what it means to "close the loop." However, there does seem to be agreement that whatever "use" might mean, there is not enough of it happening let alone with regularity (Kuh et al., 2015). This paper provides an overview of an alternative conception of use through the lens of evidence-based storytelling--an approach that has been used at the National Institute for Learning Outcomes Assessment (NILOA) to refine and encourage evidence-based stories in assessment (Jankowski & Baker, 2019). This occasional paper serves two main purposes: to re-examine what is meant by use of assessment results and to unpack evidence-based storytelling and its connection to assessment.

For researchers seeking to improve education, a common goal is to identify teaching practices that have causal benefits in classroom settings. To test whether an instructional practice exerts a causal influence on an outcome measure, the most straightforward and compelling method is to conduct an experiment. While experimentation is common in laboratory studies of learning, experimentation is increasingly rare in classroom settings, and to date, researchers have argued it is prohibitively expensive and difficult to conduct experiments on education in situ. To address this challenge, we present Terracotta (Tool for Education Research with RAndomized COnTrolled TriAls), an open-source web application that integrates with a learning management system to provide a comprehensive experimental research platform within an online class site. Terracotta automates randomization, informed consent, experimental manipulation of different versions of learning activities, and export of de-identified research data. Here we describe these features, and the results of a live classroom demonstration study using Terracotta, a preregistered replication of McDaniel et al. ("Journal of Applied Research in Memory and Cognition," 1(1), 18-26, 2012). Using Terracotta, we experimentally manipulated online review assignments so that consenting students alternated, on a weekly basis, between taking multiple-choice quizzes (retrieval practice) and reading answers to these quizzes (restudy). Students' performance on subsequent exams was significantly improved for items that had been in retrieval practice review assignments. This successful replication demonstrates that Terracotta can be used to experimentally manipulate consequential aspects of students' experiences in education settings. [This is online version of an article published in "Behavior Research Methods."]

In March 2020, institutions abruptly pivoted to remote instruction, sending students, faculty, and staff away from college campuses in response to COVID-19. In June 2020, the National Institute for Learning Outcomes Assessment (NILOA) launched a survey to capture a snapshot of assessment-related changes made during Spring 2020 in response to the sudden shift to remote instruction and to help determine remaining professional development needs. The brief questionnaire focused on changes that were made, potential impacts of those changes on assessment culture, and the role student voice and equity concerns did or did not play in Spring decisions. This report provides an overview of findings from the survey; couples those findings with other reports released from March through July; and provides guidance in the form of "do's" and "do not's" for higher education and the field of assessment--looking beyond Fall 2020 toward what needs to be done.

Stronger connections are needed between employers and higher education, however, questions abound about how best to work in partnership. Stemming from various conversations with members of the higher education and employer communities, this Occasional Paper provides tips for fostering successful partnerships and collaboration. Three lessons learned from successful employer and higher education partnerships are introduced: the importance of context, tips for sparking conversation between faculty and employers, and which questions to ask one another to find common values and deeper understanding. We hope you consider some of the tips and possible conversation foci presented in this guide as you engage in relationship building and find them useful in your employer partnerships. [For "Co-Designing Assessment and Learning: Rethinking Employer Engagement in a Changing World. Occasional Paper #39," see ED598871.]

Traditionally, transcripts present a collection of lists of courses and grades in a format valuable for trading information between and among institutions of higher education. What if transcripts were reimagined to not only provide information on credits earned, but also provide information on learning acquired both inside and outside the classroom? What if the transcript documented learning in a digital portable record, regardless of where the learning took place or was acquired? Beginning from a partnership between the American Association of Collegiate Registrars and Admissions Officers (AACRAO) and NASPA: Association of Student Personnel Administrators, the work to develop Comprehensive Learner Records (CLR) sought to address these questions with the help of a group of pilot institutions. The National Institute for Learning Outcomes Assessment (NILOA) provided professional development support to the pilot institutions as the pilot participants mapped learning and identified assessments of learning from both within and outside of the classroom for inclusion in a formal record. This occasional paper provides an overview of the Comprehensive Learner Record as well as connection points to assessment.

This Occasional Paper outlines lessons learned about mapping and assessing learning in student affairs and student employment. Over the last three years, the National Institute for Learning Outcomes Assessment (NILOA) assisted institutions developing Comprehensive Learner Records and scaling high-impact practices. In each of these initiatives, mapping of learning, redesigning assessments, and creating assignments was a staple of practice. Further, assessment practitioners were reminded that before staff jump into implementing assessment and writing the perfect learning outcome statement with just the right verb, time is needed to think about what is being built, why, how, and which students are best served by it all. The discussion centered upon: What is the role of student affairs as part of a larger system of interlocking and supporting learning for students throughout an institution that builds towards common or shared learning outcomes? This resource provides inroads to such discussions, and offers resources to inform practice and for use in institutional professional development.

Entering into a new decade with an even more diversified college student population will not only require more assessment models involving students but also deeper professional development of institutional representatives key to student learning. Reflecting upon the conversations over the last three years around culturally responsive assessment and related equity and assessment discussions, this occasional paper highlights questions, insights, and future directions for the decade ahead by exploring what equitable assessment is and is not; the challenges and barriers to equitable assessment work; where the decade ahead may lead; and next steps in the conversation on equity and assessment. [For "Equity and Assessment: Moving towards Culturally Responsive Assessment. Occasional Paper #29," see ED574461.]

Learning frameworks are tools that specify learning outcomes and/or competencies that define, classify, and recognize educational, learner, and industry expectations of knowledge, skills, and abilities at increasing levels of complexity and difficulty. They allow for alignment, translation, and mapping of learning through various spaces in order to capture learning that can be valued and recognized by education, industry, and the military. This paper outlines the roles that learning frameworks play in the emerging ecosystem of connected learning--why they matter and how they can bring disparate pieces of the learning ecosystem together for greater portability and documentation of learning in all the places it unfolds. It concludes with a focus on technological innovations as a source of future directions for learning frameworks connectivity.

Over the past decade, the National Institute for Learning Outcomes Assessment (NILOA) has been documenting what colleges and universities are doing to gather evidence about student learning and helping institutions to productively use assessment data to strengthen undergraduate education. NILOA also has been monitoring how institutions communicate with policy makers, families, and others about their efforts to enhance student learning and institutional effectiveness. One mechanism utilized to understand the landscape of assessment practices in United States higher education has been national surveys of senior academic leaders about what is being done to measure student learning outcomes and how results are used to improve teaching and learning. This report summarizes the findings from NILOA's third and most recent survey which was conducted in 2017. It is based on data collected from provosts between April and September 2017. The sample included provosts/chief academic officers (or their designees) at 2,781 regionally accredited, undergraduate degree-granting institutions. The questionnaire was completed by representatives of 811 institutions for a response rate of 29%. Nearly 80% of the survey respondents were from within the office of the provost, with the remainder of the surveys completed by those responsible for assessment within the institution. Appendix A contains additional information about the sample and data analysis. The 2017 questionnaire asked respondents about institution-level assessment, repeating many questions from the first and second survey efforts on assessment methods, uses, drivers of assessment practice, availability of assessment information, while adding a few new questions about initiatives to improve student learning.

Tuning is a bottom-up, faculty-led process which leads to the creation of a discipline-specific learning outcome document along with a degree profile that is used to communicate the value of a particular degree to a variety of audiences including students, employers, policy makers, and the general public. Tuning, introduced in the United States in 2009, involves a five-part process to develop a discipline core: defining the discipline core, mapping career pathways, consulting stakeholders, honing core competencies and learning outcomes, and implementing results locally and writing degree specifications. In contrast to the description of the process of Tuning outlined in the Tuning process report (Marshall, 2017), this study presents the outcomes from the work of Tuning in the United States from 2009-2016. The findings in this report emerge from a review of interviews, surveys, institutional activity reports, grantee interim and final reports, project evaluation reports, observations of project meetings, conference presentations, journals, and materials developed from the Tuning process. This Tuning impact study explores the intended outcomes of Tuning as well as the assumptions behind the benefits of engaging in the Tuning process. The five sections of this report present the findings of the Tuning impact study addressing reaching consensus, fostering faculty-led cross-institution discussions, learning-focused transfer and pathway conversations, expanded conversations on educational redesign and assessment, and student focused and employer engagement. The report concludes with a review of common misconceptions and challenges as well as final thoughts.

As colleges educate a more diverse and global student population, there is increased need to ensure every student succeeds regardless of their differences. This paper explores the relationship between equity and assessment, addressing the question: how consequential can assessment be to learning when assessment approaches may not be inclusive of diverse learners? The paper argues that for assessment to meet the goal of improving student learning and authentically document what students know and can do, a culturally responsive approach to assessment is needed. In describing what culturally responsive assessment entails, this paper offers a rationale as to why change is necessary, proposes a way to conceptualize the place of students and culture in assessment, and introduces three ways to help make assessment culturally responsive.

Examples from the field indicates that effectively using the Degree Qualifications Profile (DQP) can benefit both students and institutions (Jankowski & Giffin, 2016). But what does "effective use" entail and how does it unfold? That is, what is the nature of the process that makes it possible for institutions to use the DQP to achieve desired ends? In a post-convening survey of participants following the October 2014 launch of the revised DQP, 91% of participants agreed or strongly agreed with Lumina's call for widespread implementation of DQP, but only 5% agreed that they understood the next steps in order to implement it. With this in mind, the National Institute for Learning Outcomes Assessment (NILOA) has been tracking campus engagement with the DQP, identifying approaches that institutions have used to implement the framework in meaningful ways. In this report, we describe those approaches and how they have been used within and across institutions.

Following the release of the Degree Qualifications Profile (DQP) in 2011, many institutions of various types tried out different ways to use the DQP. Although over 680 institutions have used the DQP to date, until now the impact of the DQP on institutions and students has not been documented in a systematic manner. To determine the effects of DQP use on institutional policies and practices, the National Institute for Learning Outcomes Assessment (NILOA) conducted a study of the more than 400 (n = 425) institutions that used the DQP between the 2011 release and the October 2014 revision. The study explored how institutions engaged with the DQP and how working with DQP was associated with changes in curriculum, instructional practices, and assessment activities. One appendix is included: Funded DQP Projects and Related Initiatives.

Most institutions are collecting evidence of student learning, but it is not clear how these results are being used to improve student outcomes. To learn more about what colleges and universities are doing to use assessment data productively to inform and strengthen undergraduate education, the National Institute for Learning Outcomes Assessment conducted nine case studies. This report synthesizes the insights from these individual studies to discern promising practices in using information about student learning. Institutional behavior appears to be generally consistent with the Principles of Effective Assessment Practice identified by Banta and Associates (2002). For example, the case study institutions took advantage of calls for accountability to leverage internal improvement efforts, communicated widely about assessment efforts and results, and took time to reflect on their assessment activities and results. Equally important, the case study institutions aligned their assessment work with organizational structures and cultures, and focused their assessment efforts on specific problems or questions. The report concludes with lessons learned and reflective questions to help institutions advance their own assessment efforts within their specific institutional contexts. Appended are: (1) Methodology; and (2) Annotated Case Study List. [Funding for this paper was provided by the College of Education at the University of Illinois.]

The Voluntary System of Accountability (VSA) is a vehicle for public four-year universities to report comparable information about the undergraduate student experience via the College Portrait, a common web reporting template. The National Institute for Learning Outcomes Assessment (NILOA) was asked by the VSA to evaluate the effectiveness of the student learning outcomes pilot project within the College Portrait. The evaluation was conducted over a five-month period, October 2011 through February 2012, drawing on a variety of data sources. These included focus groups, interviews with leaders from the policy arena and regional accreditation agencies, institutional surveys, and analyses of results from the VSA-conducted survey of participating institutions, and College Portrait database statistics and Google Analytics. The authors' review found widespread agreement that the launch of the VSA in 2007 was a necessary response to the demands of the time. Accountability demands and public interests are transitory in nature and while the press for transparency persists, the environment in which the VSA exists is different from when it began. While seen as "necessary," many observers also expect the VSA to continue to evolve. They also found that the standardized tests of student learning originally approved for inclusion in the pilot lack credibility and acceptance within a broad sweep of the higher education community which, in turn, serves to undermine institutional participation in the VSA. Given this mixed picture, they asked, "Should the VSA College Portrait be continued?" With some exceptions, the response was affirmative, suggesting "mend it, don't end it" regarding the value of the VSA and the College Portrait. They recommend the VSA College Portrait be recast as a state-of-the-art electronic communication tool targeted for specific audiences, including prospective and current students, parents, and guidance counselors; alumni; faculty and staff; trustees; employers; accreditors; public policy makers; and media. They also urge that information presented on the College Portrait be presented around questions of particular interest to students and other relevant audiences to tell a contextualized, institution-specific, evidence-based story--possibly incorporating video or other media. Finally, to expand viewership of the College Portrait, they recommend consideration be given to the creation of a College Portrait template that could be adapted and used by all postsecondary institutions, public and private, community colleges and others. A common template for all of higher education, while challenging to achieve, would serve as one access point for the public and thereby attract increased viewer traffic to the site. Appended are: (1) Voluntary System of Accountability: An Overview; and (2) Methodology. (Contains 5 figures and 4 footnotes.)

This booklet provides information to multiple stakeholders on the implementation of career clusters in Illinois. The booklet is an extension of the previous edition titled "An Introduction to Illinois CTE Programs of Study" (2008), and provides a resource for partners to understand Illinois' Career Cluster Model as its own adaptation of the States Career Cluster Framework. It is intended for individuals familiar with career clusters and programs of study and those beginning to learn about them. The booklet is divided into seven sections: (1) Perkins IV: main themes of the Carl D. Perkins Career and Technical Education Improvement Act of 2006; (2) Key Concepts: terms, definitions and benefits of the career cluster approach, to help partners become familiar with the States Career Cluster framework; (3) Career Cluster Framework: graphic representation of the essential knowledge and skills, the 5 career and technical education areas at the secondary level, the 16 career clusters, and the 79 career pathways; (4) Illinois' Career Cluster Model: visual example of the Health Science cluster and how different elements of the States Career Cluster Framework fit together in Illinois; (5) Implementation: information useful to preparing partners to implement programs of study, including the six guiding principles and partner roles in the process; (6) Acronyms: list for reference and ease of communication between multiple partners on programs, initiatives, and organizations; and (7) Resources: links to tools, research, and national organizations related to Perkins IV, career development, career clusters, and other information specific to Illinois. (Contains a list of resources and acronyms.)

This document reports on federal Research and Development (R&D) funding trends for the last 10 years and explains the sources of Federal R&D revisions. The data are obtained from an annual census of approximately 30 federal agencies that report obligation data to the National Science Foundation Survey of Federal Funds for R&D. (YDS)

This monograph proposes an approach to discipline that emphasizes school community building and offers examples from the Kendall Demonstration Elementary School (KDES) and the Model Secondary School for the Deaf (MSSD), both in Washington, DC. Following discussion of the problem of violence in today's schools, the paper addresses the importance of developing students' emotional, interpersonal, and intrapersonal skills; of developing students' work skills; and of encouraging parent participation. Suggestions for schools focus on practices at KDES and MSSD, where the emphasis is on building a school community with a caring and cooperative environment. Strategies used in such schools include identification of community building goals; development of a community building mini-manual at the MSSD; the Enhancing Student Success Program, an alternative to suspensions; identification of expected student outcomes; and a positive discipline program that offers teachers strategies for classroom behavior management. The slogan, "It All Connects," and four classroom examples illustrate how these principles and strategies have been applied at KDES. Eight specific suggestions for teachers are offered. An appendix provides further detail on community building at these two schools. (Contains approximately 50 references.) (DB)

There is no shortage of challenges facing postsecondary institutions in the US. One that cuts to the core of the enterprise is whether they are preparing their graduates to live productive, civically responsible lives in a dynamic global marketplace mapped onto diverse, yet increasingly interdependent, social and cultural systems. Much of the evidence presented in recent "Change" articles suggests that what undergraduate students know and are able to do falls well short of what employers, policymakers, and educational leaders say is needed. Whether one accepts the available evidence as sufficient to draw such a sweeping conclusion ultimately rests on resolving two non-trivial issues. First, key stakeholders--those mentioned above and others, including students--must agree on the constellation of knowledge, skills, competencies, and dispositions that need to be signaled by postsecondary degrees and credentials if they are to be attuned to the demands of the times. Second, what evidence do we have that students have achieved the desired levels of proficiency in the respective outcome areas? Countries throughout the world--in Europe, Australasia, and Central and South America--have made considerable progress in addressing these challenges by developing degree qualifications frameworks that articulate what outcomes graduates of their colleges and universities should have, along with behaviorally anchored indicators and other measures that mark the extent to which students have acquired them. A handful of institutions in the US--such as Alverno College, the military academies, and Western Governor's University--have done something akin to this. But it is only recently that concerted efforts have been mounted to bring greater clarity and more widespread agreement about what credentials and degrees should represent by more precisely defining what college students in this country need to know and be able to do and at what level of proficiency. This article is about the status and aspirations of one such effort, Lumina Foundation's Degree Qualifications Profile (DQP).