Search

Across OECD countries, the increasing demand for evidence-based policy making has further led governments to design policies jointly with clear measurable objectives, and to define relevant indicators to monitor their achievement. This paper discusses the importance of such indicators in supporting the implementation of education policies. Building on the OECD education policy implementation framework, the paper reviews the role of indicators along each of the dimensions of the framework, namely smart policy design, inclusive stakeholder engagement, and conducive environment. It draws some lessons to improve the contribution of indicators to the implementation of education policies, while taking into account some of their perennial challenges pertaining to the unintended effects of accountability. This paper aims to provide insights to policy makers and various education stakeholders, to initiate a discussion on the use and misuse of indicators in education, and to guide future actions towards a better contribution of indicators to education policy implementation.

Since 2012, the Wisconsin Department of Public Instruction (DPI) has maintained a statewide predictive analytics system providing schools with an early warning in middle grades of students at risk for not completing high school. DPI is considering extending and enhancing this system, known as the Dropout Early Warning System (DEWS). The proposed enhancements include better understanding how and why schools use a tool like DEWS, supports and training necessary to translate DEWS into school change, and extending DEWS into other domains such as college and career readiness. This paper identifies national models of predictive analytic systems in education, including a focus on the Early Warning Implementation Monitoring System (EWIMS) (National High School Center, 2013). The paper explores how such policies might succeed in achieving their goals (e.g., dropout prevention and reduction of predictive at-risk behaviors), ways that districts and schools can make the policies more successful, and how states and state agencies like DPI might strengthen the policies, thereby facilitating local success. The paper recommends that DPI consider: (1) fostering a network of schools for professional development and support of implementation of predictive analytics like DEWS and EWIMS; (2) developing modifications of predictive analytic indicators to measure short-term change and progress; (3) merging predictive analytics with findings of current research funded by the statewide longitudinal data system grant that will identify effective strategies for supporting students with different at-risk profiles; (4) soliciting schools for voluntary implementation of the full DEW/EWIMS model; and (5) sponsoring research on existing practices of how schools identify and intervene on behalf of at-risk students. The analysis and recommendations of the paper should not be considered as final but rather as material for further discussion and deliberation--in essence as food for thought and inquiry. The paper is organized as follows. First is a description of the details of DEWS as an example of implementation of a predictive analytics tool. Second is a logic model of the policy, which is the theory of change underlying its intended positive effects on outcomes. Third, beginning an initial assessment of the theory of change tracing the policy from schools and students, is an analysis of the strength of predictive analytic policies, using a framework developed by Porter, Floden, Freeman, Schmidt, & Schwille (1988). Fourth, following the logic model to the school level, is an analysis of the characteristics of organization and process required for successful implementation in schools, using a framework developed by Gamoran and colleagues (2003). Finally, the paper turns to the question of how outside agencies might enable successful implementation of predictive analytics, with a description of the results of a study of how three school districts supported use of college readiness indicators, followed by a discussion of how DPI might strengthen its own policies.

Providing accurate and useful information on student achievement is a rising challenge for state educational agencies. With the passage of the No Child Left Behind Act (NCLB) in 2001, such challenges have become more pressing. A centerpiece of the educational accountability movement, NCLB has prompted states to improve their reporting on student outcomes. Although not mandated by the law, in recent years states have actively moved to revamp their student data systems (SDS) to better address both federal mandates and inform local stakeholders. Many states now utilize unique statewide student identifiers and seek to collect longitudinal data on students' progress throughout their K-12 educational system and beyond. To date, however, there remains considerable variation across states in quality and comprehensiveness of student data systems. How then can states successfully develop and establish their statewide longitudinal student data systems? This report provides an overview of longitudinal student data systems (LSDS) across the 50 states before focusing on cases of the creation, implementation and expansion of SDS in three states. "Emergent State" has just begun assigning unique student identifiers to state assessment data. "Accelerated State" has rapidly developed and implemented a longitudinal student data system and is moving towards including data warehousing capabilities. "Established State" has a comprehensive K-20 longitudinal student data system that includes a data warehouse. These cases illuminate the challenges and successes that state educational agencies have encountered in their efforts to develop, implement and expand LSDS. The report finds that the state education agencies faced similar challenges in the following areas: (1) building state agency capacity; (2) developing reciprocal and efficient relationships with districts and schools; (3) clarifying governance of the data system and ownership of data; and (4) garnering and sustaining state legislative support. The authors identify the variety of strategies that state education agencies employed to meet these challenges, highlighting those strategies that hold the most promise for the development of comprehensive LSDS. The authors conclude with recommendations for state education agency leaders interested in creating and expanding their states' LSDS. Appended are: (1) What do we know about the Evolution of Longitudinal Student Data Systems?; and (2) State LSDS Rankings. (Contains 3 tables and 2 footnotes. This work was supported in part by funds from the Education Policy Center at Michigan State University. )

A long-held ambition for many educators and assessment experts has been to integrate summative and formative assessments so that data from external assessments used for system monitoring may also be used to shape teaching and learning in classrooms. In turn, classroom-based assessments may provide valuable data for decision makers at school and system levels. Currently there are important technical barriers to this kind of seamless integration. Nevertheless there are a number of promising developments in the field. Ongoing research and development aims at improving testing and measurement technologies, as well strengthening classroom-based formative assessment practices. Improved integration of formative and summative assessment will require investments in new testing technologies, teacher training and professional development, and further research and development.

Decentralization and an increased emphasis on community and parent participation represent significant education reform trends over the past decade. These reforms take place in the context of increased emphasis within Education for All (EFA) on improving education quality and outcomes and on strengthening accountability for results. They require that substantial information be available to local and regional stakeholders, school officials, and communities in order to increase transparency, establish a basis for accountability, and provide tools for effective management at the local level. Parents, teachers, school officials, and other stakeholders must be able to assess school performance and status. A number of countries are experimenting with school-level information systems known as "school report cards" to increase accountability and transparency. These systems have different formats and purposes, ranging from strict accountability systems that measure student performance to participatory diagnostic and management tools that support school managers. Efforts are relatively novel, and substantial evaluation information is not yet available. The purpose of this report is to present the various types of school report cards and information systems currently being used and establish a typology for understanding the range of audiences and purposes for such systems, as well as the continuum of cost and sophistication involved.

Institutional researchers play a key role in an environment where data warehouses are used to store and retrieve vast amounts of data. Along with the benefits of increased access for more users, improved reporting capabilities, and less reliance on centralized information, come questions of appropriate data usage, data access and security, training, and on-going support. This paper discusses the development of a student decision support system at a university and the role played by the institutional research staff. Following the successful implementation of a data warehouse serving an employee and financial decision support system, development of a student decision support system was begun in January 1996. The paper outlines each step in the process, beginning with the decision model for the data to be included and a brief description of the technology. The following sections discuss implementation of the system, which included determining who should have access, security guidelines, and training potential users. Also discussed are data usage, analysis and reporting, on-going support, the student steering committee, standard queries, data validation, other support services, and metadata. Issues still to be resolved include an issue log, adding new data, software upgrades, and who organizationally will be responsible for on-going maintenance and enhancements. A user survey is appended. (Contains 6 references.) (CH)

This paper describes a methodology for helping institutional research assess its roles and its effectiveness in supporting decision-making at all institutional levels. The paper discusses how to meet this goal by planning, doing, checking, and acting. Planning requires the understanding of the roles of the data custodian, the broker, and the information user. Doing involves performing these roles through five sequential steps. Checking requires understanding and overcoming barriers to effectiveness. Acting involves using strategies to integrate and interpret information in order to enhance its value to the institution. Individual sections of the paper define the term institutional research, establish the framework of the model, and discuss management of the various institutional research functions. Recommendations and suggestions for overcoming barriers are discussed. A table listing barriers to effective institutional research is appended. (Contains 15 references.) (CH)

In response to changing management and budgetary strategies that have been driving decision making to the lowest administrative levels, institutional researchers have been attempting to identify indicators which can be used to evaluate institutional performance and progress. This paper reviews efforts to introduce departmental operational performance indicators at the University of Minnesota. A decision support schematic illustrates the instructional resource management tool that was developed. The diagram shows the relationships of the broad indicators, such as demand, fiscality, and productivity, to each other and to specific lower-level measures such as tuition revenue management, instructional cost management, curriculum management, enrollment and tuition management, and faculty course management. Two other programs, a "U2000 critical measures" process and incentives for managed growth, are also discussed. The former, an attempt to demonstrate to faculty how daily planning and operational decisions affect overall institutional goals, was judged to have failed. The initial attempts to establish an "incentives for managed growth" program (also termed responsibility-centered management), a financial strategy that emphasizes entrepreneurial activities and encourages decision making at the lowest administrative levels, were disappointing, and few of the university's departments were deemed able to use this strategy. (CH)

This paper presents a model for obtaining and organizing managment information for decision making in university planning, developed by the Bureau for Management Information of the University of South Africa. The model identifies the fundamental entities of the university as environment, finance, physical facilities, assets, personnel, and students. The model proposes a computerized decision support system which analyzes the mathematical relationships between different components of the system. The support system's data flow plan is presented as a plan that can be constructed independently within each entity, that generates output that can be used by another model as input, and that enables the measurement of variables in one submodel compared to another. The model emphasizes integrated scenario construction, in which a single run can potentially produce outputs of enrollment projections and projections of the needs for personnel, physical facilities, and finances. The paper describes the hardware and software requirements; the structure of files in the system; and the application of the model to enrollment, personnel planning, physical planning, long-term physical planning, and subsidy planning. A final section treats the establishment and use of efficiency measurements. Included are 15 figures and 6 references. (JB)

A statewide system of reporting faculty workload for the 18 Maryland state community colleges is described, and study findings on faculty workload are presented. Data are collected for full-time teaching faculty who are tenured or on continuous contract. Major findings include: 89 percent of faculty workload is spent in formal classroom instruction; the statewide average for released time is three hours annually per faculty member (approximately one class); 83 percent of full-time faculty teach eight or more classes annually; a typical full-time faculty member generates an average of 547 student credit hours annually, and a typical full-time faculty member teaches an average of 21 students per class. The format for the faculty workload report is illustrated, and data are presented for the specific community colleges, which are grouped by size (large, medium, small). The data collected cover: number of faculty, contractual teaching hours, released time hours, contractual workload hours, teaching overload hours, total workload hours, contractual sections taught, overload sections taught, total students taught, average section size, and student credit hours. (SW)

Total quality management (TQM) was employed to improve the collection, storage, and use of racial/ethnic information at Cornell University. The initiative began with the task of standardizing ethnic codes across a number of university systems. The project to improve race/ethnicity reporting was undertaken by a team representing various university offices. The work of the team and issues encountered are discussed, with attention to a seven-step problem-solving methodology. This approach involved: (1) identifying the problem, (2) breaking into smaller segments to determine where improvement is needed, (3) identifying root causes of the problem and potential corrective actions, (4) planning steps to reduce the problem's severity, (5) reexamining the indicator and assessing efficacy of countermeasures, (6) preventing recurrences of the problem, and (7) reviewing what has been learned and possible applications to other problems. After implementing two of three proposed countermeasures, the incidence of unreported racial/ethnic data among entering freshmen was reduced from approximately 15 percent to less than 5 percent. Initial barriers are also noted, which resulted from team members' not having a shared perspective and understanding of the infrastructure surrounding racial/ethnic student data. A questionnaire for obtaining information on race/ethnicity is appended. (Contains 12 references, 5 tables and 5 figures.) (SW)

The issue of increasing the effectiveness of alumni development and fund raising programs to ensure quality in academic programs and campus life for the coming decade was examined. The study's primary purposes were to: (1) describe the development of a comprehensive decision support system for alumni and fund raising activities at one independent university--the University of Hartford, Connecticut; (2) examine specific research and management applications of that system; and (3) feature examples of effective analyses and reports it can generate. Discussions focused on early collaboration between institutional researchers and advancement professionals which facilitated the study and led to construction of a multipurpose alumni survey. Preliminary results indicated that administrators desiring to strengthen fund raising programs should concentrate on the following areas: improve community and donor relationships; gain the confidence of the institution and the participation of its president; build commitment among staff to improve outcomes; interact with people (including staff); and participate in other cultivation/solicitation activities. Contains 18 references. (GLR)

The development of an information system to help College of Education faculty at Florida State University campuses determine the criteria and procedures for awarding merit pay is described. Faculty groups need information that address both the objectives of individual faculty members and institutional goals. Data needed from the following five sources were examined: five-year salary history, faculty activity for current term, faculty assignment form, types of publications individual faculty completed over the previous five years; and the ranking of journals by faculty. Data collected on instruction and research from existing records from 22 programs, six departments, and for the college as a whole are summarized. Findings include: productivity appears unrelated to rewards; instructional and research productivity appear highly related; and the presence of active Ph.D. candidates appears unrelated to research productivity. Using the information system to identify the most productive programs can help direct both rewards and other resources to them. Institutional issues of faculty equity and motivation to maintain an academic culture are considered in light of a salary model. Conclusions are drawn about an information system to monitor faculty rewards. A salary increase form and faculty activity form are included. (SW)

The development of an unusual database at Virginia Polytechnic Institute and State University (Virginia Tech) is described. The Student Census-date Report File (STUCENFL) was designed to meet both internal and external data recipients' needs for student-related information. Attention is directed to needs for the database, underlying design concepts, the development process, and benefits. STUCENFL yields reports and ad hoc analyses that contain valid and internally-consistent information, and that cross-total to related reports and studies. The data items of STUCENFL are a subset of the student, academic, and timetable portions of the Student Data Base. One STUCENFL record is extracted for each student, containing demographic, diploma, admissions, academic history, and current course registration data. The data are extensively edited and updated to achieve valid, consistent, and reliable information. Details of the system, including data elements (BSR and USDS), derived report files, and records for six student populations are presented, along with a glossary of terms. Information for users of STUCENFL as a data source for automated systems is also included. (SW)

The development of a faculty information system at Carnegie Mellon University is traced using a case study approach. Attention is focused on the process of system design by committee, the resulting relational database, and its impact on institutional research and university reporting. Technical implementation and details of the system design are covered. Goals for the system were to provide: access to both current and historical information; and an integrated database of faculty information from which queries, reports, and analyses could be easily generated. The ultimate goal was to provide information to support faculty-related decision-making processes, both centrally and in colleges and departments. The relational database management system, INGRES, was chosen for development of the information system. Included in the system were biographical, salary, teaching, and research data. Future directions in the ongoing project include the development of a university information system that will integrate faculty, student and space data, and the transfer of data from this system through the campus-wide network of personal computers. (SW)

A study of the fund raising program design aspects commonly considered important in fund raising success and the strength of these aspects in institutions with various levels of fund-raising success is reported. The 398 institutions studied include public and private research universities, public and private doctoral universities, public comprehensive universities, and public and private four-year colleges. Fund-raising data from the years 1985-86 through 1987-88 were used to determine institutional success ratings. Development officers were asked to rate the importance of each program aspect to fund-raising success and evaluate each aspect at their own institutions. Program aspects investigated include the donor research system, donor tracking system, management information system, market position analysis, planning and goal-setting, program evaluation, staff evaluation, chief development officer role, chief development officer leadership, close relationship with donors, commitment to excellence, commitment to the institution, entrepreneurial atmosphere, institutional commitment to fund-raising, institutional image defined and well-known, presidential leadership in fund-raising, and trustee participation and support. Results are presented and interpreted. They suggest general agreement on important program aspects, with some differences in perception by institution type. Includes 12 references. (MSE)

The Executive information System (EIS), a decision support system for the executive, is defined, a comparison is made between EIS and its predecessors, and the differences between EIS in academic institutions vis-a-vis private business firms are discussed. An outline is also provided of the technological and data requirements for executive information systems in higher education as well as a table that lists Executive Information/Support Systems, the companies that provide the systems, the area of business the systems are designed for, and the EIS product. Finally, examples of EIS as they are used within specific universities and colleges are described. Contains 5 references. (GLR)