Search

The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality. We propose the Model-in-the-Loop (MILO) framework, which integrates AI/ML models into the annotation process. Our research introduces a collaborative paradigm that leverages the strengths of both professional human annotators and large language models (LLMs). By employing LLMs as pre-annotation and real-time assistants, and judges on annotator responses, MILO enables effective interaction patterns between human annotators and LLMs. Three empirical studies on multimodal data annotation demonstrate MILO's efficacy in reducing handling time, improving data quality, and enhancing annotator experiences. We also introduce quality rubrics for flexible evaluation and fine-grained feedback on open-ended annotations. The MILO framework has implications for accelerating AI/ML development, reducing reliance on human annotation alone, and promoting better alignment between human and machine values.

In this work, we introduce a personalised and age-specific Net Benefit function, composed of benefits and costs, to recommend optimal timing of risk assessments for cardiovascular disease prevention. We extend the 2-stage landmarking model to estimate patient-specific CVD risk profiles, adjusting for time-varying covariates. We apply our model to data from the Clinical Practice Research Datalink, comprising primary care electronic health records from the UK. We find that people at lower risk could be recommended an optimal risk-assessment interval of 5 years or more. Time-varying risk-factors are required to discriminate between more frequent schedules for higher-risk people.

In 2013 and 2015 Oregon enacted legislation that shifted school discipline policies from a zero-tolerance approach to one that emphasizes preventing behavioral problems and reducing unnecessary suspensions and expulsions. Suspensions and expulsions are often referred to as exclusionary discipline because they remove students from classroom instruction. This study examined the association between state-level policy reforms and suspension and expulsion rates for grades K-12 in Oregon public schools. The findings suggest that the policy shift has led to some short-term progress on two of the state's main goals--reducing unnecessary removal of students from classroom instruction for disciplinary reasons and reducing exclusionary discipline for weapons offenses that do not involve firearms. Across all grade spans the use of exclusionary discipline declined from school year 2008/09 to 2016/17, with greater reductions for grades 9-12 and 6-8 than for grades K-5. Oregon's policy reforms were associated with short-term reductions in some forms of exclusionary discipline after pre-policy trends and other factors were adjusted for. For example, the 2013 policy reforms were associated with short-term reductions in the number of out-of-school suspensions per student for grades 9-12 and 6-8, and the 2015 policy reforms were associated with a reduction in out-of-school suspensions for grades K-5. For grades 6-8 and K-5 the policy reforms were also associated with reductions in the number of expulsions per student. However, many disciplinary actions reverted--or appeared to be reverting--toward pre-policy trends within a few years. Meanwhile, for all grade spans the policy reforms were not associated with reductions in the use of in-school suspensions after other factors were adjusted for. The declining rates of exclusionary discipline indicate progress, but growth in out-of-school suspensions in recent years suggests the need for further monitoring and additional support. For example, strengthening efforts to reduce suspensions for minor infractions, especially for grades 9-12 and 6-8, could reduce unnecessary suspensions overall--a priority of Oregon's school discipline policy reforms. [For the appendixes, see ED607761; for the study snapshot, see ED607762.]

This white paper submitted for the 2020 Canadian Long-Range Planning process (LRP2020) presents the prospects for Canada and the Square Kilometre Array (SKA) from 2020-2030, focussing on the first phase of the project (SKA1) scheduled to begin construction early in the next decade. SKA1 will make transformational advances in our understanding of the Universe across a wide range of fields, and Canadians are poised to play leadership roles in several. Canadian key SKA technologies will ensure a good return on capital investment in addition to strong scientific returns, positioning Canadian astronomy for future opportunities well beyond 2030. We therefore advocate for Canada's continued scientific and technological engagement in the SKA from 2020-2030 through participation in the construction and operations phases of SKA1., Comment: 14 pages, 4 figures, 2020 Canadian Long-Range Plan (LRP2020) white paper

This paper studies the notion of W-measurable sensitivity in the context of semigroup actions. W-measurable sensitivity is a measurable generalization of sensitive dependence on initial conditions. In 2012, Grigoriev et. al. proved a classification result of conservative ergodic dynamical systems that states all are either W-measurably sensitive or act by isometries with respect to some metric and have refined structure. We generalize this result to a class of semigroup actions. Furthermore, a counterexample is provided that shows W-measurable sensitivity is not preserved under factors. We also consider the restriction of W-measurably sensitive semigroup actions to sub-semigroups and show that the restriction remains W-measurably sensitive when the sub-semigroup is large enough (e.g. when the sub-semigroups are syndetic or thick)., Comment: 13 pages, 1 Figure. Theorem 5.3 strengthened, typos fixed, some new references added

Most U.S. school districts (88 percent) offer credit recovery courses or programs for students. In rural states such as Montana, online credit recovery options are especially popular because they allow schools to serve students in remote areas throughout the year, across a range of subjects, and with few additional resources (Carver & Lewis, 2011; Oliver, Osborne, Patel, & Kleiman, 2009). Such programs offer students greater flexibility and choice, which results in more opportunities to make up classes and a greater likelihood that students will stay in school and stay on track to graduate (Oliver et al., 2009). Despite the growing popularity of online credit recovery courses, there is little research about which students take these courses or how well they perform. This report addresses this information gap by examining 2013/14 data from the Montana Digital Academy (MTDA), the only state-funded program offering online credit recovery courses in Montana. The report provides a descriptive analysis of course enrollment and passing rates. It also draws on interviews with eight district leaders and one MTDA representative to provide context for the analysis and to describe other credit recovery strategies in Montana. The report offers educators an early look at the potential of online credit recovery courses to help struggling students get back on track to graduation. It can also help state leaders compare the MTDA with other online programs and identify possible areas for investigation or improvement when designing credit recovery options. The report's key findings are: (1) More male students than female students enrolled in MTDA online credit recovery courses, and students in grades 10 and 11 made up a larger proportion of MTDA student enrollments than those in grades 9 and 12; (2) Enrollments in MTDA online credit recovery courses were higher in English language arts than in any other subject; (3) The passing rate in MTDA online credit recovery courses was 57 percent of enrollments; (4) Passing rates in MTDA online credit recovery courses were lowest in math (49 percent) and English language arts (52 percent); and (5) Passing rates were lower for students who took one MTDA online credit recovery course per semester (40 percent) than for students who took multiple courses in a semester (nearly 80 percent). Appended are: (1) Study methodology; (2) Local management; and (3) Interview protocol. [This report was written with Biraj Bisht and Rashea Hamilton.]

We show that all rank-one transformations are subsequence boundedly rationally ergodic and that there exist rank-one transformations that are not weakly rationally ergodic., Comment: Added references, minor updates

For nearly a decade, the Chicago High School Redesign Initiative (CHSRI) has worked to improve educational opportunities for Chicago adolescents by opening and supporting small high schools across the city. While the overall results of the initiative have been mixed, several CHSRI schools have successfully improved students' outcomes. This latest research brief in the CHSRI series describes both the practices and characteristics that such high performing CHSRI schools share. Drawing on qualitative data, authors W. David Stevens, Sue Sporte, Sara Ray Stoelinga, and Alissa Bolz discuss common elements of their classroom environments and reveal similarities in how they organize instructional leadership and improvement activities. By highlighting shared practices across these schools, CCSR hopes to identify general lessons that other schools may use to create productive teaching and learning environments. (Contains 22 endnotes and 1 table.)

Policy options for increasing the earnings of the young welfare recipients were explored by analyzing the incomes of nearly 12,000 young women in Baltimore, Maryland, whose 19th birthday fell between April 1, 1985, and March 31, 1989, and who had at least one spell of welfare dependency between their 19th and 29th birthdays. An analysis of the earnings of 24 subpopulations (based on race and short-term, cyclic, or long-term welfare receipt) established that, as a group, the welfare recipients have demonstrated an ability and willingness to work at least some of the time. However, a reasonable estimate of how much they would earn if they devoted more time to work, without factoring in accumulation of and reward for additional human capital, indicated that most would fall short of a full-time federal minimum wage benchmark. The following welfare policy changes were recommended: (1) abandon job entry as a performance indicator; (2) stop overuse of the career ladder for the crafting of advice to those who seek to enter or advance in the labor market and substitute a "climbing wall" metaphor; and (3) revisit the off-the-books earnings issue to gain a better understanding of how it affects the incidence and distribution of employee benefits coverage. (Contains 17 references.) (MN)

The passage of the Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998 (Perkins III) and the Workforce Investment Act of 1998 (WIA) marked a new era in the performance accountability partnership among the states, the U.S. Department of Education, and the U.S. Department of Labor (DOL). Evaluation of vocational education programs' performance is now characterized by quality management principles promoted by the Baldrige National Quality Program and the International Organization for Standardization. The following are among the key elements of the accountability process mandated by Perkins III and WIA: (1) identification of the customers and purposes for performance data; (2) the Office of Vocational and Adult Education's Data Quality Initiative; (3) principles of outcome-based performance review; and (4) application of quality management principles to the Data Quality Initiative. A review of recent federal- and state-level evaluation initiatives reveals that outcome-based performance measurement systems consist of the following core components: (1) a clear goal statement; (2) valid and reliable measurement; (3) standards adjusted for "relevant" differences in circumstances; and (4) performance-based consequences. Before undertaking the design of a high-quality performance measurement system, stakeholders must agree that relevant program activities and reliable performance measures can be linked in a practical way. (Contains 43 references.) (MN)

We describe an atom interferometer to study the coherence of atoms reflected from an evanescent wave mirror. The interferometer is sensitive to the loss of phase coherence induced by the defects in the mirror. The results are consistent with and complementary to recent measurements of specular reflection.

The Steering Committee of the National Postsecondary Education Cooperative established a Unit Record versus Aggregate Data Working Group in 1996 to identify and assess data sharing methods that are needed to advance the understanding of how postsecondary students are served, and with what outcomes. Most mandated record sharing has been through aggregate data methodologies, but some members of the Working Group advocate more widespread development of unit record methodologies. This paper adopts a practical approach to the unit record versus aggregate data topic. It does not argue the universal superiority of unit records over aggregated data, or vice versa. Rather, it examines and evaluates both approaches and the issues that will need to be addressed. After an overview of aggregate data, which refers to data collected at the institutional rather than the unit (student) record level, chapter 2 identifies three stages of data handling that usually involve different responsible parties, legal stipulations, and funding streams: collection, sharing, and release to users. Chapter 3 discusses new approaches, and chapter 4 considers the reporting and sharing process. Chapter 5 presents some recommendations for continued work by the Working Group, with a focus on potential linkages of unit record data to assist postsecondary entities in the development of statistics that will meet current demands for accountability. (SLD)

The school-to-work transition of high school and community college vocational program completers in 1990-1992 was examined by analyzing administrative records and employment and earnings data of vocational program completers from state education agencies in Colorado, Florida, Missouri, and Washington. A consistently high percentage of vocational program completers at both the high school and postsecondary levels continued an uninterrupted affiliation with the same employer during the bridge period encompassing their last months in school and first few months after leaving school; however, substantial movement between/among employers during the first years after the former students left school was observed. Former students who continued with the same employer through the bridge period were consistently found to have higher earnings than their classmates while they were still in school, shortly after leaving school, and at the end of the postschool reference period. It was concluded that knowledge about a former student's occupational assignment within a place of employment is not needed to predict that employee's earnings; rather, awareness of the person's industry affiliation is an acceptable substitute for that purpose. (Forty tables/figures and 86 endnotes are included. Appended are additional information on the wage-record components examined and calculation of a full-time earnings threshold amount.) (MN)

This report presents findings of a demonstration project that tested the concepts and applications of linking student records using social security numbers with state unemployment insurance (UI) wage and other data sources. It also contains information of value to policymakers, state and local administrators, practitioners, and consumers of postsecondary vocational education. Chapter I explores the value of wage data sources in examining trends regarding former students in the workplace. Chapter II provides the historical perspective and looks at the changing legislative environment and factors influencing the use of state UI wage data. Chapter III identifies needs of various audiences for employment and earnings information. Chapter IV describes concepts tested by the demonstration project as well as the data sources and elements used. Chapter V presents findings from the two-state demonstration to illustrate how wage data can be used and presented. Chapter VI provides a profile of the strength of the longitudinal perspective. Chapter VII discusses five steps for state implementation of a system to link school-based information with state UI and other state and federal wage data records: (1) plan and determine data priorities, (2) identify state and federal administrative data sources, (3) make data collection decisions, (4) develop data processing procedures, and (5) conduct data analysis. Appendixes include 91 endnotes and a discussion of the differences between outcome measures and performance standards. (YLB)

State programs offer greatest promise for practical strategic actions to enhance the quality of the nation's adult work force. Three steps in the recursive process that create a model of state action are as follows: (1) documentation of the platform for action, which consists of current adult work force attributes and current institutional capacities to advance the quality of a state's labor force; (2) initial decision with respect to short-term work force quality goals; and (3) subsequent (recurring) decisions. The maturation of initiatives in two states--California and Florida--offers documentation of the instability that characterizes state efforts to address adult work force quality issues. According to the first step of the model, the state's role is to translate national demographic predictions into state-specific labor force quality implications. The goal should be to establish a combined early warning system and accessible intelligence network. An important component of the latter is an accurate inventory of current institutional capabilities. Recommendations for work force quality enhancement include the following: (1) investigate better uses of existing data systems; (2) document current institutional capacities to respond to work force needs; (3) explore the applicability of Canada's JOBSCAN concept; (4) expand the scope of reciprocal expectations from individuals and organizations receiving state benefits; (5) consider the consequences of expanding competency certification; and (6) join the Department of Defense in recommitment of military training funds to adult work force renewal. (Contains 58 bibliographic references and 60 endnotes.) (YLB)

This publication discusses and analyzes issues related to job training in the high school. Section 1 describes the nature of the job training problem by presenting views of constituencies affected by high school programs. The interests of these groups are examined: students, instructional staff members, school administrators, parents, employers, and elected officials. Section 2 provides information on forces affecting criteria for evaluating policy options. These forces are summarized as four views of job training: human capital, signaling, training opportunity, and core and periphery. Section 3 discusses these criteria for assessing policy options: action oriented, equity effect, and efficiency effect. Section 4 presents three policy options for reforming job training in the high school--eliminate job training from the high schools, reorganize high school job training to provide only general prerequisite skills to master any specific occupational area (diminish distinctions between academic and vocational curricula), and reorganize job training in the high school to enhance occupationally specific instruction (sharpen the distinctions between academic and vocational curricula). Discussion then focuses on student access reforms. Section 5 states recommendations intended to aid policymakers in their pursuit of effective and equitable job training programs. (YLB)

This document reports on a systematic examination of the boundary between public and private delivery of vocational education and other employment and training opportunities, in order to find opportunities to shift the boundary toward greater private involvement without sacrificing public responsibilities. In addition to an executive summary, introduction, and 54-item bibliography, the document contains six sections. The first section distinguishes between provision and production. The second section addresses several provision issues in vocational education and employment and training services. The third section is entitled Public and Private Production of Federally Provided Employment and Training Services. The fourth section describes privatization approaches for restructuring provision and production arrangements, including deregulation, tax incentives, vouchers, user fees, contracting out, total withdrawal of government involvement, and sale of assets. The market alternative, and its limitations, for production arrangements for federal employment and training programs is taken up in the fifth section. The last section contains conclusions and recommendations related to performance standards, consumer information, competitive contracting requirements and procedures, and public agency coordination and promotion of increased use of market incentives. (CML)

Correlates of employer-provided training while in the most recent full-time job held were explored. Respondents were nearly 10,000 members of the 1980 high school seniors cohort of the High School and Beyond survey data set. Two types of forces expected to influence participation in employer-provided training were distinguished: incentive and opportunity. Five types of factors were components of these incentive and opportunity sets: education and work experience, personal and household characteristics, personal attitudes, location, and industry/occupational affiliation. Findings indicated 40 percent of the respondents had participated in some type of employer-provided training. No important difference in the overall participation patterns in training appeared between females and males. Females spent 52 percent as many hours and 68 percent as many weeks in training compared to males. Conclusions were that different factors affect the incentives and opportunities upon which females and males acted, education and work experience appear to be more consistent correlates of female participation in training, the industry and occupational affiliation factors appear to dominate as correlates of participation in training, and employers offer training support as a substitute for alternative external sources of work skills enhancement. (YLB)

This two-part paper attempts to establish the facts about, and to debate the merits of, using individual earnings information collected quarterly by states from employers as a source for vocational education performance assessment. The first part of the paper explores three topics: (1) who wants to know about the earnings of vocational education program leavers and why do they want to know? (2) what problems are encountered in using other available data sources that are not found in working with wage-records data? and (3) what specific data items does the quarterly wage-record source offer? The second part of the paper then examines process issues that are encountered in using the wage-record data. Brief mention is made of complementary data sources that enhance the usefulness of the wage-record data. The paper concludes that it is only a matter of time before the wage-record data are used and that procedures must be worked out for the most effective and efficient use of the data. A 31-item bibliography is included. (KC)

An examination of the recent performance of the United States Employment Service (USES) and the State Employment Security Agency (SESA) results in the following conclusions: (1) there are compelling public responsibilities that require the government to provide some employment services; (2) developing demographic, production, and budgetary forces have created an urgent need to redefine federal responsibilities; and (3) the 1990s will challenge the USES to target very limited federal resources in ways that ensure that the SESAs offer reliable competency certification and efficient referrral of candidates to jobs. Policy alternatives include maintaining the status quo; withdrawing from public responsibility for labor-exchange services; giving up state control in the management of labor-exchange responsibilities; abandoning federal oversight of the management of labor-exchange functions; and adopting a hybrid approach that uses market incentives to achieve efficiencies akin to good business practices, but that retains public responsibility for defining appropriate system goals and performance standards. The last option seems best and may include decentralizing through providing block grants to states, offering states competitive grants for innovative practices, and establishing a National Center for Employment Service Training and Research. (Forty-one references are included.) (CML)

Issues and difficulties in the management of formal affiliations between medical schools and managed health-care organizations are discussed, based on the experience of the Case Western Reserve University (Ohio) medical school and the Henry Ford Health System. Difficulties include differing institutional cultures, departmental authority over curriculum and faculty appointments, and competition for preeminence and for patients. (Author/MSE)

Data from four states on vocational education outcomes are used to explore three themes: (1) the range of possible research findings is predetermined by the data collected; (2) frequency, duration, and detail of observation should be justified in terms of the decisions affected; and (3) more diagnostics should be undertaken to complement mandated performance measures. (SK)

Recent satellite observations of ship tracks surprisingly indicate that cloud water decreases with increasing droplet concentrations. However, we find by analyzing detailed simulations of stratocumulus that the reported trend is likely an artifact of sampling, only overcast clouds. The simulations instead show cloud coverage increasing with droplet concentrations, accounting for 25% of cloud albedo increase at moderate droplet concentrations. Our simulations also show that increases in cloud water from drizzle suppression (by increasing droplet concentrations) are favored only at night or at extremely low droplet concentrations, suggesting that the indirect aerosol forcing is overestimated in climate change projections by many general circulation models.

The processes controlling production of ice crystals in deep, rapidly ascending convective columns are poorly understood due to the difficulties involved with either modeling or in situ sampling of these violent clouds. A large number of ice crystals are no doubt generated when droplets freeze at about -40 C. However, at higher levels, these crystals are likely depleted due to precipitation and detrainment. As the ice surface area decreases, the relative humidity can increase well above ice saturation, resulting in bursts of ice nucleation. We will present simulations of these processes using a large-eddy simulation model with detailed microphysics. Size bins are included for aerosols, liquid droplets, ice crystals, and mixed-phase (ice/liquid) hydrometers. Microphysical processes simulated include droplet activation, freezing, melting, homogeneous freezing of sulfate aerosols, and heterogeneous ice nucleation. We are focusing on the importance of ice nucleation events in the upper part of the cloud at temperatures below -40 C. We will show that the ultimate evolution of the cloud in this region (and the anvil produced by the convection) is sensitive to these ice nucleation events, and hence to the composition of upper tropospheric aerosols that get entrained into the convective column.

This study, an interim assessment, has attempted to understand the role of the Federal-State public employment service as a job search intermediary for insured unemployed workers. The conceptual role, the actual role, and claimant responsiveness to role change have been studied of three selected local offices in providing job search services to claimants. The reader is provided a better understanding of efficiency and equity aspects of the employment service role in helping claimants find jobs the extent of variation in claimant circumstances, and the relative importance of particular personal/market characteristics associated with job search processes and the length of time a claimant remains out of work. Employment service, on the average has filled fewer than one out of every five job openings; for the three groups studied, employment service was not influential in facilitating their return to work. Since most claimants obtain new jobs through some other method of search, implications of the study would be a shift from placement to assisted job search. Factors associated with different search strategies might be identified and used as screening criteria before offering more varied job search services to selected claimants. (EA)