Search

Openness in education is not a consistent term or value since "open" is used to describe various things and often means different things to different individuals. In a research context, it is important to identify the many interpretation(s) and perspectives of openness being investigated, especially since the underlying ideas behind these different interpretations and contexts can yield different results. Not much empirical research on the implementation aspects of open education exists, especially comparing open educational resources (OER) and open online education (OOE). This empirical study addresses this gap, exploring identification and prioritization of organizational challenges and opportunities of two subgroups of projects (i.e. OER focused or OOE focused) within various higher education institutions in The Netherlands. The main research question in this study is: Does the project character (OER focus vs. OOE focus) of innovation projects lead to perceived differences by actors involved in their implementation? Findings indicate that there are differences in conceptual as well as practical representation between the two groups. These findings imply that higher education institutions need to internally adapt to the needs of various manifestations of "openness" to be able to fully benefit from opportunities and overcome challenges.

Background and Objectives: The Youth Engagement Network (YEN) was a three-year project focused on strengthening meaningful youth engagement in sexual health programs. It sought to answer the question, "What is a useful framework for meaningful youth engagement of vulnerable populations in sexual health programming?" Methods: The YEN chose Group Concept Mapping (GCM) as the approach to answer their learning question. GCM is a mixed-methods process that can be used to integrate the unique perspectives of a group of individuals. The approach entails 1) brainstorming ideas related to a prompt, 2) individual sorting and rating of the ideas, and 3) the application of multi-dimensional scaling (MDS) and hierarchical cluster analysis to integrate the individual sorting information and develop a series of concept maps. Results: Seventy-four young people and adults generated 141 responses to our prompt. These were pared down to 82 statements that were then sorted and/or rated on importance by 113 participants. Our final cluster map focused on 9 concepts: Facilitators, Diversity, Relevance, Access, Connection, Comprehensive Sex Ed, Co-development, and Space. Conclusions: We found GCM to be useful for identifying potential components of a framework to move the field toward more meaningful engagement of young people in sexual health programming. [ABSTRACT FROM AUTHOR]

State early intervention programs (EIPs) have been encouraged to develop and implement comprehensive outcomes measurement systems. As the number of children with autism spectrum disorders (ASD) and their families receiving services in state EIPs increases, disability-specific outcomes data are needed to better understand issues of access, costs, and benefits. The New York State Early Intervention Program (NYSEIP) used concept mapping with parent and professional stakeholders to define child and family outcomes expected from early intervention services for children with ASD and their families. Stakeholders generated more than 700 statements (synthesized to 105), then individually sorted (17 parents, 39 professionals) these statements into categories. Stakeholders rated each outcome on importance (20 parents, 48 professionals) and likelihood (18 parents, 46 professionals) that EIP services will yield a positive impact. The aggregated sort data were subjected to two separate sets of multivariate statistical analyses, producing an 11-cluster map of professionals' conceptualization and a 7-cluster map of parent/family members' conceptualization of the outcomes. Possible explanations for the parent/professional differences are discussed. The results will be used as part of ongoing efforts to develop a comprehensive evaluation system for state Part C EIPs that is grounded in stakeholders' perspectives on expected outcomes for children and families.

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed. (Contains 2 tables and 3 figures.)

Background: Although models such as the coordinated school health program (CSHP) are widely available to address student health needs, school professionals have been unconvinced that scarce resources should be allocated to improving student health. Concern that attention may be diverted from meeting academic accountability goals is often seen as a reason not to attend to student health. Despite continuing calls for the study of multicomponent health programs in relation to educational achievement, the understanding of the extent to which adherence to the characteristics of CSHP contributes to or compromises academic outcomes over time remains incomplete. Methods: A retrospective study was conducted of CSHP implementation across 158 public schools in Delaware, serving grades K-12. Using a doubly multivariate design, this study examined 3 levels of CSHP implementation across 5 school-level academic indicators for 3 years. Indicators included school performance, school progress, and aggregated student performance in 3 content areas--reading, mathematics, and writing. Data for the years prior to, during, and following implementation of CSHP were analyzed. Results: Multivariate main effects of year by implementation level were detected. CSHP schools with high levels of implementation had better school-level performance and progress ratings. CSHP implementation did not have an effect on reading, math, and writing indicators, though all groups showed significant improvements over time in these areas. Conclusions: Results of this study suggest that quality implementation of CSHP does not adversely impact school-level academic indicators over time. Moreover, findings suggest a better fit with school-wide accountability indicators than with specific content-based achievement indicators. (Contains 1 figure and 3 tables.)

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a fully integrated part of scale development. As part of a multi-site evaluation of family support programs, we integrated concept mapping with traditional scale-development processes to strengthen the creation of a scale for inclusion in an evaluation instrument. Using concept mapping, we engaged staff and managers in the development of a framework of intended benefits of program participation and used the information to systematically select the scale's content. The psychometric characteristics of the scale were then formally assessed using a sample of program participants. The implications of the approach for supporting construct validity, inclusion of staff and managers, and theory-driven evaluation are discussed. (Contains 3 tables and 2 figures.)

The emergence of theory-driven evaluation has drawn attention to the need for practical ways for evaluators to construct program theory to achieve the benefits of the approach. This article illustrates the use of concept mapping as a technique to explicate underlying program theory in the context of family support programs. In this example, 29 professional staff members from a statewide family support program brainstormed 96 specific benefits for participant families. Fourteen direct service staff members sorted and rated the items and interpreted the map produced by multidimensional scaling analysis and hierarchical cluster analysis. Some of the benefits of the technique for evaluators conducting theory-driven evaluations are highlighted. These include helping evaluators decide what to measure, capturing the complexity of programs in the context of implementation, and managing good stakeholder-evaluator relations. The general limitations and issues of concept mapping as it applies to theory development are also discussed.

Abstract Globally, health research organizations are called upon to re-examine their policies and practices to more efficiently and effectively address current scientific and social needs, as well as increasing public demands for accountability. Through a case study approach, the authors examine an effort undertaken by the National Institute of Allergy & Infectious Diseases (part of the National Institutes of Health, Department of Health & Human Services, United States Government) to develop an evaluation system for its recently restructured HIV/AIDS clinical trials program. The challenges in designing, operationalizing, and managing global clinical trials programs are considered in the context of large scale scientific research initiatives. Through a process of extensive stakeholder input, a framework of success factors was developed that enables both a prospective view of the elements that must be addressed in an evaluation of this research and a current state assessment of the extent to which the goals of the restructuring are understood by stakeholders across the DAIDS clinical research networks.

Persistent challenges to public health and well-being are being addressed though complex interventions. Complex interventions are intentional processes designed to shift and realign the form and function of targeted system patterns by changing underlying system dynamics, structures, and conditions. The IM40 initiative was a complex health promotion intervention designed to address adolescent health and well-being through a developmental assets approach. The initiative sought to influence several levels simultaneously (e.g. individual, professional, procedural, and policy), compounding the complexity for implementation and evaluation. The purpose of this case study is to present the development and initiation of an evaluation approach based on systems thinking and complexity science principles, concepts, and methods. We used three systems methods, Group Model Building (GMB), viable systems model (VSM) assessment, and Social Network Analysis (SNA) to identify and examine underlying patterns and structures that influenced system-wide behaviors. Each method focused on a general systems-oriented question and illuminated stakeholder views of the IM40 theory of change, adaptive organization of multiple independent institutions, and relational ties among system actors. Collectively, the methods enabled us to describe systems change over time and examine shifts in boundaries, relationships, perspectives, and dynamics. The implications of this approach relative to typical evaluation activities in public health practice are contrasted and discussed. [ABSTRACT FROM AUTHOR]

Establishing a system of care requires communities to identify ways to successfully implement strategies and support positive outcomes for children and their families. Such community transformation is complex and communities vary in terms of their readiness for implementing sustainable community interventions. Assessing community readiness and guiding implementation, specifically for the funded communities implementing a system of care, requires a well-designed tool with sound psychometric properties. This scale development study used the results of a previously published concept mapping study to create, administer, and assess the psychometric characteristics of the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS). The results indicate the SOC-RIMS possesses excellent internal consistency characteristics, measures clearly discernible dimensions of community readiness, and demonstrates the target constructs exist within a broad network of content. The SOC-RIMS can be a useful part of a comprehensive assessment in communities where system of care practices, principles, and philosophies are implemented and evaluated. [ABSTRACT FROM AUTHOR]