Search

Setting: COVID-19 has highlighted the need for credible epidemiological models to inform pandemic policy. Traditional mechanisms of commissioning research are ill-suited to guide policy during a rapidly evolving pandemic. At the same time, contracting with a single centre of expertise has been criticized for failing to reflect challenges inherent in specific modelling approaches., Intervention: This report describes an alternative approach to mobilizing scientific expertise. Ontario's COVID-19 Modelling Consensus Table (MCT) was created in March 2020 to enable rapid communication of credible estimates of the impact of COVID-19 and to accelerate learning on how the disease is spreading and what could slow its transmission. The MCT is a partnership between the province and academic modellers and consists of multiple groups of experts, health system leaders, and senior decision-makers. Armed with Ministry of Health data, the MCT meets once per week to share results from modelling exercises, generate consensus judgements of the likely future impact of COVID-19, and discuss decision-makers' priorities., Outcomes: The MCT has enabled swift access to data for participants, a structure for developing consensus estimates and communicating these to decision-makers, credible models to inform health system planning, and increased transparency in public reporting of COVID-19 data. It has also facilitated the rapid publication of research findings and its incorporation into government policy., Implications: The MCT approach is one way to quickly draw on scientific advice outside of government and public health agencies. Beyond speed, this approach allows for nimbleness as experts from different organizations can be added as needed. It also shows how universities and research institutes have a role to play in crisis situations, and how this expertise can be marshalled to inform policy while respecting academic freedom and confidentiality., (© 2021. The Canadian Public Health Association.)

The impact of policy ambiguity on implementation is a perennial concern in policy circles. The degree of ambiguity of policy goals and the means to achieve them influences the likelihood that a policy will be uniformly understood and implemented across implementation sites. We argue that the application of institutional and organisational theories to policy implementation must be supplemented by a socio-cognitive lens in which stakeholders' interpretations of policy are investigated and compared. We borrow the concept of 'Shared Mental Models' from the literature on industrial psychology to examine the microprocesses of policy implementation. Drawing from interviews with 45 key informants involved in the implementation of a hospital funding reform, known as Quality-Based Procedures in Ontario, Canada, we identify divergent mental models and explain how these divergences may have affected implementation and change management. We close with considerations for future research and practice.

Objective: Radiology is a finite health care resource in high demand at most health centers. However, anticipating fluctuations in demand is a challenge because of the inherent uncertainty in disease prognosis. The aim of this study was to explore the potential of natural language processing (NLP) to predict downstream radiology resource utilization in patients undergoing surveillance for hepatocellular carcinoma (HCC)., Materials and Methods: All HCC surveillance CT examinations performed at our institution from January 1, 2010, to October 31, 2017 were selected from our departmental radiology information system. We used open source NLP and machine learning software to parse radiology report text into bag-of-words and term frequency-inverse document frequency (TF-IDF) representations. Three machine learning models-logistic regression, support vector machine (SVM), and random forest-were used to predict future utilization of radiology department resources. A test data set was used to calculate accuracy, sensitivity, and specificity in addition to the area under the curve (AUC)., Results: As a group, the bag-of-word models were slightly inferior to the TF-IDF feature extraction approach. The TF-IDF + SVM model outperformed all other models with an accuracy of 92%, a sensitivity of 83%, and a specificity of 96%, with an AUC of 0.971., Conclusions: NLP-based models can accurately predict downstream radiology resource utilization from narrative HCC surveillance reports and has potential for translation to health care management where it may improve decision making, reduce costs, and broaden access to care., (Copyright © 2018 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

Purpose: The purpose of this paper is to investigate the relationship between hospital adoption and use of computed tomography (CT) scanners, and magnetic resonance imaging (MRI) machines and in-patient mortality and length of stay., Design/methodology/approach: This study used panel data (2007-2010) from 124 hospital corporations operating in Ontario, Canada. Imaging use focused on medical patients accounting for 25 percent of hospital discharges. Main outcomes were in-hospital mortality rates and average length of stay. A model for each outcome-technology combination was built, and controlled for hospital structural characteristics, market factors and patient characteristics., Findings: In 2010, 36 and 59 percent of hospitals had adopted MRI machines and CT scanners, respectively. Approximately 23.5 percent of patients received CT scans and 3.5 percent received MRI scans during the study period. Adoption of these technologies was associated with reductions of up to 1.1 percent in mortality rates and up to 4.5 percent in length of stay. The imaging use-mortality relationship was non-linear and varied by technology penetration within hospitals. For CT, imaging use reduced mortality until use reached 19 percent in hospitals with one scanner and 28 percent in hospitals with 2+ scanners. For MRI, imaging use was largely associated with decreased mortality. The use of CT scanners also increased length of stay linearly regardless of technology penetration (4.6 percent for every 10 percent increase in use). Adoption and use of MRI was not associated with length of stay., Research Limitations/implications: These results suggest that there may be some unnecessary use of imaging, particularly in small hospitals where imaging is contracted out. In larger hospitals, the results highlight the need to further investigate the use of imaging beyond certain thresholds. Independent of the rate of imaging use, the results also indicate that the presence of CT and MRI devices within a hospital benefits quality and efficiency., Originality/value: To the authors' knowledge, this study is the first to investigate the combined effect of adoption and use of medical imaging on outcomes specific to CT scanners and MRI machines in the context of hospital in-patient care.

Background: Since 2011, the Government of Ontario, Canada, has phased in hospital funding reforms hoping to encourage standardised, evidence-based clinical care processes to both improve patient outcomes and reduce system costs. One aspect of the reform - quality-based procedures (QBPs) - replaced some of each hospital's global budget with a pre-set price per episode of care for patients with specific diagnoses or procedures. The QBP initiative included publication and dissemination of a handbook for each of these diagnoses or procedures, developed by an expert technical group. Each handbook was intended to guide hospitals in reducing inappropriate variation in patient care and cost by specifying an evidence-based episode of care pathway. We explored whether, how and why hospitals implemented these episode of care pathways in response to this initiative., Methods: We interviewed key informants at three levels in the healthcare system, namely individuals who conceived and designed the QBP policy, individuals and organisations supporting QBP adoption, and leaders in five case-study hospitals responsible for QBP implementation. Analysis involved an inductive approach, incorporating framework analysis to generate descriptive and explanatory themes from data., Results: The 46 key informants described variable implementation of best practice episode of care pathways across QBPs and across hospitals. Handbooks outlining evidence-based clinical pathways did not address specific barriers to change for different QBPs nor differences in hospitals' capacity to manage change. Hospitals sometimes found it easier to focus on containing and standardising costs of care than on implementing standardised care processes that adhered to best clinical practices., Conclusion: Implementation of QBPs in Ontario's hospitals depended on the interplay between three factors, namely complexity of changes required, internal capacity for organisational change, and availability and appropriateness of targeted external facilitators and supports to manage change. Variation in these factors across QBPs and hospitals suggests the need for more tailored and flexible implementation supports designed to fit all elements of the policy, rather than one-size-fits-all handbooks alone. Without such supports, hospitals may enact quick fixes aimed mainly at preserving budgets, rather than pursue evidence- and value-based changes in care management. Overestimating hospitals' change management capacity increases the risk of implementation failure.

Background: Methods to measure or quantify equity in health care remain scarce, if not difficult to interpret. A novel method to measure health equity is presented, applied to gender health equity, and illustrated with an example of timing of angiography in patients following a hospital admission for an acute coronary syndrome., Methods: Linked administrative hospital discharge and survey data was used to identify a retrospective cohort of patients hospitalized with Acute Coronary Syndrome (ACS) between 2002 and 2008 who also responded to the Canadian Community Health Survey (CCHS), was analyzed using decision trees to determine whether gender impacted the delay to angiography following an ACS., Results: Defining a delay to angiography as 1 day or more, resulted in a non-significant difference in an equity score of 0.14 for women and 0.12 for men, where 0 and 1 represents perfect equity and inequity respectively. Using 2 and 3 day delays as a secondary outcome resulted in women and men producing scores of 0.19 and 0.17 for a 2 day delay and 0.22 and 0.23 for a 3 day delay., Conclusions: A technique developed expressly for measuring equity suggests that men and women in Ontario receive equitable care in access to angiography with respect to timeliness following an ACS.

Context: As evidence-based medicine grows in influence and scope, its applicability to health policy prompts two questions: Can the principles and, more specifically, the tools used to bring research into the clinical world apply to civil servants offering advice to politicians? If not, what approach should the evidence-oriented health policy organization take to improve the use of research?, Methods: This article reviews evidence-based medicine and models of research use in policy. Then it reports the results of interviews with civil servants in the Ontario Ministry of Health, which recently adopted a stewardship rather than an operational role, incorporating many evidence-oriented strategies. The interviews focused on functional roles for research-based evidence in policy advice., Findings: The clinical context and tools for evidence-based medicine can rarely be generalized to policy. Most current models of research use offer lessons to researchers wishing to apply their work to policy but little help for civil servants wishing to become more evidence oriented. The interviews revealed functional roles for research in setting agendas (noting upcoming issues and screening interest groups' claims), developing new policies (reducing uncertainty, helping speak truth to power, and preventing repetition and duplication), and monitoring or modifying existing policies (continuously improving programs and creating a culture of inquiry). Each area requires different tools to help filter the push of evidence from researchers and set agendas, to facilitate the urgent pull on relevant research by civil servants developing new policy, and to support ongoing linkage and exchange between civil servants and researchers for monitoring and modifying existing policy., Conclusions: A functional framework for evidence-informed policy advice is useful for distinguishing the activity from evidence-based medicine and "auditing" the balance of efforts across the different functional roles of research in policy.

Growing interest in pay-for-performance and the level of chief executive officers' (CEOs') pay raises questions about the link between performance and compensation in the health sector. This study compares the compensation of nonprofit hospital CEOs in Ontario, Canada to the three longest reported and most used measures of hospital financial performance. Our sample consisted of 132 CEOs from 92 hospitals between 1999 and 2006. Unbalanced panel data were analyzed using fixed effects regression. Results suggest that CEO compensation was largely unrelated to hospital financial performance. Inflation-adjusted salaries appeared to increase over time independent of hospital performance, and hospital size was positively correlated with CEO compensation. The apparent upward trend in salary despite some declines in financial performance challenges the fundamental assumption underlying this article, that is, financial performance is likely linked to CEO compensation in Ontario. Further research is needed to understand long-term performance related to compensation incentives.

This study examined relationships between financial indicators for nurse staffing and organizational system integration and change indicators. These indicators, along with hospital location and type, were examined in relation to the nursing financial indicators. Results showed that different indicators predicted each of the outcome variables. Nursing care hours were predicted by the hospital type, geographic location, and the system. Both nursing and patient care hours were significantly related to dissemination and benchmarking of clinical data.

Objective: To compare patient reports about hospital care between western New York State and southern Ontario using a random intercept model., Method: Cross-sectional survey of 3923 patients who received medical or surgical care between August and October 2004 at 28 hospitals (14 hospitals per jurisdiction). Thirty-five questions were combined to calculate eight indicators with scores ranging from 0 to 100 (best care experience). For each indicator, a model was built where the region (western New York vs. southern Ontario) was included as a fixed effect with hospital as random within region. A number of patient characteristics were also included as fixed effects., Results: The effect of the region was statistically significant (P < 0.05) only for the models predicting the 'continuity and transition', 'involvement of family' and 'physical comfort' indicator scores. The differences were 10.66, 4.05 and -3.23 points, respectively. In all three models, the random intercepts were not statistically significant, indicating that the differences above did not vary by hospitals. The model predicting 'overall impression' scores, however, showed a random intercept statistically significant (P = 0.026). The individual-level explained proportion of variance ranged from 5.68 to 11.22%, and the hospital-within-region-level explained proportion of variance ranged from 2.19 to 52.28%., Conclusion: The difference observed on the 'continuity and transition' indicator might be the only one somewhat meaningful, and might be explained by health maintenance organization reimbursements' mechanisms and hospital quality improvement initiatives available in western New York, as well as by the fact that occupancy rates in western New York border the 60% compared with the 95% in southern Ontario.

Background: The Ontario health care system is devolving planning and funding authority to community based organizations and moving from steering through rules and regulations to steering on performance. As part of this transformation, the Ontario Ministry of Health and Long-Term Care (MOHLTC) are interested in using incentives as a strategy to ensure alignment - that is, health service providers' goals are in accord with the goals of the health system. The objective of the study was to develop a decision framework to assist policymakers in choosing and designing effective incentive systems., Methods: The first part of the study was an extensive review of the literature to identify incentives models that are used in the various health care systems and their effectiveness. The second part was the development of policy principles to ensure that the used incentive models are congruent with the values of the Ontario health care system. The principles were developed by reviewing the Ontario policy documents and through discussions with policymakers. The validation of the principles and the suggested incentive models for use in Ontario took place at two meetings. The first meeting was with experts from the research and policy community, the second with senior policymakers from the MOHLTC. Based on the outcome of those two meetings, the researchers built a decision framework for incentives. The framework was send to the participants of both meetings and four additional experts for validation., Results: We identified several models that have proven, with a varying degree of evidence, to be effective in changing or enabling a health provider's performance. Overall, the literature suggests that there is no single best approach to create incentives yet and the ability of financial and non-financial incentives to achieve results depends on a number of contextual elements. After assessing the initial set of incentive models on their congruence with the four policy principles we defined nine incentive models to be appropriate for use in Ontario and potentially other health care systems that want to introduce incentives to improve performance. Subsequently, the models were incorporated in the resulting decision framework., Conclusion: The design of an incentive must reflect the values and goals of the health care system, be well matched to the performance objectives and reflect a range of contextual factors that can influence the effectiveness of even well-designed incentives. As a consequence, a single policy recommendation around incentives is inappropriate. The decision framework provides health care policymakers and purchasers with a tool to support the selection of an incentive model that is the most appropriate to improve the targeted performance.

Objective: Patient involvement in health service planning and evaluation is considered important yet not widely practiced. This study explored stakeholder beliefs about patient participation in performance indicator selection to better understand hypothesized barriers., Methods: Interviews with 30 cancer patients and health professionals from two teaching hospitals were analyzed qualitatively., Results: All groups believed patients, not members of the public, should be involved in the selection of indicators. Ongoing, interactive methods such as committee involvement, rather than single, passive efforts such as surveys were preferred. Health professionals recommended patients assume a consultative, rather than decision-making role. Older patients agreed with this., Conclusion: Variable patient interest, health professional attitudes, and a lack of insight on appropriate methods may be limiting patient involvement in this, and other service planning and evaluation activities. More research is required to validate expressed views among the populations these stakeholders represent, and to establish effective methods for engaging patients., Practice Implications: Efforts to encourage a change in health professional attitude may be required, along with dedicated organizational resources, coordinators and training. Methods to engage patients should involve deliberation, which can be achieved through modified Delphi panel or participatory research approaches.

Objective: This study outlines predictors of cancer patients' overall perceptions of the quality of care., Design and Setting: Our sample included 2790 patients who received cancer care services during 2004 in 15 comprehensive cancer programmes across Ontario, Canada. Patients were classified into three groups: those receiving both chemotherapy and radiotherapy (n = 752), those receiving only chemotherapy (n = 1044), and those receiving only radiotherapy (n = 994). An ordinal logistic regression model for each patient group was performed to determine which variables most affected the probabilities of the patients' overall evaluations of the quality of care. Potential control variables were patients' age, sex, type of cancer, self-assessed health, and who completed the survey., Results: Among seven common predictors of the overall quality perception across the three models, four should be of particular interest because patients perceived them as relatively problematic aspects of care. These are 'was informed about follow-up care after completing treatment', 'knew next step in care', 'knew who to go to with questions', and 'providers were aware of test results'. These predictors explained between 25 and 34% of the variance (depending on the model) of the overall perception of quality. The explanatory power of these predictors did not change across sex and age group. 'Self-assessed health' was the only control variable that remained in all three models., Conclusions: From a practical perspective, improvement efforts are best focused on factors that are strong predictors as well as on those for which there is a low score. Thus, on the basis of this study, practitioners' improvement efforts might be constructively focused on the four predictors mentioned above.

This article surveyed the processes and policies underlying the routine collection of clinical data in acute care hospitals in Ontario, Canada. Although there is evidence of a small shortfall in the availability of human resources, most health records departments employ experienced staff with health records certification. However, there is much more important variation in the documented and undocumented processes used to generate routinely collected clinical data. Current guidelines and coding schedules are helpful but insufficient to guide the production of good quality data. The variations in the processes used to produce clinical data have important implications for the management, reimbursement, and planning of healthcare. This is particularly critical at a time when hospitals and other stakeholders, such as governments, are relying more and more on accurate, reliable, and comparable data.

Background: The goal of this study was to identify aspects of care (predictors) that can most easily be modified to produce an improvement in the score of patients' overall evaluations of the quality of care received., Patients and Methods: Our sample consisted of 2247 cancer patients hospitalized in Ontario acute care hospitals in 1999/2000. We sought predictors of patients' overall perceptions of the quality of care by applying a methodology that minimizes the improvement of the predictors while gaining a desired increase in the score of the dependent variable. This approach tends to ignore items that rate relatively high and focuses on those for which hospitals can more easily modify the score. Two main subgroups were analyzed in this study: patients with malignant and benign neoplasms., Results: 'Skills of nursing staff', 'courtesy of nursing staff', 'courtesy of people who drew blood' and 'cleanliness of hospital in general' were consistently selected by our approach in both cancer groups., Conclusions: This study identifies an efficient approach to improving the score of patients' overall perceptions of the quality of care received. By focusing on these aspects of care, hospitals may be able to improve the allocation of scarce resources when planning patient satisfaction improvement initiatives.

Since the inception of the Hospital Reports in 1998, researchers have focused on three separate but related problems--how to measure performance; how to evaluate performance; and how to transfer knowledge about excellent performance to the field. This article describes a method to address the second problem--how to evaluate performance by benchmarking two indicators of financial performance and condition through three years of recent data for Ontario hospitals.

The dominant system-level strategic priorities facing Ontario hospitals were elicited and validated. Researchers employed a multi-stage survey and focus group process to solicit the opinions of senior hospital managers and other healthcare stakeholders. Four shared, system-level priorities emerged: (i) improved clinical and staff recruitment; (ii) stronger inter-hospital partnership and vertical integration along the continuum of care; (iii) improved patient safety; and (iv) the implementation of decision support systems. A subsequent CEO validation survey showed strong endorsement of these system priorities. The authors conclude that a survey, focus group, and validation process can reveal shared system priorities and can highlight emergent organizational strategies designed to resolve them. This process, in which a hospital learns of the priorities facing other hospitals in a system, is a potentially useful managerial tool to promote "strategic synapse"-whereby management can adjust internal organizational strategy and local scorecards to align with shared system priorities.

Background: Little performance measurement has been undertaken in the area of oncology, particularly for surgery, which is a pivotal event in the continuum of cancer care. This work was conducted to develop indicators of quality for colorectal cancer surgery, using a 3-step modified Delphi approach., Methods: A multidisciplinary panel, comprising surgical and methodological co-chairs, 9 surgeons, a medical oncologist, a radiation oncologist, a nurse and a pathologist, reviewed potential indicators extracted from the medical literature through 2 consecutive rounds of rating followed by consensus discussion. The panel then prioritized the indicators selected in the previous 2 rounds., Results: Of 45 possible indicators that emerged from 30 selected articles, 15 were prioritized by the panel as benchmarks for assessing the quality of surgical care. The 15 indicators represent 3 levels of measurement (provincial/regional, hospital, individual provider) across several phases of care (diagnosis, surgery, adjuvant therapy, pathology and follow-up), as well as broad measures of access and outcome. The indicators selected by the panel were more often supported by evidence than those that were discarded., Conclusions: This project represents a unique initiative, and the results may be applicable to colorectal cancer surgery in any jurisdiction.

Study Objective: Patient satisfaction is an important performance measure for emergency departments (EDs), but the most efficient ways of improving satisfaction are unclear. This study uses optimization techniques to identify the best possible combination of predictors of overall patient satisfaction to help guide improvement efforts., Methods: The results of a satisfaction survey from 20,500 patients who visited 123 EDs were used to develop ordinal logistic regression models for overall quality of care, overall medical treatment, willingness to recommend the ED to others, and willingness to return to the same ED. Originally, 68,981 surveys were mailed, and 20,916 were returned, representing an overall response rate of 30.3%. We then incorporated these regressions into an optimization model to select the most efficient combination of predictors necessary to increase the 4 overall satisfaction measures by 5%. A sensitivity analysis was also conducted to explore differences across hospital peer groups and regions., Results: Results differ slightly for each of the 4 overall satisfaction measures. However, 4 predictors were common to all of these measures: "perceived waiting time to receive treatment," "courtesy of the nursing staff," "courtesy of the physicians," and "thoroughness of the physicians." The selected predictors were not necessarily the strongest predictors identified through regression models. The optimization model suggests that most of these predictors must be improved by 15% to increase the overall satisfaction measures by 5%., Conclusion: This study introduces the use of optimization techniques to study ED patient satisfaction and highlights an opportunity to apply this technique to widely collected data to help inform hospitals' improvement strategies. The results suggest that hospitals should focus most of their improvement efforts on the 4 predictors mentioned above.

Objectives: To develop a set of scientifically sound and managerially useful system-level cancer care performance indicators for public reporting in Ontario, Canada., Implementation: Using a modified Delphi panel method, comprising a systematic literature review and multiple rounds of structured feedback from 34 experts, the Cancer Quality Council of Ontario developed a set of quality indicators spanning cancer prevention through to end-of-life care. To be useful to decision-makers and providers, indicator selection criteria included a clear focus on the cancer system, relevance to a diversity of cancer providers, a strong link to the mission and strategic objectives of the cancer system, clear directionality of indicator results, presence of targets and/or benchmarks, feasibility of populating the indicator, and credibility of the measure as an indicator of quality. To ensure that the selected indicators would measure progress over time against specific and widely accepted goals, we created a strategy map based on the five strategic objectives of the Ontario cancer system: (i) to improve the measurement and reporting of cancer quality, (ii) to increase the use of evidence and innovation in decision-making, (iii) to improve access to cancer services and reduce waiting times, (iv) to increase efficiency across the system, (v) to reduce the burden of cancer. An analysis of the mean indicator ratings by experts, and the strategy mapping exercise resulted in the identification of 36 indicators deemed suitable for routine performance measurement of the Ontario cancer system., Lessons Learned: The resulting instrument incorporates a credible evidence basis for performance measurement aligned to the five strategic goals for the Ontario cancer system. It represents the integrating of a management culture, focused on the implementation of a new strategic direction for the cancer system, with the underlying evidence-based culture of clinicians.

To explore the current and pending strategic agenda of Ontario hospitals (the largest consumers of the provincial healthcare budget), a survey of Ontario acute care hospital CEOs was conducted in January 2004. The survey, with an 82% response rate, identifies 29 strategic priorities under seven key strategic themes consistent across different hospital types. These themes include (1) human resources cultivation, (2) service integration and partnerships, (3) consumer engagement, (4) corporate governance and management, (5) organizational efficiency and redesign, (6) improved information use for decision-making, (7) patient care management. The extent to which an individual hospital's control over strategic resolutions is perceived may affect multilevel strategic priority-setting and action-planning. In addition to supporting ongoing development of meaningful performance measures and information critical to strategic decision-making, this study's findings may facilitate a better understanding of hospitals' key resource commitments, the extent of competition and collaboration for key resources, the perceived degree of individual control over strategic issue resolution and where systemic resolutions may be required.

Public performance reporting is an increasingly common health policy tool to support accountability and quality improvement but there are few formal evaluations of this trend. In this review, we suggest that performance reporting may be an effective way to support improvements in performance when it is directed towards groups of providers rather than individuals and that there is enough evidence to support the use and further development of public performance reporting. However, the true impact of performance reporting depends on the policy context around reporting including clarity of strategy, incentives, and support for performance improvement.

The Women's Health Report was a preliminary study that formed part of the Hospital Report 2001 Series; an evaluation of hospital performance in the province of Ontario using a balanced scorecard framework. The Report reflects the importance of including a women's health perspective in the evaluation of health system performance and the need for increased accountability for equity in access to, and delivery of, care. This paper outlines the methodology that was used to develop the Women's Health Report and highlights important findings from this early stage in the development of performance measurement for hospital care for women in Canada.