Search

There has been considerable growth in the development of machine learning algorithms for clinical applications. The authors survey recent machine learning models developed with the use of large health administrative databases at ICES and highlight three areas of ongoing development that are particularly important for health system applications.

The COVID-19 pandemic has demonstrated the need for real-time, open-access epidemiological information to inform public health decision-making and outbreak control efforts. In Canada, authority for healthcare delivery primarily lies at the provincial and territorial level; however, at the outset of the pandemic no definitive pan-Canadian COVID-19 datasets were available. The COVID-19 Canada Open Data Working Group was created to fill this crucial data gap. As a team of volunteer contributors, we collect daily COVID-19 data from a variety of governmental and non-governmental sources and curate a line-list of cases and mortality for all provinces and territories of Canada, including information on location, age, sex, travel history, and exposure, where available. We also curate time series of COVID-19 recoveries, testing, and vaccine doses administered and distributed. Data are recorded systematically at a fine sub-national scale, which can be used to support robust understanding of COVID-19 hotspots. We continue to maintain this dataset, and an accompanying online dashboard, to provide a reliable pan-Canadian COVID-19 resource to researchers, journalists, and the general public., Measurement(s) COVID-19 Cases • COVID-19 Mortalities • COVID-19 vaccination Technology Type(s) digital curation Sample Characteristic - Organism Homo sapiens Sample Characteristic - Location Canada Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.14331311

BACKGROUND Public health professionals regularly engage in complex tasks involving data and models on large population segments, and use various tools to help support their data-driven decision-making. Human factors methods can be employed in the design of such tools to better support public health professionals in decision-making tasks. OBJECTIVE While human factors methods have been applied to the design of some individually-based clinical health tools, their applications are limited in the design of public health systems. We conducted focus groups with public health professionals in Ontario, Canada to understand their informational needs in decision support tools to support a human factors approach to designing data-based tools for public health. METHODS Three focus groups were completed, each consisting of a group of four participants. The discussion included a critique of pre-existing public health tools and their interfaces, a discussion of decision support tools and population risk tools familiar to the participants, and sought out insights on developing a chronic disease prediction dashboard as a use case. RESULTS Data was recorded, transcribed, and coded by three independent reviewers to extract themes using affinity diagramming. We present system features identified by public health professionals, and provide an example of how to use and prioritize these suggested features by applying them to our use case. CONCLUSIONS Focus groups revealed that users want a tool that identifies the end user early in the design process, has quality visualizations, provides transparency in data inputs and outputs, and incorporates equity considerations. This research provides a critical perspective on the potential for human factors to improve the use of data and analytics for public health applications.

Estimates in a 1989 study indicated that physicians in the United States were unable to reach a diagnosis that accounted for their patient's symptoms in up to 90% of outpatient patient encounters. Many proponents of artificial intelligence (AI) see the current process of moving from clinical data gathering to medical diagnosis as being limited by human analytic capability and expect AI to be a valuable tool to refine this process. The use of AI fundamentally calls into question the extent to which uncertainty in medical decision making is tolerated. Uncertainty is perceived by some as fundamentally undesirable and thus, for them, optimal decision making should be based on minimizing uncertainty. However, uncertainty cannot be reduced to zero; thus, relative uncertainty can be used as a metric to weigh the likelihood of various diagnoses being correct and the appropriateness of treatments. Here, the authors make the argument, using as examples the experiences of 2 AI systems, IBM Watson on Jeopardy and Watson for Oncology, that medical decision making based on relative uncertainty provides a better lens for understanding the application of AI to medicine than one that minimizes uncertainty. This approach to uncertainty has significant implications for how health care leaders consider the benefits and trade-offs of AI-assisted and AI-driven decision tools and ultimately integrate AI into medical practice.

Purpose of review Machine learning (ML), a branch of artificial intelligence, is influencing all fields in medicine, with an abundance of work describing its application to adult practice. ML in pediatrics is distinctly unique with clinical, technical, and ethical nuances limiting the direct translation of ML tools developed for adults to pediatric populations. To our knowledge, no work has yet focused on outlining the unique considerations that need to be taken into account when designing and implementing ML in pediatrics. Recent findings The nature of varying developmental stages and the prominence of family-centered care lead to vastly different data-generating processes in pediatrics. Data heterogeneity and a lack of high-quality pediatric databases further complicate ML research. In order to address some of these nuances, we provide a common pipeline for clinicians and computer scientists to use as a foundation for structuring ML projects, and a framework for the translation of a developed model into clinical practice in pediatrics. Throughout these pathways, we also highlight ethical and legal considerations that must be taken into account when working with pediatric populations and data. Summary Here, we describe a comprehensive outline of special considerations required of ML in pediatrics from project ideation to implementation. We hope this review can serve as a high-level guideline for ML scientists and clinicians alike to identify applications in the pediatric setting, generate effective ML solutions, and subsequently deliver them to patients, families, and providers.

Objective: To predict older adults’ risk of avoidable hospitalisation related to ambulatory care sensitive conditions (ACSC) using machine learning applied to administrative health data of Ontario, Canada. Design, setting and participants: A retrospective cohort study was conducted on a large cohort of all residents covered under a single-payer system in Ontario, Canada over the period of 10 years (2008– 2017). The study included 1.85 million Ontario residents between 65 and 74 years old at any time throughout the study period. Data sources: Administrative health data from Ontario, Canada obtained from the (ICES formerly known as the Institute for Clinical Evaluative Sciences Data Repository. Main outcome measures: Risk of hospitalisations due to ACSCs 1 year after the observation period. Results: The study used a total of 1 854 116 patients, split into train, validation and test sets. The ACSC incidence rates among the data points were 1.1% for all sets. The final XGBoost model achieved an area under the receiver operating curve of 80.5% and an area under precision–recall curve of 0.093 on the test set, and the predictions were well calibrated, including in key subgroups. When ranking the model predictions, those at the top 5% of risk as predicted by the model captured 37.4% of those presented with an ACSC-related hospitalisation. A variety of features such as the previous number of ambulatory care visits, presence of ACSC-related hospitalisations during the observation window, age, rural residence and prescription of certain medications were contributors to the prediction. Our model was also able to capture the geospatial heterogeneity of ACSC risk in Ontario, and especially the elevated risk in rural and marginalised regions. Conclusions: This study aimed to predict the 1-year risk of hospitalisation from ambulatory-care sensitive conditions in seniors aged 65–74 years old with a single, large-scale machine learning model. The model shows the potential to inform population health planning and interventions to reduce the burden of ACSC-related hospitalisations. Published version This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). This work was supported by the New Frontiers in Research Fund (NFRFE2018-00662), a Canada Research Chair in Population Health Analytics (950- 230702) (LR), Ontario Graduate Scholarship (number N/A) (VH), Canadian Institutes of Health Research Banting and Best Canada Graduate Scholarship Master’s and Doctoral awards (numbers N/A) (VH), and Vector Institute Post-graduate Fellowship (number N/A) (VH).

Background As rates of international travel increase, more individuals are at risk of travel-acquired infections (TAIs). We aimed to review all microbiologically confirmed cases of malaria, dengue, chikungunya, and enteric fever (Salmonella enterica serovar Typhi/Paratyphi) in Ontario, Canada between 2008-2020 to identify high-resolution geographical clusters that could be targeted for pre-travel prevention. Methods Retrospective cohort study of over 174,000 unique tests for the four above TAIs from Public Health Ontario Laboratories. Test-level data were processed to calculate annual case counts and crude population-standardized incidence ratios (SIRs) at the forward sortation area (FSA) level. Moran’s I statistic was used to test for global spatial autocorrelation. Smoothed SIRs and 95% posterior credible intervals (CIs) were estimated using a spatial Bayesian hierarchical model, which accounts for statistical instability and uncertainty in small-area incidence. Posterior CIs were used to identify high- and low-risk areas, which were described using sociodemographic data from the 2016 Census. Finally, a second model was used to estimate the association between drivetime to the nearest travel clinic and risk of TAI within high-risk areas. Results There were 5962 cases of the four TAIs across Ontario over the study period. Smoothed FSA-level SIRs are shown in Figure 1a, with an inset for the Greater Toronto Area (GTA) in 1b. There was spatial clustering of TAIs (Moran’s I=0.61, p< 2.2e-16). Identified high- and low-risk areas are shown in panels c and d. Compared to low-risk areas, high-risk areas were significantly more likely to have higher proportions of immigrants (p< 0.0001), lower household after-tax income (p=0.04), more university education (p< 0.0001), and were less knowledgeable of English/French (p< 0.0001). In the high-risk GTA, each minute increase in drivetime to the closest travel clinic was associated with a 4% reduction in TAI risk (95% CI 2 - 6%). Bayesian hierarchical model (BHM) smoothed standardized incidence ratios (SIRs) for travel-acquired infections (TAIs) and estimated risk levels (a and c) with insets for the Greater Toronto Area (b and d). High-risk areas are defined as those with smoothed SIR 95% CIs greater than 2, and low-risk areas with smoothed SIR 95% CIs less than 0.25. Conclusion Urban neighbourhoods in the GTA had elevated risks of becoming ill with TAIs. However, geographic proximity to a travel clinic was not associated with an area-level risk reduction in TAI, suggesting other barriers to seeking and adhering to pre-travel advice. Disclosures Isaac Bogoch, MD, MSc, BlueDot (Consultant)National Hockey League Players' Association (Consultant) Andrea Boggild, MSc MD DTMH FRCPC, Nothing to disclose Shaun Morris, MD, MPH, DTM&H, FRCPC, FAAP, GSK (Speaker's Bureau)Pfizer (Advisor or Review Panel member)Pfizer (Grant/Research Support)

Key Points Question Can a machine learning model trained on routinely collected administrative health data be used to accurately predict the onset of type 2 diabetes at the population level? Findings In this decision analytical model study of 2.1 million residents in Ontario, Canada, a machine learning model was developed with high discrimination, population-level calibration, and calibration across population subgroups. Meaning Study results suggest that machine learning and administrative health data can be used to create population health planning tools that accurately discriminate between high- and low-risk groups to guide investments and targeted interventions for diabetes prevention., This decision analytical model study assesses whether a machine learning model trained on routinely collected administrative health data from a single-payer health system in Canada can be used to predict the onset of type 2 diabetes in the population., Importance Systems-level barriers to diabetes care could be improved with population health planning tools that accurately discriminate between high- and low-risk groups to guide investments and targeted interventions. Objective To develop and validate a population-level machine learning model for predicting type 2 diabetes 5 years before diabetes onset using administrative health data. Design, Setting, and Participants This decision analytical model study used linked administrative health data from the diverse, single-payer health system in Ontario, Canada, between January 1, 2006, and December 31, 2016. A gradient boosting decision tree model was trained on data from 1 657 395 patients, validated on 243 442 patients, and tested on 236 506 patients. Costs associated with each patient were estimated using a validated costing algorithm. Data were analyzed from January 1, 2006, to December 31, 2016. Exposures A random sample of 2 137 343 residents of Ontario without type 2 diabetes was obtained at study start time. More than 300 features from data sets capturing demographic information, laboratory measurements, drug benefits, health care system interactions, social determinants of health, and ambulatory care and hospitalization records were compiled over 2-year patient medical histories to generate quarterly predictions. Main Outcomes and Measures Discrimination was assessed using the area under the receiver operating characteristic curve statistic, and calibration was assessed visually using calibration plots. Feature contribution was assessed with Shapley values. Costs were estimated in 2020 US dollars. Results This study trained a gradient boosting decision tree model on data from 1 657 395 patients (12 900 257 instances; 6 666 662 women [51.7%]). The developed model achieved a test area under the curve of 80.26 (range, 80.21-80.29), demonstrated good calibration, and was robust to sex, immigration status, area-level marginalization with regard to material deprivation and race/ethnicity, and low contact with the health care system. The top 5% of patients predicted as high risk by the model represented 26% of the total annual diabetes cost in Ontario. Conclusions and Relevance In this decision analytical model study, a machine learning model approach accurately predicted the incidence of diabetes in the population using routinely collected health administrative data. These results suggest that the model could be used to inform decision-making for population health planning and diabetes prevention.

[ ]we must seek and use data for precision and population-level interventions 6 Accurate, timely, and actionable data are necessary both to understand the extent of issues threatening community health and to measure the effectiveness of solutions for health promotion and protection [ ]holistically addressing health emergencies requires collaboration across disciplines [ ]as the next generation of public health professionals, we must not hesitate to champion our communities in the political arena

ObjectiveTo predict older adults’ risk of avoidable hospitalization related to ambulatory care sensitive conditions (ACSC) using machine learning applied to administrative health data of Ontario, Canada.Design, Setting, and ParticipantsA retrospective cohort study was conducted on a large cohort of all residents covered under a single-payer system in Ontario, Canada over the period of 10 years, between 2008 and 2017. The study included 1.85 million Ontario residents between 65 and 74 years old at any time throughout the study period.Data sourcesAdministrative health data from Ontario, Canada obtained from the ICES Data Repository.Main outcome measuresRisk of hospitalizations due to ACSCs one year after the observation period.ResultsThe study used a total of 1,854,116 patients, split into train, validation, and test sets. The ACSC incidence rates among the data points were 1.1% for all sets. The final XGBoost model achieved an AUC of 80.5% on the held-out test set, and the predictions were well-calibrated. When ranking the predictions made by the model, those at the top 5% of risk as predicted by the model captured 37.4% of those presented with an ACSC-related hospitalization. A variety of features such as the previous number of ambulatory care visits, presence of ACSC-related hospitalizations during the observation window, age, rural residence, and prescription of certain medications were contributors to the prediction. Our model was also able to capture the geospatial heterogeneity of ACSC risk in the province of Ontario, and especially the elevated risk in rural and marginalized regions.ConclusionsThis study aimed to predict the 1-year risk of hospitalization from a series of ambulatory-care sensitive conditions in seniors aged 65 to 74 years old with a single, large-scale machine learning model. The model shows the potential to inform population health planning and interventions to reduce the burden of ACSC-related hospitalizations.

BackgroundThere has been growing acknowledgement that undergraduate medical education (UME) must play a formal role in instructing future physicians on the promises and limitations of artificial intelligence (AI), as these tools are integrated into medical practice.MethodsWe conducted an exploratory survey of medical students’ knowledge of AI, perceptions on the role of AI in medicine, and preferences surrounding the integration of AI competencies into medical education. The survey was completed by 321 medical students (13.4% response rate) at four medical schools in Ontario.ResultsMedical students are generally optimistic regarding AI’s capabilities to carry out a variety of healthcare functions, from clinical to administrative, with reservations about specific task types such as personal counselling and empathetic care. They believe AI will raise novel ethical and social challenges. Students are concerned about how AI will affect the medical job market, with 25% responding that it was actively impacting their choice of specialty. Students agree that medical education must do more to prepare them for the impact of AI in medicine (79%), and the majority (68%) believe that this training should begin at the UME level.ConclusionsMedical students expect AI will be widely integrated into healthcare and are enthusiastic to obtain AI competencies in undergraduate medical education.

Across jurisdictions, government and health insurance providers hold a large amount of data from patient interactions with the healthcare system. We aimed to develop a machine learning-based model for predicting adverse outcomes due to diabetes complications using administrative health data from the single-payer health system in Ontario, Canada. A Gradient Boosting Decision Tree model was trained on data from 1,029,366 patients, validated on 272,864 patients, and tested on 265,406 patients. Discrimination was assessed using the AUC statistic and calibration was assessed visually using calibration plots overall and across population subgroups. Our model predicting three-year risk of adverse outcomes due to diabetes complications (hyper/hypoglycemia, tissue infection, retinopathy, cardiovascular events, amputation) included 700 features from multiple diverse data sources and had strong discrimination (average test AUC = 77.7, range 77.7–77.9). Through the design and validation of a high-performance model to predict diabetes complications adverse outcomes at the population level, we demonstrate the potential of machine learning and administrative health data to inform health planning and healthcare resource allocation for diabetes management.

Non-pharmaceutical interventions (NPIs) have been the primary tool used by governments and organizations to mitigate the spread of the ongoing pandemic of COVID-19. Natural experiments are currently being conducted on the impact of these interventions, but most of these occur at the subnational level - data not available in early global datasets. We describe the rapid development of the first comprehensive, labelled dataset of 1640 NPIs implemented at federal, provincial/territorial and municipal levels in Canada to guide COVID-19 research. For each intervention, we provide: a) information on timing to aid in longitudinal evaluation, b) location to allow for robust spatial analyses, and c) classification based on intervention type and target population, including classification aligned with a previously developed measure of government response stringency. This initial dataset release (v1.0) spans January 1st, and March 31st, 2020; bi-weekly data updates to continue for the duration of the pandemic. This novel dataset enables robust, inter-jurisdictional comparisons of pandemic response, can serve as a model for other jurisdictions and can be linked with other information about case counts, transmission dynamics, health care utilization, mobility data and economic indicators to derive important insights regarding NPI impact.

PURPOSE: Electromagnetic tracking is used in image-­‐guided interventions to monitor the position of surgical equipment. However, it is prone to error. During navigation procedures, the measurement and visualization of error should take place to ensure precision and accuracy. Our goal was to extend open-­‐source software such that no programming from the user is needed for error monitoring. METHODS: The electromagnetic tracking error was defined as the difference in position readings of a surgical stylus that was optically and electromagnetically tracked. The optical tracking reading was considered ground truth as it is unaffected by metal objects. The stylus was moved freehandedly within a region of interest to quickly sample electromagnetic tracking error, including error caused by field-­‐distortion inducing ferromagnetic materials and random noise. All tracked devices were used in a plug-­‐and-­‐play manner. Measurements were visualized in real-­‐time in 3D Slicer, an open-­‐ source platform for medical image computing (www.slicer.org). RESULTS: To determine the reproducibility of the data collected by our system, the tracking error was measured in regions of interest representing a surgeon’s workspace. Tests were done with and without metal objects placed in the workspace. A quick freehand sampling procedure was sufficient to detect error. Freehand measurements within a controlled environment reported distortion values of 1.16 mm (STD 0.71 mm). Measurements taken with a metal rod and parts from a surgical retractor kit were 4.27 mm (STD 2.92 mm) and 8.26 mm (STD 2.08 mm), respectively. CONCLUSION: Our system shows promise for the real-­‐time visualization of electromagnetic tracking error during computer-­‐assisted surgical procedures

PURPOSE: The intraoperative measurement of tracking error is crucial to ensure the reliability of electromagnetically navigated procedures. For intraoperative use, methods need to be quick to set up, easy to interpret, and not interfere with the ongoing procedure. Our goal was to evaluate the feasibility of using redundant electromagnetic sensors to alert users to tracking error in a navigated intervention setup. METHODS: Electromagnetic sensors were fixed to a rigid frame around a region of interest and on surgical tools. A software module was designed to detect tracking error by comparing real-time measurements of the differences between inter-sensor distances and angles to baseline measurements. Once these measurements were collected, a linear support vector machine-based classifier was used to predict tracking errors from redundant sensor readings. RESULTS: Measuring the deviation in the reported inter-sensor distance and angle between the needle and cautery served as a valid indicator for electromagnetic tracking error. The highest classification accuracy, 86%, was achieved based on readings from the cautery when the two sensors on the cautery were close together. The specificity of this classifier was 93% and the sensitivity was 82%. CONCLUSION: Placing redundant electromagnetic sensors in a workspace seems to be feasible for the intraoperative detection of electromagnetic tracking error in controlled environments. Further testing should be performed to optimize the measurement error threshold used for classification in the support vector machine, and improve the sensitivity of our method before application in real procedures.

PURPOSE: Optical pose tracking of medical instruments is often used in image-guided interventions. Unfortunately, compared to commonly used computing devices, optical trackers tend to be large, heavy, and expensive devices. Compact 3D vision systems, such as Intel RealSense cameras can capture 3D pose information at several magnitudes lower cost, size, and weight. We propose to use Intel SR300 device for applications where it is not practical or feasible to use conventional trackers and limited range and tracking accuracy is acceptable. We also put forward a vertebral level localization application utilizing the SR300 to reduce risk of wrong-level surgery. METHODS: The SR300 was utilized as an object tracker by extending the PLUS toolkit to support data collection from RealSense cameras. Accuracy of the camera was tested by comparing to a high-accuracy optical tracker. CT images of a lumbar spine phantom were obtained and used to create a 3D model in 3D Slicer. The SR300 was used to obtain a surface model of the phantom. Markers were attached to the phantom and a pointer and tracked using Intel RealSense SDK’s built-in object tracking feature. 3D Slicer was used to align CT image with phantom using landmark registration and display the CT image overlaid on the optical image. RESULTS: Accuracy of the camera yielded a median position error of 3.3mm (95th percentile 6.7mm) and orientation error of 1.6° (95th percentile 4.3°) in a 20x16x10cm workspace, constantly maintaining proper marker orientation. The model and surface correctly aligned demonstrating the vertebral level localization application. CONCLUSION: The SR300 may be usable for pose tracking in medical procedures where limited accuracy is acceptable. Initial results suggest the SR300 is suitable for vertebral level localization.

PURPOSE: The measurement of tracking error is crucial to ensure the safety and feasibility of electromagnetically tracked, image-guided procedures. Measurement should occur in a clinical environment because electromagnetic field distortion depends on positioning relative to the field generator and metal objects. However, we could not find an accessible and open-source system for calibration, error measurement, and visualization. We developed such a system and tested it in a navigated breast surgery setup. METHODS: A pointer tool was designed for concurrent electromagnetic and optical tracking. Software modules were developed for automatic calibration of the measurement system, real-time error visualization, and analysis. The system was taken to an operating room to test for field distortion in a navigated breast surgery setup. Positional and rotational electromagnetic tracking errors were then calculated using optical tracking as a ground truth. RESULTS: Our system is quick to set up and can be rapidly deployed. The process from calibration to visualization also only takes a few minutes. Field distortion was measured in the presence of various surgical equipment. Positional and rotational error in a clean field was approximately 0.90 mm and 0.31°. The presence of a surgical table, an electrosurgical cautery, and anesthesia machine increased the error by up to a few tenths of a millimeter and tenth of a degree. CONCLUSION: In a navigated breast surgery setup, measurement and visualization of tracking error defines a safe working area in the presence of surgical equipment. Our system is available as an extension for the open-source 3D Slicer platform.

Electromagnetic tracking allows for increased flexibility in designing image-guided interventions, however it is well understood that electromagnetic tracking is prone to error. Visualization and assessment of the tracking error should take place in the operating room with minimal interference with the clinical procedure. The goal was to achieve this ideal in an open-source software implementation in a plug and play manner, without requiring programming from the user. We use optical tracking as a ground truth. An electromagnetic sensor and optical markers are mounted onto a stylus device, pivot calibrated for both trackers. Electromagnetic tracking error is defined as difference of tool tip position between electromagnetic and optical readings. Multiple measurements are interpolated into the thin-plate B-spline transform visualized in real time using 3D Slicer. All tracked devices are used in a plug and play manner through the open-source SlicerIGT and PLUS extensions of the 3D Slicer platform. Tracking error was measured multiple times to assess reproducibility of the method, both with and without placing ferromagnetic objects in the workspace. Results from exhaustive grid sampling and freehand sampling were similar, indicating that a quick freehand sampling is sufficient to detect unexpected or excessive field distortion in the operating room. The software is available as a plug-in for the 3D Slicer platforms. Results demonstrate potential for visualizing electromagnetic tracking error in real time for intraoperative environments in feasibility clinical trials in image-guided interventions.