Search

Objectives: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. Methods: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. Results: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. Conclusions: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Electronic health records (EHR) interoperability is a complex topic that continues to gain traction in the digital health landscape. We facilitated a qualitative workshop consisting of domain experts in EHR implementation and health IT managers. The workshop aimed to identify critical barriers to achieving interoperability, priorities for new EHR implementations and lessons learned from managing existing implementations. The workshop highlighted that data modelling and interoperability standards are vital priorities for maternal and child health data services in low- and middle-income countries (LMICs).

UNSTRUCTURED There is an urgent need to incorporate theory-informed health information technology evaluation frameworks into existing and emerging guidelines for the evaluation of Artificial Intelligence. Such frameworks can help developers, implementers, and strategic decision makers to build on existing experience and the existing empirical evidence base. We here provide concrete examples on how existing theory-informed health information technology evaluation frameworks may be used to inform the safe implementation of AI in healthcare settings considering technology, design, use by health and care professionals and organisations, and implementation strategy within and across organisations over time.

The health and social care sector has experienced an optimistic turn in the last decade. There has been substantial growth in recent years due to the COVID-19 pandemic that forced the entire sector to identify digital methods of delivering a better level of care than before the pandemic. This paper used the Theory to Change (ToC) approach to demonstrate how the digital skills development of the health and care workforce can be achieved in specific contexts. The paper offers background on digital technologies used in healthcare and outlines the steps and methods used in developing a ToC map. The impact of the proposed ToC approach provides a measurable and predictable way to onboard the health and social care workforce into digital technologies, providing a more digitally skilled and literate workforce.

Computable knowledge artefact development is challenging and often culminates in the development of unique single usage solutions. Libraries of computable knowledge artefacts have the possibility to enhance the Learning Health System in order to improve the benefits of innovation and the decision making of clinicians. This paper aims to discuss the process of creating the use cases and the tutorial material that would enable students to both understand how the interaction between the dataset and the outcome occurs as well as how HL7 Clinical Quality Language can be used to create artefacts of re-usable code.

Computable knowledge artefact development is challenging and often culminates in the development of unique single usage solutions. Libraries of computable knowledge artefacts have the possibility to enhance the Learning Health System in order to improve the benefits of innovation and the decision making of clinicians. This paper aims to discuss the process of creating the use cases and the tutorial material that would enable students to both understand how the interaction between the dataset and the outcome occurs as well as how HL7 Clinical Quality Language can be used to create artefacts of re-usable code.

Objectives: To highlight the role of technology assessment in the management of the COVID-19 pandemic. Method: An overview of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. Results: Evaluation of digital health technologies for COVID-19 should be based on their technical maturity as well as the scale of implementation. For mature technologies like telehealth whose efficacy has been previously demonstrated, pragmatic, rapid evaluation using the complex systems paradigm which accounts for multiple sociotechnical factors, might be more suitable to examine their effectiveness and emerging safety concerns in new settings. New technologies, particularly those intended for use on a large scale such as digital contract tracing, will require assessment of their usability as well as performance prior to deployment, after which evaluation should shift to using a complex systems paradigm to examine the value of information provided. The success of a digital health technology is dependent on the value of information it provides relative to the sociotechnical context of the setting where it is implemented. Conclusion: Commitment to evaluation using the evidence-based medicine and complex systems paradigms will be critical to ensuring safe and effective use of digital health technologies for COVID-19 and future pandemics. There is an inherent tension between evaluation and the imperative to urgently deploy solutions that needs to be negotiated.

Introduction Coronavirus disease 2019 (COVID-19) placed increased burdens on National Health Service hospitals and necessitated significant adjustments to their structures and processes. This research investigated if and how these changes affected the patterns of vital sign recording and staff compliance with expected monitoring schedules on general wards. Methods We compared the pattern of vital signs and early warning score (EWS) data collected from admissions to a single hospital during the initial phase of the COVID-19 pandemic with those in three control periods from 2018, 2019 and 2020. Main outcome measures were weekly and monthly hospital admissions; daily and hourly patterns of recorded vital signs and EWS values; time to next observation and; proportions of ‘on time’, ‘late’ and ‘missed’ vital signs observations sets. Results There were large falls in admissions at the beginning of the COVID-19 era. Admissions were older, more unwell on admission and throughout their stay, more often required supplementary oxygen, spent longer in hospital and had a higher in-hospital mortality compared to one or more of the control periods. More daily observation sets were performed during the COVID-19 era than in the control periods. However, there was no clear evidence that COVID-19 affected the pattern of vital signs collection across the 24-h period or the week. Conclusions The increased burdens of the COVID-19 pandemic, and the alterations in healthcare structures and processes necessary to respond to it, did not adversely affect the hospitals’ ability to monitor patients under its care and to comply with expected monitoring schedules.

National Health Service (NHS) policy suggests that increasing usage of electronic personal health records (PHR) by patients will result in cost savings and improved public health, especially for people with long-term conditions. PHR design features are inevitably important, since a good PHR design should make the users achieve their health goals effortlessly, which is understandable and usable. Three original theoretical models were developed using realist evaluation, one per long-term condition cohort, describing the interaction between the PHR design features and the patient and disease specific factors, to help determine what works for whom in what circumstances.

BACKGROUND Digital health has been a tool of transformation for the delivery of healthcare services globally. An electronic health record (EHR) system can solve the bottlenecks of paper documentation in health service delivery if it is successfully implemented, but poor implementation can lead to a waste of resources. The study of EHR systems implementation in low- and middle-income countries (LMICs) is of particular interest to health stakeholders such as policymakers, funding agencies and care providers due to the efficiencies and evidence base that could result from the appropriate evaluation of such systems. OBJECTIVE We aimed to develop a theory of change (ToC) to assess the feasibility of EHR implementation for the delivery of maternal and child health (MCH) in LMICs. ToC is an outcomes-based approach that starts with the long-term goals and works backwards to the inputs and mediating components required to achieve those goals for complex programmes. METHODS We used the ToC approach to guide the pilot study and identify the preconditions needed to realize the study’s long-term goal in Festac Primary Health Centre (PHC), Lagos, Nigeria. To evaluate the maturity of the implementation, we adapted previously defined success factors to supplement the ToC approach. RESULTS We developed three ToC maps, which showed that the long-term change was the improved MCH service delivery in Festac PHC with the introduction of EHR. The EHR adoption and usage data from the system go-live revealed that the number of registered patients and patient registration forms completed on the EHR system was 1790. Antenatal care (ANC) and immunization encounter forms (n=198 and n=309) were completed. Vital signs (n=325) and visit notes (n=177) were entered into the EHR. We proposed a generic ToC map that LMIC implementers can use to introduce an optimized EHR with the assumptions about sustainability and other relevant factors. The critical success factors were the sustainability, financial and organizational categories. CONCLUSIONS This contribution shows that ToC is a rewarding approach in framing dialogue with stakeholders and serves as a framework for planning, evaluation, learning and reflection. We hypothesized that any future health IT implementation in primary care could adapt our ToC approach to their contexts with necessary modifications based on inherent characteristics.

Background Digital health has been a tool of transformation for the delivery of health care services globally. An electronic health record (EHR) system can solve the bottleneck of paper documentation in health service delivery if it is successfully implemented, but poor implementation can lead to a waste of resources. The study of EHR system implementation in low- and middle-income countries (LMICs) is of particular interest to health stakeholders such as policy makers, funders, and care providers because of the efficiencies and evidence base that could result from the appropriate evaluation of such systems. Objective We aimed to develop a theory of change (ToC) for the implementation of EHRs for maternal and child health care delivery in LMICs. The ToC is an outcomes-based approach that starts with the long-term goals and works backward to the inputs and mediating components required to achieve these goals for complex programs. Methods We used the ToC approach for the whole implementation’s life cycle to guide the pilot study and identify the preconditions needed to realize the study’s long-term goal at Festac Primary Health Centre in Lagos, Nigeria. To evaluate the maturity of the implementation, we adapted previously defined success factors to supplement the ToC approach. Results The initial ToC map showed that the long-term goal was an improved service delivery in primary care with the introduction of EHRs. The revised ToC revealed that the long-term change was the improved maternal and child health care delivery at Festac Primary Health Center using EHRs. We proposed a generic ToC map that implementers in LMICs can use to introduce an optimized EHR system, with assumptions about sustainability and other relevant factors. The outcomes from the critical success factors were sustainability: the sustained improvements included trained health care professionals, a change in mindset from using paper systems toward digital health transformation, and using the project’s laptops to collect aggregate data for the District Health Information System 2–based national health information management system; financial: we secured funding to procure IT equipment, including servers, laptops, and networking, but the initial cost of implementation was high, and funds mainly came from the funding partner; and organizational: the health professionals, especially the head of nursing and health information officers, showed significant commitment to adopting the EHR system, but certain physicians and midwives were unwilling to use the EHR system initially until they were persuaded or incentivized by the management. Conclusions This study shows that the ToC is a rewarding approach to framing dialogue with stakeholders and serves as a framework for planning, evaluation, learning, and reflection. We hypothesized that any future health IT implementation in primary care could adapt our ToC approach to their contexts with necessary modifications based on inherent characteristics.

ObjectiveTo provide an overview of the effects of inter-organisational electronic health records on inpatient diagnosis and treatment decisions by hospital physicians and pharmacists.Materials and MethodsFive-stage scoping review, using distributed cognition and the information value chain as guiding conceptual models. Eligibility criteria: empirical studies addressing how shared health records were used in inpatient clinical decision-making, published 2008-18. Sources: Healthcare Databases Advanced Search, covering nine sources including PubMed. Charting methods: data extraction form completed by one author, with inter-rater reliability assessment at title and abstract review.ResultsQuantitative studies (n=14) often reported relatively low usage of shared records (6.8% to 37.1% of cases). Usage is associated with reduction in diagnostic testing and readmission and variable effects on admissions and overall costs. Qualitative studies (n=6) reported avoidance of duplicate diagnostics, changing clinical decisions, the value of historical laboratory results and optimising the timeliness of care. We found no explicit use of explanatory theoretical models, but there is implicit evidence of an information value chain. We found only one study specifically about pharmacists.DiscussionRelatively low usage is due to clinical judgement whether “extra” data is needed, given current knowledge of the presenting condition and relative complexity. We suggest that extensive EHRs need recommender systems to highlight (sometimes unexpected) relevant content, in parallel with professional guidance on indications for consulting shared records.ConclusionsClinicians only consult shared health records when they must. Mixed effects on process outcomes are due to the hidden variables of patient complexity, clinician judgement and organisational context.

Twenty-four four-year-old children were given a perceptual task (color matching), a memory task (color recognition), and a verbal task (color naming). The relation between the children's performance on the nonverbal and verbal tasks was demonstrated by the fact that nearly all the matching and recognition responses, even when incorrect, were from the same naming class as the stimulus color. Contrary to expectations, this tendency was stronger on the matching task than on the recognition task. Only those children who spontaneously named the colors as they were presented showed this effect on the recognition task. As a result of this experiment, a model is proposed in which perception of colors leads to two distinct encodings: one representational and one symbolic. The two can have different internal histories, but if the symbolic encoding is present at the time of response selection, it will influence the output. However, the symbolic encoding is very short-lived for these young children, and it will be forgotten unless rehearsed overtly. This model is considered in terms of mediation; and it is argued that the non-namers in the recognition task do not suffer from production deficiency, but rather from mediational deficiency, resulting from a lack of memory for the mediator. These conclusions differ from the weak form of the Whorf hypothesis in that two distinct, concurrent encodings are assumed. (DO)

Objectives: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. Method: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. Results: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. Conclusion: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

In a few short weeks, the COVID-19 pandemic radically changed the way health and care services are delivered. The rapid acceleration of digital transformation has been one of the most dramatic changes. Professionals and members of the public have welcomed the benefits that virtual working has

Introduction Since the introduction of the UK’s National Early Warning Score (NEWS) and its modification, NEWS2, coronavirus disease 2019 (COVID-19), has caused a worldwide pandemic. NEWS and NEWS2 have good predictive abilities in patients with other infections and sepsis, however there is little evidence of their performance in COVID-19. Methods Using receiver-operating characteristics analyses, we used the area under the receiver operating characteristic (AUROC) curve to evaluate the performance of NEWS or NEWS2 to discriminate the combined outcome of either death or intensive care unit (ICU) admission within 24 h of a vital sign set in five cohorts (COVID-19 POSITIVE, n = 405; COVID-19 NOT DETECTED, n = 1716; COVID-19 NOT TESTED, n = 2686; CONTROL 2018, n = 6273; CONTROL 2019, n = 6523). Results The AUROC values for NEWS or NEWS2 for the combined outcome were: COVID-19 POSITIVE, 0.882 (0.868−0.895); COVID-19 NOT DETECTED, 0.875 (0.861−0.89); COVID-19 NOT TESTED, 0.876 (0.85−0.902); CONTROL 2018, 0.894 (0.884−0.904); CONTROL 2019, 0.842 (0.829−0.855). Conclusions The finding that NEWS or NEWS2 performance was good and similar in all five cohorts (range = 0.842−0.894) suggests that amendments to NEWS or NEWS2, such as the addition of new covariates or the need to change the weighting of existing parameters, are unnecessary when evaluating patients with COVID-19. Our results support the national and international recommendations for the use of NEWS or NEWS2 for the assessment of acute-illness severity in patients with COVID-19.