Search

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Lung cancer screening with low-dose computed tomography (CT) could help avert thousands of deaths each year. Since the implementation of screening is complex and underspecified, there is a need for systematic and theory-based strategies. Explore the implementation of lung cancer screening in primary care, in the context of integrating a decision aid into the electronic health record. Design implementation strategies that target hypothesized mechanisms of change and context-specific barriers. The study had two phases. The Qualitative Analysis phase included semi-structured interviews with primary care physicians to elicit key task behaviors (e.g. ordering a low-dose CT) and understand the underlying behavioral determinants (e.g. social influence). The Implementation Strategy Design phase consisted of defining implementation strategies and hypothesizing causal pathways to improve screening with a decision aid. Three key task behaviors and four behavioral determinants emerged from 14 interviews. Implementation strategies were designed to target multiple levels of influence. Strategies included increasing provider self-efficacy toward performing shared decision making and using the decision aid, improving provider performance expectancy toward ordering a low-dose CT, increasing social influence toward performing shared decision making and using the decision aid, and addressing key facilitators to using the decision aid. This study contributes knowledge about theoretical determinants of key task behaviors associated with lung cancer screening. We designed implementation strategies according to causal pathways that can be replicated and tested at other institutions. Future research is needed to evaluate the effectiveness of these strategies and to determine the contexts in which they can be effectively applied. [ABSTRACT FROM AUTHOR]

Introduction: Poor clinical trial (CT) recruitment is a significant barrier to translating basic science discoveries into medical practice. Improving support for primary care provider (PCP) referral of patients to CTs may be an important part of the solution. However, implementing CT referral support in primary care is not only technically challenging, but also presents challenges at the person and organization levels. Methods: The objectives of this study were (1) to characterize provider and clinical supervisor attitudes and perceptions regarding CT research, recruitment, and referrals in primary care and (2) to identify perceived workflow strategies and facilitators relevant to designing a technology-supported primary care CT referral program. Focus groups were conducted with PCPs, directors, and supervisors. Results: Analysis indicated widespread support for the intrinsic scientific value of CTs, while at the same time deep concerns regarding protecting patient well-being, perceived loss of control when patients participate in trials, concern about the impact of point-of-care referrals on clinic workflow, the need for standard processes, and the need for CT information that enables referring providers to quickly confirm that the burdens are justified by the benefits at both patient and provider levels. PCP suggestions pertinent to implementing a CT referral decision support system are reported. Conclusion: The results from this work contribute to developing an implementation approach to support increased referral of patients to CTs. [ABSTRACT FROM AUTHOR]

SMART on FHIR applications are standards-based tools integrated with electronic health record (EHR) systems and intended for dissemination across EHR platforms. A key challenge for disseminating many apps is that EHR vendors provide different levels of support for FHIR. Thus, app developers must balance functionality versus portability. In this case study, a feature-rich app for neonatal bilirubin management was developed prioritizing physician-requested functionality, with custom FHIR interfaces implemented within the EHR as needed. Following wide intra-institutional use, several approaches are being pursued for adapting the app for cross-institutional dissemination: user surveys and interviews to identify least-valued app features which could potentially be omitted; enabling the application to provide differential features depending on available EHR FHIR capabilities; replacing custom FHIR interfaces with native EHR FHIR interfaces as they became available; and using a canonical logical data model known as QUICK that can be mapped to different FHIR versions and profiles., (©2019 AMIA - All rights reserved.)

Introduction . Preventable adverse drug events are a significant patient-safety concern, yet most medication alerts are disregarded. Pharmacists encounter the highest number of medication alerts and likely have developed behaviors to cope with alerting inefficiencies. The study objective was to better understand alert override behavior relating to a motivational construct framework. Methods . Mixed-methods study of 10 pharmacists (567 verifications) with eye-tracking observations and retrospective think aloud interviews. Results . Pharmacists spent on average 14 seconds longer verifying orders with alerts than orders without alerts (p<0.001). Verification occurred before alerts were triggered, and no order changes occurred after alerts. Pharmacists reported 62% of alerts as unhelpful and 21% as frustrating. Alert interactions took on average 3.9 seconds. Discussion . Pharmacists anticipate alerts by making appropriate checks and changes before alert prompts. Medication alerts seem to be useful. However, the observed pharmacists' behavior suggests changes in the alert context are needed to match cognition.

Documentation processes have changed substantially with EHR adoption. User satisfaction studies have focused on usability or cognitive analysis perspectives. Few studies have provided useful information to developers to improve designs. The purpose of this study is to report a 3-pronged approach to deepen understanding of the documentation process, with the intent to provide useful information for future design. This study was conducted in two phases, beginning with cognitive task interviews and observations, followed by post-observation interviews. Twenty-five constructs were identified across the phases, and we observed several patterns of note writing. Participants provided useful information to potentially inform future design. Our study illustrates how electronic documentation serves many clinical processes and is at the core of the medical record. Providers need multiple kinds of notes and ways to display notes. In order to meet provider goals, we must completely re-think the way electronic documentation is composed and displayed.

Introduction . Implementations of electronic health records (EHR) have been met with mixed outcome reviews. Complaints about these systems have led to many attempts to have useful measures of end-user satisfaction. However, most user satisfaction assessments do not focus on high-level reasoning, despite the complaints of many physicians. Our study attempts to identify some of these determinants. Method . We developed a user satisfaction survey instrument, based on pre-identified and important clinical and non-clinical clinician tasks. We surveyed a sample of in-patient physicians and focused on using exploratory factor analyses to identify underlying high-level cognitive tasks. We used the results to create unique, orthogonal variables representative of latent structure predictive of user satisfaction. Results. Our findings identified 3 latent high-level tasks that were associated with end-user satisfaction: a) High- level clinical reasoning b) Communicate/coordinate care and c) Follow the rules/compliance. Conclusion : We were able to successfully identify latent variables associated with satisfaction. Identification of communicability and high-level clinical reasoning as important factors determining user satisfaction can lead to development and design of more usable electronic health records with higher user satisfaction.

Objectives: Transitions in patient care pose an increased risk to patient safety. One way to reduce this risk is to ensure accurate medication reconciliation during the transition. Here we present an evaluation of an electronic medication reconciliation module we developed to reduce the transition risk in patients referred for home healthcare., Methods: Nineteen physicians with experience in managing home health referrals were recruited to participate in this within-subjects experiment. Participants completed medication reconciliation for three clinical cases in each of two conditions. The first condition (paper-based) simulated current practice - reconciling medication discrepancies between a paper plan of care (CMS 485) and a simulated Electronic Health Record (EHR). For the second condition (electronic) participants used our medication reconciliation module, which we integrated into the simulated EHR. To evaluate the effectiveness of our medication reconciliation module, we employed repeated measures ANOVA to test the hypotheses that the module will: 1) Improve accuracy by reducing the number of unaddressed medication discrepancies, 2) Improve efficiency by reducing the reconciliation time, 3) have good perceived usability., Results: The improved accuracy hypothesis is supported. Participants left more discrepancies unaddressed in the paper-based condition than the electronic condition, F (1,1) = 22.3, p < 0.0001 (Paper Mean = 1.55, SD = 1.20; Electronic Mean = 0.45, SD = 0.65). However, contrary to our efficiency hypothesis, participants took the same amount of time to complete cases in the two conditions, F (1, 1) =0.007, p = 0.93 (Paper Mean = 258.7 seconds, SD = 124.4; Electronic Mean = 260.4 seconds, SD = 158.9). The usability hypothesis is supported by a composite mean ability and confidence score of 6.41 on a 7-point scale, 17 of 19 participants preferring the electronic system and an SUS rating of 86.5., Conclusion: We present the evaluation of an electronic medication reconciliation module that increases detection and resolution of medication discrepancies compared to a paper-based process. Further work to integrate medication reconciliation within an electronic medical record is warranted.

We present a novel user-centric visual analytics system that supports investigation of simulated disease outbreak and the study of decision-making. We developed Epinome as part of our research on decision making in public health and in particular, on the evaluation of information search strategies in public health practice. Epinome is a highly dynamic web-based system that provides a platform to track and study subjects' decision making and information search strategies, under controlled and repeatable conditions using simulated disease outbreaks. In this paper we focus on the design and implementation of Epinome and present relevant results from field tests we conducted in Utah and Colorado.