Search

Background Temporal dataset shift can cause degradation in model performance as discrepancies between training and deployment data grow over time. The primary objective was to determine whether parsimonious models produced by specific feature selection methods are more robust to temporal dataset shift as measured by out-of-distribution (OOD) performance, while maintaining in-distribution (ID) performance. Methods Our dataset consisted of intensive care unit patients from MIMIC-IV categorized by year groups (2008–2010, 2011–2013, 2014–2016, and 2017–2019). We trained baseline models using L2-regularized logistic regression on 2008–2010 to predict in-hospital mortality, long length of stay (LOS), sepsis, and invasive ventilation in all year groups. We evaluated three feature selection methods: L1-regularized logistic regression (L1), Remove and Retrain (ROAR), and causal feature selection. We assessed whether a feature selection method could maintain ID performance (2008–2010) and improve OOD performance (2017–2019). We also assessed whether parsimonious models retrained on OOD data performed as well as oracle models trained on all features in the OOD year group. Results The baseline model showed significantly worse OOD performance with the long LOS and sepsis tasks when compared with the ID performance. L1 and ROAR retained 3.7 to 12.6% of all features, whereas causal feature selection generally retained fewer features. Models produced by L1 and ROAR exhibited similar ID and OOD performance as the baseline models. The retraining of these models on 2017–2019 data using features selected from training on 2008–2010 data generally reached parity with oracle models trained directly on 2017–2019 data using all available features. Causal feature selection led to heterogeneous results with the superset maintaining ID performance while improving OOD calibration only on the long LOS task. Conclusions While model retraining can mitigate the impact of temporal dataset shift on parsimonious models produced by L1 and ROAR, new methods are required to proactively improve temporal robustness.

Background Deep generative models (DGMs) present a promising avenue for generating realistic, synthetic data to augment existing health care datasets. However, exactly how the completeness of the original dataset affects the quality of the generated synthetic data is unclear. Objectives In this paper, we investigate the effect of data completeness on samples generated by the most common DGM paradigms. Methods We create both cross-sectional and panel datasets with varying missingness and subset rates and train generative adversarial networks, variational autoencoders, and autoregressive models (Transformers) on these datasets. We then compare the distributions of generated data with original training data to measure similarity. Results We find that increased incompleteness is directly correlated with increased dissimilarity between original and generated samples produced through DGMs. Conclusions Care must be taken when using DGMs to generate synthetic data as data completeness issues can affect the quality of generated data in both panel and cross-sectional datasets.

Introduction In the health care environment, a huge volume of data is produced on a daily basis. However, the processes of collecting, storing, sharing, analyzing, and reporting health data usually face with numerous challenges that lead to producing incomplete, inaccurate, and untimely data. As a result, data quality issues have received more attention than before. Objective The purpose of this article is to provide an insight into the data quality definitions, dimensions, and assessment methodologies. Methods In this article, a scoping literature review approach was used to describe and summarize the main concepts related to data quality and data quality assessment methodologies. Search terms were selected to find the relevant articles published between January 1, 2012 and September 31, 2022. The retrieved articles were then reviewed and the results were reported narratively. Results In total, 23 papers were included in the study. According to the results, data quality dimensions were various and different methodologies were used to assess them. Most studies used quantitative methods to measure data quality dimensions either in paper-based or computer-based medical records. Only two studies investigated respondents' opinions about data quality. Conclusion In health care, high-quality data not only are important for patient care, but also are vital for improving quality of health care services and better decision making. Therefore, using technical and nontechnical solutions as well as constant assessment and supervision is suggested to improve data quality.

Background The short time frame between the coronavirus disease 2019 (COVID-19) pandemic declaration and the vaccines authorization led to concerns among public regarding the safety and efficacy of the vaccines. The Food and Drug Administration uses the Vaccine Adverse Events Reporting System (VAERS) where general population can report their vaccine side effects in the text box. This information could be utilized to determine self-reported vaccine side effects. Objective To develop a supervised and unsupervised natural language processing (NLP) pipeline to extract self-reported COVID-19 vaccination side effects, location of the side effects, medications, and possibly false/misinformation seeking further investigation in a structured format for analysis and reporting. Methods We utilized the VAERS dataset of COVID-19 vaccine reports from November 2020 to August 2022 of 725,246 individuals. We first developed a gold-standard (GS) dataset of randomly selected 1,500 records. Second, the GS was split into training, testing, and validation sets. The training dataset was used to develop the NLP applications (supervised and unsupervised) and testing and validation datasets were used to test the performances of the NLP application. Results The NLP application automatically extracted vaccine side effects, body locations of the side effects, medication, and possibly misinformation with moderate to high accuracy (84% sensitivity, 82% specificity, and 83% F-1 measure). We found that 23% people (386,270) faced arm soreness, 31% body swelling (226,208), 23% fatigue/body weakness (168,160), and 22% (159,873) cold/flue-like symptoms. Most of the complications occurred in the body locations such as the arm, back, chest, neck, face, and head. Over-the-counter pain medications such as Tylenol and Ibuprofen and allergy medication like Benadryl were most reported self-reported medications. Death due to COVID-19, changes in the DNA, and infertility were possible false/misinformation reported by people. Conclusion Some self-reported side effects such as syncope, arthralgia, and blood clotting need further clinical investigations. Our NLP application may help in extracting information from big free-text electronic datasets to help policy makers and other researchers with decision making.

Background: Privacy management is a key issue when dealing with storage and distribution of health information. However, FAIR (Findability, Accessibility, Interoperability, and Reusability) principles when sharing information are in increasing demand in several organizations, especially for information generated in public-funded research projects. Objectives: The two main objectives of the presented work are the definition of a secure and interoperable modular architecture to manage different kinds of medical content (xIPAMS [x, for Any kind of content, Information Protection And Management System] and HIPAMS [Health Information Protection And Management System]), and the application of FAIR principles to that architecture in such a way that privacy and security are compatible with FAIR. Methods: We propose the concept of xIPAMS as a modular architecture, following standards for interoperability, which defines mechanisms for privacy, protection, storage, search, and access to health-related information. Results: xIPAMS provides FAIR principles and preserves patient's privacy. For each module, we identify how FAIR principles apply. Conclusions: We have analyzed how xIPAMS, and in particular HIPAMS (Health content), support the FAIR principles focusing on security and privacy. We have identified the FAIR principles supported by the different xIPAMS modules, concluding that the four principles are supported. Our analysis has also considered a possible implementation based on the concept of DACS (Document Access and Communication System), a system storing medical documents in a private and secure way. In addition, we have analyzed security aspects of the FAIRification process and how they are provided by xIPAMS modules. The work presented in this article has been partially supported by the Spanish Government under the project: GenClinLab-Sec (Mechanisms for secure and efficient management of genomic information tailored to clinical laboratories: Security Aspects, PID2020-114394RB-C31) funded by MCIN/AEI/10.13039/501100011033 and by the Generalitat de Catalunya (2017 SGR 1749).

Background While there is a rapid increase in digital health initiatives focusing on the processing of personal data for strengthening the delivery of reproductive, maternal, newborn, and child health (RMNCH) services in fragile settings, these are often unaccompanied at both the policy and operational levels with adequate legal and regulatory frameworks.Objective The main aim was to understand the maturity level of digital personal data initiatives for RMNCH services within fragile contexts. This aim was performed by choosing digital health initiatives from each country (two in Jordan and three in Palestine) based on RMNCH.Methods A qualitative study design was adopted. We developed a digital maturity assessment tool assessing two maturity levels: the information and communications technology digital infrastructure, and data governance and interoperability in place for the five selected RMNCH initiatives in Jordan and Palestine.Results Overall, the digital infrastructure and technological readiness components are more advanced and show higher maturity levels compared with data governance and interoperability components in Jordan and Palestine. In Jordan, the overall Jordan stillbirths and neonatal deaths surveillance initiative maturity indicators are somehow less advanced than those of the Electronic Maternal and Child Health Handbook-Jordan (EMCH-J) application. In Palestine, the Electronic Maternal and Child Health-registry initiative maturity indicators are more advanced than both Avicenna and EMCH-Palestine initiatives.Conclusion The findings highlighted several challenges and opportunities around the application and implementation of selected digital health initiatives in the provision of RMNCH in Jordan and Palestine. Our findings shed lights on the maturity level of these initiatives within fragile contexts. The maturity level of the five RMNCH initiatives in both countries is inadequate and requires further advancement before they can be scaled up and scaled out. Taking the World Health Organization recommendations into account when developing, implementing, and scaling digital health initiatives in low- and middle-income countries can result in successful and sustainable initiatives, thus meeting health needs and improving the quality of health care received by individuals especially those living in fragile contexts.

Background Since COVID-19 (coronavirus disease 2019) was discovered in December 2019, it has spread worldwide. Early isolation and medical observation management of cases and their close contacts are the key to controlling the spread of the epidemic. However, traditional medical observation requires medical staff to measure body temperature and other vital signs face to face and record them manually. There is a general shortage of human and personal protective equipment and a high risk of occupational exposure, which seriously threaten the safety of medical staff. Methods We designed an intelligent crowd isolation medical observation management system framework based on the Internet of Things using wireless telemetry and big data cloud platform remote management technology. Through a smart wearable device with built-in sensors, vital sign data and geographical locations of medical observation subjects are collected and automatically uploaded to the big data monitoring platform on demand. According to the comprehensive analysis of the set threshold parameters, abnormal subjects are screened out, and activity tracking and health status monitoring for medical observation and management objectives are performed through monitoring and early warning management and post-event data traceability. In the trial of this system, the subjects wore the wristwatches designed in this study and real-time monitoring was conducted throughout the whole process. Additionally, for comparison, the traditional method was also used for these people. Medical staff came to measure their temperature twice a day. The subjects were 1,128 returned overseas Chinese from Europe. Results Compared with the traditional vital sign detection method, the system designed in this study has the advantages of a fast response, low error, stability, and good endurance. It can monitor the temperature, pulse, blood pressure, and heart rate of the monitored subject in real time. The system designed in this study and the traditional vital sign detection method were both used to monitor 1,128 close contacts with COVID-19. There were six cases of abnormal body temperature that were missed by traditional manual temperature measurement in the morning and evening, and these six cases (0.53%) were sent to the hospital for further diagnosis. The abnormal body temperature of these six cases was not found in time when the medical staff came to check the temperature on a twice-a-day basis. The system designed in this study, however, can detect the abnormal body temperature of all these six people. The sensitivity and specificity of our system were both 100%. Conclusion The system designed in this study can monitor the body temperature, blood oxygen, blood pressure, heart rate, and geographical location of the monitoring subject in real time. It can be extended to COVID-19 medical observation isolation points, shelter hospitals, infectious disease wards, and nursing homes.

Objectives The TMF (Technology, Methods, and Infrastructure for Networked Medical Research) Data Protection Guide (TMF-DP) makes path-breaking recommendations on the subject of data protection in research projects. It includes comprehensive requirements for applications such as patient lists, pseudonymization services, and consent management services. Nevertheless, it lacks a structured, categorized list of requirements for simplified application in research projects and systematic evaluation. The 3LGM2IHE (“Three-layer Graphbased meta model – Integrating the Healthcare Enterprise [IHE] ” ) project is funded by the German Research Foundation (DFG). 3LGM2IHE aims to define modeling paradigms and implement modeling tools for planning health care information systems. In addition, one of the goals is to create and publish 3LGM2 information system architecture design patterns (short “design patterns”) for the community as design models in terms of a framework. A structured list of data protection-related requirements based on the TMF-DP is a precondition to integrate functions (3LGM2 Domain Layer) and building blocks (3LGM2 Logical Tool Layer) in 3LGM2 design patterns. Methods In order to structure the continuous text of the TMF-DP, requirement types were defined in a first step. In a second step, dependencies and delineations of the definitions were identified. In a third step, the requirements from the TMF-DP were systematically extracted. Based on the identified lists of requirements, a fourth step included the comparison of the identified requirements with exemplary open source tools as provided by the “Independent Trusted Third Party of the University Medicine Greifswald” (TTP tools). Results As a result, four lists of requirements were created, which contain requirements for the “patient list”, the “pseudonymization service”, and the “consent management”, as well as cross-component requirements from the TMF-DP chapter 6 in a structured form. Further to requirements (1), possible variants (2) of implementations (to fulfill a single requirement) and recommendations (3) were identified. A comparison of the requirements lists with the functional scopes of the open source tools E-PIX (record linkage), gPAS (pseudonym management), and gICS (consent management) has shown that these fulfill more than 80% of the requirements. Conclusions A structured set of data protection-related requirements facilitates a systematic evaluation of implementations with respect to the fulfillment of the TMF-DP guidelines. These re-usable lists provide a decision aid for the selection of suitable tools for new research projects. As a result, these lists form the basis for the development of data protection-related 3LGM2 design patterns as part of the 3LGM2IHE project.

Objective To provide high-quality data for coronavirus disease 2019 (COVID-19) research, we validated derived COVID-19 clinical indicators and 22 associated machine learning phenotypes, in the Mass General Brigham (MGB) COVID-19 Data Mart. Methods Fifteen reviewers performed a retrospective manual chart review for 150 COVID-19-positive patients in the data mart. To support rapid chart review for a wide range of target data, we offered a natural language processing (NLP)-based chart review tool, the Digital Analytic Patient Reviewer (DAPR). For this work, we designed a dedicated patient summary view and developed new 127 NLP logics to extract COVID-19 relevant medical concepts and target phenotypes. Moreover, we transformed DAPR for research purposes so that patient information is used for an approved research purpose only and enabled fast access to the integrated patient information. Lastly, we performed a survey to evaluate the validation difficulty and usefulness of the DAPR. Results The concepts for COVID-19-positive cohort, COVID-19 index date, COVID-19-related admission, and the admission date were shown to have high values in all evaluation metrics. However, three phenotypes showed notable performance degradation than the positive predictive value in the prepandemic population. Based on these results, we removed the three phenotypes from our data mart. In the survey about using the tool, participants expressed positive attitudes toward using DAPR for chart review. They assessed that the validation was easy and DAPR helped find relevant information. Some validation difficulties were also discussed. Conclusion Use of NLP technology in the chart review helped to cope with the challenges of the COVID-19 data validation task and accelerated the process. As a result, we could provide more reliable research data promptly and respond to the COVID-19 crisis. DAPR's benefit can be expanded to other domains. We plan to operationalize it for wider research groups.

Background Behavior and health are inextricably linked. As a result, continuous wearable sensor data offer the potential to predict clinical measures. However, interruptions in the data collection occur, which create a need for strategic data imputation. Objective The objective of this work is to adapt a data generation algorithm to impute multivariate time series data. This will allow us to create digital behavior markers that can predict clinical health measures. Methods We created a bidirectional time series generative adversarial network to impute missing sensor readings. Values are imputed based on relationships between multiple fields and multiple points in time, for single time points or larger time gaps. From the complete data, digital behavior markers are extracted and are mapped to predicted clinical measures. Results We validate our approach using continuous smartwatch data for n = 14 participants. When reconstructing omitted data, we observe an average normalized mean absolute error of 0.0197. We then create machine learning models to predict clinical measures from the reconstructed, complete data with correlations ranging from r = 0.1230 to r = 0.7623. This work indicates that wearable sensor data collected in the wild can be used to offer insights on a person's health in natural settings.

Background Allergy is a hypersensitive reaction that occurs when the allergen reacts with the immune system. The prevalence and severity of the allergies are uprising in South Asian countries. Allergy often occurs in combinations which becomes difficult for physicians to diagnose. Objectives This work aims to develop a decision-making model which aids physicians in diagnosing allergy comorbidities. The model intends to not only provide rational decisions, but also explainable knowledge about all alternatives. Methods The allergy data gathered from real-time sources contain a smaller number of samples for comorbidities. Decision-making model applies three sampling strategies, namely, ideal, single, and complete, to balance the data. Bayes theorem-based probabilistic approaches are used to extract knowledge from the balanced data. Preference weights for attributes with respect to alternatives are gathered from a group of domain-experts affiliated to different allergy testing centers. The weights are combined with objective knowledge to assign confidence values to alternatives. The system provides these values along with explanations to aid decision-makers in choosing an optimal decision. Results Metrics of explainability and user satisfaction are used to evaluate the effectiveness of the system in real-time diagnosis. Fleiss' Kappa statistic is 0.48, and hence the diagnosis of experts is said to be in moderate agreement. The decision-making model provides a maximum of 10 suitable and relevant pieces of evidence to explain a decision alternative. Clinicians have improved their diagnostic performance by 3% after using CDSS (77.93%) with a decrease in 20% of time taken. Conclusion The performance of less-experienced clinicians has improved with the support of an explainable decision-making model. The code for the framework with all intermediate results is available at https://github.com/kavya6697/Allergy-PT.git.

Background Breast cancer is the most prevailing heterogeneous disease among females characterized with distinct molecular subtypes and varied clinicopathological features. With the emergence of various artificial intelligence techniques especially machine learning, the breast cancer research has attained new heights in cancer detection and prognosis. Objective Recent development in computer driven diagnostic system has enabled the clinicians to improve the accuracy in detecting various types of breast tumors. Our study is to develop a computer driven diagnostic system which will enable the clinicians to improve the accuracy in detecting various types of breast tumors. Methods In this article, we proposed a breast cancer classification model based on the hybridization of machine learning approaches for classifying triple-negative breast cancer and non-triple negative breast cancer patients with clinicopathological features collected from multiple tertiary care hospitals/centers. Results The results of genetic algorithm and support vector machine (GA-SVM) hybrid model was compared with classics feature selection SVM hybrid models like support vector machine-recursive feature elimination (SVM-RFE), LASSO-SVM, Grid-SVM, and linear SVM. The classification results obtained from GA-SVM hybrid model outperformed the other compared models when applied on two distinct hospital-based datasets of patients investigated with breast cancer in North West of African subcontinent. To validate the predictive model accuracy, 10-fold cross-validation method was applied on all models with the same multicentered datasets. The model performance was evaluated with well-known metrics like mean squared error, logarithmic loss, F1-score, area under the ROC curve, and the precision–recall curve. Conclusion The hybrid machine learning model can be employed for breast cancer subtypes classification that could help the medical practitioners in better treatment planning and disease outcome.

Background Timely and reliable data are crucial for clinical, epidemiologic, and program management decision making. Electronic health information systems provide platforms for managing large longitudinal patient records. Nigeria implemented the National Data Repository (NDR) to create a central data warehouse of all people living with human immunodeficiency virus (PLHIV) while providing useful functionalities to aid decision making at different levels of program implementation. Objective We describe the Nigeria NDR and its development process, including its use for surveillance, research, and national HIV program monitoring toward achieving HIV epidemic control. Methods Stakeholder engagement meetings were held in 2013 to gather information on data elements and vocabulary standards for reporting patient-level information, technical infrastructure, human capacity requirements, and information flow. Findings from these meetings guided the development of the NDR. An implementation guide provided common terminologies and data reporting structures for data exchange between the NDR and the electronic medical record (EMR) systems. Data from the EMR were encoded in extensible markup language and sent to the NDR over secure hypertext transfer protocol after going through a series of validation processes. Results By June 30, 2021, the NDR had up-to-date records of 1,477,064 (94.4%) patients receiving HIV treatment across 1,985 health facilities, of which 1,266,512 (85.7%) patient records had fingerprint template data to support unique patient identification and record linkage to prevent registration of the same patient under different identities. Data from the NDR was used to support HIV program monitoring, case-based surveillance and production of products like the monthly lists of patients who have treatment interruptions and dashboards for monitoring HIV test and start. Conclusion The NDR enabled the availability of reliable and timely data for surveillance, research, and HIV program monitoring to guide program improvements to accelerate progress toward epidemic control.

Background In recent years, researchers have used many computerized interventions to reduce medical errors, the third cause of death in developed countries. One of such interventions is using differential diagnosis generators in primary care, where physicians may encounter initial symptoms without any diagnostic presuppositions. These systems generate multiple diagnoses, ranked by their likelihood. As such, these reports' accuracy can be determined by the location of the correct diagnosis in the list. Objective This study aimed to design and evaluate a novel practical web-based differential diagnosis generator solution in primary care. Methods In this research, a new online clinical decision support system, called DxGenerator, was designed to improve diagnostic accuracy; to this end, an attempt was made to converge a semantic database with the unified medical language system (UMLS) knowledge base, using MetaMap tool and natural language processing. In this regard, 120 diseases of gastrointestinal organs causing abdominal pain were modeled into the database. After designing an inference engine and a pseudo-free-text interactive interface, 172 patient vignettes were inputted into DxGenerator and ISABEL, the most accurate similar system. The Wilcoxon signed ranked test was used to compare the position of correct diagnoses in DxGenerator and ISABEL. The α level was defined as 0.05. Results On a total of 172 vignettes, the mean and standard deviation of correct diagnosis positions improved from 4.2 ± 5.3 in ISABEL to 3.2 ± 3.9 in DxGenerator. This improvement was significant in the subgroup of uncommon diseases (p-value Conclusion Using UMLS knowledge base and MetaMap Tools can improve the accuracy of diagnostic systems in which terms are entered in a free text manner. Applying these new methods will help the medical community accept medical diagnostic systems better.

Background: MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward.Objective: To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance.Methods: MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated.Results: The percentages of exact matches and missing gold-standard terms were 0.6–0.79 and 0.09–0.3 for one behavior option, and 0.56–0.8 and 0.09–0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively.Conclusion: We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.

Background Data protection policies might prohibit the transfer of existing study data to interested research groups. To overcome legal restrictions, simulated data can be transferred that mimic the structure but are different from the existing study data. Objectives The aim of this work is to introduce the simple-to-use R package Mock Data Generation (modgo) that may be used for simulating data from existing study data for continuous, ordinal categorical, and dichotomous variables. Methods The core is to combine rank inverse normal transformation with the calculation of a correlation matrix for all variables. Data can then be simulated from a multivariate normal and transferred back to the original scale of the variables. Unique features of modgo are that it allows to change the correlation between variables, to perform perturbation analysis, to handle multicenter data, and to change inclusion/exclusion criteria by selecting specific values of one or a set of variables. Simulation studies on real data demonstrate the validity and flexibility of modgo. Results modgo mimicked the structure of the original study data. Results of modgo were similar with those from two other existing packages in standard simulation scenarios. modgo's flexibility was demonstrated on several expansions. Conclusion The R package modgo is useful when existing study data may not be shared. Its perturbation expansion permits to simulate truly anonymized subjects. The expansion to multicenter studies can be used for validating prediction models. Additional expansions can support the unraveling of associations even in large study data and can be useful in power calculations.

Background Owing to the linguistic situation, Japanese natural language processing (NLP) requires morphological analyses for word segmentation using dictionary techniques. Objective We aimed to clarify whether it can be substituted with an open-end discovery-based NLP (OD-NLP), which does not use any dictionary techniques. Methods Clinical texts at the first medical visit were collected for comparison of OD-NLP with word dictionary-based-NLP (WD-NLP). Topics were generated in each document using a topic model, which later corresponded to the respective diseases determined in International Statistical Classification of Diseases and Related Health Problems 10 revision. The prediction accuracy and expressivity of each disease were examined in equivalent number of entities/words after filtration with either term frequency and inverse document frequency (TF-IDF) or dominance value (DMV). Results In documents from 10,520 observed patients, 169,913 entities and 44,758 words were segmented using OD-NLP and WD-NLP, simultaneously. Without filtering, accuracy and recall levels were low, and there was no difference in the harmonic mean of the F-measure between NLPs. However, physicians reported OD-NLP contained more meaningful words than WD-NLP. When datasets were created in an equivalent number of entities/words with TF-IDF, F-measure in OD-NLP was higher than WD-NLP at lower thresholds. When the threshold increased, the number of datasets created decreased, resulting in increased values of F-measure, although the differences disappeared. Two datasets near the maximum threshold showing differences in F-measure were examined whether their topics were associated with diseases. The results showed that more diseases were found in OD-NLP at lower thresholds, indicating that the topics described characteristics of diseases. The superiority remained as much as that of TF-IDF when filtration was changed to DMV. Conclusion The current findings prefer the use of OD-NLP to express characteristics of diseases from Japanese clinical texts and may help in the construction of document summaries and retrieval in clinical settings.

Introduction Pneumonia is caused by microbes that establish an infectious process in the lungs. The gold standard for pneumonia diagnosis is radiologist-documented pneumonia-related features in radiology notes that are captured in electronic health records in an unstructured format. Objective The study objective was to develop a methodological approach for assessing validity of a pneumonia diagnosis based on identifying presence or absence of key radiographic features in radiology reports with subsequent rendering of diagnostic decisions into a structured format. Methods A pneumonia-specific natural language processing (NLP) pipeline was strategically developed applying Clinical Text Analysis and Knowledge Extraction System (cTAKES) to validate pneumonia diagnoses following development of a pneumonia feature–specific lexicon. Radiographic reports of study-eligible subjects identified by International Classification of Diseases (ICD) codes were parsed through the NLP pipeline. Classification rules were developed to assign each pneumonia episode into one of three categories: “positive,” “negative,” or “not classified: requires manual review” based on tagged concepts that support or refute diagnostic codes. Results A total of 91,998 pneumonia episodes diagnosed in 65,904 patients were retrieved retrospectively. Approximately 89% (81,707/91,998) of the total pneumonia episodes were documented by 225,893 chest X-ray reports. NLP classified and validated 33% (26,800/81,707) of pneumonia episodes classified as “Pneumonia-positive,” 19% as (15401/81,707) as “Pneumonia-negative,” and 48% (39,209/81,707) as “episode classification pending further manual review.” NLP pipeline performance metrics included accuracy (76.3%), sensitivity (88%), and specificity (75%). Conclusion The pneumonia-specific NLP pipeline exhibited good performance comparable to other pneumonia-specific NLP systems developed to date.

Background Easy identification of immunocompromised hosts (ICHs) would allow for stratification of culture results based on host type. Methods We utilized antimicrobial stewardship program (ASP) team notes written during handshake stewardship rounds in the pediatric intensive care unit (PICU) as the gold standard for host status; clinical notes from the primary team, medication orders during the encounter, problem list, and billing diagnoses documented prior to the ASP documentation were extracted to develop models that predict host status. We calculated performance for three models based on diagnoses/medications, with and without natural language processing from clinical notes. The susceptibility of pathogens causing bacteremia to commonly used empiric antibiotic regimens was then stratified by host status. Results We identified 844 antimicrobial episodes from 666 unique patients; 160 (18.9%) were identified as ICHs. We randomly selected 675 initiations (80%) for model training and 169 initiations (20%) for testing. A rule-based model using diagnoses and medications alone yielded a sensitivity of 0.87 (08.6–0.88), specificity of 0.93 (0.92–0.93), and positive predictive value (PPV) of 0.74 (0.73–0.75). Adding clinical notes into XGBoost model led to improved specificity of 0.98 (0.98–0.98) and PPV of 0.9 (0.88–0.91), but with decreased sensitivity 0.77 (0.76–0.79). There were 77 bacteremia episodes during the study period identified and a host-specific visualization was created. Conclusions An electronic health record–based phenotype based on notes, diagnoses, and medications identifies ICH in the PICU with high specificity.

Background Prediction models inform decisions in many areas of medicine. Most models are fitted once and then applied to new (future) patients, despite the fact that model coefficients can vary over time due to changes in patients' clinical characteristics and disease risk. However, the optimal method to detect changes in model parameters has not been rigorously assessed. Methods We simulated data, informed by post–lung transplant mortality data and tested the following two approaches for detecting model change: (1) the “Direct Approach,” it compares coefficients of the model refit on recent data to those at baseline; and (2) “Calibration Regression,” it fits a logistic regression model of the log-odds of the observed outcomes versus the linear predictor from the baseline model (i.e., the log-odds of the predicted probabilities obtained from the baseline model) and tests whether the intercept and slope differ from 0 and 1, respectively. Four scenarios were simulated using logistic regression for binary outcomes as follows: (1) we fixed all model parameters, (2) we varied the outcome prevalence between 0.1 and 0.2, (3) we varied the coefficient of one of the ten predictors between 0.2 and 0.4, and (4) we varied the outcome prevalence and coefficient of one predictor simultaneously. Results Calibration regression tended to detect changes sooner than the Direct Approach, with better performance (e.g., larger proportion of true claims). When the sample size was large, both methods performed well. When two parameters changed simultaneously, neither method performed well. Conclusion Neither change detection method examined here proved optimal under all circumstances. However, our results suggest that if one is interested in detecting a change in overall incidence of an outcome (e.g., intercept), the Calibration Regression method may be superior to the Direct Approach. Conversely, if one is interested in detecting a change in other model covariates (e.g., slope), the Direct Approach may be superior.

Background Public health emergencies leave little time to develop novel surveillance efforts. Understanding which preexisting clinical datasets are fit for surveillance use is of high value. Coronavirus disease 2019 (COVID-19) offers a natural applied informatics experiment to understand the fitness of clinical datasets for use in disease surveillance. Objectives This study evaluates the agreement between legacy surveillance time series data and discovers their relative fitness for use in understanding the severity of the COVID-19 emergency. Here fitness for use means the statistical agreement between events across series. Methods Thirteen weekly clinical event series from before and during the COVID-19 era for the United States were collected and integrated into a (multi) time series event data model. The Centers for Disease Control and Prevention (CDC) COVID-19 attributable mortality, CDC's excess mortality model, national Emergency Medical Services (EMS) calls, and Medicare encounter level claims were the data sources considered in this study. Cases were indexed by week from January 2015 through June of 2021 and fit to Distributed Random Forest models. Models returned the variable importance when predicting the series of interest from the remaining time series. Results Model r2 statistics ranged from 0.78 to 0.99 for the share of the volumes predicted correctly. Prehospital data were of high value, and cardiac arrest (CA) prior to EMS arrival was on average the best predictor (tied with study week). COVID-19 Medicare claims volumes can predict COVID-19 death certificates (agreement), while viral respiratory Medicare claim volumes cannot predict Medicare COVID-19 claims (disagreement). Conclusions Prehospital EMS data should be considered when evaluating the severity of COVID-19 because prehospital CA known to EMS was the strongest predictor on average across indices.

Background Data quality issues can cause false decisions of clinical decision support systems (CDSSs). Analyzing local data quality has the potential to prevent data quality-related failure of CDSS adoption. Objectives To define a shareable set of applicable measurement methods (MMs) for a targeted data quality assessment determining the suitability of local data for our CDSS. Methods We derived task-specific MMs using four approaches: (1) a GUI-based data quality analysis using the open source tool openCQA. (2) Analyzing cases of known false CDSS decisions. (3) Data-driven learning on MM-results. (4) A systematic check to find blind spots in our set of MMs based on the HIDQF data quality framework. We expressed the derived data quality-related knowledge about the CDSS using the 5-tuple-formalization for MMs. Results We identified some task-specific dataset characteristics that a targeted data quality assessment for our use case should inspect. Altogether, we defined 394 MMs organized in 13 data quality knowledge bases. Conclusions We have created a set of shareable, applicable MMs that can support targeted data quality assessment for CDSS-based systemic inflammatory response syndrome (SIRS) detection in critically ill, pediatric patients. With the demonstrated approaches for deriving and expressing task-specific MMs, we intend to help promoting targeted data quality assessment as a commonly recognized usual part of research on data-consuming application systems in health care.

Background Synthetic tabular data generation is a potentially valuable technology with great promise for data augmentation and privacy preservation. However, prior to adoption, an empirical assessment of generated synthetic tabular data is required across dimensions relevant to the target application to determine its efficacy. A lack of standardized and objective evaluation and benchmarking strategy for synthetic tabular data in the health domain has been found in the literature. Objective The aim of this paper is to identify key dimensions, per dimension metrics, and methods for evaluating synthetic tabular data generated with different techniques and configurations for health domain application development and to provide a strategy to orchestrate them. Methods Based on the literature, the resemblance, utility, and privacy dimensions have been prioritized, and a collection of metrics and methods for their evaluation are orchestrated into a complete evaluation pipeline. This way, a guided and comparative assessment of generated synthetic tabular data can be done, categorizing its quality into three categories (“Excellent,” “Good,” and “Poor”). Six health care-related datasets and four synthetic tabular data generation approaches have been chosen to conduct an analysis and evaluation to verify the utility of the proposed evaluation pipeline. Results The synthetic tabular data generated with the four selected approaches has maintained resemblance, utility, and privacy for most datasets and synthetic tabular data generation approach combination. In several datasets, some approaches have outperformed others, while in other datasets, more than one approach has yielded the same performance. Conclusion The results have shown that the proposed pipeline can effectively be used to evaluate and benchmark the synthetic tabular data generated by various synthetic tabular data generation approaches. Therefore, this pipeline can support the scientific community in selecting the most suitable synthetic tabular data generation approaches for their data and application of interest.

Background Digitalization is playing a major role in mastering the current coronavirus 2019 (COVID-19) pandemic. However, several outbreaks of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in German hospitals last year have shown that many of the surveillance and warning mechanisms related to infection control (IC) in hospitals need to be updated. Objectives The main objective of the following work was to assess the state of information technology (IT) systems supporting IC and surveillance in German university hospitals in March 2021, almost a year into the SARS-CoV-2 pandemic. Methods As part of the National Research Network for Applied Surveillance and Testing project within the Network University Medicine, a cross-sectional survey was conducted to assess the situation of IC IT systems in 36 university hospitals in Germany. Results Among the most prominent findings were the lack of standardization of IC IT systems and the predominant use of commercial IC IT systems, while the vast majority of hospitals reported inadequacies in the features their IC IT systems provide for their daily work. However, as the pandemic has shown that there is a need for systems that can help improve health care, several German university hospitals have already started this upgrade independently. Conclusions The deep challenges faced by the German health care sector regarding the integration and interoperability of IT systems designed for IC and surveillance are unlikely to be solved through punctual interventions and require collaboration between educational, medical, and administrative institutions.

Introduction The informed consent is the legal basis for research with human subjects. Therefore, the consent form (CF) as legally binding document must be valid, that is, be completely filled-in stating the person's decision clearly and signed by the respective person. However, especially paper-based CFs might have quality issues and the transformation into machine-readable information could add to low quality. This paper evaluates the quality and arising quality issues of paper-based CFs using the example of the Baltic Fracture Competence Centre (BFCC) fracture registry. It also evaluates the impact of quality assurance (QA) measures including giving site-specific feedback. Finally, it answers the question whether manual data entry of patients' decisions by clinical staff leads to a significant error rate in digitalized paper-based CFs. Methods Based on defined quality criteria, monthly QA including source data verification was conducted by two individual reviewers since the start of recruitment in December 2017. Basis for the analyses are the CFs collected from December 2017 until February 2019 (first recruitment period). Results After conducting QA internally, the sudden increase of quality issues in May 2018 led to site-specific feedback reports and follow-up training regarding the CFs' quality starting in June 2018. Specific criteria and descriptions on how to correct the CFs helped in increasing the quality in a timely matter. Most common issues were missing pages, decisions regarding optional modules, and signature(s). Since patients' datasets without valid CFs must be deleted, QA helped in retaining 65 datasets for research so that the final datapool consisted of 840 (99.29%) patients. Conclusion All quality issues could be assigned to one predefined criterion. Using the example of the BFCC fracture registry, CF-QA proved to significantly increase CF quality and help retain the number of available datasets for research. Consequently, the described quality indicators, criteria, and QA processes can be seen as the best practice approach.

Background As a national effort to better understand the current pandemic, three cohorts collect sociodemographic and clinical data from coronavirus disease 2019 (COVID-19) patients from different target populations within the German National Pandemic Cohort Network (NAPKON). Furthermore, the German Corona Consensus Dataset (GECCO) was introduced as a harmonized basic information model for COVID-19 patients in clinical routine. To compare the cohort data with other GECCO-based studies, data items are mapped to GECCO. As mapping from one information model to another is complex, an additional consistency evaluation of the mapped items is recommended to detect possible mapping issues or source data inconsistencies. Objectives The goal of this work is to assure high consistency of research data mapped to the GECCO data model. In particular, it aims at identifying contradictions within interdependent GECCO data items of the German national COVID-19 cohorts to allow investigation of possible reasons for identified contradictions. We furthermore aim at enabling other researchers to easily perform data quality evaluation on GECCO-based datasets and adapt to similar data models. Methods All suitable data items from each of the three NAPKON cohorts are mapped to the GECCO items. A consistency assessment tool (dqGecco) is implemented, following the design of an existing quality assessment framework, retaining their-defined consistency taxonomies, including logical and empirical contradictions. Results of the assessment are verified independently on the primary data source. Results Our consistency assessment tool helped in correcting the mapping procedure and reveals remaining contradictory value combinations within COVID-19 symptoms, vital signs, and COVID-19 severity. Consistency rates differ between the different indicators and cohorts ranging from 95.84% up to 100%. Conclusion An efficient and portable tool capable of discovering inconsistencies in the COVID-19 domain has been developed and applied to three different cohorts. As the GECCO dataset is employed in different platforms and studies, the tool can be directly applied there or adapted to similar information models.

Background Multisite research networks such as the project “Collaboration on Rare Diseases” connect various hospitals to obtain sufficient data for clinical research. However, data quality (DQ) remains a challenge for the secondary use of data recorded in different health information systems. High levels of DQ as well as appropriate quality assessment methods are needed to support the reuse of such distributed data. Objectives The aim of this work is the development of an interoperable methodology for assessing the quality of data recorded in heterogeneous sources to improve the quality of rare disease (RD) documentation and support clinical research. Methods We first developed a conceptual framework for DQ assessment. Using this theoretical guidance, we implemented a software framework that provides appropriate tools for calculating DQ metrics and for generating local as well as cross-institutional reports. We further applied our methodology on synthetic data distributed across multiple hospitals using Personal Health Train. Finally, we used precision and recall as metrics to validate our implementation. Results Four DQ dimensions were defined and represented as disjunct ontological categories. Based on these top dimensions, 9 DQ concepts, 10 DQ indicators, and 25 DQ parameters were developed and applied to different data sets. Randomly introduced DQ issues were all identified and reported automatically. The generated reports show the resulting DQ indicators and detected DQ issues. Conclusion We have shown that our approach yields promising results, which can be used for local and cross-institutional DQ assessments. The developed frameworks provide useful methods for interoperable and privacy-preserving assessments of DQ that meet the specified requirements. This study has demonstrated that our methodology is capable of detecting DQ issues such as ambiguity or implausibility of coded diagnoses. It can be used for DQ benchmarking to improve the quality of RD documentation and to support clinical research on distributed data.

Background Receiver operating characteristic (ROC) analysis is commonly used for comparing models and humans; however, the exact analytical techniques vary and some are flawed. Objectives The aim of the study is to identify common flaws in ROC analysis for human versus model performance, and address them. Methods We review current use and identify common errors. We also review the ROC analysis literature for more appropriate techniques. Results We identify concerns in three techniques: (1) using mean human sensitivity and specificity; (2) assuming humans can be approximated by ROCs; and (3) matching sensitivity and specificity. We identify a technique from Provost et al using dominance tables and cost-prevalence gradients that can be adapted to address these concerns. Conclusion Dominance tables and cost-prevalence gradients provide far greater detail when comparing performances of models and humans, and address common failings in other approaches. This should be the standard method for such analyses moving forward.

Background Value sets are lists of terms (e.g., opioid medication names) and their corresponding codes from standard clinical vocabularies (e.g., RxNorm) created with the intent of supporting health information exchange and research. Value sets are manually-created and often exhibit errors. Objectives The aim of the study is to develop a semi-automatic, data-centric natural language processing (NLP) method to assess medication-related value set correctness and evaluate it on a set of opioid medication value sets. Methods We developed an NLP algorithm that utilizes value sets containing mostly true positives and true negatives to learn lexical patterns associated with the true positives, and then employs these patterns to identify potential errors in unseen value sets. We evaluated the algorithm on a set of opioid medication value sets, using the recall, precision and F1-score metrics. We applied the trained model to assess the correctness of unseen opioid value sets based on recall. To replicate the application of the algorithm in real-world settings, a domain expert manually conducted error analysis to identify potential system and value set errors. Results Thirty-eight value sets were retrieved from the Value Set Authority Center, and six (two opioid, four non-opioid) were used to develop and evaluate the system. Average precision, recall, and F1-score were 0.932, 0.904, and 0.909, respectively on uncorrected value sets; and 0.958, 0.953, and 0.953, respectively after manual correction of the same value sets. On 20 unseen opioid value sets, the algorithm obtained average recall of 0.89. Error analyses revealed that the main sources of system misclassifications were differences in how opioids were coded in the value sets—while the training value sets had generic names mostly, some of the unseen value sets had new trade names and ingredients. Conclusion The proposed approach is data-centric, reusable, customizable, and not resource intensive. It may help domain experts to easily validate value sets.

Objective Data harmonization is essential to integrate individual participant data from multiple sites, time periods, and trials for meta-analysis. The process of mapping terms and phrases to an ontology is complicated by typographic errors, abbreviations, truncation, and plurality. We sought to harmonize medical history (MH) and adverse events (AE) term records across 21 randomized clinical trials in pulmonary arterial hypertension and chronic thromboembolic pulmonary hypertension. Methods We developed and applied a semi-automated harmonization pipeline for use with domain-expert annotators to resolve ambiguous term mappings using exact and fuzzy matching. We summarized MH and AE term mapping success, including map quality measures, and imputation of a generalizing term hierarchy as defined by the applied Medical Dictionary for Regulatory Activities (MedDRA) ontology standard. Results Over 99.6% of both MH (N = 37,105) and AE (N = 58,170) records were successfully mapped to MedDRA low-level terms. Automated exact matching accounted for 74.9% of MH and 85.5% of AE mappings. Term recommendations from fuzzy matching in the pipeline facilitated annotator mapping of the remaining 24.9% of MH and 13.8% of AE records. Imputation of the generalized MedDRA term hierarchy was unambiguous in 85.2% of high-level terms, 99.4% of high-level group terms, and 99.5% of system organ class in MH, and 75% of high-level terms, 98.3% of high-level group terms, and 98.4% of system organ class in AE. Conclusion This pipeline dramatically reduced the burden of manual annotation for MH and AE term harmonization and could be adapted to other data integration efforts.

Background Automated clinical decision support for risk assessment is a powerful tool in combating cardiovascular disease (CVD), enabling targeted early intervention that could avoid issues of overtreatment or undertreatment. However, current CVD risk prediction models use observations at baseline without explicitly representing patient history as a time series. Objective The aim of this study is to examine whether by explicitly modelling the temporal dimension of patient history event prediction may be improved. Methods This study investigates methods for multivariate sequential modelling with a particular emphasis on long short-term memory (LSTM) recurrent neural networks. Data from a CVD decision support tool is linked to routinely collected national datasets including pharmaceutical dispensing, hospitalization, laboratory test results, and deaths. The study uses a 2-year observation and a 5-year prediction window. Selected methods are applied to the linked dataset. The experiments performed focus on CVD event prediction. CVD death or hospitalization in a 5-year interval was predicted for patients with history of lipid-lowering therapy. Results The results of the experiments showed temporal models are valuable for CVD event prediction over a 5-year interval. This is especially the case for LSTM, which produced the best predictive performance among all models compared achieving AUROC of 0.801 and average precision of 0.425. The non-temporal model comparator ridge classifier (RC) trained using all quarterly data or by aggregating quarterly data (averaging time-varying features) was highly competitive achieving AUROC of 0.799 and average precision of 0.420 and AUROC of 0.800 and average precision of 0.421, respectively. Conclusion This study provides evidence that the use of deep temporal models particularly LSTM in clinical decision support for chronic disease would be advantageous with LSTM significantly improving on commonly used regression models such as logistic regression and Cox proportional hazards on the task of CVD event prediction.

Background FAIR Guiding Principles present a synergy with the use cases for digital health records, in that clinical data need to be found, accessible within a range of environments, and data must interoperate between systems and subsequently reused. The use of HL7 FHIR, openEHR, IHE XDS, and SNOMED CT (FOXS) together represents a specification to create an open digital health platform for modern health care applications. Objectives To describe where logical FOXS components align to the European Open Science Cloud Interoperability Framework (EOSC-IF) reference architecture for semantic interoperability. This should provide a means of defining if FOXS aligns to FAIR principles and to establish the data models and structures that support longitudinal care records as being fit to underpin scientific research. Methods The EOSC-IF Semantic View is a representation of semantic interoperability where meaning is preserved between systems and users. This was analyzed and cross-referenced with FOXS architectural components, mapping concepts, and objects that describe content such as catalogues and semantic artifacts. Results Majority of conceptual Semantic View components were featured within the FOXS architecture. Semantic Business Objects are composed of a range of elements such as openEHR archetypes and templates, FHIR resources and profiles, SNOMED CT concepts, and XDS document identifiers. Semantic Functional Content comprises catalogues of metadata that were also supported by openEHR and FHIR tools. Conclusions Despite some elements of EOSC-IF being vague (e.g., FAIR Digital Object), there was a broad conformance to the framework concepts and the components of a FOXS platform. This work supports a health-domain-specific view of semantic interoperability and how this may be achieved to support FAIR data for health research via a standardized framework.

Objective Our objective was to phenotype periodontal disease (PD) diagnoses from three different sections (diagnosis codes, clinical notes, and periodontal charting) of the electronic dental records (EDR) by developing two automated computer algorithms. Methods We conducted a retrospective study using EDR data of patients (n = 27,138) who received care at Temple University Maurice H. Kornberg School of Dentistry from January 1, 2017 to August 31, 2021. We determined the completeness of patient demographics, periodontal charting, and PD diagnoses information in the EDR. Next, we developed two automated computer algorithms to automatically diagnose patients' PD statuses from clinical notes and periodontal charting data. Last, we phenotyped PD diagnoses using automated computer algorithms and reported the improved completeness of diagnosis. Results The completeness of PD diagnosis from the EDR was as follows: periodontal diagnosis codes 36% (n = 9,834), diagnoses in clinical notes 18% (n = 4,867), and charting information 80% (n = 21,710). After phenotyping, the completeness of PD diagnoses improved to 100%. Eleven percent of patients had healthy periodontium, 43% were with gingivitis, 3% with stage I, 36% with stage II, and 7% with stage III/IV periodontitis. Conclusions We successfully developed, tested, and deployed two automated algorithms on big EDR datasets to improve the completeness of PD diagnoses. After phenotyping, EDR provided 100% completeness of PD diagnoses of 27,138 unique patients for research purposes. This approach is recommended for use in other large databases for the evaluation of their EDR data quality and for phenotyping PD diagnoses and other relevant variables.

Introduction Health care information systems can generate and/or record huge volumes of data, some of which may be reused for research, clinical trials, or teaching. However, these databases can be affected by data quality problems; hence, an important step in the data reuse process consists in detecting and rectifying these issues. With a view to facilitating the assessment of data quality, we developed a taxonomy of data quality problems in operational databases. Material We searched the literature for publications that mentioned “data quality problems,” “data quality taxonomy,” “data quality assessment,” or “dirty data.” The publications were then reviewed, compared, summarized, and structured using a bottom-up approach, to provide an operational taxonomy of data quality problems. The latter were illustrated with fictional examples (though based on reality) from clinical databases. Results Twelve publications were selected, and 286 instances of data quality problems were identified and were classified according to six distinct levels of granularity. We used the classification defined by Oliveira et al to structure our taxonomy. The extracted items were grouped into 53 data quality problems. Discussion This taxonomy facilitated the systematic assessment of data quality in databases by presenting the data's quality according to their granularity. The definition of this taxonomy is the first step in the data cleaning process. The subsequent steps include the definition of associated quality assessment methods and data cleaning methods. Conclusion Our new taxonomy enabled the classification and illustration of 53 data quality problems found in hospital databases.

Background In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear. Objectives The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics. Methods We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics). Results Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon—only moderate or low consensus was achieved for those metrics. Conclusion The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multifaceted to ensure acceptability.

Background Patient safety event reports provide valuable insight into systemic safety issues but deriving insights from these reports requires computational tools to efficiently parse through large volumes of qualitative data. Natural language processing (NLP) combined with predictive learning provides an automated approach to evaluating these data and supporting the work of patient safety analysts. Objectives The objective of this study was to use NLP and machine learning techniques to develop a generalizable, scalable, and reliable approach to classifying event reports for the purpose of driving improvements in the safety and quality of patient care. Methods Datasets for 14 different labels (themes) were vectorized using a bag-of-words, tf-idf, or document embeddings approach and then applied to a series of classification algorithms via a hyperparameter grid search to derive an optimized model. Reports were also analyzed for terms strongly associated with each theme using an adjusted F-score calculation. Results F1 score for each optimized model ranged from 0.951 (“Fall”) to 0.544 (“Environment”). The bag-of-words approach proved optimal for 12 of 14 labels, and the naïve Bayes algorithm performed best for nine labels. Linear support vector machine was demonstrated as optimal for three labels and XGBoost for four of the 14 labels. Labels with more distinctly associated terms performed better than less distinct themes, as shown by a Pearson's correlation coefficient of 0.634. Conclusions We were able to demonstrate an analytical pipeline that broadly applies NLP and predictive modeling to categorize patient safety reports from multiple facilities. This pipeline allows analysts to more rapidly identify and structure information contained in patient safety data, which can enhance the evaluation and the use of this information over time.

Objectives A thread is the most common information aggregation unit in a health forum, so effective thread recommendation is critical for improving the user experience in an online health community (OHC). This paper proposes an OHC thread recommendation method based on topic model and network embedding, which recommends threads to users by training a classifier and predicting user reply behavior. Methods The proposed model uses the network structure to describe valid information in OHCs and treats a recommendation as the task of predicting links between users and threads in the network. Topic nodes are added to the information network to better represent the features of users and threads. The results of the latent Dirichlet allocation (LDA) model describe thread topics and user interests from the perspectives of consumer health vocabulary in OHCs and social support types. The large-scale information network embedding technology LINE is used to mine the node's contextual information from the network structure to obtain the low-dimensional vectors of nodes. We optimize the representation method and similarity calculation of network nodes and enrich the network structure information contained in the recommended features to improve the recommendation effect. Results To verify the proposed model, we collected data from the diabetes forum “Sweet Home.” The experimental results show that the proposed model can effectively extract user interests in threads from the information network and optimize thread recommendation in OHCs.

Objectives The aim of the study is to design an ontology model for the representation of assets and its features in distributed health care environments. Allow the interchange of information about these assets through the use of specific vocabularies based on the use of ontologies. Methods Ontologies are a formal way to represent knowledge by means of triples composed of a subject, a predicate, and an object. Given the sensitivity of network assets in health care institutions, this work by using an ontology-based representation of information complies with the FAIR principles. Federated queries to the ontology systems, allow users to obtain data from multiple sources (i.e., several hospitals belonging to the same public body). Therefore, this representation makes it possible for network administrators in health care institutions to have a clear understanding of possible threats that may emerge in the network. Results As a result of this work, the “Software Defined Networking Description Language—CUREX Asset Discovery Tool Ontology” (SDNDL-CAO) has been developed. This ontology uses the main concepts in network assets to represent the knowledge extracted from the distributed health care environments: interface, device, port, service, etc. Conclusion The developed SDNDL-CAO ontology allows to represent the aforementioned knowledge about the distributed health care environments. Network administrators of these institutions will benefit as they will be able to monitor emerging threats in real-time, something critical when managing personal medical information.

Background Unambiguous sharing of data requires information models and terminology in combination, but there is a lack of knowledge as to how they should be combined, leading to impaired interoperability. Objectives To facilitate creation of guidelines for SNOMED CT terminology binding we have performed a literature review to find existing recommendations and expose knowledge gaps. The primary audience is practitioners and researchers working with terminology binding. Methods PubMed, Scopus, and Web of Science were searched for papers containing “terminology binding,” “subset,” “map,” “information model” or “implement” and the term “SNOMED.” Results The search yielded 616 unique papers published from 2004 to 2020, from which 55 papers were selected and analyzed inductively. Topics described in the papers include problems related to input material, SNOMED CT, information models, and lack of appropriate tools as well as recommendations regarding competence. Conclusion Recommendations are given for practitioners and researchers. Many of the stated problems can be solved by better co-operation between domain experts and informaticians and better knowledge of SNOMED CT. Settings where these competences either work together or where staff with knowledge of both act as brokers are well equipped for terminology binding. Tooling is not thoroughly researched and might be a possible way to facilitate terminology binding.

Background Unplanned hospital readmission after discharge reflects low satisfaction and reliability in care and the possibility of potential medical accidents, and is thus indicative of the quality of patient care and the appropriateness of discharge plans. Objectives The purpose of this study was to develop and validate prediction models for all-cause unplanned hospital readmissions within 30 days of discharge, based on a common data model (CDM), which can be applied to multiple institutions for efficient readmission management. Methods Retrospective patient-level prediction models were developed based on clinical data of two tertiary general university hospitals converted into a CDM developed by Observational Medical Outcomes Partnership. Machine learning classification models based on the LASSO logistic regression model, decision tree, AdaBoost, random forest, and gradient boosting machine (GBM) were developed and tested by manipulating a set of CDM variables. An internal 10-fold cross-validation was performed on the target data of the model. To examine its transportability, the model was externally validated. Verification indicators helped evaluate the model performance based on the values of area under the curve (AUC). Results Based on the time interval for outcome prediction, it was confirmed that the prediction model targeting the variables obtained within 30 days of discharge was the most efficient (AUC of 82.75). The external validation showed that the model is transferable, with the combination of various clinical covariates. Above all, the prediction model based on the GBM showed the highest AUC performance of 84.14 ± 0.015 for the Seoul National University Hospital cohort, yielding in 78.33 in external validation. Conclusions This study showed that readmission prediction models developed using machine-learning techniques and CDM can be a useful tool to compare two hospitals in terms of patient-data features.

Background Interpretations of the electrocardiogram (ECG) are often prepared using software outside the electronic health record (EHR) and imported via an interface as a narrative note. Thus, natural language processing is required to create a computable representation of the findings. Challenges include misspellings, nonstandard abbreviations, jargon, and equivocation in diagnostic interpretations. Objectives Our objective was to develop an algorithm to reliably and efficiently extract such information and map it to the standardized ECG ontology developed jointly by the American Heart Association, the American College of Cardiology Foundation, and the Heart Rhythm Society. The algorithm was to be designed to be easily modifiable for use with EHRs and ECG reporting systems other than the ones studied. Methods An algorithm using natural language processing techniques was developed in structured query language to extract and map quantitative and diagnostic information from ECG narrative reports to the cardiology societies' standardized ECG ontology. The algorithm was developed using a training dataset of 43,861 ECG reports and applied to a test dataset of 46,873 reports. Results Accuracy, precision, recall, and the F1-measure were all 100% in the test dataset for the extraction of quantitative data (e.g., PR and QTc interval, atrial and ventricular heart rate). Performances for matches in each diagnostic category in the standardized ECG ontology were all above 99% in the test dataset. The processing speed was approximately 20,000 reports per minute. We externally validated the algorithm from another institution that used a different ECG reporting system and found similar performance. Conclusion The developed algorithm had high performance for creating a computable representation of ECG interpretations. Software and lookup tables are provided that can easily be modified for local customization and for use with other EHR and ECG reporting systems. This algorithm has utility for research and in clinical decision-support where incorporation of ECG findings is desired.

Background The COVID-19 pandemic has changed health care systems and clinical workflows in many countries, including the United States. This public health crisis has accelerated the transformation of health care delivery through the use of telehealth. Due to the coronavirus' severity and pathogenicity, telehealth services are considered the best platforms to meet suddenly increased patient care demands, reduce the transformation of the virus, and protect patients and health care workers. However, many hospitals, clinicians, and patients are not ready to switch to virtual care completely. Objectives We designed six experiments to examine how people (as an actual beneficiary of telehealth) evaluate five telehealth encounters versus face-to-face visits. Methods We used an online survey to collect data from 751 individuals (patients) in the United States. Results Findings demonstrate that significant factors for evaluating five types of telehealth encounters are perceived convenience expected from telehealth encounters, perceived psychological risks associated with telehealth programs, and perceived attentive care services delivered by telehealth platforms. However, significant elements for comparing telehealth services with traditional face-to-face clinic visits are perceived cost-saving, perceived time-saving, perceived hygienic services, perceived technical errors, perceived information completeness, perceived communication barriers, perceived trust in medical care platforms' competency, and perceived privacy concerns. Conclusion Although the in-person visit was reported as the most preferred care practice, there was no significant difference between people's willingness to use face-to-face visits versus virtual care. Nevertheless, before the widespread rollout of telehealth platforms, health care systems need to determine and address the challenges of implementing virtual care to improve patient engagement in telehealth services. This study also provides practical implications for health care providers to deploy telehealth effectively during the pandemic and postpandemic phases.

Background The European Medical Device Regulation 2017/745 (MDR) has its date of application in May 2021. This new legislation has refined and expanded the need of manufacturers to have a postmarket surveillance (PMS) system. According to this legislation, a postmarket clinical follow-up (PMCF) plan is also required. Manufacturers of high-risk medical devices are obliged to conduct both PMCF and PMS studies. There is thus the need to generate evidence from clinical data. Objectives The conduct of several studies for PMS and PMCF can be cumbersome. We therefore aim to present a modular approach to combine PMS and PMCF studies into a single study. Materials and Methods We extracted the topics listed in the MDR, especially Annex XV, Section 3, the Good Clinical Practice for medical devices (EN 14155:2020, Annex A). In addition, we added topics according to the SPIRIT and the SPIRIT-PRO statement and created a draft clinical investigation plan (CIP). Results The CIP template is provided as part of the manuscript. The modular concept has passed the required regulatory and legal requirements for one specific study. Conclusion A modular approach for combining PMCF and PMS studies in a single CIP has been developed and implemented, and it is ready for use. The provided CIP template should enable other researchers and groups to adopt this concept according to their needs.

Background Health care services are increasingly being digitized, but extant literature shows that digital technologies and applications are often developed without careful consideration of user needs. Research is needed to identify and investigate best-in-class methods to support user-centered design of mHealth applications. Objectives The article investigates how the Kano model can be adapted and used for the purpose of eliciting child patients' information needs during the design phase of mHealth application development. The aim is to demonstrate its applicability for collecting and analyzing patient-centered data that are key to designing technology-supported solutions for health management. Methods The article is based on a mixed-methods case study, which includes interviews with 21 patients aged 6 to 18. Structured interviews are analyzed based on prescriptions of the Kano model. Semi-structured interviews about child patients' information needs are analyzed thematically. Results The results demonstrate several improvements to the Kano model that take into account the difficulties of effectively communicating with child patients. The combination of two types of interviews offers unique insights into the what, how, and why of patients' needs. Adaptation of the Kano model, simplification of response options, and participation of child patients' parents in interviews facilitate data collection. Conclusion The article shows how the Kano model can be adapted to provide an effective means of eliciting child patients' needs. Adapting the model by combining structured and semi-structured interviews makes it a powerful tool in designing mHealth applications.

Background During the COVID-19 pandemic, several methodologies were designed for obtaining electronic health record (EHR)-derived datasets for research. These processes are often based on black boxes, on which clinical researchers are unaware of how the data were recorded, extracted, and transformed. In order to solve this, it is essential that extract, transform, and load (ETL) processes are based on transparent, homogeneous, and formal methodologies, making them understandable, reproducible, and auditable. Objectives This study aims to design and implement a methodology, according with FAIR Principles, for building ETL processes (focused on data extraction, selection, and transformation) for EHR reuse in a transparent and flexible manner, applicable to any clinical condition and health care organization. Methods The proposed methodology comprises four stages: (1) analysis of secondary use models and identification of data operations, based on internationally used clinical repositories, case report forms, and aggregated datasets; (2) modeling and formalization of data operations, through the paradigm of the Detailed Clinical Models; (3) agnostic development of data operations, selecting SQL and R as programming languages; and (4) automation of the ETL instantiation, building a formal configuration file with XML. Results First, four international projects were analyzed to identify 17 operations, necessary to obtain datasets according to the specifications of these projects from the EHR. With this, each of the data operations was formalized, using the ISO 13606 reference model, specifying the valid data types as arguments, inputs and outputs, and their cardinality. Then, an agnostic catalog of data was developed through data-oriented programming languages previously selected. Finally, an automated ETL instantiation process was built from an ETL configuration file formally defined. Conclusions This study has provided a transparent and flexible solution to the difficulty of making the processes for obtaining EHR-derived data for secondary use understandable, auditable, and reproducible. Moreover, the abstraction carried out in this study means that any previous EHR reuse methodology can incorporate these results into them.

Objective The aim of the study is to identify the important clinical variables found in both pregnant and non-pregnant women who tested positive for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, using an artificial intelligence (AI) platform. Methods This was a retrospective cohort study of all women between the ages of 18 to 45, who were admitted to Maimonides Medical Center between March 10, 2020 and December 20, 2021. Patients were included if they had nasopharyngeal PCR swab positive for SARS-CoV-2. Safe People Artificial Intelligence (SPAI) platform, developed by Gynisus, Inc., was used to identify key clinical variables predicting a positive test in pregnant and non-pregnant women. A list of mathematically important clinical variables was generated for both non-pregnant and pregnant women. Results Positive results were obtained in 1,935 non-pregnant women and 1,909 non-pregnant women tested negative for SARS-CoV-2 infection. Among pregnant women, 280 tested positive, and 1,000 tested negative. The most important clinical variable to predict a positive swab result in non-pregnant women was age, while elevated D-dimer levels and presence of an abnormal fetal heart rate pattern were the most important clinical variable in pregnant women to predict a positive test. Conclusion In an attempt to better understand the natural history of the SARS-CoV-2 infection we present a side-by-side analysis of clinical variables found in pregnant and non-pregnant women who tested positive for COVID-19. These clinical variables can help stratify and highlight those at risk for SARS-CoV-2 infection and shed light on the individual patient risk for testing positive.

Background Clinical procedures are often performed in outpatient clinics without prior scheduling at the administrative level, and documentation of the procedure often occurs solely in free-text clinical electronic notes. Natural language processing (NLP), particularly named entity recognition (NER), may provide a solution to extracting procedure data from free-text electronic notes. Methods Free-text notes from outpatient ophthalmology visits were collected from the electronic clinical records at a single institution over 3 months. The Prodigy low-code annotation tool was used to create an annotation dataset and train a custom NER model for clinical procedures. Clinical procedures were extracted from the entire set of clinical notes. Results There were a total of 5,098 clinic notes extracted for the study period; 1,923 clinic notes were used to build the NER model, which included a total of 231 manual annotations. The NER model achieved an F-score of 0.767, a precision of 0.810, and a recall of 0.729. The most common procedures performed included intravitreal injections of therapeutic substances, removal of corneal foreign bodies, and epithelial debridement of corneal ulcers. Conclusion The use of a low-code annotation software tool allows the rapid creation of a custom annotation dataset to train a NER model to identify clinical procedures stored in free-text electronic clinical notes. This enables clinicians to rapidly gather previously unidentified procedural data for quality improvement and auditing purposes. Low-code annotation tools may reduce time and coding barriers to clinician participation in NLP research.

Background One Digital Health (ODH) aims to propose a framework that merges One Health's and Digital Health's specific features into an innovative landscape. FAIR (Findable, Accessible, Interoperable, and Reusable) principles consider applications and computational agents (or, in other terms, data, metadata, and infrastructures) as stakeholders with the capacity to find, access, interoperate, and reuse data with none or minimal human intervention. Objectives This paper aims to elicit how the ODH framework is compliant with FAIR principles and metrics, providing some thinking guide to investigate and define whether adapted metrics need to be figured out for an effective ODH Intervention setup. Methods An integrative analysis of the literature was conducted to extract instances of the need—or of the eventual already existing deployment—of FAIR principles, for each of the three layers (keys, perspectives and dimensions) of the ODH framework. The scope was to assess the extent of scatteredness in pursuing the many facets of FAIRness, descending from the lack of a unifying and balanced framework. Results A first attempt to interpret the different technological components existing in the different layers of the ODH framework, in the light of the FAIR principles, was conducted. Although the mature and working examples of workflows for data FAIRification processes currently retrievable in the literature provided a robust ground to work on, a nonsuitable capacity to fully assess FAIR aspects for highly interconnected scenarios, which the ODH-based ones are, has emerged. Rooms for improvement are anyway possible to timely deal with all the underlying features of topics like the delivery of health care in a syndemic scenario, the digital transformation of human and animal health data, or the digital nature conservation through digital technology-based intervention. Conclusions ODH pillars account for the availability (findability, accessibility) of human, animal, and environmental data allowing a unified understanding of complex interactions (interoperability) over time (reusability). A vision of integration between these two worlds, under the vest of ODH Interventions featuring FAIRness characteristics, toward the development of a systemic lookup of health and ecology in a digitalized way, is therefore auspicable.