Search

OBJECTIVE: Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations variability on patient characteristics. MATERIALS AND METHODS: Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics. RESULTS: On average, the teams interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5. CONCLUSIONS: Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

Attempts to scale up instructional interventions confront implementation challenges that mitigate their ultimate impact on teaching and learning. In this article, we argue that learning about adaptation during the design and implementation phases of reform is critical to the development of interventions that can be implemented with integrity at scale. Through analysis of data generated during a mathematics instructional coaching initiative, we examine the adaptations coaches made to diverse relational and organizational contexts. Findings from two studies of adaptation illustrate the need to attend to the extent to which adaptations are consistent with the core features of a reform. Based on our findings, we posit a generalizable model that supports evidence-based mutual adaptation. [For the corresponding grantee submission, see ED599340.]

BACKGROUND: We investigated whether we could use influenza data to develop prediction models for COVID-19 to increase the speed at which prediction models can reliably be developed and validated early in a pandemic. We developed COVID-19 Estimated Risk (COVER) scores that quantify a patients risk of hospital admission with pneumonia (COVER-H), hospitalization with pneumonia requiring intensive services or death (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis using historical data from patients with influenza or flu-like symptoms and tested this in COVID-19 patients. METHODS: We analyzed a federated network of electronic medical records and administrative claims data from 14 data sources and 6 countries containing data collected on or before 4/27/2020. We used a 2-step process to develop 3 scores using historical data from patients with influenza or flu-like symptoms any time prior to 2020. The first step was to create a data-driven model using LASSO regularized logistic regression, the covariates of which were used to develop aggregate covariates for the second step where the COVER scores were developed using a smaller set of features. These 3 COVER scores were then externally validated on patients with 1) influenza or flu-like symptoms and 2) confirmed or suspected COVID-19 diagnosis across 5 databases from South Korea, Spain, and the United States. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. RESULTS: Overall, 44,507 COVID-19 patients were included for model validation. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, kidney disease) which combined with age and sex discriminated which patients would experience any of our three outcomes. The models achieved good performance in influenza and COVID-19 cohorts. For COVID-19 the AUC ranges were, COVER-H: 0.69-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.72-0.90. Calibration varied across the validations with some of the COVID-19 validations being less well calibrated than the influenza validations. CONCLUSIONS: This research demonstrated the utility of using a proxy disease to develop a prediction model. The 3 COVER models with 9-predictors that were developed using influenza data perform well for COVID-19 patients for predicting hospitalization, intensive services, and fatality. The scores showed good discriminatory performance which transferred well to the COVID-19 population. There was some miscalibration in the COVID-19 validations, which is potentially due to the difference in symptom severity between the two diseases. A possible solution for this is to recalibrate the models in each location before use.

[This corrects the article DOI: 10.3389/fphar.2022.893484.].

Background: Routinely collected healthcare data such as administrative claims and electronic health records (EHR) can complement clinical trials and spontaneous reports to detect previously unknown risks of vaccines, but uncertainty remains about the behavior of alternative epidemiologic designs to detect and declare a true risk early. Methods: Using three claims and one EHR database, we evaluate several variants of the case-control, comparative cohort, historical comparator, and self-controlled designs against historical vaccinations using real negative control outcomes (outcomes with no evidence to suggest that they could be caused by the vaccines) and simulated positive control outcomes. Results: Most methods show large type 1 error, often identifying false positive signals. The cohort method appears either positively or negatively biased, depending on the choice of comparator index date. Empirical calibration using effect-size estimates for negative control outcomes can bring type 1 error closer to nominal, often at the cost of increasing type 2 error. After calibration, the self-controlled case series (SCCS) design most rapidly detects small true effect sizes, while the historical comparator performs well for strong effects. Conclusion: When applying any method for vaccine safety surveillance we recommend considering the potential for systematic error, especially due to confounding, which for many designs appears to be substantial. Adjusting for age and sex alone is likely not sufficient to address differences between vaccinated and unvaccinated, and for the cohort method the choice of index date is important for the comparability of the groups. Analysis of negative control outcomes allows both quantification of the systematic error and, if desired, subsequent empirical calibration to restore type 1 error to its nominal value. In order to detect weaker signals, one may have to accept a higher type 1 error.

Post-marketing vaccine safety surveillance aims to detect adverse events following immunization in a population. Whether certain methods of surveillance are more precise and unbiased in generating safety signals is unclear. Here, we synthesized information from existing literature to provide an overview of the strengths, weaknesses, and clinical applications of epidemiologic and analytical methods used in vaccine monitoring, focusing on cohort, case-control and self-controlled designs. These designs are proposed to be evaluated in the EUMAEUS (Evaluating Use of Methods for Adverse Event Under Surveillance-for vaccines) study because of their widespread use and potential utility. Over the past decades, there have been an increasing number of epidemiological study designs used for vaccine safety surveillance. While traditional cohort and case-control study designs remain widely used, newer, novel designs such as the self-controlled case series and self-controlled risk intervals have been developed. Each study design comes with its strengths and limitations, and the most appropriate study design will depend on availability of resources, access to records, number and distribution of cases, and availability of population coverage data. Several assumptions have to be made while using the various study designs, and while the goal is to mitigate any biases, violations of these assumptions are often still present to varying degrees. In our review, we discussed some of the potential biases (i.e., selection bias, misclassification bias and confounding bias), and ways to mitigate them. While the types of epidemiological study designs are well established, a comprehensive comparison of the analytical aspects (including method evaluation and performance metrics) of these study designs are relatively less well studied. We summarized the literature, reporting on two simulation studies, which compared the detection time, empirical power, error rate and risk estimate bias across the above-mentioned study designs. While these simulation studies provided insights on the analytic performance of each of the study designs, its applicability to real-world data remains unclear. To bridge that gap, we provided the rationale of the EUMAEUS study, with a brief description of the study design; and how the use of real-world multi-database networks can provide insights into better methods evaluation and vaccine safety surveillance.

[Figure: see text].

[Figure: see text].

Using real-world data and past vaccination data, we conducted a large-scale experiment to quantify bias, precision and timeliness of different study designs to estimate historical background (expected) compared to post-vaccination (observed) rates of safety events for several vaccines. We used negative (not causally related) and positive control outcomes. The latter were synthetically generated true safety signals with incident rate ratios ranging from 1.5 to 4. Observed vs. expected analysis using within-database historical background rates is a sensitive but unspecific method for the identification of potential vaccine safety signals. Despite good discrimination, most analyses showed a tendency to overestimate risks, with 20%-100% type 1 error, but low (0% to 20%) type 2 error in the large databases included in our study. Efforts to improve the comparability of background and post-vaccine rates, including age-sex adjustment and anchoring background rates around a visit, reduced type 1 error and improved precision but residual systematic error persisted. Additionally, empirical calibration dramatically reduced type 1 to nominal but came at the cost of increasing type 2 error.

There is a dearth of research in aquatic motor competency, a key requirement for primary physical education in order to become physically literate. This study proposes a new assessment protocol for aquatic motor competence and sets out to examine the validity of the Aquatic Movement Protocol (AMP) in children between 7 and 9 years of age. Testing of Gross Motor Development--second edition was implemented to assess general motor competence, including a composite of 10-m running sprint time and standing long jump distance. Aquatic motor competence was assessed by the AMP. Univariate analyses of covariance were used to examine whether assessment of general motor competence differed as opposed to aquatic motor competence. Process and product measures of dryland motor competence were analyzed using male and female subjects measuring three aquatic motor competences (low, medium, and high). Cronbach's alpha and exploratory factor analyses were implemented to show both construct and concurrent validity of the AMP. Children classified as high for aquatic motor competence had significantly higher general motor competence (p = 0.001). Those who achieved a higher composite score for faster sprint speeds and longer jump distances had significantly higher aquatic motor competence (p = 0.001). Cronbach's alpha of 0.908, showed internal consistency of the AMP. Results extracted one factor from analysis with an eigenvalue = 6.2; %variance = 62.1, with loadings higher than 0.5. This data suggests that the items on the AMP measure a single construct that we would call "Aquatic Motor Competence." This study demonstrates that the AMP is a valid measure of aquatic motor competence in primary aged children.

ObjectivesTo demonstrate the application of the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) principles described in our companion article to hypertension treatments and assess internal and external validity of the generated evidence.Materials and methodsLEGEND defines a process for high-quality observational research based on 10 guiding principles. We demonstrate how this process, here implemented through large-scale propensity score modeling, negative and positive control questions, empirical calibration, and full transparency, can be applied to compare antihypertensive drug therapies. We assess internal validity through covariate balance, confidence-interval coverage, between-database heterogeneity, and transitivity of results. We assess external validity through comparison to direct meta-analyses of randomized controlled trials (RCTs).ResultsFrom 21.6 million unique antihypertensive new users, we generate 6 076 775 effect size estimates for 699 872 research questions on 12 946 treatment comparisons. Through propensity score matching, we achieve balance on all baseline patient characteristics for 75% of estimates, observe 95.7% coverage in our effect-estimate 95% confidence intervals, find high between-database consistency, and achieve transitivity in 84.8% of triplet hypotheses. Compared with meta-analyses of RCTs, our results are consistent with 28 of 30 comparisons while providing narrower confidence intervals.ConclusionWe find that these LEGEND results show high internal validity and are congruent with meta-analyses of RCTs. For these reasons we believe that evidence generated by LEGEND is of high quality and can inform medical decision-making where evidence is currently lacking. Subsequent publications will explore the clinical interpretations of this evidence.

Evidence derived from existing health-care data, such as administrative claims and electronic health records, can fill evidence gaps in medicine. However, many claim such data cannot be used to estimate causal treatment effects because of the potential for observational study bias; for example, due to residual confounding. Other concerns include P hacking and publication bias. In response, the Observational Health Data Sciences and Informatics international collaborative launched the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) research initiative. Its mission is to generate evidence on the effects of medical interventions using observational health-care databases while addressing the aforementioned concerns by following a recently proposed paradigm. We define 10 principles of LEGEND that enshrine this new paradigm, prescribing the generation and dissemination of evidence on many research questions at once; for example, comparing all treatments for a disease for many outcomes, thus preventing publication bias. These questions are answered using a prespecified and systematic approach, avoiding P hacking. Best-practice statistical methods address measured confounding, and control questions (research questions where the answer is known) quantify potential residual bias. Finally, the evidence is generated in a network of databases to assess consistency by sharing open-source analytics code to enhance transparency and reproducibility, but without sharing patient-level information. Here we detail the LEGEND principles and provide a generic overview of a LEGEND study. Our companion paper highlights an example study on the effects of hypertension treatments, and evaluates the internal and external validity of the evidence we generate.

BACKGROUND:Uncertainty remains about the optimal monotherapy for hypertension, with current guidelines recommending any primary agent among the first-line drug classes thiazide or thiazide-like diuretics, angiotensin-converting enzyme inhibitors, angiotensin receptor blockers, dihydropyridine calcium channel blockers, and non-dihydropyridine calcium channel blockers, in the absence of comorbid indications. Randomised trials have not further refined this choice. METHODS:We developed a comprehensive framework for real-world evidence that enables comparative effectiveness and safety evaluation across many drugs and outcomes from observational data encompassing millions of patients, while minimising inherent bias. Using this framework, we did a systematic, large-scale study under a new-user cohort design to estimate the relative risks of three primary (acute myocardial infarction, hospitalisation for heart failure, and stroke) and six secondary effectiveness and 46 safety outcomes comparing all first-line classes across a global network of six administrative claims and three electronic health record databases. The framework addressed residual confounding, publication bias, and p-hacking using large-scale propensity adjustment, a large set of control outcomes, and full disclosure of hypotheses tested. FINDINGS:Using 4·9 million patients, we generated 22 000 calibrated, propensity-score-adjusted hazard ratios (HRs) comparing all classes and outcomes across databases. Most estimates revealed no effectiveness differences between classes; however, thiazide or thiazide-like diuretics showed better primary effectiveness than angiotensin-converting enzyme inhibitors: acute myocardial infarction (HR 0·84, 95% CI 0·75-0·95), hospitalisation for heart failure (0·83, 0·74-0·95), and stroke (0·83, 0·74-0·95) risk while on initial treatment. Safety profiles also favoured thiazide or thiazide-like diuretics over angiotensin-converting enzyme inhibitors. The non-dihydropyridine calcium channel blockers were significantly inferior to the other four classes. INTERPRETATION:This comprehensive framework introduces a new way of doing observational health-care science at scale. The approach supports equivalence between drug classes for initiating monotherapy for hypertension-in keeping with current guidelines, with the exception of thiazide or thiazide-like diuretics superiority to angiotensin-converting enzyme inhibitors and the inferiority of non-dihydropyridine calcium channel blockers. FUNDING:US National Science Foundation, US National Institutes of Health, Janssen Research & Development, IQVIA, South Korean Ministry of Health & Welfare, Australian National Health and Medical Research Council.

Registries have significant roles in assessing the comparative performance of devices. Ideally, early identification of outliers should use a time-to-event outcome while reducing the confounding effects of other components in the device and patient characteristics. Machine learning (ML), which contains self-learning algorithms, is one approach to consider many variables simultaneously to reduce the impact of confounding. The principal objective of this study was to investigate the effectiveness of using either random survival forest (RSF) or regularised/unregularised Cox regression to account for patient and associated device confounding factors in comparison with current standard techniques. This study evaluated RSF and regularised/unregularised Cox regression using data from the Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) to detect outlier devices among 213 individual primary total hip components performed in 163,356 primary procedures from 1 January 2015 to the end of 2019. Device components and patient characteristics were the inputs, and time to first revision surgery was the primary outcome treated as a censored case for death. The effectiveness of the ML approaches was assessed based on the ability to detect the outliers identified by the AOANJRR standard approach. In the study cohort, the standardised AOANJRR approach identified three acetabular components and seven femoral stems as outliers. The ML approaches identified some but not all the outliers detected by the AOANJRR. Both the methods identified three of the same femoral stems, and the RSF identified the other five components, including two of the same acetabular cups and three of the same femoral stems. In addition, both the RSF and Cox techniques detected a number of additional device components that were not previously identified by the standard approach. The results showed that ML may be able to offer a supplementary approach to enhance the early identification of outlier devices. Random survival forest was a more comparable technique to the AOANJRR standard than the Cox regression, but further studies are required to better understand the potential of ML to improve the early identification of outliers. [ABSTRACT FROM AUTHOR]

Based on data from the Longitudinal Study of Australian Children linked with pharmacy dispensing data from the Australian Government's Pharmaceutical Benefits Scheme, we calculated the 1-year prevalence of psychotropic medicine supply in children and adolescents with Autism Spectrum Disorder (ASD) as reported by parents in 2014. The majority of children and adolescents with ASD in Australia were not treated with psychotropic medicine. One-third had claims for at least one psychotropic medication, most commonly medications for attention-deficit/hyperactivity disorder (ADHD), and antidepressants. Antipsychotics were supplied to less than one in twenty children and approximately one in ten adolescents. In line with findings from North America, psychotropic medicine was more often supplied to children and adolescents with ASD and comorbid ADHD.

Background: There are variations in the performance of individual prostheses used in hip replacements. Some of which have unexpectedly higher revision rates – outliers. The Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) has established a standardised multi-stage approach for identifying these devices. This is done by comparing the revision rates of individual prostheses to all other prostheses in class, with the exception of large head metal-on-metal (LHMoM) prostheses. However, improvements in device design and performance over time have required a need to reconsider the comparator group. This study aimed to identify a more specific comparator to better reflect contemporary surgical practice. Methods: The time to first revision was estimated on the data of 413,417 primary total conventional hip replacements undertaken for osteoarthritis (OA) from 01 January 2003 to 31 December 2019. Survivorship analyses with stepwise exclusions were undertaken. The first exclusion was LHMoM, followed by other non-modern bearing surfaces (defined as all the bearing couples except metal or ceramic heads on cross-linked polyethylene and mixed ceramic-on-ceramic), and then devices with modular neck-stem design or used for specific purposes (incl. constrained, dual-mobility, and head size <28 mm). Lastly, all remaining prostheses previously identified as having a higher than anticipated rate of revision (HTARR) were also excluded. Results: These exclusions progressively reduced the cumulative percent revision (CPR) rate. The final comparator, which only includes satisfactory-performed prostheses of contemporary design and use, has a 10-year CPR of 4.30% (95% CI, 4.2–4.41) which is lower than 4.93% (95% CI, 4.84–5.02) for the current comparator used by the AOANJRR (all prostheses excluding LHMOM). Over the study period, 13 additional components were identified utilising the modified comparator. Conclusions: The calculation of the comparator revision rate should be re-evaluated to include only modern prosthesis constructs to ensure that poorly performing prostheses are identified early. [ABSTRACT FROM AUTHOR]