Search

Despite research investment and a growing body of diverse evidence there has been no comprehensive review and grading of evidence for public health emergency preparedness and response practices comparable to those in medicine and other public health fields. The National Academies of Sciences, Engineering, and Medicine convened an ad hoc committee to develop and use methods for grading and synthesizing diverse types of evidence to create a single certainty of intervention-related evidence to support recommendations for Public Health Emergency Preparedness and Response Research. A 13-step consensus building method was used. Experts were first canvassed in public meetings, and a comprehensive review of existing methods was undertaken. Although aspects of existing review methodologies and evidence grading systems were relevant, none adequately covered all requirements for this specific context. Starting with a desire to synthesize diverse sources of evidence not usually included in systematic reviews and using GRADE for assessing certainty and confidence in quantitative and qualitative evidence as the foundation, we developed a mixed-methods synthesis review and grading methodology that drew on (and in some cases adapted) those elements of existing frameworks and methods that were most applicable. Four topics were selected as test cases. The process was operationalized with a suite of method-specific reviews of diverse evidence types for each topic. Further consensus building was undertaken through stakeholder engagement and feedback The NASEM committee's GRADE adaption for mixed-methods reviews will further evolve over time and has yet to be endorsed by the GRADE working group.

BackgroundWhether the presence of dementia in patients makes it difficult for physicians to assess the risk such patients might have for serious conditions such as pulmonary embolism (PE) is unknown. Our objective was to examine the differential association of four clinical factors (deep venous thrombosis (DVT)/PE, malignancy, recent surgery, and tachycardia) with PE testing for patients with dementia compared to patients without dementia.MethodsWe performed a cross-sectional study of emergency department (ED) visits to 104 Veterans Affairs (VA) hospitals from 2011 to 2018 by patients aged 60 years and over presenting with shortness of breath (SOB). Our outcomes were PE testing (CT scan and/or D-dimer) and subsequently diagnosed acute PE.ResultsThe sample included 593,001 patient visits for SOB across 7124 ED physicians; 5.6% of the sample had dementia, and 10.6% received PE testing. Three of the four clinical factors examined had a lower association with PE testing for patients with dementia. For example, after taking into account that at baseline, physicians were 0.9 percentage points less likely to test patients with dementia than patients without dementia for PE, physicians were an additional 2.6 percentage points less likely to test patients with dementia who had tachycardia than patients without dementia who had tachycardia. We failed to find evidence that any clinical factor examined had a differentially lower association with a subsequently diagnosed acute PE for patients with dementia.ConclusionsClinical factors known to be predictive of PE risk had a lower association with PE testing for patients with dementia compared to patients without dementia. These results may be consistent with physicians missing these clinical factors more often when evaluating patients with dementia, but also with physicians recognizing such factors but not using them in the decision-making process. Further understanding how physicians evaluate patients with dementia presenting with common acute symptoms may help improve the care delivered to such patients.

Robotic ventral hernia repair (VHR) has seen rapid adoption, but with limited data assessing clinical outcome or cost. This systematic review compared robotic VHR with laparoscopic and open approaches. This systematic review was undertaken in accordance with PRISMA guidelines. PubMed, MEDLINE, Embase, and Cochrane databases were searched for articles with terms relating to 'robot-assisted', 'cost effectiveness', and 'ventral hernia' or 'incisional hernia' from 1 January 2010 to 10 November 2020. Intraoperative and postoperative outcomes, pain, recurrence, and cost data were extracted for narrative analysis. Of 25 studies that met the inclusion criteria, three were RCTs and 22 observational studies. Robotic VHR was associated with a longer duration of operation than open and laparoscopic repairs, but with fewer transfusions, shorter hospital stay, and lower complication rates than open repair. Robotic VHR was more expensive than laparoscopic repair, but not significantly different from open surgery in terms of cost. There were no significant differences in rates of intraoperative complication, conversion to open surgery, surgical-site infection, readmission, mortality, pain, or recurrence between the three approaches. Robotic VHR was associated with a longer duration of operation, fewer transfusions, a shorter hospital stay, and fewer complications compared with open surgery. Robotic VHR had higher costs and a longer operating time than laparoscopic repair. Randomized or matched data with standardized reporting, long-term outcomes, and cost-effectiveness analyses are still required to weigh the clinical benefits against the cost of robotic VHR.

Abstract Background The evidence base for quality improvement (QI) interventions is expanding rapidly. The diversity of the initiatives and the inconsistency in labeling these as QI interventions makes it challenging for researchers, policymakers, and QI practitioners to access the literature systematically and to identify relevant publications. Methods We evaluated search strategies developed for MEDLINE (Ovid) and PubMed based on free text words, Medical subject headings (MeSH), QI intervention components, continuous quality improvement (CQI) methods, and combinations of the strategies. Three sets of pertinent QI intervention publications were used for validation. Two independent expert reviewers screened publications for relevance. We compared the yield, recall rate, and precision of the search strategies for the identification of QI publications and for a subset of empirical studies on effects of QI interventions. Results The search yields ranged from 2,221 to 216,167 publications. Mean recall rates for reference publications ranged from 5% to 53% for strategies with yields of 50,000 publications or fewer. The 'best case' strategy, a simple text word search with high face validity ('quality' AND 'improv*' AND 'intervention*') identified 44%, 24%, and 62% of influential intervention articles selected by Agency for Healthcare Research and Quality (AHRQ) experts, a set of exemplar articles provided by members of the Standards for Quality Improvement Reporting Excellence (SQUIRE) group, and a sample from the Cochrane Effective Practice and Organization of Care Group (EPOC) register of studies, respectively. We applied the search strategy to a PubMed search for articles published in 10 pertinent journals in a three-year period which retrieved 183 publications. Among these, 67% were deemed relevant to QI by at least one of two independent raters. Forty percent were classified as empirical studies reporting on a QI intervention. Conclusions The presented search terms and operating characteristics can be used to guide the identification of QI intervention publications. Even with extensive iterative development, we achieved only moderate recall rates of reference publications. Consensus development on QI reporting and initiatives to develop QI-relevant MeSH terms are urgently needed.

Abstract Background Fall prevention interventions for community-dwelling older adults have been found to reduce falls in some research studies. However, wider implementation of fall prevention activities in routine care has yielded mixed results. We implemented a theory-driven program to improve care for falls at our Veterans Affairs healthcare facility. The first project arising from this program used a nurse advice telephone line to identify patients' risk factors for falls and to triage patients to appropriate services. Here we report the formative evaluation of this project. Methods To evaluate the intervention we: 1) interviewed patient and employee stakeholders, 2) reviewed participating patients' electronic health record data and 3) abstracted information from meeting minutes. We describe the implementation process, including whether the project was implemented according to plan; identify barriers and facilitators to implementation; and assess the incremental benefit to the quality of health care for fall prevention received by patients in the project. We also estimate the cost of developing the pilot project. Results The project underwent multiple changes over its life span, including the addition of an option to mail patients educational materials about falls. During the project's lifespan, 113 patients were considered for inclusion and 35 participated. Patient and employee interviews suggested support for the project, but revealed that transportation to medical care was a major barrier in following up on fall risks identified by nurse telephone triage. Medical record review showed that the project enhanced usual medical care with respect to home safety counseling. We discontinued the program after 18 months due to staffing limitations and competing priorities. We estimated a cost of $9194 for meeting time to develop the project. Conclusions The project appeared feasible at its outset but could not be sustained past the first cycle of evaluation due to insufficient resources and a waning of local leadership support due to competing national priorities. Future projects will need both front-level staff commitment and prolonged high-level leadership involvement to thrive.

BackgroundRapid adoption of robotic-assisted general surgery procedures, particularly for cholecystectomy, continues while questions remain about its benefits and utility. The objective of this study was to compare the clinical effectiveness of robot-assisted cholecystectomy for benign gallbladder disease as compared with the laparoscopic approach.MethodsA literature search was performed from January 2010 to March 2020, and a narrative analysis was performed as studies were heterogeneous.ResultsOf 887 articles screened, 44 met the inclusion criteria (range 20-735,537 patients). Four were randomized controlled trials, and four used propensity-matching. There were variable comparisons between operative techniques with only 19 out of 44 studies comparing techniques using the same number of ports. Operating room time was longer for the robot-assisted technique in the majority of studies (range 11-55 min for 22 studies, p < 0.05; 15 studies showed no difference; two studies showed shorter laparoscopic times), while conversion rates and intraoperative complications were not different. No differences were detected for the length of stay, surgical site infection, or readmissions. Across studies comparing single-port robot-assisted to multi-port laparoscopic cholecystectomy, there was a higher rate of incisional hernia; however, no differences were noted when comparing single-port robot-assisted to single-port laparoscopic cholecystectomy.ConclusionsClinical outcomes were similar for benign, elective gallbladder disease for robot-assisted compared with laparoscopic cholecystectomy. Overall, the rates of complications were low. More high-quality studies are needed as the robot-assisted technique expands to more complex gallbladder disease, where its utility may prove increasingly beneficial.Systematic review registrationPROSPERO CRD42020156945.

Abstract Background Implementing quality improvement programs that require behavior change on the part of health care professionals and patients has proven difficult in routine care. Significant randomized trial evidence supports creating fall prevention programs for community-dwelling older adults, but adoption in routine care has been limited. Nationally-collected data indicated that our local facility could improve its performance on fall prevention in community-dwelling older people. We sought to develop a sustainable local fall prevention program, using theory to guide program development. Methods We planned program development to include important stakeholders within our organization. The theory-derived plan consisted of 1) an initial leadership meeting to agree on whether creating a fall prevention program was a priority for the organization, 2) focus groups with patients and health care professionals to develop ideas for the program, 3) monthly workgroup meetings with representatives from key departments to develop a blueprint for the program, 4) a second leadership meeting to confirm that the blueprint developed by the workgroup was satisfactory, and also to solicit feedback on ideas for program refinement. Results The leadership and workgroup meetings occurred as planned and led to the development of a functional program. The focus groups did not occur as planned, mainly due to the complexity of obtaining research approval for focus groups. The fall prevention program uses an existing telephonic nurse advice line to 1) place outgoing calls to patients at high fall risk, 2) assess these patients' risk factors for falls, and 3) triage these patients to the appropriate services. The workgroup continues to meet monthly to monitor the progress of the program and improve it. Conclusion A theory-driven program development process has resulted in the successful initial implementation of a fall prevention program.

IntroductionMost older Americans use drug therapies for chronic conditions. Several are associated with risk of Alzheimer's disease and related dementias (ADRD).MethodsA scoping review was used to identify drug classes associated with increasing or decreasing ADRD risk. We analyzed size, type, and findings of the evidence.ResultsWe identified 29 drug classes across 11 therapeutic areas, and 404 human studies. Most common were studies on drugs for hypertension (93) or hyperlipidemia (81). Fewer than five studies were identified for several anti-diabetic and anti-inflammatory drugs. Evidence was observational only for beta blockers, proton pump inhibitors, benzodiazepines, and disease-modifying anti-rheumatic drugs. For 13 drug classes, 50% or more of the studies reported consistent direction of effect on risk of ADRD.DiscussionFuture research targeting drug classes with limited/non-robust evidence, examining sex, racial heterogeneity, and separating classes by molecule, will facilitate understanding of associated risk, and inform clinical and policy efforts to alleviate the growing impact of ADRD.

Abstract Background While falls and urinary incontinence are prevalent among older patients, who sometimes rely on proxies to provide their health information, the validity of proxy reports of concern about falls and urinary incontinence remains unknown. Methods Telephone interviews with 43 consecutive patients with falls or fear of falling and/or bothersome urinary incontinence and their proxies chosen by patients as most knowledgeable about their health. The questionnaire included items derived from the Medical Outcomes Study Short Form 12 (SF-12), a scale assessing concerns about urinary incontinence (UI), and a measure of fear of falling, the Falls Efficacy Scale (FES). Scores were estimated using items asking the proxy perspective (6 items from the SF-12, 10 items from a UI scale, and all 10 FES items). Proxy and patient scores were compared using intraclass correlation coefficients (ICC, one-way model). Variables associated with absolute agreement between patients and proxies were explored. Results Patients had a mean age of 81 years (range 75–93) and 67% were female while proxies had a mean age of 70 (range 42–87) and 49% were female. ICCs were 0.63 for the SF-12, 0.52 for the UI scale, and 0.29 for the FES. Proxies tended to understate patients' general health and incontinence concern, but overstate patients' concern about falling. Proxies who lived with patients and those who more often see patients more closely reflected patient FES scores compared to those who lived apart or those who saw patients less often. Internal consistency reliability of proxy responses was 0.62 for the SF-12, 0.86 for the I-QOL, and 0.93 for the FES. In addition, construct validity of the proxy FES scale was supported by greater proxy-perceived fear of falling for patients who received medical care after a fall during the past 12 months (p < .05). Conclusion Caution should be exercised when using proxies as a source of information about older patients' health perceptions. Questions asking about proxies' views yield suboptimal agreement with patient responses. However, proxy scales of UI and fall concern are internally consistent and may provide valid independent information.

BackgroundPayers and policy makers across the international healthcare market are increasingly using publicly available summary measures to designate providers as "high-performing", but no consistently-applied approach exists to identifying high performers. This paper uses publicly available data to examine how different classification approaches influence which providers are designated as "high-performers".MethodsWe conducted a quantitative analysis of cross-sectional publicly-available performance data in the U.S. We used 2014 Minnesota Community Measurement data from 58 medical groups to classify performance across 4 domains: quality (two process measures of cancer screening and 2 composite measures of chronic disease management), total cost of care, access (a composite CAHPS measure), and patient experience (3 CAHPS measures). We classified medical groups based on performance using either relative thresholds or absolute values of performance on all included measures.ResultsUsing relative thresholds, none of the 58 medical groups achieved performance in the top 25% or 35% in all 4 performance domains. A relative threshold of 40% was needed before one group was classified as high-performing in all 4 domains. Using absolute threshold values, two medical groups were classified as high-performing across all 4 domains. In both approaches, designating "high performance" using fewer domains led to more groups designated as high-performers, though there was little to moderate concordance across identified "high-performing" groups.ConclusionsClassification of medical groups as high performing is sensitive to the domains of performance included, the classification approach, and choice of threshold. With increasing focus on achieving high performance in healthcare delivery, the absence of a consistently-applied approach to identify high performers impedes efforts to reliably compare, select and reward high-performing providers.

Objectives: Massage therapy has been proposed for painful conditions, but it can be difficult to understand the breadth and depth of evidence, as various painful conditions may respond differently to massage. The authors conducted an evidence mapping process and generated an "evidence map" to visually depict the distribution of evidence available for massage and various pain indications to identify gaps in evidence and to inform future research priorities. Design: The authors searched PubMed, Embase, and Cochrane for systematic reviews reporting pain outcomes for massage therapy. The authors assessed the quality of each review using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) criteria. The authors used a bubble plot to depict the number of included articles, pain indication, effect of massage for pain, and strength of findings for each included systematic review. Results: The authors identified 49 systematic reviews, of which 32 were considered high quality. Types of pain frequently included in systematic reviews were cancer pain, low back pain, and neck pain. High quality reviews concluded that there was low strength of evidence of potential benefits of massage for labor, shoulder, neck, low back, cancer, arthritis, postoperative, delayed onset muscle soreness, and musculoskeletal pain. Reported attributes of massage interventions include style of massage, provider, co-interventions, duration, and comparators, with 14 high-quality reviews reporting all these attributes in their review. Conclusion: Prior reviews have conclusions of low strength of evidence because few primary studies of large samples with rigorous methods had been conducted, leaving evidence gaps about specific massage type for specific pain. Primary studies often do not provide adequate details of massage therapy provided, limiting the extent to which reviews are able to draw conclusions about characteristics such as provider type.

BACKGROUND:The correct perioperative management of antiplatelet therapy (APT) in patients undergoing non-cardiac surgery (NCS) is often debated by clinicians. American College of Cardiology (ACC) and American Heart Association (AHA) guidelines recommend postponing elective NCS at least 3 months after stent implantation. Regardless of the timing of surgery, ACC/AHA guidelines recommend continuing at least ASA throughout the perioperative period and ideally continuing dual APT (DAPT) therapy "unless surgery demands discontinuation." The objective of this review was to ascertain the risks and benefits of APT in the perioperative period, to assess how these risks and benefits vary by APT management, and the significance of length of time since stent implantation before operative intervention. METHODS:PubMed, Web of Science, and Scopus were searched from inception through October 2017. Articles were included if patients were post PCI with stent placement (bare metal [BMS] or drug eluting [DES]), underwent elective NCS, and had rates of major adverse cardiac events (MACE) or bleeding events associated with pre and perioperative APT therapy. RESULTS:Of 4882 screened articles, we included 16 studies in the review (1 randomized controlled trial and 15 observational studies). Studies were small (150: n = 6). All studies included DES with 7 of 16 also including BMS. Average time from stent to NCS was variable ( 12 months: n = 6). At least six different APT strategies were described. Six studies further utilized bridging protocols using three different pharmacologic agents. Studies typically included multiple surgical fields with varying degrees of invasiveness. Across all APT strategies, rates of MACE/bleeding ranged from 0 to 21% and 0 to 22%. There was no visible trend in MACE/bleeding rates within a given APT strategy. Stratifying the articles by type of surgery, timing of discontinuation of APT therapy, bridging vs. no bridging, and time since stent placement did not help explain the heterogeneity. CONCLUSIONS:Evidence regarding perioperative APT management in patients with cardiac stents undergoing NCS is insufficient to guide practice. Other clinical factors may have a greater impact than perioperative APT management on MACE and bleeding events. SYSTEMATIC REVIEW REGISTRATION:PROSPERO CRD42016036607.

BackgroundRural areas have historically struggled with shortages of healthcare providers; however, advanced communication technologies have transformed rural healthcare, and practice in underserved areas has been recognized as a policy priority. This systematic review aims to assess reasons for current providers' geographic choices and the success of training programs aimed at increasing rural provider recruitment.MethodsThis systematic review (PROSPERO: CRD42015025403) searched seven databases for published and gray literature on the current cohort of US rural healthcare practitioners (2005 to March 2017). Two reviewers independently screened citations for inclusion; one reviewer extracted data and assessed risk of bias, with a senior systematic reviewer checking the data; quality of evidence was assessed using the GRADE approach.ResultsOf 7276 screened citations, we identified 31 studies exploring reasons for geographic choices and 24 studies documenting the impact of training programs. Growing up in a rural community is a key determinant and is consistently associated with choosing rural practice. Most existing studies assess physicians, and only a few are based on multivariate analyses that take competing and potentially correlated predictors into account. The success rate of placing providers-in-training in rural practice after graduation, on average, is 44% (range 20-84%; N = 31 programs). We did not identify program characteristics that are consistently associated with program success. Data are primarily based on rural tracks for medical residents.DiscussionThe review provides insight into the relative importance of demographic characteristics and motivational factors in determining which providers should be targeted to maximize return on recruitment efforts. Existing programs exposing students to rural practice during their training are promising but require further refining. Public policy must include a specific focus on the trajectory of the healthcare workforce and must consider alternative models of healthcare delivery that promote a more diverse, interdisciplinary combination of providers.

BackgroundThe prevalence of knee osteoarthritis (OA)/degenerative joint disease (DJD) is increasing in the USA. Systematic reviews of treatment efficacy and adverse events (AEs) of hyaluronic acid (HA) injections report conflicting evidence about the balance of benefits and harms. We review evidence on efficacy and AEs of intraarticular viscosupplementation with HA in older individuals with knee osteoarthritis and account for differences in these conclusions from another systematic review.MethodsWe searched PubMed and eight other databases and gray literature sources from 1990 to December 12, 2014. Double-blind placebo-controlled randomized controlled trials (RCTs) reporting functional outcomes or quality-of-life; RCTs and observational studies on delay/avoidance of arthroplasty; RCTs, case reports, and large cohort studies and case series assessing safety; and systematic reviews reporting on knee pain were considered for inclusion. A standardized, pre-defined protocol was applied by two independent reviewers to screen titles and abstracts, review full text, and extract details on study design, interventions, outcomes, and quality. We compared our results with those of a prior systematic review and found them to be discrepant; our analysis of why this discrepancy occurred is the focus of this manuscript.ResultsEighteen RCTs reported functional outcomes: pooled analysis of ten placebo-controlled, blinded trials showed a standardized mean difference of -0.23 (95 % confidence interval (CI) -0.45 to -0.01) favoring HA at 6 months. Studies reported few serious adverse events (SAEs) and no significant differences in non-serious adverse events (NSAEs) (relative risk (RR) [95 % CI] 1.03 [0.93-1.15] or SAEs (RR [95 % CI] 1.39 [0.78-2.47]). A recent prior systematic review reported similar functional outcomes, but significant SAE risk. Differences in SAE inclusion and synthesis accounted for the disparate conclusions.ConclusionsTrials show a small but significant effect of HA on function on which recent systematic reviews agree, but lack of AE synthesis standardization leads to opposite conclusions about the balance of benefits and harms. A limitation of the re-analysis of the prior systematic review is that it required imputation of missing data.

An editorial is presented which highlights the "Making Healthcare Safer IV" initiative, reviewing and updating patient safety practices after 25 years. It emphasizes the importance of adopting effective strategies for reducing medication errors, improving opioid stewardship, and minimizing adverse drug events.

BackgroundThis evidence map describes the volume and focus of Tai Chi research reporting health outcomes. Originally developed as a martial art, Tai Chi is typically taught as a series of slow, low-impact movements that integrate the breath, mind, and physical activity to achieve greater awareness and a sense of well-being.MethodsThe evidence map is based on a systematic review of systematic reviews. We searched 11 electronic databases from inception to February 2014, screened reviews of reviews, and consulted with topic experts. We used a bubble plot to graphically display clinical topics, literature size, number of reviews, and a broad estimate of effectiveness.ResultsThe map is based on 107 systematic reviews. Two thirds of the reviews were published in the last five years. The topics with the largest number of published randomized controlled trials (RCTs) were general health benefits (51 RCTs), psychological well-being (37 RCTs), interventions for older adults (31 RCTs), balance (27 RCTs), hypertension (18 RCTs), fall prevention (15 RCTs), and cognitive performance (11 RCTs). The map identified a number of areas with evidence of a potentially positive treatment effect on patient outcomes, including Tai Chi for hypertension, fall prevention outside of institutions, cognitive performance, osteoarthritis, depression, chronic obstructive pulmonary disease, pain, balance confidence, and muscle strength. However, identified reviews cautioned that firm conclusions cannot be drawn due to methodological limitations in the original studies and/or an insufficient number of existing research studies.ConclusionsTai Chi has been applied in diverse clinical areas, and for a number of these, systematic reviews have indicated promising results. The evidence map provides a visual overview of Tai Chi research volume and content.Systematic review registrationPROSPERO CRD42014009907.

ObjectiveValid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.MethodsApproaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.ResultsThe QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.ConclusionsWe developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

The expanded use of clinical process-of-care measures to assess the quality of health care in the context of public reporting and pay-for-performance applications has led to a desire to demonstrate the value of such efforts in terms of improved patient outcomes. The inability to observe associations between improved delivery of clinical processes and improved clinical outcomes in practice has raised concerns about the value of holding providers accountable for delivery of clinical processes of care. Analyses that attempt to investigate this relationship are fraught with many challenges, including selection of an appropriate outcome, the proximity of the outcome to the receipt of the clinical process, limited power to detect an effect, small expected effect sizes in practice, potential bias due to unmeasured confounding factors, and difficulties due to changes in measure specification over time. To avoid potentially misleading conclusions about an observed or lack of observed association between a clinical process of care and an outcome in the context of observational studies, individuals conducting and interpreting such studies should carefully consider, evaluate, and acknowledge these types of challenges.

BackgroundThere is an increasing push for 'evidence-based' decision making in global health policy circles. However, at present there are no agreed upon standards or guidelines for how to evaluate evidence in global health. Recent evaluations of existing evidence frameworks that could serve such a purpose have identified details of program context and project implementation as missing components needed to inform policy. We performed a pilot study to assess the current state of reporting of context and implementation in studies of global health interventions.MethodsWe identified three existing criteria sets for implementation reporting and selected from them 10 criteria potentially relevant to the needs of policy makers in global health contexts. We applied these 10 criteria to 15 articles included in the evidence base for three global health interventions chosen to represent a diverse set of advocated global health programs or interventions: household water chlorination, prevention of mother-to-child transmission of HIV, and lay community health workers to reduce child mortality. We used a good-fair-poor/none scale for the ratings.ResultsThe proportion of criteria for which reporting was poor/none ranged from 11% to 54% with an average of 30%. Eight articles had 'good' or 'fair' documentation for greater than 75% of criteria, while five articles had 'poor or none' documentation for 50% of criteria or more. Examples of good reporting were identified.ConclusionsReporting of context and implementation information in studies of global health interventions is mostly fair or poor, and highly variable. The idiosyncratic variability in reporting indicates that global health investigators need more guidance about what aspects of context and implementation to measure and how to report them. This lack of context and implementation information is a major gap in the evidence needed by global health policy makers to reach decisions.

BackgroundThe Health Information Technology for Economic and Clinical Health (HITECH) Act subsidizes implementation by hospitals of electronic health records with computerized provider order entry (CPOE), which may reduce patient injuries caused by medication errors (preventable adverse drug events, pADEs). Effects on pADEs have not been rigorously quantified, and effects on medication errors have been variable. The objectives of this analysis were to assess the effectiveness of CPOE at reducing pADEs in hospital-related settings, and examine reasons for heterogeneous effects on medication errors.MethodsArticles were identified using MEDLINE, Cochrane Library, Econlit, web-based databases, and bibliographies of previous systematic reviews (September 2013). Eligible studies compared CPOE with paper-order entry in acute care hospitals, and examined diverse pADEs or medication errors. Studies on children or with limited event-detection methods were excluded. Two investigators extracted data on events and factors potentially associated with effectiveness. We used random effects models to pool data.ResultsSixteen studies addressing medication errors met pooling criteria; six also addressed pADEs. Thirteen studies used pre-post designs. Compared with paper-order entry, CPOE was associated with half as many pADEs (pooled risk ratio (RR) = 0.47, 95% CI 0.31 to 0.71) and medication errors (RR = 0.46, 95% CI 0.35 to 0.60). Regarding reasons for heterogeneous effects on medication errors, five intervention factors and two contextual factors were sufficiently reported to support subgroup analyses or meta-regression. Differences between commercial versus homegrown systems, presence and sophistication of clinical decision support, hospital-wide versus limited implementation, and US versus non-US studies were not significant, nor was timing of publication. Higher baseline rates of medication errors predicted greater reductions (P

BackgroundSystematic reviews are a cornerstone of evidence-based medicine but are useful only if up-to-date. Methods for detecting signals of when a systematic review needs updating have face validity, but no proposed method has had an assessment of predictive validity performed.MethodsThe AHRQ Comparative Effectiveness Review program had produced 13 comparative effectiveness reviews (CERs), a subcategory of systematic reviews, by 2009, 11 of which were assessed in 2009 using a surveillance system to determine the degree to which individual conclusions were out of date and to assign a priority for updating each report. Four CERs were judged to be a high priority for updating, four CERs were judged to be medium priority for updating, and three CERs were judged to be low priority for updating. AHRQ then commissioned full update reviews for 9 of these 11 CERs. Where possible, we matched the original conclusions with their corresponding conclusions in the update reports, and compared the congruence between these pairs with our original predictions about which conclusions in each CER remained valid. We then classified the concordance of each pair as good, fair, or poor. We also made a summary determination of the priority for updating each CER based on the actual changes in conclusions in the updated report, and compared these determinations with the earlier assessments of priority.ResultsThe 9 CERs included 149 individual conclusions, 84% with matches in the update reports. Across reports, 83% of matched conclusions had good concordance, and 99% had good or fair concordance. The one instance of poor concordance was partially attributable to the publication of new evidence after the surveillance signal searches had been done. Both CERs originally judged as being low priority for updating had no substantive changes to their conclusions in the actual updated report. The agreement on overall priority for updating between prediction and actual changes to conclusions was Kappa = 0.74.ConclusionsThese results provide some support for the validity of a surveillance system for detecting signals indicating when a systematic review needs updating.

Our objective was to conduct a rigorous, structured process to create a new model of well-child care (WCC) in collaboration with a multisite community health center and 2 small, independent practices serving predominantly Medicaid-insured children. Working groups of clinicians, staff, and parents (called "Community Advisory Boards" [CABs]) used (1) perspectives of WCC stakeholders and (2) a literature review of WCC practice redesign to create 4 comprehensive WCC models for children ages 0 to 3 years. An expert panel, following a modified version of the Rand/UCLA Appropriateness Method, rated each model for potential effectiveness on 4 domains: (1) receipt of recommended services, (2) family-centeredness, (3) timely and appropriate follow-up, and (4) feasibility and efficiency. Results were provided to the CABs for selection of a final model to implement. The newly developed models rely heavily on a health educator for anticipatory guidance and developmental, behavioral, and psychosocial surveillance and screening. Each model allots a small amount of time with the pediatrician to perform a brief physical examination and to address parents' physical health concerns. A secure Web-based tool customizes the visit to parents' needs and facilitates previsit screening. Scheduled, non-face-to-face methods (text, phone) for parent communication with the health care team are also critical to these new models of care. A structured process that engages small community practices and community health centers in clinical practice redesign can produce comprehensive, site-specific, and innovative models for delivery of WCC. This process, as well as the models developed, may be applicable to other small practices and clinics interested in practice redesign.

BackgroundThere are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses.MethodsWe generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions.ResultsThe power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively.ConclusionsAlthough study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

BackgroundSystematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program.MethodsTwenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating.ResultsTwenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts.ConclusionIn this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.

BackgroundFall prevention interventions for community-dwelling older adults have been found to reduce falls in some research studies. However, wider implementation of fall prevention activities in routine care has yielded mixed results. We implemented a theory-driven program to improve care for falls at our Veterans Affairs healthcare facility. The first project arising from this program used a nurse advice telephone line to identify patients' risk factors for falls and to triage patients to appropriate services. Here we report the formative evaluation of this project.MethodsTo evaluate the intervention we: 1) interviewed patient and employee stakeholders, 2) reviewed participating patients' electronic health record data and 3) abstracted information from meeting minutes. We describe the implementation process, including whether the project was implemented according to plan; identify barriers and facilitators to implementation; and assess the incremental benefit to the quality of health care for fall prevention received by patients in the project. We also estimate the cost of developing the pilot project.ResultsThe project underwent multiple changes over its life span, including the addition of an option to mail patients educational materials about falls. During the project's lifespan, 113 patients were considered for inclusion and 35 participated. Patient and employee interviews suggested support for the project, but revealed that transportation to medical care was a major barrier in following up on fall risks identified by nurse telephone triage. Medical record review showed that the project enhanced usual medical care with respect to home safety counseling. We discontinued the program after 18 months due to staffing limitations and competing priorities. We estimated a cost of $9194 for meeting time to develop the project.ConclusionsThe project appeared feasible at its outset but could not be sustained past the first cycle of evaluation due to insufficient resources and a waning of local leadership support due to competing national priorities. Future projects will need both front-level staff commitment and prolonged high-level leadership involvement to thrive.

Abstract Background The evidence base for quality improvement (QI) interventions is expanding rapidly. The diversity of the initiatives and the inconsistency in labeling these as QI interventions makes it challenging for researchers, policymakers, and QI practitioners to access the literature systematically and to identify relevant publications. Methods We evaluated search strategies developed for MEDLINE (Ovid) and PubMed based on free text words, Medical subject headings (MeSH), QI intervention components, continuous quality improvement (CQI) methods, and combinations of the strategies. Three sets of pertinent QI intervention publications were used for validation. Two independent expert reviewers screened publications for relevance. We compared the yield, recall rate, and precision of the search strategies for the identification of QI publications and for a subset of empirical studies on effects of QI interventions. Results The search yields ranged from 2,221 to 216,167 publications. Mean recall rates for reference publications ranged from 5% to 53% for strategies with yields of 50,000 publications or fewer. The 'best case' strategy, a simple text word search with high face validity ('quality' AND 'improv*' AND 'intervention*') identified 44%, 24%, and 62% of influential intervention articles selected by Agency for Healthcare Research and Quality (AHRQ) experts, a set of exemplar articles provided by members of the Standards for Quality Improvement Reporting Excellence (SQUIRE) group, and a sample from the Cochrane Effective Practice and Organization of Care Group (EPOC) register of studies, respectively. We applied the search strategy to a PubMed search for articles published in 10 pertinent journals in a three-year period which retrieved 183 publications. Among these, 67% were deemed relevant to QI by at least one of two independent raters. Forty percent were classified as empirical studies reporting on a QI intervention. Conclusions The presented search terms and operating characteristics can be used to guide the identification of QI intervention publications. Even with extensive iterative development, we achieved only moderate recall rates of reference publications. Consensus development on QI reporting and initiatives to develop QI-relevant MeSH terms are urgently needed.

BackgroundImplementing quality improvement programs that require behavior change on the part of health care professionals and patients has proven difficult in routine care. Significant randomized trial evidence supports creating fall prevention programs for community-dwelling older adults, but adoption in routine care has been limited. Nationally-collected data indicated that our local facility could improve its performance on fall prevention in community-dwelling older people. We sought to develop a sustainable local fall prevention program, using theory to guide program development.MethodsWe planned program development to include important stakeholders within our organization. The theory-derived plan consisted of 1) an initial leadership meeting to agree on whether creating a fall prevention program was a priority for the organization, 2) focus groups with patients and health care professionals to develop ideas for the program, 3) monthly workgroup meetings with representatives from key departments to develop a blueprint for the program, 4) a second leadership meeting to confirm that the blueprint developed by the workgroup was satisfactory, and also to solicit feedback on ideas for program refinement.ResultsThe leadership and workgroup meetings occurred as planned and led to the development of a functional program. The focus groups did not occur as planned, mainly due to the complexity of obtaining research approval for focus groups. The fall prevention program uses an existing telephonic nurse advice line to 1) place outgoing calls to patients at high fall risk, 2) assess these patients' risk factors for falls, and 3) triage these patients to the appropriate services. The workgroup continues to meet monthly to monitor the progress of the program and improve it.ConclusionA theory-driven program development process has resulted in the successful initial implementation of a fall prevention program.

BackgroundWhile falls and urinary incontinence are prevalent among older patients, who sometimes rely on proxies to provide their health information, the validity of proxy reports of concern about falls and urinary incontinence remains unknown.MethodsTelephone interviews with 43 consecutive patients with falls or fear of falling and/or bothersome urinary incontinence and their proxies chosen by patients as most knowledgeable about their health. The questionnaire included items derived from the Medical Outcomes Study Short Form 12 (SF-12), a scale assessing concerns about urinary incontinence (UI), and a measure of fear of falling, the Falls Efficacy Scale (FES). Scores were estimated using items asking the proxy perspective (6 items from the SF-12, 10 items from a UI scale, and all 10 FES items). Proxy and patient scores were compared using intraclass correlation coefficients (ICC, one-way model). Variables associated with absolute agreement between patients and proxies were explored.ResultsPatients had a mean age of 81 years (range 75-93) and 67% were female while proxies had a mean age of 70 (range 42-87) and 49% were female. ICCs were 0.63 for the SF-12, 0.52 for the UI scale, and 0.29 for the FES. Proxies tended to understate patients' general health and incontinence concern, but overstate patients' concern about falling. Proxies who lived with patients and those who more often see patients more closely reflected patient FES scores compared to those who lived apart or those who saw patients less often. Internal consistency reliability of proxy responses was 0.62 for the SF-12, 0.86 for the I-QOL, and 0.93 for the FES. In addition, construct validity of the proxy FES scale was supported by greater proxy-perceived fear of falling for patients who received medical care after a fall during the past 12 months (p < .05).ConclusionCaution should be exercised when using proxies as a source of information about older patients' health perceptions. Questions asking about proxies' views yield suboptimal agreement with patient responses. However, proxy scales of UI and fall concern are internally consistent and may provide valid independent information.