Search

We revisit the results of the recent Reproducibility Project: Psychology by the Open Science Collaboration. We compute Bayes factors—a quantity that can be used to express comparative evidence for an hypothesis but also for the null hypothesis—for a large subset (N = 72) of the original papers and their corresponding replication attempts. In our computation, we take into account the likely scenario that publication bias had distorted the originally published results. Overall, 75% of studies gave qualitatively similar results in terms of the amount of evidence provided. However, the evidence was often weak (i.e., Bayes factor < 10). The majority of the studies (64%) did not provide strong evidence for either the null or the alternative hypothesis in either the original or the replication, and no replication attempts provided strong evidence in favor of the null. In all cases where the original paper provided strong evidence but the replication did not (15%), the sample size in the replication was smaller than the original. Where the replication provided strong evidence but the original did not (10%), the replication sample size was larger. We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. We further conclude that traditional sample sizes are insufficient and that a more widespread adoption of Bayesian methods is desirable.

Introduction: Reproducible research is increasingly gaining interest in the research community. Automating the production of research manuscript tables from statistical software can help increase the reproducibility of findings. Logistic regression is used in studying disease prevalence and associated factors in epidemiological studies and can be easily performed using widely available software including SAS, SUDAAN, Stata or R. However, output from these software must be processed further to make it readily presentable. There exists a number of procedures developed to organize regression output, though many of them suffer limitations of flexibility, complexity, lack of validation checks for input parameters, as well as inability to incorporate survey design. Methods: We developed a SAS macro, %svy_logistic_regression, for fitting simple and multiple logistic regression models. The macro also creates quality publication-ready tables using survey or non-survey data which aims to increase transparency of data analyses. It further significantly reduces turn-around time for conducting analysis and preparing output tables while also addressing the limitations of existing procedures. In addition, the macro allows for user-specific actions to handle missing data as well as use of replication-based variance estimation methods. Results: We demonstrate the use of the macro in the analysis of the 2013–2014 National Health and Nutrition Examination Survey (NHANES), a complex survey designed to assess the health and nutritional status of adults and children in the United States. The output presented here is directly from the macro and is consistent with how regression results are often presented in the epidemiological and biomedical literature, with unadjusted and adjusted model results presented side by side. Conclusions: The SAS code presented in this macro is comprehensive, easy to follow, manipulate and to extend to other areas of interest. It can also be incorporated quickly by the statistician for immediate use. It is an especially valuable tool for generating quality, easy to review tables which can be incorporated directly in a publication. [ABSTRACT FROM AUTHOR]

Shear-wave electrography (SWE) is a method used to assess tissue elasticity. Recently, it has been used to assess muscle stiffness, but the reliability of SWE for this purpose has not been thoroughly investigated. The purpose of this study was to evaluate the repeatability and reproducibility of SWE on porcine meat specimens and the human biceps femoris muscle. Measurements on meat specimens (n = 20) were performed by three raters and with a custom-built device that allowed constant application force. Measurements on human participants (n = 20) were performed by two raters in relaxed and stretched muscle positions on two visits. Most aspects of repeatability and reproducibility were good or high, with intra-class correlation coefficient (ICC) values above 0.70. Minimal detectable changes were lower in a relaxed (6–10%) than stretched (15-16%) muscle position. In conclusion, SWE is a reliable tool for assessing muscle stiffness if the muscle is examined in relaxed condition, while changing the force applied with the probe for as little as 1.5 N results in significantly lower repeatability. [ABSTRACT FROM AUTHOR]

Background: Near infrared spectroscopy (NIRS) is currently complementing techniques to age-grade mosquitoes. NIRS classifies lab-reared and semi-field raised mosquitoes into < or ≥ 7 days old with an average accuracy of 80%, achieved by training a regression model using partial least squares (PLS) and interpreted as a binary classifier. Methods and findings: We explore whether using an artificial neural network (ANN) analysis instead of PLS regression improves the current accuracy of NIRS models for age-grading malaria transmitting mosquitoes. We also explore if directly training a binary classifier instead of training a regression model and interpreting it as a binary classifier improves the accuracy. A total of 786 and 870 NIR spectra collected from laboratory reared An. gambiae and An. arabiensis, respectively, were used and pre-processed according to previously published protocols. The ANN regression model scored root mean squared error (RMSE) of 1.6 ± 0.2 for An. gambiae and 2.8 ± 0.2 for An. arabiensis; whereas the PLS regression model scored RMSE of 3.7 ± 0.2 for An. gambiae, and 4.5 ± 0.1 for An. arabiensis. When we interpreted regression models as binary classifiers, the accuracy of the ANN regression model was 93.7 ± 1.0% for An. gambiae, and 90.2 ± 1.7% for An. arabiensis; while PLS regression model scored the accuracy of 83.9 ± 2.3% for An. gambiae, and 80.3 ± 2.1% for An. arabiensis. We also find that a directly trained binary classifier yields higher age estimation accuracy than a regression model interpreted as a binary classifier. A directly trained ANN binary classifier scored an accuracy of 99.4 ± 1.0 for An. gambiae and 99.0 ± 0.6% for An. arabiensis; while a directly trained PLS binary classifier scored 93.6 ± 1.2% for An. gambiae and 88.7 ± 1.1% for An. arabiensis. We further tested the reproducibility of these results on different independent mosquito datasets. ANNs scored higher estimation accuracies than when the same age models are trained using PLS. Regardless of the model architecture, directly trained binary classifiers scored higher accuracies on classifying age of mosquitoes than regression models translated as binary classifiers. Conclusion: We recommend training models to estimate age of An. arabiensis and An. gambiae using ANN model architectures (especially for datasets with at least 70 mosquitoes per age group) and direct training of binary classifier instead of training a regression model and interpreting it as a binary classifier. [ABSTRACT FROM AUTHOR]

The contemporary scientific community places a growing emphasis on the reproducibility of research. The projects R package is a free, open-source package created in the interest of facilitating reproducible research workflows. It adds to existing software tools for reproducible research and introduces several practical features that are helpful for scientists and their collaborative research teams. For each individual project, it supplies a framework for storing raw and cleaned study data sets, and it provides script templates for protocol creation, data cleaning, data analysis and manuscript development. Internal databases of project and author information are generated and displayed, and manuscript title pages containing author lists and their affiliations are automatically generated from the internal database. File management tools allow teams to organize multiple projects. When used on a shared file system, multiple researchers can harmoniously contribute to the same project in a less punctuated manner, reducing the frequency of misunderstandings and the need for status updates. [ABSTRACT FROM AUTHOR]

Background: Two manufacturers, Maxim Biomedical and Sedia Biosciences Corporation, supply CDC-approved versions of the HIV-1 Limiting Antigen Avidity EIA (LAg) for detecting ‘recent’ HIV infection in cross-sectional incidence estimation. This study assesses and compares the performance of the two assays for incidence surveillance. Methods: We ran both assays on a panel of 2,500 well-characterized HIV-1-infected specimens. We analysed concordance of assay results, assessed reproducibility using repeat testing and estimated mean durations of recent infection (MDRIs) and false-recent rates (FRRs) for a range of normalized optical density (ODn) thresholds, alone and in combination with viral load thresholds. We defined three hypothetical surveillance scenarios, similar to the Kenyan and South African epidemics, and a concentrated epidemic. These scenarios allowed us to evaluate the precision of incidence estimates obtained by means of various recent infection testing algorithms (RITAs) based on each of the two assays. Results: The Maxim assay produced lower ODn values than the Sedia assay on average, largely as a result of higher calibrator readings (mean OD of 0.749 vs. 0.643), with correlation of normalized readings lower (R2 = 0.908 vs. R2 = 0.938). Reproducibility on blinded control specimens was slightly better for Maxim. The MDRI of a Maxim-based algorithm at the ‘standard’ threshold (ODn ≤1.5 & VL >1,000) was 201 days (95% CI: 180,223) and for Sedia 171 (152,191). The difference Differences in MDRI were estimated at 32.7 (22.9,42.8) and 30.9 days (21.7,40.7) for the two algorithms, respectively. Commensurately, the Maxim algorithm had a higher FRR in treatment-naive subjects (1.7% vs. 1.1%). The two assays produced similar precision of incidence estimates in the three surveillance scenarios. Conclusions: Differences between the assays can be primarily attributed to the calibrators supplied by the manufacturers. Performance for surveillance was extremely similar, although different thresholds were optimal (i.e. produced the lowest variance of incidence estimates) and at any given ODn threshold, different estimates of MDRI and FRR were obtained. The two assays cannot be treated as interchangeable: assay and algorithm-specific performance characteristic estimates must be used for survey planning and incidence estimation. [ABSTRACT FROM AUTHOR]

Background: Quantifying reproducibility of native T1 and T2 mapping over a long period (> 1 year) is necessary to assess whether changes in T1 and T2 over repeated sessions in a longitudinal study are associated with variability due to underlying tissue composition or technical confounders. Objectives: To carry out a single-center phantom study to 1) investigate measurement reproducibility of slice-interleaved T1 (STONE) and T2 mapping over 20 months, 2) quantify sources of variability, and 3) compare reproducibility and measurements against reference spin-echo measurements. Methods: MR imaging was performed on a 1.5 Tesla Philips Achieva scanner every 2–3 weeks over 20 months using the T1MES phantom. In each session, slice-interleaved T1 and T2 mapping was repeated 3 times for 5 slices, and maps were reconstructed using both 2-parameter and 3-parameter fit models. Reproducibility between sessions, and repeatability between repetitions and slices were evaluated using coefficients of variation (CV). Different sources of variability were quantified using variance decomposition analysis. The slice-interleaved measurement was compared to the spin-echo reference and MOLLI. Results: Slice-interleaved T1 had excellent reproducibility and repeatability with a CV < 2%. The main sources of T1 variability were temperature in 2-parameter maps, and slice in 3-parameter maps. Superior between-session reproducibility to the spin-echo T1 was shown in 2-parameter maps, and similar reproducibility in 3-parameter maps. Superior reproducibility to MOLLI T1 was also shown. Similar measurements to the spin-echo T1 were observed with linear regression slopes of 0.94–0.99, but slight underestimation. Slice-interleaved T2 showed good reproducibility and repeatability with a CV < 7%. The main source of T2 variability was slice location/orientation. Between-session reproducibility was lower than the spin-echo T2 reference and showed good measurement agreement with linear regression slopes of 0.78–1.06. Conclusions: Slice-interleaved T1 and T2 mapping sequences yield excellent long-term reproducibility over 20 months. [ABSTRACT FROM AUTHOR]

Introduction: It is crucial to understand the factors that introduce variability before applying metabolomics to clinical and biomarker research. Objectives: We quantified technical and biological variability of both fasting and postprandial metabolite concentrations measured using 1H NMR spectroscopy in plasma samples. Methods: In the Netherlands Epidemiology of Obesity study (n = 6,671), 148 metabolite concentrations (101 metabolites belonging to lipoprotein subclasses) were measured under fasting and postprandial states (150 minutes after a mixed liquid meal). Technical variability was evaluated among 265 fasting and 851 postprandial samples, with the identical blood plasma sample being measured twice by the same laboratory protocol. Biological reproducibility was assessed by measuring 165 individuals twice across time for evaluation of short- (<6 months) and long-term (>3 years) biological variability. Intra-class coefficients (ICCs) were used to assess variability. The ICCs of the fasting metabolites were compared with the postprandial metabolites using two-sided paired Wilcoxon test separately for short- and long-term measurements. Results: Both fasting and postprandial metabolite concentrations showed high technical reproducibility using 1H NMR spectroscopy (median ICC = 0.99). Postprandial metabolite concentrations revealed slightly higher ICC scores than fasting ones in short-term repeat measures (median ICC in postprandial and fasting metabolite concentrations 0.72 versus 0.67, Wilcoxon p-value = 8.0×10−14). Variability did not increase further in a long-term repeat measure, with median ICC in postprandial of 0.64 and in fasting metabolite concentrations 0.66. Conclusion: Technical reproducibility is excellent. Biological reproducibility of postprandial metabolite concentrations showed a less or equal variability than fasting metabolite concentrations over time. [ABSTRACT FROM AUTHOR]

Directly observed therapy (DOT) is almost universally used for the treatment of TB. Several meta-analyses using different methods have assessed the effectiveness of DOT compared to self-administered therapy (SAT). The results of these meta-analyses often conflict with some concluding DOT is superior and others that there is little or no difference. Meta-analyses can guide policymaking, but such analyses must be reliable. To assess the validity of a previous meta-analysis, we tried to reproduce it. We encountered problems with the previous analysis that did not allow for a meaningful reproduction. We describe the issues we encountered here. We then performed a new meta-analysis comparing the treatment outcomes of adults given treatment with SAT versus DOT. Outcomes in the new analysis are loss to follow-up, treatment failure, cure, treatment completed, and all-cause mortality. All data, documentation, and code used to generate our results is provided. Our new analysis included four randomized and three observational studies with 1603 and 1626 individuals respectively. The pooled relative risks (RR) are as follows: Lost to follow-up (RR = 1.2, 95% CI 0.9, 1.7), Treatment Failure (RR = 1.1, 95% CI 0.6, 2), Cure (RR = 0.9, 95% CI 0.8, 1.1), Treatment Completion (RR = 1, 95% CI 0.9, 1.1), Mortality (RR = 0.9, 95% CI 0.6, 1.3). Based on data from our new meta-analysis, the magnitude of the difference between DOT and SAT for all reported outcomes is small, and none of the differences are statistically significant. [ABSTRACT FROM AUTHOR]

Consistent confirmations obtained independently of each other lend credibility to a scientific result. We refer to results satisfying this consistency as reproducible and assume that reproducibility is a desirable property of scientific discovery. Yet seemingly science also progresses despite irreproducible results, indicating that the relationship between reproducibility and other desirable properties of scientific discovery is not well understood. These properties include early discovery of truth, persistence on truth once it is discovered, and time spent on truth in a long-term scientific inquiry. We build a mathematical model of scientific discovery that presents a viable framework to study its desirable properties including reproducibility. In this framework, we assume that scientists adopt a model-centric approach to discover the true model generating data in a stochastic process of scientific discovery. We analyze the properties of this process using Markov chain theory, Monte Carlo methods, and agent-based modeling. We show that the scientific process may not converge to truth even if scientific results are reproducible and that irreproducible results do not necessarily imply untrue results. The proportion of different research strategies represented in the scientific population, scientists’ choice of methodology, the complexity of truth, and the strength of signal contribute to this counter-intuitive finding. Important insights include that innovative research speeds up the discovery of scientific truth by facilitating the exploration of model space and epistemic diversity optimizes across desirable properties of scientific discovery. [ABSTRACT FROM AUTHOR]

Background: The Tuberculin Skin Test (TST) is a relatively simple test for detecting latent tuberculosis infection (LTBI) but requires regular quality assurance to ensure proper technique for administration and reading. The objective of this study was to estimate the accuracy and reproducibility of an mhealth approach (the mTST) to measure the size of swelling immediately following TST administration (TST injection bleb) and after 48–72 hours (TST induration). Methods: Five non-clinical and one clinical reviewer measured the size of TST injection blebs, and TST indurations using smartphone acquired photos of sites of TST administration and readings in patients, or saline injections in volunteers. The reference standard was the onsite measurement (measured by an experienced TB nurse) of the actual TST injection bleb, or induration. Agreement of reviewers’ measurements with the reference standard, as well as agreement within and between reviewers, was estimated using Cohen's kappa coefficient. Results: Using the mTST method to assess bleb size in 64 photos of different TST injections, agreement between reviewers, and the reference standard was very good to excellent (κ ranged from 0.75 to 0.87), and within-reviewer reproducibility of readings was excellent (κ ranged from 0.86 to 0.96). Using the mTST method to assess TST induration in 72 photos, reviewers were able to detect no induration (<5mm) and induration of 15mm or greater with accuracy of 95% and 92% respectively, but accuracy was only 20% and 77% for reactions of 5-9mm and 10-14mm respectively. Conclusion: The mTST approach appears to be a reliable tool to assess TST administration. The mTST approach was accurate to read indurations of 0-4mm or 15+mm, but less accurate for reactions of 5-14mm. We believe the mTST approach could be useful for training and quality assurance in locations where on-site supervision is not possible. [ABSTRACT FROM AUTHOR]

Big biomedical data create exciting opportunities for discovery, but make it difficult to capture analyses and outputs in forms that are findable, accessible, interoperable, and reusable (FAIR). In response, we describe tools that make it easy to capture, and assign identifiers to, data and code throughout the data lifecycle. We illustrate the use of these tools via a case study involving a multi-step analysis that creates an atlas of putative transcription factor binding sites from terabytes of ENCODE DNase I hypersensitive sites sequencing data. We show how the tools automate routine but complex tasks, capture analysis algorithms in understandable and reusable forms, and harness fast networks and powerful cloud computers to process data rapidly, all without sacrificing usability or reproducibility—thus ensuring that big data are not hard-to-(re)use data. We evaluate our approach via a user study, and show that 91% of participants were able to replicate a complex analysis involving considerable data volumes. [ABSTRACT FROM AUTHOR]

To what extent is it possible to use machine learning to predict the outcome of a relationship, based on the personality of both partners? In the present study, relationship satisfaction, conflicts, and separation (intents) of 192 partners four years after the completion of questionnaires concerning their personality traits was predicted. A 10x10-fold cross-validation was used to ensure that the results of the linear regression models are reproducible. The findings indicate that machine learning techniques can improve the prediction of relationship quality (37% of variance explained), and that the perceived relationship quality of a partner is mostly dependent on his or her own individual personality traits. Additionally, the influences of different sets of variables on predictions are shown: partner and similarity effects did not incrementally predict relationship quality beyond actor effects and general personality traits predicted relationship quality less strongly than relationship-related personality. [ABSTRACT FROM AUTHOR]

Background: Changes in the microcirculation may be used as a surrogate outcome in studies on cardiovascular disease. We assessed the reliability characteristics of the sublingual microcirculation parameters Vascular Density (VD), Red Blood Cell Filling (RBCF), and Perfused Boundary Region (PBR) as obtained by sidestream darkfield imaging. Methods: For each of the three parameters, the variance components of measurement, the Intraclass Correlation Coefficient (ICC), the Standard Error of Measurement, and the limits of agreement were estimated for the intra-rater setting (N = 50) and the inter-rater setting (N = 48). Subsequently, as a proof of concept, the reliability measures were used for a power analysis to design studies to evaluate the effect of acute stimuli–i.e. having a meal (N = 50) and cigarette smoking (N = 21) on the three parameters. Results: Reproducibility was poor for all three parameters. The intra-rater ICC for 2 measurements was 0.28 (95% CI: 0.04, 0.53) for the VD, 0.51 (95% CI: 0.27, 0.69) for the RBCF, and 0.33 (95% CI: 0.08–0.56) for the PBR. The standard errors of measurement and the limits of agreement for all three parameters were larger than most statistically significant intra-individual or inter-individual differences reported in previous studies. The proofs of concept showed that sample sizes in excess of 600 subjects are necessary to reach statistical significance for the observed effects of having a meal or smoking on VD and PBR. Conclusions: The reliability of the three sublingual microcirculation parameters in their current form appears to be low and a large sample size is advisable for their use in conditions similar to those we describe. [ABSTRACT FROM AUTHOR]

Objectives: To assess the influence of gray-level discretization on inter- and intra-observer reproducibility of texture radiomics features on clinical MR images. Materials and methods: We studied two independent MRI datasets of 74 lacrymal gland tumors and 30 breast lesions from two different centers. Two pairs of readers performed three two-dimensional delineations for each dataset. Texture features were extracted using two radiomics softwares (Pyradiomics and an in-house software). Reproducible features were selected using a combination of intra-class correlation coefficient (ICC) and concordance and coherence coefficient (CCC) with 0.8 and 0.9 as thresholds, respectively. We tested six absolute and eight relative gray-level discretization methods and analyzed the distribution and highest number of reproducible features obtained for each discretization. We also analyzed the number of reproducible features extracted from computer simulated delineations representative of inter-observer variability. Results: The gray-level discretization method had a direct impact on texture feature reproducibility, independent of observers, software or method of delineation (simulated vs. human). The absolute discretization consistently provided statistically significantly more reproducible features than the relative discretization. Varying the bin number of relative discretization led to statistically significantly more variable results than varying the bin size of absolute discretization. Conclusions: When considering inter-observer reproducible results of MRI texture radiomics features, an absolute discretization should be favored to allow the extraction of the highest number of potential candidates for new imaging biomarkers. Whichever the chosen method, it should be systematically documented to allow replicability of results. [ABSTRACT FROM AUTHOR]

This research-on-research study describes efforts to develop non-Cochrane systematic reviews (SRs) by analyzing demographical and time-course collaborations between international institutions using protocols registered in the International Prospective Register of Systematic Reviews (PROSPERO) or published in scientific journals. We have published an a priori protocol to develop this study. Protocols published in scientific journals were searched using the MEDLINE and Embase databases; the query terms “Systematic review” [Title] AND “protocol” [Title] were searched from February 2011 to December 2017. Protocols registered at PROSPERO during the same period were obtained by web scraping all non-Cochrane records with a Python script. After excluding protocols that had a fulfillment or duplication rate of less than 90%, they were classified as published “only in PROSPERO”, “only in journals”, or in “journals and PROSPERO”. Results of data and metadata extraction using text mining processes were curated by two reviewers. These Datasets and R scripts are freely available to facilitate reproducibility. We obtained 20,814 protocols of non-Cochrane SRs. While “unique protocols” by reviewers’ institutions from 60 countries were the most frequent, a median of 6 (2-150) institutions from 130 different countries were involved in the preparation of “collaborative protocols”. The highest Ranked countries involved in overall protocol production were the UK, the U.S., Australia, Brazil, China, Canada, the Netherlands, Germany, and Italy. Most protocols were registered only in PROSPERO. However, the number of protocols published in scientific journals (924) or in both PROSPERO and journals (807) has increased over the last three years. Syst Rev and BMJ Open published more than half of the total protocols. While the more productive countries were involved in “unique” and “collaborative protocols”, less productive countries only participated in “collaborative protocols” that were mainly published in PROSPERO. Our results suggest that, although most countries were involved in solitary production of protocols for non-Cochrane SRs during the study period, it would be useful to develop new strategies to promote international collaborations, especially with less productive countries. [ABSTRACT FROM AUTHOR]

Objective: Clinical code sets are vital to research using routinely-collected electronic healthcare data. Existing code set engineering methods pose significant limitations when considering reproducible research. To improve the transparency and reusability of research, these code sets must abide by FAIR principles; this is not currently happening. We propose ‘term sets’, an equivalent alternative to code sets that are findable, accessible, interoperable and reusable. Materials and methods: We describe a new code set representation, consisting of natural language inclusion and exclusion terms (term sets), and explain its relationship to code sets. We formally prove that any code set has a corresponding term set. We demonstrate utility by searching for recently published code sets, representing them as term sets, and reporting on the number of inclusion and exclusion terms compared with the size of the code set. Results: Thirty-one code sets from 20 papers covering diverse disease domains were converted into term sets. The term sets were on average 74% the size of their equivalent original code set. Four term sets were larger due to deficiencies in the original code sets. Discussion: Term sets can concisely represent any code set. This may reduce barriers for examining and reusing code sets, which may accelerate research using healthcare databases. We have developed open-source software that supports researchers using term sets. Conclusion: Term sets are independent of clinical code terminologies and therefore: enable reproducible research; are resistant to terminology changes; and are less error-prone as they are shorter than the equivalent code set. [ABSTRACT FROM AUTHOR]

Mathematical models for signaling pathways are helpful for understanding molecular mechanism in the pathways and predicting dynamic behavior of the signal activity. To analyze the robustness of such models, local sensitivity analysis has been implemented. However, such analysis primarily focuses on only a certain parameter set, even though diverse parameter sets that can recapitulate experiments may exist. In this study, we performed sensitivity analysis that investigates the features in a system considering the reproducible and multiple candidate values of the model parameters to experiments. The results showed that although different reproducible model parameter values have absolute differences with respect to sensitivity strengths, specific trends of some relative sensitivity strengths exist between reactions regardless of parameter values. It is suggested that (i) network structure considerably influences the relative sensitivity strength and (ii) one might be able to predict relative sensitivity strengths specified in the parameter sets employing only one of the reproducible parameter sets. [ABSTRACT FROM AUTHOR]

Background: Cardiovascular magnetic resonance feature tracking (CMR-FT) is increasingly used for myocardial deformation assessment including ventricular strain, showing prognostic value beyond established risk markers if used in experienced centres. Little is known about the impact of appropriate training on CMR-FT performance. Consequently, this study aimed to evaluate the impact of training on observer variance using different commercially available CMR-FT software. Methods: Intra- and inter-observer reproducibility was assessed prior to and after dedicated one-hour observer training. Employed FT software included 3 different commercially available platforms (TomTec, Medis, Circle). Left (LV) and right (RV) ventricular global longitudinal as well as LV circumferential and radial strains (GLS, GCS and GRS) were studied in 12 heart failure patients and 12 healthy volunteers. Results: Training improved intra- and inter-observer reproducibility. GCS and LV GLS showed the highest reproducibility before (ICC >0.86 and >0.81) and after training (ICC >0.91 and >0.92). RV GLS and GRS were more susceptible to tracking inaccuracies and reproducibility was lower. Inter-observer reproducibility was lower than intra-observer reproducibility prior to training with more pronounced improvements after training. Before training, LV strain reproducibility was lower in healthy volunteers as compared to patients with no differences after training. Whilst LV strain reproducibility was sufficient within individual software solutions inter-software comparisons revealed considerable software related variance. Conclusion: Observer experience is an important source of variance in CMR-FT derived strain assessment. Dedicated observer training significantly improves reproducibility with most profound benefits in states of high myocardial contractility and potential to facilitate widespread clinical implementation due to optimized robustness and diagnostic performance. [ABSTRACT FROM AUTHOR]

Recent years have seen a surge of studies in machine learning in health and biomedicine, driven by digitalization of healthcare environments and increasingly accessible computer systems for conducting analyses. Many of us believe that these developments will lead to significant improvements in patient care. Like many academic disciplines, however, progress is hampered by lack of code and data sharing. In bringing together this PLOS ONE collection on machine learning in health and biomedicine, we sought to focus on the importance of reproducibility, making it a requirement, as far as possible, for authors to share data and code alongside their papers. [ABSTRACT FROM AUTHOR]

Objective: To assess the reproducibility of adverse event evaluation by a medical record review committee. Design: Cross-sectional reanalysis of medical records. Intervention: Reviewers re-examined fifty medical records of deceased patients regarding the presence of adverse events, their potential preventability and their possible contribution to death. Also we investigated the root causes of the preventable AEs. Differences between the first and second assessment were calculated. Results: The Kappa on the presence of an adverse event was 0.64 and 0.32 for the potential preventability. The intrarater agreement showed a Kappa of 0.61 on the adverse event presence and 0.64 for the potential preventability. Interrater agreement showed a Kappa of 0.66 for the adverse event presence and 0.03 for the potential preventability. Conclusion: We found a fair reproducibility for the detection of adverse events, but a poor reproducibility for the potential preventability. Possibly this was caused by lack of a definition for the preventability of adverse events. We think giving feedback to professionals using the results of medical record review remains valuable, but an improvement of its reproducibility is essential. To our opinion an international consensus on what exactly constitutes preventability of adverse events and agreement on a definition is necessary. This would result in more comparable studies in this field and could then be more informative on the ideal procedure to avoid certain potentially preventable adverse events in the future. [ABSTRACT FROM AUTHOR]

Measurement of pH in IVF-media using the blood gas analyzer (BGA) requires validation, because IVF-media is outside the intended scope of the BGA. To determine whether the Siemens Rapidpoint 500 BGA is suitable for pH measurements in IVF-media this study will validate the BGA and assess its accuracy. In this method comparison study, the pH of over three hundred IVF-media samples was measured with the BGA and a pH electrode (Hanna pH checker). The precision of both the BGA and the pH electrode were excellent (coefficient variation <1.4%). However, the closeness of agreement between measured values of both devices were not equivalent to each other in the tested IVF-media, showing 15% to 85% accordance between devices. The pH measured with the blood gas analyzer was also significantly higher in the tested media, compared to that measured by the pH electrode. One of the tested media did not reach its target pH when it was measured with the BGA, even at 9% CO2. The results show that the validated blood gas analyzer produces excellent results in terms of precision but not in terms of accuracy. Inaccurate measurement may lead to misinterpretation of results and consequently to suboptimal culture conditions. Therefore, each laboratory is encouraged to perform a validation of their BGA. [ABSTRACT FROM AUTHOR]

Background: This study examined the short-term reproducibility of non-invasive estimates of central and peripheral blood pressure and markers of central systolic loading (augmentation index [AIx; a measure of central systolic loading] and AIx75 [AIx standardised to 75 b·min-1 heart rate]) and the effect of posture and fasting state on these variables in patients with acute stroke. Methods: Twenty-two acute stroke patients (72 ± 10y) had blood pressure measured using the SphygmoCor XCEL in supine and seated postures and whilst fasted and non-fasted. Results: Acceptable short-term reproducibility (ICC >0.75) was reported for all peripheral and central variables in all conditions (ICC = 0.77–0.90) and for AIx and AIx75 in both fasted postures (ICC = 0.78–0.81). Food consumption significantly lowered all blood pressures (p <0.05; η2p = 0.20–0.55). The seated posture resulted in a significantly greater AIx than supine (p <0.05; η2p = 0.22). Fasting state had significant main effects on AIx and AIx75 (p <0.05; η2p = 0.14–0.22). Conslusions: Oscillometric estimates of central blood pressure have high short-term reproducibility in different postures and fasting states but markers of systolic load should be assessed whilst fasted. Fasting state has a large effect on central and peripheral blood pressures and on measures of systolic loading. It is important for clinicians to be aware of optimal assessment conditions without this impacting on patient wellbeing. Trial registration: Clinical trial registry name: . [ABSTRACT FROM AUTHOR]

In recent years there is increasing evidence that elevated progesterone levels during ovarian stimulation for IVF / ICSI have a negative impact on the ART-outcome. However, different progesterone assays were used in the previous studies and different assays might produce varying results. This retrospective study evaluated the reproducibility and reliability of different progesterone assays with a special focus on progesterone levels below 1.5 ng/ml, as this range is crucial for early detection of progesterone rise during ovarian stimulation for IVF. A total of 413 blood samples were categorized in different progesterone ranges and whether they were retrieved on the day of final oocyte maturation and the results were compared regarding their reproducibility and reliability. To compare the reproducibility between the different progesterone assays, the Intraclass Correlation Coefficient (ICC) was calculated and interpretation of the ICC results was done according to Cicchetti, ranging from poor to excellent. The correlation of the assays was excellent when all samples were compared including samples retrieved on day of final oocyte maturation, however in the ranges of progesterone levels 1.0 ng/ml to < 1.5 ng/ml, 0.8 ng/ml to < 1.0 ng/ml and < 0.8 ng/ml, the ICC varied between poor and excellent. The assays “gen III” and “Architect” showed an excellent reproducibility of progesterone results throughout all ranges of progesterone levels. This analysis demonstrates, that different progesterone assays have a limited reproducibility and that the results depend on the assay used and the range of progesterone level. This fact leads to two important conclusions. Firstly the limited reproducibility might lead to substantially different treatment decisions in ovarian stimulation treatment for IVF and secondly critical interpretation of thresholds, provided by meta-analysis, is crucial despite the risk that the so far gained clinical experience might become irrelevant and has to be adjusted to the results, obtained by each assay. [ABSTRACT FROM AUTHOR]

Mammalian cell culture is foundational to biomedical research, and the reproducibility of research findings across the sciences is drawing increasing attention. While many components contribute to reproducibility, the reporting of factors that impact oxygen delivery in the general biomedical literature has the potential for both significant impact, and immediate improvement. The relationship between the oxygen consumption rate of cells and the diffusive delivery of oxygen through the overlying medium layer means parameters such as medium depth and cell type can cause significant differences in oxygenation for cultures nominally maintained under the same conditions. While oxygenation levels are widely understood to significantly impact the phenotype of cultured cells in the abstract, in practise the importance of the above parameters does not appear to be well recognized in the non-specialist research community. On analyzing two hundred articles from high-impact journals we find a large majority missing at least one key piece of information necessary to ensure consistency in replication. We propose that explicitly reporting these values should be a requirement for publication. [ABSTRACT FROM AUTHOR]

Purpose: To evaluate the uncertainty of radiomics features from contrast-enhanced breath-hold helical CT scans of non-small cell lung cancer for both manual and semi-automatic segmentation due to intra-observer, inter-observer, and inter-software reliability. Methods: Three radiation oncologists manually delineated lung tumors twice from 10 CT scans using two software tools (3D-Slicer and MIM Maestro). Additionally, three observers without formal clinical training were instructed to use two semi-automatic segmentation tools, Lesion Sizing Toolkit (LSTK) and GrowCut, to delineate the same tumor volumes. The accuracy of the semi-automatic contours was assessed by comparison with physician manual contours using Dice similarity coefficients and Hausdorff distances. Eighty-three radiomics features were calculated for each delineated tumor contour. Informative features were identified based on their dynamic range and correlation to other features. Feature reliability was then evaluated using intra-class correlation coefficients (ICC). Feature range was used to evaluate the uncertainty of the segmentation methods. Results: From the initial set of 83 features, 40 radiomics features were found to be informative, and these 40 features were used in the subsequent analyses. For both intra-observer and inter-observer reliability, LSTK had higher reliability than GrowCut and the two manual segmentation tools. All observers achieved consistently high ICC values when using LSTK, but the ICC value varied greatly for each observer when using GrowCut and the manual segmentation tools. For inter-software reliability, features were not reproducible across the software tools for either manual or semi-automatic segmentation methods. Additionally, no feature category was found to be more reproducible than another feature category. Feature ranges of LSTK contours were smaller than those of manual contours for all features. Conclusion: Radiomics features extracted from LSTK contours were highly reliable across and among observers. With semi-automatic segmentation tools, observers without formal clinical training were comparable to physicians in evaluating tumor segmentation. [ABSTRACT FROM AUTHOR]

We present an improved image analysis pipeline to detect the percent brain volume change (PBVC) using SIENA (tructural mage valuation, using ormalization, of trophy) in populations with Alzheimer’s dementia. Our proposed approach uses the improved brain extraction mask from BEaST (rain xtraction based on nonlocl egmentation echnique) instead of the conventional BET (rain xtraction ool) for SIENA. We compared four varying options of BET as well as BEaST and applied these five methods to analyze scan-rescan MRIs in ADNI from 332 subjects, longitudinal ADNI MRIs from the same 332 subjects, their repeat scans over time, and OASIS longitudinal MRIs from 123 subjects. The results showed that BEaST brain masks were consistent in scan-rescan reproducibility. The cross-sectional scan-rescan error in the absolute percent brain volume difference measured by SIENA was smallest (p≤0.0187) with the proposed BEaST-SIENA. We evaluated the statistical power in terms of effect size, and the best performance was achieved with BEaST-SIENA (1.2789 for ADNI and 1.095 for OASIS). The absolute difference in PBVC between scan-dataset (volume change from baseline to year-1) and rescan-dataset (volume change from baseline repeat scan to year-1 repeat scan) was also the smallest with BEaST-SIENA compared to the BET-based SIENA and had the highest correlation when compared to the BET-based SIENA variants. In conclusion, our study shows that BEaST was robust in terms of reproducibility and consistency and that SIENA’s reproducibility and statistical power are improved in multiple datasets when used in combination with BEaST. [ABSTRACT FROM AUTHOR]

Objective: Use of reproducible research practices improves the quality of science and the speed of scientific development. We sought to understand use of reproducible research practices in public health and associated barriers and facilitators. Methods: In late 2017, we surveyed members of the American Public Health Association Applied Public Health Statistics section and others; 247 of 278 who screened eligible answered the survey, and 209 answered every applicable question. The survey included questions about file management, code annotation and documentation, reproducibility of analyses, and facilitators and barriers of using reproducible practices. Results: Just 14.4% of participants had shared code, data, or both. Many participants reported their data (33%) and code (43.2%) would be difficult for colleagues to find if they left their institution. Top reported barriers to using reproducible practices were data privacy (49.8%) and lack of time (41.7%). Participants suggested training (50.9%) and requirements by journals (44.4%) and funders (40.2%) to increase use of reproducible research practices. Conclusions: Increasing use of reproducible research practices is important for public health and requires action from researchers, training programs, funders, and journals. [ABSTRACT FROM AUTHOR]

We developed a new probabilistic model to assess the impact of recommendations rectifying the reproducibility crisis (by publishing both positive and ‘negative‘ results and increasing statistical power) on competing objectives, such as discovering causal relationships, avoiding publishing false positive results, and reducing resource consumption. In contrast to recent publications our model quantifies the impact of each single suggestion not only for an individual study but especially their relation and consequences for the overall scientific process. We can prove that higher-powered experiments can save resources in the overall research process without generating excess false positives. The better the quality of the pre-study information and its exploitation, the more likely this beneficial effect is to occur. Additionally, we quantify the adverse effects of both neglecting good practices in the design and conduct of hypotheses-based research, and the omission of the publication of ‘negative‘ findings. Our contribution is a plea for adherence to or reinforcement of the good scientific practice and publication of ‘negative‘ findings. [ABSTRACT FROM AUTHOR]

We present Direct Assessment, a method for manually assessing the quality of automatically-generated captions for video. Evaluating the accuracy of video captions is particularly difficult because for any given video clip there is no definitive ground truth or correct answer against which to measure. Metrics for comparing automatic video captions against a manual caption such as BLEU and METEOR, drawn from techniques used in evaluating machine translation, were used in the TRECVid video captioning task in 2016 but these are shown to have weaknesses. The work presented here brings human assessment into the evaluation by crowd sourcing how well a caption describes a video. We automatically degrade the quality of some sample captions which are assessed manually and from this we are able to rate the quality of the human assessors, a factor we take into account in the evaluation. Using data from the TRECVid video-to-text task in 2016, we show how our direct assessment method is replicable and robust and scales to where there are many caption-generation techniques to be evaluated including the TRECVid video-to-text task in 2017. [ABSTRACT FROM AUTHOR]

Linear description (LD) of conformation traits was introduced in horse breeding to minimise subjectivity in scoring. However, recent studies have shown that LD traits show essentially the same problems as traditionally scored traits, such as data converging around the mean value with very small standard deviations. To improve the assessment of conformation traits of horses, we investigated the application of the recently described horse shape space model based upon 403 digitised photographs of 243 Franches-Montagnes (FM) stallions and extracted joint angles based on specific landmark triplets. Repeatability, reproducibility and consistency of the resulting shape data and joint angles were assessed with Procrustes ANOVA (Rep) and intra-class correlation coefficients (ICC). Furthermore, we developed a subjective score to classify the posture of the horses on each photograph. We derived relative warp scores (PCs) based upon the digitised photos conducting a principal component analysis (PCA). The PCs of the shapes and joint angles were compared to the posture scores and to the linear description data using linear mixed effect models including significant posture scores as random factors. The digitisation process was highly repeatable and reproducible for the shape (Rep = 0.72–0.99, ICC = 0.99). The consistency of the shape was limited by the age and posture (p < 0.05). The angle measurements were highly repeatable within one digitiser. Between digitisers, we found a higher variability of ICC values (ICC = 0.054–0.92), indicating digitising error in specific landmarks (e.g. shoulder point). The posture scores were highly repeatable (Fleiss’ kappa = 0.713–0.857). We identified significant associations (p(X2) < 0.05) with traits describing the withers height, shoulder length and incline, overall leg conformation, walk and trot step length. The horse shape data and angles provide additional information to explore the morphology of horses and therefore can be applied to improve the knowledge of the genetic architecture of LD traits. [ABSTRACT FROM AUTHOR]

Kinomics is an emerging field of science that involves the study of global kinase activity. As kinases are essential players in virtually all cellular activities, kinomic testing can directly examine protein function, distinguishing kinomics from more remote, upstream components of the central dogma, such as genomics and transcriptomics. While there exist several different approaches for kinomic research, peptide microarrays are the most widely used and involve kinase activity assessment through measurement of phosphorylation of peptide substrates on the array. Unfortunately, bioinformatic tools for analyzing kinomic data are quite limited necessitating the development of accessible open access software in order to facilitate standardization and dissemination of kinomic data for scientific use. Here, we examine and present tools for data analysis for the popular PamChip® (PamGene International) kinomic peptide microarray. As a result, we propose (1) a procedural optimization of kinetic curve data capture, (2) new methods for background normalization, (3) guidelines for the detection of outliers during parameterization, and (4) a standardized data model to store array data at various analytical points. In order to utilize the new data model, we developed a series of tools to implement the new methods and to visualize the various data models. In the interest of accessibility, we developed this new toolbox as a series of JavaScript procedures that can be utilized as either server side resources (easily packaged as web services) or as client side scripts (web applications running in the browser). The aggregation of these tools within a Kinomics Toolbox provides an extensible web based analytic platform that researchers can engage directly and web programmers can extend. As a proof of concept, we developed three analytical tools, a technical reproducibility visualizer, an ANOVA based detector of differentially phosphorylated peptides, and a heatmap display with hierarchical clustering. [ABSTRACT FROM AUTHOR]

One of the main objectives of orthodontic treatment is to achieve an esthetic smile. This study set out to analyze differences in the perception of smile esthetics among patients before and after receiving orthodontic treatment. 250 Spanish patients analyzed a single photograph in which, by means of computer software, midline diastema, black triangle, gingival margin of the left central incisor, and gingival (“gummy”) smile were altered. Each patient analyzed these images before and after undergoing orthodontic treatment. Patients scored the photographs on a scale from 1 to 10. Statistical analyses of each group’s level of perception were carried out, identifying significant differences in evaluations before and after treatment, and in relation to subjects’ gender and age. Patients presented significant differences in the esthetic perception of midline diastema and gummy smile anomalies after they had completed orthodontic treatment. Gender influenced the perception of smile esthetics, whereby women were significantly more critical of midline diastema, black triangle and gingival margin of the upper central incisor than men. The age variable also showed significant differences in the perception of midline diastema and black triangle anomalies. The perception of smile esthetics of some dental anomalies changes as a result of orthodontic treatment. Gender influences the perception of some of the dental anomalies studied. [ABSTRACT FROM AUTHOR]

The World Health Organization suggests that approximately 10% of medicines worldwide are either falsified or substandard with higher figures in low and middle income countries. Such poor quality medicines can seriously harm patients and pose a threat to the economy worldwide. This study investigates attenuated total reflectance-fourier transform infrared (ATR-FTIR) spectroscopy as a simple and rapid method for determination of drug content in tablet dosage forms. Paracetamol was used as the model pharmaceutical ingredient. Spectra of standard mixtures of paracetamol with different excipients formed the basis for multivariate PLS based quantitative analysis of simulated tablet content using different selected infrared absorbance bands. Calibration methods using ATR-FTIR were compared with the ATR-FTIR and conventional ultraviolet spectroscopic analyses of real tablet samples and showed that the paracetamol/microcrystalline cellulose mixtures gave optimum results for all spectral bands tested. The quantitative data for band 1524–1493cm-1 was linear (R2 ˃ 0.98; LOQ ≥ 10%w/w tablet). Global examples of paracetamol tablets were tested using this protocol and 12% of the tablet samples examined was identified as substandard. Each sample analysis was completed in just a few minutes. ATR-FTIR can therefore be used in the rapid screening of tablet formulations. The simplicity of the proposed method makes it appropriate for use in low and middle income countries where analytical facilities are not available. [ABSTRACT FROM AUTHOR]

Purpose: To evaluate the repeatability and reproducibility of retinal and choroidal thickness measured with Swept source Optical Coherence Tomography (SS-OCT) in eyes with Diabetic Macular Edema (DME). Methods: 42 DME eyes were imaged using SS-OCT standard Macular scanning protocols. Retinal and choroidal thickness were measured in the Total macular circle (TMC) and foveal central subfield (FCS) using device-integrated specific software. The coefficient of repeatability (CR) and intraclass correlation coefficient (ICC) were determined as a measure of repeatability and relative reliability within graders. Reproducibility was assessed using Bland-Altman plots and 95% limits of agreement (LoA) were determined as a measure of interobserver variability. Results: Intragrader CR of retinal and choroidal thickness were 8.37 and 12.20 microns for TMC and 22.24 and 32.40 microns for FCS, and intergrader 95% LoA were 7.37–8.69 and -27.2–27.71 microns for TMC and -34.21–41.93 and -30.46–24.84 for FCS, respectively. Retinal and choroidal thickness showed very good intraobserver reliability for both TMC and FCS (ICC 0.99, LoA 0.98–0.99 in all cases). Intraobserver and interobserver variability for retinal and choroidal thickness was not significantly different for TMC (p = 0.98 and p = 0.90, p = 0.98 and p = 0.91) or FCS (p = 0.97 and p = 0.85, p = 0.78 and p = 0.73), respectively. Conclusions: Retinal and choroidal thickness in DME eyes can be quantified with good reliability, repeatability and reproducibility using new OCT devices that incorporate swept source technology. The technical advantages of this technology may provide new insights in the understanding of the choroidal changes related with DME. [ABSTRACT FROM AUTHOR]

Neuroimaging methods such as magnetic resonance imaging (MRI) involve complex data collection and analysis protocols, which necessitate the establishment of good research data management (RDM). Despite efforts within the field to address issues related to rigor and reproducibility, information about the RDM-related practices and perceptions of neuroimaging researchers remains largely anecdotal. To inform such efforts, we conducted an online survey of active MRI researchers that covered a range of RDM-related topics. Survey questions addressed the type(s) of data collected, tools used for data storage, organization, and analysis, and the degree to which practices are defined and standardized within a research group. Our results demonstrate that neuroimaging data is acquired in multifarious forms, transformed and analyzed using a wide variety of software tools, and that RDM practices and perceptions vary considerably both within and between research groups, with trainees reporting less consistency than faculty. Ratings of the maturity of RDM practices from ad-hoc to refined were relatively high during the data collection and analysis phases of a project and significantly lower during the data sharing phase. Perceptions of emerging practices including open access publishing and preregistration were largely positive, but demonstrated little adoption into current practice. [ABSTRACT FROM AUTHOR]

Objective: Quantitative saccadic testing is a non-invasive method of evaluating the neural networks involved in the control of eye movements. The aim of this study is to provide a standardized and reproducible protocol for infrared oculography measurements of eye movements and analysis, which can be applied for various diseases in a multicenter setting. Methods: Development of a protocol to Demonstrate Eye Movement Networks with Saccades (DEMoNS) using infrared oculography. Automated analysis methods were used to calculate parameters describing the characteristics of the saccadic eye movements. The two measurements of the subjects were compared with descriptive and reproducibility statistics. Results: Infrared oculography measurements of all subjects were performed using the DEMoNS protocol and various saccadic parameters were calculated automatically from 28 subjects. Saccadic parameters such as: peak velocity, latency and saccade pair ratios showed excellent reproducibility (intra-class correlation coefficients > 0.9). Parameters describing performance of more complex tasks showed moderate to good reproducibility (intra-class correlation coefficients 0.63–0.78). Conclusions: This study provides a standardized and transparent protocol for measuring and analyzing saccadic eye movements in a multicenter setting. The DEMoNS protocol details outcome measures for treatment trial which are of excellent reproducibility. The DEMoNS protocol can be applied to the study of saccadic eye movements in various neurodegenerative and motor diseases. [ABSTRACT FROM AUTHOR]

Using 3D anatomical landmarks from adult human head MRIs, we assessed the magnitude of inter-operator differences in Procrustes-based geometric morphometric analyses. An in depth analysis of both absolute and relative error was performed in a subsample of individuals with replicated digitization by three different operators. The effect of inter-operator differences was also explored in a large sample of more than 900 individuals. Although absolute error was not unusual for MRI measurements, including bone landmarks, shape was particularly affected by differences among operators, with up to more than 30% of sample variation accounted for by this type of error. The magnitude of the bias was such that it dominated the main pattern of bone and total (all landmarks included) shape variation, largely surpassing the effect of sex differences between hundreds of men and women. In contrast, however, we found higher reproducibility in soft-tissue nasal landmarks, despite relatively larger errors in estimates of nasal size. Our study exemplifies the assessment of measurement error using geometric morphometrics on landmarks from MRIs and stresses the importance of relating it to total sample variance within the specific methodological framework being used. In summary, precise landmarks may not necessarily imply negligible errors, especially in shape data; indeed, size and shape may be differentially impacted by measurement error and different types of landmarks may have relatively larger or smaller errors. Importantly, and consistently with other recent studies using geometric morphometrics on digital images (which, however, were not specific to MRI data), this study showed that inter-operator biases can be a major source of error in the analysis of large samples, as those that are becoming increasingly common in the 'era of big data'. [ABSTRACT FROM AUTHOR]

Background: Mitral annular plane systolic excursion (MAPSE) is an increasingly used echocardiography technique to assess left ventricular (LV) function. However, reproducibility and dependence on echocardiographic image quality for MAPSE in pediatric patients have not been studied to date. Methods: We analyzed 284 transthoracic echocardiograms performed on consecutive normotensive children without structural heart disease (mean age 12.6±3.1 years, 50.4% female). B-mode and M-mode derived MAPSE measurements were performed and analyzed regarding inter- and intraobserver reliability and the influence of echocardiographic image quality. Results: Overall, MAPSE measurements were highly reproducible with only minor bias. Both inter- and intraobserver reliability were significantly better for M-mode derived MAPSE (p<0.001). Echocardiographic image quality did not significantly influence M-mode MAPSE reproducibility (p>0.235). In contrast, B-mode lateral MAPSE was significantly better reproducible in optimal image quality (-0.07±1.04) when compared to suboptimal echocardiographic images (0.42±1.59, p<0.001). Moreover, poor quality images yielded significantly lower M-mode MAPSE values (14.3±2 mm) than near-optimal (15.2±1.9 mm, p<0.001) or optimal images (15.1±2.2 mm, p = 0.006). Conclusion: Echocardiographic image quality essentially has a negligible effect on MAPSE reproducibility and measurements. Consequently, MAPSE is a robust echocardiographic parameter with convincing reproducibility for the assessment of LV function in children—even in patients with substandard imaging conditions. [ABSTRACT FROM AUTHOR]

A number of publishers and funders, including PLOS, have recently adopted policies requiring researchers to share the data underlying their results and publications. Such policies help increase the reproducibility of the published literature, as well as make a larger body of data available for reuse and re-analysis. In this study, we evaluate the extent to which authors have complied with this policy by analyzing Data Availability Statements from 47,593 papers published in PLOS ONE between March 2014 (when the policy went into effect) and May 2016. Our analysis shows that compliance with the policy has increased, with a significant decline over time in papers that did not include a Data Availability Statement. However, only about 20% of statements indicate that data are deposited in a repository, which the PLOS policy states is the preferred method. More commonly, authors state that their data are in the paper itself or in the supplemental information, though it is unclear whether these data meet the level of sharing required in the PLOS policy. These findings suggest that additional review of Data Availability Statements or more stringent policies may be needed to increase data sharing. [ABSTRACT FROM AUTHOR]

Visual assessments are used for evaluating the quality of food products, such as fresh-cut lettuce packaged in bags with modified atmosphere. We have compared the accuracy and the reliability of visual evaluations of decay on fresh-cut lettuce performed with experienced and inexperienced raters. In addition, we have analyzed decay data from over 4.5 thousand bags to determine the optimum timing for evaluations to detect differences among accessions. Lin’s concordance coefficient (ρc) that takes into consideration both the closeness of the data and the conformance to the identity line showed high repeatability (intra-rater reliability, ρc = 0.97), reproducibility (inter-rater reliability, ρc = 0.92), and accuracy (ρc = 0.96) for experienced raters. Inexperienced raters did not perform as well and their ratings showed decreased repeatability (ρc = 0.93), but even larger reduction in reproducibility (ρc = 0.80) and accuracy (ρc = 0.90). We have detected that 5.3% of ratings were outside of the 95% limits of agreement. These under- or overestimates were predominantly found for bags with intermediate levels of decay, which corresponds to the middle of the rating scale. This occurs because intermediate amounts of decay are more difficult to discriminate than extremes. The frequencies of aberrant ratings for experienced raters ranged from 0.6% to 4.4% (mean = 2.1%), for inexperienced raters the frequencies were substantially higher, ranging from 6.1% to 15.6% (mean = 9.4%). Therefore, we recommend that new raters receive training that includes practical examples in this range of decay, use of standard area diagrams, and continuing interaction with experienced raters (consultation during actual rating). Very high agreement among experienced raters indicate that visual ratings can be successfully used for evaluations of decay, until a more objective, rapid, and affordable method is developed. We recommend evaluating samples at multiple time points until 42 days after processing (about 80% decay on average) and then combining these individual ratings into the area under the decay progress stairs (AUDePS) score. Applying this approach, experienced evaluators can accurately detect difference among lettuce accessions and identify lettuce cultivars with reduced decay. [ABSTRACT FROM AUTHOR]

We introduce a publication policy that incorporates “conditional equivalence testing” (CET), a two-stage testing scheme in which standard NHST is followed conditionally by testing for equivalence. The idea of CET is carefully considered as it has the potential to address recent concerns about reproducibility and the limited publication of null results. In this paper we detail the implementation of CET, investigate similarities with a Bayesian testing scheme, and outline the basis for how a scientific journal could proceed to reduce publication bias while remaining relevant. [ABSTRACT FROM AUTHOR]

The “publish or perish” incentive drives many researchers to increase the quantity of their papers at the cost of quality. Lowering quality increases the number of false positive errors which is a key cause of the reproducibility crisis. We adapted a previously published simulation of the research world where labs that produce many papers are more likely to have “child” labs that inherit their characteristics. This selection creates a competitive spiral that favours quantity over quality. To try to halt the competitive spiral we added random audits that could detect and remove labs with a high proportion of false positives, and also improved the behaviour of “child” and “parent” labs who increased their effort and so lowered their probability of making a false positive error. Without auditing, only 0.2% of simulations did not experience the competitive spiral, defined by a convergence to the highest possible false positive probability. Auditing 1.35% of papers avoided the competitive spiral in 71% of simulations, and auditing 1.94% of papers in 95% of simulations. Audits worked best when they were only applied to established labs with 50 or more papers compared with labs with 25 or more papers. Adding a ±20% random error to the number of false positives to simulate peer reviewer error did not reduce the audits’ efficacy. The main benefit of the audits was via the increase in effort in “child” and “parent” labs. Audits improved the literature by reducing the number of false positives from 30.2 per 100 papers to 12.3 per 100 papers. Auditing 1.94% of papers would cost an estimated $15.9 million per year if applied to papers produced by National Institutes of Health funding. Our simulation greatly simplifies the research world and there are many unanswered questions about if and how audits would work that can only be addressed by a trial of an audit. [ABSTRACT FROM AUTHOR]

Mormyrid fish rely on reafferent input for active electrolocation. Their electrosensory input consists of phase and amplitude information. These are encoded by differently tuned receptor cells within the Mormyromasts, A- and B-cells, respectively, which are distributed over the animal’s body. These convey their information to two topographically ordered medullary zones in the electrosensory lateral line lobe (ELL). The so-called medial zone receives only amplitude information, while the dorsolateral zone receives amplitude and phase information. Using both sources of information, Mormyrid fish can disambiguate electrical impedances. Where and how this disambiguation takes place is presently unclear. We here investigate phase-sensitivity downstream from the electroreceptors. We provide first evidence of phase-sensitivity in the medial zone of ELL. In this zone I-cells consistently decreased their rate to positive phase-shifts (6 of 20 cells) and increased their rate to negative shifts (11/20), while E-cells of the medial zone (3/9) responded oppositely to I-cells. In the dorsolateral zone the responses of E- and I-cells were opposite to those found in the medial zone. Tracer injections revealed interzonal projections that interconnect the dorsolateral and medial zones in a somatotopic manner. In summary, we show that phase information is processed differently in the dorsolateral and the medial zones. This is the first evidence for a mechanism that enhances the contrast between two parallel sensory channels in Mormyrid fish. This could be beneficial for impedance discrimination that ultimately must rely on a subtractive merging of these two sensory streams. [ABSTRACT FROM AUTHOR]

Purpose: To investigate the accuracy, reproducibility and sensitivity to respiratory gating, field strength and ventricle segmentation of hemodynamic force quantification in the left and right ventricles of the heart (LV and RV) using 4D-flow magnetic resonance imaging (MRI), and to provide free hemodynamic force analysis software. Materials and methods: A pulsatile flow phantom was imaged using 4D flow MRI and laser-based particle image velocimetry (PIV). Cardiac 4D flow MRI was performed in healthy volunteers at 1.5T (n = 23). Reproducibility was investigated using MR scanners from two different vendors on the same day (n = 8). Subsets of volunteers were also imaged without respiratory gating (n = 17), at 3T on the same day (n = 6), and 1–12 days later on the same scanner (n = 9, median 6 days). Agreement was measured using the intraclass correlation coefficient (ICC). Results: Phantom validation showed good accuracy for both scanners (Scanner 1: bias -14±9%, y = 0.82x+0.08, R2 = 0.96, Scanner 2: bias -12±8%, y = 0.99x-0.08, R2 = 1.00). Force reproducibility was strong in the LV (0.09±0.07 vs 0.09±0.07 N, bias 0.00±0.04 N, ICC = 0.87) and RV (0.09±0.06 vs 0.09±0.05 N, bias 0.00±0.03, ICC = 0.83). Strong to very strong agreement was found for scans with and without respiratory gating (LV/RV: ICC = 0.94/0.95), scans on different days (ICC = 0.92/0.87), and 1.5T and 3T scans (ICC = 0.93/0.94). Conclusion: Software for quantification of hemodynamic forces in 4D-flow MRI was developed, and results show high accuracy and strong to very strong reproducibility for both the LV and RV, supporting its use for research and clinical investigations. The software including source code is released freely for research. [ABSTRACT FROM AUTHOR]

Competitive pressure to maximize the current bibliometric measures of productivity is jeopardizing the integrity of the scientific literature. Efforts are underway to address the ‘reproducibility crisis’ by encouraging the use of more rigorous, confirmatory methods. However, as long as productivity continues to be defined by the number of discoveries scientists publish, the impact factor of the journals they publish in and the number of times their papers are cited, they will be reluctant to accept high quality methods and consistently conduct and publish confirmatory/replication studies. This exploratory study examined a sample of rigorous Phase II-IV clinical trials, including unpublished studies, to determine if more appropriate metrics and incentives can be developed. The results suggest that rigorous procedures will help reduce false positives, but to the extent that higher quality methods are accepted as the standard of practice, the current bibliometric incentives will discourage innovative studies and encourage scientists to shift their research to less informative studies of subjects that are already being more actively investigated. However, the results also suggest that it is possible to develop a more appropriate system of rewards. In contrast to the current bibliometric incentives, evaluations of the quality of the methods and reproducibility of the results, innovation and diversity of thought, and amount of information produced may serve as measures and incentives that maintain the integrity of the scientific literature and maximize scientific progress. [ABSTRACT FROM AUTHOR]

A sensitive and innovative assay system based on a micro-MEMS-fluxgate sensor and immunomagnetic beads-labels was developed for the rapid analysis of C-reactive proteins (CRP). The fluxgate sensor presented in this study was fabricated through standard micro-electro-mechanical system technology. A multi-loop magnetic core made of Fe-based amorphous ribbon was employed as the sensing element, and 3-D solenoid copper coils were used to control the sensing core. Antibody-conjugated immunomagnetic microbeads were strategically utilized as signal tags to label the CRP via the specific conjugation of CRP to polyclonal CRP antibodies. Separate Au film substrates were applied as immunoplatforms to immobilize CRP-beads labels through classical sandwich assays. Detection and quantification of the CRP at different concentrations were implemented by detecting the stray field of CRP labeled magnetic beads using the newly-developed micro-fluxgate sensor. The resulting system exhibited the required sensitivity, stability, reproducibility, and selectivity. A detection limit as low as 0.002 μg/mL CRP with a linearity range from 0.002 μg/mL to 10 μg/mL was achieved, and this suggested that the proposed biosystem possesses high sensitivity. In addition to the extremely low detection limit, the proposed method can be easily manipulated and possesses a quick response time. The response time of our sensor was less than 5 s, and the entire detection period for CRP analysis can be completed in less than 30 min using the current method. Given the detection performance and other advantages such as miniaturization, excellent stability and specificity, the proposed biosensor can be considered as a potential candidate for the rapid analysis of CRP, especially for point-of-care platforms. [ABSTRACT FROM AUTHOR]

Aim: Since cardiovascular magnetic resonance feature-tracking (CMR-FT) has been demonstrated to be of incremental clinical merit we investigated the interchangeability of global left and right ventricular strain parameters between different CMR-FT software solutions. Material and methods: CMR-cine images of 10 patients without significant reduction in LVEF and RVEF and 10 patients with a significantly impaired systolic function were analyzed using two different types of FT-software (TomTec, Germany; QStrain, Netherlands). Global longitudinal strains (LV GLS, RV GLS), global left ventricular circumferential (GCS) and radial strains (GRS) were assessed. Differences in intra- and inter-observer variability within and between software types based on single and up to three repeated and subsequently averaged measurements were evaluated. Results: Inter-vendor agreement was highest for GCS followed by LV GLS. GRS and RV GLS showed lower inter-vendor agreement. Variability was consistently higher in healthy volunteers as compared to the patient group. Intra-vendor reproducibility was excellent for GCS, LV GLS and RV GLS, but lower for GRS. The impact of repeated measurements was most pronounced for GRS and RV GLS on an intra-vendor level. Conclusion: Cardiac pathology has no influence on CMR-FT reproducibility. LV GLS and GCS qualify as the most robust parameters within and between individual software types. Since both parameters can be interchangeably assessed with different software solutions they may enter the clinical arena for optimized diagnostic and prognostic evaluation of cardiovascular morbidity and mortality in various pathologies. [ABSTRACT FROM AUTHOR]

Selenium (Se) is an important micronutrient but also a strong toxin with a narrow tolerance range for many organisms. As such, a globally heterogeneous Se distribution in soils is responsible for various disease patterns (i.e. Se excess and deficiency) and environmental problems, whereby plants play a key role for the Se entrance into the biosphere. Selenium isotope variations were proved to be a powerful tracer for redox processes and are therefore promising for the exploration of the species dependent Se metabolism in plants and the Se cycling within the Critical Zone. Plant cultivation setups enable systematic controlled investigations, but samples derived from them–plant tissue and phytoagar–are particularly challenging and require specific preparation and purification steps to ensure precise and valid Se isotope analytics performed with HG-MC-ICP-MS. In this study, different methods for the entire process from solid tissue preparation to Se isotope measurements were tested, optimized and validated. A particular microwave digestion procedure for plant tissue and a vacuum filtration method for phytoagar led to full Se recoveries, whereby unfavorable organic residues were reduced to a minimum. Three purification methods predominantly described in the literature were systematically tested with pure Se solution, high concentrated multi-element standard solution as well as plant and phytoagar as target matrices. All these methods efficiently remove critical matrix elements, but differ in Se recovery and organic residues. Validation tests doping Se-free plant material and phytoagar with a reference material of known Se isotope composition revealed the high impact of organic residues on the accuracy of MC-ICP-MS measurements. Only the purification method with no detectable organic residues, hydride generation and trapping, results in valid mass bias correction for plant samples with an average deviation to true δ82/76Se values of 0.2 ‰ and a reproducibility (2 SD) of ± 0.2 ‰. For phytoagar this test yields a higher deviation of 1.1 ‰ from the true value and a 2 SD of ± 0.1 ‰. The application of the developed methods to cultivated plants shows sufficient accuracy and precision and is a promising approach to resolve plant internal Se isotope fractionations, for which respective δ82/76Se values of +2.3 to +3.5 ‰ for selenate and +1.2 to +1.9 ‰ for selenite were obtained. [ABSTRACT FROM AUTHOR]

Background: Iron overload-related heart failure is the principal cause of death in transfusion dependent patients, including those with Thalassemia Major. Linking cardiac siderosis measured by T2* to therapy improves outcomes. T1 mapping can also measure iron; preliminary data suggests it may have higher sensitivity for iron, particularly for early overload (the conventional cut-point for no iron by T2* is 20ms, but this is believed insensitive). We compared T1 mapping to T2* in cardiac iron overload. Methods: In a prospectively large single centre study of 138 Thalassemia Major patients and 32 healthy controls, we compared T1 mapping to dark blood and bright blood T2* acquired at 1.5T. Linear regression analysis was used to assess the association of T2* and T1. A “moving window” approach was taken to understand the strength of the association at different levels of iron overload. Results: The relationship between T2* (here dark blood) and T1 is described by a log-log linear regression, which can be split in three different slopes: 1) T2* low, <20ms, r2 = 0.92; 2) T2* = 20-30ms, r2 = 0.48; 3) T2*>30ms, weak relationship. All subjects with T2*<20ms had low T1; among those with T2*>20ms, 38% had low T1 with most of the subjects in the T2* range 20-30ms having a low T1. Conclusions: In established cardiac iron overload, T1 and T2* are concordant. However, in the 20-30ms T2* range, T1 mapping appears to detect iron. These data support previous suggestions that T1 detects missed iron in 1 out of 3 subjects with normal T2*, and that T1 mapping is complementary to T2*. The clinical significance of a low T1 with normal T2* should be further investigated. [ABSTRACT FROM AUTHOR]