Search

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. The present observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1000 papers/field per year) were randomly selected from PubMed Central and screened (n=227998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4–58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. The present study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

Timely publication of clinical trial results is central for evidence-based medicine. In this follow-up study we benchmark the performance of German university medical centers (UMCs) regarding timely dissemination of clinical trial results in recent years.Following the same search and tracking methods used in our previous study for the years 2009 - 2013, we identified trials led by German UMCs completed between 2014 and 2017 and tracked results dissemination for the identified trials.We identified 1,658 trials in the 2014 -2017 cohort. Of these trials, 43% published results as either journal publication or summary results within 24 months after completion date, which is an improvement of 3.8% percentage points compared to the previous study. At the UMC level, the proportion published after 24 months ranged from 14% to 71%. Five years after completion, 30% of the trials still remained unpublished.Despite minor improvements compared to the previously investigated cohort, the proportion of timely reported trials led by German UMCs remains low. German UMCs should take further steps to improve the proportion of timely reported trials.

There are several definitions for open data, largely in agreement with each other. However, they have so far not been translated into operationalizations which would allow to detect in a structured and reproducible way, whether for a specific research article underlying data have been shared. Here, we propose a detailed set of criteria to enable such an assessment, focusing on biomedical research. We have used this assessment for the distribution of performance-oriented funding at a large university hospital, as well as for monitoring of data sharing practices in a dashboard. In addition to fully open data, we include a separate set of criteria for datasets with restricted access, which we also reward. The criteria are in part inspired by the FAIR principles, with a focus on findability and accessibility, but do not map onto individual principles. The criteria attribute open data status in a binary fashion, both to individual datasets and, ultimately, the article with which they were shared. The criteria allow a verifiable assessment, based on an automated and a manual screening step, which we have implemented and validated, as described elsewhere. In the present article, we focus on the conceptual level of assessing the presence of shared data.

BackgroundMandates and recommendations related to embedding open science practices within the research lifecycle are increasingly common. Few stakeholders, however, are monitoring compliance to their mandates or recommendations. It is necessary to monitor the current state of open science to track changes over time and to identify areas to create interventions to drive improvements.Monitoring open science practices requires that they are defined and operationalized. Involving the biomedical community, we sought to reach consensus on a core set of open science practices to monitor at biomedical research institutions.Methods and FindingsTo establish consensus in a structured and systematic fashion, we conducted a modified 3-round Delphi study. Participants in Round 1 were 80 individuals from 20 biomedical research institutions that exhibit interest in or actively support open science. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In Rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices that could be important and meaningful to monitor in an automated institutional open science dashboard. Participants voted on the inclusion of each item and provided a rationale for their choice. We defined consensus as 80% agreement. Between rounds, participants received aggregated voting scores for each item and anonymized comments from all participants, and were asked to re-vote on items that did not reach consensus. For Round 3, we hosted two half- day virtual meetings with 21 and 17 participants respectively to discuss and vote on all items that had not reached consensus after Round 2. Ultimately, participants reached consensus to include a 19 open science practices.ConclusionsA group of international stakeholders used a modified Delphi process to agree upon open science practices to monitor in a proposed open science dashboard for biomedical institutions. The core set of 19 open science practices identified by participants will form the foundation for institutional dashboards that display compliance with open science practices. They will now be assessed and tested for automatic inclusion in terms of technical feasibility. Using user-centered design, participating institutions will be involved in creating a dashboard prototype, which can then be implemented to monitor rates of open science practices at biomedical institutions. Our methods and approach may also transfer to other research settings–other disciplines could consider using our consensus list as a starting point for agreement upon a discipline-specific set of open science practices to monitor. The findings may also be of broader value to the development of policy, education, and interventions.

Western blotting is a standard laboratory method used to detect proteins and assess their expression levels. Unfortunately, poor western blot image display practices and a lack of detailed methods reporting can limit a reader’s ability to evaluate or reproduce western blot results. While several groups have studied the prevalence of image manipulation or provided recommendations for improving western blotting, data on the prevalence of common publication practices are scarce. We systematically examined 551 articles published in the top 25% of journals in neurosciences (n=151) and cell biology (n=400) that contained western blot images, focusing on practices that may omit important information. Our data show that most published western blots are cropped and blot source data are not made available to readers in the supplement. Publishing blots with visible molecular weight markers is rare, and many blots additionally lack molecular weight labels. Western blot methods sections often lack information on the amount of protein loaded on the gel, blocking steps and antibody labeling protocol. Important antibody identifiers like source, catalog number or RRID were omitted frequently for primary antibodies, and regularly for secondary antibodies. We present detailed descriptions and visual examples to help scientists, peer reviewers and editors to publish more informative western blot figures and methods. Additional resources include a toolbox to help scientists produce more reproducible western blot data, teaching slides in English and Spanish and an antibody reporting template.

Promoting translational research as a means to overcoming chasms in the translation of knowledge through successive fields of research from basic science to public health impacts and back is a central challenge for research managers and policymakers. Organizational leaders need to assess baseline conditions, identify areas needing improvement, and to judge the impact of specific initiatives to sustain or improve translational research practices at their institutions. Currently, there is a lack of such an assessment tool addressing the specific context of translational biomedical research. To close this gap, we have developed a new survey for assessing the organizational climate for translational research. This self-assessment tool measures employees’ perceptions of translational research climate and underlying research practices in organizational environments and builds on the established Survey of Organizational Research Climate, assessing research integrity. Using this tool, we show that scientists at a large university hospital (Charité Berlin) perceive translation as a central and important component of their work. Importantly, local resources and direct support are main contributing factors for the practical implementation of translation into their own research practice. We identify and discuss potential leverage points for an improvement of research climate to foster successful translational research. Electronic supplementary material The online version of this article (10.1007/s11948-020-00234-0) contains supplementary material, which is available to authorized users.

ObjectiveTimely publication of clinical trial results is central for evidence-based medicine. The performance of university medical centers (UMCs) regarding timely results publication informs the local and national decision makers, patients, and the public about the need to improve trial results dissemination.Study Design and SettingFollowing the same search and tracking methods used in our previous study for the years 2009 - 2013, we identified trials led by German UMCs completed between 2014 and 2017 and tracked results dissemination for the identified trials.ResultsWe identified 1,658 trials in the 2014-2017 cohort. Of these trials, 43% published results as either journal publication or summary results within 24 months after completion date, which is an improvement of 3.8% percentage points compared to the previous study. At the UMC level, the 24-month publication rates varied from 14% to 71%. Five years after completion, still 30% of the trials remained unpublished.ConclusionDespite minor improvements compared to the previously investigated cohort, the rate of timely reporting of trials led by German UMCs remains low. German UMCs should take further steps to improve timely reporting rates.

Timely and comprehensive reporting of clinical trial results builds the backbone of evidence-based medicine and responsible research. The proportion of timely disseminated trial results can inform alternative national and international benchmarking of university medical centers (UMCs).For all German UMCs, we tracked all registered trials completed between 2009 and 2013. The results and an interactive website benchmark German UMCs regarding their performance in result dissemination.We identified and tracked 2,132 clinical trials. For 1,509 trials, one of the German UMCs took the academic lead. Of these 1,509 "lead trials," 39% published their results (mostly via journal publications) in a timely manner (24 months after completion). More than 6 years after study completion, 26% of all eligible lead trials still had not disseminated results.Despite substantial attention from many stakeholders to the topic, there is still a strong delay or even absence of result dissemination for many trials. German UMCs have several opportunities to improve this situation. Further research should evaluate whether and how a transparent benchmarking of UMC performance in result dissemination helps to increase value and reduce waste in medical research.

International audience; The COVID-19 pandemic has thrust preprints into the spotlight, highlighting their advantages and disadvantages. The lack of peer review allows publication to occur with unprecedented speed, but this has raised concerns among biomedical scientists about the quality of the reported research. The study had three primary objectives in regard to COVID-19 preprints: 1. Test the feasibility of automatically and publicly evaluating preprints on a large scale; 2. Assess the prevalence of common quality problems in preprints; and 3. Compare the quality of preprints to published papers. While a substantial fraction (36.2%) of preprints addressed study limitations, the proportion that met other quality criteria was much lower, with only 20% addressing sex as a variable, ~14% sharing open code or data, 7.6% including non-colorblind safe images, and 7.3% showing misleading bar graphs. Both authors and non-authors interacted with the automated Tweets containing reports for preprints. This project shows that it is feasible to conduct large-scale automated screening of preprints for common quality criteria and provide feedback to study authors and readers before publication. These reports can publicly raise awareness about factors that affect study quality and reproducibility, while helping authors to present their research in a more transparent and reproducible manner.

Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify five indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central. Our results indicate remarkable improvements in some (e.g. conflict of interest disclosures, funding disclosures), but not other (e.g. protocol registration, code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand and promote transparency and reproducibility in science.

Statistically significant findings are more likely to be published than non-significant or null findings, leaving scientists and healthcare personnel to make decisions based on distorted scientific evidence. Continuously expanding ´file drawers’ of unpublished data from well-designed experiments waste resources creates problems for researchers, the scientific community and the public. There is limited awareness of the negative impact that publication bias and selective reporting have on the scientific literature. Alternative publication formats have recently been introduced that make it easier to publish research that is difficult to publish in traditional peer reviewed journals. These include micropublications, data repositories, data journals, preprints, publishing platforms, and journals focusing on null or neutral results. While these alternative formats have the potential to reduce publication bias, many scientists are unaware that these formats exist and don’t know how to use them. Our open source file drawer data liberation effort (fiddle) tool (RRID:SCR_017327 available at: http://s-quest.bihealth.org/fiddle/) is a match-making Shiny app designed to help biomedical researchers to identify the most appropriate publication format for their data. Users can search for a publication format that meets their needs, compare and contrast different publication formats, and find links to publishing platforms. This tool will assist scientists in getting otherwise inaccessible, hidden data out of the file drawer into the scientific community and literature. We briefly highlight essential details that should be included to ensure reporting quality, which will allow others to use and benefit from research published in these new formats.

The reproducibility crisis triggered worldwide initiatives to improve rigor, reproducibility, and transparency in biomedical research. There are many examples of scientists, journals, and funding agencies adopting responsible research practices. The QUEST (Quality-Ethics-Open Science-Translation) Center offers a unique opportunity to examine the role of institutions. The Berlin Institute of Health founded QUEST to increase the likelihood that research conducted at this large academic medical center would be trustworthy, useful for scientists and society, and ethical. QUEST researchers perform “science of science” studies to understand problems with standard practices and develop targeted solutions. The staff work with institutional leadership and local scientists to incentivize and support responsible practices in research, funding, and hiring. Some activities described in this paper focus on the institution, whereas others may benefit the national and international scientific community. Our experience, approaches, and recommendations will be informative for faculty leadership, administrators, and researchers interested in improving scientific practice., This Community Page article describes an ongoing large-scale structured initiative at the Berlin Institute of Health which aims at improving the robustness, reproducibility and transparency of research in an academic biomedical setting.

The need for replication of initial results has been rediscovered only recently in many fields of research. In preclinical biomedical research, it is common practice to conduct exact replications with the same sample sizes as those used in the initial experiments. Such replication attempts, however, have lower probability of replication than is generally appreciated. Indeed, in the common scenario of an effect just reaching statistical significance, the statistical power of the replication experiment assuming the same effect size is approximately 50%-in essence, a coin toss. Accordingly, we use the provocative analogy of "replicating" a neuroprotective drug animal study with a coin flip to highlight the need for larger sample sizes in replication experiments. Additionally, we provide detailed background for the probability of obtaining a significant p value in a replication experiment and discuss the variability of p values as well as pitfalls of simple binary significance testing in both initial preclinical experiments and replication studies with small sample sizes. We conclude that power analysis for determining the sample size for a replication study is obligatory within the currently dominant hypothesis testing framework. Moreover, publications should include effect size point estimates and corresponding measures of precision, e.g., confidence intervals, to allow readers to assess the magnitude and direction of reported effects and to potentially combine the results of initial and replication study later through Bayesian or meta-analytic approaches.

ObjectivesTo establish the rates of publication and reporting of results for interventional clinical trials across Polish academic medical centres (AMCs) completed between 2009 and 2013. We aim also to compare the publication and reporting success between adult and paediatric trials.DesignCross-sectional study.SettingAMCs in Poland.ParticipantsAMCs with interventional trials registered on ClinicalTrials.gov.Main outcome measureResults reporting on ClinicalTrials.gov and publishing via journal publication.ResultsWe identified 305 interventional clinical trials registered on ClinicalTrials.gov, completed between 2009 and 2013 and affiliated with at least one AMC. Overall, 243 of the 305 trials (79.7%) had been published as articles or posted their summary results on ClinicalTrials.gov. Results were posted within a year of study completion and/or published within 2 years of study completion for 131 trials (43.0%). Dissemination by both posting and publishing results in a timely manner was achieved by four trials (1.3%).ConclusionsOur cross-sectional analysis revealed that Polish AMCs fail to meet the expectation for timely disseminating the findings of all interventional clinical trials. Delayed dissemination and non-dissemination of trial results negatively affects decisions in healthcare.

ObjectiveTimely and comprehensive reporting of clinical trial results build the backbone of evidence-based medicine and responsible research. The proportion of timely disseminated trial results can inform alternative national and international benchmarking of university medical centers (UMCs).Study Design and SettingFor all German UMCs we tracked all registered trials completed between 2009 and 2013. The results and an interactive website benchmark German UMCs regarding their performance in results dissemination.ResultsWe identified and tracked 2,132 clinical trials. For 1,509 trials, one of the German UMCs took the academic lead. Of these 1,509 “lead trials”, 39% published their results (mostly via journal publications) in a timely manner (ConclusionDespite substantial attention from many stakeholders to the topic, there is still a strong delay or even absence of results dissemination for many trials. German UMCs have several opportunities to improve this situation. Further research should evaluate whether and how a transparent benchmarking of UMC performance in results dissemination helps to increase value and reduce waste in medical research.

The concept of evolution, which was introduced by Charles Darwin in 1859, and also its mathematical description by the theory of population genetics are well-established. Population genetics describes the development of a population under the influence of mutations, creating new genetic variants, and natural selection, increasing the frequency of favorable phenotypes. Yet, the experimental verification of selective forces acting on species has proven difficult. With new experimental techniques that have been established in the field of quantitative genetics, like the sequencing of DNA or measurements of gene expression levels, it has become possible to find signs of natural selection on the level of the genome. In this thesis, I develop a statistical test based on population genetics theory that can infer lineage-specific differences in selection between multiple lines of a species. The test employs data from quantitative trait experiments and uses a log-likelihood scoring to quantify the evidence for different selective scenarios. I show that the use of multiple lines increases both the power and the scope of selection inference. Extensive numerical simulations demonstrate that the test can distinguish selection from neutral evolution as well as different scenarios of lineage-specific evolution. The principle of maximum entropy is used to derive a modified version of the selection test that accounts for the multiple testing problem arising when many traits are tested for selection at the same time. The developed test is applied to two published plant datasets and a published dataset of gene expression levels in three yeast lines. In all cases, I find signs of selection not seen with a two-line test. For the yeast dataset I find pervasive adaptation linked to stress resistance both on the level of individual genes as well as for larger gene modules consisting of several genes, like protein complexes and pathways. This adaptation signal is also reflected on the protein levels.

Trait differences between species may be attributable to natural selection. However, quantifying the strength of evidence for selection acting on a particular trait is a difficult task. Here we develop a population-genetic test for selection acting on a quantitative trait which is based on multiple-line crosses. We show that using multiple lines increases both the power and the scope of selection inference. First, a test based on three or more lines detects selection with strongly increased statistical significance, and we show explicitly how the sensitivity of the test depends on the number of lines. Second, a multiple-line test allows to distinguish different lineage-specific selection scenarios. Our analytical results are complemented by extensive numerical simulations. We then apply the multiple-line test to QTL data on floral character traits in plant species of the Mimulus genus and on photoperiodic traits in different maize strains, where we find a signatures of lineage-specific selection not seen in a two-line test., 21 pages, 11 figures; to appear in Genetics

In a multicellular organism different cell types express a gene in different amounts. Samples from which gene expression levels can be measured typically contain a mixture of different cell types, the resulting measurements thus give only averages over the different cell types present. Based on fluctuations in the mixture proportions from sample to sample it is in principle possible to reconstruct the underlying expression levels of each cell type: to deconvolute the sample. We use a statistical mechanics approach to the problem of deconvoluting such partial concentrations from mixed samples, give analytical results for when and how well samples can be unmixed, and suggest an algorithm for sample deconvolution., Comment: 8 pages, 4 figures