Search

Purpose: To present a nomogram for prediction of overall survival (OS) in patients with locally advanced cervical cancer (LACC) undergoing definitive radiochemotherapy including image-guided adaptive brachytherapy (IGABT). Methods and Materials: Seven hundred twenty patients with LACC treated with radiochemotherapy including IGABT in 12 institutions (median follow-up 56 months) were analyzed; 248 deaths occurred. Thirteen candidate predictors for OS were a priori chosen on the basis of the literature and expert knowledge. Missing data (7.2%) were imputed using multiple imputation and predictive mean matching. Univariate analysis with a multivariable Cox regression model for OS stratified by center was performed. Stepwise selection of predictive factors with the Akaike Information Criterion was used to obtain a predictive model and construct a nomogram for OS predictions 60 months from diagnosis; this was internally validated by concordance probability as a measure of discrimination and a calibration plot. Results: Thirteen potential predictive factors were evaluated; 10 factors reached statistical significance in univariate analysis (age, Hemoglobin, FIGO Stage2009, tumor width, corpus involvement, lymph node involvement, concurrent chemotherapy, dose to 90% of the high-risk clinical target volume, volume of CTV at the first brachytherapy [CTVHRVolumeBT], overall treatment time [OTT]). Four factors were confirmed significant within the multivariable Cox regression model (FIGO Stage2009, lymph node involvement, concurrent chemotherapy, CTVHRVolumeBT). The predictive model and corresponding nomogram were based on 7 Akaike Information Criterion–selected factors (age, corpus involvement, FIGO Stage2009, lymph node involvement, concurrent chemotherapy, CTVHRVolumeBT, OTT) and showed promising calibration and discrimination (cross-validated concordance probability c = 0.73). Conclusions: This is the first nomogram to predict OS in patients with LACC treated with IGABT. In addition to previously reported factors (age, FIGO2009 stage, corpus involvement, chemotherapy delivery, OTT, lymph node involvement), status of primary tumor at the time of brachytherapy seems to be an essential outcome predictor. These results can facilitate individualized tailoring of treatment and patient counseling during the treatment.

Background and Objectives: The funnel plot is the most widely used diagnostic plot for meta-analysis. Numerous variants exist to visualize small-study effects, heterogeneity, and the sensitivity of the meta-analytic summary estimates to new evidence (Langan, Higgins, Gregory, & Sutton, 2012). What is currently missing is a funnel plot variant which incorporates information on statistical study-level power to detect an effect of interest. To fill this gap, we here introduce the sunset funnel plot, which, in essence, is a power-enhanced funnel plot (Figure 1). Visual funnel plot examinations for small-study effects include checks whether smaller studies in particular (i.e., those with larger standard errors and associated lower analytic power) tend to yield larger effect sizes. When such an association evidently is driven by conventional criteria of statistical significance, then publication bias is considered to be a likely explanation for the phenomenon, and preferred to other causes, such as true heterogeneity or chance alone (Peters, Sutton, Jones, Abrams, & Rushton, 2008). Information on the power of studies can further support such evaluations of potential publication bias. The test for excess significance (Ioannidis & Trikalinos, 2007) is a widely used evidentiality test to check whether there is a higher number of statistically significant studies than expected, considering their power to detect an effect of interest. Such an excess of significant findings indicates bias in the set of studies under consideration. In the same spirit, if an implausible excess of significant, but at the same time underpowered, studies is visible and potentially drives small-study effects in the funnel plot, this can further weaken the credibility of these results and indicate bias. In addition, significant effects observed in low-powered studies more likely are false positive findings (Forstmeier, Wagenmakers, & Parker, 2017). Power can therefore be seen as an indicator for the replicability of research findings. Indeed, for a set of studies, the deviation of (or, gap between) the proportion of actually observed significant studies and twice the median study power has been proposed as the R-index of replicability (Schimmack, 2016). All in all, study-level power is one useful information to assess the credibility and evidentiality of a set of studies potentially included in a meta-analysis. Consequently, a power-enhanced funnel plot is one means to visualize and communicate this information by incorporating information on study-level power in the well-known, classic funnel plot display. Methods: The sunset (power-enhanced) funnel plot assumes normally distributed effect sizes and regards variances of these effect sizes as known. These assumptions are common in the context of meta-analysis and standard effect sizes for meta-analysis are suitable for the sunset funnel plot as well (e.g., Cohen d, Hedges g, log OR, Fisher’s z-transformed r). For a true population effect size δ, the power of a two-sided Wald test with significance level α testing the null hypothesis δ = 0 is given by Power = 1 - Φ(z1-α/2 - δ/SE(d)) + Φ(-z1-α/2 - δ/SE(d)) with Φ the cumulative distribution function of the standard normal distribution, z1-α/2 the 1-α/2 quantile of the standard normal distribution, and SE(d) the standard error of the study effect size d. The sunset (power-enhanced) funnel plot visualizes these power estimates corresponding to specific standard errors on a second y-axis and with color-coded power regions (Figure 1). Color regions range from an alarming dark red for highly underpowered studies to a relaxing dark green for appropriately powered studies to detect the underlying true effect of interest. The color palette used in the graphic display is vividly remindful of a colorful sunset; hence, the denomination sunset funnel plot. Figure 1: Sunset (power-enhanced) funnel plot, using data from a published meta-analysis (Mathie et al., 2017) comparing homeopathic treatment with placebo. 95% confidence contours are shown, with the black vertical reference line marking the observed summary effect (fixed-effect model) used for power analysis. Significance contours at the .05 and .01 levels are indicated through dark shaded areas. Power estimates are computed for a two-tailed test with significance level .05. R code to reproduce the figure: https://osf.io/967bh/?view_only=e659e4eb1cfa46c2bfe4c8ceb622e922. The underlying true population effect size can either be determined theoretically (e.g., by assuming a smallest effect of interest), or empirically, using meta-analytic estimates of the summary effect. For the latter, the fixed-effect model estimator is one natural default choice, giving less weight (and therefore being less sensitive) to small, biased studies, as compared to random-effects meta-analytic modeling. A number of related power-based statistics can be presented alongside the power-enhanced funnel plot and support its evaluation. These include (i) the median power of studies, (ii) the true underlying effect size necessary for achieving certain levels of median power (e.g., 33% or 66%), (iii) the results of the test for excess significance (Ioannidis & Trikalinos, 2007), and (iv) the R-index as measure for the expected replicability of findings (Schimmack, 2016). To create sunset (power-enhanced) funnel plots and to compute statistics related to these, we provide the tailored function viz_sunset in the package metaviz (Kossmeier, Tran, & Voracek, 2018) within the statistical software R (R Core Team, 2018), and a corresponding online application available at https://metaviz.shinyapps.io/sunset/. Results: For the following illustration example, we use data from a recent published meta-analysis on the effect of homeopathic treatment vs. placebo for numerous medical conditions (Mathie et al., 2017). In this systematic review and meta-analysis, bias assessment suggested high risk of bias for the majority of the 54 randomized controlled trials (RCTs) considered for meta-analysis; only three RCTs were judged as reliable evidence. For illustration purposes, we use the totality of these 54 effect sizes. Visual examination of the corresponding funnel plot shows clear small-study effects, such that imprecise, smaller studies (those with larger standard errors) report larger effects in favor of homeopathy than more precise, larger studies (those with smaller standard errors). This association seems to be driven by studies reporting imprecise, but significant estimates, in particular. Incorporating power information in these considerations (with the fixed-effect estimate δ = -0.25 in favor of homeopathy) additionally reveals that a non-trivial, implausible high, and thus worrisome, number of the significant studies evidently are drastically underpowered (with power values lower than 10%) to detect this effect of interest, thus further suggesting bias (Figure 1). Accordingly, there is an excess of significant findings among the primary studies included in this meta-analysis (15 nominally significant studies observed, but, under these circumstances, only 9.45 significant studies expected; p = .047). The median power of this set of primary studies merely amounts to 14.3% (IQR: 11.1-20.6%), and the true effects needed to reach typical (i.e., median) power levels of 33% or 66% would be substantial (absolute δ values of 0.43 or 0.67, respectively). The expected replicability of findings, as quantified with the R-Index, is extremely low (0.8%). Conclusions and Implications: We introduce the sunset (power-enhanced) funnel plot as a new, useful display for the meta-analytic visualization toolbox. First and foremost, the sunset funnel plot allows to incorporate power considerations into classic funnel plot assessments for small-study effects. In the same spirit as testing for an excess of significant findings (Ioannidis & Trikalinos, 2007), the credibility of findings can further be critically examined by checking whether small-study effects are especially driven by an implausible large number of significant, but at the same time underpowered, studies. Second, the display allows to visually explore and communicate the distribution and typical values of study power for an effect of interest. This visualization is not only informative for meta-analyses, but also in the broader context of meta-scientific investigations into the power of studies of whole scientific fields (e.g., Szucs, & Ioannidis, 2017). Third, changes of power values for a set of studies can be visually examined by varying the true underlying effect. This directly corresponds to questions regarding the necessary true effect size, such that the power of individual or typical studies would reach desired levels. Software to create sunset (power-enhanced) funnel plots is provided. References: Forstmeier, W., Wagenmakers, E. J., & Parker, T. H. (2017). Detecting and avoiding likely false‐positive finding: A practical guide. Biological Reviews, 92, 1941-1968. Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. Kossmeier, M., Tran, U. S., & Voracek, M. (2018). metaviz [R software package]. Retrieved from https://github.com/Mkossmeier/metaviz Langan, D., Higgins, J. P., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. Journal of Clinical Epidemiology, 65, 511-519. Mathie, R. T., Ramparsad, N., Legg, L. A., Clausen, J., Moss, S., Davidson, J. R., ... McConnachie, A. (2017). Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: Systematic review and meta-analysis. Systematic Reviews, 6, 63. R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/ Schimmack, U. (2016). The replicability-index: Quantifying statistical research integrity. Retrieved from https://replicationindex.wordpress.com/2016/01/31/a-revised-introduction-to-the-r-index/ Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15, e2000797. Forstmeier, W., Wagenmakers, E. J., & Parker, T. H. (2017). Detecting and avoiding likely false‐positive finding: A practical guide. Biological Reviews, 92, 1941-1968. Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. Kossmeier, M., Tran, U. S., & Voracek, M. (2018). metaviz [R software package]. Retrieved from https://github.com/Mkossmeier/metaviz Langan, D., Higgins, J. P., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. Journal of Clinical Epidemiology, 65, 511-519. Mathie, R. T., Ramparsad, N., Legg, L. A., Clausen, J., Moss, S., Davidson, J. R., ... McConnachie, A. (2017). Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: Systematic review and meta-analysis. Systematic Reviews, 6, 63. R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/ Schimmack, U. (2016). The replicability-index: Quantifying statistical research integrity. Retrieved from https://replicationindex.wordpress.com/2016/01/31/a-revised-introduction-to-the-r-index/ Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15, e2000797.

Objectives: Reimbursement decisions on new medicines require an assessment of their value. In Austria, when applying for reimbursement of new medicines, pharmaceutical companies are also obliged to submit forecasts of future sales. We systematically examined the accuracy of these pharmaceutical sales forecasts and hence the usefulness of these forecasts for reimbursement evaluations. Methods: We retrospectively analyzed reimbursement applications of 102 new drugs submitted between 2005 and 2014, which were accepted for reimbursement outside of hospitals, and for which actual reimbursed sales were available for at least 3 years. The main outcome variable was the accuracy ratio, defined as the ratio of forecasted sales submitted by pharmaceutical companies when applying for reimbursement to actual sales from reimbursement data. Results: The median accuracy ratio [95% confidence interval] was 1.33 [1.03; 1.74, range 0.15–37.5], corresponding to a median overestimation of actual sales by 33%. Forecasts of actual sales for 55.9% of all examined products either overestimated actual sales by more than 100% or underestimated them by more than 50%. The accuracy of sales forecasts did not show systematic change over the analyzed decade nor was it discernibly influenced by reimbursement status (restricted or unrestricted), the degree of therapeutic benefit, or the therapeutic area of the pharmaceutical product. Sales forecasts of drugs with a higher degree of innovation and those within a dynamic market tended to be slightly more accurate. Conclusions: The majority of sales forecasts provided by applicants for reimbursement evaluations in Austria were highly inaccurate and were on average too optimistic. This is in line with published results for other jurisdictions and highlights the need for caution when using such forecasts for reimbursement procedures.

For a quarter of a century researchers investigating the origins of sexual orientation have largely ascribed to the fraternal birth order effect (FBOE) as a fact, holding that older brothers increase the odds of homosexual orientation among men through an immunoreactivity process. Here, we triangulate the empirical foundations of the FBOE from three distinct, informative perspectives: First, drawing on basic probability calculus, we deduce mathematically that the body of statistical evidence of the FBOE rests on the false assumptions that effects of family size should be controlled for and that this could be achieved through the use of ratio variables. Second, using a data-simulation approach, we demonstrate that by using ratio variables, researchers are bound to falsely declare corroborating evidence of an excess of older brothers at a rate of up to 100%, and that valid approaches attempting to quantify a potential excess of older brothers among homosexual men must control for the confounding effects of the number of older siblings. And third, we re-examine the empirical evidence of the FBOE by using a novel specification-curve and multiverse approach to meta-analysis. This yielded highly inconsistent and moreover similarly-sized effects across 64 male and 17 female samples (N = 2,778,998), compatible with an excess as well as with a lack of older brothers in both groups, thus, suggesting that almost no variation in the number of older brothers in men is attributable to sexual orientation.

The funnel plot is widely used in meta-analyses to assess potential publication bias. However, experimental evidence suggests that informal, mere visual, inspection of funnel plots is frequently prone to incorrect conclusions, and formal statistical tests (Egger regression and others) entirely focus on funnel plot asymmetry. We suggest using the visual inference framework with funnel plots routinely, including for didactic purposes. In this framework, the type I error is controlled by design, while the explorative, holistic, and open nature of visual graph inspection is preserved. Specifically, the funnel plot of the actually observed data is presented simultaneously, in a lineup, with null funnel plots showing data simulated under the null hypothesis. Only when the real data funnel plot is identifiable from all the funnel plots presented, funnel plot-based conclusions might be warranted. Software to implement visual funnel plot inference is provided via a tailored R function.

Abstract. Which data to analyze, and how, are fundamental questions of all empirical research. As there are always numerous flexibilities in data-analytic decisions (a “garden of forking paths”), this poses perennial problems to all empirical research. Specification-curve analysis and multiverse analysis have recently been proposed as solutions to these issues. Building on the structural analogies between primary data analysis and meta-analysis, we transform and adapt these approaches to the meta-analytic level, in tandem with combinatorial meta-analysis. We explain the rationale of this idea, suggest descriptive and inferential statistical procedures, as well as graphical displays, provide code for meta-analytic practitioners to generate and use these, and present a fully worked real example from digit ratio (2D:4D) research, totaling 1,592 meta-analytic specifications. Specification-curve and multiverse meta-analysis holds promise to resolve conflicting meta-analyses, contested evidence, controversial empirical literatures, and polarized research, and to mitigate the associated detrimental effects of these phenomena on research progress.

BackgroundData-visualization methods are essential to explore and communicate meta-analytic data and results. With a large number of novel graphs proposed quite recently, a comprehensive, up-to-date overview of available graphing options for meta-analysis is unavailable.MethodsWe applied a multi-tiered search strategy to find the meta-analytic graphs proposed and introduced so far. We checked more than 150 retrievable textbooks on research synthesis methodology cover to cover, six different software programs regularly used for meta-analysis, and the entire content of two leading journals on research synthesis. In addition, we conducted Google Scholar and Google image searches and cited-reference searches of prior reviews of the topic. Retrieved graphs were categorized into a taxonomy encompassing 11 main classes, evaluated according to 24 graph-functionality features, and individually presented and described with explanatory vignettes.ResultsWe ascertained more than 200 different graphs and graph variants used to visualize meta-analytic data. One half of these have accrued within the past 10 years alone. The most prevalent classes were graphs for network meta-analysis (45 displays), graphs showing combined effect(s) only (26), funnel plot-like displays (24), displays showing more than one outcome per study (19), robustness, outlier and influence diagnostics (15), study selection andp-value based displays (15), and forest plot-like displays (14). The majority of graphs (130, 62.5%) possessed a unique combination of graph features.ConclusionsThe rich and diverse set of available meta-analytic graphs offers a variety of options to display many different aspects of meta-analyses. This comprehensive overview of available graphs allows researchers to make better-informed decisions on which graphs suit their needs and therefore facilitates using the meta-analytic tool kit of graphs to its full potential. It also constitutes a roadmap for a goal-driven development of further graphical displays for research synthesis.

Summary Citations are widely used for measuring scientific impact. The goal of the present study was to predict citation counts of manuscripts submitted to Transplant International (TI) in the two calendar years following publication. We considered a comprehensive set of 21 manuscript, author, and peer‐review‐related predictor variables available early in the peer‐review process. We also evaluated how successfully the peer‐review process at TI identified and accepted the most promising manuscripts for publication. A developed predictive model with nine selected variables showed acceptable test performance to identify often cited articles (AUROC = 0.685). Particularly important predictors were the number of pages, month of publication, publication type (review versus other), and study on humans (yes versus no). Accepted manuscripts at TI were cited more often than rejected but elsewhere published manuscripts (median 4 vs. 2 citations). The predictive model did not outperform the actual editorial decision. Both findings suggest that the peer‐review process at TI, in its current form, was successful in selecting submitted manuscripts with a high scientific impact in the future. Predictive models might have the potential to support the review process when decisions are made under great uncertainty.

The Dark Triad of personality (Machiavellianism, narcissism, psychopathy) is widely considered conceptually important for individual differences research into personality and interpersonal behavior. Recent research suggests to add trait sadism to its defining constructs (i.e., to form a Dark Tetrad), and that a single common dimension (the Dark Core) underlies these dark personality traits. Taxometric studies suggest the Dark Triad traits are dimensional (i.e., quantitative), but investigations on the facet level are lacking and sex differences in dark personality traits have not been considered. Utilizing widely-used scales, this study investigated the Dark Triad traits, sadism, as well as the Dark Core of personality, with taxometric methods on both the aggregate and facet level and separately among men and women (total N = 2463, 56% women, mean age: 41.4 years). Dark personality traits mostly were dimensional, on both the aggregate and the facet level, and for both sexes. The Dark Core appeared to be taxonic among men, but dimensional among women. Taxon members were characterized by uniformly elevated dark personality traits and younger age. Future studies might profitably investigate the incremental predictive validity of this identified taxon and focus on further sex differences in dark personality traits.

Abbreviations: 2D:4D = digit ratio; CI = confidence interval; F = female; FtM = female-to-male transgender; M = male; MtF = male-to-female transgender; TGI = transgender identity

Abstract. Recent metascience investigations of the N-pact factor (NF; median sample size of studies published in a journal) have revealed NFs of merely about 100 in fields like social, sport, and exercise psychology. Journal NF has also been shown to correlate negatively with journal impact factors (JIF), implying that smaller studies appear in more prestigious journals. In this first long-term and largest NF analysis to date (3,699 articles coded), annual NFs of two personality psychology journals were tracked over 38 years since their inception in 1980. Overall NF was about 190, gradually increased over time, and within-journal NF-JIF correlations were positive. Online samples and articles featuring supplemental files presented larger NFs, whereas those involving student samples had smaller ones. Sample size distributions showed multimodality, and surplusses of even-numbered sample sizes and of those just beyond 100 were evident. An NF statement, accompanying authors’ submitted papers, is suggested.

While the conventional forest plot is useful to present results within subgroups of patients in clinical studies, it has been criticized for several reasons. First, small subgroups are visually overemphasized by long confidence interval lines, which is misleading. Second, the point estimates of large subgroups are difficult to discern because of the large box representing the precision of the estimate within subgroups. Third, confidence intervals depicted by lines might incorrectly convey the impression that all points within the interval are equally likely. Rainforest plots have been proposed to overcome these potentially misleading aspects of conventional forest plots. The metaviz package enables to generate rainforest plots for meta-analysis within the statistical computing environment R. We suggest the application of rainforest plots for the depiction of subgroup analysis in clinical trials. In this tutorial, detailed step-by-step guidance on the generation of rainforest plot for this purpose is provided.

Non-right-handedness appears to be more common among bisexuals and homosexuals than among heterosexuals, which might be indirect evidence of effects of prenatal androgen exposure. Current data suggest higher prenatal testosterone levels among bisexual and homosexual women, but are inconclusive for men. This study examined the association between sexual orientation and non-right-handedness for sex differences and whether higher rates of mixed-handedness, rather than left-handedness, might be the driving factor. This allowed for more specific tests regarding the predictions of two competing theories of prenatal androgen exposure, the Geschwind–Galaburda theory and the callosal hypothesis, than in previous research. Being a potentially better indicator of cerebral lateralization than handedness, associations with footedness were also explored. To counter inconsistencies and shortcomings of previous research, we utilized two large discovery and replication datasets (ns = 2368 and 1565) and applied latent variable analysis to reliably classify lateral preferences (i.e., handedness, footedness). This maximized the statistical conclusion validity and allowed for direct tests of replicability. Sexual orientation was differentially associated with lateral preferences among men and women. Associations among women were consistent with predictions of the Geschwind–Galaburda theory, whereas among men they were consistent with predictions of the callosal hypothesis. The results were further consistent with models of homosexuality that suggest a role of parental epigenetic marks on sexually dimorphic fetal development. Research efforts should be increased with regard to footedness and epigenetic theories of homosexuality.