Search

Abstract When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. We demonstrate that segmentation models trained with a small set of synthetic image data reach accuracy levels comparable to those of generalist models trained with a large and diverse collection of manually annotated image data, thereby offering a streamlined and specialized application of segmentation models.

Mitosis is the process by which eukaryotic cells divide to produce two similar daughter cells with identical genetic material. Research into the process of mitosis is therefore of critical importance both for the basic understanding of cell biology and for the clinical approach to manifold pathologies resulting from its malfunctioning, including cancer. In this paper, we propose an approach to study mitotic progression automatically using deep learning. We used neural networks to predict different mitosis stages. We extracted video sequences of cells undergoing division and trained a Recurrent Neural Network (RNN) to extract image features. The use of RNN enabled better extraction of features. The RNN-based approach gave better performance compared to classifier based feature extraction methods which do not use time information. Evaluation of precision, recall, and F-score indicates the superiority of the proposed model compared to the baseline. To study the loss in performance due to confusion between adjacent classes, we plotted the confusion matrix as well. In addition, we visualized the feature space to understand why RNNs are better at classifying the mitosis stages than other classifier models, which indicated the formation of strong clusters for the different classes, clearly confirming the advantage of the proposed RNN-based approach.

Abstract Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding "realistic image appearance", "anatomical correctness", and "consistency between slices". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).

Inability to image large numbers of growth plate chondrocytes while retaining their spatial context during analysis has hindered the study of bone development. Here, the authors present a pipeline called 3D MAPs and use it to uncover morphogenic behaviors and growth strategies in normal bones as well as aberrations in Gdf5 KO bones.

Liver cancer is one of the most frequently diagnosed and fatal cancers worldwide, with hepatocellular carcinoma (HCC) being the most common primary liver cancer. Hundreds of studies involving thousands of patients have now been analysed across different cancer types, including HCC, regarding the effects of immune infiltrates on the prognosis of cancer patients. However, for these analyses, an unambiguous delineation of the cancer area is paramount, which is difficult due to the strong heterogeneity and considerable inter-operator variability induced by qualitative visual assessment and manual assignment. Nowadays, however, multiplex analyses allow the simultaneous evaluation of multiple protein markers, which, in conjunction with recent machine learning approaches, may offer great potential for the objective, enhanced identification of cancer areas with further in situ analysis of prognostic immune parameters. In this study, we, therefore, used an exemplary five-marker multiplex immunofluorescence panel of commonly studied markers for prognosis (CD3 T, CD4 T helper, CD8 cytotoxic T, FoxP3 regulatory T, and PD-L1) and DAPI to assess which analytical approach is best suited to combine morphological and immunohistochemical data into a cancer score to identify the cancer area that best matches an independent pathologist’s assignment. For each cell, a total of 68 individual cell features were determined, which were used as input for 4 different approaches for computing a cancer score: a correlation-based selection of individual cell features, a MANOVA-based selection of features, a multilayer perceptron, and a convolutional neural network (a U-net). Accuracy was used to evaluate performance. With a mean accuracy of 75%, the U-net was best capable of identifying the cancer area. Although individual cell features showed a strong heterogeneity between patients, the spatial representations obtained with the computed cancer scores delineate HCC well from non-cancer liver tissues. Future analyses with larger sample sizes will help to improve the model and enable direct, in-depth investigations of prognostic parameters, ultimately enabling precision medicine.

Positional information is a central concept in developmental biology. In developing organs, positional information can be idealized as a local coordinate system that arises from morphogen gradients controlled by organizers at key locations. This offers a plausible mechanism for the integration of the molecular networks operating in individual cells into the spatially coordinated multicellular responses necessary for the organization of emergent forms. Understanding how positional cues guide morphogenesis requires the quantification of gene expression and growth dynamics in the context of their underlying coordinate systems. Here, we present recent advances in the MorphoGraphX software (Barbier de Reuille et al., 2015⁠) that implement a generalized framework to annotate developing organs with local coordinate systems. These coordinate systems introduce an organ-centric spatial context to microscopy data, allowing gene expression and growth to be quantified and compared in the context of the positional information thought to control them.

Abstract Accurate quantification of heartbeats in fish models is an important readout to study cardiovascular biology, disease states and pharmacology. However, dependence on anaesthesia, laborious sample orientation or requirement for fluorescent reporters have hampered the use of high-throughput heartbeat analysis. To overcome these limitations, we established an efficient screening assay employing automated label-free heart rate determination of randomly oriented, non-anesthetized medaka (Oryzias latipes) and zebrafish (Danio rerio) embryos in microtiter plates. Automatically acquired bright-field data feeds into an easy-to-use HeartBeat software with graphical user interface for automated quantification of heart rate and rhythm. Sensitivity of the assay was demonstrated by profiling heart rates during entire embryonic development. Our analysis revealed rapid adaption of heart rates to temperature changes, which has implications for standardization of experimental layout. The assay allows scoring of multiple embryos per well enabling a throughput of >500 embryos per 96-well plate. In a proof of principle screen for compound testing, we captured concentration-dependent effects of nifedipine and terfenadine over time. Our novel assay permits large-scale applications ranging from phenotypic screening, interrogation of gene functions to cardiovascular drug development.

Foreign bodies such as fibers of a surgical mesh induce a typical reaction with an inflammatory infiltrate that forms a surrounding granuloma. This infiltrate is dominated by macrophages, lymphocytes, and neutrophils, whereas its extent of collaboration is widely unknown. In this study, we analyzed 12 samples of surgical meshes explanted from humans by multiplex analyses with three different 5-marker panels – 1. macrophage panel: CD68, CD86, CD105, CD163, and CD206; 2. lymphocyte panel: CD3, CD4, CD8, CD20, and CD68; and 3. neutrophil panel: CD15, histone, MPO, NE, and CD68. Measurement of fluorescence intensity within nuclear masks resulting from DAPI nuclear staining allows exact quantification of cells considered “positive” at a user-defined mean intensity threshold of > 100. Obviously, however, there is no natural threshold as a biological criterion for an intensity that separates “positive” stained cells from unstained cells (“negative”). Multiplex staining of 5 markers always reveals a high rate of coexpression for almost all of the 25 possible marker combinations (= 32 combinations, when using 5 markers simultaneously). The present staining results demonstrate that various morphological and functional subtypes of macrophages, lymphocytes, and neutrophils are abundant in the foreign body granuloma (FBG), which were investigated by regions of interest (ROI) with an area of 1 mm2. The widespread coexpression of two or more markers underscores the complex collaboration network of the inflammatory infiltrate. The ability to combine spatial distribution with exact numerical analysis may offer new perspectives for our understanding of the complex interactions in this multidimensional process.

Mechanobiology requires precise quantitative information on processes taking place in specific 3D microenvironments. Connecting the abundance of microscopical, molecular, biochemical, and cell mechanical data with defined topologies has turned out to be extremely difficult. Establishing such structural and functional 3D maps needed for biophysical modeling is a particular challenge for the cytoskeleton, which consists of long and interwoven filamentous polymers coordinating subcellular processes and interactions of cells with their environment. To date, useful tools are available for the segmentation and modeling of actin filaments and microtubules but comprehensive tools for the mapping of intermediate filament organization are still lacking. In this work, we describe a workflow to model and examine the complete 3D arrangement of the keratin intermediate filament cytoskeleton in canine, murine, and human epithelial cells both, in vitro and in vivo. Numerical models are derived from confocal airyscan high-resolution 3D imaging of fluorescence-tagged keratin filaments. They are interrogated and annotated at different length scales using different modes of visualization including immersive virtual reality. In this way, information is provided on network organization at the subcellular level including mesh arrangement, density and isotropic configuration as well as details on filament morphology such as bundling, curvature, and orientation. We show that the comparison of these parameters helps to identify, in quantitative terms, similarities and differences of keratin network organization in epithelial cell types defining subcellular domains, notably basal, apical, lateral, and perinuclear systems. The described approach and the presented data are pivotal for generating mechanobiological models that can be experimentally tested.

Live-cell imaging has become state of the art to accurately identify the nature of mitotic and cell cycle defects. Low- and high-throughput microscopy setups have yield huge data amounts of cells recorded in different experimental and pathological conditions. Tailored semi-automated and automated image analysis approaches allow the analysis of high-content screening data sets, saving time and avoiding bias. However, they were mostly designed for very specific experimental setups, which restricts their flexibility and usability. The general need for dedicated experiment-specific user-annotated training sets and experiment-specific user-defined segmentation parameters remains a major bottleneck for fully automating the analysis process. In this work we present LiveCellMiner, a highly flexible open-source software tool to automatically extract, analyze and visualize both aggregated and time-resolved image features with potential biological relevance. The software tool allows analysis across high-content data sets obtained in different platforms, in a quantitative and unbiased manner. As proof of principle application, we analyze here the dynamic chromatin and tubulin cytoskeleton features in human cells passing through mitosis highlighting the versatile and flexible potential of this tool set.

Automated image processing approaches are indispensable for many biomedical experiments and help to cope with the increasing amount of microscopy image data in a fast and reproducible way. Especially state-of-the-art deep learning-based approaches most often require large amounts of annotated training data to produce accurate and generalist outputs, but they are often compromised by the general lack of those annotated data sets. In this work, we propose how conditional generative adversarial networks can be utilized to generate realistic image data for 3D fluorescence microscopy from annotation masks of 3D cellular structures. In combination with mask simulation approaches, we demonstrate the generation of fully-annotated 3D microscopy data sets that we make publicly available for training or benchmarking. An additional positional conditioning of the cellular structures enables the reconstruction of position-dependent intensity characteristics and allows to generate image data of different quality levels. A patch-wise working principle and a subsequent full-size reassemble strategy is used to generate image data of arbitrary size and different organisms. We present this as a proof-of-concept for the automated generation of fully-annotated training data sets requiring only a minimum of manual interaction to alleviate the need of manual annotations.

The glucose-sensing Mondo pathway regulates expression of metabolic genes in mammals. Here, we characterized its function in the zebrafish and revealed an unexpected role of this pathway in vertebrate embryonic development. We showed that knockdown of mondoa impaired the early morphogenetic movement of epiboly in zebrafish embryos and caused microtubule defects. Expression of genes in the terpenoid backbone and sterol biosynthesis pathways upstream of pregnenolone synthesis was coordinately downregulated in these embryos, including the most downregulated gene nsdhl. Loss of Nsdhl function likewise impaired epiboly, similar to MondoA loss of function. Both epiboly and microtubule defects were partially restored by pregnenolone treatment. Maternal-zygotic mutants of mondoa showed perturbed epiboly with low penetrance and compensatory changes in the expression of terpenoid/sterol/steroid metabolism genes. Collectively, our results show a novel role for MondoA in the regulation of early vertebrate development, connecting glucose, cholesterol and steroid hormone metabolism with early embryonic cell movements.

Cranial neural crest (NC) contributes to the developing vertebrate eye. By multidimensional, quantitative imaging, we traced the origin of the ocular NC cells to two distinct NC populations that differ in the maintenance of sox10 expression, Wnt signalling, origin, route, mode and destination of migration. The first NC population migrates to the proximal and the second NC cell group populates the distal (anterior) part of the eye. By analysing zebrafish pax6a/b compound mutants presenting anterior segment dysgenesis, we demonstrate that Pax6a/b guide the two NC populations to distinct proximodistal locations. We further provide evidence that the lens whose formation is pax6a/b-dependent and lens-derived TGFβ signals contribute to the building of the anterior segment. Taken together, our results reveal multiple roles of Pax6a/b in the control of NC cells during development of the anterior segment.

Signaling filopodia, termed cytonemes, are dynamic actin-based membrane structures that regulate the exchange of signaling molecules and their receptors within tissues. However, how cytoneme formation is regulated remains unclear. Here, we show that Wnt/planar cell polarity (PCP) autocrine signaling controls the emergence of cytonemes, and that cytonemes subsequently control paracrine Wnt/β-catenin signal activation. Upon binding of the Wnt family member Wnt8a, the receptor tyrosine kinase Ror2 becomes activated. Ror2/PCP signaling leads to the induction of cytonemes, which mediate the transport of Wnt8a to neighboring cells. In the Wnt-receiving cells, Wnt8a on cytonemes triggers Wnt/β-catenin-dependent gene transcription and proliferation. We show that cytoneme-based Wnt transport operates in diverse processes, including zebrafish development, murine intestinal crypt and human cancer organoids, demonstrating that Wnt transport by cytonemes and its control via the Ror2 pathway is highly conserved in vertebrates.

State-of-the-art light-sheet and confocal microscopes allow recording of entire embryos in 3D and over time (3D+t) for many hours. Fluorescently labeled structures can be segmented and tracked automatically in these terabyte-scale 3D+t images, resulting in thousands of cell migration trajectories that provide detailed insights to large-scale tissue reorganization at the cellular level. Here we present EmbryoMiner, a new interactive open-source framework suitable for in-depth analyses and comparisons of entire embryos, including an extensive set of trajectory features. Starting at the whole-embryo level, the framework can be used to iteratively focus on a region of interest within the embryo, to investigate and test specific trajectory-based hypotheses and to extract quantitative features from the isolated trajectories. Thus, the new framework provides a valuable new way to quantitatively compare corresponding anatomical regions in different embryos that were manually selected based on biological prior knowledge. As a proof of concept, we analyzed 3D+t light-sheet microscopy images of zebrafish embryos, showcasing potential user applications that can be performed using the new framework.

[This corrects the article DOI: 10.1371/journal.pone.0187535.].

Many automatically analyzable scientific questions are well-posed and a variety of information about expected outcomes is available a priori. Although often neglected, this prior knowledge can be systematically exploited to make automated analysis operations sensitive to a desired phenomenon or to evaluate extracted content with respect to this prior knowledge. For instance, the performance of processing operators can be greatly enhanced by a more focused detection strategy and by direct information about the ambiguity inherent in the extracted data. We present a new concept that increases the result quality awareness of image analysis operators by estimating and distributing the degree of uncertainty involved in their output based on prior knowledge. This allows the use of simple processing operators that are suitable for analyzing large-scale spatiotemporal (3D+t) microscopy images without compromising result quality. On the foundation of fuzzy set theory, we transform available prior knowledge into a mathematical representation and extensively use it to enhance the result quality of various processing operators. These concepts are illustrated on a typical bioimage analysis pipeline comprised of seed point detection, segmentation, multiview fusion and tracking. The functionality of the proposed approach is further validated on a comprehensive simulated 3D+t benchmark data set that mimics embryonic development and on large-scale light-sheet microscopy data of a zebrafish embryo. The general concept introduced in this contribution represents a new approach to efficiently exploit prior knowledge to improve the result quality of image analysis pipelines. The generality of the concept makes it applicable to practically any field with processing strategies that are arranged as linear pipelines. The automated analysis of terabyte-scale microscopy data will especially benefit from sophisticated and efficient algorithms that enable a quantitative and fast readout.

Automated analysis of multi-dimensional microscopy images has become an integral part of modern research in life science. Most available algorithms that provide sufficient segmentation quality, however, are infeasible for a large amount of data due to their high complexity. In this contribution we present a fast parallelized segmentation method that is especially suited for the extraction of stained nuclei from microscopy images, e.g., of developing zebrafish embryos. The idea is to transform the input image based on gradient and normal directions in the proximity of detected seed points such that it can be handled by straightforward global thresholding like Otsu's method. We evaluate the quality of the obtained segmentation results on a set of real and simulated benchmark images in 2D and 3D and show the algorithm's superior performance compared to other state-of-the-art algorithms. We achieve an up to ten-fold decrease in processing times, allowing us to process large data sets while still providing reasonable segmentation results.

Mathematical modeling of biochemical processes significantly contributes to a better understanding of biological functionality and underlying dynamic mechanisms. To support time consuming and costly lab experiments, kinetic reaction equations can be formulated as a set of ordinary differential equations, which in turn allows to simulate and compare hypothetical models in silico. To identify new experimental designs that are able to discriminate between investigated models, the approach used in this work solves a semi-infinite constrained nonlinear optimization problem using derivative based numerical algorithms. The method takes into account parameter variabilities such that new experimental designs are robust against parameter changes while maintaining the optimal potential to discriminate between hypothetical models. In this contribution we present a newly developed software tool that offers a convenient graphical user interface for model discrimination. We demonstrate the beneficial operation of the discrimination approach and the usefulness of the software tool by analyzing a realistic benchmark experiment from literature. New robust optimal designs that allow to discriminate between the investigated model hypotheses of the benchmark experiment are successfully calculated and yield promising results. The involved robustification approach provides maximally discriminating experiments for the worst parameter configurations, which can be used to estimate the meaningfulness of upcoming experiments. A major benefit of the graphical user interface is the ability to interactively investigate the model behavior and the clear arrangement of numerous variables. In addition to a brief theoretical overview of the discrimination method and the functionality of the software tool, the importance of robustness of experimental designs against parameter variability is demonstrated on a biochemical benchmark problem. The software is licensed under the GNU General Public License and freely available at http://sourceforge.net/projects/mdtgui/.