Search

The field of 3D medical image segmentation is witnessing a growing trend in the utilization of combined networks that integrate convolutional neural networks and transformers. Nevertheless, prevailing hybrid networks are confronted with limitations in their straightforward serial or parallel combination methods and lack an effective mechanism to fuse channel and spatial feature attention. To address these limitations, we present a robust multi-scale 3D medical image segmentation network, the Transformer-Driven Pyramid Attention Fusion Network, which is denoted as TPAFNet, leveraging a hybrid structure of CNN and transformer. Within this framework, we exploit the characteristics of atrous convolution to extract multi-scale information effectively, thereby enhancing the encoding results of the transformer. Furthermore, we introduce the TPAF block in the encoder to seamlessly fuse channel and spatial feature attention from multi-scale feature inputs. In contrast to conventional skip connections that simply concatenate or add features, our decoder is enriched with a TPAF connection, elevating the integration of feature attention between low-level and high-level features. Additionally, we propose a low-level encoding shortcut from the original input to the decoder output, preserving more original image features and contributing to enhanced results. Finally, the deep supervision is implemented using a novel CNN-based voxel-wise classifier to facilitate better network convergence. Experimental results demonstrate that TPAFNet significantly outperforms other state-of-the-art networks on two public datasets, indicating that our research can effectively improve the accuracy of medical image segmentation, thereby assisting doctors in making more precise diagnoses.

Glioma segmentation is a crucial task in computer-aided diagnosis, requiring precise discrimination between lesions and normal tissue at the pixel level. Popular methods neglect crucial edge information, leading to inaccurate contour delineation. Moreover, global information has been proven beneficial for segmentation. The feature representations extracted by convolution neural networks often struggle with local-related information owing to the limited receptive fields. To address these issues, we propose a novel edge-aware segmentation network that incorporates a dual-path gradient-guided training strategy with Fourier edge-enhancement for precise glioma segmentation, a.k.a. GFNet. First, we introduce a Dual-path Gradient-guided Training strategy (DGT) based on a Siamese network guiding the optimizing direction of one path by the gradient from the other path. DGT pays attention to the indistinguishable pixels with large weight-updating gradient, such as the pixels near the boundary, to guide the network training, addressing hard samples. Second, to further perceive the edge information, we derive a Fourier Edge-enhancement Module (FEM) to augment feature edges with high-frequency representations from the spectral domain, providing global information and edge details. Extensive experiments on public glioma segmentation datasets, BraTS2020 and Medical Segmentation Decathlon (MSD) glioma and prostate segmentation, demonstrate that GFNet achieves competitive performance compared to other state-of-the-art methods, both qualitatively and quantitatively.

Magneto-acousto-electrical tomography (MAET) is a hybrid imaging method that combines the high spatial resolution of ultrasonography with the high contrast of electrical impedance tomography (EIT). While most previous studies on MAET have focused on two-dimensional imaging, our recent research proposed a novel three-dimensional (3D) MAET method using B-mode and translational scanning. This method has been the first to reconstruct a 3D volume image of conductivity interfaces. However, this method has its limitations in mapping irregular shapes of conductivity. To address this challenge, we propose a 3D magneto-acousto-electrical computed tomography (3D MAE-CT) method utilizing an ultrasound linear array transducer in this work. Both phantom and in vitro experiments were conducted to validate our proposed method. The results from the phantom experiments demonstrate that our method can map the 3D volume conductivity with high spatial resolution. The oblique angles extracted from the 3D image closely match practical value, with the relative error ranging between -2.80% and 4.07%. Furthermore, the in vitro experiment successfully obtained a 3D image of a chicken heart, marking the first MAET 3D conductivity image of a tissue sample to date.

Objective: The paper proposes a non-invasive original methodology to directly and automatically identify the spine line from the external position of vertebral apophyses, which are key anatomical landmarks., Methods: Apophyses are detected directly on discrete high-density geometric models of human backs acquired by a 3D scanner. The methodology is inspired by the posturologist's approach that detects the spine line through the identification, by manual palpation, of the spinal apophyses. For this purpose, an appropriate shape index is used to identify vertebral positions. The shape index estimates the local differential geometric properties of the back surface. This index is very discriminating in locating both pronounced and blurred apophyses. To validate the method, the research involved the analysis of 21 healthy human backs acquired in both standing and asymmetric postures. For each of them, a skilled operator detected the spinal apophyses by tactile investigation and located them through cutaneous marking. Markers have been used as the reference for spinal apophyses' positions., Results: A comparison of the proposed approach with state-of-the-art methods has been conducted. This study evidences the high accuracy of the methodology proposed here and the capability to recognize also blurred apophyses., Conclusion: The method automatically performs the spine line identification and accurately locates apophyses along both vertical and coronal directions., Significance: The proposed inexpensive and easy-to-use approach significantly advances over other non-invasive methods. Its ability to detect the apophyses' location potentially offers new capabilities in detecting, diagnosing, and monitoring spinal disorders.

Objective: In clinical ultrasound, current 2-D strain imaging faces challenges in quantifying three orthogonal normal strain components. This requires separate image acquisitions based on the pixel-dependent cardiac coordinate system, leading to additional computations and estimation discrepancies due to probe orientation. Most systems lack shear strain information, as displaying all components is challenging to interpret., Methods: This paper presents a 3-D high-spatial-resolution, coordinate-independent strain imaging approach based on principal stretch and axis estimation. All strain components are transformed into three principal stretches along three normal principal axes, enabling direct visualization of the primary deformation. We devised an efficient 3-D speckle tracking method with tilt filtering, incorporating randomized searching in a two-pass tracking framework and rotating the phase of the 3-D correlation function for robust filtering. The proposed speckle tracking approach significantly improves estimates of displacement gradients related to the axial displacement component. Non-axial displacement gradient estimates are enhanced using a correlation-weighted least-squares method constrained by tissue incompressibility., Results: Simulated and in vivo canine cardiac datasets were evaluated to estimate Lagrangian strains from end-diastole to end-systole. The proposed speckle tracking method improves displacement estimation by a factor of 4.3 to 10.5 over conventional 1-pass processing. Maximum principal axis/direction imaging enables better detection of local disease regions than conventional strain imaging., Conclusion: Coordinate-independent tracking can identify myocardial abnormalities with high accuracy., Significance: This study offers enhanced accuracy and robustness in strain imaging compared to current methods.

Accurately reconstructing 4D critical organs contributes to the visual guidance in X-ray image-guided interventional operation. Current methods estimate intraoperative dynamic meshes by refining a static initial organ mesh from the semantic information in the single-frame X-ray images. However, these methods fall short of reconstructing an accurate and smooth organ sequence due to the distinct respiratory patterns between the initial mesh and X-ray image. To overcome this limitation, we propose a novel dual-stage complementary 4D organ reconstruction (DSC-Recon) model for recovering dynamic organ meshes by utilizing the preoperative and intraoperative data with different respiratory patterns. DSC-Recon is structured as a dual-stage framework: 1) The first stage focuses on addressing a flexible interpolation network applicable to multiple respiratory patterns, which could generate dynamic shape sequences between any pair of preoperative 3D meshes segmented from CT scans. 2) In the second stage, we present a deformation network to take the generated dynamic shape sequence as the initial prior and explore the discriminate feature (i.e., target organ areas and meaningful motion information) in the intraoperative X-ray images, predicting the deformed mesh by introducing a designed feature mapping pipeline integrated into the initialized shape refinement process. Experiments on simulated and clinical datasets demonstrate the superiority of our method over state-of-the-art methods in both quantitative and qualitative aspects.

Objective: A novel small form factor circular electrode array was designed specifically for electrical impedance tomography (EIT) based assessment of surgical margins during robot assisted radical prostatectomy (RARP)., Methods: The electrode array consists of 33 gold-plated electrodes arranged within a 9.5 mm diameter circular footprint on the end of a surgical probe that can be introduced through a standard 12 mm laparoscopic port used during RARP. The electrode array contains 8 larger, low-contact impedance outer electrodes dedicated for current drive and an internal grid of 25 smaller electrodes for simultaneous voltage measurement. Separating electrode geometry by function is designed to improve current delivery, speed, and resolution while reducing hardware requirements., Results: Simulations demonstrated that 1 mm diameter hemispherical prostate cancer inclusions could be localized within regions of adipose and benign prostate tissue; 1.5 mm diameter inclusions were required for localization within muscle tissue. A 2.38 mm diameter aluminum rod in 0.2 S/m saline could be localized throughout the imaging domain with a position error of less than 2.5 mm for depths from the electrode array surface of up to 1.7 mm. Ex vivo tissue experiments with a bovine model demonstrate visual congruence of muscle and adipose tissue locations between the sample and reconstructed images., Conclusion: Simulation and experimental results indicate good detection and location of inclusions., Significance: These results suggest the proposed electrode array design can provide sufficient accuracy in the detection and localization of prostate cancer against clinically relevant background tissues for use during RARP.

Objective: Paraspinal muscle segmentation and reconstruction from MR images are critical to implement quantitative assessment of chronic and recurrent low back pains. Due to unclear muscle boundaries and shape variations, current segmentation methods demonstrate suboptimal performance with insufficient training samples. This work proposes a novel approach to modeling and inferring muscle shapes that enhances segmentation accuracy and efficiency with few training data., Methods: Firstly, a probabilistic shape model (PSM) based on Fourier basis functions and Gaussian processes (GPs) is designed to encode 3D muscle shapes, where anatomical meanings are attributed to the model's geometric parameters. Muscle shape variations and correlations are described by the GPs of the geometric parameters, which allow a small size of parameters to model the distribution of muscle shapes. Secondly, a Bayesian framework is developed to achieve entire muscle segmentation by posterior estimations. The framework fuses the geometric prior of the PSM with observations of deep-learning-based edge detections (DED) and sparse manual annotations, by which issues of unclear boundaries and shape variations can be compensated., Results and Conclusion: Experiments on public and clinical datasets demonstrate that, with just three manually annotated slices, our method achieves a Dice similarity coefficient exceeding 90%, which outperforms other methods. Meanwhile, our method needs only a small training dataset and offers rapid inference speeds in clinical applications., Significance: Our study enables precise assessment of paraspinal muscles in 2D and 3D, aiding clinicians and researchers in understanding muscle changes in various conditions, potentially enhancing treatment outcomes.

Rigid motion tracking is paramount in many medical imaging applications where movements need to be detected, corrected, or accounted for. Modern strategies rely on convolutional neural networks (CNN) and pose this problem as rigid registration. Yet, CNNs do not exploit natural symmetries in this task, as they are equivariant to translations (their outputs shift with their inputs) but not to rotations. Here we propose EquiTrack, the first method that uses recent steerable SE(3)-equivariant CNNs (E-CNN) for motion tracking. While steerable E-CNNs can extract corresponding features across different poses, testing them on noisy medical images reveals that they do not have enough learning capacity to learn noise invariance. Thus, we introduce a hybrid architecture that pairs a denoiser with an E-CNN to decouple the processing of anatomically irrelevant intensity features from the extraction of equivariant spatial features. Rigid transforms are then estimated in closed-form. EquiTrack outperforms state-of-the-art learning and optimisation methods for motion tracking in adult brain MRI and fetal MRI time series. Our code is available at https://github.com/BBillot/EquiTrack.

The self-supervised monocular depth estimation framework is well-suited for medical images that lack ground-truth depth, such as those from digestive endoscopes, facilitating navigation and 3D reconstruction in the gastrointestinal tract. However, this framework faces several limitations, including poor performance in low-texture environments, limited generalisation to real-world datasets, and unclear applicability in downstream tasks like visual servoing. To tackle these challenges, we propose MonoLoT, a self-supervised monocular depth estimation framework featuring two key innovations: point matching loss and batch image shuffle. Extensive ablation studies on two publicly available datasets, namely C3VD and SimCol, have shown that methods enabled by MonoLoT achieve substantial improvements, with accuracies of 0.944 on C3VD and 0.959 on SimCol, surpassing both depth-supervised and self-supervised baselines on C3VD. Qualitative evaluations on real-world endoscopic data underscore the generalisation capabilities of our methods, outperforming both depth-supervised and self-supervised baselines. To demonstrate the feasibility of using monocular depth estimation for visual servoing, we have successfully integrated our method into a proof-of-concept robotic platform, enabling real-time automatic intervention and control in digestive endoscopy. In summary, our method represents a significant advancement in monocular depth estimation for digestive endoscopy, overcoming key challenges and opening promising avenues for medical applications.

Deep learning methods have recently achieved remarkable performance in vessel segmentation applications, yet require numerous labor-intensive labeled data. To alleviate the requirement of manual annotation, transfer learning methods can potentially be used to acquire the related knowledge of tubular structures from public large-scale labeled vessel datasets for target vessel segmentation in other anatomic sites of the human body. However, the cross-anatomy domain shift is a challenging task due to the formidable discrepancy among various vessel structures in different anatomies, resulting in the limited performance of transfer learning. Therefore, we propose a cross-anatomy transfer learning framework for 3D vessel segmentation, which first generates a pre-trained model on a public hepatic vessel dataset and then adaptively fine-tunes our target segmentation network initialized from the model for segmentation of other anatomic vessels. In the framework, the adaptive fine-tuning strategy is presented to dynamically decide on the frozen or fine-tuned filters of the target network for each input sample with a proxy network. Moreover, we develop a Gaussian-based signed distance map that explicitly encodes vessel-specific shape context. The prediction of the map is added as an auxiliary task in the segmentation network to capture geometry-aware knowledge in the fine-tuning. We demonstrate the effectiveness of our method through extensive experiments on two small-scale datasets of coronary artery and brain vessel. The results indicate the proposed method effectively overcomes the discrepancy of cross-anatomy domain shift to achieve accurate vessel segmentation for these two datasets.

This paper introduces an innovative methodology for producing high-quality 3D lung CT images guided by textual information. While diffusion-based generative models are increasingly used in medical imaging, current state-of-the-art approaches are limited to low-resolution outputs and underutilize radiology reports' abundant information. The radiology reports can enhance the generation process by providing additional guidance and offering fine-grained control over the synthesis of images. Nevertheless, expanding text-guided generation to high-resolution 3D images poses significant memory and anatomical detail-preserving challenges. Addressing the memory issue, we introduce a hierarchical scheme that uses a modified UNet architecture. We start by synthesizing low-resolution images conditioned on the text, serving as a foundation for subsequent generators for complete volumetric data. To ensure the anatomical plausibility of the generated samples, we provide further guidance by generating vascular, airway, and lobular segmentation masks in conjunction with the CT images. The model demonstrates the capability to use textual input and segmentation tasks to generate synthesized images. Algorithmic comparative assessments and blind evaluations conducted by 10 board-certified radiologists indicate that our approach exhibits superior performance compared to the most advanced models based on GAN and diffusion techniques, especially in accurately retaining crucial anatomical features such as fissure lines and airways. This innovation introduces novel possibilities. This study focuses on two main objectives: (1) the development of a method for creating images based on textual prompts and anatomical components, and (2) the capability to generate new images conditioning on anatomical elements. The advancements in image generation can be applied to enhance numerous downstream tasks.

Medical imaging is limited by acquisition time and scanning equipment. CT and MR volumes, reconstructed with thicker slices, are anisotropic with high in-plane resolution and low through-plane resolution. We reveal an intriguing phenomenon that due to the mentioned nature of data, performing slice-wise interpolation from the axial view can yield greater benefits than performing super-resolution from other views. Based on this observation, we propose an Inter-Intra-slice Interpolation Network ( [Formula: see text]Net), which fully explores information from high in-plane resolution and compensates for low through-plane resolution. The through-plane branch supplements the limited information contained in low through-plane resolution from high in-plane resolution and enables continual and diverse feature learning. In-plane branch transforms features to the frequency domain and enforces an equal learning opportunity for all frequency bands in a global context learning paradigm. We further propose a cross-view block to take advantage of the information from all three views online. Extensive experiments on two public datasets demonstrate the effectiveness of [Formula: see text]Net, and noticeably outperforms state-of-the-art super-resolution, video frame interpolation and slice interpolation methods by a large margin. We achieve 43.90dB in PSNR, with at least 1.14dB improvement under the upscale factor of ×2 on MSD dataset with faster inference. Code is available at https://github.com/DeepMed-Lab-ECNU/Medical-Image-Reconstruction.

Image-guided interventional oncology procedures can greatly enhance the outcome of cancer treatment. As an enhancing procedure, oncology smart material delivery can increase cancer therapy's quality, effectiveness, and safety. However, the effectiveness of enhancing procedures highly depends on the accuracy of smart material placement procedures. Inaccurate placement of smart materials can lead to adverse side effects and health hazards. Image guidance can considerably improve the safety and robustness of smart material delivery. In this study, we developed a novel generative deep-learning platform that highly prioritizes clinical practicality and provides the most informative intra-operative feedback for image-guided smart material delivery. XIOSIS generates a patient-specific 3D volumetric computed tomography (CT) from three intraoperative radiographs (X-ray images) acquired by a mobile C-arm during the operation. As the first of its kind, XIOSIS (i) synthesizes the CT from small field-of-view radiographs;(ii) reconstructs the intra-operative spacer distribution; (iii) is robust; and (iv) is equipped with a novel soft-contrast cost function. To demonstrate the effectiveness of XIOSIS in providing intra-operative image guidance, we applied XIOSIS to the duodenal hydrogel spacer placement procedure. We evaluated XIOSIS performance in an image-guided virtual spacer placement and actual spacer placement in two cadaver specimens. XIOSIS showed a clinically acceptable performance, reconstructed the 3D intra-operative hydrogel spacer distribution with an average structural similarity of 0.88 and Dice coefficient of 0.63 and with less than 1 cm difference in spacer location relative to the spinal cord.

Adapting a medical image segmentation model to a new domain is important for improving its cross-domain transferability, and due to the expensive annotation process, Unsupervised Domain Adaptation (UDA) is appealing where only unlabeled images are needed for the adaptation. Existing UDA methods are mainly based on image or feature alignment with adversarial training for regularization, and they are limited by insufficient supervision in the target domain. In this paper, we propose an enhanced Filtered Pseudo Label (FPL+)-based UDA method for 3D medical image segmentation. It first uses cross-domain data augmentation to translate labeled images in the source domain to a dual-domain training set consisting of a pseudo source-domain set and a pseudo target-domain set. To leverage the dual-domain augmented images to train a pseudo label generator, domain-specific batch normalization layers are used to deal with the domain shift while learning the domain-invariant structure features, generating high-quality pseudo labels for target-domain images. We then combine labeled source-domain images and target-domain images with pseudo labels to train a final segmentor, where image-level weighting based on uncertainty estimation and pixel-level weighting based on dual-domain consensus are proposed to mitigate the adverse effect of noisy pseudo labels. Experiments on three public multi-modal datasets for Vestibular Schwannoma, brain tumor and whole heart segmentation show that our method surpassed ten state-of-the-art UDA methods, and it even achieved better results than fully supervised learning in the target domain in some cases.

Objective: Major challenges for clinical applications of in silico medicine are limitations in time and computational resources. Computational approaches should therefore be tailored to specific applications with relatively low complexity and must be verified and validated against clinical gold standards., Methods: This study performed computational fluid dynamics simulations of left ventricular hemodynamics of different complexity based on shape reconstruction from steady state gradient echo magnetic resonance imaging (MRI) data. Computed flow results of a rigid wall model (RWM) and a prescribed motion fluid-structure interaction (PM-FSI) model were compared against phase-contrast MRI measurements for three healthy subjects., Results: Extracted boundary conditions from the steady state MRI sequences as well as computed metrics, such as flow rate, valve velocities, and kinetic energy show good agreement with in vivo flow measurements. Regional flow analysis reveals larger differences., Conclusion: Basic flow structures are well captured with RWM and PM-FSI. For the computation of further biomarkers like washout or flow efficiency, usage of PM-FSI is required. Regarding boundary-near flow, more accurate anatomical models are inevitable., Significance: These results delineate areas of application of both methods and lay a foundation for larger validation studies and sensitivity analysis for healthy and diseased cases, being an essential step upon clinical translations.

Owing to the success of transformer models, recent works study their applicability in 3D medical segmentation tasks. Within the transformer models, the self-attention mechanism is one of the main building blocks that strives to capture long-range dependencies, compared to the local convolutional-based design. However, the self-attention operation has quadratic complexity which proves to be a computational bottleneck, especially in volumetric medical imaging, where the inputs are 3D with numerous slices. In this paper, we propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features using a pair of inter-dependent branches based on spatial and channel attention. Our spatial attention formulation is efficient and has linear complexity with respect to the input. To enable communication between spatial and channel-focused branches, we share the weights of query and key mapping functions that provide a complimentary benefit (paired attention), while also reducing the complexity. Our extensive evaluations on five benchmarks, Synapse, BTCV, ACDC, BraTS, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy. On Synapse, our UNETR++ sets a new state-of-the-art with a Dice Score of 87.2%, while significantly reducing parameters and FLOPs by over 71%, compared to the best method in the literature. Our code and models are available at: https://tinyurl.com/2p87x5xn.

Self-supervised learning (SSL) has emerged as a powerful technique for improving the efficiency and effectiveness of deep learning models. Contrastive methods are a prominent family of SSL that extract similar representations of two augmented views of an image while pushing away others in the representation space as negatives. However, the state-of-the-art contrastive methods require large batch sizes and augmentations designed for natural images that are impractical for 3D medical images. To address these limitations, we propose a new longitudinal SSL method, 3DTINC, based on non-contrastive learning. It is designed to learn perturbation-invariant features for 3D optical coherence tomography (OCT) volumes, using augmentations specifically designed for OCT. We introduce a new non-contrastive similarity loss term that learns temporal information implicitly from intra-patient scans acquired at different times. Our experiments show that this temporal information is crucial for predicting progression of retinal diseases, such as age-related macular degeneration (AMD). After pretraining with 3DTINC, we evaluated the learned representations and the prognostic models on two large-scale longitudinal datasets of retinal OCTs where we predict the conversion to wet-AMD within a six-month interval. Our results demonstrate that each component of our contributions is crucial for learning meaningful representations useful in predicting disease progression from longitudinal volumetric scans.

3D Imaging of the human heart at high frame rate is of major interest for various clinical applications. Electronic complexity and cost has prevented the dissemination of 3D ultrafast imaging into the clinic. Row column addressed (RCA) transducers provide volumetric imaging at ultrafast frame rate by using a low electronic channel count, but current models are ill-suited for transthoracic cardiac imaging due to field-of-view limitations. In this study, we proposed a mechanically curved RCA with an aperture adapted for transthoracic cardiac imaging ( 24×16 mm2). The RCA has a toroidal curved surface of 96 elements along columns (curvature radius rC = 4.47 cm) and 64 elements along rows (curvature radius rR = 3 cm). We implemented delay and sum beamforming with an analytical calculation of the propagation of a toroidal wave which was validated using simulations (Field II). The imaging performance was evaluated on a calibrated phantom. Experimental 3D imaging was achieved up to 12 cm deep with a total angular aperture of 30° for both lateral dimensions. The Contrast-to-Noise ratio increased by 12 dB from 2 to 128 virtual sources. Then, 3D Ultrasound Localization Microscopy (ULM) was characterized in a sub-wavelength tube diameter. Finally, 3D ULM was demonstrated on a perfused ex-vivo swine heart to image the coronary microcirculation.

Delineating 3D blood vessels of various anatomical structures is essential for clinical diagnosis and treatment, however, is challenging due to complex structure variations and varied imaging conditions. Although recent supervised deep learning models have demonstrated their superior capacity in automatic 3D vessel segmentation, the reliance on expensive 3D manual annotations and limited capacity for annotation reuse among different vascular structures hinder their clinical applications. To avoid the repetitive and costly annotating process for each vascular structure and make full use of existing annotations, this paper proposes a novel 3D shape-guided local discrimination (3D-SLD) model for 3D vascular segmentation under limited guidance from public 2D vessel annotations. The primary hypothesis is that 3D vessels are composed of semantically similar voxels and often exhibit tree-shaped morphology. Accordingly, the 3D region discrimination loss is firstly proposed to learn the discriminative representation measuring voxel-wise similarities and cluster semantically consistent voxels to form the candidate 3D vascular segmentation in unlabeled images. Secondly, the shape distribution from existing 2D structure-agnostic vessel annotations is introduced to guide the 3D vessels with the tree-shaped morphology by the adversarial shape constraint loss. Thirdly, to enhance training stability and prediction credibility, the highlighting-reviewing-summarizing (HRS) mechanism is proposed. This mechanism involves summarizing historical models to maintain temporal consistency and identifying credible pseudo labels as reliable supervision signals. Only guided by public 2D coronary artery annotations, our method achieves results comparable to SOTA barely-supervised methods in 3D cerebrovascular segmentation, and the best DSC in 3D hepatic vessel segmentation, demonstrating the effectiveness of our method.

Multimodal neuroimaging provides complementary information critical for accurate early diagnosis of Alzheimer's disease (AD). However, the inherent variability between multimodal neuroimages hinders the effective fusion of multimodal features. Moreover, achieving reliable and interpretable diagnoses in the field of multimodal fusion remains challenging. To address them, we propose a novel multimodal diagnosis network based on multi-fusion and disease-induced learning (MDL-Net) to enhance early AD diagnosis by efficiently fusing multimodal data. Specifically, MDL-Net proposes a multi-fusion joint learning (MJL) module, which effectively fuses multimodal features and enhances the feature representation from global, local, and latent learning perspectives. MJL consists of three modules, global-aware learning (GAL), local-aware learning (LAL), and outer latent-space learning (LSL) modules. GAL via a self-adaptive Transformer (SAT) learns the global relationships among the modalities. LAL constructs local-aware convolution to learn the local associations. LSL module introduces latent information through outer product operation to further enhance feature representation. MDL-Net integrates the disease-induced region-aware learning (DRL) module via gradient weight to enhance interpretability, which iteratively learns weight matrices to identify AD-related brain regions. We conduct the extensive experiments on public datasets and the results confirm the superiority of our proposed method. Our code will be available at: https://github.com/qzf0320/MDL-Net.

In the field of medical imaging, the fusion of data from diverse modalities plays a pivotal role in advancing our understanding of pathological conditions. Sparse representation (SR), a robust signal modeling technique, has demonstrated noteworthy success in multi-dimensional (MD) medical image fusion. However, a fundamental limitation appearing in existing SR models is their lack of directionality, restricting their efficacy in extracting anatomical details from different imaging modalities. To tackle this issue, we propose a novel directional SR model, termed complex sparse representation (ComSR), specifically designed for medical image fusion. ComSR independently represents MD signals over directional dictionaries along specific directions, allowing precise analysis of intricate details of MD signals. Besides, current studies in medical image fusion mostly concentrate on addressing either 2D or 3D fusion problems. This work bridges this gap by proposing a MD medical image fusion method based on ComSR, presenting a unified framework for both 2D and 3D fusion tasks. Experimental results across six multi-modal medical image fusion tasks, involving 93 pairs of 2D source images and 20 pairs of 3D source images, substantiate the superiority of our proposed method over 11 state-of-the-art 2D fusion methods and 4 representative 3D fusion methods, in terms of both visual quality and objective evaluation.

Objective: Investigate the capacity of MRI to evaluate efficacy of radiofrequency (RF) ablations delivered to MRI-defined arrhythmogenic substrates., Methods: Baseline MRI was performed at 3 T including 3D LGE in a swine model of chronic myocardial infarct (N = 8). MRI-derived maps of scar and heterogeneous tissue channels (HTCs) were generated using ADAS 3D. Animals underwent electroanatomic mapping and ablation of the left ventricle in CARTO3, guided by MRI-derived scar maps. Post-ablation MRI (in vivo at 3 T in 5/8 animals; ex vivo at 1.5 T in 3/8) included 3D native T1-weighted IR-SPGR (TI = 700-800 ms) to visualize RF lesions. T1-derived RF lesions were compared against excised tissue. The locations of T1-derived RF lesions were compared against CARTO ablation tags, and segment-wise sensitivity and specificity of lesion detection were calculated within the AHA 17-segment model., Results: RF lesions were clearly visualized in HTCs, scar, and myocardium. Ablation patterns delivered in CARTO matched T1-derived RF lesion patterns with high sensitivity (88.9%) and specificity (94.7%), and were closely matched in registered MR-EP data sets, with a displacement of 5.4 ±3.8 mm (N = 152 ablation tags)., Conclusion: Integrating MRI into ablative procedures for RF lesion assessment is feasible. Patterns of RF lesions created using a standard 3D EAM system are accurately reflected by MRI visualization in healthy myocardium, scar, and HTCs comprising the MRI-defined arrhythmia substrate., Significance: MRI visualization of RF lesions can provide near-immediate ( 24 h) assessment of ablation, potentially indicating whether critical MRI-defined ventricular tachycardia substrates have been adequately ablated.

Global single-valued biomarkers, such as ejection fraction, are widely used in clinical practice to assess cardiac function. However, they only approximate the heart's true 3D deformation process, thus limiting diagnostic accuracy and the understanding of cardiac mechanics. Metrics based on 3D shape have been proposed to alleviate these shortcomings. In this work, we present the Point Cloud Deformation Network (PCD-Net) as a novel geometric deep learning approach for direct modeling of 3D cardiac mechanics of the biventricular anatomy between the extreme ends of the cardiac cycle. Its encoder-decoder architecture combines a low-dimensional latent space with recent advances in point cloud deep learning for effective multi-scale feature learning directly on flexible and memory-efficient point cloud representations of the cardiac anatomy. We first evaluate the PCD-Net's predictive capability for both cardiac contraction and relaxation on a large UK Biobank dataset of over 10,000 subjects and find average Chamfer distances between the predicted and ground truth anatomies below the pixel resolution of the underlying image acquisition. We then show the PCD-Net's ability to capture subpopulation-specific differences in 3D cardiac mechanics between normal and myocardial infarction (MI) subjects and visualize abnormal phenotypes between predicted normal 3D shapes and corresponding observed ones. Finally, we demonstrate that the PCD-Net's learned 3D deformation encodings outperform multiple clinical and machine learning benchmarks by 11% in terms of area under the receiver operating characteristic curve for the tasks of prevalent MI detection and incident MI prediction and by 7% in terms of Harrell's concordance index for MI survival analysis.

Coronary artery disease (CAD) is one of the leading causes of death globally. Currently, diagnosis and intervention in CAD are typically performed via minimally invasive cardiac catheterization procedures. Using current diagnostic technology, such as angiography and fractional flow reserve (FFR), interventional cardiologists must decide which patients require intervention and which can be deferred; 10% of patients with stable CAD are incorrectly deferred using current diagnostic best practices. By developing a forward-viewing intravascular ultrasound (FV-IVUS) 2-D array capable of simultaneously evaluating morphology, hemodynamics, and plaque composition, physicians would be better able to stratify risk of major adverse cardiac events in patients with intermediate stenosis. For this application, a forward-viewing, 16-MHz 2-D array transducer was designed and fabricated. A 2-mm-diameter aperture consisting of 140 elements, with element dimensions of 98×98×70 μ m ( w×h×t ) and a nominal interelement spacing of 120 μ m, was designed for this application based on simulations. The acoustic stack for this array was developed with a designed center frequency of 16 MHz. A novel via-less interconnect was developed to enable electrical connections to fan-out from a 140-element 2-D array with 120- μ m interelement spacing. The fabricated array transducer had 96/140 functioning elements operating at a center frequency of 16 MHz with a -6-dB fractional bandwidth of 62% ± 7 %. Single-element SNR was 23 ± 3 dB, and the measured electrical crosstalk was - 33 ± 3 dB. In imaging experiments, the measured lateral resolution was 0.231 mm and the measured axial resolution was 0.244 mm at a depth of 5 mm. Finally, the transducer was used to perform 3-D B-mode imaging of a 3-mm-diameter spring and 3-D B-mode and power Doppler imaging of a tissue-mimicking phantom.

The demand for cone-beam computed tomography (CBCT) imaging in clinics, particularly in dentistry, is rapidly increasing. Preoperative surgical planning is crucial to achieving desired treatment outcomes for imaging-guided surgical navigation. However, the lack of surface texture hinders effective communication between clinicians and patients, and the accuracy of superimposing a textured surface onto CBCT volume is limited by dissimilarity and registration based on facial features. To address these issues, this study presents a CBCT imaging system integrated with a monocular camera for reconstructing the texture surface by mapping it onto a 3D surface model created from CBCT images. The proposed method utilizes a geometric calibration tool for accurate mapping of the camera-visible surface with the mosaic texture. Additionally, a novel approach using 3D-2D feature mapping and surface parameterization technology is proposed for texture surface reconstruction. Experimental results, obtained from both real and simulation data, validate the effectiveness of the proposed approach with an error reduction to 0.32 mm and automated generation of integrated images. These findings demonstrate the robustness and high accuracy of our approach, improving the performance of texture mapping in CBCT imaging.

Objective: Low-cost, portable RGB-D cameras with integrated motion tracking functionality enable easy-to-use 3D motion analysis without requiring expensive facilities and specialized personnel. However, the accuracy of existing systems is insufficient for most clinical applications, particularly when applied to children. In previous work, we developed an RGB-D camera-based motion tracking method and showed that it accurately captures body joint positions of children and young adults in 3D. In this study, the validity and accuracy of clinically relevant motion parameters that were computed from kinematics of our motion tracking method are evaluated in children and young adults., Methods: Twenty-three typically developing children and healthy young adults (5-29 years, 110-189 cm) performed five movement tasks while being recorded simultaneously with a marker-based Vicon system and an Azure Kinect RGB-D camera. Motion parameters were computed from the extracted kinematics of both methods: time series measurements, i.e., measurements over time, peak measurements, i.e., measurements at a single time instant, and movement smoothness. The agreement of these parameter values was evaluated using Pearson's correlation coefficients r for time series data, and mean absolute error (MAE) and Bland-Altman plots with limits of agreement for peak measurements and smoothness., Results: Time series measurements showed strong to excellent correlations (r-values between 0.8 and 1.0), MAE for angles ranged from 1.5 to 5 degrees and for smoothness parameters (SPARC) from 0.02-0.09, while MAE for distance-related parameters ranged from 9 to 15 mm., Conclusion: Extracted motion parameters are valid and accurate for various movement tasks in children and young adults, demonstrating the suitability of our tracking method for clinical motion analysis., Clinical Impact: The low-cost portable hardware in combination with our tracking method enables motion analysis outside of specialized facilities while providing measurements that are close to those of the clinical gold-standard., (© 2024 The Authors.)

Artificial intelligence (AI) in healthcare, especially in medical imaging, faces challenges due to data scarcity and privacy concerns. Addressing these, we introduce Med-DDPM, a diffusion model designed for 3D semantic brain MRI synthesis. This model effectively tackles data scarcity and privacy issues by integrating semantic conditioning. This involves the channel-wise concatenation of a conditioning image to the model input, enabling control in image generation. Med-DDPM demonstrates superior stability and performance compared to existing 3D brain imaging synthesis methods. It generates diverse, anatomically coherent images with high visual fidelity. In terms of dice score in the tumor segmentation task, Med-DDPM achieves 0.6207, close to the 0.6531 dice score of real images, and outperforms baseline models. Combined with real images, it further increases segmentation accuracy to 0.6675, showing the potential of the proposed method for data augmentation. This model represents the first use of a diffusion model in 3D semantic brain MRI synthesis, producing high-quality images. Its semantic conditioning feature also shows potential for image anonymization in biomedical imaging, addressing data and privacy issues.

Power Doppler ultrasound (PD-US) is the ideal modality to assess tissue perfusion as it is cheap, patient-friendly and does not require ionizing radiation. However, meaningful inter-patient comparison only occurs if differences in tissue-attenuation are corrected for. This can be done by standardizing the PD-US signal to a blood vessel assumed to have 100% vascularity. The original method to do this is called fractional moving blood volume (FMBV). We describe a novel, fully-automated method combining image processing, numerical modelling, and deep learning to estimate three-dimensional single vessel fractional moving blood volume (3D-svFMBV). We map the PD signals to a characteristic intensity profile within a single large vessel to define the standardization value at the high shear vessel margins. This removes the need for mathematical correction for background signal which can introduce error. The 3D-svFMBV was first tested on synthetic images generated using the characteristics of uterine artery and physiological ultrasound noise levels, demonstrating prediction of standardization value close to the theoretical ideal. Clinical utility was explored using 143 first-trimester placental ultrasound volumes. More biologically plausible perfusion estimates were obtained, showing improved prediction of pre-eclampsia compared with those generated with the semi-automated original 3D-FMBV technique. The proposed 3D-svFMBV method overcomes the limitations of the original technique to provide accurate and robust placental perfusion estimation. This not only has the potential to provide an early pregnancy screening tool but may also be used to assess perfusion of different organs and tumors.

In 3D freehand ultrasound imaging, operator dependent variations in applied forces and movements can lead to errors in the reconstructed images. In this paper, we introduce an automated 3D ultrasound system, which enables acquisitions with controlled movement trajectories by using motors, which electrically move the probe. Due to integrated encoders there is no need of position sensors. An included force control mechanism ensures a constant contact force to the skin. We conducted 8 trials with the automated 3D ultrasound system on 2 different phantoms with 3 force settings and 10 trials on a human tibialis anterior muscle with 2 force settings. For comparison, we also conducted 8 freehand 3D ultrasound scans from 2 operators (4 force settings) on one phantom and 10 with one operator on the tibialis anterior muscle. Both freehand and automated trials showed small errors in volume and length computations of the reconstructions, however the freehand trials showed larger standard deviations. We also computed the thickness of the phantom and the tibialis anterior muscle. We found significant differences in force settings for the operators and higher coefficients of variation for the freehand trials. Overall, the automated 3D ultrasound system shows a high accuracy in reconstruction. Due to the smaller coefficients of variation, the automated 3D ultrasound system enables more reproducible ultrasound examinations than the freehand scanning. Therefore, the automated 3D ultrasound system is a reliable tool for 3D investigations of skeletal muscle.

Histological analysis is the core of follicular thyroid carcinoma (FTC) classification. The histopathological criteria of capsular and vascular invasion define malignancy and aggressiveness of FTC. Analysis of multiple sections is cumbersome and as only a minute tissue fraction is analyzed during histopathology, under-sampling remains a problem. Application of an efficient tool for complete tissue imaging in 3D would speed-up diagnosis and increase accuracy. We show that X-ray propagation-based imaging (XPBI) of paraffin-embedded tissue blocks is a valuable complementary method for follicular thyroid carcinoma diagnosis and assessment. It enables a fast, non-destructive and accurate 3D virtual histology of the FTC resection specimen. We demonstrate that XPBI virtual slices can reliably evaluate capsular invasions. Then we discuss the accessible morphological information from XPBI and their significance for vascular invasion diagnosis. We show 3D morphological information that allow to discern vascular invasions. The results are validated by comparing XPBI images with clinically accepted histology slides revised by and under supervision of two experienced endocrine pathologists.

Fairness (also known as equity interchangeably) in machine learning is important for societal well-being, but limited public datasets hinder its progress. Currently, no dedicated public medical datasets with imaging data for fairness learning are available, though underrepresented groups suffer from more health issues. To address this gap, we introduce Harvard Glaucoma Fairness (Harvard-GF), a retinal nerve disease dataset including 3,300 subjects with both 2D and 3D imaging data and balanced racial groups for glaucoma detection. Glaucoma is the leading cause of irreversible blindness globally with Blacks having doubled glaucoma prevalence than other races. We also propose a fair identity normalization (FIN) approach to equalize the feature importance between different identity groups. Our FIN approach is compared with various state-of-the-art fairness learning methods with superior performance in the racial, gender, and ethnicity fairness tasks with 2D and 3D imaging data, demonstrating the utilities of our dataset Harvard-GF for fairness learning. To facilitate fairness comparisons between different models, we propose an equity-scaled performance measure, which can be flexibly used to compare all kinds of performance metrics in the context of fairness. The dataset and code are publicly accessible via https://ophai.hms.harvard.edu/datasets/harvard-gf3300/.

Accurately delineating and categorizing individual hand bones in 3D ultrasound (US) is a promising technology for precise digital diagnostic analysis. However, this is a challenging task due to the inherent imaging limitations of the US and the insignificant feature differences among numerous bones. In this study, we have proposed a novel deep learning-based solution for pediatric hand bone segmentation in the US. Our method is unique in that it allows for effective detailed feature mining through an adaptive multi-dimensional weighting attention mechanism. It innovatively implements a category-aware contrastive learning method to highlight inter-class semantic feature differences, thereby enhancing the category discrimination performance of the model. Extensive experiments on the challenging pediatric clinical hand 3D US datasets show the outstanding performance of the proposed method in segmenting thirty-eight bone structures, with the average Dice coefficient of 90.0%. The results outperform other state-of-the-art methods, demonstrating its effectiveness in fine-grained hand bone segmentation. Our method will be globally released as a plugin in the 3D Slicer, providing an innovative and reliable tool for relevant clinical applications.

The accurate segmentation of brain tumor is significant in clinical practice. Convolutional Neural Network (CNN)-based methods have made great progress in brain tumor segmentation due to powerful local modeling ability. However, brain tumors are frequently pattern-agnostic, i.e. variable in shape, size and location, which can not be effectively matched by traditional CNN-based methods with local and regular receptive fields. To address the above issues, we propose a shape-scale co-awareness network (S2CA-Net) for brain tumor segmentation, which can efficiently learn shape-aware and scale-aware features simultaneously to enhance pattern-agnostic representations. Primarily, three key components are proposed to accomplish the co-awareness of shape and scale. The Local-Global Scale Mixer (LGSM) decouples the extraction of local and global context by adopting the CNN-Former parallel structure, which contributes to obtaining finer hierarchical features. The Multi-level Context Aggregator (MCA) enriches the scale diversity of input patches by modeling global features across multiple receptive fields. The Multi-Scale Attentive Deformable Convolution (MS-ADC) learns the target deformation based on the multiscale inputs, which motivates the network to enforce feature constraints both in terms of scale and shape for optimal feature matching. Overall, LGSM and MCA focus on enhancing the scale-awareness of the network to cope with the size and location variations, while MS-ADC focuses on capturing deformation information for optimal shape matching. Finally, their effective integration prompts the network to perceive variations in shape and scale simultaneously, which can robustly tackle the variations in patterns of brain tumors. The experimental results on BraTS 2019, BraTS 2020, MSD BTS Task and BraTS2023-MEN show that S2CA-Net has superior overall performance in accuracy and efficiency compared to other state-of-the-art methods. Code: https://github.com/jiangyu945/S2CA-Net.

Vascular structure segmentation plays a crucial role in medical analysis and clinical applications. The practical adoption of fully supervised segmentation models is impeded by the intricacy and time-consuming nature of annotating vessels in the 3D space. This has spurred the exploration of weakly-supervised approaches that reduce reliance on expensive segmentation annotations. Despite this, existing weakly supervised methods employed in organ segmentation, which encompass points, bounding boxes, or graffiti, have exhibited suboptimal performance when handling sparse vascular structure. To alleviate this issue, we employ maximum intensity projection (MIP) to decrease the dimensionality of 3D volume to 2D image for efficient annotation, and the 2D labels are utilized to provide guidance and oversight for training 3D vessel segmentation model. Initially, we generate pseudo-labels for 3D blood vessels using the annotations of 2D projections. Subsequently, taking into account the acquisition method of the 2D labels, we introduce a weakly-supervised network that fuses 2D-3D deep features via MIP to further improve segmentation performance. Furthermore, we integrate confidence learning and uncertainty estimation to refine the generated pseudo-labels, followed by fine-tuning the segmentation network. Our method is validated on five datasets (including cerebral vessel, aorta and coronary artery), demonstrating highly competitive performance in segmenting vessels and the potential to significantly reduce the time and effort required for vessel annotation. Our code is available at: https://github.com/gzq17/Weakly-Supervised-by-MIP.

Germectomy is a common surgery in pediatric dentistry to prevent the potential dangers caused by impacted mandibular wisdom teeth. Segmentation of mandibular wisdom teeth is a crucial step in surgery planning. However, manually segmenting teeth and bones from 3D volumes is time-consuming and may cause delays in treatment. Deep learning based medical image segmentation methods have demonstrated the potential to reduce the burden of manual annotations, but they still require a lot of well-annotated data for training. In this paper, we initially curated a Cone Beam Computed Tomography (CBCT) dataset, NKUT, for the segmentation of pediatric mandibular wisdom teeth. This marks the first publicly available dataset in this domain. Second, we propose a semantic separation scale-specific feature fusion network named WTNet, which introduces two branches to address the teeth and bones segmentation tasks. In WTNet, We design a Input Enhancement (IE) block and a Teeth-Bones Feature Separation (TBFS) block to solve the feature confusions and semantic-blur problems in our task. Experimental results suggest that WTNet performs better on NKUT compared to previous state-of-the-art segmentation methods (such as TransUnet), with a maximum DSC lead of nearly 16%.

The widely used Vermon 1024-element matrix array for 3-D ultrasound imaging has three blank rows in the elevational direction, which breaks the elevation periodicity, thus degrading volumetric image quality. To bypass the blank rows in elevation while maintaining the steering capability in azimuth, we proposed a row-transmission (RT) scheme to improve 3-D spatial resolution. Specifically, we divided the full array into four apertures, each with multiple rows along the elevation. Each multirow aperture (MRA) was further divided into subapertures to transmit diverging waves (DWs) sequentially. Coherent DW compounding (CDWC) was realized in azimuth, while the elevation was multielement synthetic aperture (M-SA) imaging by regarding each row as an array of dashed line elements. An in-house spatiotemporal coding strategy, cascaded synthetic aperture (CaSA), was incorporated into the RT scheme as RT-CaSA to increase the signal-to-noise ratio (SNR). We compared the proposed RT with conventional bank-by-bank transmission-reception (Bank) and sparse-random-aperture compounding (SRAC) in a wire phantom and the in vivo human abdominal aorta (AA) to assess the performance of anatomical imaging and aortic wall motion estimation. Phantom results demonstrated superior lateral resolution achieved by our RT scheme (+19.52% and +16.88% versus Bank, +15.32% and +19.72% versus SRAC, in the azimuth-depth and elevation-depth planes, respectively). Our RT-CaSA showed excellent contrast ratios (CRs) (+8.19 and +8.08 dB versus Bank, +6.81 and +5.85 dB versus SRAC, +0.99 and +0.90 dB versus RT) and the highest in vivo aortic wall motion estimation accuracy. The RT scheme was demonstrated to have potential for various matrix array-based 3-D imaging research.

Accurate and automatic delineation of the left atrium (LA) is crucial for computer-aided diagnosis of atrial fibrillation-related diseases. However, effective model training typically requires a large amount of labeled data, which is time-consuming and labor-intensive. In this study, we propose a novel LA delineation framework. The region of LA is first detected using an actor-critic based deep reinforcement learning method with a shape-adaptive detection strategy using only box-level annotations, bypassing the need for voxel-level labeling. With the effectively detected LA, the impacts of class-imbalance and interference from surrounding tissues are significantly reduced. Subsequently, a semi-supervised segmentation scheme is coined to precisely delineate the contour of LA in 3D volume. The scheme integrates two independent networks with distinct structures, enabling implicit consistency regularization, capturing more spatial features, and avoiding the error accumulation present in current mainstream semi-supervised frameworks. Specifically, one network is combined with Transformer to capture latent spatial features, while the other network is based on pure CNN to capture local features. The difference prediction between these two sub-networks is exploited to mutually provide high-quality pseudo-labels and correct the cognitive bias. Experimental results on two public datasets demonstrate that our proposed strategy outperforms several state-of-the-art methods in terms of accuracy and clinical convenience.

Semi-supervised segmentation is highly significant in 3D medical image segmentation. The typical solutions adopt a teacher-student dual-model architecture, and they constrain the two models' decision consistency on the same segmentation task. However, the scarcity of medical samples can lower the diversity of tasks, reducing the effectiveness of consistency constraint. The issue can further worsen as the weights of the models gradually become synchronized. In this work, we have proposed to construct diverse joint-tasks using masked image modelling for enhancing the reliability of the consistency constraint, and develop a novel architecture consisting of a single teacher but multiple students to enjoy the additional knowledge decoupled from the synchronized weights. Specifically, the teacher and student models 'see' varied randomly-masked versions of an input, and are trained to segment the same targets but reconstruct different missing regions concurrently. Such joint-task of segmentation and reconstruction can have the two learners capture related but complementary features to derive instructive knowledge when constraining their consistency. Moreover, two extra students join the original one to perform an inter-student learning. The three students share the same encoding but different decoding designs, and learn decoupled knowledge by constraining their mutual consistencies, preventing themselves from suboptimally converging to the biased predictions of the dictatorial teacher. Experimental on four medical datasets show that our approach performs better than six mainstream semi-supervised methods. Particularly, our approach achieves at least 0.61% and 0.36% higher Dice and Jaccard values, respectively, than the most competitive approach on our in-house dataset. The code will be released at https://github.com/zxmboshi/DDL.

Optical coherence tomography (OCT) can perform non-invasive high-resolution three-dimensional (3D) imaging and has been widely used in biomedical fields, while it is inevitably affected by coherence speckle noise which degrades OCT imaging performance and restricts its applications. Here we present a novel speckle-free OCT imaging strategy, named toward-ground-truth OCT ( t GT-OCT), that utilizes unsupervised 3D deep-learning processing and leverages OCT 3D imaging features to achieve speckle-free OCT imaging. Specifically, our proposed t GT-OCT utilizes an unsupervised 3D-convolution deep-learning network trained using random 3D volumetric data to distinguish and separate speckle from real structures in 3D imaging volumetric space; moreover, t GT-OCT effectively further reduces speckle noise and reveals structures that would otherwise be obscured by speckle noise while preserving spatial resolution. Results derived from different samples demonstrated the high-quality speckle-free 3D imaging performance of t GT-OCT and its advancement beyond the previous state-of-the-art. The code is available online: https://github.com/Voluntino/tGT-OCT.

Position emission tomography (PET) is widely used in clinics and research due to its quantitative merits and high sensitivity, but suffers from low signal-to-noise ratio (SNR). Recently convolutional neural networks (CNNs) have been widely used to improve PET image quality. Though successful and efficient in local feature extraction, CNN cannot capture long-range dependencies well due to its limited receptive field. Global multi-head self-attention (MSA) is a popular approach to capture long-range information. However, the calculation of global MSA for 3D images has high computational costs. In this work, we proposed an efficient spatial and channel-wise encoder-decoder transformer, Spach Transformer, that can leverage spatial and channel information based on local and global MSAs. Experiments based on datasets of different PET tracers, i.e., 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE, were conducted to evaluate the proposed framework. Quantitative results show that the proposed Spach Transformer framework outperforms state-of-the-art deep learning architectures.

This paper studies 3D low-dose computed tomography (CT) imaging. Although various deep learning methods were developed in this context, typically they focus on 2D images and perform denoising due to low-dose and deblurring for super-resolution separately. Up to date, little work was done for simultaneous in-plane denoising and through-plane deblurring, which is important to obtain high-quality 3D CT images with lower radiation and faster imaging speed. For this task, a straightforward method is to directly train an end-to-end 3D network. However, it demands much more training data and expensive computational costs. Here, we propose to link in-plane and through-plane transformers for simultaneous in-plane denoising and through-plane deblurring, termed as LIT-Former, which can efficiently synergize in-plane and through-plane sub-tasks for 3D CT imaging and enjoy the advantages of both convolution and transformer networks. LIT-Former has two novel designs: efficient multi-head self-attention modules (eMSM) and efficient convolutional feed-forward networks (eCFN). First, eMSM integrates in-plane 2D self-attention and through-plane 1D self-attention to efficiently capture global interactions of 3D self-attention, the core unit of transformer networks. Second, eCFN integrates 2D convolution and 1D convolution to extract local information of 3D convolution in the same fashion. As a result, the proposed LIT-Former synergizes these two sub-tasks, significantly reducing the computational complexity as compared to 3D counterparts and enabling rapid convergence. Extensive experimental results on simulated and clinical datasets demonstrate superior performance over state-of-the-art models. The source code is made available at https://github.com/hao1635/LIT-Former.

In recent years, an increasing number of medical engineering tasks, such as surgical navigation, pre-operative registration, and surgical robotics, rely on 3D reconstruction techniques. Self-supervised depth estimation has attracted interest in endoscopic scenarios because it does not require ground truth. Most existing methods depend on expanding the size of parameters to improve their performance. There, designing a lightweight self-supervised model that can obtain competitive results is a hot topic. We propose a lightweight network with a tight coupling of convolutional neural network (CNN) and Transformer for depth estimation. Unlike other methods that use CNN and Transformer to extract features separately and then fuse them on the deepest layer, we utilize the modules of CNN and Transformer to extract features at different scales in the encoder. This hierarchical structure leverages the advantages of CNN in texture perception and Transformer in shape extraction. In the same scale of feature extraction, the CNN is used to acquire local features while the Transformer encodes global information. Finally, we add multi-head attention modules to the pose network to improve the accuracy of predicted poses. Experiments demonstrate that our approach obtains comparable results while effectively compressing the model parameters on two datasets.

Three-dimensional images are frequently used in medical imaging research for classification, segmentation, and detection. However, the limited availability of 3D images hinders research progress due to network training difficulties. Generative methods have been proposed to create medical images using AI techniques. Nevertheless, 2D approaches have difficulty dealing with 3D anatomical structures, which can result in discontinuities between slices. To mitigate these discontinuities, several 3D generative networks have been proposed. However, the scarcity of available 3D images makes training these networks with limited samples inadequate for producing high-fidelity 3D images. We propose a data-guided generative adversarial network to provide high fidelity in 3D image generation. The generator creates fake images with noise using reference code obtained by extracting features from real images. The generator also creates decoded images using reference code without noise. These decoded images are compared to the real images to evaluate fidelity in the reference code. This generation process can create high-fidelity 3D images from only a small amount of real training data. Additionally, our method employs three types of discriminator: volume (evaluates all the slices), slab (evaluates a set of consecutive slices), and slice (evaluates randomly selected slices). The proposed discriminator enhances fidelity by differentiating between real and fake images based on detailed characteristics. Results from our method are compared with existing methods by using quantitative analysis such as Fréchet inception distance and maximum mean discrepancy. The results demonstrate that our method produces more realistic 3D images than existing methods.

The augmented intra-operative real-time imaging in vascular interventional surgery, which is generally performed by projecting preoperative computed tomography angiography images onto intraoperative digital subtraction angiography (DSA) images, can compensate for the deficiencies of DSA-based navigation, such as lack of depth information and excessive use of toxic contrast agents. 3D/2D vessel registration is the critical step in image augmentation. A 3D/2D registration method based on vessel graph matching is proposed in this study. For rigid registration, the matching of vessel graphs can be decomposed into continuous states, thus 3D/2D vascular registration is formulated as a search tree problem. The Monte Carlo tree search method is applied to find the optimal vessel matching associated with the highest rigid registration score. For nonrigid registration, we propose a novel vessel deformation model based on manifold regularization. This model incorporates the smoothness constraint of vessel topology into the objective function. Furthermore, we derive simplified gradient formulas that enable fast registration. The proposed technique undergoes evaluation against seven rigid and three nonrigid methods using a variety of data - simulated, algorithmically generated, and manually annotated - across three vascular anatomies: the hepatic artery, coronary artery, and aorta. Our findings show the proposed method's resistance to pose variations, noise, and deformations, outperforming existing methods in terms of registration accuracy and computational efficiency. The proposed method demonstrates average registration errors of 2.14 mm and 0.34 mm for rigid and nonrigid registration, and an average computation time of 0.51 s.

In recent years, 4D medical image involving structural and motion information of tissue has attracted increasing attention. The key to the 4D image reconstruction is to stack the 2D slices based on matching the aligned motion states. In this study, the distribution of the 2D slices with the different motion states is modeled as a manifold graph, and the reconstruction is turned to be the graph alignment. An embedding-alignment fusion-based graph convolution network (GCN) with a mixed-learning strategy is proposed to align the graphs. Herein, the embedding and alignment processes of graphs interact with each other to realize a precise alignment with retaining the manifold distribution. The mixed strategy of self- and semi-supervised learning makes the alignment sparse to avoid the mismatching caused by outliers in the graph. In the experiment, the proposed 4D reconstruction approach is validated on the different modalities including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Ultrasound (US). We evaluate the reconstruction accuracy and compare it with those of state-of-the-art methods. The experiment results demonstrate that our approach can reconstruct a more accurate 4D image.

Biomedical image segmentation of organs, tissues and lesions has gained increasing attention in clinical treatment planning and navigation, which involves the exploration of two-dimensional (2D) and three-dimensional (3D) contexts in the biomedical image. Compared to 2D methods, 3D methods pay more attention to inter-slice correlations, which offer additional spatial information for image segmentation. An organ or tumor has a 3D structure that can be observed from three directions. Previous studies focus only on the vertical axis, limiting the understanding of the relationship between a tumor and its surrounding tissues. Important information can also be obtained from sagittal and coronal axes. Therefore, spatial information of organs and tumors can be obtained from three directions, i.e. the sagittal, coronal and vertical axes, to understand better the invasion depth of tumor and its relationship with the surrounding tissues. Moreover, the edges of organs and tumors in biomedical image may be blurred. To address these problems, we propose a three-direction fusion volumetric segmentation (TFVS) model for segmenting 3D biomedical images from three perspectives in sagittal, coronal and transverse planes, respectively. We use the dataset of the liver task provided by the Medical Segmentation Decathlon challenge to train our model. The TFVS method demonstrates a competitive performance on the 3D-IRCADB dataset. In addition, the t-test and Wilcoxon signed-rank test are also performed to show the statistical significance of the improvement by the proposed method as compared with the baseline methods. The proposed method is expected to be beneficial in guiding and facilitating clinical diagnosis and treatment.

Three-dimensional (3D) ultrasound imaging technique has been applied for scoliosis assessment, but the current assessment method only uses coronal projection images and cannot illustrate the 3D deformity and vertebra rotation. The vertebra detection is essential to reveal 3D spine information, but the detection task is challenging due to complex data and limited annotations. We propose VertMatch to detect vertebral structures in 3D ultrasound volume containing a detector and classifier. The detector network finds the potential positions of structures on transverse slice globally, and then the local patches are cropped based on detected positions. The classifier is used to distinguish whether the patches contain real vertebral structures and screen the predicted positions from the detector. VertMatch utilizes unlabeled data in a semi-supervised manner, and we develop two novel techniques for semi-supervised learning: 1) anatomical prior is used to acquire high-quality pseudo labels; 2) inter-slice consistency is used to utilize more unlabeled data by inputting multiple adjacent slices. Experimental results demonstrate that VertMatch can detect vertebra accurately in ultrasound volume and outperforms state-of-the-art methods. Moreover, VertMatch is also validated in automatic spinous process angle measurement on forty subjects with scoliosis, and the results illustrate that it can be a promising approach for the 3D assessment of scoliosis.

Accurate vascular segmentation from High Resolution 3-Dimensional (HR3D) medical scans is crucial for clinicians to visualize complex vasculature and diagnose related vascular diseases. However, a reliable and scalable vessel segmentation framework for HR3D scans remains a challenge. In this work, we propose a High-resolution Energy-matching Segmentation (HrEmS) framework that utilizes deep learning to directly process the entire HR3D scan and segment the vasculature to the finest level. The HrEmS framework introduces two novel components. Firstly, it uses the real-order total variation operator to construct a new loss function that guides the segmentation network to obtain the correct topology structure by matching the energy of the predicted segment to the energy of the manual label. This is different from traditional loss functions such as dice loss, which matches the pixels between predicted segment and manual label. Secondly, a curvature-based weight-correction module is developed, which directs the network to focus on crucial and complex structural parts of the vasculature instead of the easy parts. The proposed HrEmS framework was tested on three in-house multi-center datasets and three public datasets, and demonstrated improved results in comparison with the state-of-the-art methods using both topology-relevant and volumetric-relevant metrics. Furthermore, a double-blind assessment by three experienced radiologists on the critical points of the clinical diagnostic processes provided additional evidence of the superiority of the HrEmS framework.

Microwave imaging is a promising method for early diagnosing and monitoring brain strokes. It is portable, non-invasive, and safe to the human body. Conventional techniques solve for unknown electrical properties represented as pixels or voxels, but often result in inadequate structural information and high computational costs. We propose to reconstruct the three dimensional (3D) electrical properties of the human brain in a feature space, where the unknowns are latent codes of a variational autoencoder (VAE). The decoder of the VAE, with prior knowledge of the brain, acts as a module of data inversion. The codes in the feature space are optimized by minimizing the misfit between measured and simulated data. A dataset of 3D heads characterized by permittivity and conductivity is constructed to train the VAE. Numerical examples show that our method increases structural similarity by 14% and speeds up the solution process by over 3 orders of magnitude using only 4.8% number of the unknowns compared to the voxel-based method. This high-resolution imaging of electrical properties leads to more accurate stroke diagnosis and offers new insights into the study of the human brain.