Search

Medical multi-modal retrieval aims to provide doctors with similar medical images from different modalities, which can greatly promote the efficiency and accuracy of clinical diagnosis. However, most existing medical retrieval methods hardly support the retrieval of multi-modal medical images, i.e., the number of modalities is greater than 2, and just convert retrieval to classification or clustering. It futilely breaks the gap between the visual information and the semantic information in different medical image modalities. To solve the problem, a Supervised Contrast Learning method based on a Multiple Pseudo-Siamese network (SCL-MPS) is proposed for multi-modal medical image retrieval. In order to make the samples with semantic similarity close neighbors on Riemann manifold, the multiple constraints based on semantic consistency and modal invariance are designed in different forward stages of SCL-MPS. We theoretically demonstrate the feasibility of the designed constraints. Finally, experiments on four benchmark datasets (ADNI1, ADNI2, ADNI3, and OASIS3) show that SCL-MPS achieves state-of-the-art performance compared to 15 retrieval methods. Especially, SCL-MPS achieves a 100% mAP score in medical cross-modal retrieval on ADNI1. [ABSTRACT FROM AUTHOR]

Image-to-image translation (I2I) has broad application prospects for assisting physicians in diagnosis of medical image missing scenarios. Considering that there is no medical I2I model constructed from a geometric view of simultaneously preserving local manifold-value and global manifold structure, we propose an I2I model based on manifold-value correction and manifold matching (MMNet) to translate one modal image to another in a paired and unpaired fashion and preserve the texture details of the target model image. For local manifold-value preservation, each manifold-value of the generated image is aligned with the corresponding real image as much as possible by jointly optimizing the distribution corrector and the distribution generator. For global manifold structure preservation, three distance metrics are defined to globally reduce the difference between the manifold of the generated images and the manifold of the real images through optimizing the manifold matching loss. Experimental results demonstrate that the proposed MMNet outperforms multiple state-of-the-art GANs-based methods for MR image translation in both qualitative and quantitative measures. [ABSTRACT FROM AUTHOR]

Because of the limitations of convolution kernel, the traditional image segmentation network is not sufficient to obtain the context information, but the image segmentation task is very dependent on the context information. Transformer's linear input can just get enough context information. In this paper, we propose a transformer segmentation network hyperfusion transformer based on a pyramid structure. First, the model divides the single-scale coding form into several-different-scale coding forms, and then fuses the decoding results. Second, in order to ensure the specificity of the output characteristics of each branch, we orthogonalize the results of a variety of different scales. By orthogonalizing in pairs, we can ensure that the results obtained by different branches are not completely similar to a certain extent, and reduce the redundancy of branch information. On the two datasets, the method in this paper surpasses a variety of classical models under multiple evaluation indexes, confirming that it is an effective segmentation method. [ABSTRACT FROM AUTHOR]

Cross-modal hashing is an efficient method to embed high-dimensional heterogeneous modal feature descriptors into a consistency-preserving Hamming space with low-dimensional. Most existing cross-modal hashing methods have been able to bridge the heterogeneous modality gap, but there are still two challenges resulting in limited retrieval accuracy: (1) ignoring the continuous similarity of samples on manifold; (2) lack of discriminability of hash codes with the same semantics. To cope with these problems, we propose a Deep Consistency-Preserving Hash Auto-encoders model, called DCPHA, based on the multi-manifold property of the feature distribution. Specifically, DCPHA consists of a pair of asymmetric auto-encoders and two semantics-preserving attention branches working in the encoding and decoding stages, respectively. When the number of input medical image modalities is greater than 2, the encoder is a multiple pseudo-Siamese network designed to extract specific modality features of different medical image modalities. In addition, we define the continuous similarity of heterogeneous and homogeneous samples on Riemann manifold from the perspective of multiple sub-manifolds, respectively, and the two constraints, i.e., multi-semantic consistency and multi-manifold similarity-preserving, are embedded in the learning of hash codes to obtain high-quality hash codes with consistency-preserving. The extensive experiments show that the proposed DCPHA has the most stable and state-of-the-art performance. We make code and models publicly available: https://github.com/Socrates023/DCPHA. [ABSTRACT FROM AUTHOR]

Benefitting from the low storage cost and high retrieval efficiency, hash learning has become a widely used retrieval technology to approximate nearest neighbors. Within it, the cross-modal medical hashing has attracted an increasing attention in facilitating efficiently clinical decision. However, there are still two main challenges in weak multi-manifold structure perseveration across multiple modalities and weak discriminability of hash code. Specifically, existing cross-modal hashing methods focus on pairwise relations within two modalities, and ignore underlying multi-manifold structures across over 2 modalities. Then, there is little consideration about discriminability, i.e., any pair of hash codes should be different. In this paper, we propose a novel hashing method named multi-manifold deep discriminative cross-modal hashing (MDDCH) for large-scale medical image retrieval. The key point is multi-modal manifold similarity which integrates multiple sub-manifolds defined on heterogeneous data to preserve correlation among instances, and it can be measured by three-step connection on corresponding hetero-manifold. Then, we propose discriminative item to make each hash code encoded by hash functions be different, which improves discriminative performance of hash code. Besides, we introduce Gaussian-binary Restricted Boltzmann Machine to directly output hash codes without using any continuous relaxation. Experiments on three benchmark datasets (AIBL, Brain and SPLP) show that our proposed MDDCH achieves comparative performance to recent state-of-the-art hashing methods. Additionally, diagnostic evaluation from professional physicians shows that all the retrieved medical images describe the same object and illness as the queried image. [ABSTRACT FROM AUTHOR]

Medical image generation from diagnostic report has important research significance for medical‐aided diagnosis. This research can improve the diagnosis speed and accuracy of doctors and effectively save the storage resources of hospitals. The research has made significant progress in the field of natural images, but rarely used in medical images. Medical images have higher requirements for image quality. In this paper, a method based on attention mechanism and content preservation loss to improve image quality is proposed. The model consists of three stages, and each stage generates feature map of different scale combined with attention features as the input of the next stage to optimise semantic consistency between image and text. The last stage will generate an image, whose size is 256 × 256. And the content preservation loss can optimise the similarity between the generated image and the real image by low‐level and high‐level features to meet the high requirements of medical image for texture details. The content preservation loss consists of MSE loss, VGG loss and TV loss. The experiments on two datasets prove that the method can achieve excellent results. [ABSTRACT FROM AUTHOR]

In different medical imaging techniques, the relative displacement between the patient and detection instrument can lead to different degrees of motion blur, which will affect the diagnosis of the patient's condition. In this paper, we propose a deblurring model with Multiple Grained Channel Normalized Fusion Networks (MG-CNFNet), which decomposes the image input by multi-grained division to cascade feature extraction from fine to coarse, constructs the encoder using semi-channel self-normalization to enrich the multi-scale information of the image, highlight the main features and improve the deblurring performance, and uses a shallow convolutional fusion block to obtain more shallow information and balance the performance and efficiency of deblurring. Compared with other state-of-the-art methods, our method is qualitatively and quantitatively evaluated on three medical image datasets (including X-ray, Ultrasound, and MRI) to demonstrate its effectiveness in removing motion blur from medical images, and yields better restoration results. • Propose a multiple grained channel normalized fusion networks model (MG-CNFNet), which is able to maintain both good spatial texture and rich semantic information. • The input image is divided into multiple grains to perform different tasks for extracting features from fine to coarse. • A semi-channel self-normalization block (Scn) is constructed as basic block of the encoder to enhance the multi-scale information of the image and highlight the main features, Using instance normalization as a feature enhancement method. • Add a shallow convolutional fusion block (SCF) based on dilated convolution and standard convolution to obtain more shallow information, and the ablation experimental results confirm SCF makes the model has a large improvement in the deblurring performance. • The deblurring effect of MG-CNFNet is very superior to the state-of-the-arts on different kinds of medical images, and subsequent experiments verify the effectiveness of our model. [ABSTRACT FROM AUTHOR]

• We propose a model which can detect the lesion areas, diagnose pathological information and output diagnostic reports for medical images simultaneously. • Our model can learn to fuse pathological information into the generation procedure of diagnostic reports, so that our model can generate reports with higher metric scores and more accurate pathological information. • To the best of our knowledge, this is the first approach attempt to improve accuracy of pathological information in generated diagnostic reports, and we have achieved a competitive result. Background and objectives: Writing diagnostic reports for medical images is a heavy and tedious work. The automatic generation of medical image diagnostic reports can assist doctors to reduce their workload and improve diagnosis efficiency. It is of great significance to introduce image caption algorithm into medical image processing. Existing approaches attempt to generate medical image diagnostic reports using image caption algorithms but without taking the accuracy of pathological information in generated diagnostic reports into account. Methods: To solve the mentioned problem, we propose a Semantic Fusion Network (SFNet) including a lesion area detection model and a diagnostic generation model. The lesion area detection model can extract visual and pathological information from medical image, and the diagnostic report generation model can learn to fuse the two kinds of information to generate reports. Thus, the pathological information in the generated diagnostic reports can be more accurate. Results: Experimental results have verified the performance of our model (Accuracy increases 1.2% on the Ultrasound Image Dataset and 2.4% on the Open-i X-ray Image Dataset), compared with the model only using visual feature to generate diagnostic reports. Conclusions: This work utilizes computer algorithms to generate the more accurate diagnostic reports for medical images automatically, which expands the application of computer-aided diagnosis and promotes the implementation of deep learning in the medical image analysis field. [ABSTRACT FROM AUTHOR]