Enhancing Histological Image Analysis and Automation through Unsupervised Attention-Guided Deep Learning for Robust Stain Normalization in H&E-Stained Images.

Histological images play a critical role in both clinical pathology and preclinical research. For instance, one way to study normal tissue radiation response is to assess the intestinal crypts in histological images. However, discrepancies in staining protocols can introduce visual inconsistencies among images, which not only challenge human interpretation but also impede the effectiveness and reliability of automated analytical systems, including deep learning (DL) models. Here, we explored an unsupervised attention-guided DL method for stain normalization in H&E-stained histological images to enhance subsequent analysis and automation process. Our dataset consists of 350 H&E-stained mouse intestinal cross-section images from a prior study on abdominal normal tissue response to radiation. Experts categorized these images into two sets: (1) 200 target images (187/13 for train/test) representing the desired target stain style, and (2) 150 source images (136/14 for train/test) with suboptimal stain styles to normalize. Within each group, stain variations are also present. CycleGAN serves as the backbone model. Additionally, to obtain a realistic stain translation preserving all content details, attention network was implemented in each generator to highlight the discriminative regions between the source and target domains, thereby focusing the generator's efforts on these relevant areas. The network was trained without the need for paired images or additional supervision, aiming to translate source images into the target stain style accurately. Furthermore, to assess the effectiveness of our stain normalization technique, we compared the performance of an in-house DL model for auto-detection of intestinal crypts on both original and normalized images. This model, specifically trained on images with the target stain style, allowed us to quantitatively evaluate the improvement in downstream histological analysis post-normalization. Qualitatively, our approach generates images that not only match the desired stain characteristics but also retain all structural details. Quantitatively, we achieved a structural similarity index (SSIM) of 0.89 between source and normalized images. Notably, applying our stain normalization method markedly enhanced the performance of the subsequent crypt auto-detection. The sensitivity of detecting intestinal crypts increased dramatically from 0 to 0.72 when utilizing the normalized images compared to the original ones with stain variations. Our approach demonstrates a robust capability to normalize stain color distributions across histological images without compromising their structural content. This method is adaptable to a wide range of histological image types, holding significant potential to enhance the accuracy, reliability, and resource utilization of subsequent image analysis processes, particularly in deep learning-based downstream tasks. [ABSTRACT FROM AUTHOR]