Search

In our dynamic world where data arrives in a continuous stream, continual learning enables us to incrementally add new tasks/domains without the need to retrain from scratch. A major challenge in continual learning of language model is catastrophic forgetting, the tendency of models to forget knowledge from previously trained tasks/domains when training on new ones. This paper studies dialog generation under the continual learning setting. We propose a novel method that 1) uses \textit{Text-Mixup} as data augmentation to avoid model overfitting on replay memory and 2) leverages Batch-Nuclear Norm Maximization (BNNM) to alleviate the problem of mode collapse. Experiments on a $37$-domain task-oriented dialog dataset and DailyDialog (a $10$-domain chitchat dataset) demonstrate that our proposed approach outperforms the state-of-the-art in continual learning., Comment: 12 pages, 2 figures

Diffusion-based text-to-image personalization have achieved great success in generating subjects specified by users among various contexts. Even though, existing finetuning-based methods still suffer from model overfitting, which greatly harms the generative diversity, especially when given subject images are few. To this end, we propose Pick-and-Draw, a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. Our approach consists of two components: appearance picking guidance and layout drawing guidance. As for the former, we construct an appearance palette with visual features from the reference image, where we pick local patterns for generating the specified subject with consistent identity. As for layout drawing, we outline the subject's contour by referring to a generative template from the vanilla diffusion model, and inherit the strong image prior to synthesize diverse contexts according to different text conditions. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image. Qualitative and quantitative experiments show that Pick-and-Draw consistently improves identity consistency and generative diversity, pushing the trade-off between subject fidelity and image-text fidelity to a new Pareto frontier.

PurposeTo extend the MR MultiTasking-based Multidimensional Assessment of Cardiovascular System (MT-MACS) technique with larger spatial coverage and water-fat separation for comprehensive aortocardiac assessment.MethodsMT-MACS adopts a low-rank tensor image model for 7D imaging, with three spatial dimensions for volumetric imaging, one cardiac motion dimension for cine imaging, one respiratory motion dimension for free-breathing imaging, one T2-prepared inversion recovery time dimension for multi-contrast assessment, and one T2*-decay time dimension for water-fat separation. Nine healthy subjects were recruited for the 3T study. Overall image quality was scored on bright-blood (BB), dark-blood (DB), and gray-blood (GB) contrasts using a 4-point scale (0-poor to 3-excellent) by two independent readers, and their interreader agreement was evaluated. Myocardial wall thickness and left ventricular ejection fraction (LVEF) were quantified on DB and BB contrasts, respectively. The agreement in these metrics between MT-MACS and conventional breath-held, electrocardiography-triggered 2D sequences were evaluated.ResultsMT-MACS provides both water-only and fat-only images with excellent image quality (average score = 3.725/3.780/3.835/3.890 for BB/DB/GB/fat-only images) and moderate to high interreader agreement (weighted Cohen's kappa value = 0.727/0.668/1.000/1.000 for BB/DB/GB/fat-only images). There were good to excellent agreements in myocardial wall thickness measurements (intraclass correlation coefficients [ICC] = 0.781/0.929/0.680/0.878 for left atria/left ventricle/right atria/right ventricle) and LVEF quantification (ICC = 0.716) between MT-MACS and 2D references. All measurements were within the literature range of healthy subjects.ConclusionThe refined MT-MACS technique provides multi-contrast, phase-resolved, and water-fat imaging of the aortocardiac systems and allows evaluation of anatomy and function. Clinical validatio

Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by a layout instruction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding to the input layout information without training auxiliary modules or finetuning diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during generative process, and assists the model to synthesize images (1) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap between consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-the-art zero-shot grounded T2I generation methods by a large margin both qualitatively and quantitatively on several benchmarks., Comment: Preprint. Under review. Project page: https://sagileo.github.io/Region-and-Boundary

Analysis of vessel morphology is important in assessing intracranial atherosclerosis disease (ICAD). Recently, magnetic resonance (MR) vessel wall imaging (VWI) has been introduced to image ICAD and characterize morphology for atherosclerotic lesions. In order to automatically perform quantitative analysis on VWI data, MR angiography (MRA) acquired in the same imaging session is typically used to localize the vessel segments of interest. However, MRA may be unavailable caused by the lack or failure of the sequence in a VWI protocol. This study aims to investigate the feasibility to infer the vessel location directly from VWI. We propose to synergize an atlas-based method to preserve general vessel structure topology with a deep learning network in the motion field domain to correct the residual geometric error. Performance is quantified by examining the agreement between the extracted vessel structures from the pair-acquired and alignment-corrected angiogram, and the estimated output using a cross-validation scheme. Our proposed pipeline yields clinically feasible performance in localizing intracranial vessels, demonstrating the promise of performing vessel morphology analysis using VWI alone.

Analysis of vessel morphology is important in assessing intracranial atherosclerosis disease (ICAD). Recently, magnetic resonance (MR) vessel wall imaging (VWI) has been introduced to image ICAD and characterize morphology for atherosclerotic lesions. In order to automatically perform quantitative analysis on VWI data, MR angiography (MRA) acquired in the same imaging session is typically used to localize the vessel segments of interest. However, MRA may be unavailable caused by the lack or failure of the sequence in a VWI protocol. This study aims to investigate the feasibility to infer the vessel location directly from VWI. We propose to synergize an atlas-based method to preserve general vessel structure topology with a deep learning network in the motion field domain to correct the residual geometric error. Performance is quantified by examining the agreement between the extracted vessel structures from the pair-acquired and alignment-corrected angiogram, and the estimated output using a cross-validation scheme. Our proposed pipeline yields clinically feasible performance in localizing intracranial vessels, demonstrating the promise of performing vessel morphology analysis using VWI alone.

Objective.To develop and test the feasibility of a novel Single ProjectIon DrivEn Real-time Multi-contrast (SPIDERM) MR imaging technique that can generate real-time 3D images on-the-fly with flexible contrast weightings and a low latency.Approach.In SPIDERM, a 'prep' scan is first performed, with sparse k-space sampling periodically interleaved with the central k-space line (navigator data), to learn a subject-specific model, incorporating a spatial subspace and a linear transformation between navigator data and subspace coordinates. A 'live' scan is then performed by repeatedly acquiring the central k-space line only to dynamically determine subspace coordinates. With the 'prep'-learned subspace and 'live' coordinates, real-time 3D images are generated on-the-fly with computationally efficient matrix multiplication. When implemented based on a multi-contrast pulse sequence, SPIDERM further allows for data-driven image contrast regeneration to convert real-time contrast-varying images into contrast-frozen images at user's discretion while maintaining motion states. Both digital phantom andin-vivoexperiments were performed to evaluate the technical feasibility of SPIDERM.Main results.The elapsed time from the input of the central k-space line to the generation of real-time contrast-frozen 3D images was approximately 45 ms, permitting a latency of 55 ms or less. Motion displacement measured from SPIDERM and reference images showed excellent correlation (R2≥0.983). Geometric variation from the ground truth in the digital phantom was acceptable as demonstrated by pancreas contour analysis (Dice ≥ 0.84, mean surface distance ≤ 0.95 mm). Quantitative image quality metrics showed good consistency between reference images and contrast-varying SPIDREM images inin-vivostudies (meanNMRSE=0.141,PSNR=30.12,SSIM=0.88).Significance.SPIDERM is capable of generating real-time multi-contrast 3D images with a low latency. An imaging framework based on SPIDERM has the potential to ser

PURPOSE: To develop an automated vessel wall segmentation method on T1-weighted intracranial vessel wall magnetic resonance images, with a focus on modeling the inclusion relation between the inner and outer boundaries of the vessel wall. METHODS: We propose a novel method that estimates the inner and outer vessel wall boundaries simultaneously, using a network with a single output channel resembling the level-set function height. The network is driven by a unique tiered loss that accounts for data fidelity of the lumen and vessel wall classes and a length regularization to encourage boundary smoothness. RESULTS: Implemented with a 2.5D UNet with a ResNet backbone, the proposed method achieved Dice similarity coefficients (DSC) in 2D of 0.925 ± 0.048, 0.786 ± 0.084, Hausdorff distance (HD) of 0.286 ± 0.436, 0.345 ± 0.419 mm, and mean surface distance (MSD) of 0.083 ± 0.037 and 0.103 ± 0.032 mm for the lumen and vessel wall, respectively, on a test set; compared favorably to a baseline UNet model that achieved DSC 0.924 ± 0.047, 0.794 ± 0.082, HD 0.298 ± 0.477, 0.394 ± 0.431 mm, and MSD 0.087 ± 0.056, 0.119 ± 0.059 mm. Our vessel wall segmentation method achieved substantial improvement in morphological integrity and accuracy compared to benchmark methods. CONCLUSIONS: The proposed method provides a systematic approach to model the inclusion morphology and incorporate it into an optimization infrastructure. It can be applied to any application where inclusion exists among a (sub)set of classes to be segmented. Improved feasibility in result morphology promises better support for clinical quantification and decision.

Training a generative adversarial network (GAN) with limited data has been a challenging task. A feasible solution is to start with a GAN well-trained on a large scale source domain and adapt it to the target domain with a few samples, termed as few shot generative model adaption. However, existing methods are prone to model overfitting and collapse in extremely few shot setting (less than 10). To solve this problem, we propose a relaxed spatial structural alignment method to calibrate the target generative models during the adaption. We design a cross-domain spatial structural consistency loss comprising the self-correlation and disturbance correlation consistency loss. It helps align the spatial structural information between the synthesis image pairs of the source and target domains. To relax the cross-domain alignment, we compress the original latent space of generative models to a subspace. Image pairs generated from the subspace are pulled closer. Qualitative and quantitative experiments show that our method consistently surpasses the state-of-the-art methods in few shot setting., Comment: Accepted by CVPR 2022

The backbone of traditional CNN classifier is generally considered as a feature extractor, followed by a linear layer which performs the classification. We propose a novel loss function, termed as CAM-loss, to constrain the embedded feature maps with the class activation maps (CAMs) which indicate the spatially discriminative regions of an image for particular categories. CAM-loss drives the backbone to express the features of target category and suppress the features of non-target categories or background, so as to obtain more discriminative feature representations. It can be simply applied in any CNN architecture with neglectable additional parameters and calculations. Experimental results show that CAM-loss is applicable to a variety of network structures and can be combined with mainstream regularization methods to improve the performance of image classification. The strong generalization ability of CAM-loss is validated in the transfer learning and few shot learning tasks. Based on CAM-loss, we also propose a novel CAAM-CAM matching knowledge distillation method. This method directly uses the CAM generated by the teacher network to supervise the CAAM generated by the student network, which effectively improves the accuracy and convergence rate of the student network., Comment: Accepted by ICCV2021

PurposeTo develop an MR multitasking-based multidimensional assessment of cardiovascular system (MT-MACS) with electrocardiography-free and navigator-free data acquisition for a comprehensive evaluation of thoracic aortic diseases.MethodsThe MT-MACS technique adopts a low-rank tensor image model with a cardiac time dimension for phase-resolved cine imaging and a T2 -prepared inversion-recovery dimension for multicontrast assessment. Twelve healthy subjects and 2 patients with thoracic aortic diseases were recruited for the study at 3 T, and both qualitative (image quality score) and quantitative (contrast-to-noise ratio between lumen and wall, lumen and wall area, and aortic strain index) analyses were performed in all healthy subjects. The overall image quality was scored based on a 4-point scale: 3, excellent; 2, good; 1, fair; and 0, poor. Statistical analysis was used to test the measurement agreement between MT-MACS and its corresponding 2D references.ResultsThe MT-MACS images reconstructed from acquisitions as short as 6 minutes demonstrated good or excellent image quality for bright-blood (2.58 ± 0.46), dark-blood (2.58 ± 0.50), and gray-blood (2.17 ± 0.53) contrast weightings, respectively. The contrast-to-noise ratios for the three weightings were 49.2 ± 12.8, 20.0 ± 5.8 and 2.8 ± 1.8, respectively. There were good agreements in the lumen and wall area (intraclass correlation coefficient = 0.993, P < .001 for lumen; intraclass correlation coefficient = 0.969, P < .001 for wall area) and strain (intraclass correlation coefficient = 0.947, P < .001) between MT-MACS and conventional 2D sequences.ConclusionThe MT-MACS technique provides high-quality, multidimensional images for a comprehensive assessment of the thoracic aorta. Technical feasibility was demonstrated in healthy subjects and patients with thoracic aortic diseases. Further clinical validation is warranted.

PurposeSegmentation of multiple organs-at-risk (OARs) is essential for magnetic resonance (MR)-only radiation therapy treatment planning and MR-guided adaptive radiotherapy of abdominal cancers. Current practice requires manual delineation that is labor-intensive, time-consuming, and prone to intra- and interobserver variations. We developed a deep learning (DL) technique for fully automated segmentation of multiple OARs on clinical abdominal MR images with high accuracy, reliability, and efficiency.MethodsWe developed Automated deep Learning-based abdominal multiorgan segmentation (ALAMO) technique based on two-dimensional U-net and a densely connected network structure with tailored design in data augmentation and training procedures such as deep connection, auxiliary supervision, and multiview. The model takes in multislice MR images and generates the output of segmentation results. 3.0-Tesla T1 VIBE (Volumetric Interpolated Breath-hold Examination) images of 102 subjects were used in our study and split into 66 for training, 16 for validation, and 20 for testing. Ten OARs were studied, including the liver, spleen, pancreas, left/right kidneys, stomach, duodenum, small intestine, spinal cord, and vertebral bodies. An experienced radiologist manually labeled each OAR, followed by reediting, if necessary, by a senior radiologist, to create the ground-truth. The performance was measured using volume overlapping and surface distance.ResultsThe ALAMO technique generated segmentation labels in good agreement with the manual results. Specifically, among the ten OARs, nine achieved high dice similarity coefficients (DSCs) in the range of 0.87-0.96, except for the duodenum with a DSC of 0.80. The inference completed within 1 min for a three-dimensional volume of 320 × 288 × 180. Overall, the ALAMO model matched the state-of-the-art techniques in performance.ConclusionThe proposed ALAMO technique allows for fully automated abdominal MR segmentation

Segmentation of multiple organs-at-risk (OARs) is essential for radiation therapy treatment planning and other clinical applications. We developed an Automated deep Learning-based Abdominal Multi-Organ segmentation (ALAMO) framework based on 2D U-net and a densely connected network structure with tailored design in data augmentation and training procedures such as deep connection, auxiliary supervision, and multi-view. The model takes in multi-slice MR images and generates the output of segmentation results. Three-Tesla T1 VIBE (Volumetric Interpolated Breath-hold Examination) images of 102 subjects were collected and used in our study. Ten OARs were studied, including the liver, spleen, pancreas, left/right kidneys, stomach, duodenum, small intestine, spinal cord, and vertebral bodies. Two radiologists manually labeled and obtained the consensus contours as the ground-truth. In the complete cohort of 102, 20 samples were held out for independent testing, and the rest were used for training and validation. The performance was measured using volume overlapping and surface distance. The ALAMO framework generated segmentation labels in good agreement with the manual results. Specifically, among the 10 OARs, 9 achieved high Dice Similarity Coefficients (DSCs) in the range of 0.87-0.96, except for the duodenum with a DSC of 0.80. The inference completes within one minute for a 3D volume of 320x288x180. Overall, the ALAMO model matches the state-of-the-art performance. The proposed ALAMO framework allows for fully automated abdominal MR segmentation with high accuracy and low memory and computation time demands., Comment: 21 pages, 4 figures, submitted to the journal Medical Physics