Search

This work presents Spacecraft Pose Network v2 (SPNv2), a Convolutional Neural Network (CNN) for pose estimation of noncooperative spacecraft across domain gap. SPNv2 is a multi-scale, multi-task CNN which consists of a shared multi-scale feature encoder and multiple prediction heads that perform different tasks on a shared feature output. These tasks are all related to detection and pose estimation of a target spacecraft from an image, such as prediction of pre-defined satellite keypoints, direct pose regression, and binary segmentation of the satellite foreground. It is shown that by jointly training on different yet related tasks with extensive data augmentations on synthetic images only, the shared encoder learns features that are common across image domains that have fundamentally different visual characteristics compared to synthetic images. This work also introduces Online Domain Refinement (ODR) which refines the parameters of the normalization layers of SPNv2 on the target domain images online at deployment. Specifically, ODR performs self-supervised entropy minimization of the predicted satellite foreground, thereby improving the CNN's performance on the target domain images without their pose labels and with minimal computational efforts. The GitHub repository for SPNv2 is available at https://github.com/tpark94/spnv2. [ABSTRACT FROM AUTHOR]

Sketch-based image retrieval (SBIR) refers to a sub-class of content-based image retrieval problems where the input queries are ambiguous sketches and the retrieval repository is a database of natural images. In the zero-shot setup of SBIR, the query sketches are drawn from classes that do not match any of those that were used in model building. The SBIR task is extremely challenging as it is a cross-domain retrieval problem, unlike content-based image retrieval problems because sketches and images have a huge domain gap. In this work, we propose an elegant retrieval methodology, StyleGen, for generating fake candidate images that match the domain of the repository images, thus reducing the domain gap for retrieval tasks. The retrieval methodology makes use of a two-stage neural network architecture known as the stacked Siamese network, which is known to provide outstanding retrieval performance without losing the generalizability of the approach. Experimental studies on the image sketch datasets TU-Berlin Extended and Sketchy Extended, evaluated using the mean average precision (mAP) metric, demonstrate a marked performance improvement compared to the current state-of-the-art approaches in the domain. [ABSTRACT FROM AUTHOR]

Deep learning has been the state-of-the-art solution to numerous remote sensing tasks, especially for building extraction. However, the performance of learning-based building extraction approaches depend to a large extent on the similarity of the source and target domain data. To alleviate the dependence on annotated data, and to exploit the potential of multimodal remote sensing data, a 3-D assisted semisupervised method for building extraction is proposed. The proposed method is based on self-training, a semisupervised method that utilizes both labeled and unlabeled data. In addition, photogrammetric digital surface model and belief function are exploited to bridge the domain gaps between the source and target data. The performance is evaluated with ISPRS Potsdam and Vaihingen benchmark datasets, and a WorldView-2 satellite multimodal dataset. Compared with the direct cross-domain test baseline, improvement of Jaccard score ranging from 8.91% to 21.39% is achieved, demonstrating the efficacy of the proposed 3-D self-training method.

Sketch-based image retrieval (SBIR) refers to a sub-class of content-based image retrieval problems where the input queries are ambiguous sketches and the retrieval repository is a database of natural images. In the zero-shot setup of SBIR, the query sketches are drawn from classes that do not match any of those that were used in model building. The SBIR task is extremely challenging as it is a cross-domain retrieval problem, unlike content-based image retrieval problems because sketches and images have a huge domain gap. In this work, we propose an elegant retrieval methodology, StyleGen, for generating fake candidate images that match the domain of the repository images, thus reducing the domain gap for retrieval tasks. The retrieval methodology makes use of a two-stage neural network architecture known as the stacked Siamese network, which is known to provide outstanding retrieval performance without losing the generalizability of the approach. Experimental studies on the image sketch datasets TU-Berlin Extended and Sketchy Extended, evaluated using the mean average precision (mAP) metric, demonstrate a marked performance improvement compared to the current state-of-the-art approaches in the domain.

Deep learning-based intelligent vehicle perception has been developing prominently in recent years to provide a reliable source for motion planning and decision making in autonomous driving. A large number of powerful deep learning-based methods can achieve excellent performance in solving various perception problems of autonomous driving. However, these deep learning methods still have several limitations, for example, the assumption that lab-training (source domain) and real-testing (target domain) data follow the same feature distribution may not be practical in the real world. There is often a dramatic domain gap between them in many real-world cases. As a solution to this challenge, deep transfer learning can handle situations excellently by transferring the knowledge from one domain to another. Deep transfer learning aims to improve task performance in a new domain by leveraging the knowledge of similar tasks learned in another domain before. Nevertheless, there are currently no survey papers on the topic of deep transfer learning for intelligent vehicle perception. To the best of our knowledge, this paper represents the first comprehensive survey on the topic of the deep transfer learning for intelligent vehicle perception. This paper discusses the domain gaps related to the differences of sensor, data, and model for the intelligent vehicle perception. The recent applications, challenges, future researches in intelligent vehicle perception are also explored.

Copyright of Telecommunication Engineering is the property of Telecommunication Engineering and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This article concerns the second Satellite Pose Estimation Competition (SPEC2021), an international competition designed to address the challenging problem of domain gap in developing spaceborne computer vision algorithms. Due to the lack of representative spaceborne images of the target object at the development stage, most algorithms are typically developed and trained on computer-generated synthetic images of the target, which causes the algorithm to overfit the features specific to the synthetic imagery. In order to promote new ideas to tackle this important problem and benchmark the robustness of different algorithms across domain gaps, SPEC2021 utilizes the next-generation spacecraft pose estimation dataset comprising images from labeled images from computer graphics and real-life sources. This article covers the dataset and competition design and analyzes the results of the competition which received submissions from 36 participating teams. Specifically, through the solutions developed by the two winning teams, this article sheds light on key components that can contribute to successfully bridging the domain gap in spaceborne computer vision applications. • Domain gap is a challenging problem for vision-based spaceborne navigation. • SPEC2021 is the first international effort to address domain gap in space applications. • The winners of SPEC2021 achieved low pose errors across domain gap. • The winners identified adversarial training to be crucial to bridging domain gap. [ABSTRACT FROM AUTHOR]

Emotion is an important area in neuroscience. Cross-subject emotion recognition based on electroencephalogram (EEG) data is challenging due to physiological differences between subjects. Domain gap, which refers to the different distributions of EEG data at different subjects, has attracted great attention for cross-subject emotion recognition. This study focuses on narrowing the domain gap between subjects through the emotional frequency bands and the relationship information between EEG channels. Emotional frequency band features represent the energy distribution of EEG data in different frequency ranges, while relationship information between EEG channels provides spatial distribution information about EEG data. To achieve this, this paper proposes a model called the Frequency Band Attention Graph convolutional Adversarial neural Network (FBAGAN). This model includes three components: a feature extractor, a classifier, and a discriminator. The feature extractor consists of a layer with a frequency band attention mechanism and a graph convolutional neural network. The mechanism effectively extracts frequency band information by assigning weights and Graph Convolutional Networks can extract relationship information between EEG channels by modeling the graph structure. The discriminator then helps minimize the gap in the frequency information and relationship information between the source and target domains, improving the model's ability to generalize. The FBAGAN model is extensively tested on the SEED, SEED-IV, and DEAP datasets. The accuracy and standard deviation scores are 88.17% and 4.88, respectively, on the SEED dataset, and 77.35% and 3.72 on the SEED-IV dataset. On the DEAP dataset, the model achieves 69.64% for Arousal and 65.18% for Valence. These results outperform most existing models. The experiments indicate that FBAGAN effectively addresses the challenges of transferring EEG channel domain and frequency band domain, leading to improved performance. • Focuses on leveraging frequency and EEG channel relationship information. • Reducing domain gap through adversarial learning. • Attention mechanism extracts specific frequency band information. • Experimental results show that FBAGAN outperforms most existing models. [ABSTRACT FROM AUTHOR]

Due to the high cost and time-consumption of artificial semantic tags, domain-based adaptive semantics segmentation is very necessary. For scenes with large gaps or pixels, it is easy to limit model training and reduce the accuracy of semantic segmentation. In this paper, a domain adaptive semantic segmentation network (DA-SSN) using the improved transformation network is proposed by eliminating the interference of large gap pictures and pixels through staged training and interpretable masks. First, in view of the problem of large domain gaps from some source graphs to target graphs and the difficulty in network model training, the training loss threshold is used to divide the source graph dataset with large gaps, and a phased transformation network training strategy is proposed. Based on the ensurance of the semantic alignment of small gap source images, the transformation quality of large gap source images is improved. In addition, in order to further reduce the gap between some pixels in the source image and the target image area, an interpretable mask is proposed. By predicting the gap between each pixel in the source image domain and the target image domain, the confidence is reduced, and the training loss of the corresponding pixel is ignored to eliminate the influence of large gap pixels on the semantic alignment of other pixels, so that model training only focuses on the domain gap of high-confidence pixels. The results show that the proposed algorithm has a higher segmentation accuracy than the original domain adaptive semantic segmentation network. Compared with the results of other popular algorithms, the proposed method obtains a higher quality semantic alignment, which shows the advantages of the proposed method with high accuracy.

The wiring harness manufacturing follows a high-variant manufacturing philosophy with a high degree of manual labor to produce customized products. Therefore, the process step of optical inspection is crucial to monitor, assess, and proactively safeguard the quality of produced wiring harnesses. To automate the optical inspection, which is currently manually conducted by operators, deep learning has become a powerful algorithm outperforming traditional computer vision approaches. Deep learning-based automated optical inspection systems can robustly and reliably detect rigid and deformable components, such as connectors, clips, single wires, and wire bundles. However, the bottleneck for scalable deep learning solutions in the industrial environment and high deep learning model performance is the database for model training and optimization. To address this research gap, we propose a deep learning-based data processing pipeline for automated optical inspection of wiring harnesses using real and synthetically generated point clouds. This paper outlines the process of real and synthetic data generation and evaluates the potential of synthetic data to enrich real data for model training. The data processing pipeline is implemented and experimental findings are generated to deduct important parameters for synthetic data generation and deep learning model training which impact model performance. [ABSTRACT FROM AUTHOR]

Deep learning has shown great power in processing remote sensing data, especially for fine-grained remote sensing ship image classification. However, the lack of a large amount of effective training data greatly limits the performance of neural networks. Based on current data augmentation methods, images of ships on the sea generated for remote sensing have the problem of distortion, blurring, and poor diversity. To tackle this problem, we propose a novel progressive remote sensing ship image data augmentation method that combines ship simulation samples and a neural style transfer (NST) based network to generate a large amount of transferred remote sensing ship images. Our method consists of two stages. The first stage uses a visible light imaging simulation system to generate ship simulation samples through three-dimensional models of real images. This stage can significantly increase the diversity of the training dataset. For the second stage, to eliminate the domain gap between real ship images and ship simulation samples, a few real images and a newly designed NST-based network called Sim2RealNet are employed to realize style transfer from simulation samples to real images. The proposed method was applied to a variety of ship targets to verify its effectiveness compared to other data augmentation methods on remote sensing image classification tasks. The experimental results demonstrate the effectiveness of the proposed method.

Even though the field of medical imaging advances, there are structures in the human body that are barely assessible with classical image acquisition modalities. One example are the three leaflets of the aortic valve due to their thin structure and high movement. However, with an increasing accuracy of biomechanical simulation, for example of the heart function, and extense computing capabilities available, concise knowledge of the individual morphology of these structures could have a high impact on personalized therapy and intervention planning as well as on clinical research. Thus, there is a high demand to estimate the individual shape of inassessible structures given only information on the geometry of the surrounding tissue. This leads to a domain adaptation problem, where the domain gap could be very large while typically only small datasets are available. Hence, classical approaches for domain adaptation are not capable of providing sufficient predictions. In this work, we present a new framework for bridging this domain gap in the scope of estimating anatomical shapes based on the surrounding tissue's morphology. Thus, we propose deep representation learning to not map from one image to another but to predict a latent shape representation. We formalize this framework and present two different approaches to solve the given problem. Furthermore, we perform a proof-of-concept study for estimating the individual shape of the aortic valve leaflets based on a volumetric ultrasound image of the aortic root. Therefore, we collect an ex-vivo porcine data set consisting of both, ultrasound volume images as well as high-resolution leaflet images, evaluate both approaches on it and perform an analysis of the model's hyperparameters. Our results show that using deep representation learning and domain mapping between the identified latent spaces, a robust prediction of the unknown leaflet shape only based on surrounding tissue information is possible, even in limited data scenarios. The concept can be applied to a wide range of modeling tasks, not only in the scope of heart modeling but also for all kinds of inassessible structures within the human body.

Extracting accurate foregrounds from natural images benefits many downstream applications such as film production and augmented reality. However, the furry characteristics and various appearance of the foregrounds, e.g., animal and portrait, challenge existing matting methods, which usually require extra user inputs such as trimap or scribbles. To resolve these problems, we study the distinct roles of semantics and details for image matting and decompose the task into two parallel sub-tasks: high-level semantic segmentation and low-level details matting. Specifically, we propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end natural image matting. Besides, due to the limitation of available natural images in the matting task, previous methods typically adopt composite images for training and evaluation, which result in limited generalization ability on real-world images. In this paper, we investigate the domain gap issue between composite images and real-world images systematically by conducting comprehensive analyses of various discrepancies between the foreground and background images. We find that a carefully designed composition route RSSN that aims to reduce the discrepancies can lead to a better model with remarkable generalization ability. Furthermore, we provide a benchmark containing 2,000 high-resolution real-world animal images and 10,000 portrait images along with their manually labeled alpha mattes to serve as a test bed for evaluating matting model's generalization ability on real-world images. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods and effectively reduces the generalization error. The code and the datasets will be released at https://github.com/JizhiziLi/GFM. [ABSTRACT FROM AUTHOR]

In this paper, we introduce and address a crucial but less accentuated problem in cross-domain retrieval task. We first highlight the challenge caused by diversity of inter-class similarities across different domains. For example, bear and teddy bear classes have almost the same instances in a sketch domain whereas they are greatly different in photos. In other words, two classes can be mixed together or even far away in the feature space depending on the modality. A key component of the problem, mapping feature representations of different domains to a common space while preserving semantic similarities, is a tough task due to the domain gap. In order to address such diversity, we introduce a novel feature learning approach by forcing the decision axes of a softmax-based classifier to be mutually orthogonal. Since the feature spaces of different domains share a regularized structure thanks to the orthogonal decision axes, we formulate the mapping to a common space as a simple linear transformation. Our proposed method is extensively evaluated on four different cross-domain retrieval benchmarks consisting of various data modalities, and we found that our method significantly outperforms the state-of-the-art techniques.

Even though neural network methodologies have been established for a long time, only recently have they achieved exceptional efficacy in practical deployments, predominantly due to improvements in hardware computational capacity and the large amounts of available data for learning. Nonetheless, substantial challenges remain in utilizing deep learning in many domains, mainly because of the lack of large amounts of labeled data that are versatile enough for deep learning models to learn useful information. For instance, in mechanical assembly inspection, annotating data for each type of mechanical part to train a deep learning model can be very labor-intensive. Additionally, it is required to annotate data after each modification of mechanical part specification. Also, the system for inspection is typically not available until the first few samples are built to collect data. This paper proposes a solution for these challenges in case of the visual mechanical assembly inspection by processing point cloud data acquired via a three-dimensional (3D) scanner. To reduce the necessity for manually labeling large amounts of data, we employed synthetically generated data for both training and validation purposes, reserving the real sensor data exclusively for the testing phase. Our approach reduces the need for large amounts of labeled data by using synthetically generated point clouds from computer-aided design models for neural network training. Domain gap is a significant challenge for the usage of synthetically generated data. To reduce the domain gap, we used different preprocessing techniques, as well as a neural network architecture that focuses more on shared features that will not significantly change between synthetically generated data and real data from the 3D sensor. [ABSTRACT FROM AUTHOR]

To ensure reliable object detection in autonomous systems, the detector must be able to adapt to changes in appearance caused by environmental factors such as time of day, weather, and seasons. Continually adapting the detector to incorporate these changes is a promising solution, but it can be computationally costly. Our proposed approach is to selectively adapt the detector only when necessary, using new data that does not have the same distribution as the current training data. To this end, we investigate three popular metrics for domain gap evaluation and find that there is a correlation between the domain gap and detection accuracy. Therefore, we apply the domain gap as a criterion to decide when to adapt the detector. Our experiments show that our approach has the potential to improve the efficiency of the detector's operation in real-world scenarios, where environmental conditions change in a cyclical manner, without sacrificing the overall performance of the detector. Our code is publicly available https://github.com/dadung/DGE-CDA. • Address the question "when to adapt" which is ignored by prior works. • Show a correlation between domain gap and detection accuracy. • Assessing domain gap helps save energy usage for continual adaption without sacrificing detection accuracy. [ABSTRACT FROM AUTHOR]

Open-set domain adaptation is a developing and practical solution to training deep networks using unlabeled data which have been collected among unknown data and are under domain shift with other labeled data. This scenario transfers knowledge from a source domain enriched with labeled data to the unlabeled target domain, meanwhile, unknown target samples which are not present in the source domain are separated. Existing methods aim to bridge the domain gap of shared classes between source and target domains in a trustworthy manner and avoid negative transfer learning using keeping away unknown data from the domain alignment step. In this review article, we present a unified framework of theory advances for network risk and a new categorization of open-set domain adaptation along with listing evaluation metrics and popular datasets. Then we accentuate challenges and gaps in existing studies to organize a road map for future research using detailed analysis of investigations. To bring things full circle, we also point out different assumptions and outlooks in the settings of this research area. • For the first time, existing methods in deep domain adaptation with open-set target data are categorized and reviewed. • To produce an optimum upper bound for network risk in the training step, needed theories are presented. • For each category of open-set domain adaptation methods, the problem assumptions along with their algorithms are pointed out. • To present a comparative framework for these methods, their performances, advantages and disadvantages are also examined. • Finally, for paving the research directions in future works, research gaps and challenges are discussed. [ABSTRACT FROM AUTHOR]

Deep neural network has achieved promising results for automatic glaucoma detection on fundus images. Nevertheless, the intrinsic discrepancy across glaucoma datasets is challenging for the data-driven neural network approaches. This discrepancy leads to the domain gap that affects model performance and declines model generalization capability. Existing domain adaptation-based transfer learning methods mostly fine-tune pretrained models on target domains to reduce the domain gap. However, this feature learning-based adaptation method is implicit, and it is not an optimal solution for transfer learning on the diverse glaucoma datasets. In this paper, we propose a mixup domain adaptation (mixDA) method that bridges domain adaptation with domain mixup to improve model performance across divergent glaucoma datasets. Specifically, the domain adaptation reduces the domain gap of glaucoma datasets in transfer learning with an explicit adaptation manner. Meanwhile, the domain mixup further minimizes the risk of outliers after domain adaptation and improves the model generalization capability. Extensive experiments show the superiority of our mixDA on several public glaucoma datasets. Moreover, our method outperforms state-of-the-art methods by a large margin on four glaucoma datasets: REFUGE, LAG, ORIGA, and RIM-ONE. [ABSTRACT FROM AUTHOR]

Even though the field of medical imaging advances, there are structures in the human body that are barely assessible with classical image acquisition modalities. One example are the three leaflets of the aortic valve due to their thin structure and high movement. However, with an increasing accuracy of biomechanical simulation, for example of the heart function, and extense computing capabilities available, concise knowledge of the individual morphology of these structures could have a high impact on personalized therapy and intervention planning as well as on clinical research. Thus, there is a high demand to estimate the individual shape of inassessible structures given only information on the geometry of the surrounding tissue. This leads to a domain adaptation problem, where the domain gap could be very large while typically only small datasets are available. Hence, classical approaches for domain adaptation are not capable of providing sufficient predictions. In this work, we present a new framework for bridging this domain gap in the scope of estimating anatomical shapes based on the surrounding tissue's morphology. Thus, we propose deep representation learning to not map from one image to another but to predict a latent shape representation. We formalize this framework and present two different approaches to solve the given problem. Furthermore, we perform a proof-of-concept study for estimating the individual shape of the aortic valve leaflets based on a volumetric ultrasound image of the aortic root. Therefore, we collect an ex-vivo porcine data set consisting of both, ultrasound volume images as well as high-resolution leaflet images, evaluate both approaches on it and perform an analysis of the model's hyperparameters. Our results show that using deep representation learning and domain mapping between the identified latent spaces, a robust prediction of the unknown leaflet shape only based on surrounding tissue information is possible, even in limited data scenarios. The concept can be applied to a wide range of modeling tasks, not only in the scope of heart modeling but also for all kinds of inassessible structures within the human body., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2022 Hagenah, Scharfschwerdt and Ernst.)

To train Deep Neural Networks (DNNs)-based methods, suitable training data are key to help DNNs learn appropriate pattern recognition features. The use of synthetic data may help in generating sufficient and balanced data. However, models trained with such data often present a domain gap when applied to real-world scenarios. Many studies focus on techniques such as domain adaptation to minimize this gap, but little attention is paid to the data generation itself. Our work shows that this gap can be minimized by enhancing the generated data features. More specifically, we generate different synthetic training datasets with particular features and use them to train a DNN for people detection in large spaces using omnidirectional cameras. Experimental results with real-world data show that proper synthetic data minimize the domain gap. We also show that expanding a training dataset to include synthetic samples in addition to real samples, can improve the model's capabilities. [Display omitted] • Training data are key for the success of Deep Neural Networks and their behaviour. • Synthetic data helps in this process, but the domain gap worsens the results. • Enhancing the generated synthetic data features minimizes the domain gap. • Suitable synthetic samples can improve a model's generalization ability. [ABSTRACT FROM AUTHOR]