Search

This paper describes an adaptation of the Local Interpretable Model-Agnostic Explanations (LIME) AI method to operate under a biometric verification setting. LIME was initially proposed for networks with the same output classes used for training, and it employs the softmax probability to determine which regions of the image contribute the most to classification. However, in a verification setting, the classes to be recognized have not been seen during training. In addition, instead of using the softmax output, face descriptors are usually obtained from a layer before the classification layer. The model is adapted to achieve explainability via cosine similarity between feature vectors of perturbated versions of the input image. The method is showcased for face biometrics with two CNN models based on MobileNetv2 and ResNet50. © 2023 IEEE., EXPLAINING - ”Project EXPLainable Artificial INtelligence systems for health and well-beING”

We introduce a non-parametric density estimator deemed Radial Voronoi Density Estimator (RVDE). RVDE is grounded in the geometry of Voronoi tessellations and as such benefits from local geometric adaptiveness and broad convergence properties. Due to its radial definition RVDE is continuous and computable in linear time with respect to the dataset size. This amends for the main shortcomings of previously studied VDEs, which are highly discontinuous and computationally expensive. We provide a theoretical study of the modes of RVDE as well as an empirical investigation of its performance on high-dimensional data. Results show that RVDE outperforms other non-parametric density estimators, including recently introduced VDEs., QC 20230821

Subterranean exploration and mapping for search and rescue robotics have become an emerging research direction since the DARPA organized Subterranean Challenge. As part of development efforts within the team CoSTAR (Collaborative SubTerranean Autonomous Robots) in the Sub-T challenge, this work establishes a novel traversable graph-based exploration strategy that utilizes frontiers for local navigation and a fast collision risk-aware graph building for global navigation. The exploration strategy extracts frontiers in an unknown area that contribute to safe navigation while maximizing information gain for the robot. The exploration problem is further bifurcated into local and global exploration for faster decision-making at junctions with the goal of rapidly exploring the area. The local exploration guarantees collision-free straight-line paths to informative frontiers for rapid forward navigation, while global re-positioning utilizes a traversable graph subject to geometrical collision checks within the occupancy map. The pathfinding in a graph is addressed using a heuristic, which combines risk margins and travel costs to assist in short yet safe paths to the global frontier in case of a dead end in local exploration. The presented exploration strategy is developed with the goal of making exploration algorithms platform agnostic in order to be able to use it with aerial, as well as ground robots. The proposed method is also evaluated against different state-of-the-art exploration planners in simulated fixed-time budget-based exploration missions on an Unmanned Aerial Vehicle (UAV) in order to benchmark the capabilities and highlight the novelty., Full text license: CC BY-NC-ND

The rapid development of social robots, designed to interact with humans, has led to increased research on user acceptance and emotions in human-robot interaction. Social acceptance is an important area to investigate if the development of social robots is to be useful. Investigating how people feel about social robots is one tool to assess acceptance toward them, and research has shown that positive emotions could invoke higher acceptance. Possible factors that have been shown to affect peoples’ attitudes regarding social robots is (1) the human-likeness and appearance of the robot and (2) the application area of the robot. Therefore, this thesis research questions address the effect of human-likeness and application areas of social robots on people's emotions. The findings indicate that in the context of companionship, people have varying emotional responses based on the appearance of the social robot. Highly human-like robots evoke more positive emotions, while low human-likeness robots elicit more negative emotions. This suggests that individuals prefer human-like social robots in intimate interactions like companionship. The results also reveal an effect of application areas, where people respond more positively to highly human-like robots used for tasks like lecturing students or companionship for older adults. Regarding less human-like social robots, people tend to respond with greater positive emotions when used within commerce. This suggests that a simpler-looking robot with low human-likeness is more suitable for commercial applications. Negative emotions expressed in the healthcare condition may reflect mistrust in robots' abilities and the sensitivity of the healthcare area. Developers and designers should consider the emotional responses that might be evoked by the task or appearance of the social robot, to ensure successful integration into society.

This thesis explores the integration of deep learning-based depth estimation models with the ORB-SLAM3 framework to address challenges in monocular Simultaneous Localization and Mapping (SLAM), particularly focusing on pure rotational movements. The study investigates the viability of using pre-trained generic depth estimation networks, and hybrid combinations of these networks, to replace traditional depth sensors and improve scale accuracy in SLAM systems. A series of experiments are conducted outdoors, utilizing a custom camera setup designed to isolate pure rotational movements. The analysis involves assessing each model's impact on the SLAM process as well as performance indicators (KPIs) on both depth estimation and 3D tracking. Results indicate a correlation between depth estimation accuracy and SLAM performance, underscoring the potential of depth estimation models in enhancing SLAM systems. The findings contribute to the understanding of the role of monocular depth estimation in integrating with SLAM, especially in applications requiring precise spatial awareness for augmented reality., Denna avhandling utforskar integrationen av djupinlärningsbaserade modeller för djupuppskattning med ORB-SLAM3-ramverket för att möta utmaningar inom monokulär Samtidig Lokalisering och Kartläggning (SLAM), med särskilt fokus på rena rotationsrörelser. Studien undersöker möjligheten att använda förtränade generiska nätverk för djupuppskattning och hybridkombinationer av dessa nätverk, för att ersätta traditionella djupsensorer och förbättra skalanoggrannheten i SLAM-system. En serie experiment genomförs med användning av en specialbyggd kamerauppställning utformad för att isolera rena rotationsrörelser. Analysen omfattar bedömning av varje modells påverkan på SLAM-processen samt kvantitativa prestandaindikatorer (KPI:er) för både djupuppskattning och följning. Resultaten visar på ett samband mellan noggrannheten i djupuppskattningen och SLAM-prestandan, vilket understryker potentialen hos modeller för djupuppskattning i förbättringen av SLAM-system. Rönen bidrar till förståelsen av rollen som monokulär djupuppskattning har i integrationen med SLAM, särskilt i tillämpningar som kräver exakt spatial medvetenhet.

This Master's Thesis delves into the application of topological regularization techniques and relative latent representations within the realm of zero-shot model stitching. Building upon the prior work of Moschella et al. (2022) that introduces relative latent representations to enhance the similarities between latent spaces of different models, we incorporate the approach of Hofer et al. (2021), which combines Topological Data Analysis (TDA) and Machine Learning techniques for topological densification of class distributions in the latent space. The main research objective is to investigate the impact of topological regularization on zero-shot stitching performance when employing relative latent representations. Theoretical foundations for the relative transformation are established based on the intertwiner groups of activation functions. Empirical analyses are conducted to validate the assumptions underlying the construction of the relative transformation in the latent space. Moreover, experiments are performed on a Large Language Model trained on multilingual Amazon Reviews datasets to evaluate the effectiveness of zero-shot stitching while using the topological densification technique and the relative transformation. The findings indicate that the proposed methodologies can enhance the performance of multilingual model stitching. Specifically, enforcing the relative transformation to preserve the H0 homology death times distributions proves beneficial. Additionally, the presence of similar topological features plays a crucial role in achieving higher model compatibility. However, a more in-depth exploration of the geometric properties of the post-relative transformation latent space is necessary to further improve the topological densification technique. Overall, this work contributes to the emerging field of Topological Machine Learning and provides valuable insights for researchers in transfer learning and representation learning domains., Denna masteruppsats undersöker tillämpningen av topologiska regleringstekniker och relativa latenta representationer inom området för zero-shot model stitching. Genom att bygga vidare på tidigare arbete av Moschella et al. (2022), som introducerade relativa latenta representationer för att förbättra likheterna mellan latenta rummet hos olika modeller, inkorporerar vi tillvägagångssättet av Hofer et al. (2021), som kombinerar topologisk dataanalys (TDA) och maskininlärningstekniker för topologisk ``förtätning'' av klassfördelningar i det latenta utrymmet. Den huvudsakliga forskningsuppgiften är att undersöka effekten av topologisk reglering på zero-shot model stitching-prestanda när man använder relativa latenta representationer. Teoretiska grunder för den relativa transformationen etableras baserat på intertwinergrupperna för aktiveringsfunktioner. Empiriska analyser genomförs för att validera antagandena som ligger till grund för konstruktionen av den relativa transformationen i det latenta rummen. Dessutom utförs experiment på en stor språkmodell tränad på multilinguella Amazon Reviews-dataset för att utvärdera effektiviteten hos zero-shot model stitching med Hofer's topologiska reglering och relativa transformation. Resultaten visar att de föreslagna metoderna kan förbättra prestationen hos zero-shot model stitching för flerspråkiga modeller. Specifikt är det fördelaktigt att tvinga den relativa transformationen att bevara H0 homologins dödstidsfördelningar. Dessutom spelar närvaron av liknande topologiska egenskaper en avgörande roll för att uppnå högre modellkompatibilitet. Dock krävs en mer ingående utforskning av de geometriska egenskaperna hos det latenta utrymmet efter den relativa transformationen för att ytterligare förbättra Hofer's topologiska reglering. Sammanfattningsvis bidrar detta arbete till det framväxande området Topologisk Maskininlärning och ger värdefulla insikter för forskare inom ``transfer-inlärning'' och representationsinlärningsdomäner.

To enable progress towards egocentric agents capable of understanding everyday tasks specified in natural language, we propose a benchmark and a synthetic dataset called Egocentric Task Verification (EgoTV). The goal in EgoTV is to verify the execution of tasks from egocentric videos based on the natural language description of these tasks. EgoTV contains pairs of videos and their task descriptions for multi-step tasks -- these tasks contain multiple sub-task decompositions, state changes, object interactions, and sub-task ordering constraints. In addition, EgoTV also provides abstracted task descriptions that contain only partial details about ways to accomplish a task. Consequently, EgoTV requires causal, temporal, and compositional reasoning of video and language modalities, which is missing in existing datasets. We also find that existing vision-language models struggle at such all-round reasoning needed for task verification in EgoTV Inspired by the needs of EgoTV, we propose a novel Neuro-Symbolic Grounding (NSG) approach that leverages symbolic representations to capture the compositional and temporal structure of tasks. We demonstrate NSG's capability towards task tracking and verification on our EgoTV dataset and a real-world dataset derived from CrossTask (CTV). We open-source the EgoTV and CTV datasets and the NSG model for future research on egocentric assistive agents.

In this paper, we study the multi-robot task assignment and path-finding problem (MRTAPF), where a number of robots are required to visit all given tasks while avoiding collisions with each other. We propose a novel two-layer algorithm SA-reCBS that cascades the simulated annealing algorithm and conflict-based search to solve this problem. Compared to other approaches in the field of MRTAPF, the advantage of SA-reCBS is that without requiring a pre-bundle of tasks to groups with the same number of groups as the number of robots, it enables a part of robots needed to visit all tasks in collision-free paths. We test the algorithm in various simulation instances and compare it with state-of-the-art algorithms. The result shows that SA-reCBS has a better performance with a higher success rate, less computational time, and better objective values., Full text license: CC BY-NC-ND

Normal Distribution Transformation (NDT) registration is a fast, learning-free point cloud registration algorithm that works well in diverse environments. It uses the compact NDT representation to represent point clouds or maps as a spatial probability function that models the occupancy likelihood in an environment. However, because of the grid discretization in NDT maps, the global minima of the registration cost function do not always correlate to ground truth, particularly for rotational alignment. In this study, we examined the NDT registration cost function in-depth. We evaluated three modifications (Student-t likelihood function, inflated covariance/heavily broadened likelihood curve, and overlapping grid cells) that aim to reduce the negative impact of discretization in classical NDT registration. The first NDT modification improves likelihood estimates for matching the distributions of small population sizes; the second modification reduces discretization artifacts by broadening the likelihood tails through covariance inflation; and the third modification achieves continuity by creating the NDT representations with overlapping grid cells (without increasing the total number of cells). We used the Pomerleau Dataset evaluation protocol for our experiments and found significant improvements compared to the classic NDT D2D registration approach (27.7% success rate) using the registration cost functions "heavily broadened likelihood NDT" (HBL-NDT) (34.7% success rate) and "overlapping grid cells NDT" (OGC-NDT) (33.5% success rate). However, we could not observe a consistent improvement using the Student-t likelihood-based registration cost function (22.2% success rate) over the NDT P2D registration cost function (23.7% success rate). A comparative analysis with other state-of-art registration algorithms is also presented in this work. We found that HBL-NDT worked best for easy initial pose difficulties scenarios making it suitable for consecutive point cloud regi

Autonomous systems, that need to operate in human environments and interact with the users, rely on understanding and anticipating human activity and motion. Among the many factors which influence human motion, semantic attributes, such as the roles and ongoing activities of the detected people, provide a powerful cue on their future motion, actions, and intentions. In this work we adapt several popular deep learning models for trajectory prediction with labels corresponding to the roles of the people. To this end we use the novel THOR-Magni dataset, which captures human activity in industrial settings and includes the relevant semantic labels for people who navigate complex environments, interact with objects and robots, work alone and in groups. In qualitative and quantitative experiments we show that the role-conditioned LSTM, Transformer, GAN and VAE methods can effectively incorporate the semantic categories, better capture the underlying input distribution and therefore produce more accurate motion predictions in terms of Top-K ADE/FDE and log-likelihood metrics.

Focusing on failure to improve human-robot interactions represents a novel approach that calls into question human expectations of robots, as well as posing ethical and methodological challenges to researchers. Fictional representations of robots (still for many non-expert users the primary source of expectations and assumptions about robots) often emphasize the ways in which robots surpass/perfect humans, rather than portraying them as fallible. Thus, to encounter robots that come too close, drop items or stop suddenly starts to close the gap between fiction and reality. These kinds of failures - if mitigated by explanation or recovery procedures - have the potential to make the robot a little more relatable and human-like. However, studying failures in human-robot interaction requires producing potentially difficult or uncomfortable interactions in which robots failing to behave as expected may seem counterintuitive and unethical. In this space, interdisciplinary conversations are the key to untangling the multiple challenges and bringing themes of power and context into view. In this workshop, we invite researchers from across the disciplines to an interactive, interdisciplinary discussion around failure in social robotics. Topics for discussion include (but are not limited to) methodological and ethical challenges around studying failure in HRI, epistemological gaps in defining and understanding failure in HRI, sociocultural expectations around failure and users' responses.

Failures are natural and unavoidable events in any form of interaction, especially in human-robot interactions (HRI). Throughout the literature, the definition and classification of failures are diverse, depending on the source and application domain. However, the tolerance to the aftereffect of these failures is low in teleoperation due to its unstructured application domains. One such type of failure is called human induced interaction failure. This is an interesting and often overlooked failure type, due to the perspective that robots are designed always to obey the instructions given by the human operators. Regardless of the degree of automation that the robot is equipped with. But what if the instructions provided are faulty, dangerous, or misleading. This paper addresses the above mentioned research gap. It introduces a framework based on the concept of Intelligent Disobedience (ID), derived from guide dog training methods, to manage human induced interaction failures in teleoperation scenarios.

In the context of robotics, accurate ground-truth positioning is the cornerstone for the development of mapping and localization algorithms. In outdoor environments and over long distances, total stations provide accurate and precise measurements, that are unaffected by the usual factors that deteriorate the accuracy of Global Navigation Satellite System (GNSS). While a single robotic total station can track the position of a target in three Degrees Of Freedom (DOF), three robotic total stations and three targets are necessary to yield the full six DOF pose reference. Since it is crucial to express the position of targets in a common coordinate frame, we present a novel extrinsic calibration method of multiple robotic total stations with field deployment in mind. The proposed method does not require the manual collection of ground control points during the system setup, nor does it require tedious synchronous measurement on each robotic total station. Based on extensive experimental work, we compare our approach to the classical extrinsic calibration methods used in geomatics for surveying and demonstrate that our approach brings substantial time savings during the deployment. Tested on more than 30 km of trajectories, our new method increases the precision of the extrinsic calibration by 25 % compared to the best state-of-the-art method, which is the one taking manually static ground control points., This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the grant CRDPJ 527642-18 SNOW (Self-driving Navigation Optimized for Winter).

Burst image processing is becoming increasingly popular in recent years. However, it is a challenging task since individual burst images undergo multiple degradations and often have mutual misalignments resulting in ghosting and zipper artifacts. Existing burst restoration methods usually do not consider the mutual correlation and non-local contextual information among burst frames, which tends to limit these approaches in challenging cases. Another key challenge lies in the robust up-sampling of burst frames. The existing up-sampling methods cannot effectively utilize the advantages of single-stage and progressive up-sampling strategies with conventional and/or recent up-samplers at the same time. To address these challenges, we propose a novel Gated Multi-Resolution Transfer Network (GMTNet) to reconstruct a spatially precise high-quality image from a burst of low-quality raw images. GMT-Net consists of three modules optimized for burst processing tasks: Multi-scale Burst Feature Alignment (MBFA) for feature denoising and alignment, Transposed-Attention Feature Merging (TAFM) for multi-frame feature aggregation, and Resolution Transfer Feature Up-sampler (RTFU) to up-scale merged features and construct a high-quality output image. Detailed experimental analysis on five datasets validate our approach and sets a state-of-the-art for burst super-resolution, burst denoising, and low-light burst enhancement. Our codes and models are available at https://github.com/nanmehta/GMTNet.

Feature matching is a challenging computer vision task that involves finding correspondences between two images of a 3D scene. In this paper we consider the dense approach instead of the more common sparse paradigm, thus striving to find all correspondences. Perhaps counter-intuitively, dense methods have previously shown inferior performance to their sparse and semi-sparse counterparts for estimation of two-view geometry. This changes with our novel dense method, which outperforms both dense and sparse methods on geometry estimation. The novelty is threefold: First, we propose a kernel regression global matcher. Secondly, we propose warp refinement through stacked feature maps and depthwise convolution kernels. Thirdly, we propose learning dense confidence through consistent depth and a balanced sampling approach for dense confidence maps. Through extensive experiments we confirm that our proposed dense method, Dense Kernelized Feature Matching, sets a new state-of-the-art on multiple geometry estimation benchmarks. In particular, we achieve an improvement on MegaDepth-1500 of +4.9 and +8.9 AUC@5° compared to the best previous sparse method and dense method respectively. Our code is provided at the following repository: https://github.com/Parskatt/DKM., This work was supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Foundation; andby the strategic research environment ELLIIT funded by the Swedish government. The computational resources were provided by the National Academic Infrastructure forSupercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by the Berzelius resource, provided bythe Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.

Pre-trained vision-language (V-L) models such as CLIP have shown excellent generalization ability to downstream tasks. However, they are sensitive to the choice of input text prompts and require careful selection of prompt templates to perform well. Inspired by the Natural Language Processing (NLP) literature, recent CLIP adaptation approaches learn prompts as the textual inputs to ne-tune CLIP for downstream tasks. We note that using prompting to adapt representations in a single branch of CLIP (language or vision) is sub-optimal since it does not allow the exibility to dynamically adjust both representation spaces on a downstream task. In this work, we propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Our design promotes strong coupling between the vision-language prompts to ensure mutual synergy and discourages learning independent uni-modal solutions. Further, we learn separate prompts across different early stages to progressively model the stage-wise feature relationships to allow rich context learning. We evaluate the effectiveness of our approach on three representative tasks of generalization to novel classes, new target datasets and unseen domain shifts. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes and 2.72% on overall harmonic-mean, averaged over 11 diverse image recognition datasets. Our code and pre-trained models are available at https://github.com/muzairkhattak/multimodal-prompt-learning.

Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Fine-tuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models will be released at https://github.com/TalalWasim/Vita-CLIP..

On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoodsf or information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed reference-based feature enrichment mechanism, which facilitates handling complex motions. Aft er multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-of-the-art methods on burst super-resolution, burst denoising and burst low-light enhancement. Our codes and pre-trained models are available at https://github.com/akshaydudhane16/Burstormer.

Most previous co-salient object detection works mainly focus on extracting co-salient cues via mining the consistency relations across images while ignore explicit exploration of background regions. In this paper, we propose a Discriminative co-saliency and background Mining Transformer framework (DMT) based on several economical multi-grained correlation modules to explicitly mine both co-saliency and background information and effectively model their discrimination. Specifically, we first propose a region-to-region correlation module for introducing inter-image relations to pixel-wise segmentation features while maintaining computational efficiency. Then, we use two types of pre-defined tokens to mine co-saliency and background information via our proposed contrast-induced pixel-to-token correlation and co-saliency token-to-token correlation modules. We also design a token-guided feature refinement module to enhance the discriminability of the segmentation features under the guidance of the learned tokens. We perform iterative mutual promotion for the segmentation feature extraction and token construction. Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method. The source code is available at: https://github.com/dragonlee258079/DMT., Funding Agencies|Key-Area Research and Development Program of Guangdong Province [2021B0101200001]; National Key R&D Program of China [2021B0101200001]; National Science Foundation of China [62036011, U20B2065, 721A0001, 62136004]

The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for high-fidelity person image synthesis with strong sample diversity and enhanced mode coverage of the learnt data distribution. Our proposed Person Image Diffusion Model (PIDM) disintegrates the complex transfer problem into a series of simpler forward-backward denoising steps. This helps in learning plausible source-to-target transformation trajectories that result in faithful textures and undistorted appearance details. We introduce a texture diffusion module based on cross-attention to accurately model the correspondences between appearance and pose information available in source and target images. Further, we propose disentangled classifier-free guidance to ensure close resemblance between the conditional inputs and the synthesized output in terms of both pose and appearance information. Our extensive results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios. We also show how our generated images can help in downstream tasks. Code is available at https://github.com/ankanbhunia/PIDM.

Transformer models have achieved outstanding results on a variety of language tasks, such as text classification, ma- chine translation, and question answering. This success in the field of Natural Language Processing (NLP) has sparked interest in the computer vision community to apply these models to vision and multi-modal learning tasks. However, visual data has a unique structure, requiring the need to rethink network designs and training methods. As a result, Transformer models and their variations have been suc- cessfully used for image recognition, object detection, seg- mentation, image super-resolution, video understanding, image generation, text-image synthesis, and visual question answering, among other applications.

Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the framelevel processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a bridge and prompt approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks. Our code and pre-trained models are available at https://github.com/muzairkhattak/ViFi-CLIP.

Recent advances in 3D-aware generative models (3D-aware GANs) combined with Neural Radiance Fields (NeRF) have achieved impressive results. However no prior works investigate 3D-aware GANs for 3D consistent multi-class image-to-image (3D-aware I2I) translation. Naively using 2D-I2I translation methods suffers from unrealistic shape/identity change. To perform 3D-aware multi-class I2I translation, we decouple this learning process into a multi-class 3D-aware GAN step and a 3D-aware I2I translation step. In the first step, we propose two novel techniques: a new conditional architecture and an effective training strategy. In the second step, based on the well-trained multi-class 3D-aware GAN architecture, that preserves view-consistency, we construct a 3D-aware I2I translation system. To further reduce the view-consistency problems, we propose several new techniques, including a U-net-like adaptor network design, a hierarchical representation constrain and a relative regularization loss. In extensive experiments on two datasets, quantitative and qualitative results demonstrate that we successfully perform 3D-aware I2I translation with multi-view consistency. Code is available in 3DI2I., Funding Agencies|Key Laboratory of Advanced Information Science and Network Technology of Beijing [XDXX2202]; Youth Foundation [62202243]; Spanish Government [PID2019-104174GB-I00, TED2021-132513B-I00]

Although existing semi-supervised learning models achieve remarkable success in learning with unannotated in-distribution data, they mostly fail to learn on unlabeled data sampled from novel semantic classes due to their closed-set assumption. In this work, we target a pragmatic but under-explored Generalized Novel Category Discovery (GNCD) setting. The GNCD setting aims to categorize unlabeled training data coming from known and novel classes by leveraging the information of partially labeled known classes. We propose a two-stage Contrastive Affinity Learning method with auxiliary visual Prompts, dubbed PromptCAL, to address this challenging problem. Our approach discovers reliable pairwise sample affinities to learn better semantic clustering of both known and novel classes for the class token and visual prompts. First, we propose a discriminative prompt regularization loss to reinforce semantic discriminativeness of prompt-adapted pre-trained vision transformer for refined affinity relationships. Besides, we propose contrastive affinity learning to calibrate semantic representations based on our iterative semi-supervised affinity graph generation method for semantically-enhanced supervision. Extensive experimental evaluation demonstrates that our PromptCAL method is more effective in discovering novel classes even with limited annotations and surpasses the current state-of-the-art on generic and fine-grained benchmarks (e.g., with nearly 11% gain on CUB-200, and 9% on ImageNet-100) on overall accuracy. Our code is available at https: // github.com/ sheng- eatamath / PromptCAL.

Ensembles of independently trained deep neural networks yield uncertainty estimates that rival Bayesian networks in performance. They also offer sizable improvements in terms of predictive performance over single models. However, deep ensembles are not commonly used in environments with limited computational budget - such as autonomous driving - since the complexity grows linearly with the number of ensemble members. An important observation that can be made for robotics applications, such as autonomous driving, is that data is typically sequential. For instance, when an object is to be recognized, an autonomous vehicle typically observes a sequence of images, rather than a single image. This raises the question, could the deep ensemble be spread over time? In this work, we propose and analyze Deep Ensembles Spread Over Time (DESOT). The idea is to apply only a single ensemble member to each data point in the sequence, and fuse the predictions over a sequence of data points. We implement and experiment with DESOT for traffic sign classification, where sequences of tracked image patches are to be classified. We find that DESOT obtains the benefits of deep ensembles, in terms of predictive and uncertainty estimation performance, while avoiding the added computational cost. Moreover, DESOT is simple to implement and does not require sequences during training. Finally, we find that DESOT, like deep ensembles, outperform single models for out-of-distribution detection.

Traditionally, monocular 3D human pose estimation employs a machine learning model to predict the most likely 3D pose for a given input image. However, a single image can be highly ambiguous and induces multiple plausible solutions for the 2D-3D lifting step, which results in overly confident 3D pose predictors. To this end, we propose DiffPose, a conditional diffusion model that predicts multiple hypotheses for a given input image. Compared to similar approaches, our diffusion model is straightforward and avoids intensive hyperparameter tuning, complex network structures, mode collapse, and unstable training. Moreover, we tackle the problem of over-simplification of the intermediate representation of the common two-step approaches which first estimate a distribution of 2D joint locations via joint-wise heatmaps and consecutively use their maximum argument for the 3D pose estimation step. Since such a simplification of the heatmaps removes valid information about possibly correct, though labeled unlikely, joint locations, we propose to represent the heatmaps as a set of 2D joint candidate samples. To extract information about the original distribution from these samples, we introduce our embedding transformer which conditions the diffusion model. Experimentally, we show that DiffPose improves upon the state of the art for multi-hypothesis pose estimation by 3-5% for simple poses and outperforms it by a large margin for highly ambiguous poses.

A deep neural classifier trained on an upstream task can be leveraged to boost the performance of another classifier in a related downstream task through the representations learned in hidden layers. However, presence of shortcuts (easy-to-learn features) in the upstream task can considerably impair the versatility of intermediate representations and, in turn, the downstream performance. In this paper, we propose a method to improve the representations learned by deep neural image classifiers in spite of a shortcut in upstream data. In our method, the upstream classification objective is augmented with a type of adversarial training where an auxiliary network, so called lens, fools the classifier by exploiting the shortcut in reconstructing images. Empirical comparisons in self-supervised and transfer learning problems with three shortcut-biased datasets suggest the advantages of our method in terms of downstream performance and/or training time.

Images fed to a deep neural network have in general undergone several handcrafted image signal processing (ISP) operations, all of which have been optimized to produce visually pleasing images. In this work, we investigate the hypothesis that the intermediate representation of visually pleasing images is sub-optimal for downstream computer vision tasks compared to the RAW image representation. We suggest that the operations of the ISP instead should be optimized towards the end task, by learning the parameters of the operations jointly during training. We extend previous works on this topic and propose a new learnable operation that enables an object detector to achieve superior performance when compared to both previous works and traditional RGB images. In experiments on the open PASCALRAW dataset, we empirically confirm our hypothesis.

A challenge in image based metrology and forensics is intrinsic camera calibration when the used camera is unavailable. The unavailability raises two questions. The first question is how to find the projection model that describes the camera, and the second is to detect incorrect models. In this work, we use off-the-shelf extended PnP-methods to find the model from 2D-3D correspondences, and propose a method for model validation. The most common strategy for evaluating a projection model is comparing different models’ residual variances—however, this naive strategy cannot distinguish whether the projection model is potentially underfitted or overfitted. To this end, we model the residual errors for each correspondence, individually scale all residuals using a predicted variance and test if the new residuals are drawn from a standard normal distribution. We demonstrate the effectiveness of our proposed validation in experiments on synthetic data, simulating 2D detection and Lidar measurements. Additionally, we provide experiments using data from an actual scene and compare non-camera access and camera access calibrations. Last, we use our method to validate annotations in MegaDepth., Funding agencies: This work was partially supported by the Wallenberg AI, Autonomous Systems, and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation; and the computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre; and a point cloud of a realistic scene was provided by the Swedish National Forensic Centre (NFC).

Dandelion Optimizer (DO) is a recently proposed swarm intelligence algorithm that coincides with the process of finding the best reproduction site for dandelion seeds. Compared with the classical Meta-heuristic algorithms, DO exhibits strong competitiveness, but it also has some drawbacks. In this paper, we proposed an adaptive hybrid dandelion optimizer called DETDO by combining three strategies of adaptive tent chaotic mapping, differential evolution (DE) strategy, and adaptive t-distribution perturbation to address the shortcomings of weak DO development, easy to fall into local optimum and slow convergence speed. Firstly, the adaptive tent chaos mapping is used in the initialization phase to obtain a uniformly distributed high-quality initial population, which helps the algorithm to enter the correct search region quickly. Secondly, the DE strategy is introduced to increase the diversity of dandelion populations to avoid algorithm stagnation, which improves the exploitation capability and the accuracy of the optimal solution. Finally, adaptive t-distribution perturbation around the elite solution successfully balances the exploration and exploitation phases while improving the convergence speed through a reasonable conversion from Cauchy to Gaussian distribution. The proposed DETDO is compared with classical or advanced optimization algorithms on CEC2017 and CEC2019 test sets, and the experimental results and statistical analysis demonstrate that the algorithm has better optimization accuracy and speed. In addition, DETDO has obtained the best results in solving six real-world engineering design problems. Finally, DETDO is applied to two bar topology optimization cases. Under a series of complex constraints, DETDO produces a lighter bar structure than the current scheme. It further illustrates the effectiveness and applicability of DETDO in practical problems. The above results mean that DETDO with strong competitiveness will become a preferred swarm intelligenc, Funding Agencies|National Natural Science Foundation of China [51875454]

In unmanned aerial vehicle photographs, object detection algorithms encounter challenges in enhancing both speed and accuracy for objects of different sizes, primarily due to complex backgrounds and small objects. This study introduces the PDWT-YOLO algorithm, based on the YOLOv7-tiny model, to improve the effectiveness of object detection across all sizes. The proposed method enhances the detection of small objects by incorporating a dedicated small-object detection layer, while reducing the conflict between classification and regression tasks through the replacement of the YOLOv7-tiny model’s detection head (IDetect) with a decoupled head. Moreover, network convergence is accelerated, and regression accuracy is improved by replacing the Complete Intersection over Union (CIoU) loss function with a Wise Intersection over Union (WIoU) focusing mechanism in the loss function. To assess the proposed model’s effectiveness, it was trained and tested on the VisDrone-2019 dataset comprising images captured by various drones across diverse scenarios, weather conditions, and lighting conditions. The experiments show that mAP@0.5:0.95 and mAP@0.5 increased by 5% and 6.7%, respectively, with acceptable running speed compared with the original YOLOv7-tiny model. Furthermore, this method shows improvement over other datasets, confirming that PDWT-YOLO is effective for multiscale object detection.

Reliable detection and classification of bacteria and other pathogens in the human body, animals, food, and water is crucial for improving and safeguarding public health. For instance, identifying the species and its antibiotic susceptibility is vital for effective bacterial infection treatment. Here we show that phase contrast time-lapse microscopy combined with deep learning is sufficient to classify four species of bacteria relevant to human health. The classification is performed on living bacteria and does not require fixation or staining, meaning that the bacterial species can be determined as the bacteria reproduce in a microfluidic device, enabling parallel determination of susceptibility to antibiotics. We assess the performance of convolutional neural networks and vision transformers, where the best model attained a class-average accuracy exceeding 98%. Our successful proof-of-principle results suggest that the methods should be challenged with data covering more species and clinically relevant isolates for future clinical use. Bacterial infections are a leading cause of premature death worldwide, and growing antibiotic resistance is making treatment increasingly challenging. To effectively treat a patient with a bacterial infection, it is essential to quickly detect and identify the bacterial species and determine its susceptibility to different antibiotics. Prompt and effective treatment is crucial for the patient's survival. A microfluidic device functions as a miniature "lab-on-chip" for manipulating and analyzing tiny amounts of fluids, such as blood or urine samples from patients. Microfluidic chips with chambers and channels have been designed for quickly testing bacterial susceptibility to different antibiotics by analyzing bacterial growth. Identifying bacterial species has previously relied on killing the bacteria and applying species-specific fluorescent probes. The purpose of the herein proposed species identification is to speed up decisions o

4D Imaging Radar is increasingly becoming a critical component in various industries due to beamforming technology and hardware advancements. However, it does not replace visual data in the form of 2D images captured by an RGB camera. Instead, 4D radar point clouds are a complementary data source that captures spatial information and velocity in a Doppler dimension that cannot be easily captured by a camera's view alone. Some discriminative features of the scene captured by the two sensors are hypothesized to have a shared representation. Therefore, a more interpretable visualization of the radar output can be obtained by learning a mapping from the empirical distribution of the radar to the distribution of images captured by the camera. To this end, the application of deep generative models to generate images conditioned on 4D radar data is explored. Two approaches that have become state-of-the-art in recent years are tested, generative adversarial networks and diffusion models. They are compared qualitatively through visual inspection and by two quantitative metrics: mean squared error and object detection count. It is found that it is easier to control the generative adversarial network's generative process through conditioning than in a diffusion process. In contrast, the diffusion model produces samples of higher quality and is more stable to train. Furthermore, their combination results in a hybrid sampling method, achieving the best results while simultaneously speeding up the diffusion process., 4D bildradar får en alltmer betydande roll i olika industrier tack vare utveckling inom strålformningsteknik och hårdvara. Det ersätter dock inte visuell data i form av 2D-bilder som fångats av en RGB-kamera. Istället utgör 4D radar-punktmoln en kompletterande datakälla som representerar spatial information och hastighet i form av en Doppler-dimension. Det antas att vissa beskrivande egenskaper i den observerade miljön har en abstrakt representation som de två sensorerna delar. Därmed kan radar-datan visualiseras mer intuitivt genom att lära en transformation från fördelningen över radar-datan till fördelningen över bilderna. I detta syfte utforskas tillämpningen av djupa generativa modeller för bilder som är betingade av 4D radar-data. Två metoder som har blivit state-of-the-art de senaste åren testas: generativa antagonistiska nätverk och diffusionsmodeller. De jämförs kvalitativt genom visuell inspektion och med kvantitativa metriker: medelkvadratfelet och antalet korrekt detekterade objekt i den genererade bilden. Det konstateras att det är lättare att styra den generativa processen i generativa antagonistiska nätverk genom betingning än i en diffusionsprocess. Å andra sidan är diffusionsmodellen stabil att träna och producerar generellt bilder av högre kvalité. De bästa resultaten erhålls genom en hybrid: båda metoderna kombineras för att dra nytta av deras respektive styrkor. de identifierade begränsningarna i de enskilda modellerna och kurera datan för att jämföra hur dessa modeller skalar med större datamängder och mer variation.

In machine learning, a sub-field of computer science, a two-tower architecture model is a specialised type of neural network model that encodes paired data from different modalities (like text and images, sound and video, or proteomics and gene expression profiles) into a shared latent representation space. However, when training these models using a specific contrastive loss function, known as the multimodalinfoNCE loss, seems to often lead to a unique geometric phenomenon known as the modality gap. This gap is a clear geometric separation of the embeddings of the modalities in the joint contrastive latent space. This thesis investigates the modality gap in multimodal machine learning, specifically in two-tower neural networks trained with multimodal-infoNCE loss. We examine the adequacy of the current definition of the modality gap, the conditions under which the modality gap phenomenon manifests, and its impact on representation quality and downstream task performance. The approach to address these questions consists of a two-phase experimental strategy. Phase I involves a series of experiments, ranging from toy synthetic simulations to true multimodal machine learning with complex datasets, to explore and characterise the modality gap under varying conditions. Phase II focuses on modifying the modality gap and analysing representation quality, evaluating different loss functions and their impact on the modality gap. This methodical exploration allows us to systematically dissect the emergence and implications of the modality gap phenomenon, providing insights into its impact on downstream tasks, measured with proxy metrics based on semantic clustering in the shared latent representation space and modality-specific linear probe evaluation. Our findings reveal that the modality gap definition proposed by W. Liang et al. 2022, is insufficient. We demonstrate that similar modality gap magnitudes can exhibit varying linear separability between modality embeddings in

3D object detection is the task of detecting the full 3D pose of objects relative to an autonomous platform. It is an important perception system that can be used to plan actions according to the behavior of other dynamic objects in an environment. Due to the poor generalization of object detectors trained and tested on different datasets, this thesis concerns the utilization of unsupervised domain adaptation to train object detectors fit for mobile robotics without any labeled training data. To tackle the problem a novel approach Unsupervised Adversarial Domain Adaptation 3D (UADA3D) is presented to adapt LiDAR-based detectors, through drawing inspiration from the success of adversarial adaptation for 2D object detection in RGB images. The method adds learnable discriminator layers that discriminate between the features and bounding box predictions in the labeled source and unlabeled target data. The gradients are then reversed through gradient reversal layers during backpropagation to the base detector, which in turn learns to extract features that are similar between the domains in order to fool the discriminator. The method works for multi-class detection by simultaneous adaptation of all classes in an end-to-end trainable network and works for both point-based and voxel-based single-stage detectors. The results show that the proposed method increases detection scores for adaptation from dense to sparse point clouds and from simulated data toward the data of a mobile delivery robot, successfully handling the two relevant domain gaps given by differences in marginal and conditional probability distributions., 3D-objektdetektering handlar om att upptäcka hela 3D-positionen för objekt i förhållande till en autonom plattform. Det är ett viktigt perceptionsystem som kan användas för att planera åtgärder baserat på beteendet hos andra dynamiska objekt i en miljö. På grund av den dåliga generaliseringen av objektavkännare som tränats och testats på olika datamängder, handlar denna avhandling om användningen av osuperviserad domänanpassning för att träna objektavkännare som är anpassade för mobila robotar utan några märkta träningsdata. För att tackla problemet presenteras ett nytt tillvägagångssätt Unsupervised Adversarial Domain Adaptation 3D (UADA3D) för att anpassa LiDAR-baserade avkännare, genom att ta inspiration från framgången av mospelaranpassning för 2D-objektdetektering i RGB-bilder. Metoden lägger till inlärbara diskriminatorlager som diskriminerar mellan egenskaperna och prediktionerna i annoterad käll- och oannoterad måldata. Gradienterna är sedan reverserae genom gradientreversering under bakåtpropagering till basdetekorn, som i sin tur lär sig att extrahera egenskaper som är liknande mellan domänerna för att lura diskriminatorn. Metoden fungerar för flerklassdetektering genom samtidig anpassning av alla klasser i ett end-to-end-träningsbart nätverk och fungerar för både punktbaserade och voxelbaserade enstegs detektorere. Resultaten visar att den föreslagna metoden förbättrar detektionen för domänanpassning från täta till glesa punktmoln och från simulerad data till data från en mobil leveransrobot, därmed hanterar metoden framgångsrikt de två relevanta domänskillnaderna i marginella- och betingade sannolikhetsfördelningar.

The computer vision field has taken big steps forwards and the amount of models and datasets that are being released is increasing. A large number of contemporary models are the result of extensive training sessions on massive datasets, reflecting a significant investment of time and computational resources. This opens up a new opportunity on utilizing the knowledge from this pre-trained models. It is possible to transfer the knowledge from one domain to a more fine-tuned solution on a custom created dataset, and this can help the field of computer vision to improve rapidly. This project utilizes the pre-trained models ResNet50,ResNet18 and DensNet121, for dealing with the challenge of fine-tuning models on a custom created dataset, that is created of grayscale images. The project’s results show how it’s possible to use a pre-trained model for transferring the learned features from one domain to another. In addition to this, the project included creating a binary classifier that is fine-tuned on a balanced dataset and another classifier that was fine-tuned on an imbalanced dataset., Området för datorseende har gjort stora framsteg senaste årtionden och mängden modeller för djup maskininlärning som har tagits fram och blivit tillgängliga har ökat, tillsammans med mängden publika dataset som har blivit skapade. Nya modeller som når state-of-the-art status är tränade på stora dataset och med mycket beräkningskraft och har blivit framtagna under lång tid. Det har skapat ett nytt område inom datorseendekallat överföringsinlärning, vilket innebär att dra nytta av en modell som är tränad på ett dataset som inte är det exakta dataset som du har samlat ihop. Detta används sedan för att träna nätverket med dina specifika bilder, och lagerna i nätverket kan anpassas för att angripa det specifika problemet. Detta skapar möjligheter att på kortare framtagnings-tid dra nytta av kunskapen modellen har lärt sig och du tränar den bara på dina bilder för att anpassa nätverket. I det här projektet används de tränade modellerna ResNet50,ResNet18 och DenseNet121, och deras kunskap från tidigare träning används för att klassificera det här projektets specifika dataset med svartvita bilder. Projektets resultat visar möjligheten att använda en för tränad modell och applicera den på ett avgränsat, specifikt problem som skiljer sig från ursprungs-uppgiften. I projektet visas resultaten av att skapa ett dataset av bilder samt en binär klassificering och en klassificering av flertalet klasser.

Simultaneous Localization And Mapping (SLAM) is an important topic within the field of roboticsaiming to localize an agent in a unknown or partially known environment while simultaneouslymapping the environment. The ability to perform robust SLAM is especially important inhazardous environments such as natural disasters, firefighting and space exploration wherehuman exploration may be too dangerous or impractical. In recent years, neuromorphiccameras have been made commercially available. This new type of sensor does not outputconventional frames but instead an asynchronous signal of events at a microsecond resolutionand is capable of capturing details in complex lightning scenarios where a standard camerawould be either under- or overexposed, making neuromorphic cameras a promising solution insituations where standard cameras struggle. This thesis explores a set of different approachesto virtual frames, a frame-based representation of events, in the context of SLAM.UltimateSLAM, a project fusing events, gray scale and IMU data, is investigated using virtualframes of fixed and varying frame rate both with and without motion compensation. Theresulting trajectories are compared to the trajectories produced when using gray scale framesand the number of detected and tracked features are compared. We also use a traditional visualSLAM project, ORB-SLAM, to investigate the Gaussian weighted virtual frames and gray scaleframes reconstructed from the event stream using a recurrent network model. While virtualframes can be used for SLAM, the event camera is not a plug and play sensor and requires agood choice of parameters when constructing virtual frames, relying on pre-existing knowledgeof the scene.

As the demand for 3D maps from LIDAR scanners increases, delivering high-quality maps becomes critical. One way to ensure the quality of such maps is through point cloud alignment classification, which aims to classify the alignment error between two registered point clouds. Specifically, we present the classifier FACT (Feature-Aware Classification Transformer), consisting of two main modules: feature extraction and classification. Descriptive features are extracted from the joint point cloud, which are then processed by a point transformer-based neural network to predict the alignment error class. In a ten-class point cloud alignment classification test, FACT achieved 92.4% accuracy, where the alignment error ranged from zero meters and radians to 0.9 meters and 0.09 radians. Remarkably, the classifier only made one misprediction beyond neighboring classes, exhibiting its ability to detect alignment errors as the classes have an inherent order. Furthermore, when benchmarked on two binary classification tasks, FACT showed significantly superior performance over the baseline and even obtained 100.0% accuracy for the easier of the two tasks. FACT not only detects potential errors in 3D maps but also estimates their magnitude, leading to more reliable 3D maps with quality estimations for each transformation.

This study explores the application of multi-agent reinforcement learning (MARL) to enhance the decision-making, safety, and passenger comfort of Autonomous Vehicles (AVs)at uncontrolled intersections. The research aims to assess the potential of MARL in modeling multiple agents interacting within a shared environment, reflecting real-world situations where AVs interact with multiple actors. The findings suggest that AVs trained using aMARL approach with global experiences can better navigate intersection scenarios than AVs trained on local (individual) experiences. This capability is a critical precursor to achieving Level 5 autonomy, where vehicles are expected to manage all aspects of the driving task under all conditions. The research contributes to the ongoing discourse on enhancing autonomous vehicle technology through multi-agent reinforcement learning and informs the development of sophisticated training methodologies for autonomous driving.

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on three datasets with different organs and modalities, where it substantially outperforms existing techniques. Our code is available at: https://github.com/histocartography/generative-appearance-replay.Previous article in issue

Inverse problems in imaging consider the reconstruction of clean images from degraded observations, like deblurring or inpainting.These inverse problems are generally ill-posed. Solving them therefore requires regularization, which exists in multiple approaches: plug-and-play (pnp) methods are designed to generically solve any inverse problem by replacing a regularizing proximal operator with a denoiser. Unrolled methods perform a fixed number of iterations and train a network end-to-end for a specific degradation, but necessitate re-training for each specific degradation. Deep equilibrium models (DEQs), on the other hand, iterate an unrolled method until convergence and thereby enable end-to-end training on the reconstruction error with simplified back-propagation. We have investigated to what extent a solution for several inverse problems can be found by employing a multi-task DEQ (MTDEQ). This MTDEQ is used to train a prior on the actual estimation error, in contrast to a theoretical noise model used for pnp methods. This has the advantage that the resulting prior is trained for a range of degradations beyond pure Gaussian denoising. The investigation also demonstrates that different search methods can be used in training (forward-backward) and in testing (alternating direction method of multipliers)., PLENOPTIMA

Realistic modelling of pedestrians in Autonomous Vehicles (AV)s and AV testing is crucial to avoid lethal collisions in deployment. The majority of AV trajectory forecasting literature do not utilize the motion cues present in 3D human pose because it is hard to gather large datasets of articulated 3D pedestrian motion. In this thesis we discuss the difficulties in data gathering and propose a pedestrian model that overcomes the issues by utilizing various datasets and learning paradigms to learn articulated semantically reasoning pedestrian motion. We show that such learnt pedestrian models can and should be utilized in AV testing, instead of heuristics as in previous work, to test AVs on realistic and hard scenarios. We propose a framework for generating varied AV test scenarios by posing AV test case generation as a visual problem. Finally we provide a method to improve existing articulated human pose forecasting by utilizing individual specific motion cues on the fly. This thesis discusses the difficulties in articulated pedestrian sensing, proposes a pedestrian model to overcome these difficulties showing a direct use of the pedestrian model in AV testing, and shows the possible further improvements to articulated pedestrian motion forecasting should articulated models be utilized in AV trajectory planning. We hope that this work aids in the further development of articulated and semantically reasoning pedestrian models in AV testing and trajectory planning.

Robust SLAM in large-scale environments requires fault resilience and awareness at multiple stages, from sensing and odometry estimation to loop closure. In this work, we present TBV (Trust But Verify) Radar SLAM, a method for radar SLAM that introspectively verifies loop closure candidates. TBV Radar SLAM achieves a high correct-loop-retrieval rate by combining multiple place-recognition techniques: tightly coupled place similarity and odometry uncertainty search, creating loop descriptors from origin-shifted scans, and delaying loop selection until after verification. Robustness to false constraints is achieved by carefully verifying and selecting the most likely ones from multiple loop constraints. Importantly, the verification and selection are carried out after registration when additional sources of loop evidence can easily be computed. We integrate our loop retrieval and verification method with a robust odometry pipeline within a pose graph framework. By evaluation on public benchmarks we found that TBV Radar SLAM achieves 65% lower error than the previous state of the art. We also show that it generalizes across environments without needing to change any parameters. We provide the open-source implementation at https://github.com/dan11003/tbv_slam_public

Convolutional neural network (CNN)-based methods have achieved state-of-the-art performance in solving several complex computer vision problems including assessment of diabetic retinopathy (DR). Despite this, CNN-based methods are often criticized as “black box” methods for providing limited to no understanding about their internal functioning. In recent years there has been an increased interest to develop explainable deep learning models, and this paper is an effort in that direction in the context of DR. Based on one of the best performing method called Grad-CAM++, we propose Advanced Grad-CAM++ to provide further improvement in visual explanations of CNN model predictions (when compared to Grad-CAM++), in terms of better localization of DR pathology as well as explaining occurrences of multiple DR pathology types in a fundus image. By keeping all the layers and operations as is, the proposed method adds an additional non-learnable bilateral convolutional layer between the input image and the very first learnable convolutional layer of Grad-CAM++. Experiments were conducted on fundus images collected from publicly available sources namely EyePACS and DIARETDB1. Intersection over Union (IoU) score between the ground truth and heatmap produced by the methods were used to quantitatively compare the performance.The overall IoU score for Advanced Grad-CAM++ is 0.179, whereas for Grad-CAM++ it is score 0.161. Thus an 11.18% improvement in agreement with the ground truths by the proposed method is inferable.

Diabetic foot ulcers (DFU) are one of the major health complications for people with diabetes. It may cause limb amputation or lead to life-threatening situations if not detected and treated properly at an early stage. A diabetic patient has a 15–25% chance of developing DFU at a later stage in his or her life if proper foot care is not taken. Because of these high-risk factors, patients with diabetes need to have regular checkups and medications which cause a huge financial burden on both the patients and their families. Hence, the necessity of a cost-effective, re-mote, and fitting DFU diagnosis technique is imminent. This paper presents a convolutional neural network (CNN)-based approach for the automated detection of diabetic foot ulcers from the pictures of a patient’s feet. ResNet50 is used as the backbone of the Faster R-CNN which performed better than the original Faster R-CNN that uses VGG16. A total of 2000 images from the Diabetic Foot Ulcer Grand Challenge 2020 (DFUC2020) dataset have been used for the experiment. The proposed method obtained precision, recall, F1-score, and mean average precision of 77.3%, 89.0%, 82.7%, and 71.3%, respectively, in DFU detection which is better than results obtained by the original Faster R-CNN.

A significant challenge in service robots is the semantic understanding of their surrounding areas. Traditional approaches addressed this problem by segmenting the environment into regions corresponding to full rooms that are assigned labels consistent with human perception, e.g. office or kitchen. However, different areas inside the same room can be used in different ways: Could the table and the chair in my kitchen become my office ? What is the category of that area now? office or kitchen? To adapt to these circumstances we propose a new paradigm where we intentionally relax the resulting labeling of place classifiers by allowing confusions, and by avoiding further filtering leading to clean full room classifications. Our hypothesis is that confusions can be beneficial to a service robot and, therefore, they can be kept and better exploited. Our approach creates a subdivision of the environment into different regions by maintaining the confusions which are due to the scene appearance or to the distribution of objects. In this paper, we present a proof of concept implemented in simulated and real scenarios, that improves efficiency in the robotic task of searching for objects by exploiting the confusions in place classifications., Funding agencies:Spanish Government RTI2018-095599-B-C21 RTI2018-095599-A-C22RoboCity2030-Madrid Robotics Digital Innovation Hub project, Spain S2018/NMT-4331

The increased popularity of Behavior Trees (BTs) in different fields of robotics requires efficient methods for learning BTs from data instead of tediously handcrafting them. Recent research in learning from demonstration reported encouraging results that this letter extends, improves and generalizes to arbitrary planning domains. We propose BT-Factor as a new method for learning expert knowledge by representing it in a BT. Execution traces of previously manually designed plans are used to generate a BT employing a combination of decision tree learning and logic factorization techniques originating from circuit design. We test BT-Factor in an industrially-relevant simulation environment from a mining scenario and compare it against a state-of-the-art BT learning method. The results show that our method generates compact BTs easy to interpret, and capable to capture accurately the relations that are implicit in the training data.

In this paper, we propose a system that stems from the integration of an autonomous mobile robot with an IoT-based monitoring system to provide monitoring, assistance, and stimulation to older adults living alone in their own houses. The creation of an Internet of Robotics Things (IoRT) based on the interplay between pervasive smart objects and autonomous robotic systems is claimed to enable the creation of innovative services conceived for assisting the final user, especially in elderly care. The synergy between IoT and a Socially Assistive Robot (SAR) was conceived to offer robustness, reconfiguration, heterogeneity, and scalability, by bringing a strong added value to both the current SAR and IoT technologies. First, we propose a method to achieve the synergy and integration between the IoT system and the robot; then, we show how our method increases the performance and effectiveness of both to provide long-term support to the older adults. To do so, we present a case-study, where we focus on the detection of signs of the frailty syndrome, a set of vulnerabilities typically conveyed by a cognitive and physical decline in older people that concur in amplifying the risks of major diseases hindering the capabilities of independent living. Experimental evaluation is performed in both controlled settings and in a long-term real-world pilot study with 9 older adults in their own apartments, where the system was deployed autonomously for, on average, 12 weeks., Funding agency:PON project SI-ROBOTICS - Italian government

Multi-robot motion coordination is essential to make robots work safely and efficiently in a shared workspace, ensuring task completion while avoiding interference between each other's motions. Achieving online replanning and scaling to large numbers of robots are particularly challenging. In this paper, we present a motion coordination method for multiple robot manipulators with given targets. The approach separates the problem into a global coordination stage and a local trajectory replanning stage. Online reactivity and scalability are achieved by means of coordination in a reduced space and efficient trajectory replanning. We first introduce the method for systems with two robots, then extend it to systems with an arbitrary number of robots. The method determines whether kinematically-feasible and non-interfering trajectories leading all the robots to their targets exist. The approach generates time-efficient trajectories if solutions exist, or provides information for switching targets if solutions cannot be found. We show formally and empirically that the method has low computational overhead and scales quadratically with the number of robots. Experiments are conducted with up to three real 7-DOF robots and up to ten simulated robots. Note to Practitioners-In underground mining, a key process is that of tunneling, i.e., drilling and blasting rock to excavate tunnels that lead to sources of ore. Drilling is carried out by rigs equipped with multiple robotic arms. A key factor affecting the efficiency of tunneling is the time to completion of drilling operations. In current industrial practice, these operations are carried out manually by an operator steering the arms on the drill rig. Time to completion can be drastically reduced if the arms could operate concurrently, intelligently optimizing and coordinating their motions. However, existing methods for multi-arm motion coordination are inadequate, as they fail to cater to one or more of the following real-wor