Search

Over the past few years, there has been a noticeable surge in efforts to design novel tools and approaches that incorporate Artificial Intelligence (AI) into rehabilitation of persons with lower-limb impairments, using robotic exoskeletons. The potential benefits include the ability to implement personalized rehabilitation therapies by leveraging AI for robot control and data analysis, facilitating personalized feedback and guidance. Despite this, there is a current lack of literature review specifically focusing on AI applications in lower-limb rehabilitative robotics. To address this gap, our work aims at performing a review of 37 peer-reviewed papers. This review categorizes selected papers based on robotic application scenarios or AI methodologies. Additionally, it uniquely contributes by providing a detailed summary of input features, AI model performance, enrolled populations, exoskeletal systems used in the validation process, and specific tasks for each paper. The innovative aspect lies in offering a clear understanding of the suitability of different algorithms for specific tasks, intending to guide future developments and support informed decision-making in the realm of lower-limb exoskeleton and AI applications.

Template matching is a fundamental problem in computer vision with applications in fields including object detection, image registration, and object tracking. Current methods rely on nearest-neighbour (NN) matching, where the query feature space is converted to NN space by representing each query pixel with its NN in the template. NN-based methods have been shown to perform better in occlusions, appearance changes, and non-rigid transformations; however, they scale poorly with high-resolution data and high feature dimensions. We present an NN-based method that efficiently reduces the NN computations and introduces filtering in the NN fields (NNFs). A vector quantization step is introduced before the NN calculation to represent the template with k features, and the filter response over the NNFs is used to compare the template and query distributions over the features. We show that state-of-the-art performance is achieved in low-resolution data, and our method outperforms previous methods at higher resolution.

Multiplexed imaging enables the simultaneous spatial profiling of dozens of biological molecules in tissues at single-cell resolution. Extracting biologically relevant information, such as the spatial distribution of cell phenotypes from multiplexed tissue imaging data, involves a number of computational tasks, including image segmentation, feature extraction and spatially resolved single-cell analysis. Here, we present an end-to-end workflow for multiplexed tissue image processing and analysis that integrates previously developed computational tools to enable these tasks in a user-friendly and customizable fashion. For data quality assessment, we highlight the utility of napari-imc for interactively inspecting raw imaging data and the cytomapper R/Bioconductor package for image visualization in R. Raw data preprocessing, image segmentation and feature extraction are performed using the steinbock toolkit. We showcase two alternative approaches for segmenting cells on the basis of supervised pixel classification and pretrained deep learning models. The extracted single-cell data are then read, processed and analyzed in R. The protocol describes the use of community-established data containers, facilitating the application of R/Bioconductor packages for dimensionality reduction, single-cell visualization and phenotyping. We provide instructions for performing spatially resolved single-cell analysis, including community analysis, cellular neighborhood detection and cell-cell interaction testing using the imcRtools R/Bioconductor package. The workflow has been previously applied to imaging mass cytometry data, but can be easily adapted to other highly multiplexed imaging technologies. This protocol can be implemented by researchers with basic bioinformatics training, and the analysis of the provided dataset can be completed within 5-6 h. An extended version is available at https://bodenmillergroup.github.io/IMCDataAnalysis/. The protocol describes the analysis of data g

Horizon Europe EIC project SymAware, grant agreement 101070802

Microscopy is an integral technique in biology to study the fundamental components of life visually. Digital microscopy and automation have enabled biologists to conduct faster and larger-scale experiments with a sharp increase in the data generated. Microscopy images contain rich but sparse information, as typically, only small regions in the images are relevant for further study. Image analysis is a crucial tool for biologists in the objective interpretation and extraction of quantitative measurements from microscopy data. Recently, deep learning techniques have shown superior performance in various image analysis tasks. The models learn feature representations from the data by optimizing for a task. However, the techniques require a significant amount of annotated data to perform well. Domain experts are required to annotate microscopy data, making it expensive and time-consuming. The models offer no insight into their prediction, and the learned features are not directly interpretable. This poses challenges to the reliable utilization of the technique in high-trust applications such as drug discovery or disease detection. High data variability in microscopy and poor generalization performance of deep learning models further increase the difficulty in general usage of the technique. The work in this thesis presents frameworks and methods to solve the practical challenges of applying deep learning in microscopy. The application-specific evaluation approaches were presented to validate the approaches, aiming to increase trust in the system. The major contributions of this work are as follows. Papers I and III present human-in-the-loop frameworks for quick adaption of deep learning to new data and for improving models' performance based on human input in visual explanations provided by the model, respectively. Paper II proposes a template-matching approach to improve user interactions in the framework proposed in Paper I. Papers III and IV present architectural mod

This paper presents a novel solution for catastrophic forgetting in lifelong learning (LL) using Dynamic Convolution Neural Network (Dy-CNN). The proposed dynamic convolution layer can adapt convolution filters by learning kernel coefficients or weights based on the input image. The suitability of the proposed Dy-CNN in a lifelong sequential learning-based scenario with multi-modal MR images is experimentally demonstrated for the segmentation of Glioma tumors from multi-modal MR images. Experimental results demonstrated the superiority of the Dy-CNN-based segmenting network in terms of learning through multi-modal MRI images and better convergence of lifelong learning-based training.

Data-driven and controllable human motion synthesis and prediction are active research areas with various applications in interactive media and social robotics. Challenges remain in these fields for generating diverse motions given past observations and dealing with imperfect poses. This paper introduces MoDiff, an autoregressive probabilistic diffusion model over motion sequences conditioned on control contexts of other modalities. Our model integrates a cross-modal Transformer encoder and a Transformer-based decoder, which are found effective in capturing temporal correlations in motion and control modalities. We also introduce a new data dropout method based on the diffusion forward process to provide richer data representations and robust generation. We demonstrate the superior performance of MoDiff in controllable motion synthesis for locomotion with respect to two baselines and show the benefits of diffusion data dropout for robust synthesis and reconstruction of high-fidelity motion close to recorded data., Part of proceedings ISBN 979-8-3503-3670-2QC 20240110

As the pretraining technique is growing in popularity, little work has been done on pretrained learning-based motion prediction methods in autonomous driving. In this paper, we propose a framework to formalize the pretraining task for trajectory prediction of traffic participants. Within our framework, inspired by the random masked model in natural language processing (NLP) and computer vision (CV), objects' positions at random timesteps are masked and then filled in by the learned neural network (NN). By changing the mask profile, our framework can easily switch among a range of motion-related tasks. We show that our proposed pretraining framework is able to deal with noisy inputs and improves the motion prediction accuracy and miss rate, especially for objects occluded over time by evaluating it on Argoverse and NuScenes datasets., Part of ISBN 979-835039946-2QC 20240315

Accurate and timely movement intention detection can facilitate exoskeleton control during transitions between different locomotion modes. Detecting movement intentions in real environments remains a challenge due to unavoidable environmental uncertainties. False movement intention detection may also induce risks of falling and general danger for exoskeleton users. To this end, in this study, we developed a method for detecting human movement intentions in real environments. The proposed method is capable of online self-correcting by implementing a decision fusion layer. Gaze data from an eye tracker and inertial measurement unit (IMU) signals were fused at the feature extraction level and used to predict movement intentions using 2 different methods. Images from the scene camera embedded on the eye tracker were used to identify terrains using a convolutional neural network. The decision fusion was made based on the predicted movement intentions and identified terrains. Four able-bodied participants wearing the eye tracker and 7 IMU sensors took part in the experiments to complete the tasks of level ground walking, ramp ascending, ramp descending, stairs ascending, and stair descending. The recorded experimental data were used to test the feasibility of the proposed method. An overall accuracy of 93.4% was achieved when both feature fusion and decision fusion were used. Fusing gaze data with IMU signals improved the prediction accuracy., Part of proceedings ISBN: 979-8-3503-4275-8QC 20240109

This paper develops an in-depth treatment concerning the problem of approximating the Gaussian smoothing and Gaussian derivative computations in scale-space theory for application on discrete data. With close connections to previous axiomatic treatments of continuous and discrete scale-space theory, we consider three main ways discretizing these scale-space operations in terms of explicit discrete convolutions, based on either (i) sampling the Gaussian kernels and the Gaussian derivative kernels, (ii) locally integrating the Gaussian kernels and the Gaussian derivative kernels over each pixel support region, to aim at suppressing some of the severe artefacts of sampled Gaussian kernels and sampled Gaussian derivatives at very fine scales, and (iii) basing the scale-space analysis on the discrete analogue of the Gaussian kernel, and then computing derivative approximations by applying small-support central difference operators to the spatially smoothed image data. We study the properties of these three main discretization methods both theoretically and experimentally, and characterize their performance by quantitative measures, including the results they give rise to with respect to the task of scale selection, investigated for four different use cases, and with emphasis on the behaviour at fine scales. The results show that the sampled Gaussian kernels and the sampled Gaussian derivatives as well as the integrated Gaussian kernels and the integrated Gaussian derivatives perform very poorly at very fine scales. At very fine scales, the discrete analogue of the Gaussian kernel with its corresponding discrete derivative approximations performs substantially better. The sampled Gaussian kernel and the sampled Gaussian derivatives do, on the other hand, lead to numerically very good approximations of the corresponding continuous results, when the scale parameter is sufficiently large, in the experiments presented in the paper, when the scale parameter is greater than a v, QC 20231121, Covariant and invariant deep networks

Reliable detection and classification of bacteria and other pathogens in the human body, animals, food, and water is crucial for improving and safeguarding public health. For instance, identifying the species and its antibiotic susceptibility is vital for effective bacterial infection treatment. Here we show that phase contrast time-lapse microscopy combined with deep learning is sufficient to classify four species of bacteria relevant to human health. The classification is performed on living bacteria and does not require fixation or staining, meaning that the bacterial species can be determined as the bacteria reproduce in a microfluidic device, enabling parallel determination of susceptibility to antibiotics. We assess the performance of convolutional neural networks and vision transformers, where the best model attained a class-average accuracy exceeding 98%. Our successful proof-of-principle results suggest that the methods should be challenged with data covering more species and clinically relevant isolates for future clinical use. Bacterial infections are a leading cause of premature death worldwide, and growing antibiotic resistance is making treatment increasingly challenging. To effectively treat a patient with a bacterial infection, it is essential to quickly detect and identify the bacterial species and determine its susceptibility to different antibiotics. Prompt and effective treatment is crucial for the patient's survival. A microfluidic device functions as a miniature "lab-on-chip" for manipulating and analyzing tiny amounts of fluids, such as blood or urine samples from patients. Microfluidic chips with chambers and channels have been designed for quickly testing bacterial susceptibility to different antibiotics by analyzing bacterial growth. Identifying bacterial species has previously relied on killing the bacteria and applying species-specific fluorescent probes. The purpose of the herein proposed species identification is to speed up decisions o

Pedestrian crossing intention prediction is an important task for intelligent vehicles to enhance safety and reduce the risk of accidents. The high prediction accuracy and fast execution speed are essential requirements for this task. Existing studies on pedestrian crossing intention prediction have achieved good performance by using complex models and multiple modalities of input data. However, these approaches are limited in practical implications due to their high computational complexity and resource requirements. To address these, the paper propose the Spatial-Temporal Graph Convolution Network (GCN) with multiple attentions for fast and robust Pedestrian Crossing Intention Prediction (STMAGCN_PedCross). It utilizes easily accessible yet robust keypoints as input for predicting pedestrian crossing intention, making it both accurate and practical. To evaluate the effectiveness of the proposed model, we compared it with state-of-the-art models on a large-scale public Joint Attention in Autonomous Driving (JAAD) dataset. The results demonstrate that the STMA-GCN_PedCross model achieves comparable accuracy perfmance to the state-of-the-art models while having a higher robustness (recall rate) and requiring significantly less execution time. The ablation analysis further confirms the effectiveness of attention mechanisms in capturing spatial-temporal features from the skeleton sequence data by giving different attentions. Moreover, the analysis also reveals the significance of different keypoints in the prediction results., QC 20240314Part of ISBN 979-835039946-2

We introduce a non-parametric density estimator deemed Radial Voronoi Density Estimator (RVDE). RVDE is grounded in the geometry of Voronoi tessellations and as such benefits from local geometric adaptiveness and broad convergence properties. Due to its radial definition RVDE is continuous and computable in linear time with respect to the dataset size. This amends for the main shortcomings of previously studied VDEs, which are highly discontinuous and computationally expensive. We provide a theoretical study of the modes of RVDE as well as an empirical investigation of its performance on high-dimensional data. Results show that RVDE outperforms other non-parametric density estimators, including recently introduced VDEs., QC 20230821

Subterranean exploration and mapping for search and rescue robotics have become an emerging research direction since the DARPA organized Subterranean Challenge. As part of development efforts within the team CoSTAR (Collaborative SubTerranean Autonomous Robots) in the Sub-T challenge, this work establishes a novel traversable graph-based exploration strategy that utilizes frontiers for local navigation and a fast collision risk-aware graph building for global navigation. The exploration strategy extracts frontiers in an unknown area that contribute to safe navigation while maximizing information gain for the robot. The exploration problem is further bifurcated into local and global exploration for faster decision-making at junctions with the goal of rapidly exploring the area. The local exploration guarantees collision-free straight-line paths to informative frontiers for rapid forward navigation, while global re-positioning utilizes a traversable graph subject to geometrical collision checks within the occupancy map. The pathfinding in a graph is addressed using a heuristic, which combines risk margins and travel costs to assist in short yet safe paths to the global frontier in case of a dead end in local exploration. The presented exploration strategy is developed with the goal of making exploration algorithms platform agnostic in order to be able to use it with aerial, as well as ground robots. The proposed method is also evaluated against different state-of-the-art exploration planners in simulated fixed-time budget-based exploration missions on an Unmanned Aerial Vehicle (UAV) in order to benchmark the capabilities and highlight the novelty., Full text license: CC BY-NC-ND

We study the joint image of lines incident to points, meaning the set of image tuples obtained from fixed cameras observing a varying 3D point-line incidence. We prove a formula for the number of complex critical points of the triangulation problem that aims to compute a 3D point-line incidence from noisy images. Our formula works for an arbitrary number of images and measures the intrinsic difficulty of this triangulation. Additionally, we conduct numerical experiments using homotopy continuation methods, comparing different approaches of triangulation of such incidences. In our setup, exploiting the incidence relations gives a notably faster point reconstruction with comparable accuracy., QC 20240307

This paper presents an approach to automatic surface defect detection by a deep learning-based object detection method, particularly in challenging scenarios where defects are rare, i.e., with limited training data. We base our approach on an object detection model YOLOv8, preceded by a few steps: 1) filtering out irrelevant information, 2) enhancing the visibility of defects, namely brightness contrast, and 3) increasing the diversity of the training data through data augmentation. We evaluated the method in an industrial case study of crown wheel surface inspection in detecting Unclean Gear as well as Deburring defects, resulting in promising performances. With the combination of the three preprocessing steps, we improved the detection accuracy by 22.2% and 37.5% respectively while detecting those two defects. We believe that the proposed approach is also adaptable to various applications of surface defect detection in other industrial environments as the employed techniques, such as image segmentation, are available off the shelf., CC BY-NC-ND 4.0 DEED© 2023 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of the scientific committee of the 56th CIRP International Conference on Manufacturing Systems 2023.Correspondence Address: X. Zhu; Scania CV AB (publ), Södertälje, SE-151 87, Sweden; email: xiaomeng.zhu@scania.comThis work is partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Multimodal sensor fusion methods for 3D object detection have been revolutionizing the autonomous driving research field. Nevertheless, most of these methods heavily rely on dense LiDAR data and accurately calibrated sensors which is often not the case in real-world scenarios. Data from LiDAR and cameras often come misaligned due to the miscalibration, decalibration, or different frequencies of the sensors. Additionally, some parts of the LiDAR data may be occluded and parts of the data may be missing due to hardware malfunction or weather conditions. This work presents a novel fusion step that addresses data corruptions and makes sensor fusion for 3D object detection more robust. Through extensive experiments, we demonstrate that our method performs on par with state-of-the-art approaches on normal data and outperforms them on misaligned data., QC 20240301

Introduction: In the last few years, several models trying to calculate the biological brain age have been proposed based on structural magnetic resonance imaging scans (T1-weighted MRIs, T1w) using multivariate methods and machine learning. We developed and validated a convolutional neural network (CNN)-based biological brain age prediction model that uses one T1w MRI preprocessing step when applying the model to external datasets to simplify implementation and increase accessibility in research settings. Our model only requires rigid image registration to the MNI space, which is an advantage compared to previous methods that require more preprocessing steps, such as feature extraction. Methods: We used a multicohort dataset of cognitively healthy individuals (age range = 32.0–95.7 years) comprising 17,296 MRIs for training and evaluation. We compared our model using hold-out (CNN1) and cross-validation (CNN2–4) approaches. To verify generalisability, we used two external datasets with different populations and MRI scan characteristics to evaluate the model. To demonstrate its usability, we included the external dataset’s images in the cross-validation training (CNN3). To ensure that our model used only the brain signal on the image, we also predicted brain age using skull-stripped images (CNN4). Results: The trained models achieved a mean absolute error of 2.99, 2.67, 2.67, and 3.08 years for CNN1–4, respectively. The model’s performance in the external dataset was in the typical range of mean absolute error (MAE) found in the literature for testing sets. Adding the external dataset to the training set (CNN3), overall, MAE is unaffected, but individual cohort MAE improves (5.63–2.25 years). Salience maps of predictions reveal that periventricular, temporal, and insular regions are the most important for age prediction. Discussion: We provide indicators for using biological (predicted) brain age as a metric for age correction in neuroimaging studies as an alternati, QC 20240208

We address the problem of learning representations from observations of a scene involving an agent and an external object the agent interacts with. To this end, we propose a representation learning framework extracting the location in physical space of both the agent and the object from unstructured observations of arbitrary nature. Our framework relies on the actions performed by the agent as the only source of supervision, while assuming that the object is displaced by the agent via unknown dynamics. We provide a theoretical foundation and formally prove that an ideal learner is guaranteed to infer an isometric representation, disentangling the agent from the object and correctly extracting their locations. We evaluate empirically our framework on a variety of scenarios, showing that it outperforms vision-based approaches such as a state-of-the-art keypoint extractor. We moreover demonstrate how the extracted representations enable the agent to solve downstream tasks via reinforcement learning in an efficient manner., Part of ISBN 9783031434204QC 20231106

Spectral unmixing poses a significant challenge within hyperspectral image processing, traditionally addressed by supervised convolutional neural network (CNN)-based approaches employing patch-to-pixel (pixel-wise) methods. However, such pixel-wise methodologies often necessitate image splitting into overlapping patches, resulting in redundant computations and potential information leakage between training and test samples, consequently yielding overoptimistic outcomes. To overcome these challenges, this paper introduces a novel patch-to-patch (patch-wise) framework with nonoverlapping splitting, mitigating the need for repetitive calculations and preventing information leakage. The proposed framework incorporates a novel neural network structure inspired by the fully convolutional network (FCN), tailored for patch-wise unmixing. A highly efficient band reduction layer is incorporated to reduce the spectral dimension, and a specialized abundance constraint module is crafted to enforce both the Abundance Nonnegativity Constraint and the Abundance Sum-to-One Constraint for unmixing tasks. Furthermore, to enhance the performance of abundance estimation, a spatial-spectral attention module is introduced to activate the most informative spatial areas and feature maps. Extensive quantitative experiments and visual assessments conducted on two synthetic datasets and three real datasets substantiate the superior performance of the proposed algorithm. Significantly, the method achieves an impressive RMSE loss of 0.007, which is at least 4.5 times lower than that of other baselines on Urban hyperspectral images. This outcome demonstrates the effectiveness of our approach in addressing the challenges of spectral unmixing., QC 20240124

In this paper, we study the multi-robot task assignment and path-finding problem (MRTAPF), where a number of robots are required to visit all given tasks while avoiding collisions with each other. We propose a novel two-layer algorithm SA-reCBS that cascades the simulated annealing algorithm and conflict-based search to solve this problem. Compared to other approaches in the field of MRTAPF, the advantage of SA-reCBS is that without requiring a pre-bundle of tasks to groups with the same number of groups as the number of robots, it enables a part of robots needed to visit all tasks in collision-free paths. We test the algorithm in various simulation instances and compare it with state-of-the-art algorithms. The result shows that SA-reCBS has a better performance with a higher success rate, less computational time, and better objective values., Full text license: CC BY-NC-ND

Integrating mobile eye-tracking and motion capture emerges as a promising approach in studying visual-motor coordination, due to its capability of expressing gaze data within the same laboratory-centered coordinate system as body movement data. In this paper, we proposed an integrated eye-tracking and motion capture system, which can record and analyze temporally and spatially synchronized gaze and motion data during dynamic movement. The accuracy of gaze measurement were evaluated on five participants while they were instructed to view fixed vision targets at different distances while standing still or walking towards the targets. Similar accuracy could be achieved in both static and dynamic conditions. To demonstrate the usability of the integrated system, several walking tasks were performed in three different pathways. Results revealed that participants tended to focus their gaze on the upcoming path, especially on the downward path, possibly for better navigation and planning. In a more complex pathway, coupled with more gaze time on the pathway, participants were also found having the longest step time and shortest step length, which led to the lowest walking speed. It was believed that the integration of eye-tracking and motion capture is a feasible and promising methodology quantifying visual-motor coordination in locomotion., Part of proceedings ISBN 979-8-3503-4275-8QC 20240109

Sampling-based motion planning algorithms excel at searching global solution paths in geometrically complex settings. However, classical approaches, such as RRT, are difficult to scale beyond low-dimensional search spaces and rely on privileged knowledge e.g. about collision detection and underlying state distances. In this work, we take a step towards the integration of sampling-based planning into the reinforcement learning framework to solve sparse-reward control tasks from high-dimensional inputs. Our method, called VELAP, determines sequences of waypoints through sampling-based exploration in a learned state embedding. Unlike other sampling-based techniques, we iteratively expand a tree-based memory of visited latent areas, which is leveraged to explore a larger portion of the latent space for a given number of search iterations. We demonstrate state-of-the-art results in learning control from offline data in the context of vision-based manipulation under sparse reward feedback. Our method extends the set of available planning tools in model-based reinforcement learning by adding a latent planner that searches globally for feasible paths instead of being bound to a fixed prediction horizon., QC 20231227

QC 20231204

Satellite remote sensing provides a cost-effective way for monitoring wildfires on a large scale, and the continuous observations and measurements have made remote sensing a primary source of unlabelled big data. Supervised deep learning has shown great success in various remote sensing applications, but it heavily relies on high-quality labels. However, burned area labels are only available for a small part of the world, supervised deep learning from limited labelled data has poor generalization performance across geographical regions and climate zones. Different satellite sensors represent the same physical objects in various ways, while multi-source satellite data often exhibits a combination of common and complementary information, such as optical and radar data. The common information makes it possible to exploit huge amounts of unlabelled multi-source data in model training through consistency regularization between multi-source predictions. In this work, we adopted an unsupervised geospatial domain adaptation (GDA) framework based Dual Stream U-Net model, which combines the supervised loss and unsupervised multi-modal consistency regularization to exploit both labelled and unlabelled multi-model data for model training in a semi-supervised learning manner. The experimental results demonstrate that unsupervised GDA has better generalization performance across geographical regions than fully supervised learning., Part of ISBN 9798350320107QC 20231214

Urbanization is progressing at an unprecedented rate in many places around the world. The Sentinel-1 synthetic aperture radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI) missions, combined with deep learning, offer new opportunities to accurately monitor urbanization at a global scale. Although the joint use of SAR and optical data has recently been investigated for urban change detection, existing data fusion methods rely heavily on the availability of sufficient training labels. Meanwhile, change detection methods addressing label scarcity are typically designed for single-sensor optical data. To overcome these limitations, we propose a semi-supervised urban change detection method that exploits unlabeled Sentinel-1 SAR and Sentinel-2 MSI data. Using bitemporal SAR and optical image pairs as inputs, the proposed multi-modal Siamese network predicts urban changes and performs built-up area segmentation for both timestamps. Additionally, we introduce a consistency loss, which penalizes inconsistent built-up area segmentation across sensor modalities on unlabeled data, leading to more robust features. To demonstrate the effectiveness of the proposed method, the SpaceNet 7 dataset, comprising multi-temporal building annotations from rapidly urbanizing areas across the globe, was enriched with Sentinel-1 SAR and Sentinel-2 MSI data. Subsequently, network performance was analyzed under label-scarce conditions by training the network on different fractions of the labeled training set. The proposed method achieved an F1 score of 0.555 when using all available training labels, and produced reasonable change detection results (F1 score of 0.491) even with as little as 10% of the labeled training data. In contrast, multi-modal supervised methods and semi-supervised methods using optical data failed to exceed an F1 score of 0.402 under this condition. Code and data are made publicly available., QC 20231128

The influence of natural image transformations on receptive field responses is crucial for modelling visual operations in computer vision and biological vision. In this regard, covariance properties with respect to geometric image transformations in the earliest layers of the visual hierarchy are essential for expressing robust image operations and for formulating invariant visual operations at higher levels. This paper defines and proves a joint covariance property under compositions of spatial scaling transformations, spatial affine transformations, Galilean transformations and temporal scaling transformations, which makes it possible to characterize how different types of image transformations interact with each other. Specifically, the derived relations show how the receptive field parameters need to be transformed, in order to match the output from spatio-temporal receptive fields with the underlying spatio-temporal image transformations., QC 20231120, Covariant and invariant deep networks

Acoustic sensors play an important role in autonomous underwater vehicles (AUVs). Sidescan sonar (SSS) detects a wide range and provides photo-realistic images in high resolution. However, SSS projects the 3D seafloor to 2D images, which are distorted by the AUV's altitude, target's range and sensor's resolution. As a result, the same physical area can show significant visual differences in SSS images from different survey lines, causing difficulties in tasks such as pixel correspondence and template matching. In this paper, a canonical transformation method consisting of intensity correction and slant range correction is proposed to decrease the above distortion. The intensity correction includes beam pattern correction and incident angle correction using three different Lambertian laws (cos, cos2, cot), whereas the slant range correction removes the nadir zone and projects the position of SSS elements into equally horizontally spaced, view-point independent bins. The proposed method is evaluated on real data collected by a HUGIN AUV, with manually-annotated pixel correspondence as ground truth reference. Experimental results on patch pairs compare similarity measures and keypoint descriptor matching. The results show that the canonical transformation can improve the patch similarity, as well as SIFT descriptor matching accuracy in different images where the same physical area was ensonified., Part of ISBN 9798350332261QC 20231116

In this work, we develop a novel input feature selection framework for ReLU-based deep neural networks (DNNs), which builds upon a mixed-integer optimization approach. While the method is generally applicable to various classification tasks, we focus on finding input features for image classification for clarity of presentation. The idea is to use a trained DNN, or an ensemble of trained DNNs, to identify the salient input features. The input feature selection is formulated as a sequence of mixed-integer linear programming (MILP) problems that find sets of sparse inputs that maximize the classification confidence of each category. These “inverse” problems are regularized by the number of inputs selected for each category and by distribution constraints. Numerical results on the well-known MNIST and FashionMNIST datasets show that the proposed input feature selection allows us to drastically reduce the size of the input to ∼ 15% while maintaining a good classification accuracy. This allows us to design DNNs with significantly fewer connections, reducing computational effort and producing DNNs that are more robust towards adversarial attacks., Part of ISBN 9783031445040QC 20231116

Photon-counting detectors are greatly improving the resolution and image quality in computed tomography (CT) technology. The drawback is, however, that the reconstruction becomes more challenging. This is because there is a considerable increment of the processing data due to the multiple energy bins and materials in the reconstruction analysis, as well as improved resolution. Yet efficient material decomposition and reconstruction methods tend to generate noisy images that do not completely satisfy the expected image quality. Therefore, there is a need for efficient denoising of the resulting material images. We present a new and fast denoiser that is based on a linear minimum mean square error (LMMSE) estimator. The LMMSE is very fast to compute, but not commonly used for CT image denoising, probably due to its inability to adapt the amount of denoising to different parts of the image and the difficulty to derive accurate statistical properties from the CT data. To overcome these problems we propose a model-based deep learning strategy, that is, a deep neural network that preserves an LMMSE structure (model-based), providing more robustness unseen data, as well as good interpretability to the result. In this way, the solution adapts to the anatomy in every point of the image and noise properties at that particular location. In order to asses the performance of the new method, we compare it to both to a conventional LMMSE estimator and to a "black-box"CNN in a simulation study with anthropomorphic phantoms., Part of ISBN 9781510660311QC 20231123

Self-supervised graph representation learning (SSGRL) is a representation learning paradigm used to reduce or avoid manual labeling. An essential part of SSGRL is graph data augmentation. Existing methods usually rely on heuristics commonly identified through trial and error and are effective only within some application domains. Also, it is not clear why one heuristic is better than another. Moreover, recent studies have argued against some techniques (e.g., dropout: that can change the properties of molecular graphs or destroy relevant signals for graph-based document classification tasks). In this study, we propose a novel data-driven SSGRL approach that automatically learns a suitable graph augmentation from the signal encoded in the graph (i.e., the nodes' predictive feature and topological information). We propose two complementary approaches that produce learnable feature and topological augmentations. The former learns multi-view augmentation of node features, and the latter learns a high-order view of the topology. Moreover, the augmentations are jointly learned with the representation. Our approach is general that it can be applied to homogeneous and heterogeneous graphs. We perform extensive experiments on node classification (using nine homogeneous and heterogeneous datasets) and graph property prediction (using another eight datasets). The results show that the proposed method matches or outperforms the SOTA SSGRL baselines and performs similarly to semi-supervised methods. The anonymised source code is available at https://github.com/AhmedESamy/dsgrl, Part of ISBN 9781643684369QC 20231116

This paper presents a vision-based localization approach for Unmanned Aerial Vehicles (UAVs) flying at low altitude over forested areas. We address the task as a point cloud registration problem using local 3D features with the intention to exploit the shape and relative arrangement of the trees. We propose a 3D descriptor called SHOT-N which is an adaptation of the state-of-the-art SHOT 3D descriptor. SHOT-N leverages constraints in the extrinsic parameters of a gimballed, nadir-looking camera. Extensive experiments were performed with semi-simulated point cloud data based on real aerial images over four forested areas. SHOT-N is shown to outperform two state-of-the-art 3D descriptors in terms of the rate of successful registrations. The results suggest a high potential of the approach for aerial localization over forested areas., Part of ISBN 9783031314346QC 20231106

Next generation X-ray computed tomography, based on photon-counting detectors, is now clinically available. These new detectors come with the promise of higher contrast-to-noise ratio and spatial resolution and improved low-dose imaging. However, the multi-bin nature of photon-counting detectors renders the image reconstruction problem more difficult. Common approaches, such as the two-step projection-based approach, may result in material basis images with an excessive degree of noise, which limits the clinical usefulness of the images. One possible solution is to "assist"the conventional image reconstruction by post-processing the reconstructed images using deep learning. Such networks are often trained using some pixel-wise loss, such as the mean squared error. This low-level per-pixel comparison is known to lead to over-smoothing and a loss of fine-grained details that are important to the perceptual quality and clinical usefulness of the image. In this abstract, we propose to tackle this issue by including an adversarial loss based on the Wasserstein generative adversarial network with gradient penalty. The adversarial loss will encourage the distribution of the processed images to be similar to that of the ground truth. This helps prevent over-smoothing and ensures that the ground truth texture is well preserved. In particular, we train a version of the UNet using a combination of the mean absolute error and an adversarial loss to correct for noise in the material basis images. We demonstrate that the proposed method can produce denoised virtual monoenergetic images, with realistic texture, at a range of energy levels., Part of ISBN 9781510660311QC 20231031

Retrieval of mental images from measured brain activity may facilitate communication, especially when verbal or muscular communication is impossible or inefficient. The existing work focuses mostly on retrieving the observed visual stimulus while our interest is on retrieving the imagined mental image. We present a closed-loop BCI framework to retrieve mental images of human faces. We utilize EEG signals as binary feedback to determine the relevance of an image to the target mental image. We employ the feedback to traverse the latent space of a generative model to propose new images closer to the actual target image. We evaluate the proposed framework on 13 volunteers. Unlike previous studies, we do not restrict the possible attributes of the resulting images to predefined semantic classes. Subjective and objective tests validate the ability of our model to retrieve face images similar to the actual target mental images., Part of ISBN 9783031350160QC 20231010

In this paper, a novel feature generator framework is proposed for handwritten digit classification. The proposed framework includes a two-stage cascaded feature generator. The first stage is based on principal component analysis (PCA), which generates projected data on principal components as features. The second one is constructed by a partially trained neural network (PTNN), which uses projected data as inputs and generates hidden layer outputs as features. The features obtained from the PCA and PTNN-based feature generator are tested on the MNIST and USPS datasets designed for handwritten digit sets. Minimum distance classifier (MDC) and support vector machine (SVM) methods are exploited as classifiers for the obtained features in association with this framework. The performance evaluation results show that the proposed framework outperforms the state-of-the-art techniques and achieves accuracies of 99.9815% and 99.9863% on the MNIST and USPS datasets, respectively. The results also show that the proposed framework achieves almost perfect accuracies, even with significantly small training data sizes., QC 20231114

In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. We present a method for analyzing datasets from a use-case scenario perspective, detecting and quantifying out-of-distribution (OOD) data on dataset level. Our main contribution is the novel use of similarity metrics for the evaluation of the robustness of a model by introducing relative Fréchet Inception Distance (FID) and relative Kernel Inception Distance (KID) measures. These relative measures are relative to a baseline in-distribution dataset and are used to estimate how the model will perform on OOD data (i.e. estimate the model accuracy drop). We find a correlation between our proposed relative FID/relative KID measure and the drop in Average Precision (AP) accuracy on unseen data., Part of ISBN 9783031429408QC 20231009

This paper is about effectively utilizing synthetic data for training deep neural networks for industrial parts classification, in particular, by taking into account the domain gap against real-world images. To this end, we introduce a synthetic dataset that may serve as a preliminary testbed for the Sim-to-Real challenge; it contains 17 objects of six industrial use cases, including isolated and assembled parts. A few subsets of objects exhibit large similarities in shape and albedo for reflecting challenging cases of industrial parts. All the sample images come with and without random backgrounds and post-processing for evaluating the importance of domain randomization. We call it Synthetic Industrial Parts dataset (SIP-17). We study the usefulness of SIP-17 through benchmarking the performance of five state-of-the-art deep network models, supervised and self-supervised, trained only on the synthetic data while testing them on real data. By analyzing the results, we deduce some insights on the feasibility and challenges of using synthetic data for industrial parts classification and for further developing larger-scale synthetic datasets. Our dataset † and code ‡ are publicly available., Part of ISBN 9798350302493QC 20231010

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on three datasets with different organs and modalities, where it substantially outperforms existing techniques. Our code is available at: https://github.com/histocartography/generative-appearance-replay.Previous article in issue

To enable progress towards egocentric agents capable of understanding everyday tasks specified in natural language, we propose a benchmark and a synthetic dataset called Egocentric Task Verification (EgoTV). The goal in EgoTV is to verify the execution of tasks from egocentric videos based on the natural language description of these tasks. EgoTV contains pairs of videos and their task descriptions for multi-step tasks -- these tasks contain multiple sub-task decompositions, state changes, object interactions, and sub-task ordering constraints. In addition, EgoTV also provides abstracted task descriptions that contain only partial details about ways to accomplish a task. Consequently, EgoTV requires causal, temporal, and compositional reasoning of video and language modalities, which is missing in existing datasets. We also find that existing vision-language models struggle at such all-round reasoning needed for task verification in EgoTV Inspired by the needs of EgoTV, we propose a novel Neuro-Symbolic Grounding (NSG) approach that leverages symbolic representations to capture the compositional and temporal structure of tasks. We demonstrate NSG's capability towards task tracking and verification on our EgoTV dataset and a real-world dataset derived from CrossTask (CTV). We open-source the EgoTV and CTV datasets and the NSG model for future research on egocentric assistive agents.

Purpose: Our purpose is to investigate the timing resolution in edge-on silicon strip detectors for photon-counting spectral computed tomography. Today, the timing for detection of individual x-rays is not measured, but in the future, timing information can be valuable to accurately reconstruct the interactions caused by each primary photon. Approach: We assume a pixel size of 12 x 500 mu m(2) and a detector with double-sided readout with low-noise CMOS electronics for pulse processing for every pixel on each side. Due to the electrode width in relation to the wafer thickness, the induced current signals are largely dominated by charge movement close to the collecting electrodes. By employing double-sided readout electrodes, at least two signals are generated for each interaction. By comparing the timing of the induced current pulses, the time of the interaction can be determined and used to identify interactions that originate from the same incident photon. Using a Monte Carlo simulation of photon interactions in combination with a charge transport model, we evaluate the performance of estimating the time of the interaction for different interaction positions. Results: Our simulations indicate that a time resolution of 1 ns can be achieved with a noise level of 0.5 keV. In a detector with no electronic noise, the corresponding time resolution is similar to 0.1 ns. Conclusions: Time resolution in edge-on silicon strip CT detectors can potentially be used to increase the signal-to-noise-ratio and energy resolution by helping in identifying Compton scattered photons in the detector. (c) The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI., Funding Agencies|Olle Engkvist Foundation

The synthesis of high-resolution remote sensing images based on text descriptions has great potential in many practical application scenarios. Although deep neural networks have achieved great success in many important remote sensing tasks, generating realistic remote sensing images from text descriptions is still very difficult. To address this challenge, we propose a novel text-to-image modern Hopfield network (Txt2Img-MHN). The main idea of Txt2Img-MHN is to conduct hierarchical prototype learning on both text and image embeddings with modern Hopfield layers. Instead of directly learning concrete but highly diverse text-image joint feature representations for different semantics, Txt2Img-MHN aims to learn the most representative prototypes from text-image embeddings, achieving a coarse-to-fine learning strategy. These learned prototypes can then be utilized to represent more complex semantics in the text-to-image generation task. To better evaluate the realism and semantic consistency of the generated images, we further conduct zero-shot classification on real remote sensing data using the classification model trained on synthesized images. Despite its simplicity, we find that the overall accuracy in the zero-shot classification may serve as a good metric to evaluate the ability to generate an image from text. Extensive experiments on the benchmark remote sensing text-image dataset demonstrate that the proposed Txt2Img-MHN can generate more realistic remote sensing images than existing methods. Code and pre-trained models are available online ( https://github.com/YonghaoXu/Txt2Img-MHN )., Funding: Institute of Advanced Research in Artificial Intelligence (IARAI)

Accurate and robust visual object tracking is one of the most challenging and fundamental computer vision problems. It entails estimating the trajectory of the target in an image sequence, given only its initial location, and segmentation, or its rough approximation in the form of a bounding box. Discriminative Correlation Filters (DCFs) and deep Siamese Networks (SNs) have emerged as dominating tracking paradigms, which have led to significant progress. Following the rapid evolution of visual object tracking in the last decade, this survey presents a systematic and thorough review of more than 90 DCFs and Siamese trackers, based on results in nine tracking benchmarks. First, we present the background theory of both the DCF and Siamese tracking core formulations. Then, we distinguish and comprehensively review the shared as well as specific open research challenges in both these tracking paradigms. Furthermore, we thoroughly analyze the performance of DCF and Siamese trackers on nine benchmarks, covering different experimental aspects of visual tracking: datasets, evaluation metrics, performance, and speed comparisons. We finish the survey by presenting recommendations and suggestions for distinguished open challenges based on our analysis., Funding Agencies|Khalifa University of Science and Technology [FSU-2022-003, 84740 00401]; Research Center for Informatics Project [CZ.02.1.01/0.0/0.0/16_019/0000765]

This thesis explores the integration of deep learning-based depth estimation models with the ORB-SLAM3 framework to address challenges in monocular Simultaneous Localization and Mapping (SLAM), particularly focusing on pure rotational movements. The study investigates the viability of using pre-trained generic depth estimation networks, and hybrid combinations of these networks, to replace traditional depth sensors and improve scale accuracy in SLAM systems. A series of experiments are conducted outdoors, utilizing a custom camera setup designed to isolate pure rotational movements. The analysis involves assessing each model's impact on the SLAM process as well as performance indicators (KPIs) on both depth estimation and 3D tracking. Results indicate a correlation between depth estimation accuracy and SLAM performance, underscoring the potential of depth estimation models in enhancing SLAM systems. The findings contribute to the understanding of the role of monocular depth estimation in integrating with SLAM, especially in applications requiring precise spatial awareness for augmented reality., Denna avhandling utforskar integrationen av djupinlärningsbaserade modeller för djupuppskattning med ORB-SLAM3-ramverket för att möta utmaningar inom monokulär Samtidig Lokalisering och Kartläggning (SLAM), med särskilt fokus på rena rotationsrörelser. Studien undersöker möjligheten att använda förtränade generiska nätverk för djupuppskattning och hybridkombinationer av dessa nätverk, för att ersätta traditionella djupsensorer och förbättra skalanoggrannheten i SLAM-system. En serie experiment genomförs med användning av en specialbyggd kamerauppställning utformad för att isolera rena rotationsrörelser. Analysen omfattar bedömning av varje modells påverkan på SLAM-processen samt kvantitativa prestandaindikatorer (KPI:er) för både djupuppskattning och följning. Resultaten visar på ett samband mellan noggrannheten i djupuppskattningen och SLAM-prestandan, vilket understryker potentialen hos modeller för djupuppskattning i förbättringen av SLAM-system. Rönen bidrar till förståelsen av rollen som monokulär djupuppskattning har i integrationen med SLAM, särskilt i tillämpningar som kräver exakt spatial medvetenhet.

In machine learning, a sub-field of computer science, a two-tower architecture model is a specialised type of neural network model that encodes paired data from different modalities (like text and images, sound and video, or proteomics and gene expression profiles) into a shared latent representation space. However, when training these models using a specific contrastive loss function, known as the multimodalinfoNCE loss, seems to often lead to a unique geometric phenomenon known as the modality gap. This gap is a clear geometric separation of the embeddings of the modalities in the joint contrastive latent space. This thesis investigates the modality gap in multimodal machine learning, specifically in two-tower neural networks trained with multimodal-infoNCE loss. We examine the adequacy of the current definition of the modality gap, the conditions under which the modality gap phenomenon manifests, and its impact on representation quality and downstream task performance. The approach to address these questions consists of a two-phase experimental strategy. Phase I involves a series of experiments, ranging from toy synthetic simulations to true multimodal machine learning with complex datasets, to explore and characterise the modality gap under varying conditions. Phase II focuses on modifying the modality gap and analysing representation quality, evaluating different loss functions and their impact on the modality gap. This methodical exploration allows us to systematically dissect the emergence and implications of the modality gap phenomenon, providing insights into its impact on downstream tasks, measured with proxy metrics based on semantic clustering in the shared latent representation space and modality-specific linear probe evaluation. Our findings reveal that the modality gap definition proposed by W. Liang et al. 2022, is insufficient. We demonstrate that similar modality gap magnitudes can exhibit varying linear separability between modality embeddings in

This Master's Thesis delves into the application of topological regularization techniques and relative latent representations within the realm of zero-shot model stitching. Building upon the prior work of Moschella et al. (2022) that introduces relative latent representations to enhance the similarities between latent spaces of different models, we incorporate the approach of Hofer et al. (2021), which combines Topological Data Analysis (TDA) and Machine Learning techniques for topological densification of class distributions in the latent space. The main research objective is to investigate the impact of topological regularization on zero-shot stitching performance when employing relative latent representations. Theoretical foundations for the relative transformation are established based on the intertwiner groups of activation functions. Empirical analyses are conducted to validate the assumptions underlying the construction of the relative transformation in the latent space. Moreover, experiments are performed on a Large Language Model trained on multilingual Amazon Reviews datasets to evaluate the effectiveness of zero-shot stitching while using the topological densification technique and the relative transformation. The findings indicate that the proposed methodologies can enhance the performance of multilingual model stitching. Specifically, enforcing the relative transformation to preserve the H0 homology death times distributions proves beneficial. Additionally, the presence of similar topological features plays a crucial role in achieving higher model compatibility. However, a more in-depth exploration of the geometric properties of the post-relative transformation latent space is necessary to further improve the topological densification technique. Overall, this work contributes to the emerging field of Topological Machine Learning and provides valuable insights for researchers in transfer learning and representation learning domains., Denna masteruppsats undersöker tillämpningen av topologiska regleringstekniker och relativa latenta representationer inom området för zero-shot model stitching. Genom att bygga vidare på tidigare arbete av Moschella et al. (2022), som introducerade relativa latenta representationer för att förbättra likheterna mellan latenta rummet hos olika modeller, inkorporerar vi tillvägagångssättet av Hofer et al. (2021), som kombinerar topologisk dataanalys (TDA) och maskininlärningstekniker för topologisk ``förtätning'' av klassfördelningar i det latenta utrymmet. Den huvudsakliga forskningsuppgiften är att undersöka effekten av topologisk reglering på zero-shot model stitching-prestanda när man använder relativa latenta representationer. Teoretiska grunder för den relativa transformationen etableras baserat på intertwinergrupperna för aktiveringsfunktioner. Empiriska analyser genomförs för att validera antagandena som ligger till grund för konstruktionen av den relativa transformationen i det latenta rummen. Dessutom utförs experiment på en stor språkmodell tränad på multilinguella Amazon Reviews-dataset för att utvärdera effektiviteten hos zero-shot model stitching med Hofer's topologiska reglering och relativa transformation. Resultaten visar att de föreslagna metoderna kan förbättra prestationen hos zero-shot model stitching för flerspråkiga modeller. Specifikt är det fördelaktigt att tvinga den relativa transformationen att bevara H0 homologins dödstidsfördelningar. Dessutom spelar närvaron av liknande topologiska egenskaper en avgörande roll för att uppnå högre modellkompatibilitet. Dock krävs en mer ingående utforskning av de geometriska egenskaperna hos det latenta utrymmet efter den relativa transformationen för att ytterligare förbättra Hofer's topologiska reglering. Sammanfattningsvis bidrar detta arbete till det framväxande området Topologisk Maskininlärning och ger värdefulla insikter för forskare inom ``transfer-inlärning'' och representationsinlärningsdomäner.

The rapid development of social robots, designed to interact with humans, has led to increased research on user acceptance and emotions in human-robot interaction. Social acceptance is an important area to investigate if the development of social robots is to be useful. Investigating how people feel about social robots is one tool to assess acceptance toward them, and research has shown that positive emotions could invoke higher acceptance. Possible factors that have been shown to affect peoples’ attitudes regarding social robots is (1) the human-likeness and appearance of the robot and (2) the application area of the robot. Therefore, this thesis research questions address the effect of human-likeness and application areas of social robots on people's emotions. The findings indicate that in the context of companionship, people have varying emotional responses based on the appearance of the social robot. Highly human-like robots evoke more positive emotions, while low human-likeness robots elicit more negative emotions. This suggests that individuals prefer human-like social robots in intimate interactions like companionship. The results also reveal an effect of application areas, where people respond more positively to highly human-like robots used for tasks like lecturing students or companionship for older adults. Regarding less human-like social robots, people tend to respond with greater positive emotions when used within commerce. This suggests that a simpler-looking robot with low human-likeness is more suitable for commercial applications. Negative emotions expressed in the healthcare condition may reflect mistrust in robots' abilities and the sensitivity of the healthcare area. Developers and designers should consider the emotional responses that might be evoked by the task or appearance of the social robot, to ensure successful integration into society.

Gaze estimation is the process of determining where a person is looking, which has recently become a popular research area due to its broad range of applications. For example, tools that estimate gaze are used for research, medical diagnosis, virtual and augmented reality, driver assistance system, and many more. Therefore, better products are sought by many. Gaze estimation methods typically use images of only the eyes or the whole face to estimate the gaze since these methods are the most practical and convenient options. Recently, Convolutional Neural Networks (CNNs) have been appealing candidates for estimating the gaze. Nevertheless, the recent success of Vision Transformers (ViTs) in image classification tasks has introduced a new potential alternative. Hence, this work investigates the potential of using ViTs to estimate the gaze on Near-Infrared (NIR) images. This is done in terms of average error and computational complexity. Furthermore, this work examines not only pure ViTs but other models, such as hybrid ViTs and CNN-Formers, which combine CNNs and ViTs. The empirical results showed that hybrid ViTs are the only models that can outperform state-of-the-art CNNs such as MobileNetV2 and ResNet-18 while maintaining similar computational complexity to ResNet-18. The results on hybrid ViTs indicate that the convolutional stem is the most crucial part of them. Improved convolutional stems lead to better outcomes. Moreover, in this work, we defined a new training algorithm for hybrid ViTs, the hybrid Data-Efficient Image Transformer (DeiT) procedure, which has shown remarkable results. It is 3.5% better than the pretrained ResNet-18 while having the same time complexity., Blickuppskattning är processen att uppskatta en persons blick, vilket nyligen har blivit ett populärt forskningsområde på grund av dess breda användningsområde. Till exempel, verktyg för blickuppskattning används inom forskning, medicinsk diagnos, virtuell och förstärkt verklighet, förarassistanssystem och för mycket mer. Därför, bättre produkter för blickuppskattning eftersträvas av många. Blickuppskattnings metoder vanligtvis använder bilder av endast ögonen eller hela ansiktet för att uppskatta blicken eftersom denna typen av metoder är de mest praktiska och lämliga alternativ. På sistånde har Convolutional Neural Networks (CNNs) varit tilltalande kandidater för att uppskatta blicken. Dock, har den senaste framgången med Vision Transformers (ViTs) i bildklassificeringsuppgifter introducerat ett nytt potentiellt alternativ. Därför undersöker detta arbete potentialen av att använda ViTs för att uppskatta blicken på Nära-infraröda (NIR) bilder. Undersökningen görs både i termer av medelfel och beräkningskomplexitet. Hursomhelst, detta arbete undersöker inte enbart rena ViTs utan andra modeller, som hybrida ViTs och CNN-Formers, som kombinerar CNNs och ViTs. De empiriska resultaten visade att hybrida ViTs är de enda modellerna som kan överträffa toppmoderna CNNs som MobileNetV2 och ResNet-18 samtidigt som de bibehåller liknande beräkningskomplexitet som ResNet-18. Resultaten på hybrida ViTs indikerar att faltningsstammen är den mest avgörande delen av dem. Det vill säga, desto bättre faltningsstamm en har desto bättre resultat kan man erhålla. Dessutom definierade vi i detta arbete en ny träningsalgoritm för hybrida ViTs, vilket vi kallar hybrida Data-Efficient Image Transformer (DeiT) procedur som har visat anmärkningsvärda resultat. Den är 3,5% bättre än den förtränade ResNet-18 samtidigt som den har samma tid komplexitet.

This study explores the application of multi-agent reinforcement learning (MARL) to enhance the decision-making, safety, and passenger comfort of Autonomous Vehicles (AVs)at uncontrolled intersections. The research aims to assess the potential of MARL in modeling multiple agents interacting within a shared environment, reflecting real-world situations where AVs interact with multiple actors. The findings suggest that AVs trained using aMARL approach with global experiences can better navigate intersection scenarios than AVs trained on local (individual) experiences. This capability is a critical precursor to achieving Level 5 autonomy, where vehicles are expected to manage all aspects of the driving task under all conditions. The research contributes to the ongoing discourse on enhancing autonomous vehicle technology through multi-agent reinforcement learning and informs the development of sophisticated training methodologies for autonomous driving.