Search

Recent advances in machine learning have highlighted Federated Learning (FL) as a promising approach that enables multiple distributed users (so-called clients) to collectively train ML models without sharing their private data. While this privacy-preserving method shows potential, it struggles when data across clients is not independent and identically distributed (non-IID) data. The latter remains an unsolved challenge that can result in poorer model performance and slower training times. Despite the significance of non-IID data in FL, there is a lack of consensus among researchers about its classification and quantification. This systematic review aims to fill that gap by providing a detailed taxonomy for non-IID data, partition protocols, and metrics to quantify data heterogeneity. Additionally, we describe popular solutions to address non-IID data and standardized frameworks employed in FL with heterogeneous data. Based on our state-of-the-art review, we present key lessons learned and suggest promising future research directions.

In this paper we propose a methodology combining Federated Learning (FL) with Cross-view Image Geo-localization (CVGL) techniques. We address the challenges of data privacy and heterogeneity in autonomous vehicle environments by proposing a personalized Federated Learning scenario that allows selective sharing of model parameters. Our method implements a coarse-to-fine approach, where clients share only the coarse feature extractors while keeping fine-grained features specific to local environments. We evaluate our approach against traditional centralized and single-client training schemes using the KITTI dataset combined with satellite imagery. Results demonstrate that our federated CVGL method achieves performance close to centralized training while maintaining data privacy. The proposed partial model sharing strategy shows comparable or slightly better performance than classical FL, offering significant reduced communication overhead without sacrificing accuracy. Our work contributes to more robust and privacy-preserving localization systems for autonomous vehicles operating in diverse environments, Comment: 6 pages, 2 figures, Preprint submitted to the IEEE 26th International Workshop on Multimedia Signal Processing (MMSP)

This paper proposes a novel localization framework based on collaborative training or federated learning paradigm, for highly accurate localization of autonomous vehicles. More specifically, we build on the standard approach of KalmanNet, a recurrent neural network aiming to estimate the underlying system uncertainty of traditional Extended Kalman Filtering, and reformulate it by the adapt-then-combine concept to FedKalmanNet. The latter is trained in a distributed manner by a group of vehicles (or clients), with local training datasets consisting of vehicular location and velocity measurements, through a global server aggregation operation. The FedKalmanNet is then used by each vehicle to localize itself, by estimating the associated system uncertainty matrices (i.e, Kalman gain). Our aim is to actually demonstrate the benefits of collaborative training for state estimation in autonomous driving, over collaborative decision-making which requires rich V2X communication resources for measurement exchange and sensor fusion under real-time constraints. An extensive experimental and evaluation study conducted in CARLA autonomous driving simulator highlights the superior performance of FedKalmanNet over state-of-the-art collaborative decision-making approaches, in localizing vehicles without the need of real-time V2X communication.

Many therapies are effective in treating multiple diseases. We present an approach that leverages methods developed in natural language processing and real-world data to prioritize potential, new indications for a mechanism of action (MoA). We specifically use representation learning to generate embeddings of indications and prioritize them based on their proximity to the indications with the strongest available evidence for the MoA. We demonstrate the successful deployment of our approach for anti-IL-17A using embeddings generated with SPPMI and present an evaluation framework to determine the quality of indication finding results and the derived embeddings.

Physics-Informed Neural Networks (PINNs) have emerged as a robust framework for solving Partial Differential Equations (PDEs) by approximating their solutions via neural networks and imposing physics-based constraints on the loss function. Traditionally, Multilayer Perceptrons (MLPs) have been the neural network of choice, with significant progress made in optimizing their training. Recently, Kolmogorov-Arnold Networks (KANs) were introduced as a viable alternative, with the potential of offering better interpretability and efficiency while requiring fewer parameters. In this paper, we present a fast JAX-based implementation of grid-dependent Physics-Informed Kolmogorov-Arnold Networks (PIKANs) for solving PDEs, achieving up to 84 times faster training times than the original KAN implementation. We propose an adaptive training scheme for PIKANs, introducing an adaptive state transition technique to avoid loss function peaks between grid extensions, and a methodology for designing PIKANs with alternative basis functions. Through comparative experiments, we demonstrate that the adaptive features significantly enhance solution accuracy, decreasing the L^2 error relative to the reference solution by up to 43.02%. For the studied PDEs, our methodology approaches or surpasses the results obtained from architectures that utilize up to 8.5 times more parameters, highlighting the potential of adaptive, grid-dependent PIKANs as a superior alternative in scientific and engineering applications.

The Lorentzian type IIB matrix model is a promising candidate for a nonperturbative formulation of superstring theory. Recently we performed complex Langevin simulations by adding a Lorentz invariant mass term as an IR regulator and found a (1+1)-dimensional expanding spacetime with a Lorentzian signature emerging dynamically at late times when the fermionic contribution is omitted. Here we find that this is merely an artifact of the Lorentz boosts by showing that the spontaneous breaking of rotational symmetry is eliminated if one chooses a Lorentz frame appropriately. On the other hand, when we include the fermionic contribution, we find some evidence suggesting the emergence of a smooth (3+1)-dimensional expanding Lorentzian spacetime., Comment: 12 pages, 9 figures, proceedings of Corfu Summer Institute 2023 "School and Workshops on Elementary Particle Physics and Gravity"

Graph neural networks (GNNs) excel in learning from network-like data but often lack interpretability, making their application challenging in domains requiring transparent decision-making. We propose the Graph Kolmogorov-Arnold Network (GKAN), a novel GNN model leveraging spline-based activation functions on edges to enhance both accuracy and interpretability. Our experiments on five benchmark datasets demonstrate that GKAN outperforms state-of-the-art GNN models in node classification, link prediction, and graph classification tasks. In addition to the improved accuracy, GKAN's design inherently provides clear insights into the model's decision-making process, eliminating the need for post-hoc explainability techniques. This paper discusses the methodology, performance, and interpretability of GKAN, highlighting its potential for applications in domains where interpretability is crucial., Comment: 7 pages, 4 figures, under review

As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant overhead to the underlying system. In this work, we introduce LLM-dCache to optimize data accesses by treating cache operations as callable API functions exposed to the tool-augmented agent. We grant LLMs the autonomy to manage cache decisions via prompting, seamlessly integrating with existing function-calling mechanisms. Tested on an industry-scale massively parallel platform that spans hundreds of GPT endpoints and terabytes of imagery, our method improves Copilot times by an average of 1.24x across various LLMs and prompting techniques., Comment: ICECS 2024 Camera-Ready

State-of-the-art sequential reasoning in Large Language Models (LLMs) has expanded the capabilities of Copilots beyond conversational tasks to complex function calling, managing thousands of API calls. However, the tendency of compositional prompting to segment tasks into multiple steps, each requiring a round-trip to the GPT APIs, leads to increased system latency and costs. Although recent advancements in parallel function calling have improved tool execution per API call, they may necessitate more detailed in-context instructions and task breakdown at the prompt level, resulting in higher engineering and production costs. Inspired by the hardware design principles of multiply-add (MAD) operations, which fuse multiple arithmetic operations into a single task from the compiler's perspective, we propose LLM-Tool Compiler, which selectively fuses similar types of tool operations under a single function at runtime, presenting them as a unified task to the LLM. This selective fusion inherently enhances parallelization and efficiency. Benchmarked on a large-scale Copilot platform, LLM-Tool Compiler achieves up to four times more parallel calls than existing methods, reducing token costs and latency by up to 40% and 12%, respectively.

Can we assess a priori how well a knowledge graph embedding will perform on a specific downstream task and in a specific part of the knowledge graph? Knowledge graph embeddings (KGEs) represent entities (e.g., "da Vinci," "Mona Lisa") and relationships (e.g., "painted") of a knowledge graph (KG) as vectors. KGEs are generated by optimizing an embedding score, which assesses whether a triple (e.g., "da Vinci," "painted," "Mona Lisa") exists in the graph. KGEs have been proven effective in a variety of web-related downstream tasks, including, for instance, predicting relationships among entities. However, the problem of anticipating the performance of a given KGE in a certain downstream task and locally to a specific individual triple, has not been tackled so far. In this paper, we fill this gap with ReliK, a Reliability measure for KGEs. ReliK relies solely on KGE embedding scores, is task- and KGE-agnostic, and requires no further KGE training. As such, it is particularly appealing for semantic web applications which call for testing multiple KGE methods on various parts of the KG and on each individual downstream task. Through extensive experiments, we attest that ReliK correlates well with both common downstream tasks, such as tail or relation prediction and triple classification, as well as advanced downstream tasks, such as rule mining and question answering, while preserving locality.

Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. In this paradigm, clients actively engage in training by exchanging models with their networked neighbors. However, DFL introduces increased costs in terms of training and communication. Existing methods focus on minimizing communication often overlooking training efficiency and data heterogeneity. To address this gap, we propose a novel \textit{sparse-to-sparser} training scheme: DA-DPFL. DA-DPFL initializes with a subset of model parameters, which progressively reduces during training via \textit{dynamic aggregation} and leads to substantial energy savings while retaining adequate information during critical learning periods. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs. We provide a theoretical analysis of DA-DPFL's convergence by solidifying its applicability in decentralized and personalized learning. The code is available at:https://github.com/EricLoong/da-dpfl, Comment: 15 pages, 9 figures, 3 pages theory

We investigate the learning dynamics of fully-connected neural networks through the lens of gradient signal-to-noise ratio (SNR), examining the behavior of first-order optimizers like Adam in non-convex objectives. By interpreting the drift/diffusion phases in the information bottleneck theory, focusing on gradient homogeneity, we identify a third phase termed ``total diffusion", characterized by equilibrium in the learning rates and homogeneous gradients. This phase is marked by an abrupt SNR increase, uniform residuals across the sample space and the most rapid training convergence. We propose a residual-based re-weighting scheme to accelerate this diffusion in quadratic loss functions, enhancing generalization. We also explore the information compression phenomenon, pinpointing a significant saturation-induced compression of activations at the total diffusion phase, with deeper layers experiencing negligible information loss. Supported by experimental data on physics-informed neural networks (PINNs), which underscore the importance of gradient homogeneity due to their PDE-based sample inter-dependence, our findings suggest that recognizing phase transitions could refine ML optimization strategies for improved generalization.

Nowadays, the use of advanced sensors, such as terrestrial 3D laser scanners, mobile LiDARs and Unmanned Aerial Vehicles (UAV) photogrammetric imaging, has become the prevalent practice for 3D Reality Modeling and digitization of large-scale monuments of Cultural Heritage (CH). In practice, this process is heavily related to the expertise of the surveying team, handling the laborious planning and time-consuming execution of the 3D mapping process that is tailored to the specific requirements and constraints of each site. To minimize human intervention, this paper introduces a novel methodology for autonomous 3D Reality Modeling for CH monuments by employing au-tonomous biomimetic quadrupedal robotic agents and UAVs equipped with the appropriate sensors. These autonomous robotic agents carry out the 3D RM process in a systematic and repeatable ap-proach. The outcomes of this automated process may find applications in digital twin platforms, facilitating secure monitoring and management of cultural heritage sites and spaces, in both indoor and outdoor environments.

Artificial Intelligence (AI) in recent years has shown an unprecedentedly impressive development, tending to play a catalytic role in all aspects of life. The interest of the academic community, but also of governments, is huge in the dynamics of AI and is reflected by the truly explosive amount of investment and research that is underway. Enthusiastic opinions and statements about AI are made every day, but at the same time they also bring to the fore alarming predictions about its effects. This paper aims to describe the opportunities emerging from the use of artificial intelligence and ChatGPT to improve education, but also to identify the challenges and ethical issues that arise., Comment: 6 pages

Preserving and safeguarding the Cultural Heritage (CH) of our world from unforeseen hazards should be viewed as a collective responsibility for humanity. Consequently, there is a growing imperative for targeted measures aimed at conserving, rejuvenating, and safeguarding historical assets that carry cultural significance. In recent times, Terrestrial Laser Scanning (TLS), Unmanned Aerial Vehicle (UAV) Photogrammetry, and applications in Mixed Reality (MR) have assumed a pivotal role in the mapping, recording, preservation, and promotion of Cultural Heritage. This ar-ticle endeavors to present a comprehensive approach spanning from 3D surveying to the 3D representation and promotion of Religious Cultural Heritage, offering an overview of the applied methodologies. Through the integration of TLS and UAV photogrammetry techniques, a comprehensive digital record of Panagia Ekatontapyliani, the adjoining Church of Agios Nikolaos, and the Baptistery, along with their wall paintings (hagiographies) and natural surroundings, has been obtained. This record serves as the foundation for historical documentation and recreation using the HBIM concept, paving the way for the development of diverse Mixed Reality applications. These applications aim to enhance the visibility, accessibility, and visitability of the Monument.

We confront massive Proca-Nuevo gravity with cosmological observations. The former is a non-linear theory involving a massive spin-1 field, that can be extended incorporating operators of the Generalized Proca class, and when coupled to gravity it can be covariantized in a way that exhibits consistent and ghost-free cosmological solutions, without experiencing instabilities and superluminalities at the perturbative level. When applied at a cosmological framework it induces extra terms in the Friedmann equations, however due to the special non-linear construction the field is eliminated in favor of the Hubble function. Thus, the resulting effective dark energy sector is dynamical, however it contains the same number of free parameters with the $\Lambda$CDM concordance model. We use data from Supernovae Ia (SNIa) and Cosmic Chronometers (CC) observations and we construct the corresponding likelihood-contours for the free parameters. Interestingly enough, application of various information criteria, such as AIC, BIC and DIC, shows that the scenario of massive Proca-Nuevo gravity, although having exactly the same number of free parameters with $\Lambda$CDM paradigm, it is more efficient in fitting the data. Finally, the reconstructed dark-energy equation-of-state parameter shows statistical compatibility with the model-independent, data-driven reconstructed one., Comment: 19 pages, 2 figures, 2 Tables, version published in JCAP

Deep Neural Networks (DNNs) have shown significant advantages in a wide variety of domains. However, DNNs are becoming computationally intensive and energy hungry at an exponential pace, while at the same time, there is a vast demand for running sophisticated DNN-based services on resource constrained embedded devices. In this paper, we target energy-efficient inference on embedded DNN accelerators. To that end, we propose an automated framework to compress DNNs in a hardware-aware manner by jointly employing pruning and quantization. We explore, for the first time, per-layer fine- and coarse-grained pruning, in the same DNN architecture, in addition to low bit-width mixed-precision quantization for weights and activations. Reinforcement Learning (RL) is used to explore the associated design space and identify the pruning-quantization configuration so that the energy consumption is minimized whilst the prediction accuracy loss is retained at acceptable levels. Using our novel composite RL agent we are able to extract energy-efficient solutions without requiring retraining and/or fine tuning. Our extensive experimental evaluation over widely used DNNs and the CIFAR-10/100 and ImageNet datasets demonstrates that our framework achieves $39\%$ average energy reduction for $1.7\%$ average accuracy loss and outperforms significantly the state-of-the-art approaches., Comment: 14 pages, 9 figures

Background: Ontologies are fundamental components of informatics infrastructure in domains such as biomedical, environmental, and food sciences, representing consensus knowledge in an accurate and computable form. However, their construction and maintenance demand substantial resources and necessitate substantial collaboration between domain experts, curators, and ontology experts. We present Dynamic Retrieval Augmented Generation of Ontologies using AI (DRAGON-AI), an ontology generation method employing Large Language Models (LLMs) and Retrieval Augmented Generation (RAG). DRAGON-AI can generate textual and logical ontology components, drawing from existing knowledge in multiple ontologies and unstructured text sources. Results: We assessed performance of DRAGON-AI on de novo term construction across ten diverse ontologies, making use of extensive manual evaluation of results. Our method has high precision for relationship generation, but has slightly lower precision than from logic-based reasoning. Our method is also able to generate definitions deemed acceptable by expert evaluators, but these scored worse than human-authored definitions. Notably, evaluators with the highest level of confidence in a domain were better able to discern flaws in AI-generated definitions. We also demonstrated the ability of DRAGON-AI to incorporate natural language instructions in the form of GitHub issues. Conclusions: These findings suggest DRAGON-AI's potential to substantially aid the manual ontology construction process. However, our results also underscore the importance of having expert curators and ontology editors drive the ontology generation process.

The unification of quantum mechanics and general relativity has long been elusive. Only recently have empirical predictions of various possible theories of quantum gravity been put to test. The dawn of multi-messenger high-energy astrophysics has been tremendously beneficial, as it allows us to study particles with much higher energies and travelling much longer distances than possible in terrestrial experiments, but more progress is needed on several fronts. A thorough appraisal of current strategies and experimental frameworks, regarding quantum gravity phenomenology, is provided here. Our aim is twofold: a description of tentative multimessenger explorations, plus a focus on future detection experiments. As the outlook of the network of researchers that formed through the COST Action CA18108 "Quantum gravity phenomenology in the multi-messenger approach (QG-MM)", in this work we give an overview of the desiderata that future theoretical frameworks, observational facilities, and data-sharing policies should satisfy in order to advance the cause of quantum gravity phenomenology., Comment: Submitted to CQG for the Focus Issue on "Quantum Gravity Phenomenology in the Multi-Messenger Era: Challenges and Perspectives". Please contact us to express interesst of endorsement of this white paper

Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve \textit{extreme} sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip., Comment: Accepted for publication at ICDM 2023 (Full version in arxiv). The associated code is available at https://github.com/EricLoong/feddip

This paper aims at a brief overview of the main impact of ChatGTP in the scientific field of programming and learning/education in computer science. It lists, covers and documents from the literature the major issues that have been identified for this topic, such as applications, advantages and limitations, ethical issues raised. Answers to the above questions were solicited from ChatGPT itself, the responses were collected, and then the recent literature was surveyed to determine whether or not the responses are supported. The paper ends with a short discussion on what is expected to happen in the near future. A future that can be extremely promising if humanity manages to have AI as a proper ally and partner, with distinct roles and specific rules of cooperation and interaction., Comment: 16 pages

Experimental data can aid in gaining insights about a system operation, as well as determining critical aspects of a modelling or simulation process. In this paper, we analyze the data acquired from an extensive experimentation process in a serverless Function as a Service system (based on the open source Apache Openwhisk) that has been deployed across 3 available cloud/edge locations with different system setups. Thus, they can be used to model distribution of functions through multi-location aware scheduling mechanisms. The experiments include different traffic arrival rates, different setups for the FaaS system, as well as different configurations for the hardware and platform used. We analyse the acquired data for the three FaaS system setups and discuss their differences presenting interesting conclusions with relation to transient effects of the system, such as the effect on wait and execution time. We also demonstrate interesting trade-offs with relation to system setup and indicate a number of factors that can affect system performance and should be taken under consideration in modelling attempts of such systems., Comment: 24 pages, 14 Figures, Journal paper

Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy properties. This has introduced application workloads that comprise of multiple DNN applications, raising new challenges regarding workload distribution. Equipped with a diverse set of accelerators, newer embedded system present architectural heterogeneity, which current run-time controllers are unable to fully utilize. To enable high throughput in multi-DNN workloads, such a controller is ought to explore hundreds of thousands of possible solutions to exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a lightweight and extensible multi-DNN manager for heterogeneous embedded devices. We leverage stochastic space exploration and we combine it with a highly accurate performance estimator to observe a x4.6 average throughput boost compared to other state-of-the-art methods. The evaluation was performed on the HiKey970 development board.

The Lorentzian type IIB matrix model is a promising candidate for a non-perturbative formulation of superstring theory. However, it was recently found that a Euclidean space-time appears in the conventional large-$N$ limit. In this work, we study the model with a Lorentz invariant mass term which can be considered as an IR regulator. By performing complex Langevin simulations to overcome the sign problem, we observe the emergence of expanding space-time with Lorentzian signature., Comment: 13 pages, 9 figures, proceedings of Corfu Summer Institute 2022 "School and Workshops on Elementary Particle Physics and Gravity"

As the number of devices being interconnected increases, so does also the demand for (lightweight) security. To this end, Physical Unclonable Functions (PUFs) have been proposed as hardware primitives that can act as roots of trust and security. Recently, a new type of PUF based on Carbon NanoTubes (CNTs) has been proposed. At the same time, attacks and testing based on direct electrical probing appear to be moving towards non-invasive techniques. In this context, this work attempts to examine the potential for practical non-invasive probing attacks against the CNT-PUF, a novel PUF based on CNTs. Our results indicate that direct probing might potentially compromise the security of this PUF. Nevertheless, we note that this holds true only in the case that the attacker can directly probe the wire corresponding to the secret value of each CNT-PUF cell. Thus, we can conclude that the examined CNT-PUFs are rather resilient to direct probing attacks, that non-invasive probing methods appear to be promising for testing such PUFs, and that, in order for the attacker to gain the full-length value of the secret, all the relevant channels would need to be probed. Nevertheless, as our work proves, practical non-invasive attacks against the CNT-PUF are feasible and adequate countermeasures need to be employed in order to address this issue.

Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs, that accelerates the convergence of dynamic or static systems. This simple yet effective attention mechanism is a function of the evolving cumulative residuals and aims to make the optimizer aware of problematic regions at no extra computational cost or adversarial learning. We illustrate that this general method consistently achieves a relative $L^{2}$ error of the order of $10^{-5}$ using standard optimizers on typical benchmark cases of the literature. Furthermore, by investigating the evolution of weights during training, we identify two distinct learning phases reminiscent of the fitting and diffusion phases proposed by the information bottleneck (IB) theory. Subsequent gradient analysis supports this hypothesis by aligning the transition from high to low signal-to-noise ratio (SNR) with the transition from fitting to diffusion regimes of the adopted weights. This novel correlation between PINNs and IB theory could open future possibilities for understanding the underlying mechanisms behind the training and stability of PINNs and, more broadly, of neural operators.

Virtual and augmented reality are currently enjoying a great deal of attention from the research community and the industry towards their adoption within industrial spaces and processes. However, the current design and implementation landscape is still very fluid, while the community as a whole has not yet consolidated into concrete design directions, other than basic patterns. Other open issues include the choice over a cloud or edge-based architecture when designing such systems. Within this work, we present our approach for a monitoring intervention inside a factory space utilizing both VR and AR, based primarily on edge computing, while also utilizing the cloud. We discuss its main design directions, as well as a basic ontology to aid in simple description of factory assets. In order to highlight the design aspects of our approach, we present a prototype implementation, based on a use case scenario in a factory site, within the context of the ENERMAN H2020 project., Comment: Preprint submitted to the 21st IEEE International Conference on Industrial Informatics, INDIN'23