Search

Following the needs of industrial applications, virtualization has emerged as one of the most effective approaches for the consolidation of mixed-criticality systems while meeting tight constraints in terms of space, weight, power, and cost (SWaP-C). In embedded platforms with homogeneous processors, a wealth of works have proposed designs and techniques to enforce spatio-temporal isolation by leveraging well-understood virtualization support. Unfortunately, achieving the same goal on heterogeneous MultiProcessor Systems-on-Chip (MPSoCs) has been largely overlooked. Modern hypervisors are designed to operate exclusively on main cores, with little or no consideration given to other co-processors within the system, such as small microcontroller-level CPUs or soft-cores deployed on programmable logic (FPGA). Typically, hypervisors consider co-processors as I/O devices allocated to virtual machines that run on primary cores, yielding full control and responsibility over them. Nevertheless, inadequate management of these resources can lead to spatio-temporal isolation issues within the system. In this paper, we propose the Omnivisor model as a paradigm for the holistic management of heterogeneous platforms. The model generalizes the features of real-time static partitioning hypervisors to enable the execution of virtual machines on processors with different Instruction Set Architectures (ISAs) within the same MPSoC. Moreover, the Omnivisor ensures temporal and spatial isolation between virtual machines by integrating and leveraging a variety of hardware and software protection mechanisms. The presented approach not only expands the scope of virtualization in MPSoCs but also enhances the overall system reliability and real-time performance for mixed-criticality applications. A full open-source reference implementation of the Omnivisor based on the Jailhouse hypervisor is provided, targeting ARM real-time processing units and RISC-V soft-cores on FPGA. Experimental results

Machine-learning (ML) technology has been a key enabler in the push towards realizing ever more sophisticated autonomous-driving features. In deploying such technology, the automotive industry has relied heavily on using "black-box" software and hardware components that were originally intended for non-safety-critical contexts, without a full understanding of their real-time capabilities. A prime example of such a component is CUDA, which is fundamental to the acceleration of ML algorithms using NVIDIA GPUs. In this paper, evidence is presented demonstrating that CUDA can cause unbounded task delays. Such delays are the result of CUDA’s usage of synchronization mechanisms in the POSIX thread (pthread) library, so the latter is implicated as a delay-prone component as well. Such synchronization delays are shown to be the source of a system failure that occurred in an actual autonomous vehicle system during testing at WeRide. Motivated by these findings, a broader experimental study is presented that demonstrates several real-time deficiencies in CUDA, the glibc pthread library, Linux, and the POSIX interface of the safety-certified QNX Operating System for Safety. Partial mitigations for these deficiencies are presented and further actions are proposed for real-time researchers and developers to integrate more complete mitigations.

Machine-learning (ML) technology has been a key enabler in the push towards realizing ever more sophisticated autonomous-driving features. In deploying such technology, the automotive industry has relied heavily on using "black-box" software and hardware components that were originally intended for non-safety-critical contexts, without a full understanding of their real-time capabilities. A prime example of such a component is CUDA, which is fundamental to the acceleration of ML algorithms using NVIDIA GPUs. In this paper, evidence is presented demonstrating that CUDA can cause unbounded task delays. Such delays are the result of CUDA’s usage of synchronization mechanisms in the POSIX thread (pthread) library, so the latter is implicated as a delay-prone component as well. Such synchronization delays are shown to be the source of a system failure that occurred in an actual autonomous vehicle system during testing at WeRide. Motivated by these findings, a broader experimental study is presented that demonstrates several real-time deficiencies in CUDA, the glibc pthread library, Linux, and the POSIX interface of the safety-certified QNX Operating System for Safety. Partial mitigations for these deficiencies are presented and further actions are proposed for real-time researchers and developers to integrate more complete mitigations.

Following the needs of industrial applications, virtualization has emerged as one of the most effective approaches for the consolidation of mixed-criticality systems while meeting tight constraints in terms of space, weight, power, and cost (SWaP-C). In embedded platforms with homogeneous processors, a wealth of works have proposed designs and techniques to enforce spatio-temporal isolation by leveraging well-understood virtualization support. Unfortunately, achieving the same goal on heterogeneous MultiProcessor Systems-on-Chip (MPSoCs) has been largely overlooked. Modern hypervisors are designed to operate exclusively on main cores, with little or no consideration given to other co-processors within the system, such as small microcontroller-level CPUs or soft-cores deployed on programmable logic (FPGA). Typically, hypervisors consider co-processors as I/O devices allocated to virtual machines that run on primary cores, yielding full control and responsibility over them. Nevertheless, inadequate management of these resources can lead to spatio-temporal isolation issues within the system. In this paper, we propose the Omnivisor model as a paradigm for the holistic management of heterogeneous platforms. The model generalizes the features of real-time static partitioning hypervisors to enable the execution of virtual machines on processors with different Instruction Set Architectures (ISAs) within the same MPSoC. Moreover, the Omnivisor ensures temporal and spatial isolation between virtual machines by integrating and leveraging a variety of hardware and software protection mechanisms. The presented approach not only expands the scope of virtualization in MPSoCs but also enhances the overall system reliability and real-time performance for mixed-criticality applications. A full open-source reference implementation of the Omnivisor based on the Jailhouse hypervisor is provided, targeting ARM real-time processing units and RISC-V soft-cores on FPGA. Experimental results

The goal of a real-time locking protocol is to reduce any priority-inversion blocking (pi-blocking) a task may incur while waiting to access a shared resource. For mutual-exclusion sharing on an m-processor platform, the best existing lower bound on per-task pi-blocking under suspension-oblivious analysis is a (trivial) lower bound of (m-1) request lengths under any job-level fixed-priority (JLFP) scheduler. Surprisingly, most asymptotically optimal locking protocols achieve a per-task pi-blocking upper bound of (2m-1) request lengths under JLFP scheduling, even though a range of very different mechanisms are used in these protocols. This paper closes the gap between these existing lower and upper bounds by establishing a lower bound of (2m-2) request lengths under global fixed-priority (G-FP) and global earliest-deadline-first (G-EDF) scheduling. This paper also shows that worst-case per-task pi-blocking can be arbitrarily close to (2m-1) request lengths for locking protocols that satisfy a certain property that is met by most (if not all) existing locking protocols. These results imply that most known asymptotically optimal locking protocols are almost truly optimal (not just asymptotic) under G-FP and G-EDF scheduling.

Scheduling real-time tasks that utilize GPUs with analyzable guarantees poses a significant challenge due to the intricate interaction between CPU and GPU resources, as well as the complex GPU hardware and software stack. While much research has been conducted in the real-time research community, several limitations persist, including the absence or limited availability of GPU-level preemption, extended blocking times, and/or the need for extensive modifications to program code. In this paper, we propose GCAPS, a GPU Context-Aware Preemptive Scheduling approach for real-time GPU tasks. Our approach exerts control over GPU context scheduling at the device driver level and enables preemption of GPU execution based on task priorities by simply adding one-line macros to GPU segment boundaries. In addition, we provide a comprehensive response time analysis of GPU-using tasks for both our proposed approach as well as the default Nvidia GPU driver scheduling that follows a work-conserving round-robin policy. Through empirical evaluations and case studies, we demonstrate the effectiveness of the proposed approaches in improving taskset schedulability and response time. The results highlight significant improvements over prior work as well as the default scheduling approach, with up to 40% higher schedulability, while also achieving predictable worst-case behavior on Nvidia Jetson embedded platforms.

Deep Neural Networks (DNNs) are becoming common in "learning-enabled" time-critical applications such as autonomous driving and robotics. One approach to protect DNN inference from adversarial actions and preserve model privacy/confidentiality is to execute them within trusted enclaves available in modern processors. However, running DNN inference inside limited-capacity enclaves while ensuring timing guarantees is challenging due to (a) large size of DNN workloads and (b) extra switching between "normal" and "trusted" execution modes. This paper introduces new time-aware scheduling schemes - DeepTrust^RT - to securely execute deep neural inferences for learning-enabled real-time systems. We first propose a variant of EDF (called DeepTrust^RT-LW) that slices each DNN layer and runs them sequentially in the enclave. However, due to extra context switch overheads of individual layer slices, we further introduce a novel layer fusion technique (named DeepTrust^RT-FUSION). Our proposed scheme provides hard real-time guarantees by fusing multiple layers of DNN workload from multiple tasks; thus allowing them to fit and run concurrently within the enclaves while maintaining real-time guarantees. We implemented and tested DeepTrust^RT ideas on the Raspberry Pi platform running OP-TEE+DarkNet-TZ DNN APIs and three DNN workloads (AlexNet-squeezed, Tiny Darknet, YOLOv3-tiny). Compared to the layer-wise partitioning approach (DeepTrust^RT-LW), DeepTrust^RT-FUSION can schedule up to 3x more tasksets and reduce context switches by up to 11.12x. We further demonstrate the efficacy of DeepTrust^RT using a flight controller (ArduPilot) case study and find that DeepTrust^RT-FUSION retains real-time guarantees where DeepTrust^RT-LW becomes unschedulable.

This article implements and validates the performance of a virtual intelligent electronic device (vIED) framework for digital substations using a real-time (RT) simulation environment. The work looks toward the future design of protection, automation, and control systems, an evolution of the digital substation design based on IEC 61850 and virtualization technology. An RT simulation setup was developed to speed up and enhance the deployment and maintenance of vIEDs with a novel well-defined test methodology. Several scenarios were tested by varying the number of vIEDs and relevant (communication, scalability, and functionality) configuration criteria. In addition, we assessed the efficiency of a software-defined communication network for the vIED framework and its adaptability to dynamic scaling of the network under transient data traffic loads. The tests demonstrated the efficient performance of vIEDs across various system configurations, particularly for requirements on the response time and network transfer latency. The findings showcase the significance of proper design and testing methodology that can be benchmarked against other virtualization platforms for substation systems.

The development of machine learning and artificial intelligence algorithms, as well as the progress of unmanned aerial vehicle swarm technology, has significantly enhanced the intelligence and autonomy of unmanned aerial vehicles in search missions, resulting in greater efficiency when searching unknown areas. However, as search scenarios become more complex, the existing unmanned aerial vehicle swarm search method lacks scalability and efficient cooperation. Furthermore, due to the increasing scale of search scenarios, the accuracy and real-time performance of global information are difficult to ensure, necessitating the provision of local information. This paper focuses on the large-scale search scenario and split it to provide both local and global information for running unmanned aerial vehicle swarm search algorithms. Since the search environment is often unknown, dynamic, and complex, it requires adaptive decision-making in a constantly changing environment, which is suitable for modeling as a Markov decision process. Considering the sequential-based scenario, we propose a distributed collaborative search method based on a multi-agent reinforcement learning algorithm, which can operate efficiently in complex and large-scale scenarios. Additionally, the proposed method can utilize a convolutional neural network to process high-dimensional map data with almost no loss of the structure information. Experimental results demonstrate that the proposed method can collaboratively search unknown areas, avoid collisions and repetitions, and find all targets faster compared with the benchmarks.

We introduce a novel learning method that can effectively perceive both the geometry structure and semantic labels of a 3D scene in real time. Existing real-time 3D scene reconstruction approaches often rely on volumetric schemes to regress a Truncated Signed Distance Function (TSDF) as the 3D representation. However, these volumetric approaches primarily focus on ensuring global coherence in the reconstructed scene, which often results in a lack of local geometric detail. To address this limitation, we propose a solution that leverages the latent geometric knowledge present in 2D image features by explicit depth prediction thereby creating anchored features, which are used to refine the learning of occupancy in the TSDF volume. Furthermore, we discover that this cross-dimensional feature refinement methodology can also be applied to the task of semantic segmentation by utilizing semantic priors. As a result, we propose an end-to-end cross-dimensional refinement neural network (CDRNet) that can extract both the 3D mesh and 3D semantic labeling of a scene in real time. Through experimental evaluation on multiple datasets, we demonstrate that our method achieves state-of-the-art 3D perception capability by boosting over 40% and 18% in 3D semantic segmentation and geometric reconstruction respectively over the prior art. These promising results indicate the significant potential of our approach for various industrial applications. Demo video and code can be found on the project page, https://hafred.github.io/cdrnet/. IEEE

The emergence of the open RISC-V instruction set architecture has enabled the design of highly efficient processor cores by extending the relatively simple architecture with custom functionality specific to the application. The design and simulation of such extensions is traditionally done using hardware description languages (HDLs) which is time-consuming and cumbersome. SyncRim, the synchronous circuit simulation tool presented in this thesis takes a high-level approach to digital circuit design, omitting irrelevant implementation details in a simulator model of a processor, which allows for rapid prototyping of anything from architecture extensions to entire processor cores. We show how a synchronous circuit simulator can be implemented using the Rust programming language, and showcase how such a simulator can be used to aid in hardware/software co-design by implementing a RISC-V core, extending it with the bleeding edge Core-Local Interrupt Controller(CLIC), and porting a Rust real-time operating system, the RTIC framework to the CLIC-extended RISC-V.

Modern edge real-time automotive applications are becoming more complex, dynamic, and distributed, moving away from conventional static operating environments to support advanced driving assistance and autonomous driving functionalities. This shift necessitates formulating more complex task models to represent the evolving nature of these applications aptly. Modeling of real-time automotive systems is typically performed leveraging Architectural Languages (ALs) such as Amalthea, which are commonly used by the industry to describe the characteristics of processing platforms, operating systems, and tasks. However, these architectural languages are originally derived for classical automotive applications and need to evolve to meet the needs of next-generation applications. This paper proposes an automatic framework for the modeling and automatic code generation of dynamic automotive applications under the QNX RTOS. To this end, we extend Amalthea to describe chains of communicating tasks with multiple operating modes and to consider the QNX's reservation-based scheduler, called APS, which allows providing temporal isolation between applications co-located on the same hardware platform. Finally, an evaluation is presented to compare different implementation alternatives under QNX that are automatically generated by our code generation framework., QC 20240813

Automated Human Activity Recognition (HAR) stems from the requirement to seamlessly integrate technology into daily life, to enhance user experience, improve healthcare, provide improved operations, ensure safety, deliver data-driven insights, and address various real-world challenges. However, unscripted Human activity faces challenges that must be understood, and require advances in sensor technology and machine learning models. This paper presents an Active Transfer Learning (ATL) approach for real-time HAR using mobile sensor data. Unlike traditional methods, our approach accounts for both the physical and habitual constraints of individuals. Existing works make an unrealistic assumption of an omniscient oracle while using the same datasets for both training and testing of the models, which makes them impractical for industry requirements. Our proposed approach addresses challenges in existing HAR algorithms, proposing a methodology to adapt models to the real-world datasets while training and testing on cross datasets. We have tailored an existing Entropy and Memory Maximization algorithm to work in a real-time environment while considering user constraints. Primarily trained in a well-labeled controlled environment dataset, we introduce noise injection to prevent the model from overfitting and enhance its generalization for scarcely labeled real-world datasets. Evaluations on publicly available datasets demonstrate our approach achieves 80% - 90% accuracy, outperforming the base algorithm accuracy of 12% - 14%. Importantly, our proposed technique outperforms with limited labeled data, making it valuable for real-time scenarios where labeling is sparse. This research advances HAR in real-world settings, offering improved accuracy and adaptability., QC 20240722

The wide integration of inverter based renewable energy sources (RESs) in modern grids may cause severe voltage violation issues due to high stochastic fluctuations of RESs. Existing centralized approaches can achieve optimal results for voltage regulation, but they have high communication burdens; existing decentralized methods only require local information, but they cannot achieve optimal results. Deep reinforcement learning (DRL) based methods are effective to deal with uncertainties, but it is difficult to guarantee secure constraints in existing DRL training. To address the above challenges, this paper proposes a projection embedded multi-agent DRL algorithm to achieve decentralized optimal control of distribution grids with guaranteed 100% safety. The safety of the DRL training is guaranteed via an embedded safe policy projection, which could smoothly and effectively restrict the DRL agent action space, and avoid any violation of physical constraints in distribution grid operations. The multi-agent implementation of the proposed algorithm enables the optimal solution achieved in a decentralized manner that does not require real-time communication for practical deployment. The proposed method is tested in modified IEEE 33-bus distribution and compared with existing methods; the results validate the effectiveness of the proposed method in achieving decentralized optimal control with guaranteed 100% safety and without the requirement of real-time communications., QC 20240429

Autonomous driving systems critically depend on accurately classifying traffic signs, a task that is jeopardized by adversarial attacks. This paper focuses on the relatively unexplored domain of Traffic Sign Classification (TSC) in the context of detecting adversarial attacks. We conduct a rigorous evaluation of six prominent adversarial example detection methods, each representing a distinct detection category and well-regarded within the research community. Our methodology involves adapting these techniques, originally developed for general image classification (IC), to the unique challenges posed by traffic sign images, characterized by their complexity due to factors like varying environmental conditions and a large number of classes. Our study reveals insights into their effectiveness, with the Natural Scene Statistics (NSS) method outperforming others with 83.42%, 86.42%, and 99.96% detection rates; 0.08%, 0.05%, and 0.04%, false positive rates; and 0.02sec, 0.01sec, and 0.02sec detection times for Chinese, Belgium and German traffic sign datasets, respectively. NSS's superiority is crucial for autonomous vehicles. Our study also sheds light on the often-neglected aspect of detection time in IC, which plays a vital role in ensuring operational efficiency and safety for autonomous vehicles. Our research highlights the need for customized defense strategies tailored to the TSC domain, considering our evaluation's findings. By identifying promising techniques for detecting adversarial attacks in TSC, we contribute to enhancing the safety and robustness of autonomous driving systems. This study fills a critical knowledge gap, providing valuable insights into understanding and defending against adversarial attacks, specifically in the TSC context.

In military surveillance and combat systems, real-time data needs to be accessible quickly and reliably by multiple processes simultaneously. In a defense industry context, reliability and performance are crucial factors. A case study is conducted on a legacy system at Saab which handles real-time data in application-specific structures. By integrating a relational database into this system, the goal is to increase the volume of data the system can handle. To evaluate this possibility, a prototype is developed that features an Ada server and a Java client from the legacy system, as well as an additional PostgreSQL database to manage the data. A challenge is that the database as well as the software handling the database connections need to be adapted to the original system. The maximum data volume of sending and modifying data through the prototype is measured and compared to the performance of the original system. This report shows 1) insights on how to integrate a database into systems featuring legacy components written in Ada and Java as well as 2) insights into the performance benefits of handling quickly changing real-time data in a relational database. The result is that the bandwidth in the prototype is 725% higher than in the original system., Militäriska övervaknings- och stridsledningssystem kräver tillgång till realtidsdata så fort som möjligt från flera processer samtidigt. Även prestanda och pålitlighet är viktiga faktorer för applikationer i försvarsindustrin. En fallstudie genomförs på ett befintligt system på Saab som numera hanterar data i applikationsspecifika strukturer. Möjligheten av integreringen av en relationsdatabas undersöks som ska öka prestandan i systemet. En prototyp utvecklas i syfte att undersöka denna fråga. Prototypen består av en Ada server och en Java klient från det befintliga systemet såväl som en ny PostgreSQL databas som hanterar realtidsdata. En utmaning är att alla processer i datahantering ska vara anpassade till det nuvarande systemet. Prestanda tester genomförs i prototypen för att utreda den maximala datavolymen som prototypen kan hantera. Resultaten jämförs med den maximala prestandan i det befintliga systemet. Denna rapport visar 1) aspekter för integrationen av en relationsdatabas i ett befintligt realtidssystem med komponenter i Ada och Java såväl som 2) resultat om prestanda ökningen genom att hantera realtidsdata i en relationsdatabas. Det visas att bandbredden i prototypen är 725% större än bandbredden i det nuvarande systemet., Militärische Überwachungs- und Kommunikationssysteme erfordern Zugriff auf Echtzeitdaten, welche schnell und von mehreren Prozessen zugleich erreichbar sind. Applikationen für die Verteidigungsindustrie erfordern einen hohen Grad von Leistung und Zuverlässigkeit. Eine Fallstudie wird an einem vorhandenen System von Saab durchgeführt, in dem momentan Echtzeitdaten in applikationsspezifischen Strukturen gelagert und verarbeitet werden. Um die Leistung dieses Systems zu verbessern, wird die Integration einer relationalen Datenbank in Betracht gezogen. Um diese Möglichkeit genauer zu untersuchen, wird ein Prototyp zu Studienzwecken entwickelt. Dieser Prototyp besteht aus einem Ada Server und einem Java Klienten des Altsystems, sowie einer PostgreSQL Datenbank, welche für das Testsystem entwickelt wird und welche die Lagerung der Echtzeitdaten abwickelt. Eine Herausforderung ist, dass alle Prozesse des Prototyps sowie das Datenformat an das Altsystem angepasst werden müssen. Die maximalen Kapazitäten des Prototyps werden durch Tests ermittelt und mit denen des Altsystems verglichen. Diesese Arbeit zeigt 1) Einblicke in die Integration eines relationalen Datenbanksystems in Altsysteme mit Komponenten in Ada und Java sowie 2) Ergebnisse über die potenzielle Leistungssteigerung eines Systems durch die Lagerung von Echtzeitdaten in einer relationalen Datenbank. Die Resultate zeigen, dass die Bandbreite des Prototyps 725% höher ist als die des bisherigen Systems.

Robot operating system (ROS) is a widely used robotic software development framework. In safety-critical applications that require timing guarantees, the first generation of ROS falls short. The introduction of ROS 2 has addressed some of these limitations, but its multithreaded executor still struggles to meet real-time requirements. To address this issue, we design a new multithreaded executor called RTeX for ROS 2. The goal of RTeX is to improve system performance in terms of both run-time efficiency and timing predictability. We have implemented RTeX in the latest version of ROS 2 and conducted experiments on a real platform. The experimental results demonstrate that RTeX outperforms both the default ROS 2 multithreaded executor and its state-of-the-art variant, achieving significant real-time performance improvements.

This study focuses on optimizing resource monitoring and management (RMM) of artificial intelligence (AI) systems. The study begins by discussing the fundamental concepts of Digital Twins and cloud collaboration platforms. Subsequently, a collaborative platform called the distributed Digital Twins cloud is designed, incorporating blockchain technology (BCT) to enhance its security. The utilization of cryptographic algorithms in the BCT ensures data security. The study then presents a comprehensive evaluation of the designed BCT-based distributed Digital Twins cloud. The evaluation results demonstrate that the proposed distributed Digital Twins cloud achieves job scheduling times ranging from approximately 35ms to 40ms during training, with the shortest and longest durations observed. The computed data consistency by the proposed platform ranges from 90% to 98%. In the test set, the proposed model achieves job allocation times ranging from approximately 30ms to 35ms, with the shortest and longest durations observed. The data consistency computed by the platform ranges from approximately 90% to 98%, with the lowest and highest values. The network resource management technology model developed in this study exhibits higher efficiency and intelligence, and its performance is comprehensively evaluated. The research findings indicate that the designed model enhances the data processing capability of the cloud platform and effectively improves the security of platform data management, providing substantial support for the advancement of Digital Twins platforms. Moreover, this study offers technical support for the application, design, and optimization of AI technology in network RMM, providing insights into the efficient application and innovation of AI and network technology for future societal development. It contributes to the profound development of AI technology.

This study aims to investigate the deployment and maintenance of Digital Twin (DT) in an industrial ring rolling manufacturing process. Three parameters were analyzed (i) data cleaning and processing, (ii) DT development and updating, and (iii) secure communication. A process-based DT model was developed using iba-ag and Python, prioritizing security throughout the process. The study underscores the significance of secure DT system, with particular attention to the challenges of real-time updates affected by process operations and environmental factors. This emphasizes the adaptability of the DT while acknowledging the expertise of process operators. The findings promote a deeper understanding of DT implementation, particularly for process developers, maintenance personnel, and operators. The research highlights the importance of security in DT systems, especially when dealing with industrial data influenced by production and environmental factors. Therefore, by focusing on security and deployment in an open-source environment, this study contributes to the practical use of DT systems for predictive maintenance and process optimization within the industrial settings.

Increasing the deployment of Renewable Energy Resources (RES) , along with innovations in Information and Communication Technologies (ICT), would allow prosumers to engage in the energy market and trade their excess energy with each other and with the main grid. To ensure an efficient and safe operation of energy trading, the Peer-to-Peer (P2P) energy trading approach has emerged as a viable paradigm to provide the necessary flexibility and coordinate the energy sharing between a pair of prosumers. The P2P approach is based on the concept of decentralized energy trading between prosumers (i.e., production capabilities or energy consumers). However, security protection and real-time transaction issues in the P2P market present serious challenges. In this paper, we propose a decentralized P2P energy trading approach for the energy market with high penetration of RE. First, the P2P energy market platform proposed coordinating the energy trading between energy providers and consumers to maximize their social welfare. A distributed algorithm is applied to solve the market-clearing problem based on the Alternating Direction Method of Multipliers (ADMM). In this way, the computational complexity can be reduced. Furthermore, a P2P Manager (P2PM) utility is introduced as an entity to solve the synchronization problem between peers during the market clearing. Finally, through a real-time application using Hardware-In-the-Loop (HIL), the effectiveness of the proposed P2PM is verified in terms of synchronizing the market participants and improving the power transaction.

An autonomous service robot should be able to interact with its environment safely and robustly without requiring human assistance. Unstructured environments are challenging for robots since the exact prediction of outcomes is not always possible. Even when the robot behaviors are well-designed, the unpredictable nature of the physical robot-object interaction may lead to failures in object manipulation. In this letter, we focus on detecting and classifying both manipulation and post-manipulation phase failures using the same exteroception setup. We cover a diverse set of failure types for primary tabletop manipulation actions. In order to detect these failures, we propose FINO-Net (Inceoglu et al., 2021), a deep multimodal sensor fusion-based classifier network architecture. FINO-Net accurately detects and classifies failures from raw sensory data without any additional information on task description and scene state. In this work, we use our extended FAILURE dataset (Inceoglu et al., 2021) with 99 new multimodal manipulation recordings and annotate them with their corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80 failure classification F1 scores. Experimental results show that FINO-Net is also appropriate for real-time use. © 2016 IEEE., Funding: The Scientific and Technological Research Council of Türkiye under Grant 119E-436.

The rising demand for electric vehicles (EVs) in the face of limited grid capacity encourages the development and implementation of smart charging (SC) algorithms. Experimental validation plays a pivotal role in advancing this field. This article formulates a hierarchical mixed integer programming EV SC algorithm designed for low voltage (LV) distribution grid applications. A flexible receding horizon scheme is introduced in response to system uncertainties. It also considers the practical constraints in protocols, such as IEC/ISO 15118 and IEC 61851-1. The proposed algorithm is verified and assessed in a power hardware-in-the-loop testbed that incorporates models of real LV distribution grids. Furthermore, the algorithm's capabilities are examined through eight scenarios, out of which four focus on the uncertainties of the input data and two address the engagement of extra grid capacity restrictions. The results demonstrate that the SC algorithm adequately lowers the EV charging cost while fulfilling the charging demand, and substantially reduces the peak power as well as the overloading duration, even when faced with input data uncertainty. The additional grid restrictions in place are proven to improve peak demand reduction and overloading mitigation further. Finally, the limitations and potentials of the developed algorithm are scrutinized., DC systems, Energy conversion & Storage

Drivers initiate a discretionary lane change when they perceive an anticipated improvement in their own driving condition from moving to another lane. However, such a lane change can slow down other vehicles on the target lane, and even worse initiate a disturbance. In this work, we argue that the blocking effect triggered by individual lane changes results from the heterogeneity in the desired speeds of vehicles, and thus using desired speed information of vehicles when regulating lane-changing decisions can improve traffic efficiency. In doing so, our work also exemplifies the usefulness of incorporating user preferences into control decisions. The proposed lane guidance system uses an optimization-based approach to update the target range of desired speeds on each lane in real time, and accordingly recommends individual lane changes. The control system coordinates the lane-changing decisions at the link level, for which the road stretch is subdivided into multiple sections that are controlled independently. We evaluate the performance of the lane guidance system in micro-simulation, for different network demands and desired speed distributions. The results highlight that the proposed approach utilizing the desired speed preferences of drivers results in positive efficiency gains for most traffic compositions in free flow. Moreover, the highest gains are expected in medium to high demand, and when the traffic composition includes a higher proportion of vehicles desiring higher speeds. The gains also increase when the desired speeds of vehicles that want to drive fast and those that want to drive slower are more separated., Transport and Planning, Transport and Planning

With the increasing penetration of distributed energy resources such as photovoltaics (PVs), frequent voltage and congestion problems are expected to occur in distribution grids. Existing solution approaches based on centralized or distributed offline optimization cannot handle the dynamics of distribution grids in real time, i.e., by the time the computations are finished, the used data are already outdated as the grid loadings evolve so fast. Moreover, the unfair distribution of costs or burdens among end-users to relieve network issues, for example with PV curtailment, has not been sufficiently considered either. To address these issues, a fairness-incorporated online feedback optimization (OFO) approach is proposed. This approach produces implementable intermediate iterates for inverter-based assets using online optimization and suitably integrated measurements, and thus can work with the distribution grid dynamics. Moreover, it addresses the unfairness issue by modifying the voltage sensitivity matrices and leverages the feedback-based nature of OFO to compensate for the resulting modeling inaccuracies. Case studies based on a 96-bus low-voltage grid demonstrate the effectiveness of the fairness-incorporated OFO to handle the distribution grid dynamics and improve fairness, in both static and dynamic cases, with balanced and unbalanced grid models.

The performance of time-predictable systems can be improved in multi-core processors using parallel programming models (e.g., OpenMP). However, schedulability analysis of parallel applications is a big challenge due to their sophisticated structure. The common drawbacks of current task-to-thread mapping approaches in OpenMP are that they (i) utilize a global queue in the mapping process, which may increase contention, (ii) do not apply heuristic techniques, which may reduce the predictability and performance of the system, and (iii) use basic analytical techniques, which may cause notable pessimism in the temporal conditions. Accordingly, this paper proposes a task-to-thread mapping method in multi-core processors based on the OpenMP framework. The mapping process is carried out through two phases: allocation and dispatching. Each thread has an allocation queue in order to minimize contention, and the allocation and dispatching processes are performed using several heuristic algorithms to enhance predictability. In the allocation phase, each task-part from the OpenMP DAG is allocated to one of the allocation queues, which includes both sibling and child task-parts. A suitable thread (i.e., allocation queue) is selected using one of the suggested heuristic allocation algorithms. In the dispatching phase, when a thread is idle, a task-part is selected from its allocation queue using one of the suggested heuristic dispatching algorithms and then dispatched to and executed by the thread. The performance of the proposed method is evaluated under different conditions (e.g., varying the number of tasks and the number of threads) in terms of application response time and overhead of the mapping process. The simulation results show that the proposed method surpasses the other methods, especially in the scenario that includes overhead of the mapping. In addition, a prototype implementation of the main heuristics is evaluated using two kernels from real-world applications, sho, This work has been co-funded by the European commission through the AMPERE (H2020 grant agreement no. 745601) project and the RESPECT project (Departament de Recerca i Universitats de la Generalitat de Catalunya record no 2021 PROD 00179)., Peer Reviewed, Postprint (author's final draft)

The number of visually impaired individuals in the world is estimated to be 1 billion, as per WHO reports. Through a thorough examination of existing assistive technology and research, this paper provides a survey of object classification and detection techniques that are used in assistive technology by visually impaired individuals. We discuss the methodology’s drawbacks, s and functionalities of these techniques, and observe how sufficient they are in meeting the needs and requirements of the targeted users, and how they can be improved. As a result of this study, we identify areas with room for improvement in object detection in the assistive technology domain.

In this paper, we consider an Internet-of-Things network supported by several satellites and multiple cache-assisted unmanned aerial vehicles (UAVs). Due to the long-distance transmission and detrimental effects from the transmission environment, the latency can be extremely high, especially in the presence of backhaul congestion. Therefore, we formulate an optimisation problem with the aim of minimising the total network latency. To reduce the complexity of the original problem, it is divided into three sub-problems, namely, clustering ground users associated with UAVs, cache placement in UAVs (to support the network in avoiding backhaul congestion), and power allocation for satellites and UAVs. We propose a distributed optimisation method consisting of: a non-cooperative game is designed to obtain the solution to the clustering problem; a genetic algorithm, which is powerful in the scenario of many variables, is employed to obtain the optimal solution to the high-complexity caching problem; and a quick estimation technique is used for power allocation. Additionally, a centralised optimisation method is presented as a benchmark. Simulation results show that although the distributed method leads to network latency of approximately 30% higher than the centralised method, it takes significantly less time to execute and is suitable for systems requiring strict real-time computing constraints. Furthermore, the numerical results prove the efficiency of our methods compared with other conventional ones. IEEE

The Internet of Things (IoT) represents a swiftly expanding sector that is pivotal in driving the innovation of today's smart services. However, the inherent resource-constrained nature of IoT nodes poses significant challenges in embedding advanced algorithms for cybersecurity, leading to an escalation in cyberattacks against these nodes. Contemporary research in Intrusion Detection Systems (IDS) predominantly focuses on enhancing IDS performance through sophisticated algorithms, often overlooking their practical applicability. This paper introduces Deep-IDS, an innovative and practically deployable Deep Learning (DL)-based IDS. It employs a Long-Short-Term-Memory (LSTM) network comprising 64 LSTM units and is trained on the CIC-IDS2017 dataset. Its streamlined architecture renders Deep-IDS an ideal candidate for edge-server deployment, acting as a guardian between IoT nodes and the Internet against Denial of Service, Distributed Denial of Service, Brute Force, Man-in-the-Middle, and Replay Attacks. A distinctive aspect of this research is the trade-off analysis between the intrusion Detection Rate (DR) and the False Alarm Rate (FAR), facilitating the real-time performance of the Deep-IDS. The system demonstrates an exemplary detection rate of 96.8% at the 70% threshold of DR-FAR trade-off and an overall classification accuracy of 97.67%. Furthermore, Deep-IDS achieves precision, recall, and F1-scores of 97.67%, 98.17%, and 97.91%, respectively. On average, Deep-IDS requires 1.49 seconds to identify and mitigate intrusion attempts, effectively blocking malicious traffic sources. The remarkable efficacy, swift response time, innovative design, and novel defense strategy of Deep-IDS not only secure IoT nodes but also their interconnected sub-networks, thereby positioning Deep-IDS as a leading IDS for IoT-enhanced computer networks.

Propelled by recent advances in Mobile Edge Computing (MEC) and the Internet of Things (IoT), the digital twin technique has been envisioned as a de-facto driving force to bridge the virtual and physical worlds through creating digital portrayals of physical objects. In virtue of the flourishing of edge intelligence and abundant IoT data, data-driven modelling facilitates the implementation and maintenance of digital twins, where simulations of physical objects are usually performed based on Deep Neural Networks (DNNs). A significant advantage of adopting digital twins is to enable decisive prediction on the behaviours of objects in near future without waiting for that really happen. To provide accurate predictions, it is vital to keep each digital twin synchronized with its physical object in real-time. However, it is challenging to maintain the real-time synchronization between a digital twin and its physical object due to the dynamics of physical objects and sensing data drift over time, i.e., the live data from a physical object diverge from the model training data of its digital twin. To address this critical issue, continual learning is a promising solution to retrain models of digital twins incrementally. In this paper, we investigate digital twin synchronization issues via continual learning in an MEC environment, with the aim to maximize the total utility gain, i.e., the enhanced model accuracy. We study two novel optimization problems: the static digital twin synchronization problem per time slot and the dynamic digital twin synchronization problem for a finite time horizon. We first formulate an Integer Linear Program (ILP) solution for the static digital twin synchronization problem when the problem size is small; otherwise, we develop a randomized approximation algorithm at the expense of bounded resource violations for it. We also devise a deterministic approximation algorithm with guaranteed performance for a special case of the static digital twin sy

With the sixth-generation (6G) vision of connecting everything, integrating real-time sensing, communication, computing, and control together in Internet of Everything (IoE) becomes essential to support intelligent services. Such a tendency is facing great challenges in terms of information freshness, energy sustainability, and energy efficiency (EE). Thus, network requires to satisfy multiple objectives, including low age of information (AoI) and high EE. Nevertheless, existing designs mainly focused on AoI and EE separately and has no unified rule in various wireless communications scenarios. In order to efficiently describe the system performances from both information timeliness and energy saving perspectives, this article presents a novel utility-aware performance metric, i.e., age-energy efficiency (AEE), to measure the achieved AoI gain by consuming per joule energy, which takes both age and energy into account. To show the evocativeness of AEE, we explore the AEE in several typical network scenarios, such as the point-to-point (P2P) network, the wireless powered communication network (WPCN), the mobile edge computing (MEC) network and the multi-user network, as examples. The optimal policies to maximize the AEE performance of such network scenarios are also presented. Numerical results show that the AEE is a concave function of the transmit power, and can be employed to pursue the maximal age gain by consuming as less as energy in 6G intelligent network design. Additionally, some insightful promising directions on AEE for research are also presented for 6G networks. IEEE

Real-time systems require locking protocols to coordinate access to shared resources. With the booming revolution of parallel processing technology in real-time systems, there has been some work addressing the problem of extending classic locking protocols for sequential real-time tasks to parallel tasks. However, it may not be most effective to trivially follow the progress mechanisms and queue orders designed for sequential tasks since the intrastructure information within a parallel task is not taken into consideration. This article investigates the design of locking protocols for parallel tasks using a novel mechanism-longest normal Section first (LNSF)-to consider the impact of normal sections on blocking behavior in parallel tasks and further improve real-time performance. LNSF is then implemented in a locking protocol for parallel tasks named POMIP, and associated blocking analysis techniques are presented. Empirical evaluations show that our proposed analysis dominated other state-of-the-art analysis-in best cases, the acceptance ratio of the task set can be improved by around 17%.

In this article, the effect a moving target has on the signal-to-interference-plus-noise-ratio (SINR) for high time-bandwidth noise radars is investigated. To compensate for cell migration we apply a computationally efficient stretch processing algorithm that is tailored for batched processing and suitable for implementation onto a real-time radar processor. The performance of the algorithm is studied using experimental data. In the experiment, pseudorandom noise, with a bandwidth of 100 MHz, is generated and transmitted in real-time. An unmanned aerial vehicle (UAV), flown at a speed of 11.5 m/s, is acting as a target. For an integration time of 1 s, the algorithm is shown to yield an increase in SINR of roughly 13 dB, compared to no compensation. It is also shown that coherent integration times of 2.5 s can be achieved., Part of ISBN 978-2-87487-074-3QC 20231218

Electronic waste (e-waste) contains toxic elements causing an important impact on environmental and human health. However, the presence of valuable metals, such as copper or gold, among others, make recycling a necessity for obtaining an alternative source of raw materials. Conventional metal recovery methods are environmentally unsound, prompting the exploration of greener alternatives like bioleaching, which utilizes the activity of microorganisms for a more sustainable recovery. However, the mechanisms involved in the process and the conditions to optimize the metabolic paths are still not completely known. Monitorization and automatization of the different stages composing the global process are crucial for advancing in the implementation of this novel technology at an industrial scale. For the first time, an open-source industrial IoT system is designed to enhance and regulate bioleaching by implementing real-time monitoring and control within the plant’s infrastructure. This system includes an Android app that displays real-time plant data from sensors and a robust server featuring a flexible application programming interface (API) for future applications. The app caters to specific needs such as remote sensor reading, actuator control, and real-time bioleaching alerts, ensuring secure access and proactive event management. By utilizing collected data, it minimizes downtime, equipment failures, and supply chain disruptions. The server maintains seamless communication with the plant controller, enabling efficient pump activation and sensor data transmission. A telegram bot demonstrates the API’s flexibility by forwarding plant alerts to users. During validation with concurrent remote user access, the application demonstrated its ability to prevent irreversible plant failures through an advanced alarm system. Ultimately, this IIoT system amplifies plant performance, safety, and efficiency by optimizing processes and decision-making capabilities. It emerges as a, Authors acknowledge the Spanish Government, through the project PID2020-117520RA-I00 for the financial support provided to perform this research. Authors also acknowledge Ramon Miquel for his contribution during the participation in the project., Peer Reviewed, Objectius de Desenvolupament Sostenible::13 - Acció per al Clima, Objectius de Desenvolupament Sostenible::9 - Indústria, Innovació i Infraestructura, Postprint (published version)

Main memory is one of the most complex resources to analyze in multicore-based embedded real-time systems, with contention in the memory controller and the timing constraints of the main memory device as the main contributors to that complexity. One of the main challenges in multicore real-time systems is producing the required evidence on the management of contention delay for the certification. This stems from the fact that current MPSoCs barely provide any event monitors on how tasks interact and delay each other in memory. Besides, even if hardware and software mechanisms are in place to mitigate contention in the memory system, it is hard - if at all possible - to provide evidence about their correctness. In this work, we cover this gap by proposing a lightweight hardware mechanism that tightly tracks inter-core contention in memory. The proposed hardware mechanism, which we evaluate in detail, improves the quality of timing-related evidence that must be provided on how contention in main memory of multicore real-time systems is handled in adherence to applicable safety standards., This work has been supported by Grant PID2019-107255GB-C21 funded by MCIN/AEI/ 10.13039/501100011033, the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No.772773), and the Natural Sciences and Engineering Research Council of Canada (NSERC). The authors also thank the support of the Department of Research and Universities of the Government of Catalonia to the Smart and Safe Autonomous Systems (SSAS) Research Group, (Code:2021 SGR 00637)., Peer Reviewed, Postprint (author's final draft)

Rapidly detecting and identifying chemical agents after a chemical release is crucial for a reliable assessment of imminent risks and provides the initial basis for defining an adequate level of protection for first responders. An unmanned / uncrewed aerial vehicle (UAV) equipped with gas detectors may quickly explore the contaminated area without exposing any first responders to the yet unknown threat. However, UAVs possess limitations in their capability concerning payload, operational range, and power requirements. Choosing the appropriate gas sensors for this application, thus, imposes significant technical challenges. In this work, we present a mobile ion mobility spectrometer (IMS) designed for operation with UAVs. The IMS is equipped with a dedicated closed gas loop, high-performance driver electronics, and wireless data transmission capabilities. The resulting performance of the UAV-mounted mobile IMS is characterized in the lab with the chemical warfare agents (CWAs) sarin (GB), tabun (GA), soman (GD), cyclosarin (GF), and sulfur mustard (HD), and further evaluated in three different field-testing scenarios using the simulation compound di(propylene glycol) methyl ether (DPM). It is thereby used to monitor the area near (i) a point source continuously emitting small quantities of a gaseous chemical, (ii) a point source suddenly releasing a limited quantity of a chemical as an aerosol, and (iii) a minor contamination of a liquid chemical on the ground. The results obtained in this work enable a well-founded estimation of the capabilities and limitations of the UAV-mounted mobile IMS concerning the real-time remote detection of chemical hazards such as CWAs.

We present MobileSky, the first automatic method for real-time high-quality sky replacement for mobile AR applications. The primary challenge of this task is how to extract sky regions in camera feed both quickly and accurately. While the problem of sky replacement is not new, previous methods mainly concern extraction quality rather than efficiency, limiting their application to our task. We aim to provide higher quality, both spatially and temporally consistent sky mask maps for all camera frames in real time. To this end, we develop a novel framework that combines a new deep semantic network called FSNet with novel post-processing refinement steps. By leveraging IMU data, we also propose new sky-aware constraints such as temporal consistency, position consistency, and color consistency to help refine the weakly classified part of the segmentation output. Experiments show that our method achieves an average of around 30 FPS on off-the-shelf smartphones and outperforms the state-of-the-art sky replacement methods in terms of execution speed and quality. In the meantime, our mask maps appear to be visually more stable across frames. Our fast sky replacement method enables several applications, such as AR advertising, art making, generating fantasy celestial objects, visually learning about weather phenomena, and advanced video-based visual effects. To facilitate future research, we also create a new video dataset containing annotated sky regions with IMU data. IEEE

Intelligent tools for creating synthetic scenes have been developed significantly in recent years. Existing techniques on interactive scene synthesis only incorporate a single object at every interaction, i.e., crafting a scene through a sequence of single-object insertions with user preferences. These techniques suggest objects by considering existent objects in the scene instead of fully picturing the eventual result, which is inherently problematic since the sets of objects to be inserted are seldom fixed during interactive processes. In this article, we introduce SceneDirector, a novel interactive scene synthesis tool to help users quickly picture various potential synthesis results by simultaneously editing groups of objects. Specifically, groups of objects are rearranged in real-time with respect to a position of an object specified by a mouse cursor or gesture, i.e., a movement of a single object would trigger the rearrangement of the existing object group, the insertions of potentially appropriate objects, and the removal of redundant objects. To achieve this, we first propose an idea of coherent group set which expresses various concepts of layout strategies. Subsequently, we present layout attributes, where users can adjust how objects are arranged by tuning the weights of the attributes. Thus, our method gives users intuitive control of both how to arrange groups of objects and where to place them. Through extensive experiments and two applications, we demonstrate the potentiality of our framework and how it enables concurrently effective and efficient interactions of editing groups of objects.

Elastic scheduling is a framework to reduce task utilizations (often by increasing periods) in response to system overload. This paper extends elastic scheduling to uniprocessor scheduling of implicit-deadline task sets for which periods must remain harmonic. We argue that for tasks with periods constrained to continuous intervals, the problem of selecting harmonic periods from those intervals is unlikely to have a polynomial time solution. However, we outline an approach that is pseudo-polynomial in the range of acceptable periods. We then show that the problem of elastic scheduling is NP-hard with harmonic constraints. Nonetheless, if a total order is imposed on task periods (a natural restriction in many applications with execution pipelines that synchronize input data sources), the problem can be reduced offline to a lookup table, enabling polynomial-time online adaptation if available CPU bandwidth changes. We implement the proposed algorithm in two real-world applications: the Fast Integrated Mobility Spectrometer (FIMS) and ORB-SLAM3. We demonstrate that elastic scheduling allows FIMS to adjust its execution to avoid missing deadlines on a SWaP-constrained computational platform, and that it improves ORB-SLAM3's localization results by as much as 10.4x when available CPU bandwidth changes dynamically during runtime.

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in realtime. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

In this paper, we propose a method for animating behavior of communication protocols/systems based on event-driven visualization scenarios (VSs) described in a subclass of E-LOTOS called real-time LOTOS. In the proposed method, first we pick up some interesting events in the original specification of the protocol, and then describe a VS for those events so that the corresponding animations are executed just after their execution. We execute the original specification and its VS in parallel under a multi-way synchronization mechanism so that the corresponding animation is activated when each event in the original specification is executed. To describe animations in real-time LOTOS, we have defined some primitive animation operations as ADT functions. The pair of the original specification and its VS is converted into a multi-threaded Java program using our real-time LOTOS compiler where animation primitives described as ADT functions are replaced in the corresponding methods, implemented as Java class libraries. In our visualization method, we can specify the VS without modifying the original specification, and we can derive a Java program which animates the original specification in real-time. We have carried out an experiment to visualize a real-time LOTOS specification of a prioritized queuing mechanism of the differentiated service where two types of VSs for algorithm animation and performance evaluation are used for visualization.

This paper proposes an augmented reality system with correct representation of shading and shadow. To realize a seamless augmented reality system, we need to resolve certain problems. The geometric and photometric registration problems are particularly important. These problems require the position of light sources and user's viewpoint. The proposed system resolves these problems using a 3D marker which combines a 2D square marker and a mirror ball. The 2D marker and the ball are used to estimate the relationship between the real and virtual worlds and the positions of light sources in the real world, respectively., ISMAR 2002 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Sep 30-Oct 1, 2002 , Darmstadt, Germany

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM., ISMAR 2013 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Oct 1-4, 2013 , Adelaide, Australia

This paper proposes a new method of diminished reality which removes AR markers from a user's view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency., ISMAR 2012: IEEE International Symposium on Mixed and Augmented Reality , Nov 5-8, 2012 , Atlanta, GA, USA

For developers concerned with a performance drop or improvement in their software, a profiler allows a developer to quickly search and identify bottlenecks and leaks that consume much execution time. Non real-time profilers analyze the history of already executed stack traces, while a real-time profiler outputs the results concurrently with the execution of software, so users can know the results instantaneously. However, a real-time profiler risks providing overly large and complex outputs, which is difficult for developers to quickly analyze. In this paper, we visualize the performance data from a real-time profiler. We visualize program execution as a three-dimensional (3D) city, representing the structure of the program as artifacts in a city (i.e., classes and packages expressed as buildings and districts) and their program executions expressed as the fluctuating height of artifacts. Through two case studies and using a prototype of our proposed visualization, we demonstrate how our visualization can easily identify performance issues such as a memory leak and compare performance changes between versions of a program. A demonstration of the interactive features of our prototype is available at https://youtu.be/eleVo19Hp4k., VISSOFT 2017 : 2017 IEEE Working Conference on Software Visualization, 18-19 Sept. 2017, Shanghai, China

This paper proposes a vision-based augmented reality system with correct representation of attached and cast shadows. To realize a seamless augmented reality system, we need to resolve a number of problems. Especially, the geometric and photometric registration problems are important. These problems require to estimate the positions of light sources and user's viewpoint. The proposed system resolves the problems using a 3D marker which combines a 2D square marker and a mirror ball. The 2D marker and the mirror ball are used to estimate the relationship between the real and virtual worlds and the positions of light sources in the real world, respectively., ICPR 2004 : 17th International Conference on Pattern Recognition , Aug 23-26, 2004 , Cambridge, UK