Search

With the advancement of autonomous driving technology, scenario-based testing has become the mainstream testing method for intelligent vehicles. However, traditional risk indicators often fail in roundabout scenarios and cannot accurately define dangerous situations. To accurately quantify driving risks in roundabout scenarios, an improved driving safety field model is proposed in this paper. First, considering the unique traffic flow characteristics of roundabouts, the dynamic characteristics of vehicles during diverging or merging were taken into account, and the driving safety field model was improved to accurately quantify the driving risks in roundabout scenarios. Second, based on data from the rounD dataset, the model parameters were calibrated using the social force model. Finally, a DENCLUE-like method was used to extract collision systems, calculate vehicle risk degree, and analyze these risks for both the temporal and the spatial dimensions, providing guidance for virtual testing. The proposed method significantly improves detection efficiency, increasing the number of identified dangerous scenarios by 175% compared to the Time to Collision (TTC) method. Moreover, this method can more accurately quantify driving risks in roundabout scenarios and enhance the efficiency of generating dangerous scenarios, contributing to promoting the safety of autonomous vehicles. [ABSTRACT FROM AUTHOR]

Few mobile robot developers already test their software on simulated robots in virtual environments or sceneries. However, the majority still shy away from simulation-based test campaigns because it remains challenging to specify and execute suitable testing scenarios, that is, models of the environment and the robots' tasks. Through developer interviews, we identified that managing the enormous variability of testing scenarios is a major barrier to the application of simulation-based testing in robotics. Furthermore, traditional CAD or 3Dmodelling tools such as SolidWorks, 3ds Max, or Blender are not suitable for specifying sceneries that vary significantly and serve different testing objectives. For some testing campaigns, it is required that the scenery replicates the dynamic (e.g., opening doors) and static features of real-world environments, whereas for others, simplified scenery is sufficient. Similarly, the task and mission specifications used for simulation-based testing range from simple point-to-point navigation tasks to more elaborate tasks that require advanced deliberation and decision-making. We propose the concept of composable and executable scenarios and associated tooling to support developers in specifying, reusing, and executing scenarios for the simulation-based testing of robotic systems. Our approach differs from traditional approaches in that it offers a means of creating scenarios that allow the addition of new semantics (e.g., dynamic elements such as doors or varying task specifications) to existing models without altering them. Thus, we can systematically construct richer scenarios that remain manageable. We evaluated our approach in a small simulation-based testing campaign, with scenarios defined around the navigation stack of a mobile robot. The scenarios gradually increased in complexity, composing new features into the scenery of previous scenarios. Our evaluation demonstrated how our approach can facilitate the reuse of models and revealed the presence of errors in the configuration of the publicly available navigation stack of our SUT, which had gone unnoticed despite its frequent use. [ABSTRACT FROM AUTHOR]

Few mobile robot developers already test their software on simulated robots in virtual environments or sceneries. However, the majority still shy away from simulation-based test campaigns because it remains challenging to specify and execute suitable testing scenarios, that is, models of the environment and the robots’ tasks. Through developer interviews, we identified that managing the enormous variability of testing scenarios is a major barrier to the application of simulation-based testing in robotics. Furthermore, traditional CAD or 3D-modelling tools such as SolidWorks, 3ds Max, or Blender are not suitable for specifying sceneries that vary significantly and serve different testing objectives. For some testing campaigns, it is required that the scenery replicates the dynamic (e.g., opening doors) and static features of real-world environments, whereas for others, simplified scenery is sufficient. Similarly, the task and mission specifications used for simulation-based testing range from simple point-to-point navigation tasks to more elaborate tasks that require advanced deliberation and decision-making. We propose the concept of composable and executable scenarios and associated tooling to support developers in specifying, reusing, and executing scenarios for the simulation-based testing of robotic systems. Our approach differs from traditional approaches in that it offers a means of creating scenarios that allow the addition of new semantics (e.g., dynamic elements such as doors or varying task specifications) to existing models without altering them. Thus, we can systematically construct richer scenarios that remain manageable. We evaluated our approach in a small simulation-based testing campaign, with scenarios defined around the navigation stack of a mobile robot. The scenarios gradually increased in complexity, composing new features into the scenery of previous scenarios. Our evaluation demonstrated how our approach can facilitate the reuse of models and revealed the presence of errors in the configuration of the publicly available navigation stack of our SUT, which had gone unnoticed despite its frequent use.

Automated Driving Systems (ADSs) bring many benefits, however their safety assurance poses challenges. Scenario-based testing has been proposed. To provide strong safety evidence for their safety assurance, the scenario-based testing process needs to consider the Operational Design Domain (ODD) of the systems, ODD combined with the behavioural elements can provide a foundation for the scenario generation workflow due to common domain elements between ODDs and scenarios. Based on such background, this paper introduces a novel framework to generate scenarios specifically target on testing the system’s claimed ODD. It includes the process going from the system’s ODD and behaviour competency, to logical scenarios generation utilising scenario construct rule sets, and to the concretisation of the logical scenarios into concrete scenarios. This paper also draws link towards the part II of the paper series, which illustrates a novel approach for scenario coverage analysis.

With the advancement of autonomous driving technology, scenario-based testing has become the mainstream testing method for intelligent vehicles. However, traditional risk indicators often fail in roundabout scenarios and cannot accurately define dangerous situations. To accurately quantify driving risks in roundabout scenarios, an improved driving safety field model is proposed in this paper. First, considering the unique traffic flow characteristics of roundabouts, the dynamic characteristics of vehicles during diverging or merging were taken into account, and the driving safety field model was improved to accurately quantify the driving risks in roundabout scenarios. Second, based on data from the rounD dataset, the model parameters were calibrated using the social force model. Finally, a DENCLUE-like method was used to extract collision systems, calculate vehicle risk degree, and analyze these risks for both the temporal and the spatial dimensions, providing guidance for virtual testing. The proposed method significantly improves detection efficiency, increasing the number of identified dangerous scenarios by 175% compared to the Time to Collision (TTC) method. Moreover, this method can more accurately quantify driving risks in roundabout scenarios and enhance the efficiency of generating dangerous scenarios, contributing to promoting the safety of autonomous vehicles.

This paper proposes a decision-making algorithm based on deep reinforcement learning to support fallback techniques in autonomous vehicles. The fallback technique attempts to mitigate or escape risky driving conditions by responding to appropriate avoidance maneuvers essential for achieving a Level 4+ autonomous driving system. However, developing a fallback technique is difficult because of the innumerable fallback situations to address and eligible optimal decision-making among multiple maneuvers. We employed a decision-making algorithm utilizing a scenario-based learning approach to address these issues. First, we crafted a specific fallback scenario encompassing the challenges to be addressed and matched the anticipated optimal maneuvers as determined by heuristic methods. In this scenario, the ego vehicle learns through trial and error to determine the most effective maneuver. We conducted 100 independent training sessions to evaluate the proposed algorithm and compared the results with those of heuristic-derived maneuvers. The results were promising; 38% of the training sessions resulted in the vehicle learning lane-change maneuvers, whereas 9% mastered slow following. Thus, the proposed algorithm successfully learned human-equivalent fallback capabilities from scratch within the provided scenario. [ABSTRACT FROM AUTHOR]

Modern highly automated and autonomous traffic systems and sub-systems require new approaches to test their functional safety in the context of validation and verification. One approach that has taken a leading role in current research is scenario-based testing. For various reasons, simulation is considered to be the most practicable solution for a wide range of test scenarios. However, this is where many existing simulation systems in research reach their limits. In order to be able to integrate the widest possible range of systems to be tested into the simulation, the use of co-simulation has proven to be particularly useful. In this work, the High-Level Architecture defined in the IEEE 1516-2010 standard is specifically addressed, and a concept is developed that establishes the foundation for the feasible use of scenario-based distributed co-simulation on its basis. The main challenge identified and addressed is the resolution of the double-sided dependency between scenario and simulation models. The solution was to fully automate the generation and instantiation of the simulation environment on the basis of a scenario instance. Finally, the developed concept was implemented as a prototype, and the resulting process for its use is presented here using an example scenario. Based on the experience gained during the creation of the concept and the prototype, the next steps for future work are outlined in conclusion. [ABSTRACT FROM AUTHOR]

Innovative testing and validation methods are prerequisites concerning Connected, Cooperative, and Automated Mobility (CCAM), as the high number of cooperating participants and concurrent processes critically increase the probability of adverse safety and security incidents. The proposed new approaches deal with this increasing complexity of not currently having generally accepted validation mechanisms. The paper introduces a novel, mathematical model based, scenario identification methodology, facilitating the selection of critical road vehicle traffic scenarios, taking into account different testing objectives, such as maximizing the safety risk of the analyzed system. The presented results verify that applying specific decision models and quantifiable indicators related to the system elements of highly automated mobility systems can significantly contribute to the systematic identification of unsafe corner cases in connected and cooperative autonomous systems.

High-level Automated Vehicles (HAVs) are expected to improve traffic safety significantly. However, verifying and evaluating HAVs remains an open problem. Scenario-based testing is a promising method for HAV testing. Boundary scenarios exist around the performance boundary between critical and non-critical scenarios. Testing HAVs in these boundary scenarios is crucial to investigate why collisions cannot be avoided due to small changes in scenario parameters. This study proposes a methodology to generate diverse boundary scenarios to test HAVs. First, an approach is proposed to obtain at least one High-Performance Classifier (HPC) based on two classification algorithms that iteratively guide each other to find uncertain scenarios to improve their performance. Then, the HPC is exploited to find candidate scenarios highly likely to be boundary scenarios. To increase the efficiency of candidate scenario generation, a strategy based on local sampling is presented to find more diverse candidate scenarios based on a small number of them. Numerical experiments show that the HPCs acquired by the method proposed in this study can achieve a classification accuracy of 98% and 99% for random car-following and cut-in scenarios, respectively. Moreover, more than 86% of 271,744 candidate cut-in scenarios derived by local sampling are near the performance boundary. [ABSTRACT FROM AUTHOR]

Background: The Operational Design Domain (ODD) of an automated driving function defines on which roads and under which environmental conditions the function is safe to operate. It plays an important role in definition, safety analysis and validation of automated driving. In many cases, users want to determine metrics about ODDs, or about ODDs in combination with other work products, like collections of validation scenarios. Such metrics could answer questions such as what percentage of the road network of a given region is inside the ODD. While language formats to specify ODDs have emerged over the last few years, a solid methodology on how to calculate different sorts of metrics is still on the roadmap for the future. Methods: This contribution suggests metrics for ODDs that are mathematically built upon a notion of ontologies, and ODDs as multi-dimensional cross-products of sets, using standard arithmetic and set operations. To illustrate the idea, a couple of possible metrics for ODDs are derived as examples and discussed in the light of some real-world use cases. Results: To illustrate the application of a ODD metric, we apply an analysis of a sample trip and calculate the theoretical availability of variants of an automated driving system with different ODDs. Conclusions: The metrics presented and the shown sample application present an important next step in discussions around ODDs of Automated Driving Systems. They make it possible to not only consider an ODD specification as a reference for a single system, but allow comparing systems with different ODDs, judging the maturity of a system with a certain ODD, or provide indicators how usable a system is within a real-word application

Given that sensor-based perception systems are utilized in autonomous vehicle applications, it is essential to validate such systems to ensure their robustness before they are deployed. In this study, we propose a comprehensive simulation-based process to verify and enhance the robustness of sensor-based perception systems in relation to corruption. Firstly, we introduce a methodology and scenario-based corruption generation tool for creating a variety of simulated test scenarios. These scenarios can effectively mimic real-world traffic environments, with a focus on corruption types that are related to safety concerns. An effective corruption similarity filtering algorithm is then proposed to eliminate corruption types with high similarity and identify representative corruption types that encompass all considered corruption types. As a result, we can create efficient test scenarios for corruption-related robustness with reduced testing time and comprehensive scenario coverage. Subsequently, we conduct vulnerability analysis on object detection models to identify weaknesses and create an effective training dataset for enhancing model vulnerability. This improves the object detection models’ tolerance to weather and noise-related corruptions, ultimately enhancing the robustness of the perception system. We use case studies to demonstrate the feasibility and effectiveness of the proposed procedures for verifying and enhancing robustness. Furthermore, we investigate the impact of various “similarity overlap threshold” parameter settings on scenario coverage, effectiveness, scenario complexity (size of training and testing datasets), and time costs.

Virtual scenario-based testing has become an acceptable method for evaluating safety effectiveness of advanced driver assistance systems (ADAS). Due to the complexity of the ADAS operating environment, the scenarios that an ADAS could face are almost infinite. Therefore, it is crucial to find critical scenarios to improve the efficiency of testing without compromising credibility. One popular method is to explore the parameterized scenario space using various intelligent search methods. Selecting parameters to parameterize the scenario space is particularly important to achieve good coverage and high efficiency. However, an extensive collection of (relevant) influence parameters is missing, which allows a thorough consideration when selecting parameters regarding specific scenarios. In addition, the general importance definition for individual influence parameters is not provided, regarding the potential influence of their variations on the safety effectiveness of ADAS, which can also be used as a reference while selecting parameters. Combining knowledge from different sources (the published literature, standardized test scenarios, accident analysis, autonomous vehicle disengagement, accident reports, and specific online surveys), this paper has summarized, in total, 94 influence parameters, given the general definitions of importance for 77 influence parameters based on cluster analysis algorithms. The list of influence parameters provides researchers and system developers a comprehensive basis for pre-selecting influence parameters for evaluating the safety effectiveness of ADAS by virtual scenario-based testing and helps check whether certain influence parameters can be a meaningful extension for the evaluation.

Copyright of Automatisierungstechnik is the property of De Gruyter and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In the past decade, automotive companies have invested significantly in autonomous vehicles (AV), but achieving widespread deployment remains a challenge in part due to the complexities of safety evaluation. Traditional distance-based testing has been shown to be expensive and time-consuming. To address this, experts have proposed scenario-based testing (SBT), which simulates detailed real-world driving scenarios to assess vehicle responses efficiently. This paper introduces a method that builds a parametric representation of a driving scenario using collected driving data. By adopting a data-driven approach, we are then able to generate realistic, concrete scenarios that correspond to high-risk situations. A reinforcement learning technique is used to identify the combination of parameter values that result in the failure of a system under test (SUT). The proposed method generates novel, simulated high-risk scenarios, thereby offering a meaningful and focused assessment of AV systems.

This paper proposes a decision-making algorithm based on deep reinforcement learning to support fallback techniques in autonomous vehicles. The fallback technique attempts to mitigate or escape risky driving conditions by responding to appropriate avoidance maneuvers essential for achieving a Level 4+ autonomous driving system. However, developing a fallback technique is difficult because of the innumerable fallback situations to address and eligible optimal decision-making among multiple maneuvers. We employed a decision-making algorithm utilizing a scenario-based learning approach to address these issues. First, we crafted a specific fallback scenario encompassing the challenges to be addressed and matched the anticipated optimal maneuvers as determined by heuristic methods. In this scenario, the ego vehicle learns through trial and error to determine the most effective maneuver. We conducted 100 independent training sessions to evaluate the proposed algorithm and compared the results with those of heuristic-derived maneuvers. The results were promising; 38% of the training sessions resulted in the vehicle learning lane-change maneuvers, whereas 9% mastered slow following. Thus, the proposed algorithm successfully learned human-equivalent fallback capabilities from scratch within the provided scenario.

There is increasing interest in automating train operations of mainline services, e.g., to increase network capacity. Automatic train operation (ATO) is already achieved by several pilot projects, but is still not implemented on a large scale. Functional, interoperability and performance tests are necessary before ATO can be introduced generally. Virtual preliminary analysis will contribute to the validation process to ensure a safe and successful implementation. This paper aims to present an approach that applies to the performance testing of ATO systems. Therefore, methods and test standards for technologies enabling automatic operation in other transport sectors are reviewed. The main findings have been adapted, transformed and combined to be used as a general strategy for virtual performance testing in the railway sector. Specifically, universal performance indicators commonly used in the railway sector, namely punctuality, accuracy, energy consumption, safety and comfort, are presented. They are refined by adding sub-indicators specific to the performance evaluation of ATO algorithms. A layer model for scenario description is adapted from the automotive sector, as well as the definition of different scenario types. Lastly, factors that can influence the performance of an ATO algorithm are identified. For demonstration purposes, a simple case study is conducted. Thereby we exemplarily show-cased the approach for ATO performance testing using a microscopic train simulator in combination with an ATO algorithm. [ABSTRACT FROM AUTHOR]

Context-aware applications are emerging applications in the modern era of computing. These applications can determine and adapt to situational context to provide better user experience. Testing these applications is not straightforward. Constantly changing nature of context makes testing context-aware application is a challenging task. To uncover a defect in context-aware application, a test engineer needs activity (sequence of actions) and context information (context data); this makes test case development a difficult task. Conventional test case development methodologies do not cater for context information. Besides, conventional applications have only one input source, but context-aware application must obtain data from many sources to infer the context. Yet another issue that these applications often face is the noisy data problem as input data collected from physical sensors could be noisy. Test adequacy criteria are used as test stoppage rule and define the quality of testing as well as for generating test suites. Test adequacy criteria is helpful to control the cost of testing as well as determining/establishing confidence in the software product quality. A number of test adequacy criteria exist for testing conventional applications, but the same is not true for context-aware applications. Defining test adequacy criteria and test coverage measures for context-aware applications warrants further research. Several techniques have been developed by researchers to generate and execute test cases for context-aware applications; however, end-to-end testing and result analysis of executed test cases still remains a grueling task for the test engineers. The aim of this study is to automate end-to-end functional testing, analysis of the generated test results as well as functional/requirement coverage assessment. Moreover, we also present a confidence assessment template for result analysis. Test engineers can use our proposed framework to assess the requirement coverage. Our proposed framework will reduce testing time, efforts and cost thus enabling test engineers to execute more testing cycles to attain higher degree of test coverage.

With more goods to be transported oversea, traffic and vessels' dimensions increase while berthing areas merely remain constant and thus challenge ship masters and pilots to maneuver in small basins with dense traffic even in bad weather situations. Too fast approaches or steep angles of attack result in damages to fenders, quay walls, or even impact the hull structure. We propose a shore-based, vessel-independent berthing assistant system to support sailors by Reference Points that are aligned to a quay's meter markings and identify the precise berthing location by measuring distance and approach speed. For this purpose, we define the concept of a Berthing Support Area (BSA), which specifies an area in which, subject to constraints, safe berthing is provided. Within this area there are Reference Points, perpendicular distance measurements at arbitrary positions, which are implemented with a set of LiDAR sensors that have been integrated into the quay wall. In a test campaign with a vessel equipped with DGPS sensors, we sailed seven different maneuvers and evaluated the precision and the accuracy of the Reference Points for speed and distance measurements. [ABSTRACT FROM AUTHOR]

High-level Automated Vehicles (HAVs) are expected to improve traffic safety significantly. However, verifying and evaluating HAVs remains an open problem. Scenario-based testing is a promising method for HAV testing. Boundary scenarios exist around the performance boundary between critical and non-critical scenarios. Testing HAVs in these boundary scenarios is crucial to investigate why collisions cannot be avoided due to small changes in scenario parameters. This study proposes a methodology to generate diverse boundary scenarios to test HAVs. First, an approach is proposed to obtain at least one High-Performance Classifier (HPC) based on two classification algorithms that iteratively guide each other to find uncertain scenarios to improve their performance. Then, the HPC is exploited to find candidate scenarios highly likely to be boundary scenarios. To increase the efficiency of candidate scenario generation, a strategy based on local sampling is presented to find more diverse candidate scenarios based on a small number of them. Numerical experiments show that the HPCs acquired by the method proposed in this study can achieve a classification accuracy of 98% and 99% for random car-following and cut-in scenarios, respectively. Moreover, more than 86% of 271,744 candidate cut-in scenarios derived by local sampling are near the performance boundary.

Automated driving is a promising tool for reducing traffic accidents. While some companies claim that many cutting-edge automated driving functions have been developed, how to evaluate the safety of automated vehicles remains an open question, which has become a crucial bottleneck. Scenario-based testing has been introduced to test automated vehicles, and much progress has been achieved. While data-driven and knowledge-based approaches are hot research topics, this survey is mainly about Data-Driven Scenario Generation (DDSG) for automated vehicle testing. Rather than describe the contributions of every study respectively, in this survey, methodologies from various studies are anatomized as solutions for several significant problems and compared with each other. This way, scholars and engineers can quickly find state-of-the-art approaches to the issues they might encounter. Furthermore, several critical challenges that might hinder DDSG are described, and responding solutions are presented at the end of this survey.

Modern Artificial Intelligence (AI) methods can produce a large quantity of accurate and richly described data, in domains such as surveillance or automation. As a result, the need to organize data at a large scale in a semantic structure has arisen for long-term data maintenance and consumption. Ontologies and graph databases have gained popularity as mechanisms to satisfy this need. Ontologies provide the means to formally structure descriptive and semantic relations of a domain. Graph databases allow efficient and well-adapted storage, manipulation, and consumption of these linked data resources. However, at present, there is no a universally defined strategy for building AI-oriented ontologies for the automotive sector. One of the key challenges is the lack of a global standardized vocabulary. Most private initiatives and large open datasets for Advanced Driver Assistance Systems (ADASs) and Autonomous Driving (AD) development include their own definitions of terms, with incompatible taxonomies and structures, thus resulting in a well-known lack of interoperability. This paper presents the Automotive Global Ontology (AGO) as a Knowledge Organization System (KOS) using a graph database (Neo4j). Two different use cases for the AGO domain ontology are presented to showcase its capabilities in terms of semantic labeling and scenario-based testing. The ontology and related material have been made public for their subsequent use by the industry and academic communities. [ABSTRACT FROM AUTHOR]

Interaction between the background vehicles (BVs) and automated vehicles (AVs) in scenario-based testing plays a critical role in evaluating the intelligence of the AVs. Current testing scenarios typically employ predefined or scripted BVs, which inadequately reflect the complexity of human-like social behaviors in real-world driving scenarios, and also lack a systematic metric for evaluating the comprehensive intelligence of AVs. Therefore, this paper proposes an evolving scenario generation method, employing deep reinforcement learning (DRL) to construct human-like BVs that interact with AVs, and this evolving scenario is designed to test and evaluate the intelligence of AVs. Firstly, a class of BV driver models with human-like competitive, mutual, and cooperative driving motivations is designed. Then, utilizing the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and an improved level-k training procedure, the three distinct driver models acquire game-based interactive driving policies. And these driver models are combined to generate evolving scenarios in which they can interact continuously and evolve diverse contents. Next, a framework including safety, driving efficiency, and interaction utility are presented to evaluate and quantify the intelligence performance of 3 systems under test (SUTs), indicating the effectiveness of the evolving scenario for intelligence testing. Finally, the complexity and fidelity of the proposed evolving testing scenario are validated. The results demonstrate that the proposed evolving scenario exhibits the highest level of complexity compared to other baseline scenarios and has more than 85% similarity to naturalistic driving data. This highlights the potential of the proposed method to facilitate the development and evaluation of high-level AVs in a realistic and challenging environment. • A method for constructing a cluster of BV driver models based on DRL is proposed. These human-like social driver models are endowed with one of the following game-like interaction policies: cooperative, mutual, or competitive. • Utilizing the BV driver models, an evolving scenario of high complexity and fidelity is generated. • A quantitative evaluation framework for the intelligence of automated driving systems is proposed. [ABSTRACT FROM AUTHOR]

There is increasing interest in automating train operations of mainline services, e.g., to increase network capacity. Automatic train operation (ATO) is already achieved by several pilot projects, but is still not implemented on a large scale. Functional, interoperability and performance tests are necessary before ATO can be introduced generally. Virtual preliminary analysis will contribute to the validation process to ensure a safe and successful implementation. This paper aims to present an approach that applies to the performance testing of ATO systems. Therefore, methods and test standards for technologies enabling automatic operation in other transport sectors are reviewed. The main findings have been adapted, transformed and combined to be used as a general strategy for virtual performance testing in the railway sector. Specifically, universal performance indicators commonly used in the railway sector, namely punctuality, accuracy, energy consumption, safety and comfort, are presented. They are refined by adding sub-indicators specific to the performance evaluation of ATO algorithms. A layer model for scenario description is adapted from the automotive sector, as well as the definition of different scenario types. Lastly, factors that can influence the performance of an ATO algorithm are identified. For demonstration purposes, a simple case study is conducted. Thereby we exemplarily show-cased the approach for ATO performance testing using a microscopic train simulator in combination with an ATO algorithm.

Assuring the safety of automated vehicles is essential for their timely introduction and acceptance by policymakers and the public. To assess their safe design and robust decision making in response to all possible scenarios, new methods that use a scenario-based testing approach are needed, as testing on public roads in normal traffic would require driving millions of kilometres. We make use of the scenario-based testing approach and propose a method to model simulated scenarios using Gaussian Process based models to predict untested scenario outcomes. This enables us to efficiently determine the performance boundary, where the safe and unsafe scenarios can be evidently distinguished from each other. We present an iterative method that optimises the parameter space of a logical scenario towards the most critical scenarios on this performance boundary. Additionally, we conduct a novel probabilistic sensitivity analysis by efficiently computing several variance-based sensitivity indices using the Gaussian Process models and evaluate the relative importance of the scenario input parameters on the scenario outcome. We critically evaluate and investigate the usefulness of the proposed Gaussian Process based approach as a very efficient surrogate model, which can model the logical scenarios effectively in the presence of uncertainty. The proposed approach is applied on an exemplary logical scenario and shows viability in finding concrete critical scenarios. The reported results, derived from the proposed approach, could pave the way to more efficient testing of automated vehicles and instruct further physical tests on the determined critical scenarios. [ABSTRACT FROM AUTHOR]

With more goods to be transported oversea, traffic and vessels’ dimensions increase while berthing areas merely remain constant and thus challenge ship masters and pilots to maneuver in small basins with dense traffic even in bad weather situations. Too fast approaches or steep angles of attack result in damages to fenders, quay walls, or even impact the hull structure. We propose a shore-based, vessel-independent berthing assistant system to support sailors by Reference Points that are aligned to a quay’s meter markings and identify the precise berthing location by measuring distance and approach speed. For this purpose, we define the concept of a Berthing Support Area (BSA), which specifies an area in which, subject to constraints, safe berthing is provided. Within this area there are Reference Points, perpendicular distance measurements at arbitrary positions, which are implemented with a set of LiDAR sensors that have been integrated into the quay wall. In a test campaign with a vessel equipped with DGPS sensors, we sailed seven different maneuvers and evaluated the precision and the accuracy of the Reference Points for speed and distance measurements.

Customer satisfaction is a priority of agile teams aiming for successful product delivery. However, many teams continue to rely on functional and regression testing built on traditional, linear test case design. This approach is rarely customer centric, and it neither serves to validate the many different emerging layers of software architecture nor establishes a conceptual understanding of what should be built into automated tests. Consequently, traditional test case design has become an antipattern for successful product delivery. A scenario-based approach, in contrast, examines user behaviors to provide a consistent narrative to accurately evaluate product functionality, and establishes a tight boundary for automated testing of acceptance criteria. Using this approach, user behavior is emphasized, and test cases are short, concise, and driven by the language of the business. This article shows how to use the test scenario design model approach to accelerate testing by designing fewer, higherquality tests, and explains how it can be used to support traceability and other quality outcomes. [ABSTRACT FROM AUTHOR]

Testing and validation of the functionalities and safety of automated vehicles shifted from a distance-based to a scenario-based method in the past decade. A number of domain-specific languages and systems were developed to support scenario-based testing. The aim of this paper is to review and compare the features and characteristics of the major scenario description languages and systems (SDLS). Each of them is designed for different purposes and with different goals; therefore, they have their strengths and weaknesses. Their characteristics are highlighted with an example nontrivial traffic scenario that we designed. We also discuss some directions for further development and research of these SDLS.

Modern Artificial Intelligence (AI) methods can produce a large quantity of accurate and richly described data, in domains such as surveillance or automation. As a result, the need to organize data at a large scale in a semantic structure has arisen for long-term data maintenance and consumption. Ontologies and graph databases have gained popularity as mechanisms to satisfy this need. Ontologies provide the means to formally structure descriptive and semantic relations of a domain. Graph databases allow efficient and well-adapted storage, manipulation, and consumption of these linked data resources. However, at present, there is no a universally defined strategy for building AI-oriented ontologies for the automotive sector. One of the key challenges is the lack of a global standardized vocabulary. Most private initiatives and large open datasets for Advanced Driver Assistance Systems (ADASs) and Autonomous Driving (AD) development include their own definitions of terms, with incompatible taxonomies and structures, thus resulting in a well-known lack of interoperability. This paper presents the Automotive Global Ontology (AGO) as a Knowledge Organization System (KOS) using a graph database (Neo4j). Two different use cases for the AGO domain ontology are presented to showcase its capabilities in terms of semantic labeling and scenario-based testing. The ontology and related material have been made public for their subsequent use by the industry and academic communities.

Given that sensor-based perception systems are utilized in autonomous vehicle applications, it is essential to validate such systems to ensure their robustness before they are deployed. In this study, we propose a comprehensive simulation-based process to verify and enhance the robustness of sensor-based perception systems in relation to corruption. Firstly, we introduce a methodology and scenario-based corruption generation tool for creating a variety of simulated test scenarios. These scenarios can effectively mimic real-world traffic environments, with a focus on corruption types that are related to safety concerns. An effective corruption similarity filtering algorithm is then proposed to eliminate corruption types with high similarity and identify representative corruption types that encompass all considered corruption types. As a result, we can create efficient test scenarios for corruption-related robustness with reduced testing time and comprehensive scenario coverage. Subsequently, we conduct vulnerability analysis on object detection models to identify weaknesses and create an effective training dataset for enhancing model vulnerability. This improves the object detection models' tolerance to weather and noise-related corruptions, ultimately enhancing the robustness of the perception system. We use case studies to demonstrate the feasibility and effectiveness of the proposed procedures for verifying and enhancing robustness. Furthermore, we investigate the impact of various "similarity overlap threshold" parameter settings on scenario coverage, effectiveness, scenario complexity (size of training and testing datasets), and time costs.

Highly automated vehicles are increasingly gaining the public’s attention. To achieve broad acceptance for the deployment of such vehicles, it is necessary to ensure their functionality and safety. One approach that has become popular in research is the scenario-based approach. However, manual testing of such complex systems is impractical and time-consuming. Using simulations to run and evaluate such scenarios appears to be the most viable approach. This, in turn, raises new challenges, especially in modeling the scenarios to be tested simulatively and incorporating the system under test as part of these. Since existing solutions do not solve these challenges satisfactorily—due to the strict separation of scenario and simulation model, among other reasons—this work addresses the need for a standardized, holistic, and extensible approach for modeling traffic scenarios to be executed simulatively. Requirements for such an approach are identified with focus on its application in simulation- and scenario-based verification and validation. Based on these, a model-based multi-layered approach is proposed. The foundations of this are then implemented utilizing a Meta Object Facility based heavyweight extension of the Unified Modeling Language metamodel. The resulting metamodel is used to demonstrate the applicability of the proposed approach by modeling a maritime traffic scenario.

As the complexity of automated driving systemss (ADSs) with automation levels above level 3 is rising, virtual testing for such systems is inevitable and necessary. The complexity of testing these levels lies in the modeling and calculation demands for the virtual environment, which consists of roads, traffic, static and dynamic objects, as well as the modeling of the car itself. An essential part of the safety and performance analysis of ADSs is the modeling and consideration of dynamic road traffic participants. There are multiple forms of traffic flow simulation software (TFSS), which are used to reproduce realistic traffic behavior and are integrated directly or over interfaces with vehicle simulation software environments. In this paper we focus on the TFSS from PTV Vissim in a co-simulation framework which combines Vissim and CarMaker. As it is a commonly used software in industry and research, it also provides complex driver models and interfaces to manipulate and develop customized traffic participants. Using the driver model DLL interface (DMDI) from Vissim it is possible to manipulate traffic participants or adjust driver models in a defined manner. Based on the DMDI, we extended the code and developed a framework for the manipulation and testing of ADSs in the traffic environment of Vissim. The efficiency and performance of the developed software framework are evaluated using the co-simulation framework for the testing of ADSs, which is based on Vissim and CarMaker.

Assuring the safety of automated vehicles is essential for their timely introduction and acceptance by policymakers and the public. To assess their safe design and robust decision making in response to all possible scenarios, new methods that use a scenario-based testing approach are needed, as testing on public roads in normal traffic would require driving millions of kilometres. We make use of the scenario-based testing approach and propose a method to model simulated scenarios using Gaussian Process based models to predict untested scenario outcomes. This enables us to efficiently determine the performance boundary, where the safe and unsafe scenarios can be evidently distinguished from each other. We present an iterative method that optimises the parameter space of a logical scenario towards the most critical scenarios on this performance boundary. Additionally, we conduct a novel probabilistic sensitivity analysis by efficiently computing several variance-based sensitivity indices using the Gaussian Process models and evaluate the relative importance of the scenario input parameters on the scenario outcome. We critically evaluate and investigate the usefulness of the proposed Gaussian Process based approach as a very efficient surrogate model, which can model the logical scenarios effectively in the presence of uncertainty. The proposed approach is applied on an exemplary logical scenario and shows viability in finding concrete critical scenarios. The reported results, derived from the proposed approach, could pave the way to more efficient testing of automated vehicles and instruct further physical tests on the determined critical scenarios.