Search

This chapter discusses the principles of calibration and validation and links these phases to the previous chapters on data collection and data enhancement. First, a generic outline of a typical simulation study is given in Section 4.1, then a generic procedure for calibration and validation is covered in Section 4.2, underlying three main choices: measures of performance, measures for goodness of fit, and optimization procedure. Section 4.3 presents examples of calibration for car following. Section 4.4 explains which role can be played by synthetic data while scrutinizing if the calibration and validation chain procedure is or not appropriate. Section 4.5 discusses conclusions.

A single source of information for researchers and professionals, Traffic Simulation and Data: Validation Methods and Applications offers a complete overview of traffic data collection, state estimation, calibration and validation for traffic modelling and simulation. It derives from the Multitude Project—a European Cost Action project that incorporates work packages defining traffic simulation practice and research; highway and network modeling; and synthesis, dissemination, and training. This book addresses the calibration and validation of traffic models, and introduces necessary frameworks and techniques. It also includes viable methods for sensitivity analyses, and incorporates relevant tools for application. The book begins with a brief summary of various data collection techniques that can be applied to collect different data types. It then showcases various data processing and enhancement techniques for improving the quality of collected data. It also introduces the techniques according to the type of estimation, for example microscopic data enhancement, traffic state estimation, feature extraction and parameter identification techniques, and origin–destination matrix estimation. The material discusses the measures of performance, data error and goodness of fit, and optimization algorithms. It also contains the sensitivity analyses of parameters in traffic models. Describes the various tasks of calibration and validation Considers the best use of available data Presents the sensitivity analysis method Discusses typical issues of data error in transportation system data and how these errors can impact simulation results Details various methodologies for data collection, sensitivity analysis, calibration, and validation Examines benefits that result from the application of these methods Traffic Simulation and Data: Validation Methods and Applications serves as a key resource for transport engineers and planners, researchers, and graduate students in transport engineering and planning.

convenient way to assess the traffic fuel consumption is to couple a microscopic traffic flow model with an instantaneous emission model. Traffic models are usually calibrated using goodness of fit indicators that are related to the traffic behaviour. Thus, this paper aims to investigate how the traffic model calibration influences the accuracy of fuel consumption estimation. Two traffic models are investigated: the Newell’s and the Gipps’s model. It appears that the Gipps’s model globally provides more accurate fuel consumption estimation than the Newell’s model. For both models, the fuel consumption estimation of a single vehicle is very sensitive to the calibration. This is confirmed by a global sensitivity analysis of the Gipp’s model that shows that non optimal parameters significantly increase the variance of the outputs. Fortunately, this is no longer the case when the fuel consumption of a whole platoon is calculated. In fact, using global mean value for the traffic model parameters lead to errors lower than 3% when estimation the consumption of a platoon of ten vehicles.

In recent time, in the field of traffic simulation, sensitivity analysis (SA) is starting to attract attention as an indispensible tool for simplifying the calibration of microscopic traffic flow models (1,2,3). These models, in facts, involve many sub-models and dozens of parameters (4) that need to be calibrated to make the model suitable to correctly reproduce local traffic conditions. Unfortunately, such models are quite computationally expensive (the typical duration of a simulation run being in the order of minutes) meanwhile the calibration in the high-dimensional space of model parameters usually requires several thousands of model evaluations. For this reason, it is common practice to carry out calibration only for a limited number of parameters. However, there is no established procedure for their selection, other than the personal experience. Therefore it is easy to imagine as the selection of an incomplete set of parameters for the calibration might lead to several issues, including but not limited to the inaccuracy and unreliability of model results as well as unrealistic values for the calibrated parameters. Therefore, a proper SA, including the initial screening of the parameters, can be very valuable for the subsequent calibration process (5), as it can provide both quantitative and qualitative information regarding the effects of the different model parameters (and their variations) on the simulation results. The sensitivity analysis of traffic simulation models is therefore carried out on its parameters, whereas the other inputs like the transportation network or the travel demand are kept constant. Since many sensitivity analysis techniques require a considerable number of model evaluations (5) and a SA needs to be repeated for each specific case study, the computational complexity still remains a problem. For this reason, the possibility of generalizing results of a SA of a specific traffic simulation model in a specific case study, is worth to be investigated. In the present work we present the preliminary results of an exploratory research in which the robustness of the results of a sensitivity analysis, carried out on the parameters of a car-following model, is assessed against the variation of the non-parametric inputs. Car-following models are the key components of all microscopic traffic simulation models. They describe the longitudinal motion of a vehicle by mimicking the reaction of its driver (the “follower”) to the stimuli perceived while interacting with the front vehicle (the “leader”). They are in the form of differential equations (sometimes delayed) whose basic inputs are, generally, the follower’s speed, the distance between the follower and the leader and their speed difference. Outputs of such models are usually the follower’s speed or acceleration. Traffic is therefore simulated through a system of chained coupled equations. Apart from the mentioned inputs, simulation outputs are strongly dependent on the values of the model parameters which vary among the population of drivers (that is along the chain of coupled equations) as deemed to capture the individual psycho-physical characteristics of each driver. In simulation practice such parameters are considered uncertain in order to cover all the uncertainty in the simulation process. They are usually calibrated through an inverse analysis, that is by looking for the value of parameters that allow the simulated time-space trajectory of a vehicle to be as near as possible to the measured one (6). Although these models have usually a quite simple formulation, their behavior, especially as the result of the parameter values, is not yet clear. Their SA is therefore an interesting and timely issue. As already mentioned, however, this is not the only objective of this work. Here, we do want also to ascertain how robust are the results of the analysis against the variation of the non-parametric inputs, that is, by varying the leader’s trajectory. To this aim, the sensitivity analysis included the leader’s trajectory as an additional factor which, in the Monte Carlo framework adopted, was sampled from a predefined dataset of trajectories. Such dataset, in particular, was built by picking trajectories measured in different roads (freeway and arterial) and in different traffic conditions. In this way it was possible to assess the relative effect on model outputs of the parameters and the input trajectories that was essential to understand whether results of a SA can be generalized (independently of the trajectory). In the experiment, we chose the sensitivity analysis technique based on the computation of the Sobol first order and total order sensitivity indices (5,7). Confidence intervals around the indices were also calculated in order to check for their stability. The car-following model used was the IDM model (8), while the trajectories considered were 101 trajectories selected from 10 databases available thanks to the NGSIM project which allowed us to capture a wide spectrum of driving behaviors. Some results are summarized in Figure 1. Figure 1. First and total order sensitivity indices for the parameters of the IDM model. They show that the input trajectory (identified by the PairID variable) has a prominent effect on the model outputs. At the same time, however, its effect is mainly played in combination with other model parameters. Overall, it can be said that there are few parameters exerting a certain effect on the output of the model no matter the input used. This is an important result as it opens the path for defining classes of problems, in traffic simulation, for which the parameters to calibrate can be defined a priori and not individuated case by case.

In the field of traffic simulation, the calibration of uncertain inputs against real data is usually taken to cover both the epistemic uncertainty regarding the un-modeled details of the phenomena and the aleatory not predicted by the models. For this reason, model parameters are usually indirectly estimated within an optimization framework which tries to maximize the fit between real and simulated measures of the traffic system. This is the case, for example, of the calibration of car-following models’ parameters against vehicle trajectory data. Only recently, it has been proven that the capability of the optimization framework to provide the parameters’ values that allow the car-following model reproducing real trajectories at its best is strictly connected to the setting of the optimization framework itself. This, in particular, entails the necessity to carefully choose an appropriate combination of optimization algorithm and measure of goodness of fit (GOF). In this study, the authors focus attention on this latter issue. Specifically, it is claimed here that the commonly used GOFs are not able to capture the dynamics of the time-series which calibration is performed against. Therefore, a spectral analysis based approach to evaluate the overall performance of the simulation model in the objective function is proposed. The new measure of goodness of fit is tested in the calibration of the Intelligent Driver Model against synthetic and real trajectory data. Results with synthetic data, in particular, confirm that such a new optimization setting is always able to find the global optimum of the problem., JRC.F.8-Sustainable Transport

This study evaluated the effectiveness of speed limits enforced with an automated section speed enforcement system (ASSES) through the analysis of compliance after the system implementation in the Italian Motorway A3 Naples-Salerno. The speed limits were selected taking into account the average of the following values: (1) minimum design speed; (2) minimum speed consistent with the available stopping sight distance; and (3) operating speeds before the ASSES implementation. The first two criteria take into account safety whereas the third criterion takes into account the mobility needs. The speed limits selected with these criteria were then refined basing on: maximum difference between the section speed limit and the minimum speed evaluated with criteria 1 and 2, evaluation of critical conditions, and consistency between the speed limits and the highway function. The system was activated on April 21, 2010 and the driving speeds were monitored in the two periods April 21, 2010 - May 11, 2010 and March 30, 2011 - April 20, 2011. Overall, non compliance to the speed limits was equal to 50%. Non compliance was higher for heavy vehicles (60-70%) than for light vehicles (32-65%). Non compliance of light vehicles was much higher in the sections with the most constrained alignments (64% in sections with speed limit equal to 80 kph vs. 34% in sections with speed limit equal to 100 kph). Higher compliance to the speed limits, according to the authors’ belief, might be achieved by a better strategy of communication and information to the road users and a speed limit management strategy synergic between the highway agency and the Police who actually manages the commitments of fines., JRC.F.8-Sustainable Transport

This paper reports on the results of a web based survey conducted as part of the MULTITUDE project (Methods and tools for supporting the Use caLibration and validaTIon of Traffic simUlation moDEls). The project, being performed in Europe, is attempting to focus and drive forward the issue of 'model validity' and thereby calibration and validation methods and their usage in simulation. As part of this, an assessment of the state-of-the-practice has been performed, examining among other things how models are applied, what procedures are used for calibration and what guidelines are followed. The basis for this state-of-the-practice, is a web based survey which ran in the latter quarter of 2010 and obtained 215 responses. The paper details the findings of this survey which, while lengthy, substantiates two clear messages. Firstly, that Calibration, Validation and Sensitivity analysis although now more widespread than was previously thought, are still far from being required procedure, and secondly that while guidelines are increasingly in use, governing for example number of simulation runs, there are still many regions where this does not occur and indeed there is still much reliance on personal experience and habit, especially in situations where network characteristics become large. The results of the survey underpin the need for further focus on the use of models and procedures for calibration and validation, and indicate, more specifically, the areas that MULTITUDE will focus on in the near future.

In this study, a methodology for the selection of speed limits in sections enforced with automated section speed enforcement systems (ASSESs) has been proposed. Results of the speed control system have been assessed by speed monitoring after the system implementation in the Italian Motorway A3 Naples-Salerno. The speed limits were selected taking into account the average of the following values: (1) minimum design speed; (2) minimum speed consistent with the available stopping sight distance; and (3) operating speeds before the ASSES implementation. The first two criteria take into account safety whereas the third criterion takes into account the mobility needs. The speed limits selected with these criteria were then refined basing on: maximum difference between the section speed limit and the minimum speed evaluated with criteria 1 and 2, evaluation of critical conditions, and consistency between the speed limits and the highway function. Overall, non compliance to the speed limits was equal to 50%. Non compliance was higher for heavy vehicles (60-70%) than for light vehicles (32-65%). Non compliance of light vehicles was much higher in the sections with the most constrained alignments (64% in sections with speed limit equal to 80 kph vs. 34% in sections with speed limit equal to 100 kph). Higher compliance to the speed limits, according to the authors’ belief, might be achieved by a better strategy of communication and information to the road users and a speed limit management strategy synergic between the highway agency and the Police.

In the last years, the expanding range of new technologies in the field of traffic management and control called for accurate modeling of traffic flows in order to evaluate their potential impact on society and environmental decision-making. The inner complexity of these applications sought for detailed stochastic traffic simulation tools which could enable their analysis, design and evaluation. In this view, microscopic traffic flow simulation models are increasingly used as cost-effective tools to support these tasks. However, despite their importance, the use of these tools is far from being trivial. Indeed, the “goodness” of a simulation study does not depend only on the expertise of the analyst/modeler but (mostly) on the “correct” use of such models which, conversely, can be challenging even for specialists. This could be due to a number of reasons, including model indeterminacy, over-parameterization, asymmetry in the importance of parametric inputs, and so on. In other words, different sources of parametric and non-parametric uncertainty may affect the performances of simulation models, and thus if not properly assessed (and possibly reduced), would inevitably undermine the reliability of results. Despite the importance of uncertainty management in scientific modeling, it is a very under investigated issue in the field of traffic flow simulation modeling. Common symptoms of neglecting the management of uncertainty in traffic flow simulation modeling may be the (un)repeatability of experiments, the (un)reliability of predictions, and the vulnerability to instrumental or otherwise unethical use of models. Above all, this turns out in the lack of effectiveness, credibility, and transparency of simulation results. Therefore, the objective of this dissertation thesis is to propose and apply a common methodological framework for the quantitative management of uncertainty in microscopic traffic flow simulation modeling. In particular, the research focused on driver behavioral models only, although generalization to other traffic flow models, or even to more general transportation systems models (e.g. public transportation models, pedestrian simulation models), might be possible with reasonable easiness. Several contributions were put forward with regards to the critical phases of i) vehicle trajectory data analysis, ii) driver behavioral model calibration, iii) model sensitivity analysis, and iv) aggregate traffic micro-simulation. Achieved results have the ambition of enabling the exploitation of the full potential of microscopic traffic flow simulation models in traffic forecasting.

This paper discusses a metamodel-based technique for model sensitivity analysis and applies it to the Aimsun mesoscopic model. Throughout the paper it is argued that the application of sensitivity analysis is crucial for the true comprehension and correct use of the traffic simulation model while also acknowledging that the main obstacle to an extensive use of the most sophisticated techniques is the high number of model runs they usually require. For this reason we have tested the possibility of performing sensitivity analysis not on a model but on its metamodel approximation. Important issues arising when estimating a metamodel have been investigated and commented on in the specific application to the Aimsun model. Among these issues is the importance of selecting a proper sampling strategy based on low discrepancy random number sequences and the importance of selecting a class of metamodels able to reproduce the inputs-ouputs relationship in a robust and reliable way. Sobol sequences and Gaussian process metamodels have been recognized as the appropriate choices. The proposed methodology has been assessed by comparing the results of the application of variance-based sensitivity analysis techniques to the simulation model and to a metamodel estimated with 512 model runs, on a variety of traffic scenarios and model outputs. Results confirm the powerfulness of the proposed methodology and also open up to a more extensive application of sensitivity analysis techniques to complex traffic simulation models., JRC.F.8-Sustainable Transport

Despite the undoubted importance of NGSIM data for the research on traffic flow theory, they were proved to be massively affected by measurement errors in the spatial coordinates of the vehicle, further amplified in the differentiation process when calculating speeds and accelerations. If not properly accounted for, these errors would make NGSIM data not usable for any study on traffic flow theory. However, we argued herein that the techniques applied so far in the literature to correct vehicle trajectory data are not suitable for the scope, as they did not treat appropriately the “cause” of the bias, while limited to smooth out the “effects”, that are the high and medium frequency disturbances in the data. Therefore, in this paper we first illustrated the mechanism at the basis of the errors in the NGSIM data, and showed the limits of currently available techniques. Secondly we clarified that extremely high errors, the outliers, need a special treatment to be fixed. In this light, we presented a multi-step filtering procedure aimed at i) eliminating the outliers that give rise to unphysical values for the acceleration by the local reconstruction of the vehicle trajectory, and ii) cut-off the residual random disturbances from the signal, still preserving the driving dynamics (vehicle stoppages, shifting gears, etc.). Both the operations were performed taking directly into account the requirement for internal consistency of the trajectory, that is the consistency among space travelled, speeds and accelerations. Though the procedure is absolute general, and can be applied to whatever vehicle’s trajectory, in this paper we first presented the results related to a single vehicle’s trajectory from NGSIM I80 dataset, and, then, those from the application to the complete set of trajectories from the same dataset. Results here presented motivated the need to correct NGSIM data before any further processing, JRC.F.8-Sustainable Transport

TRB 2012 - Transportation Research Board 91rd Annual Meeting, WASHINGTON D.C, ETATS-UNIS, 22-/01/2012 - 26/01/2012; This paper reports on the results of a web based survey conducted as part of the MULTITUDE project (Methods and tools for supporting the Use caLibration and validaTIon of Traffic simUlation moDEls). The project, being performed in Europe, is attempting to focus and drive forward the issue of 'model validity' and thereby calibration and validation methods and their usage in simulation. As part of this, an assessment of the state-of-the-practice has been performed, examining among other things how models are applied what procedures are used for calibration and what guidelines are followed. The basis for this state-of-the-practice, is a web based survey which ran in the latter quarter of 2010 and obtained 215 responses. The paper details the finding of this survey which, while lengthy, substantiates two clear messages. Firstly, that Calibration, Validation and Sensitivity analysis although now more widespread than was previously thought, are still far from being required procedure, and secondly that while guidelines are increasingly in use, governing for example number of simulation runs, there are still many regions where this does not occur and indeed there is still much reliance on personal experience and habit, especially in situations where network characteristics become large. The results of the survey underpin the need for further focus on the use of models and procedures for calibration and validation, and indicate, more specifically, the areas that MULTITUDE will focus on in the near future.

Calibration of car-following models against trajectory data has been widely applied as the basis for several type of studies ranging from the investigation and benchmarking of models, to the study of parameters correlation, or other theoretical issues like the inter/intra driver heterogeneity or the multi-anticipative driving behaviour. However, very few of these studies attempted also to analyze and quantify the uncertainty entailed in the calibration process and its impacts on the accuracy and reliability of results. A thorough understanding of the whole calibration problem (against trajectory data), as well as of the mutual effect of the specific problems raised in the field literature, indeed, does not yet exist. In this view, a general methodology to assess a calibration procedure was proposed and applied to the calibration of the Gipps’ car-following model. Compact indicators were proposed to evaluate the capability of a calibration setting to find the “known” global solution, in terms of both the accuracy and the robustness as to the variation of the starting conditions of the optimisation algorithm. Then, a graphical inspection method, based on the so called cobweb plots, was proposed to explore the existence and the nature of the local minima found by the algorithms, as well as to give insights into the measure of performances and the goodness of fit functions used in the calibration experiments. The methodology has been then applied to all the calibration settings (i.e. combinations of algorithms, measure of performances and goodness of fit functions) applied in the field literature within the past ten years. The study allowed us to highlight and motivate the limits (sometime even the unsuitability) of some calibration settings so far applied in the scientific literature. Research lines towards the definition of robust settings for the problem of car-following models calibration based on real trajectory data, are finally outlined., JRC.F.8-Sustainable Transport

This paper reports on the results of a web based survey conducted as part of the MULTITUDE project (Methods and tools for supporting the Use caLibration and validaTIon of Traffic simUlation moDEls). The project, being performed in Europe, is attempting to focus and drive forward the issue of 'model validity' and thereby calibration and validation methods and their usage in simulation. As part of this, an assessment of the state-of-the-practice has been performed, examining among other things how models are applied what procedures are used for calibration and what guidelines are followed. The basis for this state-of-the-practice, is a web based survey which ran in the latter quarter of 2010 and obtained 215 responses. The paper details the finding of this survey which, while lengthy, substantiates two clear messages. Firstly, that Calibration, Validation and Sensitivity analysis although now more widespread than was previously thought, are still far from being required procedure, and secondly that while guidelines are increasingly in use, governing for example number of simulation runs, there are still many regions where this does not occur and indeed there is still much reliance on personal experience and habit, especially in situations where network characteristics become large. The results of the survey underpin the need for further focus on the use of models and procedures for calibration and validation, and indicate, more specifically, the areas that MULTITUDE will focus on in the near future., JRC.F.8-Sustainable Transport

In the last decades, simulation optimization has received considerable attention from both researchers and practitioners. Simulation optimization is the process of finding the best values of some decision variables for a system whose performance is evaluated using the output of a simulation model. A possible example of simulation optimization is the model calibration. In traffic modelling this topic is particularly relevant since the solutions to the methodological issues arising when setting up a calibration study cannot be posed independently. This calls for methodologies able to check the robustness of a calibration framework as well as further investigations of the issue, in order to identify possible “classes” of problems to be treated in a similar way. Therefore in the present work, first a general method for verifying a traffic micro-simulation calibration procedure (suitable in general for simulation optimization) is described, based on a test with synthetic data. Then it is applied, my means of two different case studies, to draw inferences on the effect that different combinations of parameters to calibrate, optimization algorithm, measures of Goodness of Fit and noise in the data may have on the optimization problem. Results showed the importance of verifying the calibration procedure with synthetic data. In addition they ascertained the need for global optimization solutions, giving new insights into the topic. Research contained within this paper benefited from the participation in EU COST Action TU0903 MULTITUDE, JRC.H.8-Sustainability Assessment

This paper proposes a numerical assessment of the performances of the CoNL route choice model [1] on a big-size network (Regione Campania). The CoNL is a particular specification of the CoRUM model [2]. Currently, its performances in terms of choice probabilities have been investigated only on some toy networks, by comparing route choice probabilities with reference to target Multinomial Probit choice probabilities. The paper provides also a discussion about different aspects of the CoNL for route choice: the possibility to take into account also non-efficient path, the computation time of the two versions of the algorithm for building the CoNL structure, and the possibility to adopt some different specifications for computing the structural parameters of the model. The comparison is based on a dataset of 195 trajectories on 145 different o-d pairs, tracked with the aid of an Android application. The trajectories have been collected through the smartphones of travellers moving within the network of Regione Campania (Italy). The results show the superiority of the CoNL route choice model in reproducing observed route choices when compared with other commonly used route choice formulations.

Transportation systems are complex sociotechnical systems and this dual nature is reflected in the literature dealing with their planning, i.e. defining courses of action from both public and private points of view. On the one hand, the social sciences literature makes it clear that most decisions related to transportation are "wicked", i.e. they cannot be tackled with traditional engineering approaches since they are poorly defined. On the other, transportation systems have a strong technical component affecting most of such decisions, as they have to (or should) comply with compelling technical and economic requirements. The literature on transport engineering and economics deals with transportation planning mostly as a rational process based on the formulation and comparison of alternative options. In this paper, we propose an approach to planning and designing transportation systems, at least in the more complex cases, meant to bridge the gap between social and technical points of view. The proposed approach is cross-disciplinary, as it integrates notions from cognitive sciences, organization and management sciences with transportation systems analysis. Transportation planning, both under public and private market-oriented viewpoints, is seen as a complex decision-making process where different actors (decision-makers, stakeholders, professionals) interact in different contexts and according to different "models" or approaches. After a brief discussion of the main elements of most planning processes and of the different decision making models, the paper proposes a decision-making model based on three parallel and intertwined processes: a cognitive rational approach to organizing the decision-making process, a five-level stakeholder engagement process, and a revised role of quantitative analyses and methods drawing on tools from engineering and economics, amongst other disciplines. We also present a real application of the proposed decision-making model to the case study of the Regional Metro System (RMS) project in Campania (southern Italy), where a complex decisional context with different stakeholders and multiple (often contrasting) interests was managed in the integrated framework for over a decade, resulting in the largest and most effective of such projects in Italy. Finally, the paper proposes new roles and new challenges for quantitative analyses and mathematical tools to support participated decision-making processes, extending their well-established functions for designing and assessing transportation solutions. Language: en

This paper proves that in traffic flow model calibration and validation the cumulative sum of a variable has to be preferred to the variable itself as a measure of performance. As shown through analytical relationships, model residuals dynamics are preserved if discrepancy measures of a model against reality are calculated on a cumulative variable, rather than on the variable itself. Keeping memory of model residuals occurrence times is essential in traffic flow modelling where the ability of reproducing the dynamics of a phenomenon – as a bottleneck evolution or a vehicle deceleration profile – may count as much as the ability of reproducing its order of magnitude. According to the aforesaid finding, in a car-following models context, calibration on travelled space is more robust than calibration on speed or acceleration. Similarly in case of macroscopic traffic flow models validation and calibration, cumulative flows are to be preferred to flows. Actually, the findings above hold for any dynamic model.

Car-following models are widely used by researchers and practitioners for road traffic studies. Although dozens of them have been presented so far, the model proposed by P.G. Gipps in 1981 is still one of the most extensively used. However, many features of the model are still rather unknown or neglected in common applications. In this context, the present paper wishes i) to summarize and analyze the main findings available in the scientific literature for the Gipps??? car-following model and ii) to introduce some novel features of the same model that may improve its capability to reproduce real trajectory data. In particular, the structure of the acceleration component of the model is analytically investigated, in particular for what concerns the meaning of some parameters that in the common practice are usually kept to some empirically derived fixed values. Different possible versions of the Gipps??? model are presented and their performances to reproduce real vehicle trajectories are evaluated and compared. The results achieved show the necessity for these parameters to be calibrated in order to improve the predictive capabilities of the model.