Search

Geospatial computation, data transformation to a relevant statistical software, and step-wise quantitative performance assessment can be cumbersome, especially when considering that the entire modelling procedure is repeatedly interrupted by several input/output steps, and the self-consistency and self-adaptive response to the modelled data and the features therein are lost while handling the data from different kinds of working environments. To date, an automated and a comprehensive validation system, which includes both the cutoff-dependent and –independent evaluation criteria for spatial modelling approaches, has not yet been developed for GIS based methodologies. This study, for the first time, aims to fill this gap by designing and evaluating a user-friendly model validation approach, denoted as Performance Measure Tool (PMT), and developed using freely available Python programming platform. The considered cutoff-dependent criteria include receiver operating characteristic (ROC) curve, success-rate curve (SRC) and prediction-rate curve (PRC), whereas cutoff-independent consist of twenty-one performance metrics such as efficiency, misclassification rate, false omission rate, F-score, threat score, odds ratio, etc. To test the robustness of the developed tool, we applied it to a wide variety of geo-environmental modelling approaches, especially in different countries, data, and spatial contexts around the world including, the USA (soil digital modelling), Australia (drought risk evaluation), Vietnam (landslide studies), Iran (flood studies), and Italy (gully erosion studies). The newly proposed PMT is demonstrated to be capable of analyzing a wide range of environmental modelling results, and provides inclusive performance evaluation metrics in a relatively short time and user-convenient framework whilst each of the metrics is used to address a particular aspect of the predictive model. Drawing on the inferences, a scenario-based protocol for model performance evaluation is suggested.

Gully erosion has been identified as an important soil degradation process and sediment source, especially in arid and semiarid areas. Thus, it is useful to identify the spatial occurrence of this form of water erosion in the landscape and the most vulnerable areas. In this study, we explored the effects of different pixel sizes on some controlling factors extracted from a digital elevation model and remote sensing data when producing a gully erosion susceptibility map (GESM) of Ekbatan Dam Basin, Hamadan, Iran. An inventory map of the gully landforms was prepared based on global positioning system routes of the gullies, extensive field surveys, and visual interpretations of satellite images obtained from Google Earth. Five data sets with pixel sizes ranging from 2 to 30 m were obtained using topographic attributes and remote sensing data comprising the elevation, slope degree, slope aspect, catchment area, plan curvature, profile curvature, stream power index, topographic position index, topographic wetness index, land use, and normalized difference vegetation index, which can affect the distribution of gully erosion. For each data set, 70% and 30% of the data were selected randomly for calibrating and validating the models, respectively. The statistical relationships between the occurrence of gully erosion and controlling factors were calculated using four machine-learning models, i.e., generalized linear model, boosted regression tree (BRT), multivariate adaptive regression spline, and artificial neural network (ANN). Statistical tests comprising the kappa coefficient and the area under the receiver operating characteristic curve (AUC) were calculated for both the calibration and validation data sets to estimate the optimal pixel size. The results showed that among the data sets with different pixel sizes, the optimal pixel size was 10 m for each model. In addition, the capacity of the four techniques for modeling gully erosion occurrence was quite stable when the calibration and validation samples were changed in the data set. Finally, based on three changes of the calibration and validation data sets with a pixel size of 10 m, the BRT and ANN models obtained outstanding performance (AUC > 0.9), where they had the highest goodness-of-fit and predictive power, and thus the greatest robustness to changes in the calibration/validation data (i.e., lowest sensitivity to altering calibration/validation data). Our results demonstrate the importance of selecting a suitable pixel size when producing a GESM for soil and water management practices.