Search

Shortcomings of the correlation coefficient (Pearson's) as a measure for estimating and calculating the accuracy of predictive model properties are analysed. Here we discuss two such cases that can often occur in the application of the model in predicting properties of a new external set of compounds. The first problem in using the correlation coefficient is its insensitivity to the systemic error that must be expected in predicting properties of a novel external set of compounds, which is not a random sample selected from the training set. The second problem is that an external set can be arbitrarily large or small and have an arbitrary and uneven distribution of the measured value of the target variable, whose values are not known in advance. In these conditions, the correlation coefficient can be an overoptimistic measure of agreement of predicted values with the corresponding experimental values and can lead to a highly optimistic conclusion about the predictive ability of the model. Due to these shortcomings of the correlation coefficient, the use of standard error (root-mean-square-error) of prediction is suggested as a better quality measure of predictive capabilities of a model. In the case of classification models, the use of the difference between the real accuracy and the most probable random accuracy of the model shows very good characteristics in ranking different models according to predictive quality, having at the same time an obvious interpretation . This work is licensed under a Creative Commons Attribution 4.0 International License.

Several quantitative structure‐activity studies for this data set containing 107 HEPT derivatives have been performed since 1997, using the same set of molecules by (more or less) different classes of molecular descriptors. Multivariate Regression (MR) and Artificial Neural Network (ANN) models were developed and in each study the authors concluded that ANN models are superior to MR ones. We re‐calculated multivariate regression models for this set of molecules using the same set of descriptors, and compared our results with the previous ones. Two main reasons for overestimation of the quality of the ANN models in previous studies comparing with MR models are: (1) wrong calculation of leave‐one‐out (LOO) cross‐validated (CV) correlation coefficient for MR models in Luco et al., J. Chem. Inf. Comput. Sci. 37 392–401 (1997), and (2) incorrect estimation/interpretation of leave‐one‐out (LOO) cross‐validated and predictive performance and power of ANN models. More precise and fairer comparison of fit and LOO ...

In this study we demonstrate use of ensembles of linear and nonlinear multivariate regression models, based on multivariate polynomials of initial descriptors, in QSAR/QSPR modeling. Data sets, which varied significantly in size regarding number of variables and number of points, were all previously referenced in literature and molecular structures were either obtained from authors of these publications or generated in our laboratories. All data sets were encoded as SMILES and converted to 3D structures (SD files) by the CORINA program (www2.chemie.uni-erlangen.de/software/corina/). All descriptors were computed by the program DRAGON 2.1 (http://www.disat.unimib.it/chm/). Linear ensembles were built with multiple linear regression models (MLR) and nonlinear ensembles consisted of multivariate polynomials, which were constructed as controlled subsets selected among linear descriptors, their two-fold cross-products and squares, as well as cubic potencies of (only) single descriptors. Ensemble responses were computed as mean or median or weighted values of all intrinsic models. Models and ensembles discussed in this paper were constructed with the application NQSAR, a Windows console application, which is available upon request. Results obtained show clear advantage of nonlinear ensembles over linear counterparts when data sets contain 4 to 5 times more points than model coefficients. On the other side linear ensembles, which in general exhibit higher robustness and stability, are better suited for small data sets with many variables outperforming nonlinear ensembles in predicting values of data points from external data set. This can be explained by the fact that the linear models are less affected by small variations than nonlinear models while they equally benefit from the key ensemble features. Primarily, we note the impact of the inclusion of more variables spread across optimized variable subsets, which are used in ensembles’ intrinsic models that individually satisfy before mentioned rule on over-fitting. The overall ensemble responses are more stable and robust with higher predictive powers than single models.

In this study we want to test whether a simple modeling procedure used in the field of QSAR/QSPR can produce simple models that will be, at the same time, as accurate as robust Neural Network Ensemble (NNE) ones. We present results of application of two procedures for generating/selecting simple linear and nonlinear multiregression (MR) models: (1) method for selecting the best possible MR models (named as CROMRsel) and (2) Genetic Function Approximation (GFA) method from the Cerius2 program package. The obtained MR models are strictly compared with several NNE models. For the comparison we selected four QSAR data sets previously studied by NNE (Tetko et al. J. Chem. Inf. Comput. Sci. 1996, 36, 794-803. Kovalishyn et al. J. Chem. Inf. Comput. Sci. 1998, 38, 651-659.): (1) 51 benzodiazepine derivatives, (2) 37 carboquinone derivatives, (3) 74 pyrimidines, and (4) 31 antimycin analogues. These data sets were parameterized with 7, 6, 27, and 53 descriptors, respectively. Modeled properties were anti-pentylenetetrazole activity, antileukemic activity, inhibition constants to dihydrofolate reductase from MB1428 E. coli, and antifilarial activity, respectively. Nonlinearities were introduced into the MR models through 2-fold and/or 3-fold cross-products of initial (linear) descriptors. Then, using the CROMRsel and GFA programs (J. Chem. Inf. Comput. Sci. 1999, 39, 121-132) the sets of I (Ior = 8, in this paper) the best descriptors (according to the fit and leave-one-out correlation coefficients) were selected for multiregression models. Two classes of models were obtained: (1) linear or nonlinear MR models which were generated starting from the complete set of descriptors, and (2) nonlinear MR models which were generated starting from the same set of descriptors that was used in the NNE modeling. In addition, the descriptor selection method from CROMRsel was compared with the GFA method included in the QSAR module of the Cerius2 program. For each data set it has been found that the MR models have better cross-validated statistical parameters than the corresponding NNE models and that CROMRsel selects somewhat better MR models than the GFA method. MR models are also much simpler than NNEs, which is the important surprising fact, and, additionally, express calculated dependencies in a functional form. Moreover, MR models were shown to be better than all other models obtained by different methods on the same data sets ("old" multivariate regressions, functional-link-net models, back-propagation neural networks, genetic algorithm, and partial least squares models). This study also indicated that the robust NNE models cannot generate good models when applied on small data sets, suggesting that it is perhaps better to apply robust methods (like NNE ones) on larger data sets.