Search

Performance measures are crucial in selecting the best machine learning model for a given problem. Estimating classical model performance measures by subsampling methods like bagging or cross-validation has several weaknesses. The most important ones are the inability to test the significance of the difference, and the lack of interpretability. Recently proposed Elo-based Predictive Power (EPP)—a meta-measure of machine learning model performance, is an attempt to address these weaknesses. However, the EPP is based on wrong assumptions, so its estimates may not be correct. This paper introduces the Probability-based Ranking Model Approach (PMRA), which is a modified EPP approach with a correction that makes its estimates more reliable. PMRA is based on the calculation of the probability that one model achieves a better result than another one, using the Mixed Effects Logistic Regression model. The empirical analysis was carried out on a real mortgage credits dataset. The analysis included a comparison of how the PMRA and state-of-the-art k-fold cross-validation ranked the 49 machine learning models, an example application of a novel method in hyperparameters tuning problem, and a comparison of PMRA and EPP indications. PMRA gives the opportunity to compare a newly developed algorithm to state-of-the-art algorithms based on statistical criteria. It is the solution to select the best hyperparameters configuration and to formulate criteria for the continuation of the hyperparameters space search. [ABSTRACT FROM AUTHOR]

A fuzzy set extension known as the hesitant fuzzy set (HFS) has increased in popularity for decision making in recent years, especially when experts have had trouble evaluating several alternatives by employing a single value for assessment when working in a fuzzy environment. However, it has a significant problem in its uses, i.e., considerable data loss. The probabilistic hesitant fuzzy set (PHFS) has been proposed to improve the HFS. It provides probability values to the HFS and has the ability to retain more information than the HFS. Previously, fuzzy regression models such as the fuzzy linear regression model (FLRM) and hesitant fuzzy linear regression model were used for decision making; however, these models do not provide information about the distribution. To address this issue, we proposed a probabilistic hesitant fuzzy linear regression model (PHFLRM) that incorporates distribution information to account for multi-criteria decision-making (MCDM) problems. The PHFLRM observes the input–output (IPOP) variables as probabilistic hesitant fuzzy elements (PHFEs) and uses a linear programming model (LPM) to estimate the parameters. A case study is used to illustrate the proposed methodology. Additionally, an MCDM technique called the technique for order preference by similarity to ideal solution (TOPSIS) is employed to compare the PHFLRM findings with those obtained using TOPSIS. Lastly, Spearman's rank correlation test assesses the statistical significance of two rankings sets. [ABSTRACT FROM AUTHOR]