Search

While computer networks and the massive amount of communication taking place on these networks grow, the amount of damage that can be done by network intrusions grows in tandem. The need is for an effective and scalable intrusion detection system (IDS) to address these potential damages that come with the growth of these networks. A great deal of contemporary research on near real-time IDS focuses on applying machine learning classifiers to labeled network intrusion datasets, but these datasets need be relevant pertaining to the currency of the network intrusions. This paper focuses on a newly created dataset, UWF-ZeekData22, that analyzes data from Zeek's Connection Logs collected using Security Onion 2 network security monitor and labelled using the MITRE ATT&CK framework TTPs. Due to the volume of data, Spark, in the big data framework, was used to run many of the well-known classifiers (naïve Bayes, random forest, decision tree, support vector classifier, gradient boosted trees, and logistic regression) to classify the reconnaissance and discovery tactics from this dataset. In addition to looking at the performance of these classifiers using Spark, scalability and response time were also analyzed. [ABSTRACT FROM AUTHOR]

In predicting energy consumption, classic econometric and statistical models are used to forecast energy consumption. These models may have limitations in an increasingly fast-changing energy market, which requires big data analysis of energy consumption patterns and relevant variables using complex mathematical tools. In current literature, there are minimal comparison studies reviewing machine learning algorithms to predict energy consumption in Korea. To bridge this gap, this paper compared three different machine learning algorithms, namely the Random Forest (RF) model, XGBoost (XGB) model, and Long Short-Term Memory (LSTM) model. These algorithms were applied in Period 1 (prior to the onset of the COVID-19 pandemic) and Period 2 (after the onset of the COVID-19 pandemic). Period 1 was characterized by an upward trend in energy consumption, while Period 2 showed a reduction in energy consumption. LSTM performed best in its prediction power specifically in Period 1, and RF outperformed the other models in Period 2. Findings, therefore, suggested the applicability of machine learning to forecast energy consumption and also demonstrated that traditional econometric approaches may outperform machine learning when there is less unknown irregularity in the time series, but machine learning can work better with unexpected irregular time series data. [ABSTRACT FROM AUTHOR]

This study introduces UWF-ZeekDataFall22, a newly created dataset labeled using the MITRE ATT&CK framework. Although the focus of this research is on classifying the never-before classified resource development tactic, the reconnaissance and discovery tactics were also classified. The results were also compared to a similarly created dataset, UWF-ZeekData22, created in 2022. Both of these datasets, UWF-ZeekDataFall22 and UWF-ZeekData22, created using Zeek Conn logs, were stored in a Big Data Framework, Hadoop. For machine learning classification, Apache Spark was used in the Big Data Framework. To summarize, the uniqueness of this work is its focus on classifying attack tactics. For UWF-ZeekdataFall22, the binary as well as the multinomial classifier results were compared, and overall, the results of the binary classifier were better than the multinomial classifier. In the binary classification, the tree-based classifiers performed better than the other classifiers, although the decision tree and random forest algorithms performed almost equally well in the multinomial classification too. Taking training time into consideration, decision trees can be considered the most efficient classifier. [ABSTRACT FROM AUTHOR]

The use of fuzzy logic to assess students' knowledge is not a completely new concept. However, despite dealing with a large quantity of data, traditional statistical methods have typically been the preferred approach. Many studies have argued that machine learning methods could offer a viable alternative for analyzing big data. Therefore, this study presents findings from a Random Forest (RF) regression analysis to understand the influence of demographic factors on students' achievements, i.e., teacher-given grades, students' outcomes on the national assessment, and fuzzy grades, which were obtained as a combination of the two. RF analysis showed that demographic factors have limited predictive power for teacher-assigned grades, unlike INVALSI scores and fuzzy grades. School type, macroregion, and ESCS are influential predictors, whereas gender and origin have a lesser impact. The study highlights regional and socio-economic disparities, influencing both student outcomes and fuzzy grades, underscoring the need for equitable education. Unexpectedly, gender's impact on achievements is minor, possibly due to gender-focused policies. Although the study acknowledges limitations, its integration of fuzzy logic and machine learning sets the foundation for future research and policy recommendations, advocating for diversified assessment approaches and data-driven policymaking. [ABSTRACT FROM AUTHOR]

Decision making under uncertainty is pivotal in real-world scenarios, such as selecting the shortest transportation route amidst variable traffic conditions or choosing the best investment portfolio during market fluctuations. In today's big data age, while the predict-then-optimize framework has become a standard method for tackling uncertain optimization challenges using machine learning tools, many prediction models overlook data intricacies such as outliers and heteroskedasticity. These oversights can degrade decision-making quality. To enhance predictive accuracy and consequent decision-making quality, we introduce a data transformation technique into the predict-then-optimize framework. Our approach transforms target values in linear regression, decision tree, and random forest models using a power function, aiming to boost their predictive prowess and, in turn, drive better decisions. Empirical validation on several datasets reveals marked improvements in decision tree and random forest models. In contrast, the benefits of linear regression are nuanced. Thus, while data transformation can bolster the predict-then-optimize framework, its efficacy is model-dependent. This research underscores the potential of tailoring transformation techniques for specific models to foster reliable and robust decision-making under uncertainty. [ABSTRACT FROM AUTHOR]

Before the internet, people acquired their news from the radio, television, and newspapers. With the internet, the news moved online, and suddenly, anyone could post information on websites such as Facebook and Twitter. The spread of fake news has also increased with social media. It has become one of the most significant issues of this century. People use the method of fake news to pollute the reputation of a well-reputed organization for their benefit. The most important reason for such a project is to frame a device to examine the language designs that describe fake and right news through machine learning. This paper proposes models of machine learning that can successfully detect fake news. These models identify which news is real or fake and specify the accuracy of said news, even in a complex environment. After data-preprocessing and exploration, we applied three machine learning models; random forest classifier, logistic regression, and term frequency-inverse document frequency (TF-IDF) vectorizer. The accuracy of the TFIDF vectorizer, logistic regression, random forest classifier, and decision tree classifier models was approximately 99.52%, 98.63%, 99.63%, and 99.68%, respectively. Machine learning models can be considered a great choice to find reality-based results and applied to other unstructured data for various sentiment analysis applications. [ABSTRACT FROM AUTHOR]

Gridded satellite precipitation datasets are useful in hydrological applications as they cover large regions with high density. However, they are not accurate in the sense that they do not agree with ground-based measurements. An established means for improving their accuracy is to correct them by adopting machine learning algorithms. This correction takes the form of a regression problem, in which the ground-based measurements have the role of the dependent variable and the satellite data are the predictor variables, together with topography factors (e.g., elevation). Most studies of this kind involve a limited number of machine learning algorithms and are conducted for a small region and for a limited time period. Thus, the results obtained through them are of local importance and do not provide more general guidance and best practices. To provide results that are generalizable and to contribute to the delivery of best practices, we here compare eight state-of-the-art machine learning algorithms in correcting satellite precipitation data for the entire contiguous United States and for a 15-year period. We use monthly data from the PERSIANN (Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks) gridded dataset, together with monthly earth-observed precipitation data from the Global Historical Climatology Network monthly database, version 2 (GHCNm). The results suggest that extreme gradient boosting (XGBoost) and random forests are the most accurate in terms of the squared error scoring function. The remaining algorithms can be ordered as follows, from the best to the worst: Bayesian regularized feed-forward neural networks, multivariate adaptive polynomial splines (poly-MARS), gradient boosting machines (gbm), multivariate adaptive regression splines (MARS), feed-forward neural networks and linear regression. [ABSTRACT FROM AUTHOR]

Low-frequency oscillations (LFO) occur in railway electrification systems due to the incorporation of new trains with switching converters. As a result, the increased harmonic content can cause catenary stability problems under certain conditions. Most of the research published on this topic to date is focused on modelling the event and analysing it using frequency spectrums. However, in recent years, due to the new technologies linked to Big Data (BD) and data mining (DM), a new opportunity to study and detect LFO events by means of machine-learning (ML) methods has emerged. Trains continuously collect data from the most important catenary variables, which offers new resources for analysing this type of event. Therefore, this article presents the design and implementation of a data-driven LFO event detection strategy for AC railway network scenarios. Compared to previous investigations, a new approach to analyse and detect LFO events, based on field data and ML, is presented. To obtain the most appropriate detection approach for the context of this application, on the one hand, this investigation includes a comparison of machine-learning algorithms (support vector machine, logistic regression, random forest, k-nearest neighbours, naïve Bayes) which have been trained with real field data. On the other hand, an analysis of key parameters and features to optimize event detection is also included. Thus, the most significant result of this work is the high metric values of the solution, reaching values above 97% in accuracy and 93% in F-1 score with the random forest algorithm. In addition, the applicability and training of data-driven methods with real field data are demonstrated. This automatic detection strategy can help with speeding up and improving LFO detection tasks that used to be performed manually. Finally, it is worth mentioning that this research has been structured based on the CRISP-DM methodology, established as the de facto approach for industrial DM projects. [ABSTRACT FROM AUTHOR]

The features of a dataset play an important role in the construction of a machine learning model. Because big datasets often have a large number of features, they may contain features that are less relevant to the machine learning task, which makes the process more time-consuming and complex. In order to facilitate learning, it is always recommended to remove the less significant features. The process of eliminating the irrelevant features and finding an optimal feature set involves comprehensively searching the dataset and considering every subset in the data. In this research, we present a distributed fuzzy cognitive map based learning-based wrapper method for feature selection that is able to extract those features from a dataset that play the most significant role in decision making. Fuzzy cognitive maps (FCMs) represent a hybrid computing technique combining elements of both fuzzy logic and cognitive maps. Using Spark's resilient distributed datasets (RDDs), the proposed model can work effectively in a distributed manner for quick, in-memory processing along with effective iterative computations. According to the experimental results, when the proposed model is applied to a classification task, the features selected by the model help to expedite the classification process. The selection of relevant features using the proposed algorithm is on par with existing feature selection algorithms. In conjunction with a random forest classifier, the proposed model produced an average accuracy above 90%, as opposed to 85.6% accuracy when no feature selection strategy was adopted. [ABSTRACT FROM AUTHOR]