Search

Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools insufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems: (1) traditional data processing techniques are inadequate to handle big data; (2) data preprocessing needs substantial manual effort; (3) domain knowledge is required before the classification; (4) semantic explanation is ignored. In this work, these challenges are overcome by using different algorithms combined with a Naïve Bayes classifier to ensure reliability and highly precise recommendations in virtualization and cloud environments. These features make HTwitt different from others in terms of having an effective and practical design for text classification in big data analytics. The main contribution of the paper is to propose a framework for building landslide early warning systems by pinpointing useful tweets and visualizing them along with the processed information. We demonstrate the results of the experiments which quantify the levels of overfitting in the training stage of the model using different sizes of real-world datasets in machine learning phases. Our results demonstrate that the proposed system provides high-quality results with a score of nearly 95% and meets the requirement of a Hadoop-based classification system. [ABSTRACT FROM AUTHOR]

Unsupervised machine learning is one of the accepted platforms for applying a broad data analytics challenge that involves the way to identify secret trends, unexplained associations, and other significant data from a wide dispersed dataset. The precise yield estimate for the various crops involved in the planning is a critical problem for agricultural planning. To achieve realistic and effective solutions to this problem, data mining techniques are an essential approach. Applying distplot combined with kernel density estimate (KDE) in this paper to visualize the probability density of disseminated datasets of vast crop deals for crop planning. This paper focuses on analyzing and segmenting agricultural data and determining optimal parameters to maximize crop yield using data mining techniques such as K-means clustering and principal component analysis (PCA). [ABSTRACT FROM AUTHOR]

Abstract Severe weather events pose a significant threat to transportation networks. This research analyzes and discusses the impact of precipitation, temperature, visibility and wind speed on hourly weekday traffic flow volume in Atlanta, Georgia. The study involves the following: determine weather variables that affect traffic volume, develop a machine learning technique to derive decision rules based on weather and traffic volume, and create a web-based decision support visualization tool using the analyzed results. The relationship between extreme weather events and traffic volume was investigated by comparing traffic volume between a base case scenario and an extreme weather scenario. Data from 48 Automatic Traffic Recorder (ATR) sites around Atlanta, GA, USA and hourly precipitation data from 4 climate measurement stations were used to conduct this study. The spatiotemporal relationships between traffic volume and weather variables were analyzed individually and evaluated using a non-parametric statistical test. A machine learning technique is applied to derive decision rules that result in reduction in traffic volume. Results show significant impacts on traffic volume from visibility, precipitation and temperature and helps in isolating hours in a typical weekday when such impacts are felt. A decision support tool was also developed to visualize traffic volume and weather interactions. The data-driven insights from this analysis is applicable to transportation planners, centralized traffic control rooms and urban infrastructure decision makers. Highlights • Paper analyzes correlations between 4 weather hazards and traffic volume by combining Big Data repositories from climate science and transportation. • The paper using extensive statistical analysis to derive hours in a typical day where traffic at a particular location is impacted due to a weather hazard. • Machine learning models were also applied to derive decision rules for traffic planners and transportation personnel to optimize and plan for weather hazards. • A decision support tool was also derived to help display the data analysis, derive insights and optimize transportation decision making. [ABSTRACT FROM AUTHOR]

As more and more data become available for machine prognostic analysis in the big data environment, effective data suitability assessment methods become highly desired to help locate data with sufficient quality for analysis. Driven by this purpose, this paper proposes a novel and systematic methodology for data suitability assessment based on the needs of prognostics and health management (PHM). In this study, the data suitability for PHM is assessed from the aspects of detectability, diagnosability, and trendability, which correspond to the three major tasks of PHM: fault detection, fault diagnosis, and degradation assessment. The proposed methodology is mainly built upon the recent research studies on maximum mean discrepancy in the field of machine learning, which include a family of test statistics that are used to test the difference between two data distributions. The effectiveness of the proposed methodology is demonstrated in diverse industrial applications, which include semiconductors, boring tool degradation, and sensorless drive diagnosis. The results in the case studies indicate that the proposed methodology can be a promising tool to evaluate whether the data under study or the extracted feature set is suitable for PHM tasks. [ABSTRACT FROM PUBLISHER]

In the scenario of big data, incorporating multiple attribute descriptions of the same subject (i.e., so-called multi-view data analysis) and the developing effective data visualization techniques have been one of the most important topics in the computer vision and machine learning community. In this paper, a multiple view source attributes embedding framework is proposed that can assemble various object attributes together and embed them into a low dimensional space to realize visual data structure demonstration. Classical sole attribute data embedding approaches such as MDS and SHE can be utilized under this framework. The core idea of this framework is to achieve a uniform contribution merging, which means every source attribute will contribute equally to the final embedded output. The underlying reason for this idea is that now that an attribute is input as a source and no prior weight is given beforehand, it should be treated equally. The most exciting advantage of this strategy is that it can avoid prejudice caused by major attributes (exerting due to their evident quantity advantage), which may lead the minor attributes be ignored. In addition, we propose a simple iterating algorithm to implement this framework, and plenty of experiments are conducted under diverse source attribute configurations to validate its effectiveness. [ABSTRACT FROM AUTHOR]

Excessive information significantly increases the mental burden on operators of critical monitoring services such as maritime and air traffic control. In these fields, vessels and aircraft have sensors that transmit data to a control center. Because of the large volume of collected data, it is infeasible for monitoring stations to display all of the information on monitoring screens that have limited sizes. This paper proposes a method for automatically selecting maritime traffic stream data for display from a large number of candidates in a context-aware manner. Safety is the most important concern in maritime traffic control, and special care must be taken to avoid collisions between vessels at sea. It presents an architecture for an adaptive information visualization system for a maritime traffic control service. The proposed system adaptively determines the information to be displayed based on the safety evaluation scores and expertise of vessel traffic service operators. It also introduces a method for safety context acquisition to assess the risk of collisions between vessels, using parallel and distributed processing of maritime stream data transmitted by sensors on the vessels at sea. It provides an information-filtering, knowledge extraction method based on the work logs of traffic service operators, using a machine learning technique to generate a decision tree. We applied the proposed system architecture to a large dataset collected at a port. Our results indicate that the proposed system can adaptively select traffic information according to port conditions and to ensure safety and efficiency. [ABSTRACT FROM AUTHOR]