Search

Field trials are one of the essential stages in agricultural product development, enabling the validation of products in real-world environments rather than controlled laboratory or greenhouse settings. With the advancement in technologies, field trials often collect a large amount of information with diverse data types from various sources. Managing and organizing extensive datasets can impose challenges for small research teams, especially with constantly evolving data collection processes with multiple collaborators and introducing new data types between studies. A practical database needs to be able to incorporate all these changes seamlessly. We present DynamoField, a flexible database framework for collecting and analyzing field trial data. The backend database for DynamoField is powered by Amazon Web Services DynamoDB, a NoSQL database, and DynamoField also provides a front-end interactive web interface. With the flexibility of the NoSQL database, researchers can modify the database schema based on the data provided by various collaborators and contract research organizations. This framework includes functions for non-technical users, including importing and exporting data, data integration and manipulation, and performing statistical analysis. Researchers can utilize cloud computing to establish a secure NoSQL database with minimum maintenance, this also enables collaboration with others worldwide and adapt to different data-collecting strategies as their research progresses. DynamoField is implemented in Python, and it is publicly available at https://github.com/ComputationalAgronomy/DynamoField. [ABSTRACT FROM AUTHOR]

In the domain of efficient management of resources and ensuring nutritional consistency, accuracy in predicting crop yields becomes crucial. The advancement of artificial intelligence techniques, synchronized with satellite imagery, has emerged as a potent approach for forecasting crop yields in modern times. We used two types of data: spatial data and temporal data. Spatial data are gathered from satellite imagery and processed using ArcGIS to extract data about crops based on several indices like NDVI and NWDI. Temporal data are gathered from agricultural sensors such as temperature sensors, rainfall sensor, precipitation sensor and soil moisture sensor. In our case we used Sentinel 2 satellite to extract vegetation indices. We have used IoT systems, especially Raspberry Pi B+ to collect and process data coming from sensors. All data collected are then stored into a NoSQL server to be analysed and processed. Several machine learning and deep learning algorithms have been used for the processing of crop recommendation system, such as logistic regression, KNN, decision tree, support vector machine, LSTM, and Bi-LSTM through the collected dataset. We used GRU deep learning model for the best performance, the RMSE and R2 for this model was 0.00036 and 0.99 respectively.The main contribution of our paper is the development of a new system that can predict several crop yields, such as wheat, maize, etc., using IoT, satellite imagery for spatial data and the use of sensors for temporal data. We are the first paper that has combined spatial data and temporal data to predict crop yield based on deep learning algorithms, unlike other works that uses only remote sensing data or temporal data. We created an E-monitoring crop yield prediction system that helps farmers track all information about crops and show the result of prediction in a mobile application. This system helps farmers with more efficient decision making to enhance crop production. The main production regions of wheat in Morocco are in the rainfed areas of the plains and plateaus of Chaouia, Abda, Haouz, Tadla, Gharb and Saïs. We studied three main regions well known for wheat production which are Rabat-Salé, Fez-Meknes, Casablanca-Settat.

Although there are numerous data modeling tools for relational databases, data modeling for NoSQL databases has seen another perspective. These databases (a) do not define any explicit schema, (b) store data in a denormalized manner, and (c) give many structure alternatives. The decision to structure the data always relies on rules of thumb, which do not guarantee an optimal structural solution. Based on this motivation, this paper offers a workload-driven model for the logical schema design of a NoSQL document database. It consists of Model input, Intermediate transformation, and Final schema generation. The proposed model takes the conceptual schema (EER model) and application workload (queries and anticipated data volume) as input and describes a procedure to convert it into a logical model for NoSQL document stores. The conversion process initially converts the application queries into query graphs. The query graphs, along with the anticipated data volume, are used to generate the query labels. The resulting query labels are assigned on the schema graph designed from the EER model. The schema graph and labels are used to transform the EER model into the appropriate logical schema model based on the actions defined for each label. We evaluate the model using a case study in the eCommerce application domain. The experimental evaluation shows the proposed model outperforms the existing conventional, optimized, and query path graphs models in multiple aspects, including query performance, storage space efficiency, aggregate pipeline efficiency, read–write latency, collection-wise performance, scalability, throughput and latency. By effectively addressing the challenges associated with managing the variety and volume of big data through a well-designed schema, our proposed model significantly reduces the time, cost, and effort required for schema development and repair. [ABSTRACT FROM AUTHOR]

Modern NoSQL database systems use log-structured merge (LSM) storage architectures to support high write throughput. LSM architectures aggregate writes in a mutable MemTable (stored in memory), which is regularly flushed to disk, creating a new immutable file called an SSTable. Some of the SSTables are chosen to be periodically merged—replaced with a single SSTable containing their union. A mergepolicy (a.k.a. compaction policy) specifies when to do merges and which SSTables to combine. A bounded depth merge policy is one that guarantees that the number of SSTables never exceeds a given parameter k, typically in the range 3–10. Bounded depth policies are useful in applications where low read latency is crucial, but they and their underlying combinatorics are not yet well understood. This paper compares several bounded depth policies, including representative policies from industrial NoSQL databases and two new ones based on recent theoretical modeling, as well as the standard Tiered policy and Leveled policy. The results validate the proposed theoretical model and show that, compared to the existing policies, the newly proposed policies can have substantially lower write amplification with comparable read amplification.

With the proliferation of data, relational database management systems face the challenge of managing big volumes of data because of the absence of dynamic data models, high functioning, high scalability, high availability, and low cost. The NoSQL technology has the features which RDBMS lacks. Therefore, many organizations are migrating from RDBMS to NoSQL databases. The migration from RDBMS to NoSQL is not always effective if the data is lost, if the data is replicated while transmission, and if the performance is degraded. For this reason, enhancement of transformation algorithm proposed by Shabana Ramzan et al. has been performed to implement the methodology which is used to successfully migrate data from structured and normalized SQL database MySQL to NoSQL graph database Neo4j. SchemaCrawler engine has been used to metamodeling from source to destination database and execute the transformation rules. This study has also evaluated and validated the performance of the enhanced algorithm after successful migration of data. The NoSQL graph database Neo4j handles the most common form of data that are unstructured, semi-structured, and structured as well as they support the ACID property of relational databases. This feature is not supported by other NoSQL databases such as Oracle NoSQL, MongoDB, and HBase databases.

Energy access is a key need for socio-economic growth. Proven to be a key enabler of development and progress, access to electricity has been prioritized by governments using grid extension actions and off-grid solutions, namely microgrids and home systems technologies, fed by renewable sources. However, achieving universal access to energy remains a huge challenge given the lack of resources and the large population currently unserved. The lack of adequate socio-economic data at granular scale and of a good understanding of demand uptake led by economic growth is a barrier for efficient energy planning. Access to conjoint demand and socio-economic data at local level is crucial, yet hard to obtain: often such data are unavailable or very difficult to collect, and current data platforms often lack the ability to conjointly store variegated socio-economic and time series data. For these reasons, in this paper, we present a comprehensive methodology that, based on an extensive literature review, draws guidelines for developing data-sharing platforms in energy access, develops a proposed architecture to support the data collection of conjoint socio-economic and time-series data, and proposes a prototype of the final application. The methodology leverages on a novel extensive literature review to identify the major determinants of demand uptake and the corresponding consuming entities: villages, households, and appliances. The proposed architecture is able to capture numeric, categorical, and time series information for all consuming entities, based on state-of-the-art NoSQL databases. Finally, a prototype implementation with a web-based interface developed with Angular and Spring is proposed and discussed.

Technological developments in the manufacturing industry have accelerated recently. Although many traditional manufacturing factories had adopted automated production, owing to the inability to control the status of individual tools, it is challenging for manufacturers to accurately track the real time status of production line tools. To bring smart production into existence, a robust database system needs to be ensured. New models of computer numerical control machines are equipped with a built-in tool management system (TMS), but most versions of this system are only able to set a machining time limit or a total number of workpieces limit for each tool to determine tool replacement intervals. Such system could likely cause early tool replacement, resulting in waste and increased tool costs. In order to implement green manufacturing, the prediction of tool wear is essential to reduce wastage of materials. In this study, we developed a TMS that is based on a not only SQL (NoSQL) database, which not only stores several different types of data, but also can store prediction models to make it suitable for the demands of varying enterprise scales. It will be more helpful to ensure processing safety and effectively reduce manufacturing costs.

With the continuous development of science and technology, we have fully entered the information age, people's entertainment life is becoming more and more abundant, and the Internet has also provided people with a lot of convenience. The advent of the Internet age means that more and more various kinds of data are appearing, and the situation is becoming more and more abundant. The use of traditional relational databases can no longer store these data, nor can it be queried. With the continuous development of voice technology, database technology based on NoSQL has become a research hotspot. The availability of NoSQL databases is very high, the scalability is also very high, and the efficiency is very high when processing data. Based on the development of 5G networks and the development of big data technology, this research proposes a brand-new business architecture. This architecture can use the network to store data on the basis of massive data. At the same time, we also described the business scenario, providing a new idea for more intelligent education services. We use this model for teaching in schools, and we can transmit some spoken language resources to the school through the Internet for students to use for learning. Nowadays, the application range of AI technology and intelligent technology has become more and more extensive, and these technologies have also been applied in the education field. We can apply this brand-new technology in teaching to promote the development of teaching and improve students' enthusiasm and learning effect. [ABSTRACT FROM AUTHOR]

Medicine is constantly generating new imaging data, including data from basic research, clinical research, and epidemiology, from health administration and insurance organizations, public health services, and non-conventional data sources such as social media, Internet applications, etc. Healthcare professionals have gained from the integration of big data in many ways, including new tools for decision support, improved clinical research methodologies, treatment efficacy, and personalized care. Finally, there are significant advantages in saving resources and reallocating them to increase productivity and rationalization. In this paper, we will explore how big data can be applied to the field of digital health. We will explain the features of health data, its particularities, and the tools available to use it. In addition, a particular focus is placed on the latest research work that addresses big data analysis in the health domain, as well as the technical and organizational challenges that have been discussed. Finally, we propose a general strategy for medical organizations looking to adopt or leverage big data analytics. Through this study, healthcare organizations and institutions considering the use of big data analytics technology, as well as those already using it, can gain a thorough and comprehensive understanding of the potential use, effective targeting, and expected impact. [ABSTRACT FROM AUTHOR]

In the last decade, the techniques of news aggregation and summarization have been increasingly gaining relevance for providing users on the web with condensed and unbiased information. Indeed, the recent development of successful machine learning algorithms, such as those based on the transformers architecture, have made it possible to create effective tools for capturing and elaborating news from the Internet. In this regard, this work proposes, for the first time in the literature to the best of the authors' knowledge, a methodology for the application of such techniques in news related to cryptocurrencies and the blockchain, whose quick reading can be deemed as extremely useful to operators in the financial sector. Specifically, cutting-edge solutions in the field of natural language processing were employed to cluster news by topic and summarize the corresponding articles published by different newspapers. The results achieved on 22,282 news articles show the effectiveness of the proposed methodology in most of the cases, with 86.8 % of the examined summaries being considered as coherent and 95.7 % of the corresponding articles correctly aggregated. This methodology was implemented in a freely accessible web application. [ABSTRACT FROM AUTHOR]

The rise of big data has led to a greater need for applications to change their schema frequently. NoSQL databases provide flexibility in organizing data and offer multiple choices for structuring and storing similar information. While schema flexibility speeds up initial development, choosing schemas wisely is crucial, as they significantly impact performance, affecting data redundancy, navigation cost, data access cost, and maintainability. This paper emphasizes the importance of schema design in NoSQL document stores. It proposes a model to analyze and evaluate different schema alternatives and suggest the best schema out of various schema alternatives. The model is divided into four phases. The model inputs the Entity-Relationship (ER) model and workload queries. In the Transformation Phase, the schema alternatives are initially developed for each ER model, and subsequently, a schema graph is generated for each alternative. Concurrently, workload queries undergo conversion into query graphs. In the Schema Evaluation phase, the Schema Rank (SRank) is calculated for each schema alternative using query metrics derived from the query graphs and path coverage generated from the schema graphs. Finally, in the Output phase, the schema with the highest SRank is recommended as the most suitable choice for the application. The paper includes a case study of a Hotel Reservation System (HRS) to demonstrate the application of the proposed model. It comprehensively evaluates various schema alternatives based on query response time, storage efficiency, scalability, throughput, and latency. The paper validates the SRank computation for schema selection in NoSQL databases through an extensive experimental study. The alignment of SRank values with each schema's performance metrics underscores this ranking system's effectiveness. The SRank simplifies the schema selection process, assisting users in making informed decisions by reducing the time, cost, and effort of identifying the optimal schema for NoSQL document stores. [ABSTRACT FROM AUTHOR]

The proposed measured data combines PV plant electrical data with associated solar and meteorological data during normal and faulty conditions. Data are collected regarding a domestic rooftop PV plant of 4 kW, located in the La Réunion Island, in the South-West of Indian Ocean. The present dataset includes healthy behavior and different types of shading faults, identified and labelled by means of a numeric variable. The electrical data (voltage, current and power at AC and DC side as well as produced energy and grid frequency) are collected thanks to PV inverters. Global and diffuse irradiance, PV temperature and ambient temperature are acquired thanks to additional sensors. Electrical and meteorological data sampling frequencies are set to 0.2 Hz and 1 Hz respectively. At present, 12 months of data are available and the database is still being updated. The data streams from each connected device require proper techniques to ensure their persistence. To be able to provide both efficient ingestion and retrieval of these time series collections, the NoSQL database management system InfluxDB has been implemented. The whole dataset is available on Zenodo repository, and can be used, for instance, for PV modeling, PV plant behavior analysis, PV production forecasting and PV Fault Detection and Diagnosis (FDD) tool development.

Tiled maps are one of the key GIS technologies used in the development and construction of WebGIS in the era of big data; there is an urgent need for high-performance tile map services hosted on big data GIS platforms. To address the current inefficiency of massive tile map data management and access, this paper proposes a massive tile map data access model that utilizes the jump consistent hash algorithm. Via the uniformity and consistency of a certain seed of a pseudo-random function, the algorithm can generate a storage slot for each tile data efficiently. By recording the slot information in the head of a row key, a uniform distribution of the tiles on the physical cluster nodes is achieved. This effectively solves the problem of hotspotting caused by the monotonicity of tile row keys in the data access process, thereby maximizing the random-access performance of a big data platform and greatly improving concurrent database access. Experiments show that this model can significantly improve the efficiency of tile map data access by more than 39% compared to a direct storage method, thereby confirming the model's advantages in accessing massive tile map data on a big data GIS platform. [ABSTRACT FROM AUTHOR]

Social media have changed the world in the last decade. In fact, the rapid and vast adoption of these platforms had an impact on human behavior, in particular, and society in general. All these channels of media are indispensable for our daily life because they enable us to easily communicate, interact and share a variety of content, with one another. In addition, companies have been interested in this phenomenon to discuss and share products with their customers, improving their production from reviews posted on these platforms. However, analyzing and understanding the volume of generated data from social media requires the use of data warehouse methods and techniques to discover new knowledge to assist companies in the decision-making process. In this research paper, we propose an approach for data warehouse construction for opinion analysis, which can help decision-makers to make decisions by analyzing the users' opinions from the data warehouse. In this paper, we are interested more specifically, in the designing of a data mart from social media accounts, such as Facebook, Twitter, and YouTube. Second, we define a set of semantic relations between the multidimensional concepts of data marts to generate the global schema of the data warehouse by integrating three data mart schemes. Then, we proposed a method of opinion analysis from social media based on machine learning techniques to identify the polarity of the users comments posted on social media, and finally, we are focusing on the implementation of our data warehouse on the NoSQL database to make the analysis task easier for the decision-maker. [ABSTRACT FROM AUTHOR]

Database technology applies to many engineering problems. Hence it is the subject of study at many university majors. Since teaching practical usage of theory at a distance is difficult or even impossible, we present novel mechanisms that significantly enrich e-learning platform functionalities in individualized learning of databases installed on the teacher-managed server. The crucial component of our proposal, Data Adapter, allows for interactive querying and manipulating of the NoSQL databases through an e-learning platform. The Adapter is complemented by the Learning Paths component, which enables teachers the creation of individualized didactic paths for students with special educational needs. The e-learning platform load tests confirm that new activities do not substantially affect the comfort of students' work. Its impact on learning effectiveness was verified by rating the competence test for the group of 104 course participants. Half of them were the participants of the e-learning NoSQL course and used the developed mechanisms, while the remaining, control, group was taught with the conventional methods. The research shows that the proposal successfully improves teaching modern NoSQL languages. The scores improvement in the e-learning group of students compared to results in the control group is noticeable, and observed differences are statistically significant. [ABSTRACT FROM AUTHOR]

The current trend, where information systems and informatization move forward in quick steps, also influenced database systems. The current trend of a data transfer from the databases with a rigid structure to the systems with a free data structure created a vast amount of challenges, including problems with backward compatibility, the inconsistency of data types, or possibly with an edit of the data during the transformation transfer. We created MigrationLayer to solve this problem, which can manage the structural and data compatibility between a relational and nonrelational database and is divided into three modules: The first module is a module of the transformation, respectively, of the data transfer from the relational database to the nonrelational database. The second module is based on the catching of the data during change that could influence the data entering the system, and the third module is a module of backward compatibility, that based on the data change evokes a consolidation edit of the structure and of course, the data. Experiments show MigrationLayer can sufficiently effectively catch the incoming data to the transformation module in real-time, and by that to reduce the time needed for the edit after the whole migrations. The module of the backward compatibility effectively helps us to maintain the backward compatibility between the structures of the relational and the nonrelational database.

• An encoding scheme named Pathed-Dewey Order for nodes in XML documents. • An XML-to-HBase mapping strategy based on Pathed-Dewey Order. • An algorithm to translate XPath into query operations on HBase tables. • Supports efficient storage and query of large collections of small XML documents. • A superior performance of XML2HBase over existing solutions in target use cases. Many big data applications such as smart transportation, healthcare, and e-commerce need to store and query large collections of small XML documents, which has become a fundamental problem. However, existing solutions are inadequate to deliver satisfactory query performance in such circumstances. In this paper, we propose a framework named XML2HBase to address this problem using HBase, a widely deployed NoSQL database. Within this framework, we design a novel encoding scheme called Pathed-Dewey Order and a two-layer mapping method to store XML documents in HBase tables. XML queries, which are represented as XPath expressions, are evaluated through their translation into queries over HBase tables. Based on an in-depth analysis of the characteristics of the proposed approach, we design and integrate four optimization strategies to reduce storage space and query response time. Extensive experiments on two well-known XML benchmarks demonstrate the superior performance of XML2HBase over three state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Medicine is constantly generating new imaging data, including data from basic research, clinical research, and epidemiology, from health administration and insurance organizations, public health services, and non-conventional data sources such as social media, Internet applications, etc. Healthcare professionals have gained from the integration of big data in many ways, including new tools for decision support, improved clinical research methodologies, treatment efficacy, and personalized care. Finally, there are significant advantages in saving resources and reallocating them to increase productivity and rationalization. In this paper, we will explore how big data can be applied to the field of digital health. We will explain the features of health data, its particularities, and the tools available to use it. In addition, a particular focus is placed on the latest research work that addresses big data analysis in the health domain, as well as the technical and organizational challenges that have been discussed. Finally, we propose a general strategy for medical organizations looking to adopt or leverage big data analytics. Through this study, healthcare organizations and institutions considering the use of big data analytics technology, as well as those already using it, can gain a thorough and comprehensive understanding of the potential use, effective targeting, and expected impact.

In the last decade, the techniques of news aggregation and summarization have been increasingly gaining relevance for providing users on the web with condensed and unbiased information. Indeed, the recent development of successful machine learning algorithms, such as those based on the transformers architecture, have made it possible to create effective tools for capturing and elaborating news from the Internet. In this regard, this work proposes, for the first time in the literature to the best of the authors’ knowledge, a methodology for the application of such techniques in news related to cryptocurrencies and the blockchain, whose quick reading can be deemed as extremely useful to operators in the financial sector. Specifically, cutting-edge solutions in the field of natural language processing were employed to cluster news by topic and summarize the corresponding articles published by different newspapers. The results achieved on 22,282 news articles show the effectiveness of the proposed methodology in most of the cases, with 86.8% of the examined summaries being considered as coherent and 95.7% of the corresponding articles correctly aggregated. This methodology was implemented in a freely accessible web application.

NoSQL database has become an important alternative to traditional relational databases. Those databases are prepared by the management of large, continuously and variably changing data sets. They are widely used in cloud databases and distributed systems. With NoSQL databases, static schemes and many other restrictions are avoided. In the era of big data, such databases provide scalable high availability solutions. Their key-value feature allows fast retrieval of data and the ability to store a lot of it. There are many kinds of NoSQL databases with various performances. Therefore, comparing those different types of databases in terms of performance and verifying the relationship between performance and database type has become very important. In this paper, we test and evaluate the Riak key-value database for big data clusters using benchmark tools, where huge amounts of data are stored and retrieved in different sizes in a distributed database environment. Execution times of the NoSQL database over different types of workloads and different sizes of data are compared. The results show that the Riak key-value is stable in execution time for both small and large amounts of data, and the throughput performance increases as the number of threads increases.

Abstract Background Studying structural and functional morphology of small organisms such as monogenean, is difficult due to the lack of visualization in three dimensions. One possible way to resolve this visualization issue is to create digital 3D models which may aid researchers in studying morphology and function of the monogenean. However, the development of 3D models is a tedious procedure as one will have to repeat an entire complicated modelling process for every new target 3D shape using a comprehensive 3D modelling software. This study was designed to develop an alternative 3D modelling approach to build 3D models of monogenean anchors, which can be used to understand these morphological structures in three dimensions. This alternative 3D modelling approach is aimed to avoid repeating the tedious modelling procedure for every single target 3D model from scratch. Result An automated 3D modeling pipeline empowered by an Artificial Neural Network (ANN) was developed. This automated 3D modelling pipeline enables automated deformation of a generic 3D model of monogenean anchor into another target 3D anchor. The 3D modelling pipeline empowered by ANN has managed to automate the generation of the 8 target 3D models (representing 8 species: Dactylogyrus primaries, Pellucidhaptor merus, Dactylogyrus falcatus, Dactylogyrus vastator, Dactylogyrus pterocleidus, Dactylogyrus falciunguis, Chauhanellus auriculatum and Chauhanellus caelatus) of monogenean anchor from the respective 2D illustrations input without repeating the tedious modelling procedure. Conclusions Despite some constraints and limitation, the automated 3D modelling pipeline developed in this study has demonstrated a working idea of application of machine learning approach in a 3D modelling work. This study has not only developed an automated 3D modelling pipeline but also has demonstrated a cross-disciplinary research design that integrates machine learning into a specific domain of study such as 3D modelling of the biological structures.

The analysis of relational and NoSQL databases leads to the conclusion that these data processing systems are to some extent complementary. In the current Big Data applications, especially where extensive analyses (so-called Big Analytics) are needed, it turns out that it is nontrivial to design an infrastructure involving data and software of both types. Unfortunately, the complementarity negatively influences integration possibilities of these data stores both at the data model and data processing levels. In terms of performance, it may be beneficial to use a polyglot persistence, a multimodel approach or multilevel modeling, or even to transform the SQL database schema into NoSQL and to perform data migration between the relational and NoSQL databases. Another possibility is to integrate a NoSQL database and relational database with the help of a third data model. The aim of the paper is to show these possibilities and present some new methods of designing such integrated database architectures.

Big data have analytical potential that was hard to realize with available technologies. After new storage paradigms intended for big data such as NoSQL databases emerged, traditional systems got pushed out of the focus. The current research is focused on their reconciliation on different levels or paradigm replacement. Similarly, the emergence of NoSQL databases has started to push traditional (relational) data warehouses out of the research and even practical focus. Data warehousing is known for the strict modelling process, capturing the essence of the business processes. For that reason, a mere integration to bridge the NoSQL gap is not enough. It is necessary to deal with this issue on a higher abstraction level during the modelling phase. NoSQL databases generally lack clear, unambiguous schema, making the comprehension of their contents difficult and their integration and analysis harder. This motivated involving semantic web technologies to enrich NoSQL database contents by additional meaning and context. This paper reviews the application of semantics in data integration and data warehousing and analyses its potential in integrating NoSQL data and traditional data warehouses with some focus on document stores. Also, it gives a proposal of the future pursuit directions for the big data warehouse modelling phases.