Search

The compilation and maintenance of experimental databases are of crucial importance in all research fields, allowing for researchers to develop and test new methodologies. In this work, we present a flat-file database of experimental dispersion curves and shear wave velocity profiles, mainly from active surface wave testing, but including also data from passive surface wave testing and invasive methods. The Polito Surface Wave flat-file Database (PSWD) is a gathering of experimental measurements collected within the past 25 years at different Italian sites. Discussion on the database content is reported in this paper to evaluate some statistical properties of surface wave test results. Comparisons with other methods for shear wave velocity measurements are also considered. The main novelty of this work is the homogeneity of the PSWD in terms of processing and interpretation methods. A common processing strategy and a new inversion approach were applied to all the data in the PSWD to guarantee consistency. The PSWD can be useful for further correlation studies and is made available as a reference benchmark for the validation and verification of novel interpretation procedures by other researchers.

With the increasing number of information resources in the World Wide Web, how to retrieve real-time information is becoming very important. The resources include traditional databases, flat files,...

Motivation Genomic data repositories like The Cancer Genome Atlas, Encyclopedia of DNA Elements, Bioconductor’s AnnotationHub and ExperimentHub etc., provide public access to large amounts of genomic data as flat files. Researchers often download a subset of data files from these repositories to perform exploratory data analysis. We developed Epiviz File Server, a Python library that implements an in situ data query system for local or remotely hosted indexed genomic files, not only for visualization but also data transformation. The File Server library decouples data retrieval and transformation from specific visualization and analysis tools and provides an abstract interface to define computations independent of the location, format or structure of the file. We demonstrate the File Server in two use cases: (i) integration with Galaxy workflows and (ii) using Epiviz to create a custom genome browser from the Epigenome Roadmap dataset. Availability and implementation Epiviz File Server is open source and is available on GitHub at http://github.com/epiviz/epivizFileServer. The documentation for the File Server library is available at http://epivizfileserver.rtfd.io.

Summary Maximizing the personal, public, research, and clinical value of genomic information will require the reliable exchange of genetic variation data. We report here the Variation Representation Specification (VRS, pronounced “verse”), an extensible framework for the computable representation of variation that complements contemporary human-readable and flat file standards for genomic variation representation. VRS provides semantically precise representations of variation and leverages this design to enable federated identification of biomolecular variation with globally consistent and unique computed identifiers. The VRS framework includes a terminology and information model, machine-readable schema, data sharing conventions, and a reference implementation, each of which is intended to be broadly useful and freely available for community use. VRS was developed by a partnership among national information resource providers, public initiatives, and diagnostic testing laboratories under the auspices of the Global Alliance for Genomics and Health (GA4GH).

Modern shotgun proteomics experiments generate gigabytes of spectra every hour, only a fraction of which were utilized to form biological conclusions. Instead of being stored as flat files in public data repositories, this large amount of data can be better organized to facilitate data reuse. Clustering these spectra by similarity can be helpful in building high-quality spectral libraries, correcting identification errors, and highlighting frequently observed but unidentified spectra. However, large-scale clustering is time-consuming. Here, we present ClusterSheep, a method utilizing Graphics Processing Units (GPUs) to accelerate the process. Unlike previously proposed algorithms for this purpose, our method performs true pairwise comparison of all spectra within a precursor mass-to-charge ratio tolerance, thereby preserving the full cluster structures. ClusterSheep was benchmarked against previously reported clustering tools, MS-Cluster, MaRaCluster, and msCRUSH. The software tool also functions as an interactive visualization tool with a persistent state, enabling the user to explore the resulting clusters visually and retrieve the clustering results as desired.

Command line annotation software tools have continuously gained popularity compared to centralized online services due to the worldwide increase of sequenced bacterial genomes. However, results of existing command line software pipelines heavily depend on taxon specific databases or sufficiently well annotated reference genomes. Here, we introduce Bakta, a new command line software tool for the robust, taxon-independent, thorough and nonetheless fast annotation of bacterial genomes. Bakta conducts a comprehensive annotation workflow including the detection of small proteins taking into account replicon metadata. The annotation of coding sequences is accelerated via an alignment-free sequence identification approach that in addition facilitates the precise assignment of public database cross references. Annotation results are exported in GFF3 and INSDC-compliant flat files as well as comprehensive JSON files facilitating automated downstream analysis. We compared Bakta to other rapid contemporary command line annotation software tools in both targeted and taxonomically broad benchmarks including isolates and metagenomic-assembled genomes. We demonstrated that Bakta outperforms other tools in terms of functional annotations, the assignment of functional categories and database cross-references whilst providing comparable wall clock runtimes. Bakta is implemented in Python 3 and runs on MacOS and Linux systems. It is freely available under a GPLv3 license at https://github.com/oschwengers/bakta. An accompanying web version is available at https://bakta.computational.bio.

Entity Resolution (ER) is the task of identifying different representations of the same real-world object. To achieve scalability and the desired level of quality, the typical ER pipeline includes multiple steps that may involve low-level coding and extensive human labor. We present SystemER, a tool for learning explainable ER models that reduces the human labor all throughout the stages of the ER pipeline. SystemER achieves explainability by learning rules that not only perform a given ER task but are human-comprehensible; this provides transparency into the learning process, and further enables verification and customization of the learned model by the domain experts. By leveraging a human in the loop and active learning, SystemER also ensures that a small number of labeled examples is sufficient to learn high-quality ER models. SystemER is a full-fledged tool that includes an easy to use interface, support for both flat files and semi-structured data, and scale-out capabilities by distributing computation via Apache Spark.

Corbion Group Netherlands B.V. is a company specialized in food ingredients such as lactic acid which involves fermentation process. Thus, a system was created to monitor on real-time fermentation process in order to gain insight and control of the process. This system is called Corbion Process Analysis System, or CorPAS. CorPAS covers many projects in Corbion, and in general, it extracts data from flat files, then transforms and loads the data into Power BI for data visualization. The data extraction still needs to be manually done by the users by opening the databases and copy all the data into flat files. CorPAS 2.0. was initiated to improve the previous system and add a new feature. Automated data extraction was successfully implemented with data transformation and visualization, alongside with implementation of a new feature, such as data science predictive modeling. The solution was built with R language and data visualization tool such as Power BI.

Cloud data migration concept deals with data warehouses used for data analysis and data validation working with the flat file from the multiple sources A record is a text file that contains data. Comma-separated fields delimits the record in the column. Flat files DB does not accommodate various record reports. ETL is a process in which a data repository can be dealing with extracts information from a variety of sources and stores it in an archive. we are happy to reveal about ETL method used for this data validation using the Talend tool.

Nowadays, distributed computing is developing significantly and comes with many strengths since it could decrease the cost of resources while increasing performance. Distributed systems also provide good storage for big data, which requires access to data at a high rate. Because of multiprocessing, this kind of system can improve the response time and number of simultaneously handled requests. However, these systems can bring high performance and the ability to serve with any kind of data, but their architecture is very complex and costs much time and effort to implement. This paper will introduce a Lightweight Distributed File System, which focuses mainly on storage and accessing data to improve the performance of a system. In the performance, measurements show that the system has good I/O performance and scalability, handles up to 5000 requests per second for small files. The paper also contributes to the proposed system three main techniques to improve the scalability and performance, which are a clique, flat file, and consistent hashing.

MetaNetX/MNXref is a reconciliation of metabolites and biochemical reactions providing cross-links between major public biochemistry and Genome-Scale Metabolic Network (GSMN) databases. The new release brings several improvements with respect to the quality of the reconciliation, with particular attention dedicated to preserving the intrinsic properties of GSMN models. The MetaNetX website (https://www.metanetx.org/) provides access to the full database and online services. A major improvement is for mapping of user-provided GSMNs to MXNref, which now provides diagnostic messages about model content. In addition to the website and flat files, the resource can now be accessed through a SPARQL endpoint (https://rdf.metanetx.org).

As exome and whole-genome sequencing cohorts grow in size, the data they produce strains the limits of current tools and data structures. The Variant Call Format (VCF) was originally created as part of the 1,000 Genomes project. Flexible and concise enough to describe the genetic variations of thousands of samples in a single flat file, the VCF has become the standard for communicating the results of large-scale sequencing experiments. Because of its static and text-based structure, VCFs remain cumbersome to parse and filter in an interactive way, even with the aid of indexing. Iterating on previous concepts, we propose here a pipeline for converting VCFs to simple SQLite databases, which allow for rapid searching and filtering of genetic variants while minimizing memory overhead. Code can be found at https://github.com/tkoomar/VCFdbR

Heterogeneous data on distributed computing sources are growing day by day. To manage the data from the distributed sources into a distinct type of application like mobile, cloud, desktop, web etc. is a challenging issue in the global information systems, particularly for cooperation and interoperability. This paper proposes a Data Service Framework, which integrates the data from distributed sources such as databases, Simple Object Access Protocol (SOAP) based web services and flat files, and performs create, read, update and delete (CRUD) operations on it through Representational State Transfer (REST) services over the Hyper Text Transfer Protocol (HTTP). The proposed data service framework also supports java database connectivity (JDBC). Detailed description of the proposed framework and experimental results are reported in this paper.

The integration of distributed data sources is one of the main problems of engineering software. The data integration process for a heterogeneous legacy system is a key aspect of the development of a computerized system and in the integration of a design framework. In this research, our approach to data integration focuses on developing system-building techniques for efficient data integration queries. Keyword-based data searching is investigated and applied within a database for a design framework. A database table connector (DTC) wrapper program is implemented based on the use of data integration processes and keyword-based searching. The DTC provides data integration for various data resources from legacy programs and database management systems using SQL querying. The DTC enables designers and developers to rapidly and efficiently develop integration frameworks for different data resources. This paper also describes the implementation and deployment of the Certification and Aircraft Design Integration System framework, which integrates various analysis and optimization codes, computer-aided design software and database management systems. Multiple data types are used within the framework, including databases, spreadsheets, flat files, XML files and personal data management. Several aircraft design and optimization problems are successfully solved using the developed framework.

Background Despite widespread use of electronic data capture (EDC) systems for research and electronic health records (EHR), most transfer of data between EHR and EDC systems is manual and error prone. Increased adoption of Health Level Seven Fast Healthcare Interoperability Resource (FHIR) application programming interfaces (APIs) in recent years by EHR systems has increased the availability of patient data for external applications such as REDCap. Objective Describe the development of the REDCap Clinical Data Interoperability Services (CDIS) module that provides seamless data exchange between the REDCap research EDC and any EHR system with a FHIR API. CDIS enables end users to independently set up their data collection projects, map EHR data to fields, and adjudicate data transfer without project-by-project involvement from Health Information Technology staff. Methods We identified two use cases for EHR data transfer into REDCap. Clinical Data Pull (CDP) automatically pulls EHR data into user-defined REDCap fields and replaces the workflow of having to transcribe or copy and paste data from the EHR. Clinical Data Mart (CDM) collects all specified data for a patient over a given time period and replaces the process of importing EHR data for registries from research databases. With an iterative process, we designed our access control, authentication, variable selection, and mapping interfaces in such a way that end users could easily set up and use CDIS. Results Since its release, the REDCap CDIS has been used to pull over 19.5 million data points for 82 projects at Vanderbilt University Medical Center. Software and documentation are available through the REDCap Consortium. Conclusions The new REDCap Clinical Data and Interoperability Services (CDIS) module leverages the FHIR standard to enable real-time and direct data extraction from the EHR. Researchers can self-service the mapping and adjudication of EHR data into REDCap. The uptake of CDIS at VUMC and other REDCap consortium sites is improving the accuracy and efficiency of EHR data collection by reducing the need for manual transcription and flat file uploads.

Document forensics remains an important field of digital forensics. To date, previously existing methods focused on the last saved version of the document file stored on the PC; however, the drawback of this approach is that this provides no indication as to how the contents have been modified. This paper provides a novel method for document forensics based on tracking the revision history of a Microsoft Word file. The proposed method concentrates on the TMP file created when the author saves the file and the ASD file created periodically by Microsoft Word during editing. A process whereby the revision history lists are generated based on metadata of the Word, TMP, and ASD files is presented. Furthermore, we describe a technique developed to link the revision history lists based on similarity. These outcomes can provide considerable assistance to a forensic investigator trying to establish the extent to which document file contents have been changed and when the file was created, modified, deleted, and copied.

The paper discusses issues of rule-based data transformation from arbitrary spreadsheet tables to a canonical (relational) form. We present a novel table object model and rule-based language for table analysis and interpretation. The model is intended to represent a physical (cellular) and logical (semantic) structure of an arbitrary table in the transformation process. The language allows drawing up this process as consecutive steps of table understanding, i. e. recovering implicit semantics. Both are implemented in our tool for spreadsheet data canonicalization. The presented case study demonstrates the use of the tool for developing a task-specific rule-set to convert data from arbitrary tables of the same genre (government statistical websites) to flat file databases. The performance evaluation confirms the applicability of the implemented rule-set in accomplishing the stated objectives of the application.

A large amount of data is stored in flat files. They are documents that are subjected to machine learning for clustering, classification and host of other operations. In all such use cases of text mining, there is need for representing documents for efficient processing. One such model which is widely used is known as latent Dirichlet allocation (LDA). This model is the basis for deriving many other models. For instance, topic models came into existence to perform document clustering. Many existing solutions suffer from the lack of sufficient word embeddings. To overcome this problem, in this paper, we propose a generative process model derived from LDA. Its novelty is that it exploits Word2vec for latent feature vector representations. These are from pre-trained word embeddings obtained from large document corpus. The proposed model leverages topic model and improves the performance of document clustering. It is evaluated with a prototype that shows utility of the proposed model.

The Industrial Modeling and Programming Language (IMPL) software designed for the process system engineering (PSE) and operations research (OR) communities is architected to be a structural unit-operation-port-state superstructure (UOPSS) and a semantic quantity-logic-quality phenomena (QLQP) modeling language embedded into a computer programming language with the capability to model and solve industrial optimization, estimation or simulation problems (or sub-problems). In such computer aided process engineering (CAPE) systems, to evolve towards the requirements of advanced modeling and programming skills in a staged fashion, there is a mixture, blend or combination of IML (Industrial Modeling Language), IPL (Industrial Programming Language) and IMPC (Industrial Modeling and Programming Code) for convenience, expressiveness and expedience. IML is how the user may configure the problem using a flat file to fill specific fields or frames by configuring (without coding) the embedded sets (from the UOPSS and QLQP), parameters, variables, and constraints. IPL is how a user may code problems by using computer programming languages to manipulate IMPL using any of its receiving and retrieving routines and with and without using the IML facilities for configuration when required. IMPC is implemented into a machine-coded language (Intel Fortran) to allow IMPL’s modeling facilities to be combined with a powerful general-purpose programming language to formulate any new type of set, parameter, variable and/or constraint. To summarize, IMPL may be considered as a confluence with the scientific disciplines of applied engineering, information and computing technologies, statistics, data analytics and decision sciences, that deliveries easy-to-handle and integrative capabilities for modeling and programming in a progressive-learning evolution and high-performance shareable work amongst PSE and OR teams.

The Consumer Health Vocabulary has been an important contribution to the health informatics field since its introduction in 2006. Many studies have utilized the vocabulary for various scientific research to bridge the gap between consumers and health experts. Given the flat file format of the Consumer Health Vocabulary dataset, we developed a SKOS-based ontology of the dataset. As an ontology, this dataset can be semantically linked to other resources to provide consumer-level meaning. In addition with this artifact, we plan to further expand the terminology.

Background The Systematic Review Data Repository (SRDR) was launched in 2012 as a collaborative, Web-based platform for extracting data from studies included in systematic reviews. The vision was to create a free, open-access repository of extracted data which could reduce unnecessary duplication of effort and facilitate the efficient production of systematic reviews. Objective To promote the entry of structured data into SRDR to promote interoperability and use of systematic review data by end-users. Methods We completed two steps. First, we determined the sources of flat file data that were recently submitted by Evidence-based Practice Centers (EPCs) to SRDR. We conducted a survey of EPC Project Leads that had recently uploaded to SRDR data from a systematic review in a flat file. Second, we determined and pilot tested approaches for importing structured systematic review data. Results The survey was completed by nine of nine eligible EPC Project Leads (100%). In addition to information about usability of SRDR, the primary result was that MS Excel, MS Word and DistillerSR were the most used tools for extracting data and creating tables. An approach for importing MS Excel files into SRDR was refined and pilot tested. An approach was also developed and pilot tested to import files from DistillerSR into SRDR. Detailed instructions, with screenshots, were developed for both approaches. Conclusions Improving the ability to search for and produce reviews that are interoperable for users requires production and storage of systematic review data in a structured format. The results from this work will promote interoperability and use of systematic data in SRDR.

In a Relational Database Management System (RDBMS), we execute queries to retrieve any information or to add, update and delete information. Nowadays, the data volume is increasing in every second, and therefore, it becomes difficult to keep a regular update of this data as in the form of a relational database. In this paper, we present a tool which can execute traditional SQ queries over a large amount of unstructured data. Using this application, the user need not write any traditional queries; therefore, the user requires no background knowledge on SQL or big data processing platform such as Hadoop. It can handle any numbers of flat files for executing queries. The tool provides a user interface which is easy to understand and use. The main contribution of our presented tool is that a user can easily execute queries while utilizing the enormous processing capabilities of Hadoop framework, without having any knowledge of the used framework.

The paper proposes a knowledge-based software platform to generate applications for spreadsheet data extraction and transformation. The platform includes a flexible table object model and a domain-specific language for expressing user-defined rules of table analysis and interpretation. They serve to represent knowledge of table layout and content features, as well as their interpretation, depended on transformation goals. The platform enables translating such user-defined rules to Java programs. The generated source code is serialized as a project prepared for building an executable application by using the Maven tool. The execution of the generated application transforms spreadsheet data from arbitrary form defined by the rules to the canonical one. The empirical results demonstrate the applicability of the software platform to develop applications for converting data from arbitrary spreadsheet tables originated from various domains to relational flat file databases.

This paper describes the methodology of a data warehouse used for analysis, validation using the flat files from various sources. A flat file is a file that stores data in plain text format. A flat file includes a record per line of a table. The columns in a record are delimited by a comma separated fields. Flat file database does not contain multiple tables. Extract Transform Load (ETL), is the process in data warehousing that deals with extracting data from different source system and placing the processed data in the data warehouse. In this paper we are going to explain about etl process, talend tool and validation of the data using talend tool.

When a file is deleted, the storage it occupies is de-allocated but the contents of the file are not erased. An extensive selection of file carving tools and techniques is available to forensic analysts - and yet existing file carving techniques cannot recover database storage because all database storage engines use proprietary and unique storage format. Database systems are widely used to store and process data - both on a large scale (e.g., enterprise networks) and for personal use (e.g., SQLite in mobile devices or Firefox). For some databases, users can purchase specialized recovery tools capable of discovering valid rows in storage and yet there are no tools that can recover deleted rows or make analysts aware of the "unseen" database content.Deletion is just one of the many operations that create de-allocated data in database storage. We use our Database Image Content Explorer tool, based on a universal database storage model, to recover a variety of phantom data: a) data that was actually deleted by a user, b) data that is marked as deleted, but was never explicitly deleted by any user and c) data that is not marked as deleted and had been de-allocated without anyone's knowledge. Data persists in active database tables, in memory, in auxiliary structures or in discarded pages. Strikingly, our tool can even recover data from inserts that were canceled, and thus never officially existed in a data table, which may be of immeasurable value to investigation of financial crimes. In this paper, we describe many recoverable database storage artifacts, investigate survival of data and empirically demonstrate across different databases what our universal, multi-database tool can recover.

With the continuous growth in the amount of data, data storage systems have come a long way from flat files systems to RDBMS, Data Warehousing (DW) and Distributed Data Warehousing systems. This paper proposes a new distributed data warehouse model. The model is built on a novel approach, for the intelligent distribution of data warehouse. Overall the model is named as Intelligent and Distributed Data Warehouse (IDDW). The proposed model has N-levels and is based on top-down hierarchical design approach of building distributed data warehouse. The building process of IDDW starts with the identification of various locations where DW may be built. Initially, a single location is considered at top-most level of IDDW where DW is built. Thereafter, DW at any other location of any level may be built. A method, to transfer concerned data from any upper level DW to concerned lower level DW, is also presented in the paper. The paper also presents IDDW modeling, its architecture based on modeling, the internal organization of IDDW via which all the operations within IDDW are performed.

The Clinical Practice Research Datalink is a nation-wide database of primary healthcare data records in England (UK) linked to several health services. A visit to a health practitioner can result in the digital storing of diagnostic and prescription therapeutic information. Access to patient primary care and linked service data depends on the research in mind; however, typically several flat files that describe patient interactions with a health practitioner are delivered. Some of these files will describe additional data such as the result of medical tests and patient lifestyles, denoted collectively into entity values. This data is used to supplement the medical notes recorded by a general practitioner. We have made available a set of R scripts that reads the clinical flat files, additional clinical flat files and entity values, and returns patient clinical data linked with the requested additional data. We have also included medcode descriptions associated with several entities along with instruction of how to extend the code for additional entities. The code is free to download under the MIT license: https://github.com/acnash/CPRD_Additional_Clinical

The Digest is the definitive historical sourcebook of Roman law compiled under the Byzantine emperor Justinian I (533 CE). This relational database includes the text of Theodor Mommsen’s authoritative edition of the Digest with accompanying information about its compositional structure and its featured jurists. The data was collected from raw text files and transformed to structured machine-readable form in a Python coding environment. Preprocessed data stored in flat files were loaded to a single super lightweight SQLite database (

We describe the implementation of the monitoring for the IT systems at the core of the autonomous driving vehicle. The role of the monitoring is to assist in decision to start the driving cycle and continuous assessment for the fitness to drive the vehicle. The requirements for the monitoring system with the increased resiliency and data replication make it sufficiently different from standard monitoring systems and warrant a unique implementation tuned for the autonomous driving requirements. The monitoring system combines the OS events and real-time measurements of sensor data. The information is stored in flat files for emergency access as well as in a Time Series Data Base (TSDB).

This paper aims towards analyzing the changes in the Internet Performance in India over a span of 3 years by using the clustering algorithm on PingER data. The paper analyzed the enhanced performance of the internet after major changes in the Telecom industry in India in the year 2016 and some insights into the level of improvement. PingER which is short for Ping End-to-end Reporting is a project started with an aim to monitor end-to-end performance of between Internet hosts by SLAC National Accelerator Laboratory, Stanford, California. And over the last decade or so they have collected a huge amount of data which is stored as space separated flat files along with sophisticated methods in order to enable searching for data fast.

Data refers to the concept, fact, or an instruction presented in a defined method. On the other hand, information is organization and interpretation of structured data. Data and information are always found to be intricately tied to each other. Combined analysis of data and information provides a deeper insight to understand what is going on with a particular situation or phenomena. Owing to the development of powerful computational methods, organization of the data and information extraction has now become effortless and expeditious. Computational algorithms have supported in developing numerous powerful databases, which are helpful in storing, accessing, viewing, analyzing, manipulating, and updating the data. Based on data types, three major classes of database models are available, i.e., logical (e.g., object-oriented and relational model), physical (e.g., flat file), and specialized (e.g., XML) databases. Among these, relational model database (SQL) is the most commonly used one for formatting and interpreting the data. At present, biological and medical science research fields have witnessed unprecedented technological advances, due to which data is growing exponentially. However, this large volume of biological data is of little use, if it is not properly stored, organized, and handled. Hence, there is a natural demand for diverse and powerful bioinformatic databases for managing this staggering volume of biological data. This chapter introduces the readers to some basic concepts about biological databases including their types and designs, as well as information retrieval procedures. This chapter basically focuses on important biological databases like GenBank.

Libraries’ shift to the semantic web has been underway for a number of years. Mellon funded projects such as Linked Data for Production (LD4P) [1] or the BIBFRAME European Workshop 2018 in Florence [2] show the commitment of national, public, and academic libraries, as well as vendors, to this transition. Libraries worldwide, however, are enmeshed in hundreds of millions of metadata records communicated through flat files (the MARC formats) [3]. The shift to linked data will require the conversion of these flat files to a semantically expressive model such as the Resource Description Framework (RDF) [4]. The conversion of such large amounts of semantically inexpressive data to semantically rich data will require automated enhancements in the conversion process. Data hidden within the flat files, such as role (author, illustrator, composer, etc.), can greatly aid with the reconciliation of entities within those files. Authify is one of the first tools available to libraries to both convert their metadata to linked data, but also enrich the reconciliation process with semantic data hidden within the MARC fields. As libraries look to convert their legacy data to linked data, Authify can help them move their data to the Web in as a semantically rich way as possible.

BackgroundThe submission of DNA sequences to public sequence databases is an essential, but insufficiently automated step in the process of generating and disseminating novel DNA sequence data. Despite the centrality of database submissions to biological research, the range of available software tools that facilitate the preparation of sequence data for database submissions is low, especially for sequences generated via plant DNA barcoding. Current submission procedures can be complex and prohibitively time expensive for any but a small number of input sequences. A user-friendly software tool is needed that streamlines the file preparation for database submissions of DNA sequences that are commonly generated in plant DNA barcoding.MethodsA Python package was developed that converts DNA sequences from the common EMBL and GenBank flat file formats to submission-ready, tab-delimited spreadsheets (so-called “checklists”) for a subsequent upload to the public sequence database of the European Nucleotide Archive (ENA). The software tool, titled “EMBL2checklists”, automatically converts DNA sequences, their annotation features, and associated metadata into the idiosyncratic format of marker-specific ENA checklists and, thus, generates output that can be uploaded via the interactive Webin submission system of ENA.ResultsEMBL2checklists provides a simple, platform-independent tool that automates the conversion of common plant DNA barcoding sequences into easily editable spreadsheets that require no further processing but their upload to ENA via the interactive Webin submission system. The software is equipped with an intuitive graphical as well as an efficient command-line interface for its operation. The utility of the software is illustrated by its application in the submission of DNA sequences of two recent plant phylogenetic investigations and one fungal metagenomic study.DiscussionEMBL2checklists bridges the gap between common software suites for DNA sequence assembly and annotation and the interactive data submission process of ENA. It represents an easy-to-use solution for plant biologists without bioinformatics expertise to generate submission-ready checklists from common plant DNA sequence data. It allows the post-processing of checklists as well as work-sharing during the submission process and solves a critical bottleneck in the effort to increase participation in public data sharing.

One of the greatest challenges in the field of biology today is the determination of unknown gene function. A number of web applications are currently being developed which aim at providing gene related information, such as gene expression and co-expression data for individual genes. Integrating gene expression networks with external data sources may give researchers additional information about these genes. A regulon is a collection of highly co-expressed genes which can be identified by clustering a network of genes obtained from transcriptomic analysis. Regulon analysis offers a potential way to evaluate functions of genes in a given gene family or for developing hypothesis about the function of unknown genes. Hence, it is important that information about regulons, genes, and co-expression data be available to all biologists in a user friendly manner. Currently, information about regulons and its associated genes are not easily accessible, and in order to view regulon information, manual look up into flat files such as text files is required. In order to avoid this, we come up with an application which aim at providing regulon, gene, and co-expression data for three important species in a user-friendly manner using a web interface. RegulonIT is a web based tool that aims at providing information about regulons and other gene related information. Currently, the tool provides regulon, gene and co-expression information for three species (Arabidopsis thaliana, Saccharomyces cerevisiae and Homo sapiens) for very large transcriptomic datasets using a web interface. RegulonIT is a user friendly platform independent web tool and is available at http://metnetdb.org:9090/regulonit.

PingER (Ping End-to-end Reporting) is a worldwide end-to-end Internet performance measurement framework running for the last 20 years and led by the SLAC National Accelerator Laboratory USA. The objective of the project is to monitor the performance of the Internet links around the world using the ubiquitous ping facility. Currently, the framework comprises of about 50 active Monitoring Agents (MAs) in 20 countries of the world. These MAs probe 700 remote sites located in 170 countries of the world. They are covering an area containing over 98% of the world's population. Currently, the size of the PingER data is about 60 GB stored in 100,000 flat files with a compression ratio of 5:1. The data is of an historical nature and very useful for fine-grained Internet performance analysis. However, the data contains missing values due to congestion, queuing overflow, faulty hardware or software and unavailability of MAs & remote sites. These missing values affect the quality of the Internet performance analysis. The objective of this paper is to substitute the missing values using the k-Nearest Neighbors algorithm (k-NN) and compare the estimation with the statistical method. Therefore, PingER historical data is first transformed into CSV format using a PingER data dimensional model. Afterward, missing values are imputed, using the statistical method and the k-NN algorithm, on data containing the different percentages of missing values. The results conclude that the k-NN algorithm is best suited for the substitution of missing values in the PingER data as compared to the method based on the statistical procedure.

The Mayall 4-meter telescope will soon host the Dark Energy Spectroscopic Instrument (DESI). To integrate DESI into Mayall’s software ecosystem we built an engineering data system called MED (Mayall Engineering Database) that meets the following requirements. (1) MED consists of proven, open-source components to minimize its development and deployment efforts. (2) MED collects data from various sources (log-rotated flat files, existing SQL databases, and our inhouse messaging protocol) to provide a single source for communication between the DESI software and Mayall’s existing subsystems (TCS, dome, primary mirror, weather stations) or future ones (e.g. new calibration lamps). (3) MED facilitates monitoring and troubleshooting by providing an easy way to graph time series in real time or graph time series collected over the past year. (4) MED facilitates generating audio-visual alarms in the control room (e.g. when the dome vignettes) as well as automated email alerts (e.g. when the computer room temperature exceeds 30oC). (5) MED facilitates semiautomated generation of end-of-night metrics (actual time spent on science, instrument overhead, time lost to technical problems or weather, etc.) that go into a night-log report. (6) MED has persistent, long-term storage for metrics that help with performance analysis and strategic planning. (7) The part of MED essential for science operations at night has builtin capability for high-availability. (8) Finally, for maximum flexibility MED is schema-less. We describe MED’s design and open-source technologies. Our requirements involve 3 timescales, each with its own database: SQL (long) is for metrics that we need to keep forever. InfluxDB (medium) is a time series database that stores variables for one year and, along with Grafana as its visualization front-end, is for monitoring and troubleshooting. Redis (short) is an in-memory data structure store that lacks history, as it keeps the latest value for each variable, and is for system operations, acting as the message broker between subsystems. We outline our Redis data structure and its high-resolution time stamp. We also describe MED’s implementation techniques and show how we tie the open-source components together with scripts and config files, and take advantage of Fluentd, another open-source tool, to convert and pipe telemetry from one source to another with ease

Introduction The Metabolomics Workbench Data Repository is a public repository of mass spectrometry and nuclear magnetic resonance data and metadata derived from a wide variety of metabolomics studies. The data and metadata for each study is deposited, stored, and accessed via files in the domain-specific ‘mwTab’ flat file format. Objectives In order to improve the accessibility, reusability, and interoperability of the data and metadata stored in ‘mwTab’ formatted files, we implemented a Python library and package. This Python package, named ‘mwtab’, is a parser for the domain-specific ‘mwTab’ flat file format, which provides facilities for reading, accessing, and writing ‘mwTab’ formatted files. Furthermore, the package provides facilities to validate both the format and required metadata elements of a given ‘mwTab’ formatted file. Methods In order to develop the ‘mwtab’ package we used the official ‘mwTab’ format specification. We used Git version control along with Python unit-testing framework as well as continuous integration service to run those tests on multiple versions of Python. Package documentation was developed using sphinx documentation generator. Results The ‘mwtab’ package provides both Python programmatic library interfaces and command-line interfaces for reading, writing, and validating ‘mwTab’ formatted files. Data and associated metadata are stored within Python dictionary- and list-based data structures, enabling straightforward, ‘pythonic’ access and manipulation of data and metadata. Also, the package provides facilities to convert ‘mwTab’ files into a JSON formatted equivalent, enabling easy reusability of the data by all modern programming languages that implement JSON parsers. The ‘mwtab’ package implements its metadata validation functionality based on a pre-defined JSON schema that can be easily specialized for specific types of metabolomics studies. The library also provides a command-line interface for interconversion between ‘mwTab’ and JSONized formats in raw text and a variety of compressed binary file formats. Conclusions The ‘mwtab’ package is an easy-to-use Python package that provides FAIRer utilization of the Metabolomics Workbench Data Repository. The source code is freely available on GitHub and via the Python Package Index. Documentation includes a ‘User Guide’, ‘Tutorial’, and ‘API Reference’. The GitHub repository also provides ‘mwtab’ package unit-tests via a continuous integration service. Electronic supplementary material The online version of this article (10.1007/s11306-018-1356-6) contains supplementary material, which is available to authorized users.

Nowadays, most of the information is stored digitally. Digital information is from a high level of view it is just an array of bits. In order to figure out its real meaning special software which interprets it is required. Therefore, if by evolution of technology this software cannot be executed anymore there is potential risk that also the data interpreted by it becomes not useful. The goal of Digital Preservation is to stop occurrences of such phenomenon. Data is commonly stored in files each file has a specific format or structure, by knowing it user can figure out the real meaning of raw data stored in the file as an array of bits. Digital Preservation considers valid file format as a perquisite for file to be in usable form, with valid is meant that a specific file is structured conform its declared file format. In this paper we throw a spotlight on the accuracy and capability of these file validation tests. Therefore, we present some open source software which are able to automatically identify and verify the file format. We focus more on file types they can identify, and how they work in large scale data sets.

The processing of massive numbers of small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs the traditional file systems such as ExtFS and may cause inefficiency when accessing small files randomly located in the disk. This paper focuses on optimizing the performance of data servers in accessing massive numbers of small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS is designed to substitute the traditional file system on data servers and can be deployed underneath distributed file systems that store massive numbers of small files. iFlatLFS can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1 to 64 KB, iFlatLFS is faster than Ext4 by 48 and 54 percent on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45 and 49 percent on average for random read access and hybrid access (the mix of read and write accesses), respectively.

This paper proposes a new approach, called file concealer (FC), to conceal files in a computer system. FC modifies metadata about a file in NTFS (New Technology File System) to hide the file. Unlike traditional hooking methods which can be easily detected by antivirus software, experimental results show that it is difficult for antivirus software to detect the files hidden by FC. Moreover, to enhance the concealment capability of FC, FC also rearranges the order of some data sectors of a hidden file. As a result, even if another person finds the original sectors used by the hidden file, it is difficult for him to recover the original content of the hidden file. Experimental results show that even data recovery tools cannot restore the content of a hidden file. All information that is required to restore a hidden file is stored in a file, called recovery file hereafter. When a user uses FC to hide a file, the user can specify any file as a host file, such as an image file, to which the recovery file will be appended. As a result, the user can easily restore a hidden file; however, it is difficult for other person to detect or restore the hidden file and the related recovery file.

Users are interested in current web pages contents in optimum time. In this paper, we describe the technique to detect multiple changes in a web page or Flat files in form of insertion or deletion of the content or structure change. This efficient approach is suitable for client application and for implementing server applications that could serve the needs of users in monitoring modifications to HTML web pages or to flat files made over the time, and that allow reporting and visualizing changes in order to gain insight about the significance and types of such changes.

At present, as the amount of digitized information is increasing geometrically, the importance of digitized information as critical clues from the perspective of criminal investigation is also increasing. The importance of digital forensics has also magnified accordingly. As a fundamental technique of digital forensics, recovery of data deleted to conceal information is extremely important and needs to be studied. Although file system types and versions are diversified, sufficient studies have not been performed. The XFS file system was designed to process huge files, and thus it is expected to be applied in server systems. In this paper, we analyze the XFS file system using its structure and metadata information; then, we propose a technique to recover the deleted files through previous analysis results, and verify it through testing.

Big data is the latest industry buzzword to describe large volume of structured and unstructured data that can be difficult to process and analyze. Most of organization looking for the best approach to manage and analyze the large volume of data especially in making a decision. XML and JSON are chosen by many organization because of powerful approach during retrieval and storage processes. However, these approaches, the execution time for retrieving large volume of data are still considerably inefficient due to several factors. In this contribution, three databases approaches namely Extensible Markup Language (XML), Java Object Notation (JSON) and Flat File database approach were investigated to evaluate their suitability for handling thousands records of publication data. The results showed flat file is the best choice for query retrieving speed and CPU usage. These are essential to cope with the characteristics of publication’s data. Whilst, XML, JSON and Flat File database approach technologies are relatively new to date in comparison to the relational database. Indeed, Text File Format technology demonstrates greater potential to become a key database technology for handling huge data due to increase of data annually.

The purpose of this paper is to realize a comparative analysis of relational and non-relational database management systems. Both conceptual and technical characteristics are introduced, presenting the benefits and disadvantages of each model, using a hybrid application. The application, named SuperstoreSalesReporting, generates comparative reports which describe the technical characteristics of both SQL and NoSQL databases based on the executed operations. The application constitutes the contribution of this study, together with the creation of the SQL Server internal processes for migrating data from a flat file into a star schema.

micro:bit provides a persistent file system that allows you to store files in the flash memory. The size of the storage reserved for the file system is approximately 30KB. However, micro:bit provides a flat file system, so you can't store files in directories to create a hierarchy. The stored files will remain intact until you either delete them or re-flash the device.

Relational DBMSs (RDBMSs) remain the most popular tool for processing structured data in data intensive domains. However, most of stand-alone data mining packages process flat files outside a RDBMS. In-database data mining avoids export-import data/results bottleneck as opposed to use stand-alone mining packages and keeps all the benefits provided by a RDBMS. The paper presents an approach to data mining inside a RDBMS based on a parallel implementation of user-defined functions (UDFs). Such an approach is implemented for PostgreSQL and modern Intel MIC (Many Integrated Core) architecture. The UDF performs a single mining task on data from the specified table and produces a resulting table. The UDF is organized as a wrapper of an appropriate mining algorithm, which is implemented in C language and is parallelized by the OpenMP technology and thread-level parallelism. The heavy-weight parts of the algorithm are additionally parallelized by intrinsic functions for MIC platforms to reach the optimal loop vectorization manually. The library of such UDFs supports a cache of precomputed mining structures to reduce costs of further computations. In the experiments, the proposed approach shows good scalability and overtakes R data mining package.

This paper describes an approach to formulate queries in the language of first order logic over data from disparate sources distributed over a network. The data sources are treated as if they were all in a common database. The data sources may incorporate different stored or computed methods of providing data– web services and REST APIs, XML/JSON repositories, web pages, full featured databases, flat files, etc. We describe the technical foundations and benefits of this approach and compare it to other extant solutions to this problem. This approach enables end users to formulate ad-hoc queries in logic to correlate the data sources. For programmers creating applications, using a logic-based language to specify correlation criteria shortens the development life cycle and reduces cost of maintenance, compared to manually coding the equivalent correlation algorithms in procedural languages.

Cloud-based Analytics-as-a-Service (CLAaaS) was developed by Zulkernine et al. with a goal to simplifying big data analytics users. It provides software-as-a-service access to a variety of back end analytics tools and data stores. One of the tools is the Workflow Instance Generation and Selection (WINGS). WINGS allows users to reuse predefined workflows and their components containing semantic meta-data to define new workflows; late binding of the workflows to data at the time of execution to enable the use of most recent data, and definition of domain specific software code as custom analytic components in workflows. How ever, the data used in WINGS for the workflows are mostly flat files that are stored on the WINGS server or shared directories. The goal of this project is to add support for big data storage systems to WINGS and validate the extensions using multiple data analytic workflows of different complexities with data residing in a variety of back end data sources. The extension allows the CLAaaS users to create, validate and execute analytic workflows in a distributed environment and use data from multiple big data storage systems. We validate our work using four big data storage systems in WINGS workflows namely, Apache HBase, MongoDB, MySQL with a front-end interface.