Search

micro:bit provides a persistent file system that allows you to store files in the flash memory. The size of the storage reserved for the file system is approximately 30KB. However, micro:bit provides a flat file system, so you can't store files in directories to create a hierarchy. The stored files will remain intact until you either delete them or re-flash the device.

Relational DBMSs (RDBMSs) remain the most popular tool for processing structured data in data intensive domains. However, most of stand-alone data mining packages process flat files outside a RDBMS. In-database data mining avoids export-import data/results bottleneck as opposed to use stand-alone mining packages and keeps all the benefits provided by a RDBMS. The paper presents an approach to data mining inside a RDBMS based on a parallel implementation of user-defined functions (UDFs). Such an approach is implemented for PostgreSQL and modern Intel MIC (Many Integrated Core) architecture. The UDF performs a single mining task on data from the specified table and produces a resulting table. The UDF is organized as a wrapper of an appropriate mining algorithm, which is implemented in C language and is parallelized by the OpenMP technology and thread-level parallelism. The heavy-weight parts of the algorithm are additionally parallelized by intrinsic functions for MIC platforms to reach the optimal loop vectorization manually. The library of such UDFs supports a cache of precomputed mining structures to reduce costs of further computations. In the experiments, the proposed approach shows good scalability and overtakes R data mining package.

The world is advancing and developing with the ticking seconds. New technologies, inventions and discoveries have remarkably changed the world as it was before, making lives a lot simpler and easier. From the hunter of gathers society till the scientific society today, many new fields have emerged to combat diseases, make earth a substantially friendly place to live in and fore mostly, making changes for the welfare of mankind. One of such newly developing field is ‘Bioinformatics ‘which is the use of computers and its software’s for acquisition, management and analysis of biological information.

This paper describes an approach to formulate queries in the language of first order logic over data from disparate sources distributed over a network. The data sources are treated as if they were all in a common database. The data sources may incorporate different stored or computed methods of providing data– web services and REST APIs, XML/JSON repositories, web pages, full featured databases, flat files, etc. We describe the technical foundations and benefits of this approach and compare it to other extant solutions to this problem. This approach enables end users to formulate ad-hoc queries in logic to correlate the data sources. For programmers creating applications, using a logic-based language to specify correlation criteria shortens the development life cycle and reduces cost of maintenance, compared to manually coding the equivalent correlation algorithms in procedural languages.

Cloud-based Analytics-as-a-Service (CLAaaS) was developed by Zulkernine et al. with a goal to simplifying big data analytics users. It provides software-as-a-service access to a variety of back end analytics tools and data stores. One of the tools is the Workflow Instance Generation and Selection (WINGS). WINGS allows users to reuse predefined workflows and their components containing semantic meta-data to define new workflows; late binding of the workflows to data at the time of execution to enable the use of most recent data, and definition of domain specific software code as custom analytic components in workflows. How ever, the data used in WINGS for the workflows are mostly flat files that are stored on the WINGS server or shared directories. The goal of this project is to add support for big data storage systems to WINGS and validate the extensions using multiple data analytic workflows of different complexities with data residing in a variety of back end data sources. The extension allows the CLAaaS users to create, validate and execute analytic workflows in a distributed environment and use data from multiple big data storage systems. We validate our work using four big data storage systems in WINGS workflows namely, Apache HBase, MongoDB, MySQL with a front-end interface.

SummaryWith the rapid accumulation of sequencing data from genomic and metagenomic studies, there is an acute need for better tools that facilitate their analyses against biological functions. To this end, we developed MetQy, an open–source R package designed for query–based analysis of functional units in [meta]genomes and/or sets of genes using the The Kyoto Encyclopedia of Genes and Genomes (KEGG) database. Furthermore, MetQy contains visualization and analysis tools and facilitates KEGG’s flat file manipulation. Thus, MetQy enables better understanding of metabolic capabilities of known genomes or user–specified [meta]genomes by using the available information and can help guide studies in microbial ecology, metabolic engineering and synthetic biology.Availability and ImplementationThe MetQy R package is freely available and can be downloaded from our group’s website (http://osslab.lifesci.warwick.ac.uk) or GitHub (https://github.com/OSS-Lab/MetQy).ContactO.Soyer@warwick.ac.uk

Big datasets are often stored in flat files and can contain contradictory data. Contradictory data undermines the soundness of the information from a noisy dataset. Traditional tools such as pie chart and bar chart are overwhelmed when used to visually identify contradictory data in multidimensional attribute-values of a big dataset. This work explains the importance of identifying contradictions in a noisy dataset. It also examines how contradictory data in a large and noisy dataset can be mined and visually analysed. The authors developed ‘ConTra’, an open source application which applies mutual exclusion rule in identifying contradictory data, existing in comma separated values (CSV) dataset. ConTra’s capability to enable the identification of contradictory data in different sizes of datasets is examined. The results show that ConTra can process large dataset when hosted in servers with fast processors. It is also shown in this work that ConTra is 100% accurate in identifying contradictory data of objects whose attribute values do not conform to the mutual exclusion rule of a dataset in CSV format. Different approaches through which ConTra can mine and identify contradictory data are also presented.

Background Bioinformatic research is increasingly dependent on large-scale datasets, accessed either from private or public repositories. An example of a public repository is National Center for Biotechnology Information's (NCBI’s) Reference Sequence (RefSeq). These repositories must decide in what form to make their data available. Unstructured data can be put to almost any use but are limited in how access to them can be scaled. Highly structured data offer improved performance for specific algorithms but limit the wider usefulness of the data. We present an alternative: lightly structured data stored in Apache Kafka in a way that is amenable to parallel access and streamed processing, including subsequent transformations into more highly structured representations. We contend that this approach could provide a flexible and powerful nexus of bioinformatic data, bridging the gap between low structure on one hand, and high performance and scale on the other. To demonstrate this, we present a proof-of-concept version of NCBI’s RefSeq database using this technology. We measure the performance and scalability characteristics of this alternative with respect to flat files. Results The proof of concept scales almost linearly as more compute nodes are added, outperforming the standard approach using files. Conclusions Apache Kafka merits consideration as a fast and more scalable but general-purpose way to store and retrieve bioinformatic data, for public, centralized reference datasets such as RefSeq and for private clinical and experimental data.

With advances in high throughput brain imaging at the cellular and sub-cellular level, there is growing demand for platforms that can support high performance, large-scale brain data processing and analysis. In this paper, we present a novel pipeline that combines Accumulo, D4M, geohashing, and parallel programming to manage large-scale neuron connectivity graphs in a cloud environment. Our brain connectivity graph is represented using vertices (fiber start/end nodes), edges (fiber tracks), and the 3D coordinates of the fiber tracks. For optimal performance, we take the hybrid approach of storing vertices and edges in Accumulo and saving the fiber track 3D coordinates in flat files. Accumulo database operations offer low latency on sparse queries while flat files offer high throughput for storing, querying, and analyzing bulk data. We evaluated our pipeline by using 250 gigabytes of mouse neuron connectivity data. Benchmarking experiments on retrieving vertices and edges from Accumulo demonstrate that we can achieve 1–2 orders of magnitude speedup in retrieval time when compared to the same operation from traditional flat files. The implementation of graph analytics such as Breadth First Search using Accumulo and D4M offers consistent good performance regardless of data size and density, thus is scalable to very large dataset. Indexing of neuron subvolumes is simple and logical with geohashing-based binary tree encoding. This hybrid data management backend is used to drive an interactive web-based 3D graphical user interface, where users can examine the 3D connectivity map in a Google Map-like viewer. Our pipeline is scalable and extensible to other data modalities.

Traditional methods of testing were based on Instrument-Centric models which included sensor and measurement capabilities inside of instruments that allowed test engineers to be able to perform measurements and collect data. Such data would then be analyzed by software to determine whether the test resulted in a pass or fail disposition. Modern technologies have significantly changed the equation into a Data-Centric model whereby data, and lots of it, are the key to gain insight into production and ensure quality. However, most test systems struggle to handle the data that is created by their systems. This paper will discuss three topics regarding data; a) Data Management — flat files, databases and the pros and cons of each, b) Data Visualization-viewing reports and charts as well as higher level analytics (pareto, cpk), and c) Data Validation — post-test limit checking and yield analysis.

Since the Human Genome Project was completed in 2003, many data scientists have developed algorithms in order to store and query high volumes of genomic data. The most common data storage techniques employed in these algorithms are flat files or relational databases. While sophisticated indexing techniques can accelerate queries, an alternative is to store biological sequence data directly in a way that supports efficient queries. Here we introduce a new algorithm that aims to compress the redundant information and improve the performance of query speed with the help of graphical databases, which have been commercial available since the mid-late 2000s. A graphical database stores information using nodes and relationships (edges). Our approach is to identify subsequences that are common among many sequences, and to store these as "common nodes" in the graphical database. This is accomplished for sequencing data as follows: split the whole sequence into k-mers: if a given k-mer is common to enough sequences, then it is labeled as a common segment; if a k-mer is unique (or common to too few sequences), then it is labeled as a single segment. Thus, common nodes and single nodes are formed from common segments and single segments, respectively. These two kinds of nodes are connected by edges in the graphical database, allowing each original sequences to be reconstructed by following edges in the graph. This graphical database model allows for fast taxonomic queries of 16S rDNA. When queried, the database can first attempt to find common nodes that match the query sequence, and subsequently follow edges to single nodes to refine the search. This approach is analogous to that of "compressive genomics", except that the compression is implicit in the graphical database storage model. Beyond simple sequence queries, this graphical database representation also supports variability analysis, which identifies highly variable vs. conserved regions of 16S sequence. Regions of low variability correspond to common nodes, while regions of high variability correspond to a variety of paths through single nodes. Figure illustrates common and single nodes, and a corresponding plot of variability. Benchmarking of sequence search indicates that query time in graphical databases is significantly faster than in flat files or relational databases. Implementation of graphical databases in genomic data analysis will allow for accelerated search, and may lend itself to other forms of efficient analysis, such as tetramer frequency analysis, which is useful in metagenomic binning.

Summary JASPAR is a widely used open-access database of curated, non-redundant transcription factor binding profiles. Currently, data from JASPAR can be retrieved as flat files or by using programming language-specific interfaces. Here, we present a programming language-independent application programming interface (API) to access JASPAR data using the Representational State Transfer (REST) architecture. The REST API enables programmatic access to JASPAR by most programming languages and returns data in eight widely used formats. Several endpoints are available to access the data and an endpoint is available to infer the TF binding profile(s) likely bound by a given DNA binding domain protein sequence. Additionally, it provides an interactive browsable interface for bioinformatics tool developers. Availability and implementation This REST API is implemented in Python using the Django REST Framework. It is accessible at http://jaspar.genereg.net/api/ and the source code is freely available at https://bitbucket.org/CBGR/jaspar under GPL v3 license. Supplementary information Supplementary data are available at Bioinformatics online.

Database community has been involved in topics related to improve data-related techniques or to solve data access efficiency. Health domain has been attracting the interest of database community as an application domain for many database research topics, including: (i) health data heterogeneity (e.g., different health bioimages protocols), (ii) data size (e.g., patient health related data), (iii) biomedical signals (e.g., electrocardiography data, ECG), (iv) geographical data (e.g., epidemiological one), and more recently (v) genomic and proteomic data as well as NGS data. In this chapter we present experiences from the last decade, made in a medical school, where we used database experiences to manage and analyse clinical, biological and health related data. The methodology is problem oriented and shows how to start from a problem defined in the medical domain and choose and apply techniques often known by the database community. In this chapter interesting results, in terms of applications to the clinical and medical domains, are reported.

As the generation and use of genomic datasets is becoming increasingly common in all areas of biology, the need for resources to collate, analyse and present data from one or more genome projects is becoming more pressing. The Ensembl platform is a powerful tool to make genome data and cross-species analyses easily accessible through a web interface and a comprehensive application programming interface. Here we introduce GenomeHubs, which provide a containerized environment to facilitate the setup and hosting of custom Ensembl genome browsers. This simplifies mirroring of existing content and import of new genomic data into the Ensembl database schema. GenomeHubs also provide a set of analysis containers to decorate imported genomes with results of standard analyses and functional annotations and support export to flat files, including EMBL format for submission of assemblies and annotations to International Nucleotide Sequence Database Collaboration. Database URL: http://GenomeHubs.org

Sensible decision-support is gaining widespread attention as health check institutions and governing bodies turn towards utilize information management for effective and efficient healthcare delivery and quality assured outcomes. Quantifiable Data can be created based on effective Data Integrity, Data Management, Rule Prediction and Work Optimization Techniques as parts of knowledge management in need environments. Data originates from collection of separate healthcare files which are then entered into individual flat file. Data Warehouse transforms this file into a multidimensional data model that can be efficiently navigated for global patterns and trends except for personal information, all other dimensions were composed of attributes recorded for each visit or test. Managing and exploring vast collections of heterogeneous data is common place in the large health check organizations.

the rapid growth of both Internet and Multimedia the subject of hidden exchange of information has gained a great importance. The current work is based on translation of information of a digitized image into a sound file. Later the image file can be retrieved by authorized personnel by performing the reverse process on the sound file. The three dimensional RGB image matrix is converted into a two dimensional sound file. During the retrieving time, the reverse procedure is performed on the sound file in order to get back the original image file.

A signature is a data pattern of interest in a large data file or set of large data files. Such signatures that need to be found arise in applications such as DNA sequence analysis, network intrusion detection, biometrics, large scientific experiments, speech recognition and sensor networks. Related to this is string matching. More specifically we envision a problem where long linear data files (i.e., flat files) contain multiple signatures that are to be found using a multiplicity of processors (parallel processor). This paper evaluates the performance of finding signatures in files residing in the nodes of parallel processors configured as trees, two dimensional meshes and hypercubes. We assume various combinations of sequential and parallel searching. A unique feature of this work is that it is assumed that data is pre-loaded onto processors, as may occur in practice, thus load distribution time need not be accounted for. Elegant expressions are found for average signature searching time and speedup, and graphical results are provided.

log file shows the behavior of user when they access the website. It is in text format and automatically generated by web server whenever user accesses the website. Entry of particular web page with date, time, cs-method, cs-uri-stem and other information is generated automatically in text file for every access of web page. Web Usage Mining is different mining technique apply on weblog file to discover the different pattern which is useful for efficient design of website, performance enhancement of server etc. Mining of web log file consist of three steps Data Preprocessing, Pattern Discovery and Pattern Analysis. Data preprocessing task convert the web log file in database by applying data extraction, data storage and data cleaning technique. In existing technique data extraction from the log file is perform line by line and store in multidimensional array and then extracted data field will be store in database and then data cleaning is performed. In this research paper, a new technique is being proposed for Data Preprocessing of web log using Microsoft Excel. The experimental results show that the proposed new technique using Microsoft excel file is an effective compare to existing technique.

Summary & PurposeAdvances in sequencing technology have made it possible to generate large numbers of long, high-accuracy sequencing reads. For instance, the new PacBio Sequel platform can generate hundreds of thousands of high-quality circular consensus sequences in a single run (Rhoads and F. 2015; Hebert et al. 2018). Good programs exist for aligning these reads for genome assembly (Chaisson and Tesler 2012; Li 2018). However, these long reads can also be used for other purposes, such as sequencing PCR amplicons that contain various features of interest. For instance, PacBio circular consensus sequences have been used to identify the mutations in influenza viruses in single cells (Russell et al. 2019), or to link barcodes to gene mutants in deep mutational scanning (Matreyek et al. 2018). For such applications, the alignment of the sequences to the targets may be fairly trivial, but it is not trivial to then parse specific features of interest (such as mutations, unique molecular identifiers, cell barcodes, and flanking sequences) from these alignments.Here we describe alignparse, a Python package for parsing complex sets of features from long sequences that map to known targets. Specifically, it allows the user to provide complex target sequences in Genbank format that contain an arbitrary number of user-defined sub-sequence features. It then aligns the sequencing reads to these targets and filters alignments based on whether the user-specified features are present with the desired identities (which can be set to different thresholds for different features). Finally, it parses out the sequences, mutations, and/or accuracy of these features as specified by the user. The flexibility of this package therefore fulfills the need for a tool to extract and analyze complex sets of features in large numbers of long sequencing reads.

Because the Excel file is often edited, the data blocks stored in the hard disk which make up the Excel file is often not continuous. Discontinuous storage fragmented files, deleted or formatted, with the current data recovery software to restore the effect is very poor, often file recovery, but can not open or is garbled. Therefore it makes sense to study an effective recovery method to recover Excel fragmented files. Based on "Laura" file format, first introduced the Windows platform fragmented files, and then given an Excel file fragmentation recognition method, final based on debris identify given Excel file fragmentation restructuring.

Based on the analysis of the structure of the 3DS file, this paper uses recursive method to collect content of 3DS file and designs the data structure to save them. Then, reading and drawing principle for 3DS file is studied by DirectX 11. Finally, the system for reading and drawing 3DS file is designed and implemented based on DirectX11 in VS2010, and achieves better results.

Constructing the tree type logic structure of a file and introducing the view mechanism of database system into the file system, the hierarchical structure of view files is created. Different files ( view files ) are provided to different users according to their levels or interests by mapping layer by layer with beginning of the minimum access unit (the logic block). It can solve the problem of the discrepancy of users views.

Forensic Analyst encounters a mixed file fragments in the absence of File Table information i.e., files" metadata. File Carving is a process of reconstructing files from mixed file fragments without using files" metadata. File Carving is an interesting and challenging problem in digital forensics and Data Recovery. Recently there have been number of research papers in the area of File Carving. In this paper authors describe File Carving and present its Literature Survey. The implementation of Greedy Sequential Unique Path File Carving algorithm for 4-bit bitmap files, with the help of function prototypes using C language and brief explanation of these prototypes is explained. The experimental results are also shown. The experimental results show that the files can be reconstructed from their fragments without knowing files" metadata. However there are limitations of these methods as mentioned in section 5.

File system metadata management has become a bottleneck for many data-intensive applications that rely on high-performance file systems. Part of the bottleneck is due to the limitations of an almost 50 year old interface standard with metadata abstractions that were designed at a time when high-end file systems managed less than 100 MB. Today's high-performance file systems store 7 to 9 orders of magnitude more data, resulting in numbers of data items for which these metadata abstractions are inadequate, such as directory hierarchies unable to handle complex relationships among data. Users of file systems have attempted to work around these inadequacies by moving application-specific metadata management to relational databases to make metadata searchable. Splitting file system metadata management into two separate systems introduces inefficiencies and systems management problems. To address this problem, we propose QMDS: a file system metadata management service that integrates all file system metadata and uses a graph data model with attributes on nodes and edges. Our service uses a query language interface for file identification and attribute retrieval. We present our metadata management service design and architecture and study its performance using a text analysis benchmark application. Results from our QMDS prototype show the effectiveness of this approach. Compared to the use of a file system and relational database, the QMDS prototype shows superior performance for both ingest and query workloads.

The concept of Big Data refers to very large datasets, sets of sizes where you need data warehouses to store the data, where you typically need sophisticated algorithms to handle the data, and distributed computations to get anywhere with it. At the very least, we talk many gigabytes of data but also are often dealing with terabytes or exabytes.

Traditionally, software systems used different methods of accessing data from different datastores. For example, in order to access data from relational databases, flat files, and XML files, developers needed to learn and master different object models. These object models were often totally different from each other in terms of usage syntax and underlying philosophy. Beginning with the .NET Framework 3.5, Microsoft introduced Language Integrated Query (LINQ) technology that simplifies the hassle involved while consuming different sources of data. LINQ comes in three main flavors: LINQ to Objects, LINQ to ADO.NET, and LINQ to XML.

Today’s world is knowledge based. In the earliest days, knowledge was gathered through observation. Later, knowledge not only was gathered through observation, but also confirmed by actually doing and then extended by experimenting further. Knowledge thus gathered was applied to practical fields and extended by analogy to other fields. Today, knowledge is gathered and applied by analyzing, or deep-diving, into the data accumulated through various computer applications, web sites, and more. The advent of computers complemented the knowledge of statistics, mathematics, and programming. The enormous storage and extended computing capabilities of the cloud, especially, have ensured that knowledge can be quickly derived from huge amounts of data and also can be used for further preventive or productive purposes. This chapter provides you with the basic knowledge of where and how business analytics is used.

Technological advancements have brought about profound improvements in the ability to capture high-quality medical images. However, full understanding of two-dimensional (2D) data is limited by the brain’s capacity to translate it into a three-dimensional (3D) representation within the mind’s eye. Digital 3D imaging and printing technology are emerging as essential for comprehension of the anatomic spatial relationships in complex congenital heart disease. With deteriorating pathologic specimens and the inherent limitations of 2D content, an opportunity to enhance medical visualization and understanding led to the creation of the 3D Heart Library, a collection within the National Institutes of Health (NIH) 3D Print Exchange. The Exchange serves as an open-source resource from the NIH containing bioscientific and medical 3D models for digital 3D imaging and printing. The purpose is to facilitate storage and dissemination of 3D representations of heart anatomy, with a special focus on congenital heart disease. Submissions are evaluated through a process that builds on the traditional peer-review model of medical case study publications. Users can access high-quality, validated models that can be used in educational curricula or as references to inform decision making in medical treatment plans. In this chapter, we describe the creation of this resource, the methodologies behind 3D image segmentation and quality control, and the utility of the 3D Heart Library today, and into the future.

Illuminating the ecological and evolutionary dynamics of parasites is one of the most pressing issues facing modern science, and is critical for basic science, the global economy, and human health. Extremely important to this effort are data on the disease-causing organisms of wild animal hosts (including viruses, bacteria, protozoa, helminths, arthropods, and fungi). Here we present an updated version of the Global Mammal Parasite Database, a database of the parasites of wild ungulates (artiodactyls and perissodactyls), carnivores, and primates, and make it available for download as complete flat files. The updated database has more than 24,000 entries in the main data file alone, representing data from over 2700 literature sources. We include data on sampling method and sample sizes when reported, as well as both "reported" and "corrected" (i.e., standardized) binomials for each host and parasite species. Also included are current higher taxonomies and data on transmission modes used by the majority of species of parasites in the database. In the associated metadata we describe the methods used to identify sources and extract data from the primary literature, how entries were checked for errors, methods used to georeference entries, and how host and parasite taxonomies were standardized across the database. We also provide definitions of the data fields in each of the four files that users can download.

A new era of Business Intelligence Applications has started in-order to solve the problems of Relational Database Management System (RDBMS). If the data is coming from different sources, the data couldn't be sent to any one of the target RDBMS. To overcome this problem BI Apps evolved, i.e. applying ETL (Extraction Transformation and Loading) on different schemas, flat files and many other sources the data need to be transformed into target area. Then the tasks (mappings) were developed and scheduled, the data need to be monitored in-order to load into target area. Finally the data is represented using pictorial representations on the dashboard pages. These dashboard pages consists of statistical representations like pie, pivot, horizontal, vertical bars so that the growth of business can be evaluated by the CEO's.

SQLite3 is a small-sized database engine and smartphones use this database to store records of communication data. With the wide application of mobile communication technology, mobile phone forensics is an effective means to crack down on mobile phone crimes. This paper focuses on analysis of the SQLite3 database file structure and format to recover valuable information. The suggested method uses extraction recovery with cyclic reading to locate and extract deleted data, then translates hex file into meaningful characters. Further, tests prove that it can also be used to recover partly overwritten records.

In this paper, we report on the analysis of data from Android mobile phones of 38 users, composed of access traces of the users' mobile file systems during 30 days. We shed new light on the file usage patterns and present the data in terms of file size distributions, file sessions, file lifetime, file access activity and read / write access patterns. We characterize different distributions and extract conclusions about usage patterns of Android file systems.

This paper details the development of the pvpaR (PV Performance Analysis in R) web application built on open-source technologies for the automated and user-friendly evaluation of photovoltaic (PV) system performance and identification and classification of faults. pvpaR's ecosystem is based on the R statistical computing project, both for the back-end as well as for the front-end which uses the Shiny web application framework. Currently, the core of the application incorporates models for synthesizing time-series of irradiance, module temperature, PV system voltage, current, DC and AC power. These are used to validate the imported field measurements and create the comparison which the fault identification function is dependent upon. The web application can currently import measurements from flat files and databases. It has been released under the GNU Affero GPL license to encourage contributions, with the goal to become a useful tool for the PV community and PV system owners.

A file can be called an atomic information element that is found all over in computer world just like dust particles in real world. Files are universally used, ranging from personal data on desktop computers, to organizational data shared over local networks and the data that exist on the web. That's why files caught the attention of ours and lead us to study the life of a file from Cradle to Grave or Eternity. In our study we investigate: the birth of a file; sources of file's birth; the tsunami of files; the good, the bad and the ugly files; file aging; file states transition; file lost and the quest of finding a file; a journey in past and future; the death of a file; file resurrection and the gods of clouds.

A file system is used to manage data on storage media. The FAT (File Allocation Table) file system was originally designed for floppy drives that were less than 500KB in size, and these drives were not capable of fast random reads and writes. FAT has been adapted to work on other types of storage devices since, and it is still widely used today. It is the standard file system used by microprocessors and embedded devices with constrained resources. Micro-controllers, like the Arduino, only officially support the FAT file system when interacting with a SD card. FAT performs well when data is read or written sequentially, but when data is read or written randomly, there is an impact on performance for large files on page based flash devices that cannot utilize caching strategies. Applications that perform random reading and writing are impacted by this architectural issue. For example, flash data structures, like a B-tree, will have poor performance since random reading is utilized to look up values. TEFS (Tiny Embedded File System) uses a simplified tree indexing structure to take advantage of the fast random reads and writes of flash storage and guarantees that the number of page reads and writes will stay constant as the file size increases when randomly reading or writing. Experimental results show that TEFS has significantly better performance than FAT on the Arduino for random I/Os, and the more efficient TEFS page interface is even slightly faster than FAT for sequential reading and writing.

We present a database of repetitive DNA elements, called Dfam (http://dfam.janelia.org). Many genomes contain a large fraction of repetitive DNA, much of which is made up of remnants of transposable elements (TEs). Accurate annotation of TEs enables research into their biology and can shed light on the evolutionary processes that shape genomes. Identification and masking of TEs can also greatly simplify many downstream genome annotation and sequence analysis tasks. The commonly used TE annotation tools RepeatMasker and Censor depend on sequence homology search tools such as cross_match and BLAST variants, as well as Repbase, a collection of known TE families each represented by a single consensus sequence. Dfam contains entries corresponding to all Repbase TE entries for which instances have been found in the human genome. Each Dfam entry is represented by a profile hidden Markov model, built from alignments generated using RepeatMasker and Repbase. When used in conjunction with the hidden Markov model search tool nhmmer, Dfam produces a 2.9% increase in coverage over consensus sequence search methods on a large human benchmark, while maintaining low false discovery rates, and coverage of the full human genome is 54.5%. The website provides a collection of tools and data views to support improved TE curation and annotation efforts. Dfam is also available for download in flat file format or in the form of MySQL table dumps.

We present the Database of Disordered Protein Prediction (D2P2), available at http://d2p2.pro (including website source code). A battery of disorder predictors and their variants, VL-XT, VSL2b, PrDOS, PV2, Espritz and IUPred, were run on all protein sequences from 1765 complete proteomes (to be updated as more genomes are completed). Integrated with these results are all of the predicted (mostly structured) SCOP domains using the SUPERFAMILY predictor. These disorder/structure annotations together enable comparison of the disorder predictors with each other and examination of the overlap between disordered predictions and SCOP domains on a large scale. D2P2 will increase our understanding of the interplay between disorder and structure, the genomic distribution of disorder, and its evolutionary history. The parsed data are made available in a unified format for download as flat files or SQL tables either by genome, by predictor, or for the complete set. An interactive website provides a graphical view of each protein annotated with the SCOP domains and disordered regions from all predictors overlaid (or shown as a consensus). There are statistics and tools for browsing and comparing genomes and their disorder within the context of their position on the tree of life. © The Author(s) 2012. Published by Oxford University Press.

managing files in a server system has the same magnitude as managing the World Wide Web due to the dynamic nature of the file system. Even searching for files over the file system is time consuming because finding a file on hard disk is a long-running task. Every file on the disk has to be read with dangling pointers to files which no longer exist because they have been changed, moved or deleted. This makes the user frustrated. The Automatic file indexing framework facilitates users to resolve file names and locate documents stored in file repositories. The main design objective of the framework is to maintain sub-indexes at the folder level that have the full knowledge of the revisions that are made at the folder level automatically. This research proposes a framework that manages the creation and maintenance of the file index, with the use of Resources Description Framework (RDF) and retrieval using semantic query languages i.e. SPARQL. The sub-indexes are maintained hierarchically starting from the leaf node to the root node recursively. The proposed framework will monitor the file system continuously and update individual folder descriptors (sub-indexes) stored on each node as the file system changes making the cached indexes resilient to any file changes. The framework is resilient of file or folder name changes. Further, the study explores avenues to build an offline semantic index that can be used by the clients to perform distribute file search without performing the search on the server itself. This is viable since the framework uses semantic languages to describe and build file descriptors that can easily integrate semantic indexing and hence this makes the index readily available for the Web.

The use of clinical imaging modalities within the pharmaceutical research space provides value and challenges. Typical clinical settings will utilize a Picture Archive and Communication System (PACS) to transmit and manage Digital Imaging and Communications in Medicine (DICOM) images generated by clinical imaging systems. However, a PACS is complex and provides many features that are not required within a research setting, making it difficult to generate a business case and determine the return on investment. We have developed a next-generation DICOM processing system using open-source software, commodity server hardware such as Apple Xserve®, high-performance network-attached storage (NAS), and in-house-developed preprocessing programs. DICOM-transmitted files are arranged in a flat file folder hierarchy easily accessible via our downstream analysis tools and a standard file browser. This next-generation system had a minimal construction cost due to the reuse of all the components from our first-generation system with the addition of a second server for a few thousand dollars. Performance metrics were gathered and the system was found to be highly scalable, performed significantly better than the first-generation system, is modular, has satisfactory image integrity, and is easier to maintain than the first-generation system. The resulting system is also portable across platforms and utilizes minimal hardware resources, allowing for easier upgrades and migration to smaller form factors at the hardware end-of-life. This system has been in production successfully for 8 months and services five clinical instruments and three pre-clinical instruments. This system has provided us with the necessary DICOM C-Store functionality, eliminating the need for a clinical PACS for day-to-day image processing.

The capability to validate and view or play binary file formats, as well as to convert binary file formats to standard or current file formats, is critically important to the preservation of digital data and records. This paper describes the extension of context-free grammars from strings to binary files. Binary files are arrays of data types, such as long and short integers, floating-point numbers and pointers, as well as characters. The concept of an attribute grammar is extended to these context-free array grammars. This attribute grammar has been used to define a number of chunk-based and directory-based binary file formats. A parser generator has been used with some of these grammars to generate syntax checkers (recognizers) for validating binary file formats. Among the potential benefits of an attribute grammar-based approach to specification and parsing of binary file formats is that attribute grammars not only support format validation, but support generation of error messages during validation of format, validation of semantic constraints, attribute value extraction (characterization), generation of viewers or players for file formats, and conversion to current or standard file formats. The significance of these results is that with these extensions to core computer science concepts, traditional parser/compiler technologies can potentially be used as a part of a general, cost effective curation strategy for binary file formats.

A method is proposed for determining chromium content in the carbon steel pipes of a nuclear power plant (NPP) to evaluate wall thinning caused by flow-accelerated corrosion (FAC). A flat file was used to obtain filings samples. To assess sampling quality, a disk form of SRM 1227 was ground with the flat file, and the amount of Cr in the filings was determined by ICP-AES. The content of chromium in the filings of SRM 1227 was estimated as six times higher than the certified value due to the contamination of chromium in the file. To eliminate chromium contamination from the file, it was coated with WC-12Co using high-velocity oxygen-fuel (HVOF) spraying systems. After obtaining filings samples using the coated file, Cr content in four types of disk-form SRMs was determined by ICP-AES. The recoveries of Cr in the disk-form SRMs were in the range of 95.4-102.6%, with relative standard deviations from 0.43 to 3.0%. The Cr contents in the filings collected from the used outlet headers of the nuclear power plants using the flat file coated were in the range of 0.110.19%.

In order to generate STEP-NC data model in keeping with the ISO14649 protocol, extend the STEP model to the field of the CNC, the paper analyzes STEP standard and its file structure, develops the display software which can be read and display the STEP format file based on open source Open CASCADE geometric kernel by using the Visual Studio 2008 software development environment, designs one instance by using Pro/E software, and stored as a STEP format file in Pro/E environment, and then, using the software developed to read and display the STEP file. The results of this study improve the product data exchange and integration capabilities, lay a good foundation for generating the STEP-NC format file in the future, extracting the file STEP-NC machining information, and achieving CAD / CAM / CAPP integration system.

The one of the most time consuming steps for association rule mining is the computation of the frequency of the occurrences of itemsets in the database. The hash table index approach converts a transaction database to an hash index tree by scanning the transaction database only once. Whenever user requests for any Uniform Resource Locator (URL), the request entry is stored in the Log File of the server. This paper presents the hash index table structure, a general and dense structure which provides web page set extraction from Log File of server. This hash table provides information about the original database. Web Page set mining (WPs-Mine) provides a complete representation of the original database. This approach works well for both sparse and dense data distributions. Web page set mining supported by hash table index shows the performance always comparable with and often better than algorithms accessing data on flat files. Incremental update is feasible without reaccessing the original transactional database.

Digital elevation models (DEMs) constitute a valuable source of data for a number of geoscience-related applications. The Shuttle Radar Topography Mission (SRTM) collected and made available to the public the world's largest DEM (composed of billions of points) until that date. The SRTM DEM is stored on the NASA repository as a well-organized collection of flat files. The retrieval of this stored topographic information about a region of interest involves one selection of a proper list of files, their downloading, data filtering in the desired region, and their processing according to user needs. With the aim to provide an easier and faster access to this data by improving its further analysis and processing, we have indexed the SRTM DEM by means of a spatial indexing based on the kd-tree data structure, called the Q-tree. This paper is the first in a two-part series that describes the method followed to build an index on such huge amounts of data, minimizing the number of insert operations. We demonstrate that our method can build a very efficient space-partitioning index, with good performance in both point and range queries on the spatial data. To the best of our knowledge, this is the only successful spatial indexing proposal in the literature that deals with such a huge volume of data.

Summary: Circleator is a Perl application that generates circular figures of genome-associated data. It leverages BioPerl to support standard annotation and sequence file formats and produces publication-quality SVG output. It is designed to be both flexible and easy to use. It includes a library of circular track types and predefined configuration files for common use-cases, including. (i) visualizing gene annotation and DNA sequence data from a GenBank flat file, (ii) displaying patterns of gene conservation in related microbial strains, (iii) showing Single Nucleotide Polymorphisms (SNPs) and indels relative to a reference genome and gene set and (iv) viewing RNA-Seq plots. Availability and implementation: Circleator is freely available under the Artistic License 2.0 from http://jonathancrabtree.github.io/Circleator/ and is integrated with the CloVR cloud-based sequence analysis Virtual Machine (VM), which can be downloaded from http://clovr.org or run on Amazon EC2. Contact: jcrabtree@som.umaryland.edu Supplementary information: Supplementary data are available at Bioinformatics online.

FlyFactorSurvey (http://pgfe.umassmed.edu/TFDBS/) is a database of DNA binding specificities for Drosophila transcription factors (TFs) primarily determined using the bacterial one-hybrid system. The database provides community access to over 400 recognition motifs and position weight matrices for over 200 TFs, including many unpublished motifs. Search tools and flat file downloads are provided to retrieve binding site information (as sequences, matrices and sequence logos) for individual TFs, groups of TFs or for all TFs with characterized binding specificities. Linked analysis tools allow users to identify motifs within our database that share similarity to a query matrix or to view the distribution of occurrences of an individual motif throughout the Drosophila genome. Together, this database and its associated tools provide computational and experimental biologists with resources to predict interactions between Drosophila TFs and target cis-regulatory sequences.

In this paper, we examine the potential to hide data in an ISO9660 file system, which is used to store data on CD-ROMs. By design, this file system allows for multiple directory trees and different byte orderings of essential data. We describe how data could be hidden in an ISO9660 file system and create test images using the described techniques. We test commonly used forensics tools to determine if the hidden data can be seen. The test results show that different tools show and hide different data. Some tools show all of the data, some tools show some of the data, and other tools show none of the data.

Feature selection is an essential data processing step to remove irrelevant and redundant attributes for shorter learning time, better accuracy, and better comprehensibility. A number of algorithms have been proposed in both data mining and machine learning areas. These algorithms are usually used in a single table environment, where data are stored in one relational table or one flat file. They are not suitable for a multi-relational environment, where data are stored in multiple tables joined to one another by semantic relationships. To address this problem, in this article, we propose a novel approach called FARS to conduct both Feature And Relation Selection for efficient multi-relational classification. Through this approach, we not only extend the traditional feature selection method to select relevant features from multi-relations, but also develop a new method to reconstruct the multi-relational database schema and eliminate irrelevant tables to improve classification performance further. The results of the experiments conducted on both real and synthetic databases show that FARS can effectively choose a small set of relevant features, thereby enhancing classification efficiency and prediction accuracy significantly.

User-Defined Functions (UDFs) represent an extensibility mechanism provided by most DBMSs, whose execution happens in main memory. Also, UDFs leverage the DBMS multi-threaded capabilities and exploit the C language speed and flexibility for mathematical computations. In this article, we study how to accelerate computation of sufficient statistics on large data sets with UDFs exploiting caching and sampling techniques. We present an aggregate UDF computing multidimensional sufficient statistics that benefit a broad array of statistical models: the linear sum of points and the quadratic sum of cross-products of point dimensions. Caching can be applied when the data set fits in main memory. Otherwise, sampling is required to accelerate processing of very large data sets. Also, sampling can be applied on data sets that can be cached, to further accelerate processing. Experiments carefully analyze performance and accuracy with real and synthetic data sets. We compare UDFs working inside the DBMS and C++ reading flat files, running on the same hardware. We show UDFs can have similar performance to C++, even if both exploit caching and multi-threading. As expected, C++ is much faster than UDFs when the data set is scanned from disk. We carefully analyze the case where sampling is required with larger data sets. We show geometric and bootstrapping sampling techniques can be faster than performing full tables scans, providing high accuracy estimation of mean, variance and correlation. Even further, sampling on cached data sets can provide accurate answers in a few seconds. Detailed experiments illustrate UDF optimizations including diagonal matrix computation, join avoidance and acceleration with a multi-core CPU, when available. A profile of UDF run-time execution shows the UDF is slowed down by I/O when reading from disk.