Search

Electrocardiograph (ECG) analysis brings a lot of technical concerns because ECG is one of the tools frequently used in the diagnosis of cardiovascular disease. According to World Health Organization (WHO) statistic in 2012, cardiovascular disease constitutes about 48% of noncommunicable deaths worldwide. Although there are many ECG related researches, there is not much efforts in big data computing for ECG analysis which involves dataset more than one gigabyte. ECG files contain graphical data and the size grows as period of data recording gets longer. Big data computing for ECG analysis is critical when many patients are involved. Recently, the implementation of MapReduce in cloud computing becomes a new trend due to its parallel computing characteristic. Since large ECG dataset consume much time in analysis processes, this project will construct a cloud computing approach for ECG analysis using MapReduce in order to investigate the effect of MapReduce in enhancing ECG analysis efficiency in cloud computing. The project is expected to reduce ECG analysis process time for large ECG dataset.

The new project gigabyte fast access to customer terminal (G.fast) has been initiated within ITU Telecommunication Standardization Sector (ITU-T) SG15 in February 2011. This new copper access technology is expected to provide aggregate (upstream and downstream) data rates up to 1 Gbps over distances up to 250 meter. This paper discusses the concept of G.fast channel modeling based on stochastic far end crosstalk (FEXT) model against 99% worst case model. The motivation to study these two type of model is for analyze accurate FEXT and speed prediction. The cable insertion loss and FEXT is measured for a 50-pairs 100 meter cable with 0.5mm cable gauge. The transmitter simulation is based on the G.fast International Telecommunication Union (ITU) specification. The bit rate performance is simulated for distance up to 250 meter. The results of the measurement and simulation show a better correlation between FEXT and bit rate. The results can be used as a base for future work on G.fast channel modelling emphasizes on FEXT.

Execution of a Global Cloud Resolving Model (GCRM) at target resolutions of 2-4 km will generate, at a minimum, 10s of Gigabytes of data per variable per snapshot. Writing this data to disk, without creating a serious bottleneck in the execution of the GCRM code, while also supporting efficient post-execution data analysis is a significant challenge. This paper discusses an Input/Output (IO) application programmer interface (API) for the GCRM that efficiently moves data from the model to disk while maintaining support for community standard formats, avoiding the creation of very large numbers of files, and supporting efficient analysis. Several aspects of the API will be discussed in detail. First, we discuss the output data layout which linearizes the data in a consistent way that is independent of the number of processors used to run the simulation and provides a convenient format for subsequent analyses of the data. Second, we discuss the flexible API interface that enables modelers to easily add variables to the output stream by specifying where in the GCRM code these variables are located and to flexibly configure the choice of outputs and distribution of data across files. The flexibility of the API is designed to allow model developers to add new data fields to the output as the model develops and new physics is added. It also provides a mechanism for allowing users of the GCRM code to adjust the output frequency and the number of fields written depending on the needs of individual calculations. Third, we describe the mapping to the NetCDF data model with an emphasis on the grid description. Fourth, we describe our messaging algorithms and IO aggregation strategies that are used to achieve high bandwidth while simultaneously writing concurrently from many processors to shared files. We conclude with initial performance results.

The quality of navigation service is determined by the accuracy of the available data. For existing navigation services, a full map update is provided in order to keep the map data of mobile devices current. As content and services of mobile devices have recently been diversifying, the size of map data managed in mobile devices has increased, reaching several gigabytes in size. It generally takes tens of minutes to write several gigabytes of data into mobile device storage. For traditional navigation systems, a complicated storage structure called a physical storage format (PSF) is used to assure maximum processing performance of map data in mobile devices within limited resources. Consequently, even though modified navigation map data actually affects only a portion of a map, the full map data is updated because partial updates are not possible. In this paper, a navigation system is studied to solve this difficult partial map update problem. The map air update navigation system, which is the result of this study, provides real-time partial map updating using wireless communications. Keywords: Mobile navigation system, navigation map update, mobile database, version control, routing service.

Modern scientific experiments can generate hundreds of gigabytes to terabytes or even petabytes of data that may be maintained in large numbers of relatively small files. Frequently, these data must be disseminated to remote collaborators or computational centers for data analysis. Moving this dataset with high performance and strong robustness and providing a simple interface for users are challenging tasks. We present a data transfer framework comprising a high-performance data transfer library based on GridFTP, an extensible data scheduler with four data scheduling policies, and a GUI that allows users to transfer their dataset easily, reliably, and securely. This system incorporates automatic tuning mechanisms to select at runtime the number of concurrent threads to be used for transfers. Also included are restart mechanisms for handling client, network, and server failures. Experimental results indicate that our data transfer system can significantly improve data transfer performance and can recover well from failures. Copyright © 2011 John Wiley & Sons, Ltd.

During the last decade pathology has benefited from the rapid progress of image digitizing technology. The improvement in this technology had led to the creation of slide scanners which are able to produce whole slide images (WSI) which can be explored by image viewers in a way comparable to the conventional microscope. The file size of the WSI ranges from a few megabytes to several gigabytes, leading to challenges in the area of image storage and management when they will be used routinely in daily clinical practice. Digital slides are used in pathology for education, diagnostic purposes (clinicopathological meetings, consultations, revisions, slide panels and, increasingly, for upfront clinical diagnostics) and archiving. As an alternative to conventional slides, WSI are generally well accepted, especially in education, where they are available to a large number of students with the full possibilities of annotations without the problem of variation between serial sections. Image processing techniques can also be applied to WSI, providing pathologists with tools assisting in the diagnosis-making process. This paper will highlight the current status of digital pathology applications and its impact on the field of pathology.

With the exponential growth of moving objects data to the Gigabyte range, it has become critical to develop effective techniques for indexing, updating, and querying these massive data sets. To meet the high update rate as well as low query response time requirements of moving object applications, this paper takes a novel approach in moving object indexing. In our approach, we do not require a sophisticated index structure that needs to be adjusted for each incoming update. Rather, we construct conceptually simple short-lived index images that we only keep for a very short period of time (sub-seconds) in main memory. As a consequence, the resulting technique MOVIES supports at the same time high query rates and high update rates, trading this property for query result staleness. Moreover, MOVIES is the first main memory method supporting time-parameterized predictive queries. To support this feature, we present two algorithms: non-predictive MOVIES and predictive MOVIES. We obtain the surprising result that a predictive indexing approach--considered state-of-the-art in an external-memory scenario--does not scale well in a main memory environment. In fact, our results show that MOVIES outperforms state-of-the-art moving object indexes such as a main-memory adapted B x -tree by orders of magnitude w.r.t. update rates and query rates. In our experimental evaluation, we index the complete road network of Germany consisting of 40,000,000 road segments and 38,000,000 nodes. We scale our workload up to 100,000,000 moving objects, 58,000,000 updates per second and 10,000 queries per second, a scenario at a scale unmatched by any previous work.

In the current scenario, around 35 billion Internet of Things (IoT) devices is connected to the internet. By 2025, it is predicted that the number will grow between 80 and 120 billion devices connected to the internet, supporting to generate 180 trillion gigabytes of new sensor data that year. The IoT sensor data is generated from various heterogeneous devices, communication protocols, and data formats that are enormous in nature. This huge amount of sensor data is unable to acquire and analyze manually. This is a significant problem for IoT application developers to make the integration of IoT sensor data automatically. However, the large amount of data has led to the inadequacy of the manual data acquisition and stressed the urgency into the research of IoT based frameworks in automatic. In this paper, we have proposed a framework for IoT sensor data acquisition and analysis (FSDAA). The FSDAA has been implemented on the ThingSpeak IoT Cloud platform for data analysis and visualizations, and compared with the state of the art schemes. Finally, the results show that the proposed FSDAA is efficient in terms of Accuracy, Precision, Recall, and F-measure.

Ocean Networks Canada (ONC) operates ocean observatories on all three of Canada's coasts. The instruments produce 300 gigabytes of data per day with over 600 terabytes archived so far. The majority of this data is acoustic, both passive (335 TB) and active (20 TB). This demonstrates the unprecedented capability of cabled observatories to provide unlimited power and data for high bandwidth, continuous data acquisition. Handling this data is a challenge. Metadata, calibration, quality control, and access must be considered. The volume of data is too great for most users to handle. Even if they could store and process it, data transfer to users' computers is a limiting, and perhaps unnecessary step. To address these challenges, ONC has developed a data portal, known as Oceans 2.0, that includes on-demand user-configurable online previewing and processing and a computing “sandbox” where users can upload their own code to process the data. The data portal is now fully accessible by web services. The sandbox is a contained, secure environment with direct access to the data. This paper will present our experience and best practices, including use cases, from acquisition to adding value to the data with these new computing methods.Ocean Networks Canada (ONC) operates ocean observatories on all three of Canada's coasts. The instruments produce 300 gigabytes of data per day with over 600 terabytes archived so far. The majority of this data is acoustic, both passive (335 TB) and active (20 TB). This demonstrates the unprecedented capability of cabled observatories to provide unlimited power and data for high bandwidth, continuous data acquisition. Handling this data is a challenge. Metadata, calibration, quality control, and access must be considered. The volume of data is too great for most users to handle. Even if they could store and process it, data transfer to users' computers is a limiting, and perhaps unnecessary step. To address these challenges, ONC has developed a data portal, known as Oceans 2.0, that includes on-demand user-configurable online previewing and processing and a computing “sandbox” where users can upload their own code to process the data. The data portal is now fully accessible by web services. The sandbox is...

Conventional interpretation workflows require a limited number of key horizons to be mapped. These are used to construct a generalized geologic model on the basis of which economic decisions are made. In the process, gigabytes of data are reduced to just a few kilobytes of interpreted data, during which potentially valuable information is often lost.

We describe the design of, and experience with, Query, a monitoring system that supports the Akamai EdgePlatform. Query is a foundation of Akamai's approach to administering its distributed computing platform, allowing administrators, operations staff, developers, customers, and automated systems near real-time access to data about activity in Akamai's network. Users extract information regarding the current state of the network via a SQL-like interface. Versions of Query have been deployed since the inception of Akamai's platform, and it has scaled to support a distributed platform of 60,000+ servers, collecting over 200 gigabytes of data and answering over 30,000 queries approximately every 2 minutes.

In order to solve the problem that when the output data stream of a certain type of high resolution aerial camera reaches 200 megabytes per second,conventional methods are difficult to record the image data in real time,a self-starting,multi-channel hybrid synchronization direct-write method of SCSI hard disk is proposed.The new designed state controller optimizes the recording time series of the SCSI hard disk,so that multiple concurrent recording channels can generate separate direct-write pulses automatically according to the data cache and the status of the hard disk.The image data can be recorded on the hard disk with high speed and hybrid synchronization.Experimental results prove that the method can record the image data at the rate of 200 megabytes per second,storage capacity attained 280 Gigabytes,and moreover,the structure is simple and practical.

What is "big data" anyway? Gigabytes? Terabytes? Petabytes? A brief personal memory may provide some perspective. In the late 1980s at Columbia University I had the chance to play around with what at the time was a truly enormous "disk": the IBM 3850 MSS (Mass Storage System). The MSS was actually a fully automatic robotic tape library and associated staging disks to make random access, if not exactly instantaneous, at least fully transparent. In Columbia’s configuration, it stored a total of around 100 GB. It was already on its way out by the time I got my hands on it, but in its heyday, the early to mid-1980s, it had been used to support access by social scientists to what was unquestionably "big data" at the time: the entire 1980 U.S. Census database.

A locally assembled image viewer system with pocket-size iPOD (80 GB) and flash (2 GB) drives for gigabyte storage, display and transfer of digital medical images, oriented towards training purposes, is described. Both the iPOD and flash drive enable storage of thousands of images from diverse medical-imaging equipments. The iPOD, in addition, can display with sufficient resolution any of these images and serves as a transportable preview device. Through the use of a computer, these devices can access/ store/ display the images/ photos from a CD, digital camera or the internet. A TV image viewing unit is also provided. The operational features and the advantages of these devices are discussed in detail. The quality assurance (QA) of the displays has been successfully carried out with standard test patterns. The image quality has been tested with dynamic and static medical images. The system will be highly useful for storage and remote display of multitude of images from several modalities in the hospital, as well as other images, from the point of view of education and training. It has good potential for use in clinical diagnosis as well. Other recent advancements using iPHONE and improved but expensive computers, integrated with picture archiving and communication system (PACS) as well as radiology and hospital information system (RHIS) for versatile applications in modern radiology, are also highlighted.This system, assembled with indigenous equipments, is much less expensive and specially suited for teaching radiologists, physicists and technologists, particularly in developing countries.

Gigabytes of personal data are stored electronically by businesses, governments, universities, hospitals, and organisations on their customers, citizens, students, patients, and employees respectively. These personal data include social security numbers (or identity card numbers), names, addresses, dates-of-birth, driver's licence numbers, credit card numbers, medical histories, phone numbers, and email addresses. Unfortunately, organisations view personal data as valuable assets and often keep them indefinitely.

Editor's note: Launched in January 2008 in direct response to the feedback we received from the recent JVIB Readers' Survey, JVIB's new "Technology Q&A" column, written by technology expert Jay Leventhal, editor in chief of AccessWorld[R]: Technology and People Who Are Blind or Visually Impaired, provides readers what they said they wanted: more practical information on assistive technology and how such products are being used in daily life, education, and employment. Readers are encouraged to submit their own technology questions to the column's editor by e-mail to: . In this month's "Technology Q&A" column, the technology expert answers a question from a mother of a teenager with low vision who wrote: My daughter Skylar will be making the transition from junior high to high school this fall. All her friends have iPods, and since Sky's birthday is coming up, her father and I are looking to purchase an MP3 player for her so she can enjoy listening to music on the go just like her friends. Skylar's macular degeneration makes it impossible for her to read the screen of an iPod, so I am wondering if the Technology Expert has any advice regarding accessible MP3 players that would work for my daughter. Thank you for any advice you can give! I give Skylar's mother a lot of credit for asking about "accessible" MP3 players. I would expect most readers of this column to be familiar with Apple's iPod, but for those who are not, a description of this device follows this introduction. As Skylar's mother indicates, iPods are not accessible to people who are blind or have low vision. Luckily for Skylar and other people who are visually impaired who wish to own an MP3 players, there are also MP3 players in the marketplace that are designed specifically for people with visual impairments. These devices are in the same price range as iPods, and they offer a lot of potential for helping, as well as entertaining, visually impaired teenagers like Skylar. IPODS The Apple iPod is by far the most popular MP3 player on the market. The catch is that there is no built-in speech or text enlargement capability included in an iPod, which means that people with visual impairments cannot read the menus, make and select playlists, or comfortably perform other functions. I happen to know a teenager who is blind who uses an iPod by counting the clicks emitted by the device's click wheel and memorizing some of the menus, but she also had a sighted person help her delete some of the menus to limit her choices. iPods are controlled by their patented click wheel, which consists of a center Select button surrounded by a larger circle. The click wheel concept comes into play when one wants to adjust the volume or scroll through a menu item, Instead of pressing buttons to accomplish these tasks, it is possible to simply run a finger along the outer circle. A clockwise movement increases the volume or moves forward in a list of menu items, and counterclockwise movement decreases the volume or moves backward in the menu list. An audible "click" sound is emitted for each unit that the volume is increased or as one goes from one menu item to another. The iPod Classic measures 4.1 by 2.4 by 0.53 inches and weighs 4.9 or 5.7 ounces, depending on the size of its hard drive. The Classic is a bit larger in size than other mainstream MP3 players, but its hard drive is available in 80 GB (gigabyte) or 160 GB versions, allowing one to carry up to 80,000 songs in the device. The 80 GB version costs $249, and the 160 GB version costs $349. Measuring 2.75 by 2.06 by 0.26 inches and weighing only 1.8 ounces, the Nano is a smaller, more portable iPod. The hard drive is also significantly smaller in size than the Classic, available in 4 GB or 8 GB versions, but it functions almost exactly like the other versions. The 4 GB version costs $149, and the 8 GB costs $199. …

Many of us in clinical medicine aren't fond of very large numbers. We deal with and intuitively grasp milligrams, grams, and kilograms, and centimeters and meters, but what sense does the difference between a megabyte and a terabyte make? By contrast, computer professionals routinely work with gigabytes, and given the popularity of multigigabyte, MP3 players, pocket-form terabytes (one thousand gigabytes) are soon to come. So what do these gargantuan numbers mean to the clinician? With more storage in less space the opportunity exists to develop new methods to house medical information in tinier packages. This should translate to ease of use, convenience, and portability that we need or might enjoy. Technical advancement however shouldn't be viewed simply as a parade of interesting devices. It's also about processes, surprises, breakthroughs, and the migration of technology from place to place in society. This movement of technology appears capricious, until innovation alights a consumer application, and then we can count on clever folks transplanting it to professional use.

Dynameomics is a project to investigate and catalog the native-state dynamics and thermal unfolding pathways of representatives of all protein folds using solvated molecular dynamics simulations, as described in the preceding paper. Here we introduce the design of the molecular dynamics data warehouse, a scalable, reliable repository that houses simulation data that vastly simplifies management and access. In the succeeding paper, we describe the development of a complementary multidimensional database. A single protein unfolding or native-state simulation can take weeks to months to complete, and produces gigabytes of coordinate and analysis data. Mining information from over 3000 completed simulations is complicated and time-consuming. Even the simplest queries involve writing intricate programs that must be built from low-level file system access primitives and include significant logic to correctly locate and parse data of interest. As a result, programs to answer questions that require data from hundreds of simulations are very difficult to write. Thus, organization and access to simulation data have been major obstacles to the discovery of new knowledge in the Dynameomics project. This repository is used internally and is the foundation of the Dynameomics portal site http://www.dynameomics.org. By organizing simulation data into a scalable, manageable and accessible form, we can begin to address substantial questions that move us closer to solving biomedical and bioengineering problems.

Geographic information systems (GIS) and remote sensing (RS) have been used extensively in many land surveying and natural resources management applications. With ever-advancing technology, geospatial data collection is done at increasingly wide extent and finer resolution. A single remotely-sensed image and other raster data can easily exceed a gigabyte (GB), which further challenges the storage, manipulation, and visualization of such data even on high-end computer systems. One approach to processing and handling display of these huge datasets is through clustered computing and rendering. In this project, we proposed and implemented a multi-panel visualization mechanism that allows dynamic visualization of large, high resolution GIS datasets using ESRI^(R) ArcGIS^(R) Engine. The multi-panel display system is composed of a seven-personal computer (PC) (or seven-node) cluster. Six PCs (referred to as, render nodes or client nodes) display output forming the tiled display and the seventh PC (master node) serves as the master control. The client nodes are controlled by the master node to achieve synchronous visualization of GIS data rendered in the ESRI^(R) ArcScene^(R) environment, which is installed on each of the client nodes. The system is capable of all of the navigation functions offered in the ArcScene^(R) environment and it can be easily expanded to include more nodes and display tiles. The detailed implementation strategy and the latest visualization techniques are discussed in this paper.

Today, the data storage industry faces a rapidly growing volume of data. Adding more primary disk capacity to manage data growth is a costly and non-sustainable strategy. Before investing in new capacity, data managers should rationalize their existing storage infrastructure to maximize the use of existing capacity. There is a growing need for achieving a high ratio of Total Performance of Ownership (TPO) to Total Cost of Ownership (TCO), or TPO/TCO. The storage infrastructure can be made more efficient by assessing data usages, eliminating unnecessary data copies, moving less critical data to less expensive disk devices and repurposing allocated but unused capacity. Optimizing existing storage assets can reduce storage costs by delaying or eliminating the need for new primary capacity to manage information growth. In this paper, we apply the notion of information lifecycle management (ILM) to achieve the above improved efficiencies and optimizations. By balancing data value and storage requirements, we aim to reduce the storage system’s dependence on expensive high-performance disk devices and lower its cost per online gigabyte, thus resulting in a higher TPO/TCO.

Freeway traffic surveillance systems currently collect large amounts of traffic data, sometimes a few gigabytes per day, to support various critical traffic management center functions such as incident detection, travel time and delay estimation, and congestion management. Reliable traffic information, however, requires applying quality control measures to the collected traffic data before archiving, dissemination to the public, or use in relevant applications. This paper presents a probabilistic data-driven methodology for real-time screening of freeway loop detector data. Two complementary approaches were developed to detect abrupt temporal changes in the traffic parameters, as well as possible inconsistencies among each pair of the three traffic parameters. A real-time data screening algorithm was devised to operate in three steps. An illustrative example is presented to explain how the algorithm can be applied to real-time data screening and how observations can be diagnosed for the most likely erroneous parameters.

The emerging gigabyte fast access to customer terminal (G.fast) project has been initiated within ITU Telecommunication Standardization Sector (ITU-T) SG15 in February 2011. This new copper access technology is expected to provide aggregate (upstream and downstream) data rates up to 1 Gbps over distances up to 250m. This paper discusses the concept of G.fast channel modelling based on copper cable measurements. The cable insertion loss and far end crosstalk (FEXT) transfer function are measured from a 10-pair cable with 0.5mm wires gauge. The G.fast transceiver is simulated based on the ITU-T specifications. The attainable data rate performance is simulated for distances up to 250m. The bit rate performance are expected to decrease proportional to distance and number of crosstalk in the lines. As a conclusion, in order to ensure successful deployment of G.Fast technology, it is essential for understanding the impact of FEXT noise towards data bit rate performance.

Can the cloud help the smallest, most vulnerable patients—those in the neonatal intensive care unit? Based upon the Artemis Cloud platform, a commercial service, researchers at McMaster Children’s Hospital built an analytical model to determine what would be needed to capture, analyze in real time, and store patient data. The Artemis cloud platform can collect up to 13 physiological measures (blood pressure, electrocardiogram, heart rate, etc.) plus infusion pump data from each patient, de-identify them, and transmit them to the cloud. There, medical algorithms—such as one that flags patients at risk for late onset neonatal sepsis and one that detects sleep apnea—are applied to streams of patient data. The data are stored for later retrospective analysis and data mining to uncover correlations and patterns of clinical relevance. Borrowing from queuing theory, the mathematical study of wait times originally used for calls routed through telephone exchanges, the researchers modeled the performance of Artemis Cloud. Patient features—gestational age, preterm/term, complex/standard—served as inputs to the model. Assumptions included: that service is rendered in order of patient arrival and that the unit operates at full capacity, 47 beds. The model predicted a mean of 47.8 patients in the entire intensive care unit, with a stay of 41.5 days per patient and 9 medical algorithms per patient per day. The researchers calculated that nearly 10 gigabytes of memory space would be needed to store both pump and patient data just for one day. A cluster of 40 central processing units could provide the required computational processing power for McMaster, even if the NICU grew from 47 to 170 beds. This study determines the infrastructure needed to support both computationally and memory-intensive activities. The idea of providing clinical support and early detection of disease through analytics is tantalizing, but only through practical implementation can we begin to reap the benefits of healthcare analytics. H. Khazaei et al ., Health informatics for neonatal intensive care units: An analytical modeling perspective. IEEE J. Transl. Eng. Health Med. 10.1109/JTEHM.2015.2485268 (2015). [[Full Text]][1] [1]: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7286745

As flash devices become ubiquitous in data centers and cost per gigabyte drops, flash systems need to provide data services similar to those of traditional storage. We present Mjolnir, a powerful and scalable engine that addresses the core problems that make efficient flash based data services challenging: multi-reference management and garbage collection. Additionally, by providing powerful primitives for address remapping, Mjolnir enables redesign of the I/O stack for greater efficiency and performance with flash. Mjolnir uses techniques from language runtimes for reference management and garbage collection; we show via prototype and experimental evaluation that this design can deliver predictable performance even with varied user workloads across a range of capacity and reference-count scales.

Hadoop has traditionally been used as a large-scale batch processing system. However, interactive applications such as Facebook Messenger are becoming increasingly prominent in the Hadoop world. A key bottleneck in adapting Hadoop to real-time processing is disk data transfer rate. The advent of Solid State Drives (SSDs) holds great promise in this regard as they provide bandwidth on the orders of magnitude better than that of rotating disks. But due to their higher cost per gigabyte, a common approach is to have heterogeneous storage types. This paper presents a Storage Resource Management technique that automatically and dynamically moves data across this tiered storage based on Data Temperature, migrating "hot" data towards faster storage and "cold" data towards inexpensive archival storage. Thus, the cluster adapts based on the characteristics of the workloads over time to make effective use of the scarce expensive storage. Finally, I evaluate my modified version of the Hadoop Distributed File System (HDFS) against the vanilla version to compare their performances. The results are promising and show an improvement in both read and write performance with a significant improvement in read performance.

Database management systems (DBMSs) carefully optimize complex multi-join queries to avoid expensive disk I/O. As servers today feature tens or hundreds of gigabytes of RAM, a significant fraction of many analytic databases becomes memory-resident. Even after careful tuning for an in-memory environment, a linear disk I/O model such as the one implemented in PostgreSQL may make query response time predictions that are up to 2× slower than the optimal multi-join query plan over memory-resident data. This paper introduces a memory I/O cost model to identify good evaluation strategies for complex query plans with multiple hash-based equi-joins over memory-resident data. The proposed cost model is carefully validated for accuracy using three different systems, including an Amazon EC2 instance, to control for hardware-specific differences. Prior work in parallel query evaluation has advocated right-deep and bushy trees for multi-join queries due to their greater parallelization and pipelining potential. A surprising finding is that the conventional wisdom from shared-nothing disk-based systems does not directly apply to the modern shared-everything memory hierarchy. As corroborated by our model, the performance gap between the optimal left-deep and right-deep query plan can grow to about 10× as the number of joins in the query increases.

The following is investigated — basics physics of hyperspectral mapping (HSM) and the tasks that are being solved with their help. The efficiency of majority prioritized projects fulfilment in tasks of territory management is usually increasing when there is an implementation of modern geo-information systems, which are founded on getting the information from space. Behind the classification of object’s on aerospace pictures there are differences in their optic-spectral characteristics — spectral signatures. The more these objects differ from each other, and also from background surface (background), the easier it is possible to find them and the more precise is the result of classification. In order to differentiate the objects of different classes one should have details information about their spectral signatures, what is actually provided by HSM. Hyper-spectral data might be used both together with information about topology of inspected(observed) objects and separately. Essential growth of possibility to get useful information about target objects when using HSM is based on spectral differences of target objects and backgrounds. But the realization (implementation) of this approach is a serious problem predetermined by requirement to process huge scope of iconic (image) information (tens and hundreds of gigabytes) practically in real time scale. Modern hyper-spectrometers allow to get highly detailed spatial and spectral information about type and condition of natural and anthropogenic objects of Earth surface, and also about different dynamic processes that are happening on it, e.g. fires, droughts, etc.

Large-scale video on demand (VOD) services use hard disk drives (HDDs) to store large video files. However, VOD performance is limited by the relatively low bandwidth access and high HDD latency access, which increase dramatically when a VOD server receives a large number of simultaneous requests. By contrast, flash-based solid-state drives (SSDs) are an attractive alternative because these ensure fast access to an intensive reading system such as VOD. A flash-based SSD only stores popular videos to provide fast access for read-intensive workloads. Unfortunately, replacing the entire HDD with SSD is costly because of its high cost per gigabyte. Therefore, hybrid HDD storage and the small capacity of a flash-based SSD can provide high but cost-effective performance in relation to the storage subsystem of a VOD server. The main goal is to improve the performance of the VOD server by maximizing the number of concurrent user requests and minimizing startup latency in a cost-effective way. Experimental results show that the hybrid HDD storage with flash-based SSD provides better performance than the HDD-based system because it can reduce the average response time for a large-scale VOD storage server by approximately 52.44%.

Cloud computing is a relatively new form of computing, which uses virtualized resources and is dynamically scalable and is often provided as pay for use service over the Internet or Intranet or both. With increasing demand for data storage in the cloud, study of data intensive applications is becoming a primary focus. Data intensive applications are those which involve a high CPU usage, processsing large volumes of data typically in size of hundreds of gigabytes, terabytes, or petabytes. This study was conducted on Amazon's Elastic Cloud Compute (EC2) and Amazon Elastic Map Reduce (EMR) using HiBench Hadoop Benchmark Suite. HiBench is a Hadoop benchmark suite and is used for performing and evaluating Hadoop based data intensive computation on both these cloud paltforms. Both quantitative and qualitative comparison was performed on both Amazon EC2 and Amazon EMR, including a study of their pricing models and measures are suggested for future studies and research.

Experiments in high-energy physics (HEP) and related fields often impose constraints and challenges on data acquisition systems. As a result, these systems are implemented as unique mixtures of custom and commercial-off-the-shelf electronics (COTS), involving and connecting radiation-hard devices, large high-performance networks, and computing farms. FELIX, the Frontend Link Exchange, is a new PC-based general purpose data routing device for the data-acquisition system of the ATLAS experiment at CERN. Performance is a very crucial point for devices like FELIX, which have to be capable of processing tens of gigabyte of data per second. Thus it is important to understand the performance limitations for typical workloads on modern hardware. In this paper the analysis of FELIX packet processing algorithm is presented. The role played by the PC system's memory architecture in the overall data throughput is discussed and motivated, both by measurements and theoretical means. Finally, optimizations increasing the processing throughput by a factor larger than 10x are analyzed.

In recent years, Structural Engineering practice has started adopting performance-based building design, a concept that produces a more resilient structural design as it accounts for the probabilistic nature of natural hazards such as earthquakes. Often times, a large number of dynamic building simulations have to be explored, producing results datasets that might typically be tens or hundreds of gigabytes. It is imperative to manage these datasets efficiently, to enable engineers run a larger number of simulations which helps produce more robust designs. Hierarchal Data Format (HDF) is a scientific data management library - to allow storing large amounts of data in hierarchal form on different platforms. SQLite is a relational database library based on the SQL language that can manage data efficiently on the local disk, without a server. In this study, the two libraries (HDF and SQLite) are explored for efficient dataset management, by comparing their efficiency using different performance metrics for storing and retrieving data in different scenarios. Two reading performance metrics were then considered, one based on data being read directly from disk (cold cache), and another for data being cached in memory (warm cache). Based on different building design simulations, performances of both libraries are compared for writing and reading with both warm and cold cache. According to the comparative study, it is concluded that the HDF library is faster in storing and retrieving big chunks of data and also required less storage space. Although the study was performed on structural simulation datasets, the conclusions are likely applicable to different kinds of datasets often encountered in many different engineering disciplines.

To the editor: Current state-of-the-art light sheet microscopes rely on sophisticated control software to perform the acquisition of gigabytes of image data per second over the course of hours or even days. Typically the microscopes acquire data in a first step, and only in a second step this data is processed and visualized offline. The delay between data acquisition and data assessment wastes time and storage space. Technology that makes it possible to view and assess the data during imaging would offer significant advantages. However, even the most advanced microscopes only display the latest image plane acquired or projection while the raw volumetric data is saved to disk [1–4].

Big data” is a universal buzzword in business and science, referring to the retrieval and handling of ever-growing amounts of information. It can be assumed, for example, that a typical hospital generates hundreds of terabytes (1 TB = 1012 bytes) of data annually in the course of patient care (1). For instance, exome sequencing, which results in 5 gigabytes (1 GB = 109 bytes) of data per patient, is on the way to becoming routine (2). The analysis of such enormous volumes of information, i.e., organization and description of the data and the drawing of (scientifically valid) conclusions, can already hardly be accomplished with the traditional tools of computer science and statistics. For example, examination of the exomes of several hundred patients requires sophisticated analytical approaches and the selection of statistical methods that optimize computation time to avoid exceeding the available storage capacity. This is a challenge for the discipline of statistics, which has traditionally analyzed data not only from clinical studies but also from observational studies. Inter alia, techniques have to cope with a number of characteristics per individual that greatly exceeds the number of individuals observed, e.g., in the acquisition of 5 million single-nucleotide polymorphisms from each of a cohort of 100 patients. In the following description of scenarios, techniques, and problems we focus on medical science, i.e., on the question of where and how big data approaches to the processing of large volumes of information can contribute to the advancement of scientific knowledge in medicine. While the description of the corresponding data analysis techniques takes a predominantly scientific perspective, the three scenarios preceding the discussion of techniques are intended to guide the reader in how these approaches can be used in handling routine data. Because clinical studies are our reference point, applications that have little in common with the structure of such studies, e.g., the prediction of disease spread from search engine data (Box), will not be discussed. Furthermore, concepts for technical implementation, e.g., cloud computing (5), will not be presented. Instead, we focus on biostatistical aspects, such as the undistorted estimation of treatment effects, which represent a crucial precondition for progress in medical science (6). Box The debate about a big data showpiece: Google Flu Trends In the Google Flu Trends project (3), the frequency of Google searches for certain terms is used to predict the influenza activity at regional level in a large number of countries. The original publication (3) shows that this method enables precise prediction of data that have traditionally been acquired in much more cumbersome fashion, e.g., by the United States Centers for Disease Control and Prevention (CDC), and did not use to be available until some time later. The possibility of rapid reaction opened up by the Google approach is often cited as a successful application of big data. However, later investigations (4) showed serious systematic deviations from predicted values in the period covered by (3). These may have been caused by modification of the search engine algorithm for business reasons, i.e., to optimize the primary function, with resulting impairment of the secondary function of influenza prediction.

The foraminifera.eu project with its 120 avocational and professional scientific contributors has built a popular online image database containing information about fossil and recent foraminifera. Foraminiferal data available from over 200 years of scientific studies and publications were first analyzed in order to create a robust database using structurable and available data-attributes. A freely accessible database with online interfaces was established to improve the accessibility of foraminiferal data. So far, the database contains 9,800+ records of specimens with images and accompanying metadata. Thirty data attributes were chosen, and discrete values were defined and applied to each dataset covering the areas of identification, geographical and stratigraphical positioning, morphology, synonyms, references, and collection related data. The attributes were chosen based on their usefulness to both avocational and professional scientists and on their general availability. A discussion and review process has been established to enlarge the database by adding more records and to implement new data attributes and interfaces. In 2015, the database was accessed on average by 220 users daily (excluding bots) and 0,8 gigabytes of data were downloaded each day. The numbers indicate the utility and relevance of the foraminifera.eu database, and this is also acknowledged by senior scientists and major institutions through their contributions.

Virtual microscopy is increasingly used for e-learning and medical online exams at universities. Traditional client-server systems support upi¾?to a few hundred of users accessing more than 10.000 large microscopic images each several Gigabyte and each being able to make interactive annotations. We have developed the first peer-to-peer based solution bringing virtual microscopy to an Internet-scale community. We address data distribution and replication by a novel overlay called Omentum, which is based on a random-graph architecture. Omentum uses a lightweight messaging service for peer communication and supports traffic-free routing-path calculation. Based on the directed random graph the system achieves path compression by walking along inbound links during the actual routing phase. The evaluation shows the efficiency and scalability of the Omentum overlay network, its replication strategy and an administrative communication overhead for creating new replicas around $$0.06\,\%$$0.06%.

One of the crucial technologies in Sony's new PlayStation 3 gaming console is the data transfer connections, or buses, that link the Cell processor to both the console's main memory and the graphics processor. The immersive experience that Sony is aiming for depends largely on the data flowing to and from the Cell processor at speeds way beyond anything achieved in a home-electronics system. The bus between the Cell and the PS3's memory will achieve a peak data-transfer rate, or bandwidth, or 25.6 gigabytes per second. Meanwhile, the bus connecting the Cell to the graphics chip will move data at 35 GB/s, or about five to 10 times what you can get with today's best PC-bus technology. To maintain the integrity of the electrical signals and keep them synchronized because of the higher frequencies required for faster data rates, Sony relies on a signaling technique that minimizes interference and attenuation and a timing mechanism that compensates for time-control discrepancies

An improved global positioning system (GPS)-based animal tracking system is needed to meet quickly evolving demands of ecological research, range livestock production, and natural resource management. Commercially available tracking systems lack the data storage capacity needed to frequently collect animal location data (e.g., 15-minute intervals or less) over long-term deployment periods (e.g., 1 year or more). Some commercial systems have remote data-download capabilities, reducing the need to recapture tagged animals for data retrieval, but these systems download data via satellite (Argos), global system for mobile communications (GSM) cellular telephone, or telemetry radio frequencies. Satellite systems are excessively expensive, and GSM cellular coverage is extremely limited within the United States. Radio-based systems use narrow-band very-high- or ultra-high frequencies requiring the user to obtain frequency allocations. None of these existing systems were designed to provide continual, real-time data access. The Clark GPS Animal Tracking System (Clark ATS) was developed to meet the evolving demands of animal ethologists, ecologists, natural resource managers, and livestock producers. The Clark ATS uses memory-card technology for expandable data storage from 16 megabytes to 8 gigabytes. Remote data downloading and program uploading is accomplished using spread-spectrum radio transceivers, which do not require narrow-band radio frequency allocations. These radios also transmit, at a user-defined time interval, a real-time, GPS-location beacon to any Clark ATS base station within range (about 24 km or 15 miles line of sight). Advances incorporated into the Clark ATS make it possible to evaluate animal behavior at very fine spatial- and temporal-resolution over long periods of time. The real-time monitoring provided by this system enables researchers to accurately examine animal distribution and activity responses to acute, short-term disturbances relative to longer- term behavioral patterns. The Clark ATS also provides a huge time- and cost-savings to researchers and natural resource managers attempting to relocate a tagged animal in the field for direct observation or other operations.

The Parallel ROOT Facility, PROOF, enables the analysis of much larger data sets on a shorter time scale. It exploits the inherent parallelism in data of uncorrelated events via a multi-tier architecture that optimizes I/O and CPU utilization in heterogeneous clusters with distributed storage. The system provides transparent and interactive access to gigabytes today. Being part of the ROOT framework PROOF inherits the benefits of a performant object storage system and a wealth of statistical and visualization tools. This paper describes the data analysis model of ROOT and the latest developments on closer integration of PROOF into that model and the ROOT user environment, e.g. support for PROOF-based browsing of trees stored remotely, and the popular TTree::Draw() interface. We also outline the ongoing developments aimed to improve the flexibility and user-friendliness of the system.

In December 1997, Sun Microsystems had just announced its new flagship machine: a 64-processor symmetric multiprocessor supporting up to 64 gigabytes of memory and thousands of I/O devices. As with any new machine launch, Sun was working feverishly on benchmarks to prove the machine’s performance. While the benchmarks were generally impressive, there was one in particular that was exhibiting unexpectedly low performance. The benchmark machine would occasionally become mysteriously distracted: Benchmark activity would practically cease, but the operating system kernel remained furiously busy. After some number of minutes spent on unknown work, the operating system would suddenly right itself: Benchmark activity would resume at full throttle and run to completion. Those running the benchmark could see that the machine was on course to break the world record, but these minutes-long periods of unknown kernel activity were enough to be the difference between first and worst.

We describe the design of a PC-to-FPGA data streaming platform that enables hardware acceleration of gigabyte scale input data. Specifically, the acceleration is an FPGA implementation of the Dialign Algorithm, which performs both global and local alignment of query biological sequences against relatively larger reference strands of biological sequences. Earlier implementations of this algorithm could not be scaled to handle gigabyte-length reference sequences, nor megabyte-length query sequences, due to the inherent limitations of available memory and logic on single-FPGA platforms. We solve these issues via the design of an Ethernet channel to stream the reference sequence, and describe the novel use of SATA based Solid State Drives (SSDs) to time multiplex the FPGA logic into handling larger query sequences as well. In doing so, this paper also presents a general method to achieve gigabyte-depth FIFOs on commercially available FPGA development boards. This benefits data-intensive acceleration even outside of the bioinformatics application domain. Through the development of our acceleration logic and careful coupling of the required IO peripherals, we have successfully demonstrated a processing time of 28.61 minutes for a 200 base-pair query-sequence aligned against a 1 GB reference-sequence, a rate that is limited only by SATA 2 SDD write speeds. The present runtime offers a 38x speedup (18.36 hours down to 28.61 minutes) compared to standalone PC based processing.

For as long as I can remember---and probably since the invention of the computer---there have been two rules about storage space for computer data: You never have enough of it, and if you have more, it's not connected to the right computer. Somehow this seems to be true even with the incredible increases in the capacity of disk drives, with retail prices in the neighborhood of 50 cents per gigabyte.

As librarians, we need to understand better how people are acquiring and using information—“start swimming,” or we may “sink like a stone.” Users from the Alexandrian Library in 331 BC would easily recognize that they were in today’s modern library. The physical attributes of the collection would be different, but they would know they were in a library. What happens to the library as place when the collection becomes increasingly electronic? New libraries continue to be built—to house the libraries’ physical collections, provide access to electronic materials, and serve as a physical place where people come to read and to study. Fifteen years ago new libraries were being built to accommodate public access terminals. It was rare to see a user with a portable computer. Now that libraries and campuses are wired, the user is becoming unwired. What is the future of the library as a physical space? In 2003, Grant Gross reported that five billion gigabytes of data were generated in 2002, equaling 800 megabytes per person or enough data to fill 500,000 Library of Congresses. Of these data, 92 percent were stored magnetically on hard drives. Between Now that libraries and campuses are wired, the user is becoming unwired. What is the future of the library as a physical space?

With world wide data predicted to exceed 40 trillion gigabytes by 2020, big data storage is a very real and escalating problem. Herein, we discuss the utility of synthetic DNA as a robust and eco-friendly archival data storage solution of the future.

The Ensembl Web site (http://www.ensembl.org/) is the principal user interface to the data of the Ensembl project, and currently serves >500,000 pages (∼2.5 million hits) per week, providing access to >80 GB (gigabyte) of data to users in more than 80 countries. Built atop an open-source platform comprising Apache/mod_perl and the MySQL relational database management system, it is modular, extensible, and freely available. It is being actively reused and extended in several different projects, and has been downloaded and installed in companies and academic institutions worldwide. Here, we describe some of the technical features of the site, with particular reference to its dynamic configuration that enables it to handle disparate data from multiple species.

This paper investigates a novel streaming architecture consisting of home-to-home online (H2O) devices that collaborate with one another to provide on-demand access to large repositories of continuous media such as audio and video clips. An H2O device is configured with a high bandwidth wireless communication component, a powerful processor, and gigabytes of storage. A key challenge of this environment is how to place data across H2O devices in order to enhance startup latency, defined as the delay observed from when a user requests a clip, to the onset of its display. Our primary contribution is a novel replication technique that enhances startup latency, while minimizing the total storage space required from an environment consisting of N H2O devices. This technique is based on the following intuition: The first few blocks of a clip are required more urgently than its last few blocks, and should be replicated more frequently in order to minimize startup latency. We develop analytical models to quantify the number of replicas required for each block. In addition, we describe two alternative distributed implementation of our replication strategy. When compared with full replication, our technique provides on average greater than 97% (i.e., several orders of magnitude) savings in storage space, while ensuring zero startup latency and a hiccup-free reception.

This article describes practical solutions to problems encountered over more than five years of operating a fully automated Web‐based digital document submission, registration, and distribution system. The author began creating the system in 1997 to support the activities of a Technical Committee of the International Committee for Information Technology Standardization (INCITS). Since then, the system has received continual enhancement and expansion. The register now contains more than 8,000 documents submitted by Committee participants, stored as PDF files occupying more than 4 gigabytes of storage, which interested parties world‐wide regularly access. A similar architecture and document registration system could be used to support any physically dispersed group cooperating on any type of project. The library is located on a public Web site and readers are encouraged to access it.

Why is my hard disk full? A question no doubt familiar to many readers, and one that has inspired our research for several years. Our goal is to provide more insight into large, hierarchical data sets, commonly known as trees. Hierarchical data sets can be found everywhere. A large number of items, such as files, products, employees, and stocks, can be handled and managed much more efficiently when they are grouped into larger entities. Recursive application of this approach results in a tree structure. A hierarchical file system is a prime example: The user can organize his disk, and only has to deal with a limited set of files while fulfilling a task. But the PC disk of an average user often contains dozens of gigabytes of data, distributed over hundreds of thousands of files. In this case it becomes difficult to maintain an overview, and to determine what is cluttering the disk. Often no single, simple answer exists. Perhaps another user of the PC has installed some large programs, or has failed to cleanup after finishing a task in his or her relief at meeting a deadline. Perhaps multiple copies of the same multimedia file are stored on the disk. How can we find large files and directories and identify patterns and structures easily in such large hierarchical data structures? Automatic methods, such as searching for the largest files, fall short, and standard file browsers, using indented lists, have not been developed with this problem in mind. We believe the best way to answer our question is to exploit the unique capabilities of the human visual system, tuned and optimized in the course of millions of years of evolution to extract information from images [9]. In other words, let us try to make synthetic images, using a wide variety of visual cues to transfer information as efficiently and effectively as possible.

For the last 50 years, tape has persisted as the media of choice when inexpensive data storage is required and speed is not critical. The cost of tape storage normalized per unit capacity (dollars per gigabyte) decreased steadily over this time, driven primarily by advances in areal density and reduction of tape thickness. This paper reports the next advance in tape storage--a demonstration of a tenfold increase in capacity over current-generation Linear Tape-Open® (LTO®) systems. One terabyte (1 TB, or 1000 GB) of uncompressed data was written on half-inch tape using the LTO form factor. This technical breakthrough involves significant advances in nearly every aspect of the recording process: heads, media, channel electronics, and recording platform.