Search

While we are witnessing rapid growth in data across the sciences and in many applications, this growth is particularly remarkable in the medical domain, be it because of higher resolution instruments and diagnostic tools (e.g. MRI), new sources of structured data like activity trackers, the wide-spread use of electronic health records and many others. The sheer volume of the data is not, however, the only challenge to be faced when using medical data for research. Other crucial challenges include data heterogeneity, data quality, data privacy and so on. In this article, we review solutions addressing these challenges by discussing the current state of the art in the areas of data integration, data cleaning, data privacy, scalable data access and processing in the context of medical data. The techniques and tools we present will give practitioners — computer scientists and medical researchers alike — a starting point to understand the challenges and solutions and ultimately to analyse medical data and gain better and quicker insights.

Key to DNA storage is encoding the information to a sequence of nucleotides before it can be synthesised for storage. Definition of such an encoding or mapping must adhere to multiple design restrictions. First, not all possible sequences of nucleotides can be synthesised. Homopolymers, e.g., sequences of the same nucleotide, of a length of more than two, for example, cannot be synthesised without potential errors. Similarly, the G-C content of the resulting sequences should be higher than 50\%. Second, given that synthesis is expensive, the encoding must map as many bits as possible to one nucleotide. Third, the synthesis (as well as the sequencing) is error prone, leading to substitutions, deletions and insertions. An encoding must therefore be designed to be resilient to errors through error correction codes or replication. Fourth, for the purpose of computation and selective retrieval, encodings should result in substantially different sequences across all data, even for very similar data. In the following we discuss the history and evolution of encodings.

As the size of data and its heterogeneity increase, traditional database system architecture becomes an obstacle to data analysis. Integrating and ingesting (loading) data into databases is quickly becoming a bottleneck in face of massive data as well as increasingly heterogeneous data formats. Still, state-of-the-art approaches typically rely on copying and transforming data into one (or few) repositories. Queries, on the other hand, are often ad-hoc and supported by pre-cooked operators which are not adaptive enough to optimize access to data. As data formats and queries increasingly vary, there is a need to depart from the current status quo of static query processing primitives and build dynamic, fully adaptive architectures. We build ViDa, a system which reads data in its raw format and processes queries using adaptive, just-in-time operators. Our key insight is use of virtualization, i.e., abstracting data and manipulating it regardless of its original format, and dynamic generation of operators. ViDa's query engine is generated just-in-time; its caches and its query operators adapt to the current query and the workload, while also treating raw datasets as its native storage structures. Finally, ViDa features a language expressive enough to support heterogeneous data models, and to which existing languages can be translated. Users therefore have the power to choose the language best suited for an analysis.