TAZeR: Hiding the Cost of Remote I/O in Distributed Scientific Workflows

Many scientific workflows access data derived from specialized instruments. When the data is analyzed, it is accessed over wide area networks, creating bottlenecks from long access latencies. We ask the question: assuming that data must be accessed remotely, can latencies be hidden without application change? We present TAZeR, a remote I/O framework that reduces effective data access latency. TAZeR transparently converts POSIX I/O into operations that interleave application work with data transfer, i.e., read prefetching and write stage-out. TAZeR ensures read data moves directly to application memory without synchronous intervention (soft zero-copy). TAZeR uses distributed bandwidth-aware staging to exploit data reuse across application tasks and to manage the capacity constraints of fast hierarchical storage. We evaluate TAZeR on a High Energy Physics workflow where two 1 Gb/s WAN links request remote data at 48 Gb/s using non-streaming access patterns. TAZeR is $12 \times $ and $22 \times $ faster than XRootD (state-of-the-art) and file copies (current approach), respectively; and within 7% of optimal. We explore conditions under which TAZeR can hide I/O accesses by showing performance as effective staging sizes change.