Search

We study how best to schedule scans of large data files, in the presence of many simultaneous requests to a common set of files. The objective is to maximize the overall rate of processing these files, by sharing scans of the same file as aggressively as possible, without imposing undue wait time on individual jobs. This scheduling problem arises in batch data processing environments such as Map-Reduce systems, some of which handle tens of thousands of processing requests daily, over a shared set of files.As we demonstrate, conventional scheduling techniques such as shortest-job-first do not perform well in the presence of cross-job sharing opportunities. We derive a new family of scheduling policies specifically targeted to sharable workloads. Our scheduling policies revolve around the notion that, all else being equal, it is good to schedule nonsharable scans ahead of ones that can share IO work with future jobs, if the arrival rate of sharable future jobs is expected to be high. We evaluate our policies via simulation over varied synthetic and real workloads, and demonstrate significant performance gains compared with conventional scheduling approaches.

The article discusses ensuring the privacy of data about individuals collected and sometimes made public by businesses, government agencies, and researchers. The authors observe that perturbation methods and data anonymization have not always safeguarded sensitive information. Consideration is given to different privacy definitions including epsilon-differential privacy and k-anonymity, post-processing, composition, convexity, and minimizing probabilistic failure. They conclude with a discussion of a class of privacy definitions called Blowfish.

Temporal models based on recurrent neural networks have proven to be quite powerful in a wide variety of applications, including language modeling and speech processing. However, training these models often relies on backpropagation through time (BPTT), which entails unfolding the network over many time steps, making the process of conducting credit assignment considerably more challenging. Furthermore, the nature of backpropagation itself does not permit the use of nondifferentiable activation functions and is inherently sequential, making parallelization of the underlying training process difficult. Here, we propose the parallel temporal neural coding network (P-TNCN), a biologically inspired model trained by the learning algorithm we call local representation alignment. It aims to resolve the difficulties and problems that plague recurrent networks trained by BPTT. The architecture requires neither unrolling in time nor the derivatives of its internal activation functions. We compare our model and learning procedure with other BPTT alternatives (which also tend to be computationally expensive), including real-time recurrent learning, echo state networks, and unbiased online recurrent optimization. We show that it outperforms these on-sequence modeling benchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing NotMNIST, and Penn Treebank. Notably, our approach can, in some instances, outperform full BPTT as well as variants such as sparse attentive backtracking. Significantly, the hidden unit correction phase of P-TNCN allows it to adapt to new data sets even if its synaptic weights are held fixed (zero-shot adaptation) and facilitates retention of prior generative knowledge when faced with a task sequence. We present results that show the P-TNCN’s ability to conduct zero-shot adaptation and online continual sequence modeling.

Noisy Max and Sparse Vector are selection algorithms for differential privacy and serve as building blocks for more complex algorithms. In this paper we show that both algorithms can release additional information for free (i.e., at no additional privacy cost). Noisy Max is used to return the approximate maximizer among a set of queries. We show that it can also release for free the noisy gap between the approximate maximizer and runner-up. This free information can improve the accuracy of certain subsequent counting queries by up to 50%. Sparse Vector is used to return a set of queries that are approximately larger than a fixed threshold. We show that it can adaptively control its privacy budget (use less budget for queries that are likely to be much larger than the threshold) in order to increase the amount of queries it can process. These results follow from a careful privacy analysis.

We consider the problem of privately releasing a class of queries that we call hierarchical count-of-counts histograms. Count-of-counts histograms partition the rows of an input table into groups (e.g., group of people in the same household), and for every integer jreport the number of groups of size j. Hierarchical count-of-counts queries report count-of-counts histograms at different granularities as per hierarchy defined on an attribute in the input data (e.g., geographical location of a household at the national, state and county levels). In this paper, we introduce this problem, along with appropriate error metrics and propose a differentially private solution that generates count-of-counts histograms that are consistent across all levels of the hierarchy.

When fitting statistical models to variables in geoscientific disciplines such as hydrology, it is a customary practice to stratify a large domain into multiple regions (or regimes) and study each region separately. Traditional wisdom suggests that models built for each region separately will have higher performance because of homogeneity within each region. However, each stratified model has access to fewer and less diverse data points. Here, through two hydrologic examples (soil moisture and streamflow), we show that conventional wisdom may no longer hold in the era of big data and deep learning (DL). We systematically examined an effect we call data synergy, where the results of the DL models improved when data were pooled together from characteristically different regions. The performance of the DL models benefitedfrom modest diversity in the training data compared to a homogeneous training set, even with similar data quantity. Moreover, allowing heterogeneous training data makes eligible much larger training datasets, which is an inherent advantage of DL. A large, diverse data set is advantageous in terms of representing extreme events and future scenarios, which has strong implications for climate change impact assessment. The results here suggest the research community should place greater emphasis on data sharing. Traditionally with statistical methods used in hydrology, we split the domain into relatively homogeneous regimes, for each of which we can create a simple model, that is, a local model. However, in the era of big data machine learning, we show that this is often the opposite of what should be done. With deep learning models, we should compile a large and heterogeneous data set and compare the local model to a model trained with all the data (global model). Including heterogeneous training samples may improve the results compared to the local model. We call this the data synergy effect, and it results from two main factors. First, deep learning models are complex enough to accommodate different training instances, inherently permitting larger training datasets with more extreme events and changing trends. Second, with a heterogeneous training data set, deep learning models may be able to learn both the underlying similarities and factors contributing to differences between regions. We introduced data synergy, where deep learning performance in a local region improves when including samples from other regionsData synergy is apparent with modestly diverse training data, partly because a larger and more diverse data set contains more extreme eventsThis work highlighted the value of samples outside a region of interest, emphasizing the need for community data sharing We introduced data synergy, where deep learning performance in a local region improves when including samples from other regions Data synergy is apparent with modestly diverse training data, partly because a larger and more diverse data set contains more extreme events This work highlighted the value of samples outside a region of interest, emphasizing the need for community data sharing

Small changes in seismic wave properties foretell frictional failure in laboratory experiments and in some cases on seismic faults. Such precursors include systematic changes in wave velocity and amplitude throughout the seismic cycle. However, the relationships between wave features and shear stress are complex. Here, we use data from lab friction experiments that include continuous measurement of elastic waves traversing the fault and build data‐driven models to learn these complex relations. We demonstrate that deep learning models accurately predict the timing and size of laboratory earthquakes based on wave features. Additionally, the transportability of models is explored by using data from different experiments. Our deep learning models transfer well to unseen datasets providing high‐fidelity models with much less training. These prediction methods can be potentially applied in the field for earthquake early warning in conjunction with long‐term time‐lapse seismic monitoring of crustal faults, CO2storage sites and unconventional energy reservoirs. Laboratory experiments and field observations show that wave velocity, amplitude and frequency vary systematically over time during seismic cycles. These wave characteristics drop before failure (shear stress drop) albeit at different times and thus are believed to contain precursory information about the upcoming failure event. Here, we continuously record ultrasonic data during a series of experiments designed to simulate earthquakes in the laboratory or laboratory quakes. We investigate whether machine learning can predict the occurrence of laboratory quakes from ultrasonic data. We apply XGBoost and a suite of deep learning methods to this data and present models that can accurately predict the laboratory quake timing, size or both. We compare the performance of different models in terms of accuracy and training time. Also, we interpret the developed models. Finally, we show that these models successfully transfer from one data set to another obtained using different experimental constraints. Consequently, the training time and amount of data necessary to develop models for new datasets are significantly reduced. The developed prediction models can be used for seismic hazard assessment and warning, safe management of CO2storage sites and unconventional energy reservoirs in conjunction with continuous and long‐term seismic monitoring. We predict size and timing of lab quakes from active source ultrasonic data and deep learning models yield the most accurate predictionsPredictions are accurate despite irregular seismic cyclesDeep learning models transfer better to other datasets; transfer learning reduces the training time and size of training data We predict size and timing of lab quakes from active source ultrasonic data and deep learning models yield the most accurate predictions Predictions are accurate despite irregular seismic cycles Deep learning models transfer better to other datasets; transfer learning reduces the training time and size of training data

Recently, recurrent deep networks have shown promise to harness newly available satellite‐sensed data for long‐term soil moisture projections. However, to be useful in forecasting, deep networks must also provide uncertainty estimates. Here we evaluated Monte Carlo dropout with an input‐dependent data noise term (MCD+N), an efficient uncertainty estimation framework originally developed in computer vision, for hydrologic time series predictions. MCD+N simultaneously estimates a heteroscedastic input‐dependent data noise term (a trained error model attributable to observational noise) and a network weight uncertainty term (attributable to insufficiently constrained model parameters). Although MCD+N has appealing features, many heuristic approximations were employed during its derivation, and rigorous evaluations and evidence of its asserted capability to detect dissimilarity were lacking. To address this, we provided an in‐depth evaluation of the scheme's potential and limitations. We showed that for reproducing soil moisture dynamics recorded by the Soil Moisture Active Passive (SMAP) mission, MCD+N indeed gave a good estimate of predictive error, provided that we tuned a hyperparameter and used a representative training data set. The input‐dependent term responded strongly to observational noise, while the model term clearly acted as a detector for physiographic dissimilarity from the training data, behaving as intended. However, when the training and test data were characteristically different, the input‐dependent term could be misled, undermining its reliability. Additionally, due to the data‐driven nature of the model, data noise also influences network weight uncertainty, and therefore the two uncertainty terms are correlated. Overall, this approach has promise, but care is needed to interpret the results. With proper hyperparameters and training data, Monte Carlo Dropout with a data noise term can effectively estimate prediction errorThe network‐predicted data noise term clearly responds to added noise while the network weight uncertainty term reacts to dissimilarityThe quality of both the data noise term and the network weight uncertainty term can be lowered by biased training data

The Soil Moisture Active Passive (SMAP) mission has delivered valuable sensing of surface soil moisture since 2015. However, it has a short time span and irregular revisit schedules. Utilizing a state‐of‐the‐art time series deep learning neural network, Long Short‐Term Memory (LSTM), we created a system that predicts SMAP level‐3 moisture product with atmospheric forcings, model‐simulated moisture, and static physiographic attributes as inputs. The system removes most of the bias with model simulations and improves predicted moisture climatology, achieving small test root‐mean‐square errors (<0.035) and high‐correlation coefficients >0.87 for over 75% of Continental United States, including the forested southeast. As the first application of LSTM in hydrology, we show the proposed network avoids overfitting and is robust for both temporal and spatial extrapolation tests. LSTM generalizes well across regions with distinct climates and environmental settings. With high fidelity to SMAP, LSTM shows great potential for hindcasting, data assimilation, and weather forecasting. Soil moisture is the water content in soil, and it is a critical component of the water cycle. It controls whether crops or wild vegetation can function properly, the risk of wildfire, and the likelihood of floods. The NASA satellite SMAP, launched in 2015, measures soil moisture near the ground surface over Earth at high accuracy. SMAP data are of great value to global communities to which soil moisture is relevant. However, since it was launched only recently, there is only little overlap with other data sets, which limits its use. To extend SMAP's observations in time, we employ a “deep learning” technology called Long Short‐Term Memory (LSTM), which is one of the pillars of artificial intelligence and is bringing revolutionary changes in many scientific fields and our daily lives. We use LSTM to learn patterns of soil moisture dynamics and where physics‐based models make mistakes in describing moisture changes. This approach shows great promise for projecting SMAP observations into the long past: because soil moisture has a short memory, 2 years of data seem sufficient to train LSTM to successfully capture its dynamics. Because LSTM can handle large and diverse data, it offers valuable alternatives to older statistical methods. With 2 years of data, SMAP L3 data can be extended at high fidelity using a deep learning network (LSTM), showing potential for hindcastingDespite significant, spatially varying bias in Land Surface Models, LSTM can remove bias, correct moisture climatology, and capture extremesLSTM is more generalizable than simpler methods, and its strength seems to derive from its memory and ability to accommodate large data