Search

The use of crowdsourced data has been finding practical use for enhancing situational awareness during disasters. While recent studies have shown promising results regarding the potential of crowdsourced data for flood mapping, little attention has been paid to data imbalances issues that could introduce biases. We examine biases present in crowdsourced reports to identify data imbalances with a goal of improving disaster situational awareness. Sample bias, spatial bias, and demographic bias are examined as we analyzed reported flooding from 3-1-1, Waze reports, and FEMA damage data collected in the aftermaths of Tropical Storm Imelda in 2019 and Hurricane Ida in 2021. Integrating other flooding related topics from 3-1-1 reports into the Global Moran's I and Local Indicator of Spatial Association (LISA) test revealed more communities that were impacted by floods. To examine spatial bias, we perform the LISA and BI-LISA tests on the three datasets at the census tract and census block group level. By looking at two geographical aggregations, we found that the larger spatial aggregations, census tracts, show less data imbalance in the results. Finally, one-way analysis of Variance (ANOVA) test performed on the clusters generated from the BI-LISA shows that data imbalance exists in areas where minority populations reside. Through a regression analysis, we found that 3-1-1 and Waze reports have data imbalance limitations in areas where minority populations reside. The findings of this study advance understanding of data imbalances and biases in crowdsourced datasets that are growingly used for disaster situational awareness., Comment: 28 Pages, 12 Figures 9 Tables

The objective of this study is to propose a system-level framework with quantitative measures to assess the resilience of road networks. The framework proposed in this paper can help transportation agencies incorporate resilience considerations into project development proactively and to understand the resilience performance of current road networks effectively. This study identified and implemented four quantitative metrics to classify the criticality of road segments based on critical dimensions of road network resilience, and two integrated metrics were proposed to combine all metrics to show the overall resilience performance of road segments. A case study was conducted on the Texas road networks to demonstrate the effectiveness of implementing this framework in a practical scenario. Since the data used in this study is available to other states and countries, the framework presented in this study can be adopted by other transportation agencies across the globe for regional transportation resilience assessments.

Model-based reinforcement learning (MBRL) is a sample efficient technique to obtain control policies, yet unavoidable modeling errors often lead performance deterioration. The model in MBRL is often solely fitted to reconstruct dynamics, state observations in particular, while the impact of model error on the policy is not captured by the training objective. This leads to a mismatch between the intended goal of MBRL, enabling good policy and value learning, and the target of the loss function employed in practice, future state prediction. Naive intuition would suggest that value-aware model learning would fix this problem and, indeed, several solutions to this objective mismatch problem have been proposed based on theoretical analysis. However, they tend to be inferior in practice to commonly used maximum likelihood (MLE) based approaches. In this paper we propose the Value-gradient weighted Model Learning (VaGraM), a novel method for value-aware model learning which improves the performance of MBRL in challenging settings, such as small model capacity and the presence of distracting state dimensions. We analyze both MLE and value-aware approaches and demonstrate how they fail to account for exploration and the behavior of function approximation when learning value-aware models and highlight the additional goals that must be met to stabilize optimization in the deep learning setting. We verify our analysis by showing that our loss function is able to achieve high returns on the Mujoco benchmark suite while being more robust than maximum likelihood based approaches.

Secure ultra-reliable low-latency communication (URLLC) has been recently investigated with the fundamental limits of finite block length (FBL) regime in mind. Analysis has revealed that when eavesdroppers outnumber BS antennas or enjoy a more favorable channel condition compared to the legitimate users, base station (BS) transmit power should increase exorbitantly to meet quality of service (QoS) constraints. Channel-induced impairments such as shadowing and/or blockage pose a similar challenge. These practical considerations can drastically limit secure URLLC performance in FBL regime. Deployment of an intelligent reflecting surface (IRS) can endow such systems with much-needed resiliency and robustness to satisfy stringent latency, availability, and reliability requirements. We address this problem and propose a joint design of IRS platform and secure URLLC network. We minimize the total BS transmit power by simultaneously designing the beamformers and artificial noise at the BS and phase-shifts at the IRS, while guaranteeing the required number of securely transmitted bits with the desired packet error probability, information leakage, and maximum affordable delay. The proposed optimization problem is non-convex and we apply block coordinate descent and successive convex approximation to iteratively solve a series of convex sub-problems instead. The proposed algorithm converges to a sub-optimal solution in a few iterations and attains substantial power saving and robustness compared to baseline schemes., Comment: Submitted to International Conference on Communications (ICC) 2022

Smart resilience is the beneficial result of the collision course of the fields of data science and urban resilience to flooding. The objective of this study is to propose and demonstrate a smart flood resilience framework that leverages heterogeneous community-scale big data and infrastructure sensor data to enhance predictive risk monitoring and situational awareness. The smart flood resilience framework focuses on four core capabilities that could be augmented by the use of heterogeneous community-scale big data and analytics techniques: (1) predictive flood risk mapping; (2) automated rapid impact assessment; (3) predictive infrastructure failure prediction and monitoring; and (4) smart situational awareness capabilities. We demonstrate the components of these core capabilities of the smart flood resilience framework in the context of the 2017 Hurricane Harvey in Harris County, Texas. First, we demonstrate the use of flood sensors for the prediction of floodwater overflow in channel networks and inundation of co-located road networks. Second, we discuss the use of social media and machine learning techniques for assessing the impacts of floods on communities and sensing emotion signals to examine societal impacts. Third, we illustrate the use of high-resolution traffic data in network-theoretic models for nowcasting of flood propagation on road networks and the disrupted access to critical facilities, such as hospitals. Fourth, we leverage location-based and credit card transaction data in spatial analyses to proactively evaluate the recovery of communities and the impacts of floods on businesses. These analyses show that the significance of core capabilities of the smart flood resilience framework in helping emergency managers, city planners, public officials, responders, and volunteers to better cope with the impacts of catastrophic flooding events., Comment: 24 pages, 22 figures

The objective of this study is to develop and test a novel structured deep-learning modeling framework for urban flood nowcasting by integrating physics-based and human-sensed features. We present a new computational modeling framework including an attention-based spatial-temporal graph convolution network (ASTGCN) model and different streams of data that are collected in real-time, preprocessed, and fed into the model to consider spatial and temporal information and dependencies that improve flood nowcasting. The novelty of the computational modeling framework is threefold; first, the model is capable of considering spatial and temporal dependencies in inundation propagation thanks to the spatial and temporal graph convolutional modules; second, it enables capturing the influence of heterogeneous temporal data streams that can signal flooding status, including physics-based features such as rainfall intensity and water elevation, and human-sensed data such as flood reports and fluctuations of human activity. Third, its attention mechanism enables the model to direct its focus on the most influential features that vary dynamically. We show the application of the modeling framework in the context of Harris County, Texas, as the case study and Hurricane Harvey as the flood event. Results indicate that the model provides superior performance for the nowcasting of urban flood inundation at the census tract level, with a precision of 0.808 and a recall of 0.891, which shows the model performs better compared with some other novel models. Moreover, ASTGCN model performance improves when heterogeneous dynamic features are added into the model that solely relies on physics-based features, which demonstrates the promise of using heterogenous human-sensed data for flood nowcasting

While volume-based grid tariffs have been the norm for residential consumers, capacity-based tariffs will become more relevant with the increasing electrification of society. A further development is capacity subscription, where consumers are financially penalised for exceeding their subscribed capacity, or alternatively their demand is limited to the subscribed level. The penalty or limitation can either be static (always active) or dynamic, meaning that it is only activated when there are active grid constraints. We investigate the cost impact for static and dynamic capacity subscription tariffs, for 84 consumers based on six years of historical load data. We use several approaches for finding the optimal subscription level ex ante. The results show that annual costs remain both stable and similar for most consumers, with a few exceptions for those that have high peak demand. In the case of a physical limitation, it is important to use a stochastic approach for the optimal subscription level to avoid excessive demand limitations. Facing increased peak loads due to electrification, regulators should consider a move to capacity-based tariffs in order to reduce cross-subsidisation between consumers and increase cost reflectivity without impacting the DSO cost recovery.

In many areas, such as the physical sciences, life sciences, and finance, control approaches are used to achieve a desired goal in complex dynamical systems governed by differential equations. In this work we formulate the problem of controlling stochastic partial differential equations (SPDE) as a reinforcement learning problem. We present a learning-based, distributed control approach for online control of a system of SPDEs with high dimensional state-action space using deep deterministic policy gradient method. We tested the performance of our method on the problem of controlling the stochastic Burgers' equation, describing a turbulent fluid flow in an infinitely large domain.

The authors provide a comprehensive overview of flexibility characterization along the dimensions of time, spatiality, resource, and risk in power systems. These dimensions are discussed in relation to flexibility assets, products, and services, as well as new and existing flexibility market designs. The authors argue that flexibility should be evaluated based on the dimensions under discussion. Flexibility products and services can increase the efficiency of power systems and markets if flexibility assets and related services are taken into consideration and used along the time, geography, technology, and risk dimensions. Although it is possible to evaluate flexibility in existing market designs, a local flexibility market may be needed to exploit the value of the flexibility, depending on the dimensions of the flexibility products and services. To locate flexibility in power grids and prevent incorrect valuations, the authors also discuss TSO-DSO coordination along the four dimensions, and they present interrelations between flexibility dimensions, products, services, and related market designs for productive usage of flexible electricity., Comment: This paper was sent to the Renewable & Sustainable Energy Reviews at August 2021. It is currently under review

The objective of this study is to predict road flooding risks based on topographic, hydrologic, and temporal precipitation features using machine learning models. Predictive flood monitoring of road network flooding status plays an essential role in community hazard mitigation, preparedness, and response activities. Existing studies related to the estimation of road inundations either lack observed road inundation data for model validations or focus mainly on road inundation exposure assessment based on flood maps. This study addresses this limitation by using crowdsourced and fine-grained traffic data as an indicator of road inundation, and topographic, hydrologic, and temporal precipitation features as predictor variables. Two tree-based machine learning models (random forest and AdaBoost) were then tested and trained for predicting road inundations in the contexts of 2017 Hurricane Harvey and 2019 Tropical Storm Imelda in Harris County, Texas. The findings from Hurricane Harvey indicate that precipitation is the most important feature for predicting road inundation susceptibility, and that topographic features are more essential than hydrologic features for predicting road inundations in both storm cases. The random forest and AdaBoost models had relatively high AUC scores (0.860 and 0.810 for Harvey respectively and 0.790 and 0.720 for Imelda respectively) with the random forest model performing better in both cases. The random forest model showed stable performance for Harvey, while varying significantly for Imelda. This study advances the emerging field of smart flood resilience in terms of predictive flood risk mapping at the road level. For example, such models could help impacted communities and emergency management agencies develop better preparedness and response strategies with improved situational awareness of road inundation likelihood as an extreme weather event unfolds., Comment: 17 pages, 7 figures

Performance-based optimization of energy dissipation devices in structures necessitates massive and repetitive dynamic analyses. In the endurance time method known as a rather fast dynamic analysis procedure, structures are subjected to intensifying dynamic excitations and their response at multiple intensity levels is estimated by a minimal number of analyses. So, this method significantly reduces computational endeavors. In this paper, the endurance time method is employed to determine the optimal placement of viscous dampers in a weak structure to achieve the desired performance at various hazard levels, simultaneously. The viscous damper is one of the energy dissipation systems which can dissipate a large amount of seismic input energy to the structure. To this end, hysteretic energy compatible endurance time excitation functions are used and the validity of the results is investigated by comparing them with the results obtained from a suite of ground motions. To optimize the placement of the dampers, the genetic algorithm is used. The damping coefficients of the dampers are considered as design variables in the optimization procedure and determined in such a way that the sum of them has a minimum value. The behavior of the weak structure before and after rehabilitation is also investigated using endurance time and nonlinear time history analysis procedures in different hazard levels., Comment: This article has been fully published in NMCE journal

Heat-Bath Algorithmic Cooling techniques (HBAC) are techniques that are used to purify a target element in a quantum system. These methods compress and transfer entropy away from the target element into auxiliary elements of the system. The performance of Algorithmic Cooling has been investigated under ideal noiseless conditions. However, realistic implementations are imperfect and for practical purposes, noise should be taken into account. Here we analyze Heat-Bath Algorithmic Cooling techniques under realistic noise models. Surprisingly, we find that noise can in some cases enhance the performance and improve the cooling limit of Heat-Bath Algorithmic Cooling techniques. We numerically simulate the noisy algorithmic cooling for the two optimal strategies, the Partner Pairing, and the Two-sort algorithms. We find that for both of them, in the presence of the generalized amplitude damping noise, the process converges and the asymptotic purity can be higher than the noiseless process. This opens up new avenues for increasing the purity beyond the heat-bath algorithmic cooling.

Various methods have been presented to improve the performance of buildings against earthquakes. Friction damper device is one of the energy dissipation devices that appropriately absorbs and dissipates the input energy and decreases displacements. In this paper, the possibility of using endurance time method to determine the efficiency index and optimum slip load for these dampers was investigated by comparing the results of endurance time and nonlinear time history analyses. The efficiency indexes acquired from the average of results for nonlinear time history and endurance time analyses were close to each other. In this research, by assuming identical optimum slip load for the dampers in all stories, the normalized damper strength was increased in a number of equal steps ranging from zero to one to determine the efficiency index of dampers in each step. Then, the optimum slip load of dampers in the steel frames was calculated according to the minimum efficiency index of dampers. As a result, employing the endurance time method instead of a high number of nonlinear time history analyses is also possible, and using the endurance time method diminishes 57% of computational endeavors. Lastly, a relation for acquiring the optimum slip load of the friction damper devices in steel structures was proposed in terms of the weight of the structures. After adding optimal FDDs to the structures and investigating the effectiveness of the dampers, it was concluded that by using endurance time excitation function with better energy consistency, the endurance time results could be improved.

We construct certain integral structures for the cores of reduced tame extended affine Lie algebras of rank at least 2. One of the main tools to achieve this is a generalization of Chevalley automorphisms in the context of extended affine Lie algebras. As an application, groups of extended affine Lie type associated to the adjoint representation are defined over arbitrary fields., Comment: 45 pages

Approximate computing is an emerging paradigm to improve the power and performance efficiency of error-resilient applications. As adders are one of the key components in almost all processing systems, a significant amount of research has been carried out towards designing approximate adders that can offer better efficiency than conventional designs, however, at the cost of some accuracy loss. In this paper, we highlight a new class of energy-efficient approximate adders, namely Heterogeneous Block-based Approximate Adders (HBAA), and propose a generic configurable adder model that can be configured to represent a particular HBAA configuration. An HBAA, in general, is composed of heterogeneous sub-adder blocks of equal length, where each sub-adder can be an approximate sub-adder and have a different configuration. The sub-adders are mainly approximated through inexact logic and carry truncation. Compared to the existing design space, HBAAs provide additional design points that fall on the Pareto-front and offer a better quality-efficiency trade-off in certain scenarios. Furthermore, to enable efficient design space exploration based on user-defined constraints, we propose an analytical model to efficiently evaluate the Probability Mass Function (PMF) of approximation error and other error metrics, such as Mean Error Distance (MED), Normalized Mean Error Distance (NMED) and Error Rate (ER) of HBAAs. The results show that HBAA configurations can provide around 15% reduction in area and up to 17% reduction in energy compared to state-of-the-art approximate adders., Comment: Accepted for publication in ACM Transactions on Embedded Computing Systems (TECS)

In this work, we investigate how flexible assets within a residential building influence the long-term impact of operation. We use a measured-peak grid tariff (MPGT) that puts a cost on the highest single-hour peak import over the month. We apply a mathematical model of a Home Energy Management System (HEMS) together with Stochastic Dynamic Programming (SDP), which calculates the long-term impact of operating as a non-linear expected future cost curve (EFCC) from the end of the scheduling period to the start. The proposed model is applied to a case study for a Norwegian building with smart control of a battery energy storage system (BESS), Electric vehicle (EV) charging and space heating (SH). Each of the flexible assets are investigated individually with MPGT and for an energy-based grid tariff. The results showed that EV charging has the highest peak-power impact in the system, decreasing the total electricity cost by 14.6% with MPGT when controllable compared to a reference case with passive charging. It is further shown how the EFCC helps achieve optimal timing and level of the peak demand, where it co-optimizes both real-time pricing and the MPGT., Comment: This paper was sent to the International Journal of Electrical Power & Energy Systems early February 2021. Currently under review

In this paper, we introduce an application of a high-performance MultiPeriod AC Optimal Power Flow (MPOPF) solver, called "BATTPOWER", to simulate active distribution grids for a near-future scenario. A large-scale Norwegian distribution grid along with a large population of Electric Vehicles (EV) are here taken as the case-study. We suggest and analyse three operational strategies (in terms of control of charge scheduling fleet of EV) for the Distribution System Operator (DSO): (a) uncoordinated/dumb charge scheduling, (b) coordinated charge scheduling with the objective of energy cost-minimisation without operational constraints of the grid, and (c) coordinated charge scheduling with the objective of energy cost-minimisation along with the operational constraints of the grid. The results demonstrate that the uncoordinated charging would lead to: 1) overloading of lines and transformers when the share of EVs is above 20%, and 2) higher operational costs than the proposed control strategies of (b) and (c). In strategy (b) operational line/transformer limits are violated when the populations of EVs are growing above 36%. This implies that current market design must be altered to allow active control of a large proportion of DERs within grid operational limits to achieve cost minimization at system level. To our knowledge, the work presented in this paper is the first ever attempt to do a comprehensive analysis of the impact of EV charging demand on a real Norwegian distribution grid. Moreover, the inference of the analysis says that the Norwegian distribution networks are more prone to congestion problems than the voltage problems for the EV demand which includes a smart charging scheme accounting for grid conditions., Comment: Preprint submitted to Electric Power Systems Research (EPSR)

With the introduction of massive renewable energy sources and storage devices, the traditional process of grid operation must be improved in order to be safe, reliable, fast responsive and cost efficient, and in this regard power flow solvers are indispensable. In this paper, we introduce an Interior Point-based (IP) Multi-Period AC Optimal Power Flow (MPOPF) solver for the integration of Stationary Energy Storage Systems (SESS) and Electric Vehicles (EV). The primary methodology is based on: 1) analytic and exact calculation of partial differential equations of the Lagrangian sub-problem, and 2) exploiting the sparse structure and pattern of the coefficient matrix of Newton-Raphson approach in the IP algorithm. Extensive results of the application of proposed methods on several benchmark test systems are presented and elaborated, where the advantages and disadvantages of different existing algorithms for the solution of MPOPF, from the standpoint of computational efficiency, are brought forward. We compare the computational performance of the proposed Schur-Complement algorithm with a direct sparse LU solver. The comparison is performed for two different applicational purposes: SESS and EV. The results suggest the substantial computational performance of Schur-Complement algorithm in comparison with that of a direct LU solver when the number of storage devices and optimisation horizon increase for both cases of SESS and EV. Also, some situations where computational performance is inferior are discussed., Comment: 24 pages, 15 figures, Accepted for publication in IEEE Transactions on Power Systems

Reinforcement Learning (RL) agents typically learn memoryless policies---policies that only consider the last observation when selecting actions. Learning memoryless policies is efficient and optimal in fully observable environments. However, some form of memory is necessary when RL agents are faced with partial observability. In this paper, we study a lightweight approach to tackle partial observability in RL. We provide the agent with an external memory and additional actions to control what, if anything, is written to the memory. At every step, the current memory state is part of the agent's observation, and the agent selects a tuple of actions: one action that modifies the environment and another that modifies the memory. When the external memory is sufficiently expressive, optimal memoryless policies yield globally optimal solutions. Unfortunately, previous attempts to use external memory in the form of binary memory have produced poor results in practice. Here, we investigate alternative forms of memory in support of learning effective memoryless policies. Our novel forms of memory outperform binary and LSTM-based memory in well-established partially observable domains.

Scheduling a residential building short-term to optimize the electricity bill can be difficult with the inclusion of capacity-based grid tariffs. Scheduling the building based on a proposed measured-peak (MP) grid tariff, which is a cost based on the highest peak power over a period, requires the user to consider the impact the current decision-making has in the future. Therefore, the authors propose a mathematical model using stochastic dynamic programming (SDP) that tries to represent the long-term impact of current decision-making. The SDP algorithm calculates non-linear expected future cost curves (EFCC) for the building based on the peak power backwards for each day over a month. The uncertainty in load demand and weather are considered using a discrete Markov chain setup. The model is applied to a case study for a Norwegian building with smart control of flexible loads, and compared against methods where the MP grid tariff is not accurately represented, and where the user has perfect information of the whole month. The results showed that the SDP algorithm performs 0.3 % better than a scenario with no accurate way of presenting future impacts, and performs 3.6 % worse compared to a scenario where the user had perfect information., Comment: Presented at the 16th International Conference on Probabilistic Methods Applied to Power Systems 2020 (PMAPS 2020) August 19th. Awarded the Roy Billinton Student Paper Gold Award

A multiple antenna base station (BS) with an intelligent reflecting surface (IRS) platform, and several single antenna users are considered in the downlink mode. Simultaneous wireless information and power transfer (SWIPT) is utilized by the BS via transmit beamforming to convey information and power to all devices. Each device applies power splitting (PS) to dedicate separate parts of received power to information decoding and energy harvesting. We formulate a total transmit power minimization problem to jointly design the BS beamforming vectors, IRS phase shifts, and PS ratios at the receivers subject to minimum rate and harvested energy quality of service (QoS) constraints at all the receivers. First, we develop a block coordinate descent algorithm, also known as alternating optimization that can decrease the objective function with every iteration with guaranteed convergence. Afterwards, two low-complexity sub-optimal algorithms that rely on well-known maximum ratio transmission and zero-forcing beamforming techniques are introduced. These algorithms are beneficial when the number of BS antennas and/or number of users are large, or coherence times of channels are small. Simulations corroborate the expectation that by deploying a passive IRS, BS power can be reduced by $10-20$ dBw while maintaining similar guaranteed QoS. Furthermore, even the proposed sub-optimal algorithms outperform the globally optimal SWIPT solution without IRS for a modest number of IRS elements.

Prioritized Experience Replay (ER) has been empirically shown to improve sample efficiency across many domains and attracted great attention; however, there is little theoretical understanding of why such prioritized sampling helps and its limitations. In this work, we take a deep look at the prioritized ER. In a supervised learning setting, we show the equivalence between the error-based prioritized sampling method for mean squared error and uniform sampling for cubic power loss. We then provide theoretical insight into why it improves convergence rate upon uniform sampling during early learning. Based on the insight, we further point out two limitations of the prioritized ER method: 1) outdated priorities and 2) insufficient coverage of the sample space. To mitigate the limitations, we propose our model-based stochastic gradient Langevin dynamics sampling method. We show that our method does provide states distributed close to an ideal prioritized sampling distribution estimated by the brute-force method, which does not suffer from the two limitations. We conduct experiments on both discrete and continuous control problems to show our approach's efficacy and examine the practical implication of our method in an autonomous driving application., Comment: Accepted to UAI2022

The objective of this study is to create and test a hybrid deep learning model, FastGRNN-FCN (Fast, Accurate, Stable and Tiny Gated Recurrent Neural Network-Fully Convolutional Network), for urban flood prediction and situation awareness using channel network sensors data. The study used Harris County, Texas as the testbed, and obtained channel sensor data from three historical flood events (e.g., 2016 Tax Day Flood, 2016 Memorial Day flood, and 2017 Hurricane Harvey Flood) for training and validating the hybrid deep learning model. The flood data are divided into a multivariate time series and used as the model input. Each input comprises nine variables, including information of the studied channel sensor and its predecessor and successor sensors in the channel network. Precision-recall curve and F-measure are used to identify the optimal set of model parameters. The optimal model with a weight of 1 and a critical threshold of 0.59 are obtained through one hundred iterations based on examining different weights and thresholds. The test accuracy and F-measure eventually reach 97.8% and 0.792, respectively. The model is then tested in predicting the 2019 Imelda flood in Houston and the results show an excellent match with the empirical flood. The results show that the model enables accurate prediction of the spatial-temporal flood propagation and recession and provides emergency response officials with a predictive flood warning tool for prioritizing the flood response and resource allocation strategies.

Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples. In this work, we propose a novel regularization approach as an alternative. To derive the regularizer, we formulate the adversarial robustness problem under the robust optimization framework and approximate the loss function using a second-order Taylor series expansion. Our proposed second-order adversarial regularizer (SOAR) is an upper bound based on the Taylor approximation of the inner-max in the robust optimization objective. We empirically show that the proposed method significantly improves the robustness of networks against the $\ell_\infty$ and $\ell_2$ bounded perturbations generated using cross-entropy-based PGD on CIFAR-10 and SVHN.

In this article we investigate rigidity properties of $S$-arithmetic Kac-Moody groups in characteristic $0$.

This paper considers the problem of learning a model in model-based reinforcement learning (MBRL). We examine how the planning module of an MBRL algorithm uses the model, and propose that the model learning module should incorporate the way the planner is going to use the model. This is in contrast to conventional model learning approaches, such as those based on maximum likelihood estimate, that learn a predictive model of the environment without explicitly considering the interaction of the model and the planner. We focus on policy gradient type of planning algorithms and derive new loss functions for model learning that incorporate how the planner uses the model. We call this approach Policy-Aware Model Learning (PAML). We theoretically analyze a generic model-based policy gradient algorithm and provide a convergence guarantee for the optimized policy. We also empirically evaluate PAML on some benchmark problems, showing promising results.

Like air flowing over a wing, optimizing the flow of electronic charge is essential to the operation of nanoscale devices. Unfortunately, the delicate interplay of charge, spin, and heat in complex devices has precluded detailed imaging of charge flow. Here, we report on the visualization of intrinsic charge current streamlines through yttrium iron garnet micromagnetic heterostructures. Scanning photovoltage microscopy of precisely designed devices leads to striking spatial patterns, with prominent photovoltage features emerging in corners and narrow constrictions. These patterns, which evolve continuously with rotation of an external magnetic field, enable rich spatial mapping of fluid-like flow. Taking inspiration from aerodynamic Clark Y airfoils, we engineer micromagnetic wing shaped devices, called electrofoils, which allow us to precisely contort, compress and decompress flowlines of electronic charge.120 (39) e2221815120, Comment: 7 Pages, 4 figures, supplemental materials attached after references

For multi-valued functions---such as when the conditional distribution on targets given the inputs is multi-modal---standard regression approaches are not always desirable because they provide the conditional mean. Modal regression algorithms address this issue by instead finding the conditional mode(s). Most, however, are nonparametric approaches and so can be difficult to scale. Further, parametric approximators, like neural networks, facilitate learning complex relationships between inputs and targets. In this work, we propose a parametric modal regression algorithm. We use the implicit function theorem to develop an objective, for learning a joint function over inputs and targets. We empirically demonstrate on several synthetic problems that our method (i) can learn multi-valued functions and produce the conditional modes, (ii) scales well to high-dimensional inputs, and (iii) can even be more effective for certain uni-modal problems, particularly for high-frequency functions. We demonstrate that our method is competitive in a real-world modal regression problem and two regular regression datasets., Comment: Accepted to NeurIPS 2020

Model-based reinforcement learning has been empirically demonstrated as a successful strategy to improve sample efficiency. In particular, Dyna is an elegant model-based architecture integrating learning and planning that provides huge flexibility of using a model. One of the most important components in Dyna is called search-control, which refers to the process of generating state or state-action pairs from which we query the model to acquire simulated experiences. Search-control is critical in improving learning efficiency. In this work, we propose a simple and novel search-control strategy by searching high frequency regions of the value function. Our main intuition is built on Shannon sampling theorem from signal processing, which indicates that a high frequency signal requires more samples to reconstruct. We empirically show that a high frequency function is more difficult to approximate. This suggests a search-control strategy: we should use states from high frequency regions of the value function to query the model to acquire more samples. We develop a simple strategy to locally measure the frequency of a function by gradient and hessian norms, and provide theoretical justification for this approach. We then apply our strategy to search-control in Dyna, and conduct experiments to show its property and effectiveness on benchmark domains., Comment: Accepted to ICLR 2020

Incorporation of high-rate internet of things (IoT) service into a massive MIMO framework is investigated. It is revealed that massive MIMO possess the inherent potential to offer such service provided it knows the channels for all devices. Our proposed method is to jointly estimate and track the channels of all devices irrespective of their current activity. Using the dynamical model for devices' channels evolution over time, optimal and sub-optimal trackers are developed for coordinated scenario. Furthermore, we introduce a new paradigm where the BS need not know the pilot access patterns of devices in advance which we refer to as uncoordinated setup. After motivating this scenario, we derive the optimal tracker which is intractable. Then, target tracking approaches are applied to address uncertainties in the measurements and derive sub-optimal trackers. Our proposed approaches explicitly address the channel aging problem and will not require downlink paging and uplink access request control channels which can become bottlenecks in crowded scenarios. The fundamental minimum mean square error (MMSE) gap between optimal coordinated and uncoordinated trackers which is defined as price of anarchy is evaluated and upper-bounded. Stability of optimal trackers is also investigated. Finally, performance of various proposed trackers are numerically compared., Comment: Submitted to IEEE Transactions on Wireless Communications