Search

Study region: The Tagus-Segura aqueduct (TSA) is a large and strategic water transfer scheme in Spain that connects Entrepeñas and Buendía reservoirs in the Tagus river headwaters to the Segura river basin, a highly stressed Mediterranean area. Study focus: The operating rules of the TSA underwent several modifications over the years, and the debate about which are the optimal parameters to meet the interests of the parties involved is still open. We employed Evolutionary Multi-Objective Direct Policy Search to jointly optimize the re-operation of the headwaters dams and the water transfer policy with respect to four conflicting objectives: Tagus and Segura water demands, hydropower production and socioeconomic benefit of the population living on the shores of the headwaters reservoirs. We tested the optimization under the baseline and the 2027 scenario, which foresees an increased environmental flow (EF) in the Tagus river. New hydrological insights for the region: The proposed operating rule presents optimized control parameters, a higher degree of freedom and a transferred volume that cyclically varies according to the hydrological stage of the year. In the 2027 scenario, despite the increased EF, the deficit in the aqueduct shows a limited increase compared to the historical solution (+10%), while the storage deficit is strongly reduced (−73%). This benefits the population living on the reservoirs shores and also ensures more stability to the aqueduct functioning.

The construction of the Akosombo and Kpong dams in the Lower Volta River Basin in Ghana changed the downstream riverine ecosystem and affected the lives of downstream communities, particularly those who lost their traditional livelihoods. In contrast to the costs borne by those in the vicinity of the river, Ghana as a whole, has enjoyed vast economic benefits from the affordable hydropower, irrigation schemes and lake tourism that developed after construction of the dams. Herein lies the challenge; there exists a trade-off between water for river ecosystems and related services on the one hand, and anthropogenic water demands such hydropower or irrigation on the other. In this study, an Evolutionary Multi-Objective Direct Policy Search (EMODPS) is used to identify the multi-sectorial trade-offs that exist in the Lower Volta River Basin. Three environmental flows, previously determined for the Lower Volta are incorporated separately as an environmental objective. The results highlight the dominance of hydropower production in the Lower Volta, but show that there is room for providing environmental flows under current climatic and water use conditions if firm energy requirement from Akosombo Dam reduces by 12% to 38% depending on the environmental flow regime that is implemented. There is uncertainty in climate change effects on runoff in this region, however multiple scenarios are investigated. It is found that climate change leading to increased annual inflows to the Akosombo Dam reduces the trade-off between hydropower and the environment while climate change resulting in lower inflows provide the opportunity to strategically provide dry season environmental flows, that is, reduce flows sufficiently to meet low flow requirements for key ecosystem services such as the clam fishery. This study not only highlights the challenges in balancing anthropogenic water demands and environmental considerations in managing existing dams, but also identifies opportunities for compromise in the Lower Volta River [ABSTRACT FROM AUTHOR]

Direct policy search is a promising reinforcement learning framework particularly for controlling continuous, high-dimensional systems. Peters et al. proposed reward-weighted regression (RWR) as a direct policy search. The RWR algorithm estimates the policy parameter based on the expectation-maximization (EM) algorithm and is therefore prone to overfitting. In this study, we focus on variational Bayesian inference to avoid overfitting and propose direct policy search reinforcement learning based on variational Bayesian inference (VBRL). The performance of the proposed VBRL is assessed in several experiments involving a mountain car and a ball batting task. These experiments demonstrate that VBRL yields a higher average return and outperforms the RWR. [ABSTRACT FROM AUTHOR]

Evolutionary Multi-Objective Direct Policy Search (EMODPS) is a prominent framework for designing control policies in multi-purpose environmental systems, combining direct policy search with multi-objective evolutionary algorithms (MOEAs) to identify Pareto approximate control policies. While EMODPS is effective, the choice of functions within its global approximator networks remains underexplored, despite their potential to significantly influence both solution quality and MOEA performance. This study conducts a rigorous assessment of a suite of Radial Basis Functions (RBFs) as candidates for these networks. We critically evaluate their ability to map system states to control actions, and assess their influence on Pareto efficient control policies. We apply this analysis to two contrasting case studies: the Conowingo Reservoir System, which balances competing water demands including hydropower, environmental flows, urban supply, power plant cooling, and recreation; and The Shallow Lake Problem, where a city navigates the trade-off between environmental and economic objectives when releasing anthropogenic phosphorus. Our findings reveal that the choice of RBF functions substantially impacts model outcomes. In complex scenarios like multi-objective reservoir control, this choice is critical, while in simpler contexts, such as the Shallow Lake Problem, the influence is less pronounced, though distinctive differences emerge in the characteristics of the prescribed control strategies. • RBF choice in EMODPS impacts tradeoffs and policies in multiobjective control. • Lake Problem: RBFs affect control policies, not objective values. • Concave RBFs excel in complex EMODPS, like Conowingo Reservoir. [ABSTRACT FROM AUTHOR]

Sea-level rise poses considerable risks to coastal communities, ecosystems, and infrastructure. Decision makers are faced with uncertain sea-level projections when designing a strategy for coastal adaptation. The traditional methods are often silent on tradeoffs as well as the effects of tail-area events and of potential future learning. Here we reformulate a simple sea-level rise adaptation model to address these concerns. We show that Direct Policy Search yields improved solution quality, with respect to Pareto-dominance in the objectives, over the traditional approach under uncertain sea-level rise projections and storm surge. Additionally, the new formulation produces high quality solutions with less computational demands than an intertemporal optimization approach. Our results illustrate the utility of multi-objective adaptive formulations for the example of coastal adaptation and point to wider-ranging application in climate change adaptation decision problems. [ABSTRACT FROM AUTHOR]

Managing socio-ecological systems is a challenge wrought by competing societal objectives, deep uncertainties, and potentially irreversible tipping points. A classic, didactic example is the shallow lake problem in which a hypothetical town situated on a lake must develop pollution control strategies to maximize its economic benefits while minimizing the probability of the lake crossing a critical phosphorus (P) threshold, above which it irreversibly transitions into a eutrophic state. Here, we explore the use of direct policy search (DPS) to design robust pollution control rules for the town that account for deeply uncertain system characteristics and conflicting objectives. The closed loop control formulation of DPS improves the quality and robustness of key management tradeoffs, while dramatically reducing the computational complexity of solving the multi-objective pollution control problem relative to open loop control strategies. These insights suggest DPS is a promising tool for managing socio-ecological systems with deeply uncertain tipping points. [ABSTRACT FROM AUTHOR]

During periods of significant water shortage or when drought is impending, it is customary to implement some kind of water supply reduction measures with a view to prevent the occurrence of severe shortages (vulnerability) in the near future. In the case of operation of a water supply reservoir, this reduction of water supply is affected by hedging schemes or hedging policies. This research work aims to compare the popular hedging policies: (i) linear two-point hedging; (ii) modified two-point hedging; and, (iii) discrete hedging based on time-varying and constant hedging parameters. A parameterization-simulation-optimization (PSO) framework is employed for the selection of the parameters of the compromising hedging policies. The multi-objective evolutionary search-based technique (Non-dominated Sorting based Genetic Algorithm-II) was used to identify the Pareto-optimal front of hedging policies that seek to obtain the trade-off between shortage ratio and vulnerability. The case example used for illustration is the Hemavathy reservoir in Karnataka, India. It is observed that the Pareto-optimal front that was obtained from time-varying hedging policies show significant improvement in reservoir performance when compared to constant hedging policies. The variation in the monthly parameters of the time-variant hedging policies shows a strong correlation with monthly inflows and available water.

Pickup and delivery problems have numerous applications in practice such as parcel delivery and passenger transportation. In the dynamic variant of the problem, not all information is available in advance but is revealed during the planning process. Thus, it is crucial to anticipate future events in order to generate high-quality solutions. Previous work has shown that the use of waiting strategies has the potential to save costs and maximize service quality. We adapt various waiting heuristics to the pickup and delivery problem with time windows. Previous research has shown, that specialized waiting heuristics utilizing anticipatory knowledge potentially outperform general heuristics. Direct policy search based on evolutionary computation and a simulation model is proposed as a methodology to automatically specialize waiting strategies to different problem characteristics. Based on the strengths of the previously introduced waiting strategies, we propose a novel waiting heuristic that can utilize historical request information based on an intensity measure which does not require an additional data preprocessing step. The performance of the waiting heuristics is evaluated on a single set of benchmark instances containing various instance classes that differ in terms of spatial and temporal properties. The diverse set of benchmark instances is used to analyze the influence of spatial and temporal instance properties as well as the degree of dynamism to the potential savings that can be achieved by anticipatory waiting and the incorporation of knowledge about future requests. [ABSTRACT FROM AUTHOR]

This paper introduces an algorithm for direct search of control policies in continuous-state discrete-action Markov decision processes. The algorithm looks for the best closed-loop policy that can be represented using a given number of basis functions (BFs), where a discrete action is assigned to each BF. The type of the BFs and their number are specified in advance and determine the complexity of the representation. Considerable flexibility is achieved by optimizing the locations and shapes of the BFs, together with the action assignments. The optimization is carried out with the cross-entropy method and evaluates the policies by their empirical return from a representative set of initial states. The return for each representative state is estimated using Monte Carlo simulations. The resulting algorithm for cross-entropy policy search with adaptive BFs is extensively evaluated in problems with two to six state variables, for which it reliably obtains good policies with only a small number of BFs. In these experiments, cross-entropy policy search requires vastly fewer BFs than value-function techniques with equidistant BFs, and outperforms policy search with a competing optimization algorithm called DIRECT. [ABSTRACT FROM AUTHOR]

During periods of significant water shortage or when drought is impending, it is customary to implement some kind of water supply reduction measures with a view to prevent the occurrence of severe shortages (vulnerability) in the near future. In the case of operation of a water supply reservoir, this reduction of water supply is affected by hedging schemes or hedging policies. This research work aims to compare the popular hedging policies: (i) linear two-point hedging; (ii) modified two-point hedging; and, (iii) discrete hedging based on time-varying and constant hedging parameters. A parameterization-simulation-optimization (PSO) framework is employed for the selection of the parameters of the compromising hedging policies. The multi-objective evolutionary search-based technique (Non-dominated Sorting based Genetic Algorithm-II) was used to identify the Pareto-optimal front of hedging policies that seek to obtain the trade-off between shortage ratio and vulnerability. The case example used for illustration is the Hemavathy reservoir in Karnataka, India. It is observed that the Pareto-optimal front that was obtained from time-varying hedging policies show significant improvement in reservoir performance when compared to constant hedging policies. The variation in the monthly parameters of the time-variant hedging policies shows a strong correlation with monthly inflows and available water. [ABSTRACT FROM AUTHOR]