Search

Quadruped robots are progressively being integrated into human environments. Despite the growing locomotion capabilities of quadrupedal robots, their interaction with objects in realistic scenes is still limited. While additional robotic arms on quadrupedal robots enable manipulating objects, they are sometimes redundant given that a quadruped robot is essentially a mobile unit equipped with four limbs, each possessing 3 degrees of freedom (DoFs). Hence, we aim to empower a quadruped robot to execute real-world manipulation tasks using only its legs. We decompose the loco-manipulation process into a low-level reinforcement learning (RL)-based controller and a high-level Behavior Cloning (BC)-based planner. By parameterizing the manipulation trajectory, we synchronize the efforts of the upper and lower layers, thereby leveraging the advantages of both RL and BC. Our approach is validated through simulations and real-world experiments, demonstrating the robot's ability to perform tasks that demand mobility and high precision, such as lifting a basket from the ground while moving, closing a dishwasher, pressing a button, and pushing a door. Project website: https://zhengmaohe.github.io/leg-manip, Comment: Project website: https://zhengmaohe.github.io/leg-manip

Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional augmentations are proposed to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, we get a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves offline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms. Our implementation is available at https://github.com/Dragon-Zhuang/BPPO., Comment: ICLR2023 Poster

Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: Can we achieve straightforward yet effective offline and online learning without introducing extra conservatism or regularization? In this study, we propose Uni-o4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent can transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-o4 leverages diverse ensemble policies to address the mismatch issues between the estimated behavior policy and the offline dataset. Through a simple offline policy evaluation (OPE) approach, Uni-o4 can achieve multi-step policy improvement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization as well as stable and rapid online fine-tuning capabilities. Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously unseen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning. Our website: https://lei-kun.github.io/uni-o4/ ., Comment: Our website: https://lei-kun.github.io/uni-o4

Rivers are under enormous threat worldwide and large amounts of money are invested in river restoration. Contrary to the costs, the benefits of river restoration are much harder to quantify. In this study, the benefits of restoring different sections of the Yongding River in Beijing, China, are estimated through a discrete choice experiment (DCE). Place attachment is measured by sampling residents upstream and downstream and using the river sections as labelled alternatives in the DCE. As expected, the improvement of water quality is valued highly by all river basin residents, and place attachment and spatial preference heterogeneity play a significant role in public willingness to pay (WTP) for river restoration. Although respondents are willing to give up only a small share of their disposable income, public WTP for improved river water quality is a factor 2 to 4 higher than the current household water bill. These findings provide important guidance for the recovery of the investment costs associated with river restoration projects.

For NP-hard combinatorial optimization problems, it is usually difficult to find high-quality solutions in polynomial time. The design of either an exact algorithm or an approximate algorithm for these problems often requires significantly specialized knowledge. Recently, deep learning methods provide new directions to solve such problems. In this paper, an end-to-end deep reinforcement learning framework is proposed to solve this type of combinatorial optimization problems. This framework can be applied to different problems with only slight changes of input (for example, for a traveling salesman problem (TSP), the input is the two-dimensional coordinates of nodes; while for a capacity-constrained vehicle routing problem (CVRP), the input is simply changed to three-dimensional vectors including the two-dimensional coordinates and the customer demands of nodes), masks and decoder context vectors. The proposed framework is aiming to improve the models in literacy in terms of the neural network model and the training algorithm. The solution quality of TSP and the CVRP up to 100 nodes are significantly improved via our framework. Specifically, the average optimality gap is reduced from 4.53\% (reported best \cite{R22}) to 3.67\% for TSP with 100 nodes and from 7.34\% (reported best \cite{R22}) to 6.68\% for CVRP with 100 nodes when using the greedy decoding strategy. Furthermore, our framework uses about 1/3$\sim$3/4 training samples compared with other existing learning methods while achieving better results. The results performed on randomly generated instances and the benchmark instances from TSPLIB and CVRPLIB confirm that our framework has a linear running time on the problem size (number of nodes) during the testing phase, and has a good generalization performance from random instance training to real-world instance testing.

A storm surge is a complex phenomenon in which waves, tide and current interact. Even though wind is the predominant force driving the surge, waves and tidal phase are also important factors that influence the mass and momentum transport during the surge. Devastating storm surges often occur in the Bohai Sea, a semi-enclosed shallow sea in North China, due to extreme storms. However, the effects of waves on storm surges in the Bohai Sea have not been quantified and the mechanisms responsible for the higher surges that affect part of the Bohai Sea have not been thoroughly studied. In this study, we set up a storm surge model, considering coupled effects of tides and waves on the surges. Validation against measured data shows that the coupled model is capable of simulating storm surges in the Bohai Sea. The simulation results indicate that the longshore currents, which are induced by the large gradient of radiation stress due to wave deformation, are one of the main contributors to the higher surges occurring in some coastal regions. The gently varying bathymetry is another factor contributing to these surges. With such bathymetry, the wave force direction is nearly uniform, and pushes a large amount of water in that direction. Under these conditions, the water accumulates in some parts of the coast, leading to higher surges in nearby coastal regions such as the south coast of the Bohai Bay and the west and south coasts of the Laizhou Bay. Results analysis also shows that the tidal phase at which the surge occurs influences the wave–current interactions, and these interactions are more evident in shallow waters. Neglecting these interactions can lead to inaccurate predictions of the storm surges due to overestimation or underestimation of wave-induced set-up.

In the framework of the EU-China Environmental Sustainability Program, the SUSTAIN H2O project was launched in 2013 and ran over 36 months until 2016. Sustain H2O is an abbreviation which stands for “Demonstration of Pollution Discharge Management for Water Quality Improvement in the Songhuajiang-Liaohe River Basin (SLRB)”. The SLRB represents China’s old industrial base with a large number of water-polluting industries. It has been selected as it is a key river basin for water pollution control and water environment management with clearly defined challenges in terms of pollution emission reduction and water quality improvement. Some of these challenges/problems include excessive discharge of pollutants, no optimal load allocation mechanism of quantity control, poor water quality and severely damaged water ecology as well as high risk of environmental pollution accidental events that threaten the safety of drinking water sources. Moreover, the SLRB is a trans-national (transboundary) river catchment, flowing from China to Russia, and therefore is well suited as a joint Sino-EU action.

This deliverables reports the “aquatic eco-function zoning map” delineation activities of the SUSTAIN H2O project consortium during the project’s reporting period.

Healthy river ecosystems maintain an integrity of their structural composition (including physical, chemical and biological composition), ecological processes and ecosystem functions. They have the ability to resist disturbance and restore their structure and function, and also provide ecological services in accordance with natural and human needs. River health assessment can identify water bodies with poor health and their belonging watersheds, and provide possible causes affecting river ecological conditions. In Europe and the United States and other developed countries, river health assessment has turned out to be an effective method for environmental water management.

This book describes the huge efforts by the Chinese Government concerning the restoration and future sustainable management of Chinese water systems. It presents the results of a Sino-European joint project concerning the Songhuajiang-Liaohe River Basin (SLRB) in Northeast China conducted by the Chinese Research Academy of Environmental Sciences (CRAES), the Helmholtz Centre for Environmental Research - UFZ, Germany, and the Natural Environment Research Council as represented by the Centre for Ecology and Hydrology (CEH), UK. The book explains in great detail the development of risk assessment and corresponding management methods for (i) controlling water pollution, (ii) assessing river health and ecological restoration options, (iii) characterizing persistent organic pollutants (POPs), and (iv) protecting fragile groundwater resources. It also describes the implemented demonstration sites of SLRB during the project course as well as lessons learnt on efficient project management and the dissemination of knowledge and technologies.

Healthy river ecosystems maintain an integrity of their structural composition (including physical, chemical and biological composition), ecological processes and ecosystem functions. They have the ability to resist disturbance and restore their structure and function, and also provide ecological services in accordance with natural and human needs. River health assessment can identify water bodies with poor health and their belonging watersheds, and provide possible causes affecting river ecological conditions. In Europe and the United States and other developed countries, river health assessment has turned out to be an effective method for environmental water management.

This deliverables reports the “aquatic eco-function zoning map” delineation activities of the SUSTAIN H2O project consortium during the project’s reporting period.

In the framework of the EU-China Environmental Sustainability Program, the SUSTAIN H2O project was launched in 2013 and ran over 36 months until 2016. Sustain H2O is an abbreviation which stands for “Demonstration of Pollution Discharge Management for Water Quality Improvement in the Songhuajiang-Liaohe River Basin (SLRB)”. The SLRB represents China’s old industrial base with a large number of water-polluting industries. It has been selected as it is a key river basin for water pollution control and water environment management with clearly defined challenges in terms of pollution emission reduction and water quality improvement. Some of these challenges/problems include excessive discharge of pollutants, no optimal load allocation mechanism of quantity control, poor water quality and severely damaged water ecology as well as high risk of environmental pollution accidental events that threaten the safety of drinking water sources. Moreover, the SLRB is a trans-national (transboundary) river catchment, flowing from China to Russia, and therefore is well suited as a joint Sino-EU action.

This book describes the huge efforts by the Chinese Government concerning the restoration and future sustainable management of Chinese water systems. It presents the results of a Sino-European joint project concerning the Songhuajiang-Liaohe River Basin (SLRB) in Northeast China conducted by the Chinese Research Academy of Environmental Sciences (CRAES), the Helmholtz Centre for Environmental Research - UFZ, Germany, and the Natural Environment Research Council as represented by the Centre for Ecology and Hydrology (CEH), UK. The book explains in great detail the development of risk assessment and corresponding management methods for (i) controlling water pollution, (ii) assessing river health and ecological restoration options, (iii) characterizing persistent organic pollutants (POPs), and (iv) protecting fragile groundwater resources. It also describes the implemented demonstration sites of SLRB during the project course as well as lessons learnt on efficient project management and the dissemination of knowledge and technologies.