Search

We study a class of reinforcement learning problems where the reward signals for policy learning are generated by a discriminator that is dependent on and jointly optimized with the policy. This interdependence between the policy and the discriminator leads to an unstable learning process because reward signals from an immature discriminator are noisy and impede policy learning, and conversely, an under-optimized policy impedes discriminator learning. We call this learning setting \textit{Internally Rewarded Reinforcement Learning} (IRRL) as the reward is not provided directly by the environment but \textit{internally} by the discriminator. In this paper, we formally formulate IRRL and present a class of problems that belong to IRRL. We theoretically derive and empirically analyze the effect of the reward function in IRRL and based on these analyses propose the clipped linear reward function. Experimental results show that the proposed reward function can consistently stabilize the training process by reducing the impact of reward noise, which leads to faster convergence and higher performance compared with baselines in diverse tasks., Comment: Accepted at ICML 2023. Project webpage at https://ir-rl.github.io

Explaining the behavior of intelligent agents such as robots to humans is challenging due to their incomprehensible proprioceptive states, variational intermediate goals, and resultant unpredictability. Moreover, one-step explanations for reinforcement learning agents can be ambiguous as they fail to account for the agent's future behavior at each transition, adding to the complexity of explaining robot actions. By leveraging abstracted actions that map to task-specific primitives, we avoid explanations on the movement level. Our proposed framework combines reward decomposition (RD) with abstracted action spaces into an explainable learning framework, allowing for non-ambiguous and high-level explanations based on object properties in the task. We demonstrate the effectiveness of our framework through quantitative and qualitative analysis of two robot scenarios, showcasing visual and textual explanations, from output artifacts of RD explanation, that are easy for humans to comprehend. Additionally, we demonstrate the versatility of integrating these artifacts with large language models for reasoning and interactive querying., Comment: https://lukaswill.github.io

Direct laser fabrication (DLF) has been widely used as an additive manufacturing technology. However, the effect of shielding gas flow rate and interaction among other parameters such as laser power, scanning speed, and powder feed rate in fabrication have not been studied. In this study, four factors were evaluated by two levels of full factorial experiment and response surface methodology (RSM). Thus, the interactions between processing parameters affecting two responses (shape factor and dilution rate) were determined, and regression models for two responses related to shielding gas flow rate were statistically evaluated for the first time. Interactive optimization was performed using contour and surface plots. Hence, the optimum combinations of four parameters were verified by metallurgical and mechanical analyses. More uniform surface profiles and finer microstructures can be obtained by adjusting the shielding gas flow rate. The results show that shielding gas flow rate and its interaction with other parameters for two responses play an important role in fabricating parts of AISI 316 L stainless steel. This fundamental study provides the significance of this parameter and better understanding of interaction between multiple processing parameters.

In this investigation, in order to improve bonding strength between superabrasive/metal matrix/grinding substrate, life span of grinding wheel, and grinding using small grits in continuous grinding simultaneously to fit for high speed and high precision machining in industry, coaxial powder feeding laser cladding method with CAD/CAM technology is introduced to manufacture textured CBN/CuSnTi–grinding wheels. The morphology of CBN grit on laser-cladding layer under optimized laser-cladding parameters and a pit created by fallen-off CBN on laser-cladding layer with lower laser–cladding energy density are observed by scanning electron microscope (SEM). The element distributions of interfaces of CBN/CuSnTi/AISI 1045 are analyzed by SEM and energy disperse spectroscope (EDS). The morphology and elements distribution of residual resultants on the surface of CBN grits etched by nitric acid are analyzed by SEM and EDS. Comparative-grinding process between laser cladding–grinding wheel (LCGW) and customized electroplated grinding wheel (EGW) is analyzed with grinding forces and temperature aspects respectively. The wear morphology of CBN grits on LCGW after grinding is observed by SEM. The results show that CBN grit with integrate cutting edges can protrude 50% height of its diameter on laser-cladding layer under optimized laser–cladding parameters. Fe, N, Ti, and B segregates attached to the interfaces of CBN/CuSnTi/AISI 1045 with Cu and Sn distributed uniformly in the laser-cladding layer. Residual resultants on CBN can be divided into two parts based on the distances from the surface of CBN grits. The grinding forces (Fz and Fy) and grinding temperature from LCGW are lower than those from EGW. The wear conditions of CBN on laser cladding are three parts: microfracture, cleavage plane, and wear-out.

Aiming at the serious problem of gap leakage between the stator and rotor of the supercritical carbon dioxide compressor, the dry gas seal scheme for centrifugal compressor of small S-CO2 power system is designed in this paper. The sealing characteristics of the single-stage spiral groove dry gas seal in the S-CO2 operating environment are studied by the large-scale commercial CFD numerical analysis software ANSYS CFX. The leakage characteristics of the S-CO2 dry gas seal at different film thicknesses are obtained. The results show that the S-CO2 dry gas seal leakage flow rate increases linearly with the increase of the film thickness. The dry gas seal leakage flow rate under different film thickness is 0.04%∼0.45% of the S-CO2 compressor flow rate, which can meet the leakage requirements of the centrifugal compressor shaft end seal in the small S-CO2 power system.

In order to study the effect of surface microarc oxidation modification on the comprehensive properties of titanium alloys in seawater environment, the metallographic structure of titanium alloy Microarc oxidation modified surface layer and the basic physical parameters such as hardness, thickness and roughness were measured, and the PTFE, filled with 15% glass fiber +5% were filled with 25% fiberglass. The PTFE of graphite and the PTFE filled with 60% tin bronze were tested for friction by friction pair materials, and the wear amount and roughness of titanium alloy and Microarc oxidation modified surface layer were test in simulated sea water as 3.5% Nacl solution, and the results showed that the surface modification of titanium alloy by Microarc oxidation could significantly improve its hardness. The surface roughness can reach 3.2μm, and the friction coefficient of different matching pairs is small, while the surface wear of microarc oxidation modification is less than that of substrate wear.

Zhao, X.; Yu, T.; Zhao, Y.; Zhang, Y.; Sun, X.; Ma, Z.; Li, M., and Zhang, T., 2018. The investigation of the morphology and failure types of pre-placed CBN on Cu-Ni-Sn-Ti bonded to steel by laser process. In: Liu, Z.L. and Mi, C. (eds.), Advances in Sustainable Port and Ocean Engineering. Journal of Coastal Research, Special Issue No. 83, pp. 486–489 . Coconut Creek (Florida), ISSN 0749-0208. Pre-placed CBN on Cu-Ni-Sn-Ti was bonded to AISI 1045 steel with various process parameters by laser in the present paper. The wettability and the oxidization condition of the CBN in laser bead were analyzed by scanning electron microscopy (SEM), energy dispersive X-ray spectroscopy (EDS) respectively. The failure types of the CBN in laser bead were observed by optical microscopy (OM). The results show that the wettability of the CBN and Cu-Ni-Sn-Ti mixed powder is pronounced because of active element Ti regardless the CBN size and laser processing parameters. Oxidization is serious after laser process even...

A new type of water mist fire extinguisher was proposed to be used in the manned spacecraft. The goal of prototype is that it can extinguish class 8B fires on the ground and its mass is less than 2.5 kg. The prototype employed only water as suppression agents. In the extinguisher, a piston separated the water from nitrogen artificially, which dispels the influence of the gravity on discharging. It possesses some advantages, such as high reliability, easy refilling of water and nitrogen, long life span, and convenient maintenance. A pressure-swirl nozzle was employed to produce water mist. There was no part to keep the pressure constant at the nozzle’s entrance; the service pressure of nozzle was determined in the range from 6 to 15 MPa. In that scope, the variation of droplet size with pressure did not have a significant impact on suppression performance, and the suppression time increment due to velocity reduction was in the acceptable level. A series of tests were conducted to verify its performance. The vibration test presented that there was no nitrogen leakage and no obvious resonance happened, which meant it possessed high reliability. The fire suppression tests indicated that it can suppress different fire types of a certain scale. The simulation results showed that the spray performance is improved in the microgravity. Conclusion can be drawn that the prototype is feasible in the microgravity and it reached the requirements of the design.

In this article, the mechanical characteristics of the squeeze of the F7 fault of the Wushaoling Tunnel are analyzed. The measurements and techniques are proposed to resist the deformation. The result indicates that the method of construction to control the further squeeze deflection is appropriate.