Search

In the post-training of large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) is an effective approach to achieve generation aligned with human preferences. Direct Preference Optimization (DPO) allows for policy training with a simple binary cross-entropy loss without a reward model. The objective of DPO is regularized by reverse KL divergence that encourages mode-seeking fitting to the reference policy. Nonetheless, we indicate that minimizing reverse KL divergence could fail to capture a mode of the reference distribution, which may hurt the policy's performance. Based on this observation, we propose a simple modification to DPO, H-DPO, which allows for control over the entropy of the resulting policy, enhancing the distribution's sharpness and thereby enabling mode-seeking fitting more effectively. In our experiments, we show that H-DPO outperformed DPO across various tasks, demonstrating superior results in pass@$k$ evaluations for mathematical tasks. Moreover, H-DPO is simple to implement, requiring only minor modifications to the loss calculation of DPO, which makes it highly practical and promising for wide-ranging applications in the training of LLMs.

While musculoskeletal humanoids have the advantages of various biomimetic structures, it is difficult to accurately control the body, which is challenging to model. Although various learning-based control methods have been developed so far, they cannot completely absorb model errors, and recognition errors are also bound to occur. In this paper, we describe a method to modify the movement of the musculoskeletal humanoid by applying external force during the movement, taking advantage of its flexible body. Considering the fact that the joint angles cannot be measured, and that the external force greatly affects the nonlinear elastic element and not the actuator, the modified motion is reproduced by the proposed muscle-based compensation control. This method is applied to a musculoskeletal humanoid, Musashi, and its effectiveness is confirmed., Comment: Acceptedt at Humanoids2020

We proposed a method for learning the actual body image of a musculoskeletal humanoid for posture generation and object manipulation using inverse kinematics with redundancy in the shoulder complex. The effectiveness of this method was confirmed by realizing automobile steering wheel operation. The shoulder complex has a scapula that glides over the rib cage and an open spherical joint, and is supported by numerous muscle groups, enabling a wide range of motion. As a development of the human mimetic shoulder complex, we have increased the muscle redundancy by implementing deep muscles and stabilize the joint drive. As a posture generation method to utilize the joint redundancy of the shoulder complex, we consider inverse kinematics based on the scapular drive strategy suggested by the scapulohumeral rhythm of the human body. In order to control a complex robot imitating a human body, it is essential to learn its own body image, but it is difficult to know its own state accurately due to its deformation which is difficult to measure. To solve this problem, we developed a method to acquire a self-body image that can be updated appropriately by recognizing the hand position relative to an object for the purpose of object manipulation. We apply the above methods to a full-body musculoskeletal humanoid, Kengoro, and confirm its effectiveness by conducting an experiment to operate a car steering wheel, which requires the appropriate use of both arms., Comment: Accepted at IEEE Robotics and Automation Letters

Musculoskeletal humanoids have various biomimetic advantages, of which redundant muscle arrangement is one of the most important features. This feature enables variable stiffness control and allows the robot to keep moving its joints even if one of the redundant muscles breaks, but this has been rarely explored. In this study, we construct a neural network that represents the relationship among sensors in the flexible and difficult-to-modelize body of the musculoskeletal humanoid, and by learning this neural network, accurate motions can be achieved. In order to take advantage of the redundancy of muscles, we discuss the use of this network for muscle rupture detection, online update of the intersensory relationship considering the muscle rupture, and body control and state estimation using the muscle rupture information. This study explains a method of constructing a musculoskeletal humanoid that continues to move and perform tasks robustly even when one muscle breaks., Comment: Accepted at Robotics and Autonomous Systems

Human-like environment recognition by musculoskeletal humanoids is important for task realization in real complex environments and for use as dummies for test subjects. Humans integrate various sensory information to perceive their surroundings, and hearing is particularly useful for recognizing objects out of view or out of touch. In this research, we aim to realize human-like auditory environmental recognition and task realization for musculoskeletal humanoids by equipping them with a human-like auditory processing system. Humans realize sound-based environmental recognition by estimating directions of the sound sources and detecting environmental sounds based on changes in the time and frequency domain of incoming sounds and the integration of auditory information in the central nervous system. We propose a human mimetic auditory information processing system, which consists of three components: the human mimetic binaural ear unit, which mimics human ear structure and characteristics, the sound source direction estimation system, and the environmental sound detection system, which mimics processing in the central nervous system. We apply it to Musashi, a human mimetic musculoskeletal humanoid, and have it perform tasks that require sound information outside of view in real noisy environments to confirm the usefulness of the proposed methods., Comment: Accepted at ROBOMECH Journal

For a robot with redundant sensors and actuators distributed throughout its body, it is difficult to construct a controller or a neural network using all of them due to computational cost and complexity. Therefore, it is effective to extract functionally related sensors and actuators, group them, and construct a controller or a network for each of these groups. In this study, the functional and spatial connections among sensors and actuators are embedded into a graph structure and a method for automatic grouping is developed. Taking a musculoskeletal humanoid with a large number of redundant muscles as an example, this method automatically divides all the muscles into regions such as the forearm, upper arm, scapula, neck, etc., which has been done by humans based on a geometric model. The functional relationship among the muscles and the spatial relationship of the neural connections are calculated without a geometric model., Comment: Accepted at IEEE Robotics and Automation Letters

We report an estimation of the injected mass composition of ultra-high energy cosmic rays (UHECRs) at energies higher than 10 EeV. The composition is inferred from an energy-dependent sky distribution of UHECR events observed by the Telescope Array surface detector by comparing it to the Large Scale Structure of the local Universe. In the case of negligible extra-galactic magnetic fields the results are consistent with a relatively heavy injected composition at E ~ 10 EeV that becomes lighter up to E ~ 100 EeV, while the composition at E > 100 EeV is very heavy. The latter is true even in the presence of highest experimentally allowed extra-galactic magnetic fields, while the composition at lower energies can be light if a strong EGMF is present. The effect of the uncertainty in the galactic magnetic field on these results is subdominant., Comment: 8 pages, 3 figures, accepted for publication in PRL

We use a new method to estimate the injected mass composition of ultrahigh cosmic rays (UHECRs) at energies higher than 10 EeV. The method is based on comparison of the energy-dependent distribution of cosmic ray arrival directions as measured by the Telescope Array experiment (TA) with that calculated in a given putative model of UHECR under the assumption that sources trace the large-scale structure (LSS) of the Universe. As we report in the companion letter, the TA data show large deflections with respect to the LSS which can be explained, assuming small extra-galactic magnetic fields (EGMF), by an intermediate composition changing to a heavy one (iron) in the highest energy bin. Here we show that these results are robust to uncertainties in UHECR injection spectra, the energy scale of the experiment and galactic magnetic fields (GMF). The assumption of weak EGMF, however, strongly affects this interpretation at all but the highest energies E > 100 EeV, where the remarkable isotropy of the data implies a heavy injected composition even in the case of strong EGMF. This result also holds if UHECR sources are as rare as $2 \times 10^{-5}$ Mpc$^{-3}$, that is the conservative lower limit for the source number density., Comment: 18 pages, 11 figures, accepted for publication in PRD

We report interfacial transport properties in in-situ grown orthorhombic perovskite oxide heterostructures consisting of an antiferromagnetic insulator DyFeO$_3$ and a paramagnetic conductor CaRuO$_3$. We observe Hall effect with a step-like increase amounting to an effective magnetic field of 30 T at 20 K. We provide a plausible explanation in the context of topological Hall effect originating from a non-coplanar spin texture and resultant emergent field in DyFeO$_3$ associated with the scalar spin chirality. Our results demonstrate that the proximity effect of the emergent field at heterointerfaces is a universal physical phenomenon, while it has been reported originally in a heterointerface composed of pyrochlore oxides. This will greatly expand the choice of materials to the heterointerfaces for the research in emergent transport phenomena, which has been limited to single compounds with both metallic properties and special spin textures. Additionally, this will pave the way for possible device application of the emergent field by designing and combining perovskite oxides with versatile functionalities such as multiferroicity.

The collisionless plasmas in space and astrophysical environments are intrinsically multiscale in nature, behaving as conducting fluids at macroscales and kinetically at microscales comparable to ion- and/or electron-gyroradii. A fundamental question in understanding the plasma dynamics is how energy is transported and dissipated across different scales. Here, we present spacecraft measurements in the solar wind upstream of the terrestrial bow shock, in which the macroscale ultra-low-frequency waves and microscale whistler waves simultaneously resonate with the ions. The ion acceleration from ultra-low-frequency waves leads to velocity distributions unstable to the growth of whistler waves, which in turn resonate with the electrons to complete cross-scale energy transfer. These observations, consistent with numerical simulations in the occurrence of phase-bunched ion and electron distributions, also highlight the importance of anomalous resonance, a nonlinear modification of the classical cyclotron resonance, in the cross-scale wave coupling and energy transfer processes., Comment: 22 pages, 7 figures and supplementary material

This paper summarizes an autonomous driving project by musculoskeletal humanoids. The musculoskeletal humanoid, which mimics the human body in detail, has redundant sensors and a flexible body structure. These characteristics are suitable for motions with complex environmental contact, and the robot is expected to sit down on the car seat, step on the acceleration and brake pedals, and operate the steering wheel by both arms. We reconsider the developed hardware and software of the musculoskeletal humanoid Musashi in the context of autonomous driving. The respective components of autonomous driving are conducted using the benefits of the hardware and software. Finally, Musashi succeeded in the pedal and steering wheel operations with recognition., Comment: Accepted at IEEE Robotics and Automation Magazine

In offline reinforcement learning, in-sample learning methods have been widely used to prevent performance degradation caused by evaluating out-of-distribution actions from the dataset. Extreme Q-learning (XQL) employs a loss function based on the assumption that Bellman error follows a Gumbel distribution, enabling it to model the soft optimal value function in an in-sample manner. It has demonstrated strong performance in both offline and online reinforcement learning settings. However, issues remain, such as the instability caused by the exponential term in the loss function and the risk of the error distribution deviating from the Gumbel distribution. Therefore, we propose Maclaurin Expanded Extreme Q-learning to enhance stability. In this method, applying Maclaurin expansion to the loss function in XQL enhances stability against large errors. This approach involves adjusting the modeled value function between the value function under the behavior policy and the soft optimal value function, thus achieving a trade-off between stability and optimality depending on the order of expansion. It also enables adjustment of the error distribution assumption from a normal distribution to a Gumbel distribution. Our method significantly stabilizes learning in online RL tasks from DM Control, where XQL was previously unstable. Additionally, it improves performance in several offline RL tasks from D4RL., Comment: Accepted at RLC 2024: The first Reinforcement Learning Conference

Wave-particle resonance, a ubiquitous process in the plasma universe, occurs when resonant particles observe a constant wave phase to enable sustained energy transfer. Here, we present spacecraft observations of simultaneous Landau and anomalous resonances between oblique whistler waves and the same group of protons, which are evidenced, respectively, by phase-space rings in parallel-velocity spectra and phase-bunched distributions in gyro-phase spectra. Our results indicate the coupling between Landau and anomalous resonances via the overlapping of the resonance islands., Comment: 13 pages, 4 figures and supplementary material

We report on ALMA ACA observations of a high-density region of the Corona Australis cloud forming a young star cluster, and the results of resolving internal structures. In addition to embedded Class 0/I protostars in continuum, a number of complex dense filamentary structures are detected in the C18O and SO lines by the 7m array. These are sub-structures of the molecular clump that are detected by the TP array as the extended emission. We identify 101 and 37 filamentary structures with a few thousand AU widths in C18O and SO, respectively, called as feathers. The typical column density of the feathers in C18O is about 10^{22} cm^{-2}, and the volume density and line mass are ~ 10^5 cm^{-3}, and a few times M_{sun} pc^{-1}, respectively. This line mass is significantly smaller than the critical line mass expected for cold and dense gas. These structures have complex velocity fields, indicating a turbulent internal property. The number of feathers associated with Class 0/I protostars is only ~ 10, indicating that most of them do not form stars but rather being transient structures. The formation of feathers can be interpreted as a result of colliding gas flow as the morphology well reproduced by MHD simulations, supported by the the presence of HI shells in the vicinity. The colliding gas flows may accumulate gas and form filaments and feathers, and trigger the active star formation of the R CrA cluster., Comment: 24 pages, 13 figures; Accepted for publication in ApJ

Musculoskeletal humanoids have been developed by imitating humans and expected to perform natural and dynamic motions as well as humans. To achieve desired motions stably in current musculoskeletal humanoids is not easy because they cannot maintain the sufficient moment arm of muscles in various postures. In this research, we discuss planar structures that spread across joint structures such as ligament and planar muscles and the application of planar interskeletal structures to humanoid robots. Next, we develop MusashiOLegs, a musculoskeletal legs which has planar interskeletal structures and conducts several experiments to verify the importance of planar interskeletal structures., Comment: accepted at Humanoids2020

In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gaussian error distribution. However, a recent study suggested that the error distribution for training the value function is often skewed because of the properties of the Bellman operator, and violates the implicit assumption of normal error distribution in the least squares method. To address this, we proposed a method called Symmetric Q-learning, in which the synthetic noise generated from a zero-mean distribution is added to the target values to generate a Gaussian error distribution. We evaluated the proposed method on continuous control benchmark tasks in MuJoCo. It improved the sample efficiency of a state-of-the-art reinforcement learning method by reducing the skewness of the error distribution., Comment: Accepted at AAAI 2024: The 38th Annual AAAI Conference on Artificial Intelligence (Main Tech Track)

Recent Atacama Large Millimeter/submillimeter Array (ALMA) observations have revealed an increasing number of compact protostellar disks with radii of less than a few tens of astronomical units and that young Class 0/I objects have an intrinsic size diversity. To deepen our understanding of the origin of such tiny disks, we performed the highest-resolution configuration observations with ALMA at a beam size of $\sim$0$''$03 (4 au) on the very low-luminosity Class 0 protostar embedded in the Taurus dense core MC 27/L1521F. The 1.3 mm continuum measurement successfully resolved a tiny, faint ($\sim$1 mJy) disk with a major axis length of $\sim$10 au, one of the smallest examples in the ALMA protostellar studies. In addition, we detected spike-like components in the northeastern direction at the disk edge. Gravitational instability or other fragmentation mechanisms cannot explain the structures, given the central stellar mass of $\sim$0.2 $M_{\odot}$ and the disk mass of $\gtrsim$10$^{-4}$ $M_{\odot}$. Instead, we propose that these small spike structures were formed by a recent dynamic magnetic flux transport event due to interchange instability that would be favorable to occur if the parental core has a strong magnetic field. The presence of complex arc-like structures on a larger ($\sim$2000 au) scale in the same direction as the spike structures suggests that the event was not single. Such episodic, dynamical events may play an important role in maintaining the compact nature of the protostellar disk in the complex gas envelope during the main accretion phase., Comment: 13 pages, 5 figures, Accepted for publication in ApJ

We present ${\sim}0.2$ arcsec ($\sim$80 au) resolution observations of the CO (2-1) and SiO (5-4) lines made with the Atacama large millimeter/submillimeter array toward an extremely young intermediate-mass protostellar source (t$_{\rm dyn}<$1000 years), MMS 1 located in the Orion Molecular Cloud-3 region. We have successfully imaged a very compact CO molecular outflow associated with MMS 1, having deprojected lobe sizes of $\sim$18000 au (red-shifted lobe) and $\sim$35000 au (blue-shifted lobe). We have also detected an extremely compact ($\lesssim$1000 au) and collimated SiO protostellar jet within the CO outflow. The maximum deprojected jet speed is measured to be as high as 93 km s$^{-1}$. The SiO jet wiggles and displays a chain of knots. Our detection of the molecular outflow and jet is the first direct evidence that MMS 1 already hosts a protostar. The position-velocity diagram obtained from the SiO emission shows two distinct structures: (i) bow-shocks associated with the tips of the outflow, and (ii) a collimated jet, showing the jet velocities linearly increasing with the distance from the driving source. Comparisons between the observations and numerical simulations quantitatively share similarities such as multiple-mass ejection events within the jet and Hubble-like flow associated with each mass ejection event. Finally, while there is a weak flux decline seen in the 850 $\mu$m light curve obtained with JCMT/SCUBA 2 toward MMS 1, no dramatic flux change events are detected. This suggests that there has not been a clear burst event within the last 8 years., Comment: 19 pages, 9 figures, Accepted for publication in ApJ

We present molecular line observations of the protostellar outflow associated with HH270mms1 in the Orion B molecular cloud with ALMA. The 12CO(J = 3 - 2) emissions show that the outflow velocity structure consists of four distinct components of low ($\gtrsim$ 10 km s-1), intermediate (~ 10 - 25 km s-1) and high ($\gtrsim$ 40 km s-1) velocities in addition to the entrained gas velocity (~ 25 - 40 km s-1). The high- and intermediate-velocity flows have well-collimated structures surrounded by the low-velocity flow. The chain of knots is embedded in the high-velocity flow or jet, which is the evidence of episodic mass ejections induced by time-variable mass accretion. We could detect the velocity gradients perpendicular to the outflow axis in both the low- and intermediate-velocity flows. We confirmed the rotation of the envelope and disk in the 13CO and C17O emission and found that their velocity gradients are the same as those of the outflow. Thus, we concluded that the velocity gradients in the low- and intermediate-velocity flows are due to the outflow rotation. Using observational outflow properties, we estimated the outflow launching radii to be 67.1 - 77.1 au for the low-velocity flow and 13.3 - 20.8 au for the intermediate-velocity flow. Although we could not detect the rotation in the jets due to the limited spatial resolution, we estimated the jet launching radii to be (2.36 - 3.14) x 10^-2 au using the observed velocity of each knots. Thus, the jet is driven from the inner disk region. We could identify the launching radii of distinct velocity components within a single outflow with all the prototypical characteristics expected from recent theoretical works., Comment: 20 pages, 12 figures, accepted for publication in the Astrophysical Journal

WL 17 is a Class I object and was considered to have a ring-hole structure. We analyzed the structure around WL 17 to investigate the detailed properties of WL 17. We used ALMA archival data, which have a higher angular resolution than previous observations. We investigated the WL 17 system with the 1.3 mm dust continuum and 12CO and C18O (J = 2-1) line emissions. The dust continuum emission showed a clear ring structure with inner and outer edges of ~11 and ~21 au, respectively. In addition, we detected an inner disk of < 5 au radius enclosing the central star within the ring, the first observation of this structure. Thus, WL 17 has a ring-gap structure, not a ring-hole structure. We did not detect any marked emission in either the gap or inner disk, indicating that there is no sign of a planet, circumplanetary disk, or binary companion. We identified the base of both blue-shifted and red-shifted outflows based on the 12CO emission, which is clearly associated with the disk around WL 17. The outflow mass ejection rate is ~3.6x10^-7 Msun yr-1 and the dynamical timescale is as short as ~ 10^4 yr. The C18O emission showed that an inhomogeneous infalling envelope, which can induce episodic mass accretion, is distributed in the region within ~1000 au from the central protostar. With these new findings, we can constrain the planet formation and dust growth scenarios in the accretion phase of star formation., Comment: 22 pages, 9 figures, Accepted for publication in the Astrophysical Journal

We study phenomenology of a light scalar dark matter (DM). In the model, there are an inert doublet scalar and a singlet Dirac fermion $\psi$, both charged under a global $Z_2$ symmetry. The mass of the lightest inert scalar $H$ can be lighter than 10 GeV by imposing appropriate relations between three scalar quartic couplings. The lightest $Z_2$ odd particle is stable and DM. In this paper, focusing on the parameter space where $H$ is lighter than $\psi$ and is DM, we discuss DM physics related to relic density, direct detection, indirect detection, collider searches and other cosmological observations. We clarify differences from the case where $\psi$ is instead DM, which has been focused on in the previous works., Comment: 19 pages + references

Transferring a significant fraction of the magnetic flux from a dense cloud core is essential in the star formation process. A ring-like structure produced by magnetic flux loss has been predicted theoretically, but no observational identification has been presented. We have performed ALMA observations of the Class I protostar IRS 2 in the Corona Australis star-forming region and resolved a distinctive gas ring in the C$^{18}$O ($J$ = 2-1) line emission. The center of this gas ring is $\sim$5,000 au away from the protostar, with a diameter of $\sim$7,000 au. The radial velocity of the gas is $\lesssim1$ km s$^{-1}$ blueshifted from that of the protostar, with a possible expanding feature judged from the velocity-field (moment 1) map and position-velocity diagram. These features are either observationally new or have been discovered but not discussed in depth because they are difficult to explain by well-studied protostellar phenomena such as molecular outflows and accretion streamers. A plausible interpretation is a magnetic wall created by the advection of magnetic flux which is theoretically expected in the Class 0/I phase during star formation as a removal mechanism of magnetic flux. Similar structures reported in the other young stellar sources could likely be candidates formed by the same mechanism, encouraging us to revisit the issue of magnetic flux transport in the early stages of star formation from an observational perspective., Comment: 9 pages, 3 figures, Accepted for publication in the Astronomical Journal Letters