Search

This document is a concise summary of many key results in single-agent reinforcement learning (RL). The intended audience are those who already have some familiarity with RL and are looking to review, reference and/or remind themselves of important ideas in the field.

Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS., Comment: 20 pages, 11 figures. Paper accepted to CoRL 2020

We consider the problem of learning to communicate using multi-agent reinforcement learning (MARL). A common approach is to learn off-policy, using data sampled from a replay buffer. However, messages received in the past may not accurately reflect the current communication policy of each agent, and this complicates learning. We therefore introduce a 'communication correction' which accounts for the non-stationarity of observed communication induced by multi-agent learning. It works by relabelling the received message to make it likely under the communicator's current policy, and thus be a better reflection of the receiver's current environment. To account for cases in which agents are both senders and receivers, we introduce an ordered relabelling scheme. Our correction is computationally efficient and can be integrated with a range of off-policy algorithms. We find in our experiments that it substantially improves the ability of communicating MARL systems to learn across a variety of cooperative and competitive tasks.

We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, we introduce Feudal Multi-agent Hierarchies (FMH). In this framework, a 'manager' agent, which is tasked with maximising the environmentally-determined reward function, learns to communicate subgoals to multiple, simultaneously-operating, 'worker' agents. Workers, which are rewarded for achieving managerial subgoals, take concurrent actions in the world. We outline the structure of FMH and demonstrate its potential for decentralised learning and control. We find that, given an adequate set of subgoals from which to choose, FMH performs, and particularly scales, substantially better than cooperative approaches that use a shared reward function.

This paper analyses the short utterance probabilistic linear discriminant analysis (PLDA) speaker verification with utterance partitioning and short utterance variance (SUV) modelling approaches. Experimental studies have found that instead of using single long-utterance as enrolment data, if long enrolled utterance is partitioned into multiple short utterances and average of short utterance i-vectors is used as enrolled data, that improves the Gaussian PLDA (GPLDA) speaker verification. This is because short utterance i-vectors have speaker, session and utterance variations, and utterance-partitioning approach compensates the utterance variation. Subsequently, SUV-PLDA is also studied with utterance partitioning approach, and utterance partitioning-based SUV-GPLDA system shows relative improvement of 9% and 16% in EER for NIST 2008 and NIST 2010 truncated 10sec-10sec evaluation condition as utterance partitioning approach compensates the utterance variation and SUV modelling approach compensates the mismatch between full-length development data and short-length evaluation data., Comment: arXiv admin note: text overlap with arXiv:1610.02831

This paper investigates the effects of limited speech data in the context of speaker verification using deep neural network (DNN) approach. Being able to reduce the length of required speech data is important to the development of speaker verification system in real world applications. The experimental studies have found that DNN-senone-based Gaussian probabilistic linear discriminant analysis (GPLDA) system respectively achieves above 50% and 18% improvements in EER values over GMM-UBM GPLDA system on NIST 2010 coreext-coreext and truncated 15sec-15sec evaluation conditions. Further when GPLDA model is trained on short-length utterances (30sec) rather than full-length utterances (2min), DNN-senone GPLDA system achieves above 7% improvement in EER values on truncated 15sec-15sec condition. This is because short length development i-vectors have speaker, session and phonetic variation and GPLDA is able to robustly model those variations. For several real world applications, longer utterances (2min) can be used for enrollment and shorter utterances (15sec) are required for verification, and in those conditions, DNN-senone GPLDA system achieves above 26% improvement in EER values over GMM-UBM GPLDA systems.

This paper explores how the in- and out-domain probabilistic linear discriminant analysis (PLDA) speaker verification behave when enrolment and verification lengths are reduced. Experiment studies have found that when full-length utterance is used for evaluation, in-domain PLDA approach shows more than 28% improvement in EER and DCF values over out-domain PLDA approach and when short utterances are used for evaluation, the performance gain of in-domain speaker verification reduces at an increasing rate. Novel modified inter dataset variability (IDV) compensation is used to compensate the mismatch between in- and out-domain data and IDV-compensated out-domain PLDA shows respectively 26% and 14% improvement over out-domain PLDA speaker verification when SWB and NIST data are respectively used for S normalization. When the evaluation utterance length is reduced, the performance gain by IDV also reduces as short utterance evaluation data i-vectors have more variations due to phonetic variations when compared to the dataset mismatch between in- and out-domain data.

We use NMR techniques to investigate the magnetic properties of BaCo2As2 single crystals, the non-superconducting end member of the Co-substituted iron-pnictide high Tc superconductor Ba(Fe1-xCox)2As2 with x = 1. We present 75As NMR evidence for enhancement of low frequency spin fluctuations below ~ 100 K. This enhancement is accompanied by that of static uniform spin susceptibility at the wave vector q = 0, suggesting that the primary channel of the spin correlations is ferromagnetic rather than antiferromagnetic. Comparison between the NMR Knight shift 75K and bulk susceptibility chi(bulk) data uncovers the presence of two separate components of spin susceptibility with distinct temperature dependences, presumably because multiple electronic bands crossing the Fermi energy play different roles in the electronic properties of BaCo2As2., Comment: Submitted to PRB

We will examine the possible link between spin fluctuations and the superconducting mechanism in the iron-based high temperature superconductor Ba(Fe,Co)2As2 based on NMR and high pressure transport measurements., Comment: Invited paper to m2s-IX (2009)

We report the first NMR investigation of spin dynamics in the overdoped non-superconducting regime of Ba(Fe[1-x]Co[x])2As2 up to x =0.26. We demonstrate that the absence of inter-band transitions with large momentum transfer Q(AF) ~ (pi, 0) between the hole and electron Fermi surfaces results in complete suppression of antiferromagnetic spin fluctuations for x > 0.15. Our experimental results provide direct evidence for a correlation between Tc and the strength of Q(AF) antiferromagnetic spin fluctuations., Comment: Accepted for publication in Phys. Rev. Lett

We report comprehensive resistivity measurements of single crystalline samples of the Ba(Fe(1-x)Co(x))2As2 high Tc superconductor under hydrostatic pressure up to 2.75 GPa and over a broad concentration range, 0 < x < 0.099. We show that application of pressure progressively suppresses the SDW transition temperature, T_SDW, in the underdoped regime. There is no sign of pressure-induced superconductivity in the undoped BaFe2As2 down to 1.8 K, but applied pressure dramatically enhances Tc in the underdoped regime. The effect of pressure on Tc is very small in the optimally and overdoped regimes. As a consequence, the dome of the superconducting phase extends to x 0.02 under pressure. We discuss the implications of our findings in the context of a possible quantum phase transition between the SDW and superconducting phases., Comment: 6 figures

We demonstrate that the anisotropy R of the paramagnetic spin fluctuations grows toward Tc at 75As sites in the optimally electron-doped superconductor Ba[(Fe0.92Co0.08)2]2As2, with stronger spin fluctuations along the c-axis. Our finding is in remarkable contrast with the case of high T$_c$ cuprates, where R is independent of temperature above Tc., Comment: 2 Figures

Unlike the parent phases of the iron-arsenide high Tc superconductors, undoped FeSe is not magnetically ordered and exhibits superconductivity with Tc~9K. Equally surprising is the fact that applied pressure dramatically enhances the modest Tc to ~37K. We investigate the electronic properties of FeSe using 77Se NMR to search for the key to the superconducting mechanism. We demonstrate that the electronic properties of FeSe are very similar to those of electron-doped FeAs superconductors, and that antiferromagnetic spin fluctuations are strongly enhanced near Tc. Furthermore, applied pressure enhances spin fluctuations. Our findings suggest a link between spin fluctuations and the superconducting mechanism in FeSe., Comment: Accepted for publication in Phys. Rev. Lett. after minor revision

We investigate the electronic properties of Ba[Fe(1-x)Co(x)]2As2 (x =0.02, 0.04) in the lightly electron-doped regime by 59Co and 75As NMR. We demonstrate that Co doping significantly suppresses the magnetic ordering temperature to the SDW state, T_{SDW}. Furthermore, ordered moments below T_{SDW} exhibit large distribution. Strong spin fluctuations remain even below T_{SDW}, persisting all the way down to 4.2 K. We find no signature of additional freezing of spin degrees of freedom unlike the case of the lightly hole-doped stripe phase of the cuprates., Comment: 4 figures are changed to color online. 4 pages

We report a systematic investigation of Ba[Fe(1-x)Co(x)]2As2 based on transport and 75-As NMR measurements, and establish the electronic phase diagram. We demonstrate that doping progressively suppresses the uniform spin susceptibility and low frequency spin fluctuations. The optimum superconducting phase emerges at x_c~0.08 when the tendency toward spin ordering completely diminishes. Our findings point toward the presence of a quantum critical point near x_c between the SDW (spin density wave) and superconducting phases., Comment: 5 Figures

Application of pressures or electron-doping through Co substitution into Fe sites transforms the itinerant antiferromagnet BaFe(2)As(2) into a superconductor with the Tc exceeding 20K. We carried out systematic transport measurements of BaFe(2-x)Co(x)As(2) superconductors in pressures up to 2.5GPa, and elucidate the interplay between the effects of electron-doping and pressures. For the underdoped sample with nominal composition x = 0.08, application of pressure strongly suppresses a magnetic instability while enhancing Tc by nearly a factor of two from 11K to 21K. In contrast, the optimally doped x=0.20 sample shows very little enhancement of Tc=22K under applied pressure. Our results strongly suggest that the proximity to a magnetic instability is the key to the mechanism of superconductivity in iron-pnictides., Comment: 5 figures

We will probe the intrinsic behavior of spin susceptibility chi_(spin) in the LaFeAsO(1-x)F(x) superconductor (x ~ 0.1, Tc ~ 27K) using 19-F and 75-As NMR techniques. Our new results firmly establish the pseudo-gap behavior with Delta_(PG)/kB ~ 140K. The estimated magnitude of chi_(spin) at 290K, ~1.8x10^(-4) [emu/mol-Fe], is approximately twice larger than that in high Tc cuprates. We also show that chi_(spin) levels off below ~50K down to Tc., Comment: Invited paper to the Proceedings of the International Symposium on Iron-oxypnictide Superconductors (June 28-29, 2008, Tokyo). J. Phys. Soc. Jpn. Suppl. (in press)

We report an NMR investigation of the superconductivity in BaFe(2)As(2) induced by Co doping (Tc=22K). We demonstrate that Co atoms form an alloy with Fe atoms and donate carriers without creating localized moments. Our finding strongly suggests that the underlying physics of iron-pnictide superconductors is quite different from the widely accepted physical picture of high Tc cuprates as doped Mott insulators. We also show a crossover of electronic properties into a low temperature pseudo-gap phase with a pseudo-gap Delta 560K, where chi(spin) constant and resisitivty T. The NMR Knight shift below Tc decreases for both along the c-axis and ab-plane, and is consistent with the singlet pairing scenario., Comment: Accepted for publication in J. Phys. Soc. Jpn. (4 pages)

We report 19-F NMR investigation of the new high temperature superconductor LaFeAsO(0.89)F(0.11) (Tc ~ 28K). We demonstrate that low frequency spin fluctuations exhibit pseudo gap behavior above Tc. We also deduce the London penetration depth lambda from NMR line broadening below Tc., Comment: Accepted for publication in Phys. Rev. B (Rapid) with the original submission date

The CoO$_{2}$ layers in sodium-cobaltates Na$_{x}$CoO$_{2}$ may be viewed as a spin $S=1/2$ triangular-lattice doped with charge carriers. The underlying physics of the cobaltates is very similar to that of the high $T_{c}$ cuprates. We will present unequivocal $^{59}$Co NMR evidence that below $T_{CO}\sim51 K$, the insulating ground state of the itinerant antiferromagnet Na$_{0.5}$CoO$_{2}$ ($T_{N}\sim 86 K$) is induced by charge ordering., Comment: Phys. Rev. Lett. 100 (2008), in press. 4 figures

We present the results of electrical resistivity and Hall effect measurements on single crystals of HfNiSn, TiPtSn, and TiNiSn. Semiconducting behavior is observed in each case, involving the transport of a small number of highly compensated carriers. Magnetization measurements suggest that impurities and site disorder create both localized magnetic moments and extended paramagnetic states, with the susceptibility of the latter increasing strongly with reduced temperature. The magnetoresistance is sublinear or linear in fields ranging from 0.01 - 9 Tesla at the lowest temperatures. As the temperature increases, the normal quadratic magnetoresistance is regained, initially at low fields, and at the highest temperatures extending over the complete range of fields. The origin of the vanishingly small field scale implied by these measurements remains unknown, presenting a challenge to existing classical and quantum mechanical theories of magnetoresistance., Comment: 15 pages, 12 figures