Search

This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning (RL) operating in a partially observable environment. We assume that the network of agents operates in a fully-decentralized manner, possessing the capability to exchange variables with their immediate neighbors. The proposed design methodology is supported by an analysis demonstrating that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is $\varepsilon$-bounded when an appropriate number of iterations of social learning updates are implemented. Unlike many existing dec-POMDP-based RL approaches, the proposed algorithm is suitable for model-free multi-agent reinforcement learning as it does not require knowledge of a transition model. Furthermore, experimental results illustrate the efficacy of the algorithm and demonstrate its superiority over the current state-of-the-art methods.

This paper proposes a theoretical framework to evaluate and compare the performance of gradient-descent algorithms for distributed learning in relation to their behavior around local minima in nonconvex environments. Previous works have noticed that convergence toward flat local minima tend to enhance the generalization ability of learning algorithms. This work discovers two interesting results. First, it shows that decentralized learning strategies are able to escape faster away from local minimizers and favor convergence toward flatter minima relative to the centralized solution in the large-batch training regime. Second, and importantly, the ultimate classification accuracy is not solely dependent on the flatness of the local minimizer but also on how well a learning algorithm can approach that minimum. In other words, the classification accuracy is a function of both flatness and optimization performance. The paper examines the interplay between the two measures of flatness and optimization error closely. One important conclusion is that decentralized strategies of the diffusion type deliver enhanced classification accuracy because it strikes a more favorable balance between flatness and optimization performance.

Communication-constrained algorithms for decentralized learning and optimization rely on local updates coupled with the exchange of compressed signals. In this context, differential quantization is an effective technique to mitigate the negative impact of compression by leveraging correlations between successive iterates. In addition, the use of error feedback, which consists of incorporating the compression error into subsequent steps, is a powerful mechanism to compensate for the bias caused by the compression. Under error feedback, performance guarantees in the literature have so far focused on algorithms employing a fusion center or a special class of contractive compressors that cannot be implemented with a finite number of bits. In this work, we propose a new decentralized communication-efficient learning approach that blends differential quantization with error feedback. The approach is specifically tailored for decentralized learning problems where agents have individual risk functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. We show that, under some general conditions on the compression noise, and for sufficiently small step-sizes $\mu$, the resulting communication-efficient strategy is stable both in terms of mean-square error and average bit rate: by reducing $\mu$, it is possible to keep the estimation errors small (on the order of $\mu$) without increasing indefinitely the bit rate as $\mu\rightarrow 0$. The results establish that, in the small step-size regime and with a finite number of bits, it is possible to attain the performance achievable in the absence of compression., Comment: arXiv admin note: text overlap with arXiv:2209.07821

Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ oracle complexity to find an $\varepsilon$-stationary point. Some works indicate that this complexity can be improved to $\mathcal{O}(\varepsilon^{-3})$ when the loss gradient is Lipschitz continuous. The question of achieving enhanced convergence rates under distinct conditions, remains unresolved. In this work, we address this question for optimization problems that are nonconvex in the minimization variable and strongly concave or Polyak-Lojasiewicz (PL) in the maximization variable. We introduce novel bias-corrected momentum algorithms utilizing efficient Hessian-vector products. We establish convergence conditions and demonstrate a lower iteration complexity of $\mathcal{O}(\varepsilon^{-3})$ for the proposed algorithms. The effectiveness of the method is validated through applications to robust logistic regression using real-world datasets.

In this paper, we consider a setting where heterogeneous agents with connectivity are performing inference using unlabeled streaming data. Observed data are only partially informative about the target variable of interest. In order to overcome the uncertainty, agents cooperate with each other by exchanging their local inferences with and through a fusion center. To evaluate how each agent influences the overall decision, we adopt a causal framework in order to distinguish the actual influence of agents from mere correlations within the decision-making process. Various scenarios reflecting different agent participation patterns and fusion center policies are investigated. We derive expressions to quantify the causal impact of each agent on the joint decision, which could be beneficial for anticipating and addressing atypical scenarios, such as adversarial attacks or system malfunctions. We validate our theoretical results with numerical simulations and a real-world application of multi-camera crowd counting.

Motivated by recent experiments reporting unconventional collective modes in the charge density-wave (CDW) state of rare-earth tritellurides $R$Te$_3$, we derive from a multi-orbital microscopic model on the square net a CDW Ginzburg-Landau theory that allows for non-trivial orbital order. Our analysis reveals unconventional CDWs where order parameters with distinct orbital character coexist due to an approximate symmetry of the low-energy model, which becomes exact in the limit of nearest-neighbor-only hopping and decoupled $p_x$, $p_y$ orbitals. Because of this coexistence, the resulting CDW pattern displays an orbital texture that generally breaks additional symmetries of the lattice besides those explicitly broken by the CDW wave-vector. In particular, we find two competing phases that differ in whether they break or preserve inversion and vertical mirror symmetries. We explain the mechanisms that favor each outcome, and discuss experimental probes that can distinguish the different phases., Comment: Added references

Non-Bayesian social learning is a framework for distributed hypothesis testing aimed at learning the true state of the environment. Traditionally, the agents are assumed to receive observations conditioned on the same true state, although it is also possible to examine the case of heterogeneous models across the graph. One important special case is when heterogeneity is caused by the presence of malicious agents whose goal is to move the agents toward a wrong hypothesis. In this work, we propose an algorithm that allows discovering the true state of every individual agent based on the sequence of their beliefs. In so doing, the methodology is also able to locate malicious behavior.

In this work, we examine a network of agents operating asynchronously, aiming to discover an ideal global model that suits individual local datasets. Our assumption is that each agent independently chooses when to participate throughout the algorithm and the specific subset of its neighbourhood with which it will cooperate at any given moment. When an agent chooses to take part, it undergoes multiple local updates before conveying its outcomes to the sub-sampled neighbourhood. Under this setup, we prove that the resulting asynchronous diffusion strategy is stable in the mean-square error sense and provide performance guarantees specifically for the federated learning setting. We illustrate the findings with numerical simulations.

The optimistic gradient method is useful in addressing minimax optimization problems. Motivated by the observation that the conventional stochastic version suffers from the need for a large batch size on the order of $\mathcal{O}(\varepsilon^{-2})$ to achieve an $\varepsilon$-stationary solution, we introduce and analyze a new formulation termed Diffusion Stochastic Same-Sample Optimistic Gradient (DSS-OG). We prove its convergence and resolve the large batch issue by establishing a tighter upper bound, under the more general setting of nonconvex Polyak-Lojasiewicz (PL) risk functions. We also extend the applicability of the proposed method to the distributed scenario, where agents communicate with their neighbors via a left-stochastic protocol. To implement DSS-OG, we can query the stochastic gradient oracles in parallel with some extra memory overhead, resulting in a complexity comparable to its conventional counterpart. To demonstrate the efficacy of the proposed algorithm, we conduct tests by training generative adversarial networks.

Superlattice potential modulation can produce flat minibands in Bernal-stacked bilayer graphene. In this work we study how band topology and interaction-induced symmetry-broken phases in this system are controlled by tuning the displacement field and the shape and strength of the superlattice potential. We use an analytic perturbative analysis to demonstrate that topological flat bands are favored by a honeycomb-lattice-shaped potential, and numerics to show that the robustness of topological bands depends on both the displacement field strength and the periodicity of the superlattice potential. At integer fillings of the topological flat bands, the strength of the displacement field and the superlattice potential tune phase transitions between quantum anomalous Hall insulator, trivial insulator, and metallic states. We present mean-field phase diagrams in a gate voltage parameter space at filling factor $\nu=1$, and discuss the prospects of realizing quantum anomalous Hall insulators and fractional Chern insulators when the superlattice potential modulation is produced by dielectric patterning or adjacent moir\'e materials.

Traditional social learning frameworks consider environments with a homogeneous state, where each agent receives observations conditioned on that true state of nature. In this work, we relax this assumption and study the distributed hypothesis testing problem in a heterogeneous environment, where each agent can receive observations conditioned on their own personalized state of nature (or truth). We particularly focus on community structured networks, where each community admits their own true hypothesis. This scenario is common in various contexts, such as when sensors are spatially distributed, or when individuals in a social network have differing views or opinions. We show that the adaptive social learning strategy is a preferred choice for nonstationary environments, and allows each cluster to discover its own truth.

Quantum geometry plays a pivotal role in the second-order response of $\cal PT$-symmetric antiferromagnets. Here we study the nonlinear response of 2D altermagnets protected by $C_n\cal T$ symmetry and show that their leading nonlinear response is third-order. Furthermore, we show that the contributions from the quantum metric and Berry curvature enter separately: the longitudinal response for all planar altermagnets \emph{only} has a contribution from the quantum metric quadrupole (QMQ), while transverse responses in general have contributions from both the Berry curvature quadrupole (BCQ) and QMQ. We show that for the well-known example of $d$-wave altermagnets the Hall response is dominated by the BCQ. Both longitudinal and transverse responses are strongly dependent on the crystalline anisotropy. While altermagnets are strictly defined in the limit of vanishing SOC, real altermagnets exhibit weak SOC, which is essential to observe this response. Specifically, SOC gaps the spin-group protected nodal line, generating a response peak that is sharpest when SOC is weak. Two Dirac nodes also contribute a divergence to the nonlinear response, whose scaling changes as a function of SOC. Finally, we apply our results to thin films of the 3D altermagnet RuO$_2$. Our work uncovers distinct features of altermagnets in nonlinear transport, providing experimental signatures as well as a guide to disentangling the different components of their quantum geometry., Comment: 4 figures, 6+12, Supplementary materials included

This paper investigates causal influences between agents linked by a social graph and interacting over time. In particular, the work examines the dynamics of social learning models and distributed decision-making protocols, and derives expressions that reveal the causal relations between pairs of agents and explain the flow of influence over the network. The results turn out to be dependent on the graph topology and the level of information that each agent has about the inference problem they are trying to solve. Using these conclusions, the paper proposes an algorithm to rank the overall influence between agents to discover highly influential agents. It also provides a method to learn the necessary model parameters from raw observational data. The results and the proposed algorithm are illustrated by considering both synthetic data and real Twitter data., Comment: Submitted for publication

This paper studies the probability of error associated with the social machine learning framework, which involves an independent training phase followed by a cooperative decision-making phase over a graph. This framework addresses the problem of classifying a stream of unlabeled data in a distributed manner. In this work, we examine the classification task with limited observations during the decision-making phase, which requires a non-asymptotic performance analysis. We establish a condition for consistent training and derive an upper bound on the probability of error for classification. The results clarify the dependence on the statistical properties of the data and the combination policy used over the graph. They also establish the exponential decay of the probability of error with respect to the number of unlabeled samples.

We propose heterostructures that realize first and second order topological superconductivity with vanishing net magnetization by utilizing altermagnetism. Such platforms may offer a significant improvement over conventional platforms with uniform magnetization since the latter suppresses the superconducting gap. We first introduce a 1D semiconductor-superconductor structure in proximity to an altermagnet which realizes end Majorana zero modes (MZMs) with vanishing net magnetization. Additionally, a coexisting Zeeman term provides a tuning knob to distinguish topological and trivial zero modes. We then propose 2D altermagnetic platforms that can realize chiral Majorana fermions or higher order corner MZMs. Our work paves the way towards realizing Majorana boundary states with an alternative source of time-reversal breaking and zero net magnetization., Comment: Submitted to journal, 5 figures

The Bloch electron energy spectrum of a crystalline solid is determined by the underlying lattice structure at the atomic level. In a 2-dimensional (2d) crystal it is possible to impose a superlattice with nanometer-scale periodicity, allowing to tune the fundamental Bloch electron spectrum, and enabling novel physical properties which are not accessible in the original crystal. In recent years, a top-down approach for creating 2d superlattices on monolayer graphene by means of nanopatterned electric gates has been studied, which allows the formation of isolated energy bands and Hofstadter Butterfly physics in quantizing magnetic fields. Within this approach, however, evidence of electron correlations which drive many problems at the forefront of physics research remains to be uncovered. In this work we demonstrate signatures of a correlated insulator phase in Bernal-stacked bilayer graphene (BLG) modulated by a gate-defined superlattice potential, manifested as a set of resistance peaks centered at carrier densities of integer multiples of a single electron per unit cell of the superlattice potential. We associate the correlated insulator phase to the formation of flat energy bands due to the superlattice potential combined with inversion symmetry breaking. Inducing correlated electron phases with nanopatterning defined electric gates paves the way to custom-designed superlattices with arbitrary geometries and symmetries for studying band structure engineering and strongly correlated electrons in 2d materials.

This paper proposes an exploration technique for multi-agent reinforcement learning (MARL) with graph-based communication among agents. We assume the individual rewards received by the agents are independent of the actions by the other agents, while their policies are coupled. In the proposed framework, neighbouring agents collaborate to estimate the uncertainty about the state-action space in order to execute more efficient explorative behaviour. Different from existing works, the proposed algorithm does not require counting mechanisms and can be applied to continuous-state environments without requiring complex conversion techniques. Moreover, the proposed scheme allows agents to communicate in a fully decentralized manner with minimal information exchange. And for continuous-state scenarios, each agent needs to exchange only a single parameter vector. The performance of the algorithm is verified with theoretical results for discrete-state scenarios and with experiments for continuous ones.

In this work we derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. Agents employ the recently proposed ACTC (adapt-compress-then-combine) diffusion strategy, where the signals exchanged locally by neighboring agents are encoded with randomized differential compression operators. We provide a detailed characterization of the mean-square estimation error, which is shown to comprise a term related to the error that agents would achieve without communication constraints, plus a term arising from compression. The analysis reveals quantitative relationships between the compression loss and fundamental attributes of the distributed regression problem, in particular, the stochastic approximation error caused by the gradient noise and the network topology (through the Perron eigenvector). We show that knowledge of such relationships is critical to allocate optimally the communication resources across the agents, taking into account their individual attributes, such as the quality of their data or their degree of centrality in the network topology. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents. Illustrative examples show that a significant performance improvement, as compared to a blind (i.e., uniform) resource allocation, can be achieved by optimizing the allocation by means of the provided mean-square-error formulas., Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

The vulnerability of machine learning models to adversarial attacks has been attracting considerable attention in recent years. Most existing studies focus on the behavior of stand-alone single-agent learners. In comparison, this work studies adversarial training over graphs, where individual agents are subjected to perturbations of varied strength levels across space. It is expected that interactions by linked agents, and the heterogeneity of the attack models that are possible over the graph, can help enhance robustness in view of the coordination power of the group. Using a min-max formulation of diffusion learning, we develop a decentralized adversarial training framework for multi-agent systems. We analyze the convergence properties of the proposed scheme for both convex and non-convex environments, and illustrate the enhanced robustness to adversarial attacks., Comment: arXiv admin note: text overlap with arXiv:2303.01936

We consider the problem of information aggregation in federated decision making, where a group of agents collaborate to infer the underlying state of nature without sharing their private data with the central processor or each other. We analyze the non-Bayesian social learning strategy in which agents incorporate their individual observations into their opinions (i.e., soft-decisions) with Bayes rule, and the central processor aggregates these opinions by arithmetic or geometric averaging. Building on our previous work, we establish that both pooling strategies result in asymptotic normality characterization of the system, which, for instance, can be utilized to derive approximate expressions for the error probability. We verify the theoretical findings with simulations and compare both strategies., Comment: Accepted to IEEE Statistical Signal Processing (SSP) Workshop 2023

This work focuses on adversarial learning over graphs. We propose a general adversarial training framework for multi-agent systems using diffusion learning. We analyze the convergence properties of the proposed scheme for convex optimization problems, and illustrate its enhanced robustness to adversarial attacks.

Most works on multi-agent reinforcement learning focus on scenarios where the state of the environment is fully observable. In this work, we consider a cooperative policy evaluation task in which agents are not assumed to observe the environment state directly. Instead, agents can only have access to noisy observations and to belief vectors. It is well-known that finding global posterior distributions under multi-agent settings is generally NP-hard. As a remedy, we propose a fully decentralized belief forming strategy that relies on individual updates and on localized interactions over a communication network. In addition to the exchange of the beliefs, agents exploit the communication network by exchanging value function parameter estimates as well. We analytically show that the proposed strategy allows information to diffuse over the network, which in turn allows the agents' parameters to have a bounded difference with a centralized baseline. A multi-sensor target tracking application is considered in the simulations., Comment: Accepted for publication in IEEE Open Journal of Control Systems, Special Section: Intersection of Machine Learning with Control

This work examines a social learning problem, where dispersed agents connected through a network topology interact locally to form their opinions (beliefs) as regards certain hypotheses of interest. These opinions evolve over time, since the agents collect observations from the environment, and update their current beliefs by accounting for: their past beliefs, the innovation contained in the new data, and the beliefs received from the neighbors. The distinguishing feature of the present work is that agents are constrained to share opinions regarding only a single hypothesis. We devise a novel learning strategy where each agent forms a valid belief by completing the partial beliefs received from its neighbors. This completion is performed by exploiting the knowledge accumulated in the past beliefs, thanks to a principled memory-aware rule inspired by a Bayesian criterion. The analysis allows us to characterize the role of memory in social learning under partial information sharing, revealing novel and nontrivial learning dynamics. Surprisingly, we establish that the standard classification rule based on selecting the maximum belief is not optimal under partial information sharing, while there exists a consistent threshold-based decision rule that allows each agent to classify correctly the hypothesis of interest. We also show that the proposed strategy outperforms previously considered schemes, highlighting that the introduction of memory in the social learning algorithm is critical to overcome the limitations arising from sharing partial information.

We study the privatization of distributed learning and optimization strategies. We focus on differential privacy schemes and study their effect on performance. We show that the popular additive random perturbation scheme degrades performance because it is not well-tuned to the graph structure. For this reason, we exploit two alternative graph-homomorphic constructions and show that they improve performance while guaranteeing privacy. Moreover, contrary to most earlier studies, the gradient of the risks is not assumed to be bounded (a condition that rarely holds in practice; e.g., quadratic risk). We avoid this condition and still devise a differentially private scheme with high probability. We examine optimization and learning scenarios and illustrate the theoretical findings through simulations.

Bernal stacked bilayer graphene subject to a superlattice potential can realize topological and stacked flat bands [1]. In the present work, we extend the study of a superlattice potential on graphene heterostructures to trilayer and quadrilayer graphene. Comparing Bernal- and chirally-stacked multilayers reveals that the latter are more suitable for realizing stacks of many flat bands. On the other hand, Bernal-stacked graphene heterostructures can realize topological flat bands. Imposing two simultaneous superlattice potentials enhances the viability of both regimes., Comment: 10 Figures

This work studies networked agents cooperating to track a dynamical state of nature under partial information. The proposed algorithm is a distributed Bayesian filtering algorithm for finite-state hidden Markov models (HMMs). It can be used for sequential state estimation tasks, as well as for modeling opinion formation over social networks under dynamic environments. We show that the disagreement with the optimal centralized solution is asymptotically bounded for the class of geometrically ergodic state transition models, which includes rapidly changing models. We also derive recursions for calculating the probability of error and establish convergence under Gaussian observation models. Simulations are provided to illustrate the theory and to compare against alternative approaches., Comment: Submitted for publication

The adaptive social learning paradigm helps model how networked agents are able to form opinions on a state of nature and track its drifts in a changing environment. In this framework, the agents repeatedly update their beliefs based on private observations and exchange the beliefs with their neighbors. In this work, it is shown how the sequence of publicly exchanged beliefs over time allows users to discover rich information about the underlying network topology and about the flow of information over the graph. In particular, it is shown that it is possible (i) to identify the influence of each individual agent to the objective of truth learning, (ii) to discover how well-informed each agent is, (iii) to quantify the pairwise influences between agents, and (iv) to learn the underlying network topology. The algorithm derived herein is also able to work under non-stationary environments where either the true state of nature or the graph topology are allowed to drift over time. We apply the proposed algorithm to different subnetworks of Twitter users, and identify the most influential and central agents by using their public tweets (posts).

Doped strong topological insulators are one of the most promising candidates to realize a fully gapped three-dimensional topological superconductor (TSC). In this letter, we revisit this system and reveal a possibility for higher-order topology which was previously missed. We find that over a finite-range of doping, the Fu-Berg superconducting pairing can give rise to both Majorana surface states, and nodal hinge states. Interestingly, we observe the coexistence of surface and hinge modes in the superconducting state only when there are both bulk and surface Fermi-surfaces in the normal state. Also, we find that the hinge modes can appear for normal states consisting of doped strong or weak topological insulators. In summary, this work may allow for the discovery of superconducting hinge modes in a well explored class of materials, i.e., doped strong or weak topological insulators., Comment: 4 figures

We study the generation of dependent random numbers in a distributed fashion in order to enable privatized distributed learning by networked agents. We propose a method that we refer to as local graph-homomorphic processing; it relies on the construction of particular noises over the edges to ensure a certain level of differential privacy. We show that the added noise does not affect the performance of the learned model. This is a significant improvement to previous works on differential privacy for distributed algorithms, where the noise was added in a less structured manner without respecting the graph topology and has often led to performance deterioration. We illustrate the theoretical results by considering a linear regression problem over a network of agents.

The article reviews significant advances in networked signal and information processing, which have enabled in the last 25 years extending decision making and inference, optimization, control, and learning to the increasingly ubiquitous environments of distributed agents. As these interacting agents cooperate, new collective behaviors emerge from local decisions and actions. Moreover, and significantly, theory and applications show that networked agents, through cooperation and sharing, are able to match the performance of cloud or federated solutions, while offering the potential for improved privacy, increasing resilience, and saving resources.