Search

The Circular Electron-Positron Collider (CEPC) can also work as a powerful and excellent synchrotron light source, which can generate high-quality synchrotron radiation. This synchrotron radiation has potential advantages in the medical field, with a broad spectrum, with energies ranging from visible light to x-rays used in conventional radiotherapy, up to several MeV. FLASH radiotherapy is one of the most advanced radiotherapy modalities. It is a radiotherapy method that uses ultra-high dose rate irradiation to achieve the treatment dose in an instant; the ultra-high dose rate used is generally greater than 40 Gy/s, and this type of radiotherapy can protect normal tissues well. In this paper, the treatment effect of CEPC synchrotron radiation for FLASH radiotherapy was evaluated by simulation. First, Geant4 simulation was used to build a synchrotron radiation radiotherapy beamline station, and then the dose rate that CEPC can produce was calculated. Then, a physicochemical model of radiotherapy response kinetics was established, and a large number of radiotherapy experimental data were comprehensively used to fit and determine the functional relationship between the treatment effect, dose rate and dose. Finally, the macroscopic treatment effect of FLASH radiotherapy was predicted using CEPC synchrotron radiation light through the dose rate and the above-mentioned functional relationship. The results show that CEPC synchrotron radiation beam is one of the best beams for FLASH radiotherapy., Comment: 18 pages, 7 figures

Large-scale generative models, such as text-to-image diffusion models, have garnered widespread attention across diverse domains due to their creative and high-fidelity image generation. Nonetheless, existing large-scale diffusion models are confined to generating images of up to 1K resolution, which is far from meeting the demands of contemporary commercial applications. Directly sampling higher-resolution images often yields results marred by artifacts such as object repetition and distorted shapes. Addressing the aforementioned issues typically necessitates training or fine-tuning models on higher-resolution datasets. However, this poses a formidable challenge due to the difficulty in collecting large-scale high-resolution images and substantial computational resources. While several preceding works have proposed alternatives to bypass the cumbersome training process, they often fail to produce convincing results. In this work, we probe the generative ability of diffusion models at higher resolution beyond their original capability and propose a novel progressive approach that fully utilizes generated low-resolution images to guide the generation of higher-resolution images. Our method obviates the need for additional training or fine-tuning which significantly lowers the burden of computational costs. Extensive experiments and results validate the efficiency and efficacy of our method. Project page: https://yhyun225.github.io/DiffuseHigh/, Comment: Project page: https://yhyun225.github.io/DiffuseHigh/

Temporal difference (TD) learning algorithms with neural network function parameterization have well-established empirical success in many practical large-scale reinforcement learning tasks. However, theoretical understanding of these algorithms remains challenging due to the nonlinearity of the action-value approximation. In this paper, we develop an improved non-asymptotic analysis of the neural TD method with a general $L$-layer neural network. New proof techniques are developed and an improved new $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is derived. To our best knowledge, this is the first finite-time analysis of neural TD that achieves an $\tilde{\mathcal{O}}(\epsilon^{-1})$ complexity under the Markovian sampling, as opposed to the best known $\tilde{\mathcal{O}}(\epsilon^{-2})$ complexity in the existing literature.

This paper considers smooth strongly convex and strongly concave (SC-SC) stochastic saddle point (SSP) problems. Suppose there is an arbitrary oracle that in expectation returns an $\epsilon$-solution in the sense of certain gaps, which can be the duality gap or its weaker variants. We propose a general PB-SSP framework to guarantee an $\epsilon$ small duality gap solution with high probability via only $\mathcal{O}\big(\log \frac{1}{p}\cdot\text{poly}(\log \kappa)\big)$ calls of this oracle, where $p\in(0,1)$ is the confidence level and $\kappa$ is the condition number. When applied to the sample average approximation (SAA) oracle, in addition to equipping the solution with high probability, our approach even improves the sample complexity by a factor of $\text{poly}(\kappa)$, since the high-probability argument enables us to circumvent some key difficulties of the uniform stability analysis of SAA.

This paper considers the discounted criterion of nonzero-sum decentralized stochastic games with prospect players. The state and action spaces are finite. The state transition probability is nonstationary. Each player independently controls their own Markov chain. The subjective behavior of players is described by the prospect theory (PT). Compared to the average criterion of stochastic games under PT studied firstly in 2018, we are concerned with the time value of utility, i.e., the utility should be discounted in the future. Since PT distorts the probability, the optimality equation that plays a significant role in proving the existence of equilibrium does not exist. On the other hand, the games change into Markov decision processes (MDPs) with nonstationary payoff function when fixing others' stationary Markov strategies, then the occupation measure and the linear programming of stationary MDPs are no longer suitable. Therefore, we explore a new technique by constructing the marginal distribution on the state-action pairs at any time, and establish the existence of Nash equilibrium. When the initial state is given, there exists a Markov Nash equilibrium. Furthermore, this novel technique can be extended to the finite horizon criterion. Then, we present an algorithm to find a Markov $\varepsilon$-equilibrium. Finally, the model is applied to a noncooperative stochastic game among prosumers who can produce and consume energy in the smart grid, and we give some simulation results.

We investigate Bregman proximal gradient (BPG) methods for solving nonconvex composite stochastic optimization problems. Instead of the standard gradient Lipschitz continuity (GLC) assumption, the objective function only satisfies a smooth-adaptability assumption w.r.t. some kernel function. An in-depth analysis of the stationarity measure is made in this paper, where we reveal an interesting fact that the widely adopted Bregman proximal gradient mapping in the existing works may not correctly depict the near stationarity of the solutions. To resolve this issue, a new Bregman proximal gradient mapping is proposed and analyzed in this paper. Second, a thorough analysis is made on the sample complexities of the stochastic Bregman proximal gradient methods under both the old and the newly proposed gradient mappings are analyzed. Note that the absence of GLC disables the standard analysis of the stochastic variance reduction techniques, existing stochastic BPG methods only obtain an $O(\epsilon^{-2})$ sample complexity under the old gradient mapping, we show that such a limitation in the existing analyses mainly comes from the insufficient exploitation of the kernel's properties. By proposing a new kernel-conditioning regularity assumption on the kernel, we show that a simple epoch bound mechanism is enough to enable all the existing variance reduction techniques for stochastic BPG methods. Combined with a novel probabilistic argument, we show that there is a high probability event conditioning on which $O(\sqrt{n}\epsilon^{-1})$ sample complexity for the finite-sum stochastic setting can be derived for the old gradient mapping. Moreover, with a novel adaptive step size control mechanism, we also show an $\tilde{O}(\sqrt{n}L_h(\mathcal{X}_\epsilon)\epsilon^{-1})$ complexity under the new gradient mapping.

In a cost sharing problem on a weighted undirected graph, all other nodes want to connect to the source node for some service. Each edge has a cost denoted by a weight and all the connected nodes should share the total cost for the connectivity. The goal of the existing solutions (e.g. folk solution and cycle-complete solution) is to design cost sharing rules with nice properties, e.g. budget balance and cost monotonicity. However, they did not consider the cases that each non-source node has a budget which is the maximum it can pay for its cost share and may cut its adjacent edges to reduce its cost share. In this paper, we design two cost sharing mechanisms taking into account the nodes' budgets and incentivizing all nodes to report all their adjacent edges so that we can minimize the total cost for the connectivity., Comment: arXiv admin note: substantial text overlap with arXiv:2201.05976

We consider a cost sharing problem on a weighted directed acyclic graph (DAG) with a source node to which all the other nodes want to connect. The cost (weight) of each edge is private information reported by multiple contractors, and among them, only one contractor is selected as the builder. All the nodes except for the source need to share the total cost of the used edges. However, they may block others' connections to the source by strategically cutting their outgoing edges to reduce their cost share, which may increase the total cost of connectivity. To minimize the total cost of connectivity, we design a cost sharing mechanism to incentivize each node to offer all its outgoing edges and each contractor to report all the edges' weights truthfully, and show the properties of the proposed mechanism. In addition, our mechanism outperforms the two benchmark mechanisms.

Multi-cellular robot design aims to create robots comprised of numerous cells that can be efficiently controlled to perform diverse tasks. Previous research has demonstrated the ability to generate robots for various tasks, but these approaches often optimize robots directly in the vast design space, resulting in robots with complicated morphologies that are hard to control. In response, this paper presents a novel coarse-to-fine method for designing multi-cellular robots. Initially, this strategy seeks optimal coarse-grained robots and progressively refines them. To mitigate the challenge of determining the precise refinement juncture during the coarse-to-fine transition, we introduce the Hyperbolic Embeddings for Robot Design (HERD) framework. HERD unifies robots of various granularity within a shared hyperbolic space and leverages a refined Cross-Entropy Method for optimization. This framework enables our method to autonomously identify areas of exploration in hyperbolic space and concentrate on regions demonstrating promise. Finally, the extensive empirical studies on various challenging tasks sourced from EvoGym show our approach's superior efficiency and generalization capability.

Synthesizing physically plausible human motions in 3D scenes is a challenging problem. Kinematics-based methods cannot avoid inherent artifacts (e.g., penetration and foot skating) due to the lack of physical constraints. Meanwhile, existing physics-based methods cannot generalize to multi-object scenarios since the policy trained with reinforcement learning has limited modeling capacity. In this work, we present a framework that enables physically simulated characters to perform long-term interaction tasks in diverse, cluttered, and unseen scenes. The key idea is to decompose human-scene interactions into two fundamental processes, Interacting and Navigating, which motivates us to construct two reusable Controller, i.e., InterCon and NavCon. Specifically, InterCon contains two complementary policies that enable characters to enter and leave the interacting state (e.g., sitting on a chair and getting up). To generate interaction with objects at different places, we further design NavCon, a trajectory following policy, to keep characters' locomotion in the free space of 3D scenes. Benefiting from the divide and conquer strategy, we can train the policies in simple environments and generalize to complex multi-object scenes. Experimental results demonstrate that our framework can synthesize physically plausible long-term human motions in complex 3D scenes. Code will be publicly released at https://github.com/liangpan99/InterScene.

We study the optimal lower and upper complexity bounds for finding approximate solutions to the composite problem $\min_x\ f(x)+h(Ax-b)$, where $f$ is smooth and $h$ is convex. Given access to the proximal operator of $h$, for strongly convex, convex, and nonconvex $f$, we design efficient first order algorithms with complexities $\tilde{O}\left(\kappa_A\sqrt{\kappa_f}\log\left(1/{\epsilon}\right)\right)$, $\tilde{O}\left(\kappa_A\sqrt{L_f}D/\sqrt{\epsilon}\right)$, and $\tilde{O}\left(\kappa_A L_f\Delta/\epsilon^2\right)$, respectively. Here, $\kappa_A$ is the condition number of the matrix $A$ in the composition, $L_f$ is the smoothness constant of $f$, and $\kappa_f$ is the condition number of $f$ in the strongly convex case. $D$ is the initial point distance and $\Delta$ is the initial function value gap. Tight lower complexity bounds for the three cases are also derived and they match the upper bounds up to logarithmic factors, thereby demonstrating the optimality of both the upper and lower bounds proposed in this paper.

Robot design aims at learning to create robots that can be easily controlled and perform tasks efficiently. Previous works on robot design have proven its ability to generate robots for various tasks. However, these works searched the robots directly from the vast design space and ignored common structures, resulting in abnormal robots and poor performance. To tackle this problem, we propose a Symmetry-Aware Robot Design (SARD) framework that exploits the structure of the design space by incorporating symmetry searching into the robot design process. Specifically, we represent symmetries with the subgroups of the dihedral group and search for the optimal symmetry in structured subgroups. Then robots are designed under the searched symmetry. In this way, SARD can design efficient symmetric robots while covering the original design space, which is theoretically analyzed. We further empirically evaluate SARD on various tasks, and the results show its superior efficiency and generalizability., Comment: The Fortieth International Conference on Machine Learning (ICML 2023)

Recent offline meta-reinforcement learning (meta-RL) methods typically utilize task-dependent behavior policies (e.g., training RL agents on each individual task) to collect a multi-task dataset. However, these methods always require extra information for fast adaptation, such as offline context for testing tasks. To address this problem, we first formally characterize a unique challenge in offline meta-RL: transition-reward distribution shift between offline datasets and online adaptation. Our theory finds that out-of-distribution adaptation episodes may lead to unreliable policy evaluation and that online adaptation with in-distribution episodes can ensure adaptation performance guarantee. Based on these theoretical insights, we propose a novel adaptation framework, called In-Distribution online Adaptation with uncertainty Quantification (IDAQ), which generates in-distribution context using a given uncertainty quantification and performs effective task belief inference to address new tasks. We find a return-based uncertainty quantification for IDAQ that performs effectively. Experiments show that IDAQ achieves state-of-the-art performance on the Meta-World ML1 benchmark compared to baselines with/without offline adaptation.

This paper studies the complexity of finding approximate stationary points for nonconvex-strongly-concave (NC-SC) smooth saddle point problem: $\min_x\max_yf(x,y)$. Under first-order smoothness conditions, and denote $\kappa_y$ the condition number for the inner strongly-concave problem, existing literature shows that the optimal complexity for first-order methods to obtain an $\epsilon$-stationary point is $\tilde{\Theta}(\sqrt{\kappa_y}\epsilon^{-2})$. However, when the objective function further satisfies certain high-order smoothness conditions, both the iteration complexity upper and lower bounds can be improved for first-order methods. In detail, given the additional assumption that the function $\Phi(x):=\max_y f(x,y)$ has Lipschitz continuous Hessian, we derive a first-order method with an improved $\tilde{O}(\sqrt{\kappa_y}\epsilon^{-7/4})$ complexity, by designing an inexact proximal point variant of the "Convex Until Proven Guilty" method that is compatible with the APPA acceleration technique for saddle point problems. In addition to an upper bound, we also construct an $\Omega(\sqrt{\kappa_y}\epsilon^{-12/7})$ lower bound for NC-SC problems with second-order smoothness. As a result, the complexity of our proposed algorithm is near-optimal and it almost matches the lower bound barring a negligible factor of $\tilde{O}(\epsilon^{-1/28})$.

We consider a cost sharing problem on a weighted undirected graph, where all the nodes want to connect to a special node called source, and they need to share the total cost (weights) of the used edges. Each node except for the source has a private valuation of the connection, and it may block others' connections by strategically cutting its adjacent edges to reduce its cost share, which may increase the total cost. We aim to design mechanisms to prevent the nodes from misreporting their valuations and cutting their adjacent edges. We first show that it is impossible for such a mechanism to further satisfy budget balance (cover the total cost) and efficiency (maximize social welfare). Then, we design two feasible cost sharing mechanisms that incentivize each node to offer all its adjacent edges and truthfully report its valuation, and also satisfy either budget balance or efficiency.

In this paper, a Gauss-Newton Temporal Difference (GNTD) learning method is proposed to solve the Q-learning problem with nonlinear function approximation. In each iteration, our method takes one Gauss-Newton (GN) step to optimize a variant of Mean-Squared Bellman Error (MSBE), where target networks are adopted to avoid double sampling. Inexact GN steps are analyzed so that one can safely and efficiently compute the GN updates by cheap matrix iterations. Under mild conditions, non-asymptotic finite-sample convergence to the globally optimal Q function is derived for various nonlinear function approximations. In particular, for neural network parameterization with relu activation, GNTD achieves an improved sample complexity of $\tilde{\mathcal{O}}(\varepsilon^{-1})$, as opposed to the $\mathcal{\mathcal{O}}(\varepsilon^{-2})$ sample complexity of the existing neural TD methods. An $\tilde{\mathcal{O}}(\varepsilon^{-1.5})$ sample complexity of GNTD is also established for general smooth function approximations. We validate our method via extensive experiments in several RL benchmarks, where GNTD exhibits both higher rewards and faster convergence than TD-type methods.

Rejuvenating galaxies are unusual galaxies that fully quench and then subsequently experience a "rejuvenation" event to become star-forming once more. Rejuvenation rates vary substantially in models of galaxy formation: 10%-70% of massive galaxies are expected to experience rejuvenation by z = 0. Measuring the rate of rejuvenation is therefore important for calibrating the strength of star formation feedback mechanisms. However, these observations are challenging because rejuvenating systems blend in with normal star-forming galaxies in broadband photometry. In this paper, we use the galaxy spectral energy distribution (SED)-fitting code Prospector to search for observational markers that distinguish normal star-forming galaxies from rejuvenating galaxies. We find that rejuvenating galaxies have smaller Balmer absorption line equivalent widths (EWs) than star-forming galaxies. This is analogous to the well-known "K + A" or post-starburst galaxies, which have strong Balmer absorption due to A-stars dominating the light: in this case, rejuvenating systems have a lack of A-stars, instead resembling "O - A" systems. We find star-forming galaxies that have H$\beta$, H$\gamma$, and/or H$\delta$ absorption EWs $\lesssim 3${\AA} corresponds to a highly pure selection of rejuvenating systems. Interestingly, while this technique is highly effective at identifying mild rejuvenation, "strongly" rejuvenating systems remain nearly indistinguishable from star-forming galaxies due to the well-known stellar outshining effect. We conclude that measuring Balmer absorption line EWs in star-forming galaxy populations is an efficient method to identify rejuvenating populations, and discuss several techniques to either remove or resolve the nebular emission which typically lies on top of these absorption lines., Comment: Published in ApJ, 16 pages, 9 figures

In this paper, we propose a unified primal-dual algorithm framework based on the augmented Lagrangian function for composite convex problems with conic inequality constraints. The new framework is highly versatile. First, it not only covers many existing algorithms such as PDHG, Chambolle-Pock (CP), GDA, OGDA and linearized ALM, but also guides us to design a new efficient algorithm called Simi-OGDA (SOGDA). Second, it enables us to study the role of the augmented penalty term in the convergence analysis. Interestingly, a properly selected penalty not only improves the numerical performance of the above methods, but also theoretically enables the convergence of algorithms like PDHG and SOGDA. Under properly designed step sizes and penalty term, our unified framework preserves the $\mathcal{O}(1/N)$ ergodic convergence while not requiring any prior knowledge about the magnitude of the optimal Lagrangian multiplier. Linear convergence rate for affine equality constrained problem is also obtained given appropriate conditions. Finally, numerical experiments on linear programming, $\ell_1$ minimization problem, and multi-block basis pursuit problem demonstrate the efficiency of our methods.

As an important framework for safe Reinforcement Learning, the Constrained Markov Decision Process (CMDP) has been extensively studied in the recent literature. However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound. In this paper, we focus on solving the CMDP problems where only offline data are available. By adopting the concept of the single-policy concentrability coefficient $C^*$, we establish an $\Omega\left(\frac{\min\left\{|\mathcal{S}||\mathcal{A}|,|\mathcal{S}|+I\right\} C^*}{(1-\gamma)^3\epsilon^2}\right)$ sample complexity lower bound for the offline CMDP problem, where $I$ stands for the number of constraints. By introducing a simple but novel deviation control mechanism, we propose a near-optimal primal-dual learning algorithm called DPDL. This algorithm provably guarantees zero constraint violation and its sample complexity matches the above lower bound except for an $\tilde{\mathcal{O}}((1-\gamma)^{-1})$ factor. Comprehensive discussion on how to deal with the unknown constant $C^*$ and the potential asynchronous structure on the offline dataset are also included.

We consider a cost sharing problem to connect all nodes in a weighted undirected graph, where the weight of each edge represents the cost to use the edge for the connectivity and the cost has to be shared among all connected nodes. There is one node called the source to which all the other nodes want to connect and it does not share the costs of the connectivity. As a node may need to go through other nodes to reach the source, the intermediate nodes may behave strategically to block the connection by cutting the edges adjacent to them. To prevent such strategical behavior, we design cost sharing mechanisms to incentivize all nodes not to cut any edge so that we can minimize the total cost for connecting all the nodes., Comment: The content are totally reorganized in a new paper

We consider the convex-concave saddle point problem $\min_{\mathbf{x}}\max_{\mathbf{y}}\Phi(\mathbf{x},\mathbf{y})$, where the decision variables $\mathbf{x}$ and/or $\mathbf{y}$ subject to a multi-block structure and affine coupling constraints, and $\Phi(\mathbf{x},\mathbf{y})$ possesses certain separable structure. Although the minimization counterpart of such problem has been widely studied under the topics of ADMM, this minimax problem is rarely investigated. In this paper, a convenient notion of $\epsilon$-saddle point is proposed, under which the convergence rate of several proposed algorithms are analyzed. When only one of $\mathbf{x}$ and $\mathbf{y}$ has multiple blocks and affine constraint, several natural extensions of ADMM are proposed to solve the problem. Depending on the number of blocks and the level of smoothness, $\mathcal{O}(1/T)$ or $\mathcal{O}(1/\sqrt{T})$ convergence rates are derived for our algorithms. When both $\mathbf{x}$ and $\mathbf{y}$ have multiple blocks and affine constraints, a new algorithm called ExtraGradient Method of Multipliers (EGMM) is proposed. Under desirable smoothness condition, an $\mathcal{O}(1/T)$ rate of convergence can be guaranteed regardless of the number of blocks in $\mathbf{x}$ and $\mathbf{y}$. In depth comparison between EGMM (fully primal-dual method) and ADMM (approximate dual method) is made over the multi-block optimization problems to illustrate the advantage of the EGMM., Comment: 25 pages

Reinforcement learning is a framework for interactive decision-making with incentives sequentially revealed across time without a system dynamics model. Due to its scaling to continuous spaces, we focus on policy search where one iteratively improves a parameterized policy with stochastic policy gradient (PG) updates. In tabular Markov Decision Problems (MDPs), under persistent exploration and suitable parameterization, global optimality may be obtained. By contrast, in continuous space, the non-convexity poses a pathological challenge as evidenced by existing convergence results being mostly limited to stationarity or arbitrary local extrema. To close this gap, we step towards persistent exploration in continuous space through policy parameterizations defined by distributions of heavier tails defined by tail-index parameter alpha, which increases the likelihood of jumping in state space. Doing so invalidates smoothness conditions of the score function common to PG. Thus, we establish how the convergence rate to stationarity depends on the policy's tail index alpha, a Holder continuity parameter, integrability conditions, and an exploration tolerance parameter introduced here for the first time. Further, we characterize the dependence of the set of local maxima on the tail index through an exit and transition time analysis of a suitably defined Markov chain, identifying that policies associated with Levy Processes of a heavier tail converge to wider peaks. This phenomenon yields improved stability to perturbations in supervised learning, which we corroborate also manifests in improved performance of policy search, especially when myopic and farsighted incentives are misaligned.

We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a \emph{general utility}. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. % We derive the {\bf D}ecentralized {\bf S}hadow Reward {\bf A}ctor-{\bf C}ritic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the "shadow reward". DSAC converges to $\epsilon$-stationarity in $\mathcal{O}(1/\epsilon^{2.5})$ (Theorem \ref{theorem:final}) or faster $\mathcal{O}(1/\epsilon^{2})$ (Corollary \ref{corollary:communication}) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy (Corollary \ref{corollary:global}). Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.

We study a finite-horizon two-person zero-sum risk-sensitive stochastic game for continuous-time Markov chains and Borel state and action spaces, in which payoff rates, transition rates and terminal reward functions are allowed to be unbounded from below and from above and the policies can be history-dependent. Under suitable conditions, we establish the existence of a solution to the corresponding Shapley equation (SE) by an approximation technique. Then, by the SE and the extension of the Dynkin's formula, we prove the existence of a Nash equilibrium and verify that the value of the stochastic game is the unique solution to the SE. Moreover, we develop a value iteration-type algorithm for approaching to the value of the stochastic game. The convergence of the algorithm is proved by a special contraction operator in our risk-sensitive stochastic game. Finally, we demonstrate our main results by two examples.

Policy gradient (PG) gives rise to a rich class of reinforcement learning (RL) methods. Recently, there has been an emerging trend to accelerate the existing PG methods such as REINFORCE by the \emph{variance reduction} techniques. However, all existing variance-reduced PG methods heavily rely on an uncheckable importance weight assumption made for every single iteration of the algorithms. In this paper, a simple gradient truncation mechanism is proposed to address this issue. Moreover, we design a Truncated Stochastic Incremental Variance-Reduced Policy Gradient (TSIVR-PG) method, which is able to maximize not only a cumulative sum of rewards but also a general utility function over a policy's long-term visiting distribution. We show an $\tilde{\mathcal{O}}(\epsilon^{-3})$ sample complexity for TSIVR-PG to find an $\epsilon$-stationary policy. By assuming the overparameterizaiton of policy and exploiting the hidden convexity of the problem, we further show that TSIVR-PG converges to global $\epsilon$-optimal policy with $\tilde{\mathcal{O}}(\epsilon^{-2})$ samples.

First-order algorithms have been popular for solving convex and non-convex optimization problems. A key assumption for the majority of these algorithms is that the gradient of the objective function is globally Lipschitz continuous, but many contemporary problems such as tensor decomposition fail to satisfy such an assumption. This paper develops a sequential local optimization (SLO) framework of first-order algorithms that can effectively optimize problems without Lipschitz gradient. Operating on the assumption that the gradients are {\it locally} Lipschitz continuous over any compact set, the proposed framework carefully restricts the distance between two successive iterates. We show that the proposed framework can easily adapt to existing first-order methods such as gradient descent (GD), normalized gradient descent (NGD), accelerated gradient descent (AGD), as well as GD with Armijo line search. Remarkably, the latter algorithm is totally parameter-free and do not even require the knowledge of local Lipschitz constants. We show that for the proposed algorithms to achieve gradient error bound of $\|\nabla f(x)\|^2\le \epsilon$, it requires at most $\mathcal{O}(\frac{1}{\epsilon}\times \mathcal{L}(Y))$ total access to the gradient oracle, where $\mathcal{L}(Y)$ characterizes how the local Lipschitz constants grow with the size of a given set $Y$. Moreover, we show that the variant of AGD improves the dependency on both $\epsilon$ and the growth function $\mathcal{L}(Y)$. The proposed algorithms complement the existing Bregman Proximal Gradient (BPG) algorithm, because they do not require the global information about problem structure to construct and solve Bregman proximal mappings., Comment: Accepted by Informs Journal on Optimization

In this paper, we propose a cubic regularized Newton (CRN) method for solving convex-concave saddle point problems (SPP). At each iteration, a cubic regularized saddle point subproblem is constructed and solved, which provides a search direction for the iterate. With properly chosen stepsizes, the method is shown to converge to the saddle point with global linear and local superlinear convergence rates, if the saddle point function is gradient Lipschitz and strongly-convex-strongly-concave. In the case that the function is merely convex-concave, we propose a homotopy continuation (or path-following) method. Under a Lipschitz-type error bound condition, we present an iteration complexity bound of $\mathcal{O}\left(\ln \left(1/\epsilon\right)\right)$ to reach an $\epsilon$-solution through a homotopy continuation approach, and the iteration complexity bound becomes $\mathcal{O}\left(\left(1/\epsilon\right)^{\frac{1-\theta}{\theta^2}}\right)$ under a H\"{o}lderian-type error bound condition involving a parameter $\theta$ ($0<\theta<1$)., Comment: 25 pages, 4 figures

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general concave utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases. Such generality invalidates the Bellman equation. As this means that dynamic programming no longer works, we focus on direct policy search. Analogously to the Policy Gradient Theorem \cite{sutton2000policy} available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the parametrized policy gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function. We develop a variational Monte Carlo gradient estimation algorithm to compute the policy gradient based on sample paths. We prove that the variational policy gradient scheme converges globally to the optimal policy for the general objective, though the optimization problem is nonconvex. We also establish its rate of convergence of the order $O(1/t)$ by exploiting the hidden convexity of the problem, and proves that it converges exponentially when the problem admits hidden strong convexity. Our analysis applies to the standard RL problem with cumulative rewards as a special case, in which case our result improves the available convergence rate.

Future wireless communication system embraces physical-layer signal detection with high sensitivity, especially in the microwave photon level. Currently, the receiver primarily adopts the signal detection based on semi-conductor devices for signal detection, while this paper introduces high-sensitivity photon-level microwave detection based on superconducting structure. We first overview existing works on the photon-level communication in the optical spectrum as well as the microwave photon-level sensing based on superconducting structure in both theoretical and experimental perspectives, including microwave detection circuit model based on Josephson junction, microwave photon counter based on Josephson junction, and two reconstruction approaches under background noise. In addition, we characterize channel modeling based on two different microwave photon detection approaches, including the absorption barrier and the dual-path Handury Brown-Twiss (HBT) experiments, and predict the corresponding achievable rates. According to the performance prediction, it is seen that the microwave photon-level signal detection can increase the receiver sensitivity compared with the state-of-the-art standardized communication system with waveform signal reception, with gain over $10$dB., Comment: 9 pages, 13 figures

We adopt artificial $\Lambda$-type three-level system with superconducting devices for microwave signal detection, where the signal intensity reaches the level of discrete photons instead of continuous waveform. Based on the state transition principles of the three-level system, we propose a statistical model for microwave signal detection. Moreover, we investigate the achievable transmission rate and signal detection based on the statistical model. It is predicted that the proposed detection can achieve significantly higher sensitivity compared with the currently deployed 4G/5G communication system. We further characterize the received signal considering the saturation phonomenon, which reveals negligible performance degradation caused by saturation under weak received power regime., Comment: 12 pages, 18 figures

We study a generic class of decentralized algorithms in which $N$ agents jointly optimize the non-convex objective $f(u):=1/N\sum_{i=1}^{N}f_i(u)$, while only communicating with their neighbors. This class of problems has become popular in modeling many signal processing and machine learning applications, and many efficient algorithms have been proposed. However, by constructing some counter-examples, we show that when certain local Lipschitz conditions (LLC) on the local function gradient $\nabla f_i$'s are not satisfied, most of the existing decentralized algorithms diverge, even if the global Lipschitz condition (GLC) is satisfied, where the sum function $f$ has Lipschitz gradient. This observation raises an important open question: How to design decentralized algorithms when the LLC, or even the GLC, is not satisfied? To address the above question, we design a first-order algorithm called Multi-stage gradient tracking algorithm (MAGENTA), which is capable of computing stationary solutions with neither the LLC nor the GLC. In particular, we show that the proposed algorithm converges sublinearly to certain $\epsilon$-stationary solution, where the precise rate depends on various algorithmic and problem parameters. In particular, if the local function $f_i$'s are $Q$th order polynomials, then the rate becomes $\mathcal{O}(1/\epsilon^{Q-1})$. Such a rate is tight for the special case of $Q=2$ where each $f_i$ satisfies LLC. To our knowledge, this is the first attempt that studies decentralized non-convex optimization problems with neither the LLC nor the GLC., Comment: 34 pages

This paper studies the generalization bounds for the empirical saddle point (ESP) solution to stochastic saddle point (SSP) problems. For SSP with Lipschitz continuous and strongly convex-strongly concave objective functions, we establish an $\mathcal{O}(1/n)$ generalization bound by using a uniform stability argument. We also provide generalization bounds under a variety of assumptions, including the cases without strong convexity and without bounded domains. We illustrate our results in two examples: batch policy learning in Markov decision process, and mixed strategy Nash equilibrium estimation for stochastic games. In each of these examples, we show that a regularized ESP solution enjoys a near-optimal sample complexity. To the best of our knowledge, this is the first set of results on the generalization theory of ESP.

We consider minimization of composite functions of the form $f(g(x))+h(x)$, where $f$ and $h$ are convex functions (which can be nonsmooth) and $g$ is a smooth vector mapping. In addition, we assume that $g$ is the average of finite number of component mappings or the expectation over a family of random component mappings. We propose a class of stochastic variance-reduced prox-linear algorithms for solving such problems and bound their sample complexities for finding an $\epsilon$-stationary point in terms of the total number of evaluations of the component mappings and their Jacobians. When $g$ is a finite average of $N$ components, we obtain sample complexity $\mathcal{O}(N+ N^{4/5}\epsilon^{-1})$ for both mapping and Jacobian evaluations. When $g$ is a general expectation, we obtain sample complexities of $\mathcal{O}(\epsilon^{-5/2})$ and $\mathcal{O}(\epsilon^{-3/2})$ for component mappings and their Jacobians respectively. If in addition $f$ is smooth, then improved sample complexities of $\mathcal{O}(N+N^{1/2}\epsilon^{-1})$ and $\mathcal{O}(\epsilon^{-3/2})$ are derived for $g$ being a finite average and a general expectation respectively, for both component mapping and Jacobian evaluations.

We propose a novel radio-frequency (RF) receiving architecture based on micro-electro-mechanical system (MEMS) and optical coherent detection module. The architecture converts the received electrical signal into mechanical vibration through the piezoelectric effect and adopts an optical detection module to detect the mechanical vibration. We analyze the response function of piezoelectric film to an RF signal, the noise limited sensitivity of the optical detection module and the system transfer function in the frequency domain. Finally, we adopt simple on-off keying (OOK) modulation with bandwidth 1 kHz and carrier frequency 1 GHz, to numerically evaluate the detection sensitivity. The result shows that, considering the main noise sources in wireless channel and circuits, the signal detection sensitivity can reach around -160 dBm with a 50 $\Omega$ impedance. Such sensitivity significantly outperforms that of the currently deployed Long Term Evolution (LTE) system, when normalizing the transmission bandwidth also to 1 kHz., Comment: 15 pages, 16 figures, 6 tables

We study the impact of finite-size effect on continuous variable source-independent quantum random number generation. The central-limit theorem and maximum likelihood estimation theorem are used to derive the formula which could output the statistical fluctuations and determine upper bound of parameters of practical quantum random number generation. With these results, we can see the check data length and confidence probability has intense relevance to the final randomness, which can be adjusted according to the demand in implementation. Besides, other key parameters, such as sampling range size and sampling resolution, have also been considered in detail. It is found that the distribution of quantified output related with sampling range size has significant effects on the loss of final randomness due to finite-size effect. The overall results indicate that the finite-size effect should be taken into consideration for implementing the continuous variable source-independent quantum random number generation in practical., Comment: 8 pages, 8 figures

We study the estimation of risk-sensitive policies in reinforcement learning problems defined by a Markov Decision Process (MDPs) whose state and action spaces are countably finite. Prior efforts are predominately afflicted by computational challenges associated with the fact that risk-sensitive MDPs are time-inconsistent. To ameliorate this issue, we propose a new definition of risk, which we call caution, as a penalty function added to the dual objective of the linear programming (LP) formulation of reinforcement learning. The caution measures the distributional risk of a policy, which is a function of the policy's long-term state occupancy distribution. To solve this problem in an online model-free manner, we propose a stochastic variant of primal-dual method that uses Kullback-Lieber (KL) divergence as its proximal term. We establish that the number of iterations/samples required to attain approximately optimal solutions of this scheme matches tight dependencies on the cardinality of the state and action spaces, but differs in its dependence on the infinity norm of the gradient of the risk measure. Experiments demonstrate the merits of this approach for improving the reliability of reward accumulation without additional computational burdens.

In this paper, we study the lower iteration complexity bounds for finding the saddle point of a strongly convex and strongly concave saddle point problem: $\min_x\max_yF(x,y)$. We restrict the classes of algorithms in our investigation to be either pure first-order methods or methods using proximal mappings. The existing lower bound result for this type of problems is obtained via the framework of strongly monotone variational inequality problems, which corresponds to the case where the gradient Lipschitz constants ($L_x, L_y$ and $L_{xy}$) and strong convexity/concavity constants ($\mu_x$ and $\mu_y$) are uniform with respect to variables $x$ and $y$. However, specific to the min-max saddle point problem these parameters are naturally different. Therefore, one is led to finding the best possible lower iteration complexity bounds, specific to the min-max saddle point models. In this paper we present the following results. For the class of pure first-order algorithms, our lower iteration complexity bound is $\Omega\left(\sqrt{\frac{L_x}{\mu_x}+\frac{L_{xy}^2}{\mu_x\mu_y}+\frac{L_y}{\mu_y}}\cdot\ln\left(\frac{1}{\epsilon}\right)\right)$, where the term $\frac{L_{xy}^2}{\mu_x\mu_y}$ explains how the coupling influences the iteration complexity. Under several special parameter regimes, this lower bound has been achieved by corresponding optimal algorithms. However, whether or not the bound under the general parameter regime is optimal remains open. Additionally, for the special case of bilinear coupling problems, given the availability of certain proximal operators, a lower bound of $\Omega\left(\sqrt{\frac{L_{xy}^2}{\mu_x\mu_y}+1}\cdot\ln(\frac{1}{\epsilon})\right)$ is established in this paper, and optimal algorithms have already been developed in the literature.

We consider multi-level composite optimization problems where each mapping in the composition is the expectation over a family of random smooth mappings or the sum of some finite number of smooth mappings. We present a normalized proximal approximate gradient (NPAG) method where the approximate gradients are obtained via nested stochastic variance reduction. In order to find an approximate stationary point where the expected norm of its gradient mapping is less than $\epsilon$, the total sample complexity of our method is $O(\epsilon^{-3})$ in the expectation case, and $O(N+\sqrt{N}\epsilon^{-2})$ in the finite-sum case where $N$ is the total number of functions across all composition levels. In addition, the dependence of our total sample complexity on the number of composition levels is polynomial, rather than exponential as in previous work.

Standard results in stochastic convex optimization bound the number of samples that an algorithm needs to generate a point with small function value in expectation. More nuanced high probability guarantees are rare, and typically either rely on "light-tail" noise assumptions or exhibit worse sample complexity. In this work, we show that a wide class of stochastic optimization algorithms for strongly convex problems can be augmented with high confidence bounds at an overhead cost that is only logarithmic in the confidence level and polylogarithmic in the condition number. The procedure we propose, called proxBoost, is elementary and builds on two well-known ingredients: robust distance estimation and the proximal point method. We discuss consequences for both streaming (online) algorithms and offline algorithms based on empirical risk minimization., Comment: 37 pages

We consider the problem of minimizing the composition of a smooth (nonconvex) function and a smooth vector mapping, where the inner mapping is in the form of an expectation over some random variable or a finite sum. We propose a stochastic composite gradient method that employs an incremental variance-reduced estimator for both the inner vector mapping and its Jacobian. We show that this method achieves the same orders of complexity as the best known first-order methods for minimizing expected-value and finite-sum nonconvex functions, despite the additional outer composition which renders the composite gradient estimator biased. This finding enables a much broader range of applications in machine learning to benefit from the low complexity of incremental variance-reduction methods.

The cubic regularized Newton method of Nesterov and Polyak has become increasingly popular for non-convex optimization because of its capability of finding an approximate local solution with second-order guarantee. Several recent works extended this method to the setting of minimizing the average of N smooth functions by replacing the exact gradients and Hessians with subsampled approximations. It has been shown that the total Hessian sample complexity can be reduced to be sublinear in N per iteration by leveraging stochastic variance reduction techniques. We present an adaptive variance reduction scheme for subsampled Newton method with cubic regularization, and show that the expected Hessian sample complexity is O(N + N^{2/3}\epsilon^{-3/2}) for finding an (\epsilon,\epsilon^{1/2})-approximate local solution (in terms of first and second-order guarantees respectively). Moreover, we show that the same Hessian sample complexity retains with fixed sample sizes if exact gradients are used. The techniques of our analysis are different from previous works in that we do not rely on high probability bounds based on matrix concentration inequalities. Instead, we derive and utilize bounds on the 3rd and 4th order moments of the average of random matrices, which are of independent interest on their own.

The performance and efficiency of distributed training of Deep Neural Networks highly depend on the performance of gradient averaging among all participating nodes, which is bounded by the communication between nodes. There are two major strategies to reduce communication overhead: one is to hide communication by overlapping it with computation, and the other is to reduce message sizes. The first solution works well for linear neural architectures, but latest networks such as ResNet and Inception offer limited opportunity for this overlapping. Therefore, researchers have paid more attention to minimizing communication. In this paper, we present a novel gradient compression framework derived from insights of real gradient distributions, and which strikes a balance between compression ratio, accuracy, and computational overhead. Our framework has two major novel components: sparsification of gradients in the frequency domain, and a range-based floating point representation to quantize and further compress gradients frequencies. Both components are dynamic, with tunable parameters that achieve different compression ratio based on the accuracy requirement and systems' platforms, and achieve very high throughput on GPUs. We prove that our techniques guarantee the convergence with a diminishing compression ratio. Our experiments show that the proposed compression framework effectively improves the scalability of most popular neural networks on a 32 GPU cluster to the baseline of no compression, without compromising the accuracy and convergence speed., Comment: 10 pages

In this paper we present a cubic regularized Newton's method to minimize a smooth function over a Riemannian manifold. The proposed algorithm is shown to reach a second-order $\epsilon$-stationary point within $\mathcal{O}(1/\epsilon^{\frac{3}{2}})$ iterations, under the condition that the pullbacks are locally Lipschitz continuous, a condition that is shown to be satisfied if the manifold is compact. Furthermore, we present a local superlinear convergence result under some additional conditions.

Computed tomography (CT) examinations are commonly used to predict lung nodule malignancy in patients, which are shown to improve noninvasive early diagnosis of lung cancer. It remains challenging for computational approaches to achieve performance comparable to experienced radiologists. Here we present NoduleX, a systematic approach to predict lung nodule malignancy from CT data, based on deep learning convolutional neural networks (CNN). For training and validation, we analyze >1000 lung nodules in images from the LIDC/IDRI cohort. All nodules were identified and classified by four experienced thoracic radiologists who participated in the LIDC project. NoduleX achieves high accuracy for nodule malignancy classification, with an AUC of ~0.99. This is commensurate with the analysis of the dataset by experienced radiologists. Our approach, NoduleX, provides an effective framework for highly accurate nodule malignancy prediction with the model trained on a large patient population. Our results are replicable with software available at http://bioinformatics.astate.edu/NoduleX.

In this paper we study nonconvex and nonsmooth multi-block optimization over Riemannian manifolds with coupled linear constraints. Such optimization problems naturally arise from machine learning, statistical learning, compressive sensing, image processing, and tensor PCA, among others. We develop an ADMM-like primal-dual approach based on decoupled solvable subroutines such as linearized proximal mappings. First, we introduce the optimality conditions for the afore-mentioned optimization models. Then, the notion of $\epsilon$-stationary solutions is introduced as a result. The main part of the paper is to show that the proposed algorithms enjoy an iteration complexity of $O(1/\epsilon^2)$ to reach an $\epsilon$-stationary solution. For prohibitively large-size tensor or machine learning models, we present a sampling-based stochastic algorithm with the same iteration complexity bound in expectation. In case the subproblems are not analytically solvable, a feasible curvilinear line-search variant of the algorithm based on retraction operators is proposed. Finally, we show specifically how the algorithms can be implemented to solve a variety of practical problems such as the NP-hard maximum bisection problem, the $\ell_q$ regularized sparse tensor principal component analysis and the community detection problem. Our preliminary numerical results show great potentials of the proposed methods.

In this paper, we consider the community detection problem under either the stochastic block model (SBM) assumption or the degree-correlated stochastic block model (DCSBM) assumption. The modularity maximization formulation for the community detection problem is NP-hard in general. In this paper, we propose a sparse and low-rank completely positive relaxation for the modularity maximization problem, we then develop an efficient row-by-row (RBR) type block coordinate descent (BCD) algorithm to solve the relaxation and prove an $\mathcal{O}(1/\sqrt{N})$ convergence rate to a stationary point where $N$ is the number of iterations. A fast rounding scheme is constructed to retrieve the community structure from the solution. Non-asymptotic high probability bounds on the misclassification rate are established to justify our approach. We further develop an asynchronous parallel RBR algorithm to speed up the convergence. Extensive numerical experiments on both synthetic and real world networks show that the proposed approach enjoys advantages in both clustering accuracy and numerical efficiency. Our numerical results indicate that the newly proposed method is a quite competitive alternative for community detection on sparse networks with over 50 million nodes.

This note provides examples of all possible equality and strict inequality relations between upper and lower Abelian and Cesaro limits of sequences bounded above or below.