Search

We present Polynomial Attention Drop-in Replacement (PADRe), a novel and unifying framework designed to replace the conventional self-attention mechanism in transformer models. Notably, several recent alternative attention mechanisms, including Hyena, Mamba, SimA, Conv2Former, and Castling-ViT, can be viewed as specific instances of our PADRe framework. PADRe leverages polynomial functions and draws upon established results from approximation theory, enhancing computational efficiency without compromising accuracy. PADRe's key components include multiplicative nonlinearities, which we implement using straightforward, hardware-friendly operations such as Hadamard products, incurring only linear computational and memory costs. PADRe further avoids the need for using complex functions such as Softmax, yet it maintains comparable or superior accuracy compared to traditional self-attention. We assess the effectiveness of PADRe as a drop-in replacement for self-attention across diverse computer vision tasks. These tasks include image classification, image-based 2D object detection, and 3D point cloud object detection. Empirical results demonstrate that PADRe runs significantly faster than the conventional self-attention (11x ~ 43x faster on server GPU and mobile NPU) while maintaining similar accuracy when substituting self-attention in the transformer models.

In this paper, we propose a novel token selective attention approach, ToSA, which can identify tokens that need to be attended as well as those that can skip a transformer layer. More specifically, a token selector parses the current attention maps and predicts the attention maps for the next layer, which are then used to select the important tokens that should participate in the attention operation. The remaining tokens simply bypass the next layer and are concatenated with the attended ones to re-form a complete set of tokens. In this way, we reduce the quadratic computation and memory costs as fewer tokens participate in self-attention while maintaining the features for all the image patches throughout the network, which allows it to be used for dense prediction tasks. Our experiments show that by applying ToSA, we can significantly reduce computation costs while maintaining accuracy on the ImageNet classification benchmark. Furthermore, we evaluate on the dense prediction task of monocular depth estimation on NYU Depth V2, and show that we can achieve similar depth prediction accuracy using a considerably lighter backbone with ToSA., Comment: Accepted at CVPRW 2024

This research introduces an innovative forest monitoring system designed to detect and mitigate the threats of forest fires. The proposed system leverages Arduino-based technology integrated with state-of-the-art sensors, including DHT11 for temperature and humidity detection and Flame sensor along with GSM module for gas and smoke detection. The integration of these sensors enables real-time data acquisition and analysis, providing a comprehensive and accurate assessment of environmental conditions within the forest ecosystem. The Arduino platform serves as the central processing unit, orchestrating the communication and synchronization of the sensor data. The DHT11 sensor monitors ambient temperature and humidity levels, crucial indicators for assessing fire risk and identifying potential deforestation activities. Simultaneously, the Flame sensor module detects the occurrence of fire flames nearby thus indicating it by a buzzer. The collected data is processed through an intelligent algorithm that employs machine learning techniques to discern patterns indicative of potential threats. The system is equipped with an adaptive threshold mechanism, allowing it to dynamically adjust to changing environmental conditions. In the event of abnormal readings or anomalies, the system triggers immediate alerts, notifying forest rangers and relevant authorities to facilitate timely response and intervention. The integration of low-cost, easily deployable Arduino-based devices makes this solution scalable and accessible for implementation across diverse forest environments. The proposed system represents a significant step towards leveraging technology to address environmental challenges and protect our forests.

In this paper, we propose a novel video depth estimation approach, FutureDepth, which enables the model to implicitly leverage multi-frame and motion cues to improve depth estimation by making it learn to predict the future at training. More specifically, we propose a future prediction network, F-Net, which takes the features of multiple consecutive frames and is trained to predict multi-frame features one time step ahead iteratively. In this way, F-Net learns the underlying motion and correspondence information, and we incorporate its features into the depth decoding process. Additionally, to enrich the learning of multiframe correspondence cues, we further leverage a reconstruction network, R-Net, which is trained via adaptively masked auto-encoding of multiframe feature volumes. At inference time, both F-Net and R-Net are used to produce queries to work with the depth decoder, as well as a final refinement network. Through extensive experiments on several benchmarks, i.e., NYUDv2, KITTI, DDAD, and Sintel, which cover indoor, driving, and open-domain scenarios, we show that FutureDepth significantly improves upon baseline models, outperforms existing video depth estimation methods, and sets new state-of-the-art (SOTA) accuracy. Furthermore, FutureDepth is more efficient than existing SOTA video depth estimation models and has similar latencies when comparing to monocular models

In this paper, we introduce a novel approach that harnesses both 2D and 3D attentions to enable highly accurate depth completion without requiring iterative spatial propagations. Specifically, we first enhance a baseline convolutional depth completion model by applying attention to 2D features in the bottleneck and skip connections. This effectively improves the performance of this simple network and sets it on par with the latest, complex transformer-based models. Leveraging the initial depths and features from this network, we uplift the 2D features to form a 3D point cloud and construct a 3D point transformer to process it, allowing the model to explicitly learn and exploit 3D geometric features. In addition, we propose normalization techniques to process the point cloud, which improves learning and leads to better accuracy than directly using point transformers off the shelf. Furthermore, we incorporate global attention on downsampled point cloud features, which enables long-range context while still being computationally feasible. We evaluate our method, DeCoTR, on established depth completion benchmarks, including NYU Depth V2 and KITTI, showcasing that it sets new state-of-the-art performance. We further conduct zero-shot evaluations on ScanNet and DDAD benchmarks and demonstrate that DeCoTR has superior generalizability compared to existing approaches., Comment: Accepted at CVPR 2024

With their shielded 4f orbitals, rare-earth ions (REIs) offer optical and electron spin transitions with good coherence properties even when embedded in a host crystal matrix, highlighting their utility as promising quantum emitters and memories for quantum information processing. Among REIs, trivalent erbium (Er$^{3+}$) uniquely has an optical transition in the telecom C-band, ideal for transmission over optical fibers, and making it well-suited for applications in quantum communication. The deployment of Er$^{3+}$ emitters into a thin film TiO$_2$ platform has been a promising step towards scalable integration; however, like many solid-state systems, the deterministic spatial placement of quantum emitters remains an open challenge. We investigate laser annealing as a means to locally tune the optical resonance of Er$^{3+}$ emitters in TiO$_2$ thin films on Si. Using both nanoscale X-ray diffraction measurements and cryogenic photoluminescence spectroscopy, we show that tightly focused below-gap laser annealing can induce anatase to rutile phase transitions in a nearly diffraction-limited area of the films and improve local crystallinity through grain growth. As a percentage of the Er:TiO$_2$ is converted to rutile, the Er$^{3+}$ optical transition blueshifts by 13 nm. We explore the effects of changing laser annealing time and show that the amount of optically active Er:rutile increases linearly with laser power. We additionally demonstrate local phase conversion on microfabricated Si structures, which holds significance for quantum photonics., Comment: 7 pages, 4 figures

The ultimate realization of a global quantum internet will require advances in scalable technologies capable of generating, storing, and manipulating quantum information. The essential devices that will perform these tasks in a quantum network are quantum repeaters, which will enable the long-range distribution of entanglement between distant network nodes. In this perspective, we provide an overview of the primary functions of a quantum repeater and discuss progress that has been made toward the development of repeaters with rare-earth ion doped materials while noting challenges that are being faced as the technologies mature. We give particular attention to erbium, which is well suited for networking applications. Finally, we provide a discussion of near-term benchmarks that can further guide rare-earth ion platforms for impact in near-term quantum networks.

Neural networks (NN) can improve standard video compression by pre- and post-processing the encoded video. For optimal NN training, the standard codec needs to be replaced with a codec proxy that can provide derivatives of estimated bit-rate and distortion, which are used for gradient back-propagation. Since entropy coding of standard codecs is designed to take into account non-linear dependencies between transform coefficients, bit-rates cannot be well approximated with simple per-coefficient estimators. This paper presents a new approach for bit-rate estimation that is similar to the type employed in training end-to-end neural codecs, and able to efficiently take into account those statistical dependencies. It is defined from a mathematical model that provides closed-form formulas for the estimates and their gradients, reducing the computational complexity. Experimental results demonstrate the method's accuracy in estimating HEVC/H.265 codec bit-rates.

Rare-earth ions (REI) have emerged as an attractive candidate for solid-state qubits, particularly as a quantum memory. Their 4f-4f transitions are shielded by filled 5s and 5p orbitals, offering a degree of protection from external electric fields. Embedded within a thin film oxide host, REIs could enable a qubit platform with significant memory capabilities. Furthermore, a silicon-compatible thin film form factor would enable the use of standard semiconductor fabrication processes to achieve chip-based integrability and scalability for functional quantum networks. Towards this goal, we have carried out optical and microstructural studies of erbium-doped polycrystalline and epitaxial TiO$_2$ thin films on Si (100), r-sapphire, and SrTiO$_3$ (100). We observe that the inhomogeneous optical linewidth of the Er photoluminescence is comparable or better for polycrystalline Er:TiO$_2$(grown on Si) in comparison to single crystal epitaxial films on sapphire or SrTiO$_3$, implying a relative insensitivity to extended defects. We investigated the effect of the film/substrate and film/air interface and found that the inhomogeneous linewidth and spectral diffusion can be significantly improved via bottom buffer and top capping layers of undoped TiO$_2$. Using such approaches, we obtain inhomogeneous linewidths of 5.2 GHz and spectral diffusion of 180 MHz in Er:TiO$_2$ /Si(100) films and have demonstrated the engineerability of quantum-relevant properties in these thin films., Comment: 13 pages, 6 figures; corrects the units for spectral diffusion values (MHz throughout the manuscript instead of GHz(typo) in some places)

In many real-world applications, multiple agents seek to learn how to perform highly related yet slightly different tasks in an online bandit learning protocol. We formulate this problem as the $\epsilon$-multi-player multi-armed bandit problem, in which a set of players concurrently interact with a set of arms, and for each arm, the reward distributions for all players are similar but not necessarily identical. We develop an upper confidence bound-based algorithm, RobustAgg$(\epsilon)$, that adaptively aggregates rewards collected by different players. In the setting where an upper bound on the pairwise similarities of reward distributions between players is known, we achieve instance-dependent regret guarantees that depend on the amenability of information sharing across players. We complement these upper bounds with nearly matching lower bounds. In the setting where pairwise similarities are unknown, we provide a lower bound, as well as an algorithm that trades off minimax regret guarantees for adaptivity to unknown similarity structure.

Inferring interactions from multi-agent trajectories has broad applications in physics, vision and robotics. Neural relational inference (NRI) is a deep generative model that can reason about relations in complex dynamics without supervision. In this paper, we take a careful look at this approach for relational inference in multi-agent trajectories. First, we discover that NRI can be fundamentally limited without sufficient long-term observations. Its ability to accurately infer interactions degrades drastically for short output sequences. Next, we consider a more general setting of relational inference when interactions are changing overtime. We propose an extension ofNRI, which we call the DYnamic multi-AgentRelational Inference (DYARI) model that can reason about dynamic relations. We conduct exhaustive experiments to study the effect of model architecture, under-lying dynamics and training scheme on the performance of dynamic relational inference using a simulated physics system. We also showcase the usage of our model on real-world multi-agent basketball trajectories., Comment: submitted to ICLR 2021

Corpus preparation for low-resource languages and for development of human language technology to analyze or computationally process them is a laborious task, primarily due to the unavailability of expert linguists who are native speakers of these languages and also due to the time and resources required. Bhojpuri, Magahi, and Maithili, languages of the Purvanchal region of India (in the north-eastern parts), are low-resource languages belonging to the Indo-Aryan (or Indic) family. They are closely related to Hindi, which is a relatively high-resource language, which is why we compare with Hindi. We collected corpora for these three languages from various sources and cleaned them to the extent possible, without changing the data in them. The text belongs to different domains and genres. We calculated some basic statistical measures for these corpora at character, word, syllable, and morpheme levels. These corpora were also annotated with parts-of-speech (POS) and chunk tags. The basic statistical measures were both absolute and relative and were exptected to indicate of linguistic properties such as morphological, lexical, phonological, and syntactic complexities (or richness). The results were compared with a standard Hindi corpus. For most of the measures, we tried to the corpus size the same across the languages to avoid the effect of corpus size, but in some cases it turned out that using the full corpus was better, even if sizes were very different. Although the results are not very clear, we try to draw some conclusions about the languages and the corpora. For POS tagging and chunking, the BIS tagset was used to manually annotate the data. The POS tagged data sizes are 16067, 14669 and 12310 sentences, respectively, for Bhojpuri, Magahi and Maithili. The sizes for chunking are 9695 and 1954 sentences for Bhojpuri and Maithili, respectively., Comment: ACM Transactions on Asian and Low-Resource Language Information Processing (Accepted)

The rapid escalation in plug-in electric vehicles (PEVs) and their uncoordinated charging patterns pose several challenges in distribution system operation. Some of the undesirable effects include overloading of transformers, rapid voltage fluctuations, and over/under voltages. While this compromises the consumer power quality, it also puts on extra stress on the local voltage control devices. These challenges demand for a well-coordinated and power network-aware charging approach for PEVs in a community. This paper formulates a real-time electric vehicle charging scheduling problem as an mixed-integer linear program (MILP). The problem is to be solved by an aggregator, that provides charging service in a residential community. The proposed formulation maximizes the profit of the aggregator, enhancing the utilization of available infrastructure. With a prior knowledge of load demand and hourly electricity prices, the algorithm uses a moving time horizon optimization approach, allowing the number of vehicles arriving unknown. In this realistic setting, the proposed framework ensures that power system constraints are satisfied and guarantees desired PEV charging level within stipulated time. Numerical tests on a IEEE 13-node feeder system demonstrate the computational and performance superiority of the proposed MILP technique., Comment: Accepted for presentation at PES-General Meeting, Montreal, 2020

The vast infrastructure development, gas flow dynamics, and complex interdependence of gas with electric power networks call for advanced computational tools. Solving the equations relating gas injections to pressures and pipeline flows lies at the heart of natural gas network (NGN) operation, yet existing solvers require careful initialization and uniqueness has been an open question. In this context, this work considers the nonlinear steady-state version of the gas flow (GF) problem. It first establishes that the solution to the GF problem is unique under arbitrary NGN topologies, compressor types, and sets of specifications. For GF setups where pressure is specified on a single (reference) node and compressors do no appear in cycles, the GF task is posed as an convex minimization. To handle more general setups, a GF solver relying on a mixed-integer quadratically-constrained quadratic program (MI-QCQP) is also devised. This solver can be used for any GF setup at any NGN. It introduces binary variables to capture flow directions; relaxes the pressure drop equations to quadratic inequality constraints; and uses a carefully selected objective to promote the exactness of this relaxation. The relaxation is provably exact in NGNs with non-overlapping cycles and a single fixed-pressure node. The solver handles efficiently the involved bilinear terms through McCormick linearization. Numerical tests validate our claims, demonstrate that the MI-QCQP solver scales well, and that the relaxation is exact even when the sufficient conditions are violated, such as in NGNs with overlapping cycles and multiple fixed-pressure nodes.

Increasing emphasis on reliability and resiliency call for advanced distribution system restoration (DSR). The integration of grid sensors, remote controls, and distributed generators (DG) brings about exciting opportunities in DSR. In this context, this work considers the task of single-step restoration of a single phase power distribution system. Different from existing works, the devised restoration scheme achieves optimal formation of islands without heuristically pre-identifying reference buses. It further facilitates multiple DGs running within the same island, and establishes a coordination hierarchy in terms of their PV/PQ operation modes. Generators without black-start capability are guaranteed to remain connected to a black-start DG or a substation. The proposed scheme models remotely-controlled voltage regulators exactly, and integrates them in the restoration process. Numerical tests on a modified IEEE 37-bus feeder demonstrate that the proposed mixed-integer linear program (MILP) takes less than four seconds to handle random outages of 1-5 lines. The scalability of this novel MILP formulation can be attributed to the unique use of cycles and paths on the grid infrastructure graph; the McCormick linearization technique; and an approximate power flow model.

In this paper we deduce a graded version of Quillen--Suslin's Local-Global Principle for the traditional classical groups, viz. general linear, symplectic and orthogonal groups and establish its equivalence of the normality property of the respective elementary subgroups. This generalizes previous result of Basu--Rao--Khanna. Then, as an application, we establish an analogue Local-Global Principle for the commutator subgroups of the special linear and symplectic groups. Finally, by using Swan-Weibel's homotopy trick, we establish graded analogue of the Local-Global Principle for the transvection subgroups of the full automorphism groups of the linear, symplectic and orthogonal modules. This generalizes the previous result of Bak--Basu--Rao., Comment: 15 pgs