Search

Vision-language navigation (VLN) is a critical domain within embedded intelligence, requiring agents to navigate 3D environments based on natural language instructions. Traditional VLN research has focused on improving environmental understanding and decision accuracy. However, these approaches often exhibit a significant performance gap when agents are deployed in novel environments, mainly due to the limited diversity of training data. Expanding datasets to cover a broader range of environments is impractical and costly. We propose the Vision-Language Navigation with Continual Learning (VLNCL) paradigm to address this challenge. In this paradigm, agents incrementally learn new environments while retaining previously acquired knowledge. VLNCL enables agents to maintain an environmental memory and extract relevant knowledge, allowing rapid adaptation to new environments while preserving existing information. We introduce a novel dual-loop scenario replay method (Dual-SR) inspired by brain memory replay mechanisms integrated with VLN agents. This method facilitates consolidating past experiences and enhances generalization across new tasks. By utilizing a multi-scenario memory buffer, the agent efficiently organizes and replays task memories, thereby bolstering its ability to adapt quickly to new environments and mitigating catastrophic forgetting. Our work pioneers continual learning in VLN agents, introducing a novel experimental setup and evaluation metrics. We demonstrate the effectiveness of our approach through extensive evaluations and establish a benchmark for the VLNCL paradigm. Comparative experiments with existing continual learning and VLN methods show significant improvements, achieving state-of-the-art performance in continual learning ability and highlighting the potential of our approach in enabling rapid adaptation while preserving prior knowledge.

Vision Language Navigation in Continuous Environments (VLN-CE) represents a frontier in embodied AI, demanding agents to navigate freely in unbounded 3D spaces solely guided by natural language instructions. This task introduces distinct challenges in multimodal comprehension, spatial reasoning, and decision-making. To address these challenges, we introduce Cog-GA, a generative agent founded on large language models (LLMs) tailored for VLN-CE tasks. Cog-GA employs a dual-pronged strategy to emulate human-like cognitive processes. Firstly, it constructs a cognitive map, integrating temporal, spatial, and semantic elements, thereby facilitating the development of spatial memory within LLMs. Secondly, Cog-GA employs a predictive mechanism for waypoints, strategically optimizing the exploration trajectory to maximize navigational efficiency. Each waypoint is accompanied by a dual-channel scene description, categorizing environmental cues into 'what' and 'where' streams as the brain. This segregation enhances the agent's attentional focus, enabling it to discern pertinent spatial information for navigation. A reflective mechanism complements these strategies by capturing feedback from prior navigation experiences, facilitating continual learning and adaptive replanning. Extensive evaluations conducted on VLN-CE benchmarks validate Cog-GA's state-of-the-art performance and ability to simulate human-like navigation behaviors. This research significantly contributes to the development of strategic and interpretable VLN-CE agents.

Superconducting qubits provide a promising approach to large-scale fault-tolerant quantum computing. However, qubit connectivity on a planar surface is typically restricted to only a few neighboring qubits. Achieving longer-range and more flexible connectivity, which is particularly appealing in light of recent developments in error-correcting codes, however usually involves complex multi-layer packaging and external cabling, which is resource-intensive and can impose fidelity limitations. Here, we propose and realize a high-speed on-chip quantum processor that supports reconfigurable all-to-all coupling with a large on-off ratio. We implement the design in a four-node quantum processor, built with a modular design comprising a wiring substrate coupled to two separate qubit-bearing substrates, each including two single-qubit nodes. We use this device to demonstrate reconfigurable controlled-Z gates across all qubit pairs, with a benchmarked average fidelity of $96.00\%\pm0.08\%$ and best fidelity of $97.14\%\pm0.07\%$, limited mainly by dephasing in the qubits. We also generate multi-qubit entanglement, distributed across the separate modules, demonstrating GHZ-3 and GHZ-4 states with fidelities of $88.15\%\pm0.24\%$ and $75.18\%\pm0.11\%$, respectively. This approach promises efficient scaling to larger-scale quantum circuits, and offers a pathway for implementing quantum algorithms and error correction schemes that benefit from enhanced qubit connectivity.

Quantum communication testbeds provide a useful resource for experimentally investigating a variety of communication protocols. Here we demonstrate a superconducting circuit testbed with bidirectional multi-photon state transfer capability using time-domain shaped wavepackets. The system we use to achieve this comprises two remote nodes, each including a tunable superconducting transmon qubit and a tunable microwave-frequency resonator, linked by a 2 m-long superconducting coplanar waveguide, which serves as a transmission line. We transfer both individual and superposition Fock states between the two remote nodes, and additionally show that this bidirectional state transfer can be done simultaneously, as well as used to entangle elements in the two nodes., Comment: Main Paper has 6 pages, 4 figures. Supplementary has 14 pages, 16 figures, 2 tables

In the field of quantum computation and communication there is a compelling need for quantum-coherent frequency conversion between microwave electronics and infra-red optics. A promising platform for this is an optomechanical crystal resonator that uses simultaneous photonic and phononic crystals to create a co-localized cavity coupling an electromagnetic mode to an acoustic mode, which then via electromechanical interactions can undergo direct transduction to electronics. The majority of work in this area has been on one-dimensional nanobeam resonators which provide strong optomechanical couplings but, due to their geometry, suffer from an inability to dissipate heat produced by the laser pumping required for operation. Recently, a quasi-two-dimensional optomechanical crystal cavity was developed in silicon exhibiting similarly strong coupling with better thermalization, but at a mechanical frequency above optimal qubit operating frequencies. Here we adapt this design to gallium arsenide, a natural thin-film single-crystal piezoelectric that can incorporate electromechanical interactions, obtaining a mechanical resonant mode at f_m ~ 4.5 GHz ideal for superconducting qubits, and demonstrating optomechanical coupling g_om/(2pi) ~ 650 kHz.

Spiking Neural Networks (SNNs) have garnered widespread interest for their energy efficiency and brain-inspired event-driven properties. While recent methods like Spiking-YOLO have expanded the SNNs to more challenging object detection tasks, they often suffer from high latency and low detection accuracy, making them difficult to deploy on latency sensitive mobile platforms. Furthermore, the conversion method from Artificial Neural Networks (ANNs) to SNNs is hard to maintain the complete structure of the ANNs, resulting in poor feature representation and high conversion errors. To address these challenges, we propose two methods: timesteps compression and spike-time-dependent integrated (STDI) coding. The former reduces the timesteps required in ANN-SNN conversion by compressing information, while the latter sets a time-varying threshold to expand the information holding capacity. We also present a SNN-based ultra-low latency and high accurate object detection model (SUHD) that achieves state-of-the-art performance on nontrivial datasets like PASCAL VOC and MS COCO, with about remarkable 750x fewer timesteps and 30% mean average precision (mAP) improvement, compared to the Spiking-YOLO on MS COCO datasets. To the best of our knowledge, SUHD is the deepest spike-based object detection model to date that achieves ultra low timesteps to complete the lossless conversion., Comment: 14 pages, 10 figures

In circuit quantum electrodynamics (QED), qubits are typically measured using dispersively-coupled readout resonators. Coupling between each readout resonator and its electrical environment however reduces the qubit lifetime via the Purcell effect. Inserting a Purcell filter counters this effect while maintaining high readout fidelity, but reduces measurement bandwidth and thus limits multiplexing readout capacity. In this letter, we develop and implement a multi-stage bandpass Purcell filter that yields better qubit protection while simultaneously increasing measurement bandwidth and multiplexed capacity. We report on the experimental performance of our transmission-line--based implementation of this approach, a flexible design that can easily be integrated with current scaled-up, long coherence time superconducting quantum processors.

Linear optical quantum computing provides a desirable approach to quantum computing, with a short list of required elements. The similarity between photons and phonons points to the interesting potential for linear mechanical quantum computing (LMQC), using phonons in place of photons. While single-phonon sources and detectors have been demonstrated, a phononic beamsplitter element remains an outstanding requirement. Here we demonstrate such an element, using two superconducting qubits to fully characterize a beamsplitter with single phonons. We further use the beamsplitter to demonstrate two-phonon interference, a requirement for two-qubit gates, completing the toolbox needed for LMQC. This advance brings linear quantum computing to a fully solid-state system, along with straightforward conversion between itinerant phonons and superconducting qubits.

This paper tackles the problem of novel category discovery (NCD), which aims to discriminate unknown categories in large-scale image collections. The NCD task is challenging due to the closeness to the real-world scenarios, where we have only encountered some partial classes and images. Unlike other works on the NCD, we leverage the prototypes to emphasize the importance of category discrimination and alleviate the issue of missing annotations of novel classes. Concretely, we propose a novel adaptive prototype learning method consisting of two main stages: prototypical representation learning and prototypical self-training. In the first stage, we obtain a robust feature extractor, which could serve for all images with base and novel categories. This ability of instance and category discrimination of the feature extractor is boosted by self-supervised learning and adaptive prototypes. In the second stage, we utilize the prototypes again to rectify offline pseudo labels and train a final parametric classifier for category clustering. We conduct extensive experiments on four benchmark datasets and demonstrate the effectiveness and robustness of the proposed method with state-of-the-art performance., Comment: In Submission

Combining parametric driving and photon-atomic memory coupling within one optical cavity, we describe a scheme for in-situ generation of multimode photon-memory entanglement. We find that precise cavity impedance matching is neither required nor optimal to achieve high-rate entanglement for quantum networks. This protocol can be realized with existing technologies based on on-chip photonic cavities integrated with a rare-earth-ion doped quantum memory. The proposed scheme shows significant advantages in entanglement generation rates compared with start-of-the-art quantum memory protocols and experiments, with predicted Ebit generation rates of tens of MHz without ideal operating conditions. Such a photon-memory entanglement system offers a versatile resource for quantum interconnect applications., Comment: 5 pages, 4 figures

Object goal visual navigation is a challenging task that aims to guide a robot to find the target object based on its visual observation, and the target is limited to the classes pre-defined in the training stage. However, in real households, there may exist numerous target classes that the robot needs to deal with, and it is hard for all of these classes to be contained in the training stage. To address this challenge, we study the zero-shot object goal visual navigation task, which aims at guiding robots to find targets belonging to novel classes without any training samples. To this end, we also propose a novel zero-shot object navigation framework called semantic similarity network (SSNet). Our framework use the detection results and the cosine similarity between semantic word embeddings as input. Such type of input data has a weak correlation with classes and thus our framework has the ability to generalize the policy to novel classes. Extensive experiments on the AI2-THOR platform show that our model outperforms the baseline models in the zero-shot object navigation task, which proves the generalization ability of our model. Our code is available at: https://github.com/pioneer-innovation/Zero-Shot-Object-Navigation.

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method., Comment: Journal extension of the previous conference paper arXiv:1901.02645, see IEEE page https://ieeexplore.ieee.org/document/9523596

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, the domain shift caused by the sim2real gap is inevitable, posing a crucial challenge to the segmentation model. In this paper, we emphasize the adaptation process across sim2real domains and model it as a learning problem on the BatchNorm parameters of a simulation-trained model. Specifically, we propose a novel non-parametric entropy objective, which formulates the learning objective for the test-time adaptation in an open-world manner. Then, a cross-modality knowledge distillation objective is further designed to encourage the test-time knowledge transfer for feature enhancement. Our approach can be efficiently implemented with only test images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on the overlap and boundary metrics, achieving state-of-the-art performance on unseen object instance segmentation., Comment: Accepted to ICRA 2023

Rapid, accurate and robust detection of looming objects in cluttered moving backgrounds is a significant and challenging problem for robotic visual systems to perform collision detection and avoidance tasks. Inspired by the neural circuit of elementary motion vision in the mammalian retina, this paper proposes a bioinspired approach-sensitive neural network (ASNN) that contains three main contributions. Firstly, a direction-selective visual processing module is built based on the spatiotemporal energy framework, which can estimate motion direction accurately via only two mutually perpendicular spatiotemporal filtering channels. Secondly, a novel approach-sensitive neural network is modeled as a push-pull structure formed by ON and OFF pathways, which responds strongly to approaching motion while insensitivity to lateral motion. Finally, a method of directionally selective inhibition is introduced, which is able to suppress the translational backgrounds effectively. Extensive synthetic and real robotic experiments show that the proposed model is able to not only detect collision accurately and robustly in cluttered and dynamic backgrounds but also extract more collision information like position and direction, for guiding rapid decision making.

Robust and accurate detection of small moving targets in cluttered moving backgrounds is a significant and challenging problem for robotic visual systems to perform search and tracking tasks. Inspired by the neural circuitry of elementary motion vision in the mammalian retina, this paper proposes a bioinspired retinal neural network based on a new neurodynamics-based temporal filtering and multiform 2-D spatial Gabor filtering. This model can estimate motion direction accurately via only two perpendicular spatiotemporal filtering signals, and respond to small targets of different sizes and velocities by adjusting the dendrite field size of the spatial filter. Meanwhile, an algorithm of directionally selective inhibition is proposed to suppress the target-like features in the moving background, which can reduce the influence of background motion effectively. Extensive synthetic and real-data experiments show that the proposed model works stably for small targets of a wider size and velocity range, and has better detection performance than other bioinspired models. Additionally, it can also extract the information of motion direction and motion energy accurately and rapidly.

Due to the non-smoothness of the Hinge loss in SVM, it is difficult to obtain a faster convergence rate with modern optimization algorithms. In this paper, we introduce two smooth Hinge losses $\psi_G(\alpha;\sigma)$ and $\psi_M(\alpha;\sigma)$ which are infinitely differentiable and converge to the Hinge loss uniformly in $\alpha$ as $\sigma$ tends to $0$. By replacing the Hinge loss with these two smooth Hinge losses, we obtain two smooth support vector machines(SSVMs), respectively. Solving the SSVMs with the Trust Region Newton method (TRON) leads to two quadratically convergent algorithms. Experiments in text classification tasks show that the proposed SSVMs are effective in real-world applications. We also introduce a general smooth convex loss function to unify several commonly-used convex loss functions in machine learning. The general framework provides smooth approximation functions to non-smooth convex loss functions, which can be used to obtain smooth models that can be solved with faster convergent optimization algorithms.

Imbalanced classification tasks are widespread in many real-world applications. For such classification tasks, in comparison with the accuracy rate, it is usually much more appropriate to use non-decomposable performance measures such as the Area Under the receiver operating characteristic Curve (AUC) and the $F_\beta$ measure as the classification criterion since the label class is imbalanced. On the other hand, the minimax probability machine is a popular method for binary classification problems and aims at learning a linear classifier by maximizing the accuracy rate, which makes it unsuitable to deal with imbalanced classification tasks. The purpose of this paper is to develop a new minimax probability machine for the $F_\beta$ measure, called MPMF, which can be used to deal with imbalanced classification tasks. A brief discussion is also given on how to extend the MPMF model for several other non-decomposable performance measures listed in the paper. To solve the MPMF model effectively, we derive its equivalent form which can then be solved by an alternating descent method to learn a linear classifier. Further, the kernel trick is employed to derive a nonlinear MPMF model to learn a nonlinear classifier. Several experiments on real-world benchmark datasets demonstrate the effectiveness of our new model.

Remote tiny face detection applied in unmanned system is a challeng-ing work. The detector cannot obtain sufficient context semantic information due to the relatively long distance. The received poor fine-grained features make the face detection less accurate and robust. To solve the problem of long-distance detection of tiny faces, we propose an enhanced network model (YOLOv3-C) based on the YOLOv3 algorithm for unmanned platform. In this model, we bring in multi-scale features from feature pyramid networks and make the features fu-sion to adjust prediction feature map of the output, which improves the sensitivity of the entire algorithm for tiny target faces. The enhanced model improves the accuracy of tiny face detection in the cases of long-distance and high-density crowds. The experimental evaluation results demonstrated the superior perfor-mance of the proposed YOLOv3-C in comparison with other relevant detectors in remote tiny face detection. It is worth mentioning that our proposed method achieves comparable performance with the state of the art YOLOv4[1] in the tiny face detection tasks.