Search

Upside Down Reinforcement Learning (UDRL) is a promising framework for solving reinforcement learning problems which focuses on learning command-conditioned policies. In this work, we extend UDRL to the task of learning a command-conditioned generator of deep neural network policies. We accomplish this using Hypernetworks - a variant of Fast Weight Programmers, which learn to decode input commands representing a desired expected return into command-specific weight matrices. Our method, dubbed Upside Down Reinforcement Learning with Policy Generators (UDRLPG), streamlines comparable techniques by removing the need for an evaluator or critic to update the weights of the generator. To counteract the increased variance in last returns caused by not having an evaluator, we decouple the sampling probability of the buffer from the absolute number of policies in it, which, together with a simple weighting strategy, improves the empirical convergence of the algorithm. Compared with existing algorithms, UDRLPG achieves competitive performance and high returns, sometimes outperforming more complex architectures. Our experiments show that a trained generator can generalize to create policies that achieve unseen returns zero-shot. The proposed method appears to be effective in mitigating some of the challenges associated with learning highly multimodal functions. Altogether, we believe that UDRLPG represents a promising step forward in achieving greater empirical sample efficiency in RL. A full implementation of UDRLPG is publicly available at https://github.com/JacopoD/udrlpg_, Comment: 4 pages in main text, 4 figures in main text; source code available at https://github.com/JacopoD/udrlpg_

Accurate depth estimation is crucial for many fields, including robotics, navigation, and medical imaging. However, conventional depth sensors often produce low-resolution (LR) depth maps, making detailed scene perception challenging. To address this, enhancing LR depth maps to high-resolution (HR) ones has become essential, guided by HR-structured inputs like RGB or grayscale images. We propose a novel sensor fusion methodology for guided depth super-resolution (GDSR), a technique that combines LR depth maps with HR images to estimate detailed HR depth maps. Our key contribution is the Incremental guided attention fusion (IGAF) module, which effectively learns to fuse features from RGB images and LR depth maps, producing accurate HR depth maps. Using IGAF, we build a robust super-resolution model and evaluate it on multiple benchmark datasets. Our model achieves state-of-the-art results compared to all baseline models on the NYU v2 dataset for $\times 4$, $\times 8$, and $\times 16$ upsampling. It also outperforms all baselines in a zero-shot setting on the Middlebury, Lu, and RGB-D-D datasets. Code, environments, and models are available on GitHub.

Language-based agentic systems have shown great promise in recent years, transitioning from solving small-scale research problems to being deployed in challenging real-world tasks. However, optimizing these systems often requires substantial manual labor. Recent studies have demonstrated that these systems can be represented as computational graphs, enabling automatic optimization. Despite these advancements, most current efforts in Graph-based Agentic System Optimization (GASO) fail to properly assign feedback to the system's components given feedback on the system's output. To address this challenge, we formalize the concept of semantic backpropagation with semantic gradients -- a generalization that aligns several key optimization techniques, including reverse-mode automatic differentiation and the more recent TextGrad by exploiting the relationship among nodes with a common successor. This serves as a method for computing directional information about how changes to each component of an agentic system might improve the system's output. To use these gradients, we propose a method called semantic gradient descent which enables us to solve GASO effectively. Our results on both BIG-Bench Hard and GSM8K show that our approach outperforms existing state-of-the-art methods for solving GASO problems. A detailed ablation study on the LIAR dataset demonstrates the parsimonious nature of our method. A full copy of our implementation is publicly available at https://github.com/HishamAlyahya/semantic_backprop, Comment: 11 pages in main text + 2 pages of references + 15 pages of appendices, 2 figures in main text + 17 figures in appendices, 2 tables in main text + 1 table in appendices, 2 algorithms in main text; source code available at https://github.com/HishamAlyahya/semantic_backprop

Optical brain imaging technologies are promising due to their relatively high temporal resolution, portability and cost-effectiveness. However, the highly scattering nature of near-infrared light in human tissue makes it challenging to collect photons emerging from more than 4 cm below the scalp, or with source-detector separation larger than several centimeters. We explore the physical limits of photon transport in the head and show that despite an extreme attenuation of ~10^(18), we can experimentally detect light that is transmitted diametrically through the entire adult human head. Analysis of various photon migration pathways through the head also indicates how the source-detector configuration can be used to isolate photons interacting with deep regions of the brain that are inaccessible with current optical techniques.

Modern imaging technologies are widely based on classical principles of light or electromagnetic wave propagation. They can be remarkably sophisticated, with recent successes ranging from single molecule microscopy to imaging far-distant galaxies. However, new imaging technologies based on quantum principles are gradually emerging. They can either surpass classical approaches or provide novel imaging capabilities that would not otherwise be possible. {Here }we provide an overview {of the most recently developed quantum imaging systems, highlighting the non-classical properties of sources such as bright squeezed light, entangled photons, and single-photon emitters that enable their functionality.} We outline potential upcoming trends and the associated challenges, all driven by a central inquiry, which is to understand whether quantum light can make visible the invisible.

In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich the information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, whose efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes the radar signals using convolutional Long Short-term Memory (LSTM) blocks and the depth signal, using convolutional operations. The combined latent features are then magnified using a deep feature magnification to learn cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to learn the location of the concealed object in the spatial domain through radar feature guidance. We demonstrate the detection of presence and inference of 3D location of concealed metal objects with an accuracy of up to 95%, using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost effective and portable sensor fusion, with strong opportunities for further development.

Recent experiments have searched for evidence of the impact of non-inertial motion on the entanglement of particles. The success of these endeavours has been hindered by the fact that such tests were performed within spatial scales that were only "local" when compared to the spatial scales over which the non-inertial motion was taking place. We propose a Sagnac-like interferometer that, by challenging such bottlenecks, is able to achieve entangled states through a mechanism induced by the mechanical rotation of a photonic interferometer. The resulting states violate the Bell-Clauser-Horne-Shimony-Holt (CHSH) inequality all the way up to the Tsirelson bound, thus signaling strong quantum nonlocality. Our results demonstrate that mechanical rotation can be thought of as resource for controlling quantum non-locality with implications also for recent proposals for experiments that can probe the quantum nature of curved spacetimes and non-inertial motion., Comment: 6 pages, 1 figure

The Value Iteration Network (VIN) is an end-to-end differentiable architecture that performs value iteration on a latent MDP for planning in reinforcement learning (RL). However, VINs struggle to scale to long-term and large-scale planning tasks, such as navigating a $100\times 100$ maze -- a task which typically requires thousands of planning steps to solve. We observe that this deficiency is due to two issues: the representation capacity of the latent MDP and the planning module's depth. We address these by augmenting the latent MDP with a dynamic transition kernel, dramatically improving its representational capacity, and, to mitigate the vanishing gradient problem, introducing an "adaptive highway loss" that constructs skip connections to improve gradient flow. We evaluate our method on both 2D maze navigation environments and the ViZDoom 3D navigation benchmark. We find that our new method, named Dynamic Transition VIN (DT-VIN), easily scales to 5000 layers and casually solves challenging versions of the above tasks. Altogether, we believe that DT-VIN represents a concrete step forward in performing long-term large-scale planning in RL environments.

In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road for gigapixel image generation at no additional cost. Our cascading method uses the image generated at the lowest resolution as a baseline to sample at higher resolutions. For the guidance, we introduce the Slider, a tunable mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands such that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that Pixelsmith not only achieves higher quality and diversity compared to existing techniques, but also reduces sampling time and artifacts. The code for our work is available at https://github.com/Thanos-DB/Pixelsmith.

Value iteration networks (VINs) enable end-to-end learning for planning tasks by employing a differentiable "planning module" that approximates the value iteration algorithm. However, long-term planning remains a challenge because training very deep VINs is difficult. To address this problem, we embed highway value iteration -- a recent algorithm designed to facilitate long-term credit assignment -- into the structure of VINs. This improvement augments the "planning module" of the VIN with three additional components: 1) an "aggregate gate," which constructs skip connections to improve information flow across many layers; 2) an "exploration module," crafted to increase the diversity of information and gradient flow in spatial dimensions; 3) a "filter gate" designed to ensure safe exploration. The resulting novel highway VIN can be trained effectively with hundreds of layers using standard backpropagation. In long-term planning tasks requiring hundreds of planning steps, deep highway VINs outperform both traditional VINs and several advanced, very deep NNs., Comment: ICML 2024

Learning from multi-step off-policy data collected by a set of policies is a core problem of reinforcement learning (RL). Approaches based on importance sampling (IS) often suffer from large variances due to products of IS ratios. Typical IS-free methods, such as $n$-step Q-learning, look ahead for $n$ time steps along the trajectory of actions (where $n$ is called the lookahead depth) and utilize off-policy data directly without any additional adjustment. They work well for proper choices of $n$. We show, however, that such IS-free methods underestimate the optimal value function (VF), especially for large $n$, restricting their capacity to efficiently utilize information from distant future time steps. To overcome this problem, we introduce a novel, IS-free, multi-step off-policy method that avoids the underestimation issue and converges to the optimal VF. At its core lies a simple but non-trivial \emph{highway gate}, which controls the information flow from the distant future by comparing it to a threshold. The highway gate guarantees convergence to the optimal VF for arbitrary $n$ and arbitrary behavioral policies. It gives rise to a novel family of off-policy RL algorithms that safely learn even when $n$ is very large, facilitating rapid credit assignment from the far future to the past. On tasks with greatly delayed rewards, including video games where the reward is given only at the end of the game, our new methods outperform many existing multi-step off-policy algorithms.

Advancements in quantum communication and sensing require improved optical transmission that ensures excellent state purity and reduced losses. While free-space optical communication is often preferred, its use becomes challenging over long distances due to beam divergence, atmospheric absorption, scattering, and turbulence, among other factors. In the case of polarization encoding, traditional silica-core optical fibers, though commonly used, struggle with maintaining state purity due to stress-induced birefringence. Hollow core fibers, and in particular nested antiresonant nodeless fibers (NANF), have recently been shown to possess unparalleled polarization purity with minimal birefringence in the telecom wavelength range using continuous-wave (CW) laser light. Here, we investigate a 1-km NANF designed for wavelengths up to the 2-$\mu$m waveband. Our results show a polarization extinction ratio between ~-30 dB and ~-70 dB across the 1520 to 1620 nm range in CW operation, peaking at ~-60 dB at the 2-$\mu$m design wavelength. Our study also includes the pulsed regime, providing insights beyond previous CW studies, e.g., on the propagation of broadband quantum states of light in NANF at 2 $\mu$m, and corresponding extinction-ratio-limited quantum bit error rates (QBER) for prepare-measure and entanglement-based quantum key distribution (QKD) protocols. Our findings highlight the potential of these fibers in emerging applications such as QKD, pointing towards a new standard in optical quantum technologies.

Fluorescence lifetime imaging microscopy (FLIM) provides detailed information about molecular interactions and biological processes. A major bottleneck for FLIM is image resolution at high acquisition speeds, due to the engineering and signal-processing limitations of time-resolved imaging technology. Here we present single-sample image-fusion upsampling (SiSIFUS), a data-fusion approach to computational FLIM super-resolution that combines measurements from a low-resolution time-resolved detector (that measures photon arrival time) and a high-resolution camera (that measures intensity only). To solve this otherwise ill-posed inverse retrieval problem, we introduce statistically informed priors that encode local and global dependencies between the two single-sample measurements. This bypasses the risk of out-of-distribution hallucination as in traditional data-driven approaches and delivers enhanced images compared for example to standard bilinear interpolation. The general approach laid out by SiSIFUS can be applied to other image super-resolution problems where two different datasets are available., Comment: 18 pages, 11 figures. To be published in Science Advances

Advanced machine learning algorithms require platforms that are extremely robust and equipped with rich sensory feedback to handle extensive trial-and-error learning without relying on strong inductive biases. Traditional robotic designs, while well-suited for their specific use cases, are often fragile when used with these algorithms. To address this gap -- and inspired by the vision of enabling curiosity-driven baby robots -- we present a novel robotic limb designed from scratch. Our design has a hybrid soft-hard structure, high redundancy with rich non-contact sensors (exclusively cameras), and easily replaceable failure points. Proof-of-concept experiments using two contemporary reinforcement learning algorithms on a physical prototype demonstrate that our design is able to succeed in a simple target-finding task even under simulated sensor failures, all with minimal human oversight during extended learning periods. We believe this design represents a concrete step toward more tailored robotic designs for achieving general-purpose, generally intelligent robots., Comment: 6 pages in main text + 2 pages of references, 8 figures in main text, 1 table in main text; source code available at https://github.com/dylanashley/robot-limb-testai

We explore the role of attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. However, self-attention initially plays a minor role but becomes crucial in the second phase. These findings yield a simple and training-free method known as temporally gating the attention (TGATE), which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experimental results show when widely applied to various existing text-conditional diffusion models, TGATE accelerates these models by 10%-50%. The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Recurrent Neural Networks (RNNs) are general-purpose parallel-sequential computers. The program of an RNN is its weight matrix. How to learn useful representations of RNN weights that facilitate RNN analysis as well as downstream tasks? While the mechanistic approach directly looks at some RNN's weights to predict its behavior, the functionalist approach analyzes its overall functionality-specifically, its input-output mapping. We consider several mechanistic approaches for RNN weights and adapt the permutation equivariant Deep Weight Space layer for RNNs. Our two novel functionalist approaches extract information from RNN weights by 'interrogating' the RNN through probing inputs. We develop a theoretical framework that demonstrates conditions under which the functionalist approach can generate rich representations that help determine RNN behavior. We release the first two 'model zoo' datasets for RNN weight representation learning. One consists of generative models of a class of formal languages, and the other one of classifiers of sequentially processed MNIST digits.With the help of an emulation-based self-supervised learning technique we compare and evaluate the different RNN weight encoding techniques on multiple downstream applications. On the most challenging one, namely predicting which exact task the RNN was trained on, functionalist approaches show clear superiority.

We propose a novel transformer-style architecture called Global-Local Filter Network (GLFNet) for medical image segmentation and demonstrate its state-of-the-art performance. We replace the self-attention mechanism with a combination of global-local filter blocks to optimize model efficiency. The global filters extract features from the whole feature map whereas the local filters are being adaptively created as 4x4 patches of the same feature map and add restricted scale information. In particular, the feature extraction takes place in the frequency domain rather than the commonly used spatial (image) domain to facilitate faster computations. The fusion of information from both spatial and frequency spaces creates an efficient model with regards to complexity, required data and performance. We test GLFNet on three benchmark datasets achieving state-of-the-art performance on all of them while being almost twice as efficient in terms of GFLOP operations.

Various human-designed prompt engineering techniques have been proposed to improve problem solvers based on Large Language Models (LLMs), yielding many disparate code bases. We unify these approaches by describing LLM-based agents as computational graphs. The nodes implement functions to process multimodal data or query LLMs, and the edges describe the information flow between operations. Graphs can be recursively combined into larger composite graphs representing hierarchies of inter-agent collaboration (where edges connect operations of different agents). Our novel automatic graph optimizers (1) refine node-level LLM prompts (node optimization) and (2) improve agent orchestration by changing graph connectivity (edge optimization). Experiments demonstrate that our framework can be used to efficiently develop, integrate, and automatically improve various LLM agents. The code can be found at https://github.com/metauto-ai/gptswarm., Comment: Project Website: https://gptswarm.org ; Github Repo: https://github.com/metauto-ai/gptswarm . In Forty-first International Conference on Machine Learning (2024)

Quantum imaging enhances imaging systems performance, potentially surpassing fundamental limits such as noise and resolution. However, these schemes have limitations and are still a long way from replacing classical techniques. Therefore, there is a strong focus on improving the practicality of quantum imaging methods, with the goal of finding real-world applications. With this in mind, in this tutorial we describe how the concepts of classical light shaping can be applied to imaging schemes based on entangled photon pairs. We detail two basic experimental configurations in which a spatial light modulator is used to shape the spatial correlations of a photon pair state and highlight the key differences between this and classical shaping. We then showcase two recent examples that expand on these concepts to perform aberration and scattering correction with photon pairs. We include specific details on the key steps of these experiments, with the goal that this can be used as a guide for building photon-pair-based imaging and shaping experiments., Comment: Total 19 pages; Main manuscript: 12 pages, 4 figures; Supplementary material: 7 pages, 5 figures

Neuromorphic processors improve the efficiency of machine learning algorithms through the implementation of physical artificial neurons to perform computations. However, whilst efficient classical neuromorphic processors have been demonstrated in various forms, practical quantum neuromorphic platforms are still in the early stages of development. Here we propose a fixed optical network for photonic quantum reservoir computing that is enabled by photon number-resolved detection of the output states. This significantly reduces the required complexity of the input quantum states while still accessing a high-dimensional Hilbert space. The approach is implementable with currently available technology and lowers the barrier to entry to quantum machine learning., Comment: Information from supplementary migrated to main body. Some rewording and new figures. Reformatted

Fluorescence Lifetime Imaging Microscopy in the time domain is typically performed by recording the arrival time of photons either by using electronic time tagging or a gated detector. As such the temporal resolution is limited by the performance of the electronics to 100's of picoseconds. Here, we demonstrate a fluorescence lifetime measurement technique based on photon-bunching statistics with a resolution that is only dependent on the duration of the reference photon or laser pulse, which can readily reach the 1-0.1 picosecond timescale. A range of fluorescent dyes having lifetimes spanning from 1.6 to 7 picoseconds have been here measured with only ~1 second measurement duration. We corroborate the effectiveness of the technique by measuring the Newtonian viscosity of glycerol/water mixtures by means of a molecular rotor having over an order of magnitude variability in lifetime, thus introducing a new method for contact-free nanorheology. Accessing fluorescence lifetime information at such high temporal resolution opens a doorway for a wide range of fluorescent markers to be adopted for studying yet unexplored fast biological processes, as well as fundamental interactions such as lifetime shortening in resonant plasmonic devices., Comment: 8 pages, 5 figures

Fingerprint traits are widely recognized for their unique qualities and security benefits. Despite their extensive use, fingerprint features can be vulnerable to puppet attacks, where attackers manipulate a reluctant but genuine user into completing the authentication process. Defending against such attacks is challenging due to the coexistence of a legitimate identity and an illegitimate intent. In this paper, we propose PUPGUARD, a solution designed to guard against puppet attacks. This method is based on user behavioral patterns, specifically, the user needs to press the capture device twice successively with different fingers during the authentication process. PUPGUARD leverages both the image features of fingerprints and the timing characteristics of the pressing intervals to establish two-factor authentication. More specifically, after extracting image features and timing characteristics, and performing feature selection on the image features, PUPGUARD fuses these two features into a one-dimensional feature vector, and feeds it into a one-class classifier to obtain the classification result. This two-factor authentication method emphasizes dynamic behavioral patterns during the authentication process, thereby enhancing security against puppet attacks. To assess PUPGUARD's effectiveness, we conducted experiments on datasets collected from 31 subjects, including image features and timing characteristics. Our experimental results demonstrate that PUPGUARD achieves an impressive accuracy rate of 97.87% and a remarkably low false positive rate (FPR) of 1.89%. Furthermore, we conducted comparative experiments to validate the superiority of combining image features and timing characteristics within PUPGUARD for enhancing resistance against puppet attacks.

In 1971, Zel'dovich predicted the amplification of electromagnetic (EM) waves scattered by a rotating metallic cylinder, gaining mechanical rotational energy from the body. Since then, this phenomenon has been believed to be unobservable with electromagnetic fields due to technological difficulties in meeting the condition of amplification, that is, the cylinder must rotate faster than the frequency of the incoming radiation. Here, we show that this key piece of fundamental physics has been hiding in plain sight for the past 60 years in the physics of induction generators. We measure the amplification of an electromagnetic field, generated by a toroid LC-circuit, scattered by an aluminium cylinder spinning in the toroid gap. We show that when the Zel'dovich condition is met, the resistance induced by the cylinder becomes negative implying amplification of the incoming EM waves. These results reveal the connection between the concept of induction generators and the physics of this fundamental effect that was believed to be unobservable, and hence open new prospects towards testing the Zel'dovich mechanism in the quantum regime, as well as related quantum friction effects., Comment: 5 pages and 3 figure plus supplementary file

The Languini Kitchen serves as both a research collective and codebase designed to empower researchers with limited computational resources to contribute meaningfully to the field of language modelling. We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. The number of tokens on which a model is trained is defined by the model's throughput and the chosen compute class. Notably, this approach avoids constraints on critical hyperparameters which affect total parameters or floating-point operations. For evaluation, we pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. On it, we compare methods based on their empirical scaling trends which are estimated through experiments at various levels of compute. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput. While the GPT baseline achieves better perplexity throughout all our levels of compute, our LSTM baseline exhibits a predictable and more favourable scaling law. This is due to the improved throughput and the need for fewer training tokens to achieve the same decrease in test perplexity. Extrapolating the scaling laws leads of both models results in an intersection at roughly 50,000 accelerator hours. We hope this work can serve as the foundation for meaningful and reproducible language modelling research.

Adaptive optics (AO) has revolutionized imaging in {fields} from astronomy to microscopy by correcting optical aberrations. In label-free microscopes, however, conventional AO faces limitations due to the absence of guidestar and the need to select an optimization metric specific to the sample and imaging process. Here, we propose an AO approach leveraging correlations between entangled photons to directly correct the point spread function (PSF). This guidestar-free method is independent of the specimen and imaging modality. We demonstrate the imaging of biological samples in the presence of aberrations using a bright-field imaging setup operating with a source of spatially-entangled photon pairs. Our approach performs better than conventional AO in correcting specific aberrations, particularly those involving significant defocus. Our work improves AO for label-free microscopy and could play a major role in the development of quantum microscopes., Comment: Main manuscript: 9 pages, 5 figures

Recent work on deep reinforcement learning (DRL) has pointed out that algorithmic information about good policies can be extracted from offline data which lack explicit information about executed actions. For example, videos of humans or robots may convey a lot of implicit information about rewarding action sequences, but a DRL machine that wants to profit from watching such videos must first learn by itself to identify and recognize relevant states/actions/rewards. Without relying on ground-truth annotations, our new method called Deep State Identifier learns to predict returns from episodes encoded as videos. Then it uses a kind of mask-based sensitivity analysis to extract/identify important critical states. Extensive experiments showcase our method's potential for understanding and improving agent behavior. The source code and the generated datasets are available at https://github.com/AI-Initiative-KAUST/VideoRLCS., Comment: This paper was accepted to ICCV23

Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each other in a "mindstorm." Recent implementations of NN-based societies of minds consist of large language models (LLMs) and other NN-based experts communicating through a natural language interface. In doing so, they overcome the limitations of single LLMs, improving multimodal zero-shot reasoning. In these natural language-based societies of mind (NLSOMs), new agents -- all communicating through the same universal symbolic language -- are easily added in a modular fashion. To demonstrate the power of NLSOMs, we assemble and experiment with several of them (having up to 129 members), leveraging mindstorms in them to solve some practical AI tasks: visual question answering, image captioning, text-to-image synthesis, 3D generation, egocentric retrieval, embodied AI, and general language-based task solving. We view this as a starting point towards much larger NLSOMs with billions of agents-some of which may be humans. And with this emergence of great societies of heterogeneous minds, many new research questions have suddenly become paramount to the future of artificial intelligence. What should be the social structure of an NLSOM? What would be the (dis)advantages of having a monarchical rather than a democratic structure? How can principles of NN economies be used to maximize the total reward of a reinforcement learning NLSOM? In this work, we identify, discuss, and try to answer some of these questions., Comment: 9 pages in main text + 7 pages of references + 38 pages of appendices, 14 figures in main text + 13 in appendices, 7 tables in appendices

Hong-Ou-Mandel (HOM) interference, the bunching of two indistinguishable photons on a balanced beam-splitter, has emerged as a promising tool for quantum sensing. There is a need for wide spectral-bandwidth photon pairs (for high-resolution sensing) with high brightness (for fast sensing). Here we show the generation of photon-pairs with flexible spectral-bandwidth even using single-frequency, continuous-wave diode laser enabling high-precision, real-time sensing. Using 1-mm-long periodically-poled KTP crystal, we produced degenerate, photon-pairs with spectral-bandwidth of 163.42$\pm$1.68 nm resulting in a HOM-dip width of 4.01$\pm$0.04 $\mu$m to measure a displacement of 60 nm, and sufficiently high brightness to enable the measurement of vibrations with amplitude of $205\pm0.75$ nm and frequency of 8 Hz. Fisher-information and maximum likelihood estimation enables optical delay measurements as small as 4.97 nm with precision (Cram\'er-Rao bound) and accuracy of 0.89 and 0.54 nm, respectively, therefore showing HOM sensing capability for real-time, precision-augmented, in-field quantum sensing applications.

Nowadays, non-privacy small-scale motion detection has attracted an increasing amount of research in remote sensing in speech recognition. These new modalities are employed to enhance and restore speech information from speakers of multiple types of data. In this paper, we propose a dataset contains 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77-GHz frequency modulated continuous wave (FMCW) data from millimetre wave (mmWave) radar, and laser data. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words and 16 sentences. The dataset has been validated and has potential for the research of lip reading and multimodal speech recognition.

For widespread adoption, public security and surveillance systems must be accurate, portable, compact, and real-time, without impeding the privacy of the individuals being observed. Current systems broadly fall into two categories -- image-based which are accurate, but lack privacy, and RF signal-based, which preserve privacy but lack portability, compactness and accuracy. Our paper proposes mmSense, an end-to-end portable miniaturised real-time system that can accurately detect the presence of concealed metallic objects on persons in a discrete, privacy-preserving modality. mmSense features millimeter wave radar technology, provided by Google's Soli sensor for its data acquisition, and TransDope, our real-time neural network, capable of processing a single radar data frame in 19 ms. mmSense achieves high recognition rates on a diverse set of challenging scenes while running on standard laptop hardware, demonstrating a significant advancement towards creating portable, cost-effective real-time radar based surveillance systems., Comment: Accepted by ICASSP 2023