Search

IoT is rapidly growing from small-scale apps to large-scale apps. Small-scale apps employ short-range radios like Zigbee,BLE while large-scale apps employ long-range radios like LoRa,NB-IoT. The other upcoming category of apps like P2P energy-trade in smart homes are termed mesoscale IoT apps. There are no specialized radios for these apps. They either use short/long-range radios. To close this gap, we explored mesoscale apps using the COTS IoT radios available. Our qualitative analysis identifies Zigbee and LoRa as potential candidates. Our quantitative analysis on single and multi-hop topologies showed that Zigbee and LoRa achieve competitive throughput at a distance of 500-1200m from the gateway. A fundamental finding of these analyses is that a multi-radio system that can efficiently switch between Zigbee and LoRa performs better than the single-radio systems. However, instantaneously selecting and switching to a high-throughput radio during transmission is not trivial because of the erratic link quality dynamics. To address this issue, we developed MARS, that uses path quality metrics to instantaneously select the high-throughput radio during transmission. However, realizing MARS on resource-constrained end devices entails the challenge of obtaining instantaneous path-quality metrics. Traditional path quality estimation is not instantaneous due to propagation and queuing delays. We overcome this challenge by showing that collecting local path metrics as input to our decision trees provides sufficient information to instantaneously identify the high-throughput radio. The radio selector of MARS is powered by TAO-CART trees. The evaluation of MARS on a large-scale mesh topology at two different locations shows that MARS can efficiently identify and switch to the high-throughput radio during transmission, leading to an average throughput gain of 48.2% and 49.79% over its competitors.

We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io

Automatic static cost analysis infers information about the resources used by programs without actually running them with concrete data, and presents such information as functions of input data sizes. Most of the analysis tools for logic programs (and many for other languages), as CiaoPP, are based on setting up recurrence relations representing (bounds on) the computational cost of predicates, and solving them to find closed-form functions. Such recurrence solving is a bottleneck in current tools: many of the recurrences that arise during the analysis cannot be solved with state-of-the-art solvers, including Computer Algebra Systems (CASs), so that specific methods for different classes of recurrences need to be developed. We address such a challenge by developing a novel, general approach for solving arbitrary, constrained recurrence relations, that uses machine-learning (sparse-linear and symbolic) regression techniques to guess a candidate closed-form function, and a combination of an SMT-solver and a CAS to check if it is actually a solution of the recurrence. Our prototype implementation and its experimental evaluation within the context of the CiaoPP system show quite promising results. Overall, for the considered benchmarks, our approach outperforms state-of-the-art cost analyzers and recurrence solvers, and solves recurrences that cannot be solved by them. Under consideration in Theory and Practice of Logic Programming (TPLP)., Comment: Under consideration in Theory and Practice of Logic Programming (TPLP). Extended, revised version of our work published in ICLP (Klemen et al. 2023, arXiv:2309.07259). arXiv admin note: text overlap with arXiv:2309.07259

To endow models with greater understanding of physics and motion, it is useful to enable them to perceive how solid surfaces move and deform in real scenes. This can be formalized as Tracking-Any-Point (TAP), which requires the algorithm to track any point on solid surfaces in a video, potentially densely in space and time. Large-scale groundtruth training data for TAP is only available in simulation, which currently has a limited variety of objects and motion. In this work, we demonstrate how large-scale, unlabeled, uncurated real-world data can improve a TAP model with minimal architectural changes, using a selfsupervised student-teacher setup. We demonstrate state-of-the-art performance on the TAP-Vid benchmark surpassing previous results by a wide margin: for example, TAP-Vid-DAVIS performance improves from 61.3% to 67.4%, and TAP-Vid-Kinetics from 57.2% to 62.5%. For visualizations, see our project webpage at https://bootstap.github.io/

The First Perception Test challenge was held as a half-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2023, with the goal of benchmarking state-of-the-art video models on the recently proposed Perception Test benchmark. The challenge had six tracks covering low-level and high-level tasks, with both a language and non-language interface, across video, audio, and text modalities, and covering: object tracking, point tracking, temporal action localisation, temporal sound localisation, multiple-choice video question-answering, and grounded video question-answering. We summarise in this report the task descriptions, metrics, baselines, and results.

We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters spanning $0.4-0.9\mu\mathrm{m}$) and novel JWST images with 14 filters spanning $0.8-5\mu\mathrm{m}$, including 7 medium-band filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data at $>2.3\mu\mathrm{m}$ to construct an ultradeep image, reaching as deep as $\approx31.4$ AB mag in the stack and 30.3-31.0 AB mag ($5\sigma$, $r=0.1"$ circular aperture) in individual filters. We measure photometric redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts $z=11.5-15$. These objects show compact half-light radii of $R_{1/2}\sim50-200$pc, stellar masses of $M_{\star}\sim10^7-10^8 M_{\odot}$, and star-formation rates of $\mathrm{SFR}\sim0.1-1\,M_{\odot}\,\mathrm{yr}^{-1}$. Our search finds no candidates at $15

We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This poses great challenges given the high correlation between consecutive video frames and there is very little prior work on it. Our framework allows us to do a first deep dive into the topic and includes a collection of streams and tasks composed from two existing video datasets, plus methodology for performance evaluation that considers both adaptation and generalization. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation as well as between arbitrary tasks, without ever requiring changes to models and always using the same pixel loss. Equipped with this framework we obtained large single-stream learning gains from pre-training with a novel family of future prediction tasks, found that momentum hurts, and that the pace of weight updates matters. The combination of these insights leads to matching the performance of IID learning with batch size 1, when using the same architecture and without costly replay buffers., Comment: CVPR camera ready version

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks., Comment: Accepted to ICLR 2024 (Best paper honorable mention). Project Page: https://shashankvkt.github.io/dora

The normal-state conductivity and superconducting critical temperature of oxygen-deficient YBa2Cu3O7-x can be persistently enhanced by illumination. Strongly debated for years, the origin of those effects -- termed persistent photoconductivity (PPC) and photosuperconductivity (PPS) -- has remained an unsolved critical problem, whose comprehension may provide key insights to harness the origin of high-temperature superconductivity itself. Here we make essential steps toward understanding PPS. While the models proposed so far assume that it is caused by a carrier-density increase (photodoping) observed concomitantly, our experiments contradict such conventional belief: we demonstrate that it is instead linked to a photo-induced decrease of the electronic scattering rate. Furthermore, we find that the latter effect and photodoping are completely disconnected and originate from different microscopic mechanisms since they present different wavelength and oxygen-content dependencies as well as strikingly different relaxation dynamics. Besides helping disentangle photodoping, PPC, and PPS, our results provide new evidence for the intimate relation between critical temperature and scattering rate, a key ingredient in modern theories on high-temperature superconductivity.

The field of Artificial Intelligence has witnessed remarkable progress in recent years, especially with the emergence of powerful large language models (LLMs) based on the transformer architecture. Cloud-based LLMs, such as OpenAI's ChatGPT, offer impressive capabilities but come with concerns regarding latency and privacy due to network dependencies. This article presents an innovative approach to LLM inference, envisioning a future where LLMs with billions of parameters can be executed directly on mobile devices without network connectivity. The article showcases a fine-tuned GPT LLM with 3 billion parameters that can operate smoothly on devices with as low as 4GB of memory. Through the integration of native code and model quantization techniques, the application not only serves as a general-purpose assistant but also facilitates seamless mobile interactions with text-to-actions features. The article provides insights into the training pipeline, implementation details, test results, and future directions of on-device LLM inference. This breakthrough technology opens up possibilities for empowering users with sophisticated AI capabilities while preserving their privacy and eliminating latency concerns.

In recent years, a certain type of problems have become of interest where one wants to query a trained classifier. Specifically, one wants to find the closest instance to a given input instance such that the classifier's predicted label is changed in a desired way. Examples of these ``inverse classification'' problems are counterfactual explanations, adversarial examples and model inversion. All of them are fundamentally optimization problems over the input instance vector involving a fixed classifier, and it is of interest to achieve a fast solution for interactive or real-time applications. We focus on solving this problem efficiently for two of the most widely used classifiers: logistic regression and softmax classifiers. Owing to special properties of these models, we show that the optimization can be solved in closed form for logistic regression, and iteratively but extremely fast for the softmax classifier. This allows us to solve either case exactly (to nearly machine precision) in a runtime of milliseconds to around a second even for very high-dimensional instances and many classes.

Automatic static cost analysis infers information about the resources used by programs without actually running them with concrete data, and presents such information as functions of input data sizes. Most of the analysis tools for logic programs (and other languages) are based on setting up recurrence relations representing (bounds on) the computational cost of predicates, and solving them to find closed-form functions that are equivalent to (or a bound on) them. Such recurrence solving is a bottleneck in current tools: many of the recurrences that arise during the analysis cannot be solved with current solvers, such as Computer Algebra Systems (CASs), so that specific methods for different classes of recurrences need to be developed. We address such a challenge by developing a novel, general approach for solving arbitrary, constrained recurrence relations, that uses machine-learning sparse regression techniques to guess a candidate closed-form function, and a combination of an SMT-solver and a CAS to check whether such function is actually a solution of the recurrence. We have implemented a prototype and evaluated it with recurrences generated by a cost analysis system (the one in CiaoPP). The experimental results are quite promising, showing that our approach can find closed-form solutions, in a reasonable time, for classes of recurrences that cannot be solved by such a system, nor by current CASs., Comment: In Proceedings ICLP 2023, arXiv:2308.14898

Large Language Models (LLMs) have found widespread applications in various domains, including web applications, where they facilitate human interaction via chatbots with natural language interfaces. Internally, aided by an LLM-integration middleware such as Langchain, user prompts are translated into SQL queries used by the LLM to provide meaningful responses to users. However, unsanitized user prompts can lead to SQL injection attacks, potentially compromising the security of the database. Despite the growing interest in prompt injection vulnerabilities targeting LLMs, the specific risks of generating SQL injection attacks through prompt injections have not been extensively studied. In this paper, we present a comprehensive examination of prompt-to-SQL (P$_2$SQL) injections targeting web applications based on the Langchain framework. Using Langchain as our case study, we characterize P$_2$SQL injections, exploring their variants and impact on application security through multiple concrete examples. Furthermore, we evaluate 7 state-of-the-art LLMs, demonstrating the pervasiveness of P$_2$SQL attacks across language models. Our findings indicate that LLM-integrated applications based on Langchain are highly susceptible to P$_2$SQL injection attacks, warranting the adoption of robust defenses. To counter these attacks, we propose four effective defense techniques that can be integrated as extensions to the Langchain framework. We validate the defenses through an experimental evaluation with a real-world use case application., Comment: 12 pages, 3 figures, 3 tables, 5 listings

An important challenge in active matter lies in harnessing useful global work from entities that produce work locally, e.g., via self-propulsion. We investigate here the active matter version of a classical capillary rise effect, by considering a non-phase separated sediment of self-propelled Janus colloids in contact with a vertical wall. We provide experimental evidence of an unexpected and dynamic adsorption layer at the wall. Additionally, we develop a complementary numerical model that recapitulates the experimental observations. We show that an adhesive and aligning wall enhances the pre-existing polarity heterogeneity within the bulk, enabling polar active particles to climb up a wall against gravity, effectively powering a global flux. Such steady-state flux has no equivalent in a passive wetting layer.

Cuprate superconductors are strongly sensitive materials to disorder and oxygen stoichiometry; even minute variations of those parameters drastically change their electronic properties. Here we exploit this characteristic to engineer a memristive device based on the high-T_C superconductor YBa2Cu3O7-x (YBCO), in which local changes of the oxygen content and induced disorder are exploited to produce memory effects. These effects are triggered electrically in a three-terminal device whose structure is reminiscent of a transistor, consisting of a YBCO channel and an Al gate. The Al/YBCO interface, which controls the gate conductance, displays a giant, bipolar, reversible switching across a continuum of non-volatile conductance states that span over two Decades. This phenomenon is controlled by the gate voltage magnitude and is caused by oxygen exchange between YBCO and Al. Concomitantly, the channel shows a gradual, irreversible superconductor-to-insulator transition that retains a memory of the power dissipated in the device, and can be explained by induced bulk disorder. The observed effects, and the understanding of the interplay between the underlying mechanisms, constitute interesting ingredients for the design and realization of novel memristors and switches for superconducting electronics.

We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model facilitates fast inference on long and high-resolution video sequences. On a modern GPU, our implementation has the capacity to track points faster than real-time, and can be flexibly extended to higher-resolution videos. Given the high-quality trajectories extracted from a large dataset, we demonstrate a proof-of-concept diffusion model which generates trajectories from static images, enabling plausible animations. Visualizations, source code, and pretrained models can be found on our project webpage., Comment: Published at ICCV 2023

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test, Comment: 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks

We present a ceramic material based on hematite (Fe$_2$O$_3$) and zirconia stabilized with yttria at 8% molar (YSZ), that exhibits stable electrical properties with transported charge and that can be tuned to the resistivities necessary to induce spark-quenching in gaseous detectors ($\rho = 10^9-10^{12}$ $\Omega \cdot$cm), from room temperature down to the liquid-vapor coexistence point of nitrogen (77 K). It, thus, allows covering the operating temperatures of most immediate interest to gaseous instrumentation. The ceramics have been produced in a region of mass concentrations far from what has been usually explored in literature: optimal characteristics are achieved for Fe$_2$O$_3$ concentrations of 75%wt (LAr boiling temperature), 35%wt (LXe boiling temperature), and 100%wt (room temperature). The nine order of magnitude enhancement observed for the electrical conductivity of the mixed phases relative to that of pure Fe$_2$O$_3$ is startling, however it can be qualitatively understood based on existing literature. Plates of 4 cm x 4 cm have been manufactured and, prior to this work, operated in-detector at the LXe boiling point (165 K), demonstrating spark-free operation. Preliminary results obtained for the first time on a spark-protected amplification structure (RP-WELL) at around the LAr boiling point (90 K) are now presented, too.

We consider finding a counterfactual explanation for a classification or regression forest, such as a random forest. This requires solving an optimization problem to find the closest input instance to a given instance for which the forest outputs a desired value. Finding an exact solution has a cost that is exponential on the number of leaves in the forest. We propose a simple but very effective approach: we constrain the optimization to only those input space regions defined by the forest that are populated by actual data points. The problem reduces to a form of nearest-neighbor search using a certain distance on a certain dataset. This has two advantages: first, the solution can be found very quickly, scaling to large forests and high-dimensional data, and enabling interactive use. Second, the solution found is more likely to be realistic in that it is guided towards high-density areas of input space., Comment: A shorter version of this paper appears in AAAI 2023

Chip-scale, high-energy optical pulse generation is becoming increasingly important as we expand activities into hard to reach areas such as space and deep ocean. Q-switching of the laser cavity is the best known technique for generating high-energy pulses, and typically such systems are in the realm of large bench-top solid-state lasers and fiber lasers, especially in the long wavelength range >1.8 um, thanks to their large energy storage capacity. However, in integrated photonics, the very property of tight mode confinement, that enables a small form factor, becomes an impediment to high energy application due to small optical mode cross-section. In this work, we demonstrate complementary metal-oxide-semiconductor (CMOS) compatible, rare-earth gain based large mode area (LMA) passively Q-switched laser in a compact footprint. We demonstrate high on-chip output pulse energy of >150 nJ in single transverse fundamental mode in the eye-safe window (1.9 um), with a slope efficiency ~ 40% in a footprint of ~9 mm2. The high energy pulse generation demonstrated in this work is comparable or in many cases exceeds Q-switched fiber lasers. This bodes well for field applications in medicine and space.

We respond to Tetlock et al. (2022) showing 1) how expert judgment fails to reflect tail risk, 2) the lack of compatibility between forecasting tournaments and tail risk assessment methods (such as extreme value theory). More importantly, we communicate a new result showing a greater gap between the properties of tail expectation and those of the corresponding probability., Comment: Accepted, International Journal of Forecasting

Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering. The resulting representations are however fully entangled throughout the network, which may not always be desirable: in learning, contrastive audio-visual self-supervised learning requires independent audio and visual features to operate, otherwise learning collapses; in inference, evaluation of audio-visual models should be possible on benchmarks having just audio or just video. In this paper, we introduce Zorro, a technique that uses masks to control how inputs from each modality are routed inside Transformers, keeping some parts of the representation modality-pure. We apply this technique to three popular transformer-based architectures (ViT, Swin and HiP) and show that with contrastive pre-training Zorro achieves state-of-the-art results on most relevant benchmarks for multimodal tasks (AudioSet and VGGSound). Furthermore, the resulting models are able to perform unimodal inference on both video and audio benchmarks such as Kinetics-400 or ESC-50.