Search

Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https://github.com/SqueezeAILab/LLM2LLM ., Comment: ACL 2024

Modern computer vision pipelines handle large images in one of two sub-optimal ways: down-sampling or cropping. These two methods incur significant losses in the amount of information and context present in an image. There are many downstream applications in which global context matters as much as high frequency details, such as in real-world satellite imagery; in such cases researchers have to make the uncomfortable choice of which information to discard. We introduce xT, a simple framework for vision transformers which effectively aggregates global context with local details and can model large images end-to-end on contemporary GPUs. We select a set of benchmark datasets across classic vision tasks which accurately reflect a vision model's ability to understand truly large images and incorporate fine details over large scales and assess our method's improvement on them. xT is a streaming, two-stage architecture that adapts existing vision backbones and long sequence language models to effectively model large images without quadratic memory growth. We are able to increase accuracy by up to 8.6% on challenging classification tasks and $F_1$ score by 11.6 on context-dependent segmentation on images as large as 29,000 x 29,000 pixels., Comment: Accepted to the 2024 International Conference on Machine Learning (ICML)

Anomalous diffusion processes pose a unique challenge in classification and characterization. Previously (Mangalam et al., 2023, Physical Review Research 5, 023144), we established a framework for understanding anomalous diffusion using multifractal formalism. The present study delves into the potential of multifractal spectral features for effectively distinguishing anomalous diffusion trajectories from five widely used models: fractional Brownian motion, scaled Brownian motion, continuous time random walk, annealed transient time motion, and L\'evy walk. To accomplish this, we generate extensive datasets comprising $10^6$ trajectories from these five anomalous diffusion models and extract multiple multifractal spectra from each trajectory. Our investigation entails a thorough analysis of neural network performance, encompassing features derived from varying numbers of spectra. Furthermore, we explore the integration of multifractal spectra into traditional feature datasets, enabling us to assess their impact comprehensively. To ensure a statistically meaningful comparison, we categorize features into concept groups and train neural networks using features from each designated group. Notably, several feature groups demonstrate similar levels of accuracy, with the highest performance observed in groups utilizing moving-window characteristics and $p$-variation features. Multifractal spectral features, particularly those derived from three spectra involving different timescales and cutoffs, closely follow, highlighting their robust discriminatory potential. Remarkably, a neural network exclusively trained on features from a single multifractal spectrum exhibits commendable performance, surpassing other feature groups. Our findings underscore the diverse and potent efficacy of multifractal spectral features in enhancing classification of anomalous diffusion., Comment: 23 pages, 6 figures

Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. Furthermore, modality-specific encoders achieve impressive performances in their respective domains. This raises a central question: does an alignment exist between uni-modal vision and language encoders since they fundamentally represent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training. We frame this as a seeded graph-matching problem exploiting the semantic similarity between graphs and propose two methods - a Fast Quadratic Assignment Problem optimization, and a novel localized CKA metric-based matching/retrieval. We demonstrate the effectiveness of this on several downstream tasks including cross-lingual, cross-domain caption matching and image classification. Code available at github.com/mayug/0-shot-llm-vision., Comment: Accepted CVPR 2024

Among the statistical models employed to approximate nonlinear interactions in biological and psychological processes, one prominent framework is that of cascades. Despite decades of empirical work using multifractal formalisms, a fundamental question has persisted: Do the observed nonlinear interactions across scales owe their origin to multiplicative interactions, or do they inherently reside within the constituent processes? This study presents the results of rigorous numerical simulations that demonstrate the supremacy of multiplicative interactions over the intrinsic nonlinear properties of component processes. To elucidate this point, we conducted simulations of cascade time series featuring component processes operating at distinct timescales, each characterized by one of four properties: multifractal nonlinearity, multifractal linearity (obtained via the Iterative Amplitude Adjusted Wavelet Transform of multifractal nonlinearity), phase-randomized linearity (obtained via the Iterative Amplitude Adjustment Fourier Transform), and phase- and amplitude-randomized (obtained via shuffling). Our findings unequivocally establish that the multiplicative interactions among components, rather than the inherent properties of the component processes themselves, decisively dictate the multifractal emergent properties. Remarkably, even component processes exhibiting purely linear traits can generate nonlinear interactions across scales when these interactions assume a multiplicative nature. In stark contrast, additivity among component processes inevitably leads to a linear outcome. These outcomes provide a robust theoretical underpinning for current interpretations of multifractal nonlinearity, firmly anchoring its roots in the domain of multiplicative interactions across scales within biological and psychological processes., Comment: 27 pages, 9 figures

Large pretrained models are increasingly crucial in modern computer vision tasks. These models are typically used in downstream tasks by end-to-end finetuning, which is highly memory-intensive for tasks with high-resolution data, e.g., video understanding, small object detection, and point cloud analysis. In this paper, we propose Dynamic Reversible Dual-Residual Networks, or Dr$^2$Net, a novel family of network architectures that acts as a surrogate network to finetune a pretrained model with substantially reduced memory consumption. Dr$^2$Net contains two types of residual connections, one maintaining the residual structure in the pretrained models, and the other making the network reversible. Due to its reversibility, intermediate activations, which can be reconstructed from output, are cleared from memory during training. We use two coefficients on either type of residual connections respectively, and introduce a dynamic training strategy that seamlessly transitions the pretrained model to a reversible network with much higher numerical precision. We evaluate Dr$^2$Net on various pretrained models and various tasks, and show that it can reach comparable performance to conventional finetuning but with significantly less memory usage.

Human trajectory prediction is typically posed as a zero-shot generalization problem: a predictor is learnt on a dataset of human motion in training scenes, and then deployed on unseen test scenes. While this paradigm has yielded tremendous progress, it fundamentally assumes that trends in human behavior within the deployment scene are constant over time. As such, current prediction models are unable to adapt to scene-specific transient human behaviors, such as crowds temporarily gathering to see buskers, pedestrians hurrying through the rain and avoiding puddles, or a protest breaking out. We formalize the problem of scene-specific adaptive trajectory prediction and propose a new adaptation approach inspired by prompt tuning called latent corridors. By augmenting the input of any pre-trained human trajectory predictor with learnable image prompts, the predictor can improve in the deployment scene by inferring trends from extremely small amounts of new data (e.g., 2 humans observed for 30 seconds). With less than 0.1% additional model parameters, we see up to 23.9% ADE improvement in MOTSynth simulated data and 16.4% ADE in MOT and Wildtrack real pedestrian data. Qualitatively, we observe that latent corridors imbue predictors with an awareness of scene geometry and scene-specific human behaviors that non-adaptive predictors struggle to capture. The project website can be found at https://neerja.me/atp_latent_corridors/., Comment: Accepted to ECCV 2024. Project website can be found at https://neerja.me/atp_latent_corridors/

Multifractal formalisms provide an apt framework to study random cascades in which multifractal spectrum width $\Delta\alpha$ fluctuates depending on the number of estimable power-law relationships. Then again, multifractality without surrogate comparison can be ambiguous: the original measurement series' multifractal spectrum width $\Delta\alpha_\mathrm{Orig}$ can be sensitive to the series length, ergodicity-breaking linear temporal correlations (e.g., fractional Gaussian noise, $fGn$), or additive cascade dynamics. To test these threats, we built a suite of random cascades that differ by the length, type of noise (i.e., additive white Gaussian noise, $awGn$, or $fGn$), and mixtures of $awGn$ or $fGn$ across generations (progressively more $awGn$, progressively more $fGn$, and a random sampling by generation), and operations applying noise (i.e., addition vs. multiplication). The so-called ``multifractal nonlinearity'' $t_\mathrm{MF}$ (i.e., a $t$-statistic comparing $\Delta\alpha_\mathrm{Orig}$ and multifractal spectra width for phase-randomized linear surrogates $\Delta\alpha_\mathrm{Surr}$) is a robust indicator of random multiplicative rather than random additive cascade processes irrespective of the series length or type of noise. $t_\mathrm{MF}$ is more sensitive to the number of generations than the series length. Furthermore, the random additive cascades exhibited much stronger ergodicity breaking than all multiplicative analogs. Instead, ergodicity breaking in random multiplicative cascades more closely followed the ergodicity-breaking of the constituent noise types -- breaking ergodicity much less when arising from ergodic $awGn$ and more so for noise incorporating relatively more correlated $fGn$. Hence, $t_\mathrm{MF}$ is a robust multifractal indicator of multiplicative cascade processes and not spuriously sensitive to ergodicity breaking., Comment: 24 pages, 9 figures

We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources such as semantic segmentations and depth reconstructions without needing any meta-knowledge beyond the pixels. Once this wide variety of visual data (comprising 420 billion tokens) is represented as sequences, the model can be trained to minimize a cross-entropy loss for next token prediction. By training across various scales of model architecture and data diversity, we provide empirical evidence that our models scale effectively. Many different vision tasks can be solved by designing suitable visual prompts at test time., Comment: Website: https://yutongbai.com/lvm.html

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than 33% (random is 20%) on the EgoSchema multi-choice question answering task, while humans achieve about 76% accuracy. We posit that \name{}{}, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced for both public and commercial use under the Ego4D license at http://egoschema.github.io, Comment: https://egoschema.github.io/

The growing size of datasets and deep learning models has made faster and memory-efficient training crucial. Reversible transformers have recently been introduced as an exciting new method for extremely memory-efficient training, but they come with an additional computation overhead of activation re-computation in the backpropagation phase. We present PaReprop, a fast Parallelized Reversible Backpropagation algorithm that parallelizes the additional activation re-computation overhead in reversible training with the gradient computation itself in backpropagation phase. We demonstrate the effectiveness of the proposed PaReprop algorithm through extensive benchmarking across model families (ViT, MViT, Swin and RoBERTa), data modalities (Vision & NLP), model sizes (from small to giant), and training batch sizes. Our empirical results show that PaReprop achieves up to 20% higher training throughput than vanilla reversible training, largely mitigating the theoretical overhead of 25% lower throughput from activation recomputation in reversible training. Project page: https://tylerzhu.com/pareprop., Comment: Spotlight paper, T4V Workshop @ CVPR 2023

Any reliable biomarker has to be specific, generalizable, and reproducible across individuals and contexts. The exact values of such a biomarker must represent similar health states in different individuals and at different times within the same individual to result in the minimum possible false-positive and false-negative rates. The application of standard cut-off points and risk scores across populations hinges upon the assumption of such generalizability. Such generalizability, in turn, hinges upon this condition that the phenomenon investigated by current statistical methods is ergodic, i.e., its statistical measures converge over individuals and time within the finite limit of observations. However, emerging evidence indicates that biological processes abound with non-ergodicity, threatening this generalizability. Here, we present a solution for how to make generalizable inferences by deriving ergodic descriptions of non-ergodic phenomena. For this aim, we proposed capturing the origin of ergodicity-breaking in many biological processes: cascade dynamics. To assess our hypotheses, we embraced the challenge of identifying reliable biomarkers for heart disease and stroke, which, despite being the leading cause of death worldwide and decades of research, lacks reliable biomarkers and risk stratification tools. We showed that raw R-R interval data and its common descriptors based on mean and variance are non-ergodic and non-specific. On the other hand, the cascade-dynamical descriptors, the Hurst exponent encoding linear temporal correlations, and multifractal nonlinearity encoding nonlinear interactions across scales described the non-ergodic heart rate variability ergodically and were specific. This study inaugurates applying the critical concept of ergodicity in discovering and applying digital biomarkers of health and disease., Comment: 24 pages, 6 figures, 1 table

Discrete Event Modelling of Embedded Systems (DEMES) is a development methodology based on the Discrete Event Systems (DEVS) specification that improves the time -to-market by simplifying the development and testing of embedded systems. CADMIUM is a C++ header-only library developed at Carleton University that helps simulate models built using the DEVS specification. RT-CADMIUM is a fork of CADMIUM that provides a development framework that helps users develop systems using the DEMES technology. RT-CADMIUM, however, has a limited scope of deployment due to the use of Mbed OS as its Hardware Abstraction Layer (HAL). This paper provides the methodology for porting the RT-CADMIUM library to a different platform (ESP32 specifically). This paper also portrays the performance improvements gained due to this adoption.

There has been a longstanding belief that generation can facilitate a true understanding of visual data. In line with this, we revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE). Our approach is capable of (i) serving as a strong initialization for downstream recognition tasks, (ii) conducting high-quality image inpainting, and (iii) being effortlessly extended to video where it produces state-of-the-art classification accuracy. We further perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders., Comment: Tech report. Project page: https://weichen582.github.io/diffmae.html

The human gut microbiota is a complex ecosystem that affects a range of human physiology. In order to explore the dynamics of the human gut microbiota, we used a system of ordinary differential equations to model mathematically the biomass of three microorganism populations: $\textit{Bacteroides thetaiotaomicron}$, $\textit{Eubacterium rectale}$, and $\textit{Methanobrevibacter smithii}$. Additionally, we modeled the concentrations of relevant nutrients necessary to sustain these populations over time. Our model highlights the interactions and the competition among these three species. These three microorganisms were specifically chosen due to the system's end product, butyrate, which is a short chain fatty acid that aids in developing and maintaining the intestinal barrier in the human gut. The basis of our mathematical model assumes the gut is structured such that bacteria and nutrients exit the gut at a rate proportional to its volume, the rate of volumetric flow, and the biomass or concentration of the particular population or nutrient. We performed global sensitivity analyses using Sobol' sensitivities to estimate the relative importance of model parameters on simulation results.

Canonical characterization techniques that rely upon mean squared displacement ($\mathrm{MSD}$) break down for non-ergodic processes, making it challenging to characterize anomalous diffusion from an individual time-series measurement. Non-ergodicity reigns when the time-averaged mean square displacement $\mathrm{TA}$-$\mathrm{MSD}$ differs from the ensemble-averaged mean squared displacement $\mathrm{EA}$-$\mathrm{MSD}$ even in the limit of long measurement series. In these cases, the typical theoretical results for ensemble averages cannot be used to understand and interpret data acquired from time averages. The difficulty then lies in obtaining statistical descriptors of the measured diffusion process that are not non-ergodic. We show that linear descriptors such as the standard deviation ($SD$), coefficient of variation ($CV$), and root mean square ($RMS$) break ergodicity in proportion to non-ergodicity in the diffusion process. In contrast, time series of descriptors addressing sequential structure and its potential nonlinearity: multifractality change in a time-independent way and fulfill the ergodic assumption, largely independent of the time series' non-ergodicity. We show that these findings follow the multiplicative cascades underlying these diffusion processes. Adding fractal and multifractal descriptors to typical linear descriptors would improve the characterization of anomalous diffusion processes. Two particular points bear emphasis here. First, as an appropriate formalism for encoding the nonlinearity that might generate non-ergodicity, multifractal modeling offers descriptors that can behave ergodically enough to meet the needs of linear modeling. Second, this capacity to describe non-ergodic processes in ergodic terms offers the possibility that multifractal modeling could unify several disparate non-ergodic diffusion processes into a common framework., Comment: 24 pages; 10 figures

The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced, Comment: NeurIPS 2023

We present Reversible Vision Transformers, a memory efficient architecture design for visual recognition. By decoupling the GPU memory requirement from the depth of the model, Reversible Vision Transformers enable scaling up architectures with efficient memory usage. We adapt two popular models, namely Vision Transformer and Multiscale Vision Transformers, to reversible variants and benchmark extensively across both model sizes and tasks of image classification, object detection and video classification. Reversible Vision Transformers achieve a reduced memory footprint of up to 15.5x at roughly identical model complexity, parameters and accuracy, demonstrating the promise of reversible vision transformers as an efficient backbone for hardware resource limited training regimes. Finally, we find that the additional computational burden of recomputing activations is more than overcome for deeper models, where throughput can increase up to 2.3x over their non-reversible counterparts. Full code and trained models are available at https://github.com/facebookresearch/slowfast. A simpler, easy to understand and modify version is also available at https://github.com/karttikeya/minREV, Comment: Oral at CVPR 2022, updated version

The Bayesian Hurst-Kolmogorov (HK) method estimates the Hurst exponent of a time series more accurately than the age-old detrended fluctuation analysis (DFA), especially when the time series is short. However, this advantage comes at the cost of computation time. The computation time increases exponentially with $N$, easily exceeding several hours for $N = 1024$, limiting the utility of the HK method in real-time paradigms, such as biofeedback and brain-computer interfaces. To address this issue, we have provided data on the estimation accuracy of $H$ for synthetic time series as a function of \textit{a priori} known values of $H$, the time series length, and the simulated sample size from the posterior distribution -- a critical step in the Bayesian estimation method. The simulated sample from the posterior distribution as small as $n = 25$ suffices to estimate $H$ with reasonable accuracy for a time series as short as $256$ measurements. Using a larger simulated sample from the posterior distribution -- i.e., $n > 50$ -- provides only marginal gain in accuracy, which might not be worth trading off with computational efficiency. We suggest balancing the simulated sample size from the posterior distribution of $H$ with the computational resources available to the user, preferring a minimum of $n = 50$ and opting for larger sample sizes based on time and resource constraints, Comment: 17 pages, 2 figures

Detrended Fluctuation Analysis (DFA) is the most popular fractal analytical technique used to evaluate the strength of long-range correlations in empirical time series in terms of the Hurst exponent, $H$. Specifically, DFA quantifies the linear regression slope in log-log coordinates representing the relationship between the time series' variability and the number of timescales over which this variability is computed. We compared the performance of two methods of fractal analysis -- the current gold standard, DFA, and a Bayesian method that is not currently well-known in behavioral sciences: the Hurst-Kolmogorov (HK) method -- in estimating the Hurst exponent of synthetic and empirical time series. Simulations demonstrate that the HK method consistently outperforms DFA in three important ways. The HK method: (i) accurately assesses long-range correlations when the measurement time series is short, (ii) shows minimal dispersion about the central tendency, and (iii) yields a point estimate that does not depend on the length of the measurement time series or its underlying Hurst exponent. Comparing the two methods using empirical time series from multiple settings further supports these findings. We conclude that applying DFA to synthetic time series and empirical time series during brief trials is unreliable and encourage the systematic application of the HK method to assess the Hurst exponent of empirical time series in behavioral sciences., Comment: 50 pages, 14 figures, 6 tables

Are multimodal inputs necessary for grammar induction? Recent work has shown that multimodal training inputs can improve grammar induction. However, these improvements are based on comparisons to weak text-only baselines that were trained on relatively little textual data. To determine whether multimodal inputs are needed in regimes with large amounts of textual training data, we design a stronger text-only baseline, which we refer to as LC-PCFG. LC-PCFG is a C-PFCG that incorporates em-beddings from text-only large language models (LLMs). We use a fixed grammar family to directly compare LC-PCFG to various multi-modal grammar induction methods. We compare performance on four benchmark datasets. LC-PCFG provides an up to 17% relative improvement in Corpus-F1 compared to state-of-the-art multimodal grammar induction methods. LC-PCFG is also more computationally efficient, providing an up to 85% reduction in parameter count and 8.8x reduction in training time compared to multimodal approaches. These results suggest that multimodal inputs may not be necessary for grammar induction, and emphasize the importance of strong vision-free baselines for evaluating the benefit of multimodal approaches., Comment: NAACL Findings 2024

Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content. Given limited GPU memory, training TAL end to end (i.e., from videos to predictions) on long videos is a significant challenge. Most methods can only train on pre-extracted features without optimizing them for the localization problem, consequently limiting localization performance. In this work, to extend the potential in TAL networks, we propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL. Re2TAL builds a backbone with reversible modules, where the input can be recovered from the output such that the bulky intermediate activations can be cleared from memory during training. Instead of designing one single type of reversible module, we propose a network rewiring mechanism, to transform any module with a residual connection to a reversible module without changing any parameters. This provides two benefits: (1) a large variety of reversible networks are easily obtained from existing and even future model designs, and (2) the reversible models require much less training effort as they reuse the pre-trained parameters of their original non-reversible versions. Re2TAL, only using the RGB modality, reaches 37.01% average mAP on ActivityNet-v1.3, a new state-of-the-art record, and mAP 64.9% at tIoU=0.5 on THUMOS-14, outperforming all other RGB-only methods.

We study the bound orbit conditions for equatorial and eccentric orbits around a Kerr black hole both in the parameter space ($E$, $L$, $a$) representing the energy, angular momentum of the test particle, and spin of the black hole, and also ($e$, $\mu$, $a$) space representing the eccentricity, inverse-latus rectum of the orbit, and spin. We apply these conditions and implement the relativistic precession (RP) model to M82X-1, which is an Intermediate-mass black hole (IMBH) system, where two high-frequency Quasi-Periodic Oscillations (HFQPOs) and a low-frequency QPO were simultaneously observed. Assuming that the QPO frequencies can also be generated by equatorial and eccentric trajectories, we calculate the probability distributions to infer $e$, $a$, and periastron distance, $r_p$, of the orbit giving rise to simultaneous QPOs. We find that an eccentric orbit solution is possible in the region between innermost stable circular orbit (ISCO) and the marginally bound circular orbit (MBCO) for $e=0.2768^{+0.0657}_{-0.0451}$, $a=0.2897 \pm 0.0087$, and $r_p=4.6164^{+0.0694}_{-0.1259}$., Comment: 7 pages, 2 figures, Conference: Proceedings of the MG15 Meeting on General Relativity

We derive the conditions for a non-equatorial eccentric bound orbit to exist around a Kerr black hole in two-parameter spaces: the energy, angular momentum of the test particle, spin of the black hole, and Carter's constant space ($E$, $L$, $a$, $Q$), and eccentricity, inverse-latus rectum space ($e$, $\mu$, $a$, $Q$). These conditions distribute various kinds of bound orbits in different regions of the ($E$, $L$) and ($e$, $\mu$) planes, depending on which pair of roots of the effective potential forms a bound orbit. We provide a prescription to select these parameters for bound orbits, which are useful inputs to study bound trajectory evolution in various astrophysical applications like simulations of gravitational wave emission from extreme-mass ratio inspirals, relativistic precession around black holes, and the study of gyroscope precession as a test of general relativity., Comment: 7 pages, 2 figures, Conference: Proceedings of the MG15 Meeting on General Relativity

This technical report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a "Frame-Clip Consistency" loss, which ensures the flow of structured information between images and videos. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error., Comment: Ego4D CVPR22 Object State Localization challenge. arXiv admin note: substantial text overlap with arXiv:2206.06346

Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable. At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task? We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a \emph{Frame-Clip Consistency} loss, which ensures the flow of structured information between images and videos. We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges. SViT shows strong performance improvements on multiple video understanding tasks and datasets. Furthermore, it won in the Ego4D CVPR'22 Object State Localization challenge. For code and pretrained models, visit the project page at \url{https://eladb3.github.io/SViT/}, Comment: Tech report

The brain-as-computer metaphor has anchored the professed computational nature of the mind, wresting it down from the intangible logic of Platonic philosophy to a material basis for empirical science. However, as with many long-lasting metaphors in science, the computer metaphor has been explored and stretched long enough to reveal its boundaries. These boundaries highlight widening gaps in our understanding of the brain's role in an organism's goal-directed, intelligent behaviors and thoughts. In search of a more appropriate metaphor that reflects the potentially noncomputable functions of mind and brain, eight author groups answer the following questions: (1) What do we understand by the computer metaphor of the brain and cognition? (2) What are some of the limitations of this computer metaphor? (3) What metaphor should replace the computational metaphor? (4) What findings support alternative metaphors? Despite agreeing about feeling the strain of the strictures of computer metaphors, the authors suggest an exciting diversity of possible metaphoric options for future research into the mind and brain., Comment: 157 pages, 19 figures

The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that captures both local and global features. However, through a series of systematic studies, we find that the Conformer architecture's design choices are not optimal. After re-examining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the-art ASR models under the same training schemes. In particular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal U-Net structure which reduces the cost of the multi-head attention modules on long sequences, and (ii) a simpler block structure of multi-head attention or convolution modules followed up by feed-forward module instead of the Macaron structure proposed in Conformer. Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the activations in the convolutional block, (ii) removes redundant Layer Normalization operations, and (iii) incorporates an efficient depthwise down-sampling layer to efficiently sub-sample the input signal. Squeezeformer achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on LibriSpeech test-other without external language models, which are 3.1%, 1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our code is open-sourced and available online., Comment: NeurIPS 2022