Search

Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at \texttt{https://github.com/snuhcs/Papez}, Comment: 5 pages. Accepted by ICASSP 2023

Spiking neural network (SNN) is studied in multidisciplinary domains to (i) enable order-of-magnitudes energy-efficient AI inference and (ii) computationally simulate neuro-scientific mechanisms. The lack of discrete theory obstructs the practical application of SNN by limiting its performance and nonlinearity support. We present a new optimization-theoretic perspective of the discrete dynamics of spiking neurons. We prove that a discrete dynamical system of simple integrate-and-fire models approximates the sub-gradient method over unconstrained optimization problems. We practically extend our theory to introduce a novel sign gradient descent (signGD)-based neuronal dynamics that can (i) approximate diverse nonlinearities beyond ReLU and (ii) advance ANN-to-SNN conversion performance in low time steps. Experiments on large-scale datasets show that our technique achieves (i) state-of-the-art performance in ANN-to-SNN conversion and (ii) is the first to convert new DNN architectures, e.g., ConvNext, MLP-Mixer, and ResMLP. We publicly share our source code at https://github.com/snuhcs/snn_signgd ., Comment: 37 pages, 41 figures, to be published as an ICML 2024 paper

In this paper, we present Mondrian, an edge system that enables high-performance object detection on high-resolution video streams. Many lightweight models and system optimization techniques have been proposed for resource-constrained devices, but they do not fully utilize the potential of the accelerators over dynamic, high-resolution videos. To enable such capability, we devise a novel Compressive Packed Inference to minimize per-pixel processing costs by selectively determining the necessary pixels to process and combining them to maximize processing parallelism. In particular, our system quickly extracts ROIs and dynamically shrinks them, reflecting the effect of the fast-changing characteristics of objects and scenes. It then intelligently combines such scaled ROIs into large canvases to maximize the utilization of inference accelerators such as GPU. Evaluation across various datasets, models, and devices shows Mondrian outperforms state-of-the-art baselines (e.g., input rescaling, ROI extractions, ROI extractions+batching) by 15.0-19.7% higher accuracy, leading to $\times$6.65 higher throughput than frame-wise inference for processing various 1080p video streams. We will release the code after the paper review.

We present EgoTAP, a heatmap-to-3D pose lifting method for highly accurate stereo egocentric 3D pose estimation. Severe self-occlusion and out-of-view limbs in egocentric camera views make accurate pose estimation a challenging problem. To address the challenge, prior methods employ joint heatmaps-probabilistic 2D representations of the body pose, but heatmap-to-3D pose conversion still remains an inaccurate process. We propose a novel heatmap-to-3D lifting method composed of the Grid ViT Encoder and the Propagation Network. The Grid ViT Encoder summarizes joint heatmaps into effective feature embedding using self-attention. Then, the Propagation Network estimates the 3D pose by utilizing skeletal information to better estimate the position of obscure joints. Our method significantly outperforms the previous state-of-the-art qualitatively and quantitatively demonstrated by a 23.9\% reduction of error in an MPJPE metric. Our source code is available in GitHub., Comment: 16 pages, 9 figures, to be published as CVPR 2024 paper

We present Ego3DPose, a highly accurate binocular egocentric 3D pose reconstruction system. The binocular egocentric setup offers practicality and usefulness in various applications, however, it remains largely under-explored. It has been suffering from low pose estimation accuracy due to viewing distortion, severe self-occlusion, and limited field-of-view of the joints in egocentric 2D images. Here, we notice that two important 3D cues, stereo correspondences, and perspective, contained in the egocentric binocular input are neglected. Current methods heavily rely on 2D image features, implicitly learning 3D information, which introduces biases towards commonly observed motions and leads to low overall accuracy. We observe that they not only fail in challenging occlusion cases but also in estimating visible joint positions. To address these challenges, we propose two novel approaches. First, we design a two-path network architecture with a path that estimates pose per limb independently with its binocular heatmaps. Without full-body information provided, it alleviates bias toward trained full-body distribution. Second, we leverage the egocentric view of body limbs, which exhibits strong perspective variance (e.g., a significantly large-size hand when it is close to the camera). We propose a new perspective-aware representation using trigonometry, enabling the network to estimate the 3D orientation of limbs. Finally, we develop an end-to-end pose reconstruction network that synergizes both techniques. Our comprehensive evaluations demonstrate that Ego3DPose outperforms state-of-the-art models by a pose estimation error (i.e., MPJPE) reduction of 23.1% in the UnrealEgo dataset. Our qualitative results highlight the superiority of our approach across a range of scenarios and challenges., Comment: 12 pages, 10 figures, to be published as SIGGRAPH Asia 2023 Conference Papers

The initial years of an infant's life are known as the critical period, during which the overall development of learning performance is significantly impacted due to neural plasticity. In recent studies, an AI agent, with a deep neural network mimicking mechanisms of actual neurons, exhibited a learning period similar to human's critical period. Especially during this initial period, the appropriate stimuli play a vital role in developing learning ability. However, transforming human cognitive bias into an appropriate shaping reward is quite challenging, and prior works on critical period do not focus on finding the appropriate stimulus. To take a step further, we propose multi-stage reinforcement learning to emphasize finding ``appropriate stimulus" around the critical period. Inspired by humans' early cognitive-developmental stage, we use multi-stage guidance near the critical period, and demonstrate the appropriate shaping reward (stage-2 guidance) in terms of the AI agent's performance, efficiency, and stability., Comment: Accepted by the ICML Complex Feedback in Online Learning Workshop (Open Problems) 2022

To acquire language, children need rich language input. However, many parents find it difficult to provide children with sufficient language input, which risks delaying their language development. To aid these parents, we design Captivate!, the first system that provides contextual language guidance to parents during play. Our system tracks both visual and spoken language cues to infer targets of joint attention, enabling the real-time suggestion of situation-relevant phrases for the parent. We design our system through a user-centered process with immigrant families--a highly vulnerable yet understudied population--as well as professional speech language therapists. Next, we evaluate Captivate! on parents with children aged 1-3 to observe improvements in responsive language use. We share insights into developing contextual guidance technology for linguistically diverse families., Comment: Published as a conference paper at CHI 2022

Critical periods are phases during which a toddler's brain develops in spurts. To promote children's cognitive development, proper guidance is critical in this stage. However, it is not clear whether such a critical period also exists for the training of AI agents. Similar to human toddlers, well-timed guidance and multimodal interactions might significantly enhance the training efficiency of AI agents as well. To validate this hypothesis, we adapt this notion of critical periods to learning in AI agents and investigate the critical period in the virtual environment for AI agents. We formalize the critical period and Toddler-guidance learning in the reinforcement learning (RL) framework. Then, we built up a toddler-like environment with VECA toolkit to mimic human toddlers' learning characteristics. We study three discrete levels of mutual interaction: weak-mentor guidance (sparse reward), moderate mentor guidance (helper-reward), and mentor demonstration (behavioral cloning). We also introduce the EAVE dataset consisting of 30,000 real-world images to fully reflect the toddler's viewpoint. We evaluate the impact of critical periods on AI agents from two perspectives: how and when they are guided best in both uni- and multimodal learning. Our experimental results show that both uni- and multimodal agents with moderate mentor guidance and critical period on 1 million and 2 million training steps show a noticeable improvement. We validate these results with transfer learning on the EAVE dataset and find the performance advancement on the same critical period and the guidance., Comment: ICMI2021 Oral Presentation, 9 pages, 9 figures

Building human-like agent, which aims to learn and think like human intelligence, has long been an important research topic in AI. To train and test human-like agents, we need an environment that imposes the agent to rich multimodal perception and allows comprehensive interactions for the agent, while also easily extensible to develop custom tasks. However, existing approaches do not support comprehensive interaction with the environment or lack variety in modalities. Also, most of the approaches are difficult or even impossible to implement custom tasks. In this paper, we propose a novel VR-based toolkit, VECA, which enables building fruitful virtual environments to train and test human-like agents. In particular, VECA provides a humanoid agent and an environment manager, enabling the agent to receive rich human-like perception and perform comprehensive interactions. To motivate VECA, we also provide 24 interactive tasks, which represent (but are not limited to) four essential aspects in early human development: joint-level locomotion and control, understanding contexts of objects, multimodal learning, and multi-agent learning. To show the usefulness of VECA on training and testing human-like learning agents, we conduct experiments on VECA and show that users can build challenging tasks for engaging human-like algorithms, and the features supported by VECA are critical on training human-like agents., Comment: 7 pages, 5 figures

One of the inherent limitations of current AI systems, stemming from the passive learning mechanisms (e.g., supervised learning), is that they perform well on labeled datasets but cannot deduce knowledge on their own. To tackle this problem, we derive inspiration from a highly intentional learning system via action: the toddler. Inspired by the toddler's learning procedure, we design an interactive agent that can learn and store task-agnostic visual representation while exploring and interacting with objects in the virtual environment. Experimental results show that such obtained representation was expandable to various vision tasks such as image classification, object localization, and distance estimation tasks. In specific, the proposed model achieved 100%, 75.1% accuracy and 1.62% relative error, respectively, which is noticeably better than autoencoder-based model (99.7%, 66.1%, 1.95%), and also comparable with those of supervised models (100%, 87.3%, 0.71%).

Real-world deployments of WiFi-based indoor localization in large public venues are few and far between as most state-of-the-art solutions require either client or infrastructure-side changes. Hence, even though high location accuracy is possible with these solutions, they are not practical due to cost and/or client adoption reasons. Majority of the public venues use commercial controller-managed WLAN solutions, %provided by Aruba, Cisco, etc., that neither allow client changes nor infrastructure changes. In fact, for such venues we have observed highly heterogeneous devices with very low adoption rates for client-side apps. In this paper, we present our experiences in deploying a scalable location system for such venues. We show that server-side localization is not trivial and present two unique challenges associated with this approach, namely Cardinality Mismatch and High Client Scan Latency. The "Mismatch" challenge results in a significant mismatch between the set of access points (APs) reporting a client in the offline and online phases, while the "Latency" challenge results in a low number of APs reporting data for any particular client. We collect three weeks of detailed ground truth data (~200 landmarks), from a WiFi setup that has been deployed for more than four years, to provide evidences for the extent and understanding the impact of these problems. Our analysis of real-world client devices reveal that the current trend for the clients is to reduce scans, thereby adversely impacting their localization accuracy. We analyze how localization is impacted when scans are minimal. We propose heuristics to alleviate reduction in the accuracy despite lesser scans. Besides the number of scans, we summarize the other challenges and pitfalls of real deployments which hamper the localization accuracy.

Recurrent neural networks (RNNs) have shown promising results in audio and speech processing applications due to their strong capabilities in modelling sequential data. In many applications, RNNs tend to outperform conventional models based on GMM/UBMs and i-vectors. Increasing popularity of IoT devices makes a strong case for implementing RNN based inferences for applications such as acoustics based authentication, voice commands, and edge analytics for smart homes. Nonetheless, the feasibility and performance of RNN based inferences on resources-constrained IoT devices remain largely unexplored. In this paper, we investigate the feasibility of using RNNs for an end-to-end authentication system based on breathing acoustics. We evaluate the performance of RNN models on three types of devices; smartphone, smartwatch, and Raspberry Pi and show that unlike CNN models, RNN models can be easily ported onto resource-constrained devices without a significant loss in accuracy.

We analyze the sequence of time intervals between consecutive stock trades of thirty companies representing eight sectors of the U. S. economy over a period of four years. For all companies we find that: (i) the probability density function of intertrade times may be fit by a Weibull distribution; (ii) when appropriately rescaled the probability densities of all companies collapse onto a single curve implying a universal functional form; (iii) the intertrade times exhibit power-law correlated behavior within a trading day and a consistently greater degree of correlation over larger time scales, in agreement with the correlation behavior of the absolute price returns for the corresponding company, and (iv) the magnitude series of intertrade time increments is characterized by long-range power-law correlations suggesting the presence of nonlinear features in the trading dynamics, while the sign series is anti-correlated at small scales. Our results suggest that independent of industry sector, market capitalization and average level of trading activity, the series of intertrade times exhibit possibly universal scaling patterns, which may relate to a common mechanism underlying the trading dynamics of diverse companies. Further, our observation of long-range power-law correlations and a parallel with the crossover in the scaling of absolute price returns for each individual stock, support the hypothesis that the dynamics of transaction times may play a role in the process of price formation., Comment: 8 pages, 5 figures. Presented at The Second Nikkei Econophysics Workshop, Tokyo, 11-14 Nov. 2002. A subset appears in "The Application of Econophysics: Proceedings of the Second Nikkei Econophysics Symposium", editor H. Takayasu (Springer-Verlag, Tokyo, 2003) pp.51-57. Submitted to Phys. Rev. E on 25 June 2003

We develop an exactly-solvable three-state discrete-time minimal Brownian ratchet (MBR), where the transition probabilities between states are asymmetric. By solving the master equations we obtain the steady-state probabilities. Generally the steady-state solution does not display detailed balance, giving rise to an induced directional motion in the MBR. For a reduced two-dimensional parameter space we find the null-curve on which the net current vanishes and detailed balance holds. A system on this curve is said to be balanced. On the null-curve, an additional source of external random noise is introduced to show that a directional motion can be induced under the zero overall driving force. We also indicate the off-balance behavior with biased random noise., Comment: 4 pages, 4 figures, RevTex source, General solution added. To be appeared in Phys. Rev. Lett

We study the cross-correlations in stock price changes between the S&P 500 companies by introducing a weighted random graph, where all vertices (companies) are fully connected, and each edge is weighted. The weight assigned to each edge is given by the normalized covariance of the two modified returns connected, so that it is ranged from -1 to 1. Here the modified return means the deviation of a return from its average over all companies. We define influence-strength at each vertex as the sum of the weights on the edges incident upon that vertex. Then we found that the influence-strength distribution in its absolute magnitude $|s|$ follows a power-law, $P(|s|)\sim |s|^{-\delta}$, with exponent $\delta \approx 1.8(1)$.

We study the flow of fluid in porous media in dimensions $d=2$ and 3. The medium is modeled by bond percolation on a lattice of $L^d$ sites, while the flow front is modeled by tracer particles driven by a pressure difference between two fixed sites (``wells'') separated by Euclidean distance $r$. We investigate the distribution function of the shortest path connecting the two sites, and propose a scaling {\it Ansatz} that accounts for the dependence of this distribution (i) on the size of the system, $L$, and (ii) on the bond occupancy probability, $p$. We confirm by extensive simulations that the {\it Ansatz} holds for $d=2$ and 3, and calculate the relevant scaling parameters. We also study two dynamical quantities: the minimal traveling time of a tracer particle between the wells and the length of the path corresponding to the minimal traveling time ``fastest path'', which is not identical to the shortest path. A scaling {\it Ansatz} for these dynamical quantities also includes the effect of finite system size $L$ and off-critical bond occupation probability $p$. We find that the scaling form for the distribution functions for these dynamical quantities for $d=2$ and 3 is similar to that for the shortest path but with different critical exponents. The scaling form is represented as the product of a power law and three exponential cutoff functions. We summarize our results in a table which contains estimates for all parameters which characterize the scaling form for the shortest path and the minimal traveling time in 2 and 3 dimensions; these parameters are the fractal dimension, the power law exponent, and the constants and exponents that characterize the exponential cutoff functions., Comment: 31 pages, 12 Figures

We study the frequency distribution of family names. From a common data base, we count the number of people who share the same family name. This is the size of the family. We find that (i) the total number of different family names in a society scales as a power-law of the population, (ii) the total number of family names of the same size decreases as the size increases with a power-law and (iii) the relation between size and rank of a family name also shows a power-law. These scaling properties are found to be consistent for five different regional communities in Japan., Comment: 11 pages including 4 figures, Physica A (accepted)

We study how the presence of correlations in physical variables contributes to the form of probability distributions. We investigate a process with correlations in the variance generated by (i) a Gaussian or (ii) a truncated L\'{e}vy distribution. For both (i) and (ii), we find that due to the correlations in the variance, the process ``dynamically'' generates power-law tails in the distributions, whose exponents can be controlled through the way the correlations in the variance are introduced. For (ii), we find that the process can extend a truncated distribution {\it beyond the truncation cutoff}, which leads to a crossover between a L\'{e}vy stable power law and the present ``dynamically-generated'' power law. We show that the process can explain the crossover behavior recently observed in the $S&P500$ stock index., Comment: 7 pages, five figures. To appear in Europhysics Letters (2000)

On two-dimensional percolation clusters at the percolation threshold, we study $<\sigma(M_B,r)>$, the average conductance of the backbone, defined by two points separated by Euclidean distance $r$, of mass $M_B$. We find that with increasing $M_B$ and for fixed r, $<\sigma(M_B,r)>$ asymptotically {\it decreases} to a constant, in contrast with the behavior of homogeneous sytems and non-random fractals (such as the Sierpinski gasket) in which conductance increases with increasing $M_B$. We explain this behavior by studying the distribution of shortest paths between the two points on clusters with a given $M_B$. We also study the dependence of conductance on $M_B$ slightly above the percolation threshold., Comment: 8 pages, 4 figures

We present a scaling hypothesis for the distribution function of the shortest paths connecting any two points on a percolating cluster which accounts for {\it (i)} the effect of the finite size of the system, and {\it (ii)} the dependence of this distribution on the site occupancy probability $p$. We test the hypothesis for the case of two-dimensional percolation., Comment: 7 pages, 3 figures

We develop a scale-invariant truncated L\'evy (STL) process to describe physical systems characterized by correlated stochastic variables. The STL process exhibits L\'evy stability for the probability density, and hence shows scaling properties (as observed in empirical data); it has the advantage that all moments are finite (and so accounts for the empirical scaling of the moments). To test the potential utility of the STL process, we analyze financial data., Comment: 4 pages, 6 figures; corrected typos; changed figure 3

We study traveling time and traveling length for tracer dispersion in porous media. We model porous media by two-dimensional bond percolation, and we model flow by tracer particles driven by a pressure difference between two points separated by Euclidean distance $r$. We find that the minimal traveling time $t_{min}$ scales as $t_{min} \sim r^{1.40}$, which is different from the scaling of the most probable traveling time, ${\tilde t} \sim r^{1.64}$. We also calculate the length of the path corresponding to the minimal traveling time and find $\ell_{min} \sim r^{1.13}$ and that the most probable traveling length scales as ${\tilde \ell} \sim r^{1.21}$. We present the relevant distribution functions and scaling relations., Comment: 4 pages including 9 figures, Minor corrections in text

We analyze the fluctuations in the gross domestic product (GDP) of 152 countries for the period 1950--1992. We find that (i) the distribution of annual growth rates for countries of a given GDP decays with ``fatter'' tails than for a Gaussian, and (ii) the width of the distribution scales as a power law of GDP with a scaling exponent $\beta \approx 0.15$. Both findings are in surprising agreement with results on firm growth. These results are consistent with the hypothesis that the evolution of organizations with complex structure is governed by similar growth mechanisms., Comment: 4 pages, 7 ps figures, using Latex2e with epsf rotate and multicol style files. Submitted to PRL