Search

With the exponential growth of multimedia data, leveraging multimodal sensors presents a promising approach for improving accuracy in human activity recognition. Nevertheless, accurately identifying these activities using both video data and wearable sensor data presents challenges due to the labor-intensive data annotation, and reliance on external pretrained models or additional data. To address these challenges, we introduce Multimodal Masked Autoencoders-Based One-Shot Learning (Mu-MAE). Mu-MAE integrates a multimodal masked autoencoder with a synchronized masking strategy tailored for wearable sensors. This masking strategy compels the networks to capture more meaningful spatiotemporal features, which enables effective self-supervised pretraining without the need for external data. Furthermore, Mu-MAE leverages the representation extracted from multimodal masked autoencoders as prior information input to a cross-attention multimodal fusion layer. This fusion layer emphasizes spatiotemporal features requiring attention across different modalities while highlighting differences from other classes, aiding in the classification of various classes in metric-based one-shot learning. Comprehensive evaluations on MMAct one-shot classification show that Mu-MAE outperforms all the evaluated approaches, achieving up to an 80.17% accuracy for five-way one-shot multimodal classification, without the use of additional data., Comment: IEEE MIPR 2024

Currently, over a thousand LLMs exist that are multi-purpose and are capable of performing real world tasks, including Q&A, text summarization, content generation, etc. However, accessibility, scale and reliability of free models prevents them from being widely deployed in everyday use cases. To address the first two issues of access and scale, organisations such as HuggingFace have created model repositories where users have uploaded model weights and quantized versions of models trained using different paradigms, as well as model cards describing their training process. While some models report performance on commonly used benchmarks, not all do, and interpreting the real world impact of trading off performance on a benchmark for model deployment cost, is unclear. Here, we show that a herd of open source models can match or exceed the performance of proprietary models via an intelligent router. We show that a Herd of open source models is able to match the accuracy of ChatGPT, despite being composed of models that are effectively 2.5x smaller. We show that in cases where GPT is not able to answer the query, Herd is able to identify a model that can, at least 40% of the time.

Humans learn by interacting with their environments and perceiving the outcomes of their actions. A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans. However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both. To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task. We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development. More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen, and provides a pathway towards more efficient progress.

We propose MASTAF, a Model-Agnostic Spatio-Temporal Attention Fusion network for few-shot video classification. MASTAF takes input from a general video spatial and temporal representation,e.g., using 2D CNN, 3D CNN, and Video Transformer. Then, to make the most of such representations, we use self- and cross-attention models to highlight the critical spatio-temporal region to increase the inter-class variations and decrease the intra-class variations. Last, MASTAF applies a lightweight fusion network and a nearest neighbor classifier to classify each query video. We demonstrate that MASTAF improves the state-of-the-art performance on three few-shot video classification benchmarks(UCF101, HMDB51, and Something-Something-V2), e.g., by up to 91.6%, 69.5%, and 60.7% for five-way one-shot video classification, respectively., Comment: WACV 2023

With the development of the Internet of Things(IoT) and Artificial Intelligence(AI) technologies, human activity recognition has enabled various applications, such as smart homes and assisted living. In this paper, we target a new healthcare application of human activity recognition, early mobility recognition for Intensive Care Unit(ICU) patients. Early mobility is essential for ICU patients who suffer from long-time immobilization. Our system includes accelerometer-based data collection from ICU patients and an AI model to recognize patients' early mobility. To improve the model accuracy and stability, we identify features that are insensitive to sensor orientations and propose a segment voting process that leverages a majority voting strategy to recognize each segment's activity. Our results show that our system improves model accuracy from 77.78\% to 81.86\% and reduces the model instability (standard deviation) from 16.69\% to 6.92\%, compared to the same AI model without our feature engineering and segment voting process.

Differences in gait patterns of children with Duchenne muscular dystrophy (DMD) and typically-developing (TD) peers are visible to the eye, but quantifications of those differences outside of the gait laboratory have been elusive. In this work, we measured vertical, mediolateral, and anteroposterior acceleration using a waist-worn iPhone accelerometer during ambulation across a typical range of velocities. Fifteen TD and fifteen DMD children from 3-16 years of age underwent eight walking/running activities, including five 25 meters walk/run speed-calibration tests at a slow walk to running speeds (SC-L1 to SC-L5), a 6-minute walk test (6MWT), a 100 meters fast-walk/jog/run (100MRW), and a free walk (FW). For clinical anchoring purposes, participants completed a Northstar Ambulatory Assessment (NSAA). We extracted temporospatial gait clinical features (CFs) and applied multiple machine learning (ML) approaches to differentiate between DMD and TD children using extracted temporospatial gait CFs and raw data. Extracted temporospatial gait CFs showed reduced step length and a greater mediolateral component of total power (TP) consistent with shorter strides and Trendelenberg-like gait commonly observed in DMD. ML approaches using temporospatial gait CFs and raw data varied in effectiveness at differentiating between DMD and TD controls at different speeds, with an accuracy of up to 100%. We demonstrate that by using ML with accelerometer data from a consumer-grade smartphone, we can capture DMD-associated gait characteristics in toddlers to teens.

With the rapid development of the internet of things (IoT) and artificial intelligence (AI) technologies, human activity recognition (HAR) has been applied in a variety of domains such as security and surveillance, human-robot interaction, and entertainment. Even though a number of surveys and review papers have been published, there is a lack of HAR overview papers focusing on healthcare applications that use wearable sensors. Therefore, we fill in the gap by presenting this overview paper. In particular, we present our projects to illustrate the system design of HAR applications for healthcare. Our projects include early mobility identification of human activities for intensive care unit (ICU) patients and gait analysis of Duchenne muscular dystrophy (DMD) patients. We cover essential components of designing HAR systems including sensor factors (e.g., type, number, and placement location), AI model selection (e.g., classical machine learning models versus deep learning models), and feature engineering. In addition, we highlight the challenges of such healthcare-oriented HAR systems and propose several research opportunities for both the medical and the computer science community.

Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disease of the brain and the spinal cord, which leads to paralysis of motor functions. Patients retain their ability to blink, which can be used for communication. Here, We present an Artificial Intelligence (AI) system that uses eye-blinks to communicate with the outside world, running on real-time Internet-of-Things (IoT) devices. The system uses a Convolutional Neural Network (CNN) to find the blinking pattern, which is defined as a series of Open and Closed states. Each pattern is mapped to a collection of words that manifest the patient's intent. To investigate the best trade-off between accuracy and latency, we investigated several Convolutional Network architectures, such as ResNet, SqueezeNet, DenseNet, and InceptionV3, and evaluated their performance. We found that the InceptionV3 architecture, after hyper-parameter fine-tuning on the specific task led to the best performance with an accuracy of 99.20% and 94ms latency. This work demonstrates how the latest advances in deep learning architectures can be adapted for clinical systems that ameliorate the patient's quality of life regardless of the point-of-care., Comment: 8 pages, 4 figures

Primate vision depends on recurrent processing for reliable perception. A growing body of literature also suggests that recurrent connections improve the learning efficiency and generalization of vision models on classic computer vision challenges. Why then, are current large-scale challenges dominated by feedforward networks? We posit that the effectiveness of recurrent vision models is bottlenecked by the standard algorithm used for training them, "back-propagation through time" (BPTT), which has O(N) memory-complexity for training an N step model. Thus, recurrent vision model design is bounded by memory constraints, forcing a choice between rivaling the enormous capacity of leading feedforward models or trying to compensate for this deficit through granular and complex dynamics. Here, we develop a new learning algorithm, "contractor recurrent back-propagation" (C-RBP), which alleviates these issues by achieving constant O(1) memory-complexity with steps of recurrent processing. We demonstrate that recurrent vision models trained with C-RBP can detect long-range spatial dependencies in a synthetic contour tracing task that BPTT-trained models cannot. We further show that recurrent vision models trained with C-RBP to solve the large-scale Panoptic Segmentation MS-COCO challenge outperform the leading feedforward approach, with fewer free parameters. C-RBP is a general-purpose learning algorithm for any application that can benefit from expansive recurrent dynamics. Code and data are available at https://github.com/c-rbp., Comment: Published at NeurIPS 2020

The late universe's matter distribution obeys the Copernican principle at only the coarsest of scales. The relative importance of such inhomogeneity is still not well understood. Because of the Einstein field equations' non-linear nature, some argue a non-perturbative approach is necessary to correctly model inhomogeneities and may even obviate any need for dark energy. We shall discuss an approach based on Regge calculus, a discrete approximation to general relativity: we shall discuss the Collins--Williams formulation of Regge calculus and its application to two toy universes. The first is a universe for which the continuum solution is well-established, the $\Lambda$-FLRW universe. The second is an inhomogeneous universe, the `lattice universe' wherein matter consists solely of a lattice of point masses with pure vacuum in between, a distribution more similar to that of the actual universe compared to FLRW universes. We shall discuss both regular lattices and one where one mass gets perturbed., Comment: 8 pages, 5 figures, to appear in the conference proceedings of HTGRG-2 2015, Quy Nhon, Vietnam

This paper examines the behaviour of closed `lattice universes' wherein masses are distributed in a regular lattice on the Cauchy surfaces of closed vacuum universes. Such universes are approximated using a form of Regge calculus originally developed by Collins and Williams to model closed FLRW universes. We consider two types of lattice universes, one where all masses are identical to each other and another where one mass gets perturbed in magnitude. In the unperturbed universe, we consider the possible arrangements of the masses in the Regge Cauchy surfaces and demonstrate that the model will only be stable if each mass lies within some spherical region of convergence. We also briefly discuss the existence of Regge models that are dual to the ones we have considered. We then model a perturbed lattice universe and demonstrate that the model's evolution is well-behaved, with the expansion increasing in magnitude as the perturbation is increased., Comment: 34 pages, 9 figures. Expanded Introduction: elaborated on cosmological context for work and referenced recent work on studying lattice universes using a wide variety of approaches. Corrected an error in the LW graphs in Fig 3

The Collins-Williams Regge calculus models of FLRW space-times and Brewin's subdivided models are applied to closed vacuum $\Lambda$-FLRW universes. In each case, we embed the Regge Cauchy surfaces into 3-spheres in $\mathbf{E}^4$ and consider possible measures of Cauchy surface radius that can be derived from the embedding. Regge equations are obtained from both global variation, where entire sets of identical edges get varied simultaneously, and local variation, where each edge gets varied individually. We explore the relationship between the two sets of solutions, the conditions under which the Regge Hamiltonian constraint would be a first integral of the evolution equation, the initial value equation for each model at its moment of time symmetry, and the performance of the various models. It is revealed that local variation does not generally lead to a viable Regge model. It is also demonstrated that the various models do satisfy their respective initial value equations. Finally, it is shown that the models reproduce the behaviour of the continuum model rather well initially, with performance improving as we increase the number of tetrahedra used to construct the Regge Cauchy surface. Eventually though, all models gradually fail to keep up with the continuum FLRW model's expansion, with the models with lower numbers of tetrahedra falling away more quickly. We believe this failure to keep up is due to the finite resolution of the Regge Cauchy surfaces trying to approximate an ever expanding continuum Cauchy surface; each Regge surface has a fixed number of tetrahedra and as the surface being approximated gets larger, the resolution would degrade. Finally, we note that all Regge models end abruptly at a point when the time-like struts of the skeleton become null, though this end-point appears to get delayed as the number of tetrahedra is increased., Comment: 35 pages, 16 figures. Version accepted for publication. Expanded context in Introduction. Elaborated what happens when time-like struts turn null. Added further comments to interpretation of results

This paper examines the properties of `lattice universes' wherein point masses are arranged in a regular lattice on space-like hypersurfaces; open, flat, and closed universes are considered. The universes are modelled using the Lindquist-Wheeler (LW) approximation scheme, which approximates the space-time in each lattice cell by Schwarz\-schild geometry. Extending Lindquist and Wheeler's work, we derive cosmological scale factors describing the evolution of all three types of universes, and we use these scale factors to show that the universes' dynamics strongly resemble those of Friedmann-Lema\^itre-Robertson-Walker (FLRW) universes. In particular, we use the scale factors to make more salient the resemblance between Clifton and Ferreira's Friedmann-like equations for the LW models and the actual Friedmann equations of FLRW space-times. Cosmological redshifts for such universes are then determined numerically, using a modification of Clifton and Ferreira's approach; the redshifts are found to closely resemble their FLRW counterparts, though with certain differences attributable to the `lumpiness' in the underlying matter content. Most notably, the LW redshifts can differ from their FLRW counterparts by as much as 30\%, even though they increase linearly with FLRW redshifts, and they exhibit a non-zero integrated Sachs-Wolfe effect, something which would not be possible in matter-dominated FLRW universes without a cosmological constant., Comment: 41 pages, 35 figures. Corrected error in graph legend of Fig 28 as well as several small typos