Search

Human activity recognition(HAR) is an important research focus in ubiquitous computing. It has been widely applied in various domains, such as smart homes, healthcare assistance, and sports training. Accurate human activity recognition directly impacts the performance of downstream tasks. Existing methods for human activity recognition primarily rely on extracting temporal or spatial features from the data. These features are used for the task of human activity recognition. The existing methods mainly use CNN or RNN models to model the Euclidean correlations among spatially adjacent sensors or channels. However, non-Euclidean pairwise correlations among all sensors or channels are even critical for accurate classification, which has been ignored by the existing methods. In this paper, we incorporate fine-grained spatial structural data into the model to overcome these limitations. A novel deep learning model for human activity recognition is proposed, which is called the fine-grained data-oriented Spatial-Temporal Graph Transformer network (STGT). The introduction of the STGT model can eliminate the limitations of existing spatial feature extraction methods by leveraging a novel data organization approach proposed in this study. This model enhances the spatial features within the time feature extraction module for effective spatio-temporal feature extraction. We conducted experiments on four large-scale real-world HAR datasets to evaluate its performance. The experimental results demonstrate the superiority of our method over state-of-the-art approaches.

Recognizing and understanding human activities in real-time videos is a challenging task due to the complex nature of video data and the need for efficient and accurate analysis. This research pioneers a breakthrough in video activity recognition by introducing a robust framework leveraging the power of a stacked Bidirectional Long Short-Term Memory (Bi-LSTM) and Gated Recurrent Unit (GRU) architecture, harmonized within a fusion-based deep model. The stacked Bi-LSTM-GRU model capitalizes on its dual recurrent architecture, capturing nuanced temporal dependencies within video sequences. The fusion-based deep architecture synergizes spatial and temporal features, enabling the model to discern intricate patterns in human activities. To further enhance the discriminative power of the model, we introduce a fusion module in the proposed deep architecture. The fusion module integrates multi-modal features extracted from different levels of the network hierarchy, allowing for a more comprehensive representation of video activities. We demonstrate the efficacy of our approach through rigorous experimentation on UCF50, UCF101, and HMDB51 datasets. In experiments on the UCF50 dataset, our model achieves an accuracy of 97.01% and 95.86% on training and validation sets respectively, showcasing its proficiency in discerning activities across a diverse range of scenarios. The evaluation extends to the UCF101 dataset, where the proposed approach achieves a competitive accuracy of 97.62% and 96.93% on training and validation sets, surpassing previous benchmarks by a margin of approx 1%. Further-more, on the challenging HMDB51 dataset, the model demonstrates a robust accuracy of 89.71%and 88.88% on training and validation sets, solidifying its efficacy in intricate action recognition tasks.

Abstract Background Accurately measuring energy expenditure during physical activity outside of the laboratory is challenging, especially on a large scale. Thigh-worn accelerometers have gained popularity due to the possibility to accurately detect physical activity types. The use of machine learning techniques for activity classification and energy expenditure prediction may improve accuracy over current methods. Here, we developed a novel composite energy expenditure estimation model by combining an activity classification model with a stride specific energy expenditure model for walking, running, and cycling. Methods We first trained a supervised deep learning activity classification model using pooled data from available adult accelerometer datasets. The composite energy expenditure model was then developed and validated using additional data based on a sample of 69 healthy adult participants (49% female; age = 25.2 ± 5.8 years) who completed a standardised activity protocol with indirect calorimetry as the reference measure. Results The activity classification model showed an overall accuracy of 99.7% across all five activity types during validation. The composite model for estimating energy expenditure achieved a mean absolute percentage error of 10.9%. For running, walking, and cycling, the composite model achieved a mean absolute percentage error of 6.6%, 7.9% and 16.1%, respectively. Conclusions The integration of thigh-worn accelerometers with machine learning models provides a highly accurate method for classifying physical activity types and estimating energy expenditure. Our novel composite model approach improves the accuracy of energy expenditure measurements and supports better monitoring and assessment methods in non-laboratory settings.

Thermal comfort of occupants is key feedback information for improving indoor environment and managing building energy use. Through analyzing inertial measurement units (IMU) data from wearable devices with machine learning, thermal comfort of occupants can be detected in a non- intrusive method. This paper proposed a dataset consisted of IMU data collected from 30 participants (14 males and 16 females, aged 23.23 ± 1.70 years, height 168.67 ± 8.02 cm, and weight 59.55 ± 10.96 kg) who wore two IMUs on their hands while performing 30 thermal comfort activities (10 cold-related, 10 hot-related, and 10 neutral activities) according to their personal habits.The database is divided into two parts: (1) Single activities data, which includes 4500 samples acquired from experiments where each participant was asked to perform 30 thermal comfort activities individually. (2) Continuous multi-activity data, which comprise 360 samples collected while participants performed a series of randomly assigned activities in a more natural and continuous manner. The combination of these two parts provides a comprehensive dataset for both the training and testing phases of machine learning models. By offering detailed labels, this database aims to serve as a foundation for research exploring machine learning applications in detecting occupant thermal comfort, ultimately contributing to improved indoor environments and more efficient building energy management.

Abstract Physical fitness is one of the most important traits a person could have for health longevity. Conducting regular exercise is fundamental to maintaining physical fitness, but with the caveat of occurring injury if not done properly. Several algorithms exists to automatically monitor and evaluate exercise using the user’s pose. However, it is not an easy task to accurately monitor and evaluate exercise poses automatically. Moreover, there are limited number of datasets exists in this area. In our work, we attempt to construct a neural network model that could be used to evaluate exercise poses based on key points extracted from exercise video frames. First, we collected several images consists of different exercise poses. We utilize the the OpenPose library to extract key points from exercise video datasets and LSTM neural network to learn exercise patterns. The result of our experiment has shown that the methods used are quite effective for exercise types of push-up, sit-up, squat, and plank. The neural-network model achieved more than 90% accuracy for the four exercise types.

Abstract Non-Insulin-Dependent Diabetes Mellitus (NIDDM) is a chronic health condition caused by high blood sugar levels, and if not treated early, it can lead to serious complications i.e. blindness. Human Activity Recognition (HAR) offers potential for early NIDDM diagnosis, emerging as a key application for HAR technology. This research introduces DiabSense, a state-of-the-art smartphone-dependent system for early staging of NIDDM. DiabSense incorporates HAR and Diabetic Retinopathy (DR) upon leveraging the power of two different Graph Neural Networks (GNN). HAR uses a comprehensive array of 23 human activities resembling Diabetes symptoms, and DR is a prevalent complication of NIDDM. Graph Attention Network (GAT) in HAR achieved 98.32% accuracy on sensor data, while Graph Convolutional Network (GCN) in the Aptos 2019 dataset scored 84.48%, surpassing other state-of-the-art models. The trained GCN analyzed retinal images of four experimental human subjects for DR report generation, and GAT generated their average duration of daily activities over 30 days. The daily activities in non-diabetic periods of diabetic patients were measured and compared with the daily activities of the experimental subjects, which helped generate risk factors. Fusing risk factors with DR conditions enabled early diagnosis recommendations for the experimental subjects despite the absence of any apparent symptoms. The comparison of DiabSense system outcome with clinical diagnosis reports in the experimental subjects was conducted using the A1C test. The test results confirmed the accurate assessment of early diagnosis requirements for experimental subjects by the system. Overall, DiabSense exhibits significant potential for ensuring early NIDDM treatment, improving millions of lives worldwide.

Many fall detection systems are being used to provide real-time responses to fall occurrences. Automated fall detection is challenging because it requires very high accuracy to be clinically acceptable. Recent research has tried to improve sensitivity while reducing the high rate of false positives. Nevertheless, there are still limitations in terms of having efficient learning approaches and proper datasets to train. To reduce false alarms, one approach is to add more nonfall data as negative samples to train the deep learning model. However, this approach increases class imbalance in the training set. To tackle this problem, we propose a multi-task deep learning approach that divides datasets into multiple training sets for multiple tasks. We prove this approach gives better results than a single-task model trained on all datasets. Many experiments are conducted to find the best combination of tasks for multi-task model training for fall detection.

Abstract Background The more accurate we can assess human physical behaviour in free-living conditions the better we can understand its relationship with health and wellbeing. Thigh-worn accelerometry can be used to identify basic activity types as well as different postures with high accuracy. User-friendly software without the need for specialized programming may support the adoption of this method. This study aims to evaluate the classification accuracy of two novel no-code classification methods, namely SENS motion and ActiPASS. Methods A sample of 38 healthy adults (30.8 ± 9.6 years; 53% female) wore the SENS motion accelerometer (12.5 Hz; ±4 g) on their thigh during various physical activities. Participants completed standardized activities with varying intensities in the laboratory. Activities included walking, running, cycling, sitting, standing, and lying down. Subsequently, participants performed unrestricted free-living activities outside of the laboratory while being video-recorded with a chest-mounted camera. Videos were annotated using a predefined labelling scheme and annotations served as a reference for the free-living condition. Classification output from the SENS motion software and ActiPASS software was compared to reference labels. Results A total of 63.6 h of activity data were analysed. We observed a high level of agreement between the two classification algorithms and their respective references in both conditions. In the free-living condition, Cohen’s kappa coefficients were 0.86 for SENS and 0.92 for ActiPASS. The mean balanced accuracy ranged from 0.81 (cycling) to 0.99 (running) for SENS and from 0.92 (walking) to 0.99 (sedentary) for ActiPASS across all activity types. Conclusions The study shows that two available no-code classification methods can be used to accurately identify basic physical activity types and postures. Our results highlight the accuracy of both methods based on relatively low sampling frequency data. The classification methods showed differences in performance, with lower sensitivity observed in free-living cycling (SENS) and slow treadmill walking (ActiPASS). Both methods use different sets of activity classes with varying definitions, which may explain the observed differences. Our results support the use of the SENS motion system and both no-code classification methods.

Abstract Human activity recognition has a wide range of applications in various fields, such as video surveillance, virtual reality and human–computer intelligent interaction. It has emerged as a significant research area in computer vision. GCN (Graph Convolutional networks) have recently been widely used in these fields and have made great performance. However, there are still some challenges including over-smoothing problem caused by stack graph convolutions and deficient semantics correlation to capture the large movements between time sequences. Vision Transformer (ViT) is utilized in many 2D and 3D image fields and has surprised results. In our work, we propose a novel human activity recognition method based on ViT (HAR-ViT). We integrate enhanced AGCL (eAGCL) in 2s-AGCN to ViT to make it process spatio-temporal data (3D skeleton) and make full use of spatial features. The position encoder module orders the non-sequenced information while the transformer encoder efficiently compresses sequence data features to enhance calculation speed. Human activity recognition is accomplished through multi-layer perceptron (MLP) classifier. Experimental results demonstrate that the proposed method achieves SOTA performance on three extensively used datasets, NTU RGB+D 60, NTU RGB+D 120 and Kinetics-Skeleton 400.

To achieve Wi-Fi cross-domain human activity perception that was not dependent on target domain data, a domain-generalization human activity recognition model based on CSI instance normalization called INDG-Fi was proposed. The instance normalization standardization was utilized to remove domain information from the representation of CSI features by INDG-Fi. Then action classifiers and domain classifiers were constructed for shared feature extraction. By employing activity bias learning and adversarial domain learning, the model biased the features extracted from the encoding layer towards signal variations caused by human actions while moving away from domain signals. To enhance the model’s focus on subcarrier signals that were more significantly influenced by human actions, a subcarrier attention module was incorporated into the encoding layer. The implemented results demonstrate that the proposed INDG-Fi achieves perceptual accuracies of 97.99% and 92.73% for unseen users and locations, respectively, thus enabling robust cross-domain perception.

Human interaction and computer vision converge in the realm of Human Activity Recognition (HAR), which is a research field dedicated to the creation of automated systems capable of observing and categorizing human activities. This domain closely aligns with machine learning, involving the development of algorithms and models adept at learning to recognize and classify patterns within data. HAR typically unfolds in two pivotal phases: data acquisition and processing, followed by activity classification. In the initial phase of data acquisition and processing, information is gathered from various sensors or video sources, such as accelerometers, smartphones, or smartwatches. Subsequently, the collected data undergo preprocessing to extract relevant features. The subsequent phase, activity classification, employs machine learning algorithms to categorize these extracted features into distinct activity types, ranging from walking and running to sitting. This paper introduces an innovative approach grounded on these two fundamental phases. For the first phase, we leverage the MediaPipe algorithm to discern human articulations. Once these poses are detected, we contribute by extracting the coordinates of each articulation. These coordinates are then transformed into graphs, where nodes signify the articulation coordinates and edges represent the connections between them. In the second phase, we enhance existing methodologies by incorporating a diverse set of machine learning models. Notably, the utilization of Graph Neural Networks (GNNs) which stands out as a significant advancement. This choice proves instrumental in effectively learning and representing complex spatial and temporal patterns, surpassing the limi-tations of conventional machine learning algorithms. The developed system undergoes evaluation on the KTH and UCF50 datasets, demonstrating state-of-the-art performance in HAR.

Abstract Smartphone sensors have gained considerable traction in Human Activity Recognition (HAR), drawing attention for their diverse applications. Accelerometer data monitoring holds promise in understanding students’ physical activities, fostering healthier lifestyles. This technology tracks exercise routines, sedentary behavior, and overall fitness levels, potentially encouraging better habits, preempting health issues, and bolstering students’ well-being. Traditionally, HAR involved analyzing signals linked to physical activities using handcrafted features. However, recent years have witnessed the integration of deep learning into HAR tasks, leveraging digital physiological signals from smartwatches and learning features automatically from raw sensory data. The Long Short-Term Memory (LSTM) network stands out as a potent algorithm for analyzing physiological signals, promising improved accuracy and scalability in automated signal analysis. In this article, we propose a feature analysis framework for recognizing student activity and monitoring health based on smartphone accelerometer data through an edge computing platform. Our objective is to boost HAR performance by accounting for the dynamic nature of human behavior. Nonetheless, the current LSTM network’s presetting of hidden units and initial learning rate relies on prior knowledge, potentially leading to suboptimal states. To counter this, we employ Bidirectional LSTM (BiLSTM), enhancing sequence processing models. Furthermore, Bayesian optimization aids in fine-tuning the BiLSTM model architecture. Through fivefold cross-validation on training and testing datasets, our model showcases a classification accuracy of 97.5% on the tested dataset. Moreover, edge computing offers real-time processing, reduced latency, enhanced privacy, bandwidth efficiency, offline capabilities, energy efficiency, personalization, and scalability. Extensive experimental results validate that our proposed approach surpasses state-of-the-art methodologies in recognizing human activities and monitoring health based on smartphone accelerometer data.

Human Activity Recognition (HAR) is crucial for the advancement of applications in smart environments, communication, IoT, security, and healthcare monitoring. Convolutional neural networks (CNNs) have made substantial contributions to human activity recognition (HAR). However, they frequently encounter difficulties in accurately discerning intricate human actions in real-time situations. This study aims to fill a significant research gap by incorporating the Convolutional Block Attention Module (CBAM) into CNN architectures. The goal is to improve the extraction of features from video sequences. The CBAM boosts the performance of the network by selectively prioritizing significant spatial and channel-wise data, resulting in improved detection of subtle activity patterns and increased stability in categorization. CBAM’s attention mechanism directly focuses and amplifies essential characteristics, which sets it apart from typical CNNs that lack a refined focus mechanism. This unique approach results in improved performance in behavior identification tests. The proposed CBAM-enhanced model has been extensively tested on benchmark datasets, yielding an accuracy of 94.23% on the HMDB51 dataset. It also achieved competitive results of 83.4% and 88.9% on the UCF-101 and UCF-50 datasets, respectively. However, there is still a lack of study in comprehending how CBAM adjusts to different CNN architectures and its suitability in varied HAR situations beyond controlled datasets. In future studies, it is imperative for researchers to investigate the integration of CBAM with other CNN frameworks, assess its efficacy in practical scenarios, and explore multi-modal sensor fusion techniques to enhance its reliability and utility. This study showcases the ability of CBAM to enhance HAR capabilities and also paves the way for future research to improve activity identification systems for wider and more practical uses.

Abstract Knowledge distillation is an effective approach for training robust multi-modal machine learning models when synchronous multimodal data are unavailable. However, traditional knowledge distillation techniques have limitations in comprehensively transferring knowledge across modalities and models. This paper proposes a multiscale knowledge distillation framework to address these limitations. Specifically, we introduce a multiscale semantic graph mapping (SGM) loss function to enable more comprehensive knowledge transfer between teacher and student networks at multiple feature scales. We also design a fusion and tuning (FT) module to fully utilize correlations within and between different data types of the same modality when training teacher networks. Furthermore, we adopt transformer-based backbones to improve feature learning compared to traditional convolutional neural networks. We apply the proposed techniques to multimodal human activity recognition and compared with the baseline method, it improved by 2.31% and 0.29% on the MMAct and UTD-MHAD datasets. Ablation studies validate the necessity of each component.

Abstract Wearable sensors are widely used in medical applications and human–computer interaction because of their portability and powerful privacy. Human activity identification based on sensor data plays a vital role in these fields. Therefore, it is important to improve the recognition performance of different types of actions. Aiming at the problems of insufficient time-varying feature extraction and gradient explosion caused by too many network layers, a time convolution network recognition model with attention mechanism (TCN-Attention-HAR) was proposed. The model effectively recognizes and emphasizes the key feature information. The ability of extracting temporal features from TCN (temporal convolution network) is improved by using the appropriate size of the receiver domain. In addition, attention mechanisms are used to assign higher weights to important information, enabling models to learn and identify human activities more effectively. The performance of the Open Data Set (WISDM, PAMAP2 and USC-HAD) is improved by 1.13%, 1.83% and 0.51%, respectively, compared with other advanced models, these results clearly show that the network model presented in this paper has excellent recognition performance. In the knowledge distillation experiment, the parameters of student model are only about 0.1% of those of teacher model, and the accuracy of the model has been greatly improved, and in the WISDM data set, compared with the teacher's model, the accuracy is 0.14% higher.

Smart and connected health (SCH) accelerates the development and integration of information science and engineering approaches to support the digital transformation of health and medicine in populated societies. Sensor data based human activity recognition (HAR) as an effective means of SCH is promising for healthcare monitoring and ambient assisted living. This paper focuses on multi-position sensor data fusion enabled HAR, and proposes a multi-channel deep learning framework. The main contributions include: (1) A multi-channel hybrid deep learning model (1DCNN-Att-BiLSTM) that merges a one-dimensional convolutional neural network, a bidirectional long short-term memory model, and an attention mechanism is proposed to significantly improve the recognition performance of HAR models by extracting and selecting local and global behavioral features in the spatial and temporal domains; (2) Publicly accessible datasets Shoaib AR, Shoaib SA, and HAPT are collected and processed to build multi-position sensor data pool for model's evaluation; (3) Thoroughly evaluating the classification performance of seven sensor data fusion patterns from different body positions within the parallel network structure of the multi-channel framework; (4) Comparing the recognition performance metrics of traditional machine learning models, deep learning models, and hybrid models with our proposed model. Extensive experiments demonstrate that our approach achieves competitive performance.

In recent years, Human activity recognition (HAR) based on wearable devices has been widely applied in health applications and other fields. Currently, most HAR models are based on the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), or their combination. Recently, there have been proposals based on Transformer and its variant models. However, due to the fact that these models have sequential network structures and are unable to simultaneously focus on local and global features, thus, resulting in a reduction in recognition performance. In addition, along with the substantial computational resources required by Transformers, they are not suitable for resource-constrained devices. In this paper, the primary distinction of our proposed model from other hybrid models that combine CNN and Transformer is that our model adopts a completely new parallel network architecture and primarily focuses on lightweight design. Particularly, We proposed the Mobile Human Activity Recognition Conformer (MobileHARC), which adopts the parallel structure with a lightweight Transformer and CNN as the backbone networks. Furthermore, we proposed the Inverted Residual Lightweight Convolution Block and Multiscale Lightweight Multi-Head Self-Attention Mechanism. We systematically evaluate the proposed models on four public datasets. Experimental results show that MobileHARC achieves superior recognition performance, and uses fewer Floating-Point Operations per Second (FLOPs) and parameters compared to current models.

Human Activity Recognition (HAR) has emerged as a critical research area due to its extensive applications in various real-world domains. Numerous CSI-based datasets have been established to support the development and evaluation of advanced HAR algorithms. However, existing CSI-based HAR datasets are frequently limited by a dearth of complexity and diversity in the activities represented, hindering the design of robust HAR models. These limitations typically manifest as a narrow focus on a limited range of activities or the exclusion of factors influencing real-world CSI measurements. Consequently, the scarcity of diverse training data can impede the development of efficient HAR systems. To address the limitations of existing datasets, this paper introduces a novel dataset that captures spatial diversity through multiple transceiver orientations over a high dimensional space encompassing a large number of subcarriers. The dataset incorporates a wider range of real-world factors including extensive activity range, a spectrum of human movements (encompassing both micro-and macro-movements), variations in body composition, and diverse environmental conditions (noise and interference). The experiment is performed in a controlled laboratory environment with dimensions of 5 m (width) × 8 m (length) × 3 m (height) to capture CSI measurements for various human activities. Four ESP32-S3-DevKitC-1 devices, configured as transceiver pairs with unique Media Access Control (MAC) addresses, collect CSI data according to the Wi-Fi IEEE 802.11n standard. Mounted on tripods at a height of 1.5 m, the transmitter devices (powered by external power banks) positioned at north and east send multiple Wi-Fi beacons to their respective receivers (connected to laptops via USB for data collection) located at south and west. To capture multi-perspective CSI data, all six participants sequentially performed designated activities while standing in the centre of the tripod arrangement for 5 s per sample. The system collected approximately 300–450 packets per sample for approximately 1200 samples per activity, capturing CSI information across the 166 subcarriers employed in the Wi-Fi IEEE 802.11n standard. By leveraging the richness of this dataset, HAR researchers can develop more robust and generalizable CSI-based HAR models. Compared to traditional HAR approaches, these CSI-based models hold the promise of significantly enhanced accuracy and robustness when deployed in real-world scenarios. This stems from their ability to capture the nuanced dynamics of human movement through the analysis of wireless channel characteristic from different spatial variations (utilizing two-diagonal ESP32 transceivers configuration) with higher degree of dimensionality (166 subcarriers).

Given the popularity of wrist-worn devices, particularly smartwatches, the identification of manual movement patterns has become of utmost interest within the research field of Human Activity Recognition (HAR) systems. In this context, by leveraging the numerous sensors natively embedded in smartwatches, the HAR functionalities that can be implemented in a watch via software and in a very cost-efficient way cover a wide variety of applications, ranging from fitness trackers to gesture detectors aimed at disabled individuals (e.g., for sending alarms), promoting behavioral activation or healthy lifestyle habits. In this regard, for the development of artificial intelligence algorithms capable of effectively discriminating these activities, it is of great importance to have repositories of movements that allow the scientific community to train, evaluate, and benchmark new proposals of movement detectors. The UMAHand dataset offers a collection of files containing the signals captured by a Shimmer 3 sensor node, which includes an accelerometer, a gyroscope, a magnetometer and a barometer, during the execution of different typical hand movements. For that purpose, the measurements from these four sensors, gathered at a sampling rate of 100 Hz, were taken from a group of 25 volunteers (16 females and 9 males), aged between 18 and 56, during the performance of 29 daily life activities involving hand mobility. Participants wore the sensor node on their dominant hand throughout the experiments.

Timed Up and Go (TUG) test is one of the most popular clinical tools aimed at the assessment of functional mobility and fall risk in older adults. The automation of the analysis of TUG movements is of great medical interest not only to speed up the test but also to maximize the information inferred from the subjects under study. In this context, this article describes a dataset collected from a cohort of 69 experimental subjects (including 30 adults over 60 years), during the execution of several repetitions of the TUG test. In particular, the dataset includes the measurements gathered with four wearables devices embedding four sensors (accelerometer, gyroscope magnetometer and barometer) located on four body locations (waist, wrist, ankle and chest). As a particularity, the dataset also includes the same measurements recorded when the young subjects repeat the test while wearing a commercial geriatric simulator, consisting of a set of weighted vests and other elements intended to replicate the limitations caused by aging. Thus, the generated dataset also enables the investigation into the potential of such tools to emulate the actual dynamics of older individuals.

Current remote sensing technologies employing Unmanned Aerial Vehicles (UAVs) for farm monitoring have shown promise in characterizing the environment through diverse sensor systems, including hyperspectral cameras, LiDAR, thermal cameras, and RGB sensors. However, these solutions often specialize in either activity recognition or crop monitoring, but not both. To address this limitation and enhance efficacy, we propose a multi-vision monitoring (MVM) framework capable of simultaneously recognizing farm activities and assessing crop health. Our approach involves computer vision techniques that transform aerial videos into sequential images to extract essential environmental features. Central to our framework are two pivotal components: the Farmer Activity Recognition (FAR) algorithm and the Crop Image Analysis (CIA). The FAR algorithm introduces a novel feature extraction method capturing motion across various maps, enabling distinct feature sets for each activity. Meanwhile, the CIA component utilizes the normalized Triangular Greenness Index (nTGI) to estimate leave chlorophyll levels, an important indicator for crop health. By unifying these components, we achieve dual functionality—activity recognition and crop health estimation—using identical input data, thereby enhancing efficiency and versatility in farm monitoring. Our framework employs a diverse range of machine learning models, demonstrating the potential of our extracted features to address the defined problem effectively in unison.

Study objectives: To develop a non-invasive and practical wearable method for long-term tracking of infants’ sleep. Methods: An infant wearable, NAPping PAnts (NAPPA), was constructed by combining a diaper cover and a movement sensor (triaxial accelerometer and gyroscope), allowing either real-time data streaming to mobile devices or offline feature computation stored in the sensor memory. A sleep state classifier (wake, N1/REM, N2/N3) was trained and tested for NAPPA recordings (N = 16649 epochs of 30 s), using hypnograms from co-registered polysomnography (PSG) as a training target in 33 infants (age 2 weeks to 18 months; Mean = 4). User experience was assessed from an additional group of 16 parents. Results: Overnight NAPPA recordings were successfully performed in all infants. The sleep state classifier showed good overall accuracy (78 %; Range 74–83 %) when using a combination of five features related to movement and respiration. Sleep depth trends were generated from the classifier outputs to visualise sleep state fluctuations, which closely aligned with PSG-derived hypnograms in all infants. Consistently positive parental feedback affirmed the effectiveness of the NAPPA-design. Conclusions: NAPPA offers a practical and feasible method for out-of-hospital assessment of infants’ sleep behaviour. It can directly support large-scale quantitative studies and development of new paradigms in scientific research and infant healthcare. Moreover, NAPPA provides accurate and informative computational measures for body positions, respiration rates, and activity levels, each with their respective clinical and behavioural value.

This work described a novel non-contact, wearable, real-time eye blink detection solution based on capacitive sensing technology. A custom-built prototype employing low-cost and low-power consumption capacitive sensors was integrated into standard glasses, with a copper tape electrode affixed to the frame. The blink of an eye induces a variation in capacitance between the electrode and the eyelid, thereby generating a distinctive capacitance-related signal. By analyzing this signal, eye blink activity can be accurately identified. The effectiveness and reliability of the proposed solution were evaluated through five distinct scenarios involving eight participants. Utilizing a user-dependent detection method with a customized predefined threshold value, an average precision of 92% and a recall of 94% were achieved. Furthermore, an efficient user-independent model based on the two-bit precision decision tree was further applied, yielding an average precision of 80% and an average recall of 81%. These results demonstrate the potential of the proposed technology for real-world applications requiring precise and unobtrusive eye blink detection.

Abstract Background Human activity Recognition (HAR) using smartphone sensors suffers from two major problems: sensor orientation and placement. Sensor orientation and sensor placement problems refer to the variation in sensor signal for a particular activity due to sensors’ altering orientation and placement. Extracting orientation and position invariant features from raw sensor signals is a simple solution for tackling these problems. Using few heuristic features rather than numerous time-domain and frequency-domain features offers more simplicity in this approach. The heuristic features are features which have very minimal effects of sensor orientation and placement. In this study, we evaluated the effectiveness of four simple heuristic features in solving the sensor orientation and placement problems using a 1D-CNN–LSTM model for a data set consisting of over 12 million samples. Methods We accumulated data from 42 participants for six common daily activities: Lying, Sitting, Walking, and Running at 3-Metabolic Equivalent of Tasks (METs), 5-METs and 7-METs from a single accelerometer sensor of a smartphone. We conducted our study for three smartphone positions: Pocket, Backpack and Hand. We extracted simple heuristic features from the accelerometer data and used them to train and test a 1D-CNN–LSTM model to evaluate their effectiveness in solving sensor orientation and placement problems. Results We performed intra-position and inter-position evaluations. In intra-position evaluation, we trained and tested the model using data from the same smartphone position, whereas, in inter-position evaluation, the training and test data was from different smartphone positions. For intra-position evaluation, we acquired 70–73% accuracy; for inter-position cases, the accuracies ranged between 59 and 69%. Moreover, we performed participant-specific and activity-specific analyses. Conclusions We found that the simple heuristic features are considerably effective in solving orientation problems. With further development, such as fusing the heuristic features with other methods that eliminate placement issues, we can also achieve a better result than the outcome we achieved using the heuristic features for the sensor placement problem. In addition, we found the heuristic features to be more effective in recognizing high-intensity activities.

Abstract Background The research gap addressed in this study is the applicability of deep neural network (NN) models on wearable sensor data to recognize different activities performed by patients with Parkinson’s Disease (PwPD) and the generalizability of these models to PwPD using labeled healthy data. Methods The experiments were carried out utilizing three datasets containing wearable motion sensor readings on common activities of daily living. The collected readings were from two accelerometer sensors. PAMAP2 and MHEALTH are publicly available datasets collected from 10 and 9 healthy, young subjects, respectively. A private dataset of a similar nature collected from 14 PwPD patients was utilized as well. Deep NN models were implemented with varying levels of complexity to investigate the impact of data augmentation, manual axis reorientation, model complexity, and domain adaptation on activity recognition performance. Results A moderately complex model trained on the augmented PAMAP2 dataset and adapted to the Parkinson domain using domain adaptation achieved the best activity recognition performance with an accuracy of 73.02%, which was significantly higher than the accuracy of 63% reported in previous studies. The model’s F1 score of 49.79% significantly improved compared to the best cross-testing of 33.66% F1 score with only data augmentation and 2.88% F1 score without data augmentation or domain adaptation. Conclusion These findings suggest that deep NN models originating on healthy data have the potential to recognize activities performed by PwPD accurately and that data augmentation and domain adaptation can improve the generalizability of models in the healthy-to-PwPD transfer scenario. The simple/moderately complex architectures tested in this study could generalize better to the PwPD domain when trained on a healthy dataset compared to the most complex architectures used. The findings of this study could contribute to the development of accurate wearable-based activity monitoring solutions for PwPD, improving clinical decision-making and patient outcomes based on patient activity levels.

RFID-based human activity recognition (HAR) attracts attention due to its convenience, non-invasiveness, and privacy protection. Existing RFID-based HAR methods use modeling, CNN, or LSTM to extract features effectively. Still, they have shortcomings: 1) requiring complex hand-crafted data cleaning processes and 2) only addressing single-person activity recognition based on specific RF signals. To solve these problems, this paper proposes a novel device-free method based on Time-streaming Multiscale Transformer called TransTM. This model leverages the Transformer's powerful data fitting capabilities to take raw RFID RSSI data as input without pre-processing. Concretely, we propose a multiscale convolutional hybrid Transformer to capture behavioral features that recognizes single-human activities and human-to-human interactions. Compared with existing CNN- and LSTM-based methods, the Transformer-based method has more data fitting power, generalization, and scalability. Furthermore, using RF signals, our method achieves an excellent classification effect on human behavior-based classification tasks. Experimental results on the actual RFID datasets show that this model achieves a high average recognition accuracy (99.1%). The dataset we collected for detecting RFID-based indoor human activities will be published.

Through-the-wall radar can penetrate walls and realize indoor human target detection. Deep learning is commonly used to extract the micro-Doppler signature of a target, which can be used to effectively identify human activities behind obstacles. However, the test accuracy of the deep-learning-based recognition methods is low with poor generalization ability when different testers are invited to generate the training set and test set. Therefore, this study proposes a method for recognition of anomalous human gait termination based on micro-Doppler corner features and Non-Local mechanism. In this method, Harris and Moravec detectors are utilized to extract the corner features of the radar image, and the corner feature dataset is established in this manner. Thereafter, multilink parallel convolutions and the Non-Local mechanism are utilized to construct the global contextual information extraction network to learn the global distribution characteristics of the image pixels. The semantic feature maps are generated by repeating four times the global contextual information extraction network. Finally, the probabilities of human activities are predicted using a multilayer perceptron. The numerical simulation and experimental results demonstrate that the proposed method can effectively identify such abnormal gait termination activities as sitting, lying down, and falling, among others, which occur in the process of indoor human walking, and successfully control the generalization accuracy error to be no more than \begin{document}$ 6.4\% $\end{document} under the premise of increasing the recognition accuracy and robustness.

Deep learning models have achieved significant success in human activity recognition, particularly in assisted living and telemonitoring. However, training these models requires substantial amounts of labeled training data, which is time-consuming and costly to acquire in real-world environments. Contrastive self-supervised learning has recently garnered attention in sensor-based activity recognition to mitigate the need for expensive large-scale data collection and annotation. Despite numerous related published papers, there remains a lack of literature reviews highlighting recent advances in contrastive self-supervised learning for sensor-based activity recognition. This paper extensively reviews 43 papers on recent contrastive self-supervised learning methods for sensor-based human activity recognition, excluding those related to video or audio sensors due to privacy concerns. First, we summarize the taxonomy of contrastive self-supervised learning, followed by a detailed description of contrastive learning models used for activity recognition and their main components. Next, we comprehensively review data augmentation methods for sensor data and commonly used benchmark datasets for activity recognition. The empirical performance comparisons of different methods are presented on benchmark datasets in linear evaluation, semi-supervised learning, and transfer learning scenarios. Through these comparisons, we derive significant insights into the selection of contrastive self-supervised models for sensor-based activity recognition. Finally, we discuss the limitations of current research and outline promising research directions for future exploration.

WiFi sensing is an emerging technology that utilizes wireless signals for various sensing applications. However, the reliance on supervised learning and the scarcity of labelled data and the incomprehensible channel state information (CSI) data pose significant challenges. These issues affect deep learning models’ performance and generalization across different environments. Consequently, self-supervised learning (SSL) is emerging as a promising strategy to extract meaningful data representations with minimal reliance on labelled samples. In this paper, we introduce a novel SSL framework called Context-Aware Predictive Coding (CAPC), which effectively learns from unlabelled data and adapts to diverse environments. CAPC integrates elements of Contrastive Predictive Coding (CPC) and the augmentation-based SSL method, Barlow Twins, promoting temporal and contextual consistency in data representations. This hybrid approach captures essential temporal information in CSI, crucial for tasks like human activity recognition (HAR), and ensures robustness against data distortions. Additionally, we propose a unique augmentation, employing both uplink and downlink CSI to isolate free space propagation effects and minimize the impact of electronic distortions of the transceiver. Our evaluations demonstrate that CAPC not only outperforms other SSL methods and supervised approaches, but also achieves superior generalization capabilities. Specifically, CAPC requires fewer labelled samples while significantly outperforming supervised learning by an average margin of 30.53% and surpassing SSL baselines by 6.5% on average in low-labelled data scenarios. Furthermore, our transfer learning studies on an unseen dataset with a different HAR task and environment showcase an accuracy improvement of 1.8% over other SSL baselines and 24.7% over supervised learning, emphasizing its exceptional cross-domain adaptability. These results mark a significant breakthrough in SSL applications for WiFi sensing, highlighting CAPC’s environmental adaptability and reduced dependency on labelled data.

Time Series Classification, is one of the very challenging areas in pattern recognition problems. As the volume of time series data increases, a multitude of TSC algorithms have been developed. Notably, only a limited number of these approaches have ventured into the realm of Deep Neural Networks to tackle this task. This paper presents a new hierarchical deep neural network architecture with a multi-section learning mechanism for the classification of multi-output time series datasets. The hierarchical structure of the proposed approach uses separate segments to detect output classes of different granular levels. The presented model does not suffer from heavy computations during training found in commonly used in other deep learning models such as convolutional neural network. Additionally, our suggested method handles data dimension reduction automatically in the hidden layers of different network sections and does not require separate pre-processing units for this purpose. To demonstrate the effectiveness of the proposed model, we have utilized it in the wearable sensor-based Human Activity Recognition problem and evaluated its performance on several benchmark datasets in TSC and HAR, such as PAMAP2 and Opportunity, with several different configurations. The empirical results on the benchmark datasets show that our proposed model outperforms several other deep learning-based solutions based on measures such as $F_{1}$ -Score, accuracy, and ROC curve.

Detecting falls in elderly individuals is crucial for their safety and well-being, especially in high-risk areas such as bathrooms. To enhance human activity recognition (HAR) while ensuring user privacy, we developed an advanced fall detection system using ultra-wideband (UWB) radar technology. The UWB radar transmits high-frequency pulse signals that are used to reflect human activities without compromising user privacy. Our innovative continuous human state classification approach analyzes the sequences of the activities identified using a single-shot recognition algorithm. This method interprets these activities as cohesive events, leveraging the preceding activities to predict and identify falls. Our algorithm meticulously organized these activities into a comprehensive chronicle of human behavior, significantly enhancing the model’s ability to accurately identify falls. The historical context provided by this chronicle boosted the model’s performance. Our research revealed that the continuous human state classification model increased the accuracy by 26 % compared with that of the single-shot approach. Furthermore, we incorporated a novel transformer-based model that surpasses previous methods employing convolutional neural networks (CNN) and long short-term memory (LSTM) models. This study provides an exhaustive account of the design, implementation, and evaluation of our approach, demonstrating its effectiveness through real-world testing. Our findings highlight the superior accuracy and reliability of the continuous human state classification model for fall detection, thereby setting a new standard in elderly care technology.

The growing popularity of wearable technology has led to a surge in smartwatch usage among the general public. These devices offer a range of features, including internet connectivity, fitness tracking, and real-time notifications, making them valuable tools for staying connected to the online world while remaining engaged in real-world activities. Smartwatches have become powerful platforms for Human Activity Recognition (HAR) applications thanks to the increasing computational power and the presence of a wide array of sensors, such as accelerometers, gyroscopes, heart rate, and step counters. Efficient real-time data collection from internal sensors is a crucial requirement for HAR applications on wearable devices due to their constraints in battery size and duration. In this paper, we introduce the implementation of three energy-efficient user-level libraries developed for real-time data collection from inertial sensors using native Wear OS APIs and different techniques: Thread, Flow, and Channel. Experiments were conducted on a commercially available Oppo smartwatch comparing them in terms of code size, memory utilization, and energy consumption. The characterization results show that the Channel implementation, which reduces code size by 45% and consumes 75% less energy, is lightweight and versatile. This makes it well-suited for wearable devices without significantly impacting battery life and system performance. Additionally, our findings indicate that choice of programming approach significantly impacts energy consumption, highlighting the importance of optimizing performance and battery life. Furthermore, understanding the interactions between application and system optimization policies is essential for improving energy efficiency in Wear OS applications.

The quest for efficient and non-intrusive human activity recognition (HAR) in indoor environments has led to the burgeoning field of WiFi-based HAR. This method promises applications from healthcare monitoring to elderly care due to its cost-effectiveness and ease of deployment compared to traditional sensor-based systems. However, WiFi-based HAR faces significant challenges in maintaining performance across diverse environments and subjects, largely due to WiFi signal variability. Addressing this issue necessitates training models on extensive datasets. Recent studies have utilized conventional models, including Convolutional Neural Networks (CNNs), and sequence-to-sequence (Seq2Seq) models such as LSTM, GRU, or Transformer. Despite their precision, these models are computationally intensive and require more training data. To tackle these limitations, we propose a novel approach that leverages a Temporal Convolutional Network with Augmentations and Attention, referred to as TCN-AA. This model enhances computational efficiency and accuracy in the face of dataset variability by combining temporal convolutional layers with data augmentation strategies and an attention mechanism to focus on critical features. Our method is computationally efficient and significantly improves accuracy, even with a threefold increase in data size through augmentation techniques. Experiments on a public dataset indicate our approach outperforms state-of-the-art methods, achieving 99.42% accuracy. This work represents a significant step forward in the practical application of WiFi-based HAR, paving the way for broader adoption in critical areas like healthcare and security.

In recent years, the remarkable advancement of WiFi sensing technologies has opened new frontiers in human activity recognition, enabling innovative solutions that transcend traditional methods and improve the capabilities of intelligent environments. Individual dynamic movements such as walking, sitting, standing, and running, as well as more complex interactions such as sports activities, are all examples of human activity. WiFi sensing has emerged as a powerful tool for human activity recognition; however, certain restrictions persist, especially when sensing activities involving multiple users across different locations. These limitations highlight the need for innovative techniques to address the intricacies of multi-user scenarios and environmental effects, ensuring the robustness and accuracy of WiFi-based sensing systems. To address multi-user effects in WiFi signals, we propose a few layering LSTM deep learning models with Raspberry Pi for edge computing solutions. The method leverages the decomposition of Channel State Information (CSI) signals through Independent Component Analysis (ICA) and Continuous Wavelet Transform (CWT). The integration of signal decomposition and deep learning holds promise for advancing WiFi sensing systems’ accuracy, reliability, and real-time capabilities in complex environments and multi-user scenarios. Experimental findings prove the system’s ability to handle complex activities with high classification accuracy. Furthermore, the system displays a remarkable ability to classify complex activities. By leveraging the power of deep learning, the model learns intricate patterns and relationships within the decomposed CSI signals, enabling it to distinguish between diverse activities with high accuracy.

Regular monitoring of physical activities is essential in mitigating the risks associated with diseases like heart problems, obesity, and diabetes. Recent studies have emphasized the significance of Human Activity Recognition (HAR) in tracking physical movements, which aids in enhancing healthcare. The detection of activities could be done by analyzing different patterns and interpreting the signal trends, and variations in the individual performance of activities can lead to inconsistent sensor signals. To address these challenges, this work utilizes Inertial Measurement Unit (IMU) data, converts it into spectrograms through time and frequency analysis, and primarily employs the short-time Fourier transform (STFT) technique. This strategy specifically implements Ghost Neural Network with Fire-Hawk Optimizer (GNet-FHO) to analyze both time-related characteristics from the spectrograms and distinct spatial attributes of each spectrogram. It also effectively identifies spatial correlations among various spectrogram types. Through this method, there is a remarkable improvement in the feature extraction and thereby enhances the accuracy in identifying human activities. Feature selection is done with the help of the FHO, and it is better than the Adam optimizer in context of robust global optimization, effective handling of complex landscapes also the performance is analyzed using the evaluation metrics. Experimental results exhibit that GNet-FHO outperforms other existing algorithms, establishing its efficacy as a lightweight model for human activity recognition. According to our findings, the algorithms demonstrated a success rate of 99.01% and 98.97% when applied to the WISDM smartwatch and smartphone dataset. Additionally, achieved a success rate of 97.6% on the MOTION SENSE dataset and 95.21% on the UCI-HAR dataset. Notably, these findings outperformed those published in previous research that used the identical datasets.

Human man Activity Recognition (HAR) using wearable sensors, particularly wrist-worn devices, has garnered significant research interest. However, challenges such as sensor displacement and variations in wearing habits can affect the accuracy of HAR systems. Two compensation strategies for sensor displacemnt are proposed to address these issues. The first strategy is hybrid data fusion, which involves merging sensor data collected from different displacement locations on the wrist. This technique aims to mitigate the discrepancies in data distribution that result from the multiple wearing positions along the wrist, thereby enhancing the overall accuracy of HAR models. The second strategy is cross-location transfer fine-tuning, which involves pretraining a model with data from typical wrist locations and then fine-tuning it with data from a new sensor location. This approach improves the model’s ability to adapt and perform accurately when the sensor is placed in a different position, significantly enhancing its performance and generalization capabilities. To verify the effectiveness of these proposed compensation strategies, we built an LSTM baseline model and introduce a new Multi-stage Feature Extraction (MSFE) model that integrates 1D CNN and attention. Experiments on common activities such as walking, standing, using stairs, and lying down, with data recorded at multiple locations along the wrist, have shown that both hybrid data fusion and cross-location transfer fine-tuning strategies notably improve the recognition accuracy of HAR models. The proposed MSFE model achieves higher recognition accuracies than the LSTM model in all six experimental scenarios, particularly in Scenario 5 involving sensor displacement, with an improvement of up to 31.65%. Additionally, thecross-location transfer fine-tuning strategy enhances the recognition accuracy by 9.19% for Subject 3 with sensor displacement at the right wrist location. These advancements in handling sensor displacement and wearing variations are crucial for developing more reliable and versatile wearable technologies.INDEX TERMS Sensor displacement compensation, human activity recognition, deep learning, wrist.

Augmented Reality (AR) technology for motion tracking is often used on mobile devices with an RGB front camera or LiDAR sensor rear camera. While there are various AR applications for exercise, there is a lack of comparative studies investigating the specific effects of these technologies on the AR exercise experience. This study evaluated the performance and usability of using the RGB camera and the LiDAR sensor for exercise monitoring with a mobile AR application. We examined their performance in different environmental room conditions: a solid wall, a glass wall, and a wall with objects. Focusing on a marching-in-place use case, we assessed accuracy and usability across participants with varying body mass index (BMI). Our application provided display and audio notifications for correct posture compared with validation by an expert physical therapist. The results indicated that differences in BMI did not significantly affect accuracy. Algorithm 1. The LiDAR provided higher accuracy in various environments, while the RGB camera provided higher scores in usability. The results suggested that standing position affected the detection both RGB and LiDAR cameras had better accuracy when standing at a 45-degree angle than directly facing the camera. This study showed the potential of both technologies for telehealth scenarios, emphasizing the significance of practical usage in households and ease of use to support exercise monitoring and empower users to achieve fitness goals and telehealth.

Advancements in smartphone sensor technologies have significantly enriched the field of human activity recognition, facilitating a wide array of applications from health monitoring to personal navigation. This study utilized such advancements to explore human locomotion and localization recognition using data from accelerometers, microphones, gyroscopes, magnetometers, and GPS, applying Deep Polynomial Neural Networks (DPNN) and Multilayer Perceptron (MLP) across three datasets: the Continuous In-The-Wild Smart Watch Activity Dataset, the Huawei Locomotion Dataset, and the Extra Sensory Dataset. We employ two distinct approaches for activity recognition: Deep Polynomial Neural Networks (DPNN) for deep learning-based feature extraction and Multilayer Perceptron (MLP) with manual feature extraction techniques, including Linear Predictive Coding Cepstral Coefficients (LPCC), step length, signal magnitude area, spectral, and sound features. Through rigorous experimentation, we achieved remarkable accuracy in recognizing both locomotion and localization activities, with DPNN consistently outperforming MLP in terms of accuracy. Specifically, for the Continuous In-The-Wild Dataset, DPNN achieved a 93% accuracy rate for localization activities and 95% for locomotion activities, while MLP recorded 86% and 91% in the respective categories. Similarly, on the Huawei Locomotion Dataset, DPNN attained 95% accuracy for localization and 97% for locomotion, with MLP achieving 88% and 91%, respectively. Furthermore, the application of these models to the Extra Sensory Dataset yielded 92% accuracy for both localization and locomotion activities with DPNN, and 90% and 89% with MLP. In our study, we observed that in terms of accuracy, DPNN emerges as the clear winner; however, it is computationally expensive. Conversely, MLP, while being less accurate, stands out for its computational efficiency. This study not only highlights the effectiveness of incorporating advanced machine learning techniques in interpreting sensor data but also emphasizes the trade-offs between computational demands and accuracy in the domain of human activity recognition. Through our comprehensive analysis, we contribute valuable insights into the potential of smartphone sensors in enhancing activity recognition systems, paving the way for future innovations in mobile sensing technology.

Human activity recognition (HAR) has attracted considerable research attention due to its essential role in various domains, ranging from healthcare to security, safety, and entertainment. HAR has undergone a paradigm shift from simple single-task detection to the more complex task of identifying multiple simultaneous activities as technology advances. A wide range of methods, including sensing modalities, identification algorithms, a specified list of recognized activities, and end application goals, have been used in the literature to investigate activities carried out by single individuals. However, there appears to be a research gap when it comes to scenarios in which several people engage in individual or concurrent activities. Although numerous reviews and surveys have previously addressed HAR, with the continual expansion of literature, there is a necessity for an updated assessment of the status of HAR literature. The system encompasses various operational modules, including data acquisition, noise elimination, and distortion reduction through preprocessing, followed by feature extraction, feature selection, and classification. Recent advancements have introduced state-of-the-art techniques for feature extraction and selection, which are categorized using traditional machine learning classifiers. However, a notable limitation is observed, as many of these techniques rely on basic feature extraction processes, hindering their capability to recognize complex activities. This article reviews 190 articles with respect to data collection, segmentation, feature extraction, energy efficiency, personalized models, and machine learning (ML) and deep learning (DL) approaches for sensor-based HAR. Open challenges and future enhancements of HAR are also discussed in this article.

The rapid advancements in wireless technologies have led to numerous research studies that provide evidence of the successful utilization of wireless signals, particularly, WiFi signals for human activity sensing in the indoor environment. As a promising technology, WiFi-based human sensing can be utilized for a variety of applications such as smart healthcare, smart homes, security, industry, office indoor environments etc., due to the availability of rich infrastructure. Furthermore, compared to other radio frequency (RF) based systems such as radio detection and ranging (RADAR) and radio frequency identification (RFID), WiFi is non-invasive, has low-cost, and provides ubiquitous coverage in the indoor setup. However, due to the limited accuracy and high complexity of the model-based approaches for human sensing, the systems empowered by the deep learning (DL) techniques have achieved remarkable performance gains and showed more robustness in dealing with complicated human sensing tasks. The article explores the physical layer parameters used in WiFi sensing such as received signal strength indicator (RSSI) and channel state information (CSI), the estimated parameters such as angle-of-arrival (AoA) and Doppler shift (DS) along with frequency modulated continuous wave (FMCW) RADAR technology. Moreover, the preliminary signal processing stages that are applied to the received WiFi signals in the DL assisted systems are discussed. This article provides a comprehensive literature survey on the recent advances in DL-empowered WiFi sensing focusing on human activity recognition and movement tracking followed by fall detection, single task-multi task classification, crowd monitoring and sensing, indoor localization, gaits recognition, and pose estimation. Furthermore, the paper highlight the challenges in the existing literature and discusses the possible future research directions in WiFi-based human sensing assisted by DL techniques.

In recent years, the use of WiFi Channel State Information (CSI) for Human Activity Recognition (HAR) has attracted widespread attention, thanks to its low cost and non-intrusive advantages. Previous research mostly used models based on Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN) for activity recognition. However, these methods fail to achieve good parallelism while learning global features and fine-grained features, so they often cannot achieve the ideal recognition effect or training speed. In light of this, we propose an ensemble deep learning model based on CNN and Transformer, ConTransEn. Specifically, we first use CNN to extract spatial features of the sequence, and then use Vision Transformer (ViT) to further extract temporal features. The Transformer introduces self-attention mechanism, enabling the model to fully consider information from other positions in the sequence, rather than being limited to the current input. Furthermore, due to the increased parallelism, Transformer has an advantage in training speed over RNN. In order to further improve the accuracy and robustness of the model, we adopt a bagging ensemble learning strategy, integrating the prediction results of multiple homogeneous base models using a soft voting method to obtain the final classification result. This ensemble learning method reduces the risk of model overfitting, and improves the overall accuracy and reliability of the model. We extensively evaluated the model on two publicly available datasets, and achieved excellent recognition results, indicating its good performance and robustness. The average recognition accuracy on the UT-HAR dataset reached 99.41%, surpassing existing solutions.

Nowadays, the advancements of wearable consumer devices have become a predominant role in healthcare gadgets. There is always a demand to obtain robust recognition of heterogeneous human activities in complicated IoT environments. The knowledge attained using these recognition models will be then combined with healthcare applications. In this way, the paper proposed a novel deep learning framework to recognize heterogeneous human activities using multimodal sensor data. The proposed framework is composed of four phases: employing dataset and processing, implementation of deep learning model, performance analysis, and application development. The paper utilized the recent KU-HAR database with eighteen different activities of 90 individuals. After preprocessing, the hybrid model integrating Extreme Learning Machine (ELM) and Gated Recurrent Unit (GRU) architecture is used. An attention mechanism is then included for further enhancing the robustness of human activity recognition in the IoT environment. Finally, the performance of the proposed model is evaluated and comparatively analyzed with conventional CNN, LSTM, GRU, ELM, Transformer and Ensemble algorithms. To the end, an application is developed using the Qt framework which can be deployed on any consumer device. In this way, the research sheds light on monitoring the activities of critical patients by healthcare professionals remotely. The proposed ELM-GRUaM model achieved supreme performance in recognizing multimodal human activities with an overall accuracy of 96.71% as compared with existing models.

Parkinson’s disease is a progressive neurodegenerative disorder with significant fluctuations throughout the day, making accurate drug treatment difficult. A home-based long-term monitoring system is essential to address this challenge. Contemporary approaches to activity monitoring have focused on wearable devices and computer vision systems. Wearable devices are often uncomfortable and not ideal for long-term monitoring, while computer vision is plagued with significant privacy concerns. In this context, Wi-Fi sensing presents itself as an advantageous alternative due to its non-invasive and privacy-preserving properties. However, current human activity recognition methodologies lack the specificity to identify disease-related symptoms within everyday activities. Furthermore, the efficiency of human activity recognition methods in processing continuous data streams in real time is a crucial aspect that needs thorough assessment. This study proposes a novel approach for human activity recognition using Wi-Fi signals. Traditional methods for signal processing are avoided by converting the ratio of channel state information from antenna pairs into images. These images are then processed using a convolutional neural network to detect movements related to diseases in a large dataset. The experiments utilize a laptop PC with Intel Wi-Fi Link 5300 and a receiver equipped with three external 12 dB omnidirectional antennas in the 2.4 GHz band and cover various daily activities. The proposed method has demonstrated remarkable accuracy, with an average recognition rate of 93.8% in validation. It also showcased a consistent accuracy range of 91.9% to 95.2% in generalization tests, proving its effectiveness in different environments, with various individuals, and under assorted Wi-Fi configurations. A performance test of our method revealed that it processes raw CSI to recognition results in just 0.65 seconds per second of data, highlighting its potential for real-time applications.

Anomaly detection is a critical task in ensuring the security and safety of infrastructure and individuals in smart environments. This paper provides a comprehensive analysis of recent anomaly detection solutions in data streams supporting smart environments, with a specific focus on multivariate time series anomaly detection in various environments, such as smart home, smart transport, and smart industry. The aim is to offer a thorough overview of the current state-of-the-art in anomaly detection techniques applicable to these environments. This includes an examination of publicly available datasets suitable for developing these techniques. The survey is designed to inform future research and practical applications in the field, serving as a valuable resource for researchers and practitioners. It not only reviews a range of state-of-the-art anomaly detection methods, from statistical and proximity-based to those adopting deep learning-methods but also covers fundamental aspects of anomaly detection. These aspects include the categorization of anomalies, detection scenarios, challenges associated, and evaluation metrics for assessing the techniques’ performance.

Parkinson’s disease affects bodily functions and there is a growing need for advanced solutions to offer therapeutic advice to patients. A framework using artificial intelligence and machine learning techniques has been proposed to address this. The proposed system employs a combination of RGB, inertial, and depth sensors data. The inertial signals have been filtered using a notch filter to obtain the optimal wearable sensor data by examining the upper and lower cutoff frequencies. Multiple features have been calculated, including mel frequency cepstral coefficients (MFCC), statistical features and Gaussian mixture model (GMM) features. On the other hand, silhouettes have been extracted from RGB and depth images, and four crucial parameters have been employed to gauge the level of accuracy with which patients with neurological disorders performed their activities, including the angle formation between the hands and lower/upper half of the body and center of the body, and the angle between the hands of the silhouette. The resulting features have been fused and classified using principle component analysis and a reweighted genetic algorithm. Evaluation using cross-validation on mRI (multi-modal 3d human pose estimation dataset using mmwave, RGB-D, and inertial sensors) and MHEALTH (mobile health) datasets showed a recognition accuracy rate of 97.29% and a 97.94%. The study highlights the need for more datasets to address challenges in rehabilitation using human activity recognition with multi-modal sensors.

Research in the field of human activity recognition is very interesting due to its potential for various applications such as in the field of medical rehabilitation. The need to advance its development has become increasingly necessary to enable efficient detection and response to a wide range of movements. Current recognition methods rely on calculating changes in joint distance to classify activity patterns. Therefore, a different approach is required to identify the direction of movement to distinguish activities exhibiting similar joint distance changes but differing motion directions, such as sitting and standing. The research conducted in this study focused on determining the direction of movement using an innovative joint angle shift approach. By analyzing the joint angle shift value between specific joints and reference points in the sequence of activity frames, the research enabled the detection of variations in activity direction. The joint angle shift method was combined with a Deep Convolutional Neural Network (DCNN) model to classify 3D datasets encompassing spatial-temporal information from RGB-D video image data. Model performance was evaluated using the confusion matrix. The results show that the model successfully classified nine activities in the Florence 3D Actions dataset, including sitting and standing, obtaining an accuracy of (96.72 ± 0.83)%. In addition, to evaluate its robustness, this model was tested on the UTKinect Action3D dataset, obtaining an accuracy of 97.44%, proving that state-of-the-art performance has been achieved.

This paper introduces AutoGCN, a generic Neural Architecture Search (NAS) algorithm for Human Activity Recognition (HAR) using Graph Convolution Networks (GCNs). HAR has enjoyed increased attention due to advances in deep learning, increased data availability, and enhanced computational capabilities. Concurrently, GCNs have shown promising abilities in modeling relationships between body key points in a skeletal graph. Typically, domain experts develop dataset-specific GCN-based methods, which limits their applicability beyond the specific context. AutoGCN seeks to address this limitation by simultaneously searching for the ideal hyperparameters and architecture combination within a versatile search space using a reinforcement controller while balancing optimal exploration and exploitation behavior with a knowledge reservoir during the search process. We conduct extensive experiments on two large datasets focused on skeleton-based action recognition to assess the proposed algorithm’s performance. Our experimental results demonstrate the effectiveness of AutoGCN in constructing optimal GCN architectures for HAR, outperforming conventional NAS and GCN methods, as well as random search. These findings highlight the significance of a diverse search space and an expressive input representation to achieve good model performance and generalizability.

Healthcare is an area of concern where the application of human-centred design practices and principles can enormously affect well-being and patient care. The provision of high-quality healthcare services requires a deep understanding of patients’ needs, experiences, and preferences. Human activity recognition (HAR) is paramount in healthcare monitoring by using machine learning (ML), sensor data, and artificial intelligence (AI) to track and discern individuals’ behaviours and physical movements. This technology allows healthcare professionals to remotely monitor patients, thereby ensuring they adhere to prescribed rehabilitation or exercise routines, and identify falls or anomalies, improving overall care and safety of the patient. HAR for healthcare monitoring, driven by deep learning (DL) algorithms, leverages neural networks and large quantities of sensor information to autonomously and accurately detect and track patients’ behaviors and physical activities. DL-based HAR provides a cutting-edge solution for healthcare professionals to provide precise and more proactive interventions, reducing the burden on healthcare systems and improving patient well-being while increasing the overall quality of care. Therefore, the study presents an improved coyote optimization algorithm with a deep learning-assisted HAR (ICOADL-HAR) approach for healthcare monitoring. The purpose of the ICOADL-HAR technique is to analyze the sensor information of the patients to determine the different kinds of activities. In the primary stage, the ICOADL-HAR model allows a data normalization process using the Z-score approach. For activity recognition, the ICOADL-HAR technique employs an attention-based long short-term memory (ALSTM) model. Finally, the hyperparameter tuning of the ALSTM model can be performed by using ICOA. The stimulation validation of the ICOADL-HAR model takes place using benchmark HAR datasets. The wide-ranging comparison analysis highlighted the improved recognition rate of the ICOADL-HAR method compared to other existing HAR approaches in terms of various measures.

Pressure insoles allow for the collection of real time pressure data inside and outside a laboratory setting as they are non-intrusive and can be simply integrated into industrial environments for occupational health and safety monitoring purposes. Activity detection is important for the safety and wellbeing of workers, and the present study aims to employ pressure insoles to detect the type of industry-related task an individual is performing by using random forest, an artificial intelligence-based classification technique. Twenty subjects wore loadsol® pressure insoles and performed five specific tasks associated with a typical workflow: standing, walking, pick and place, assembly, and manual handling. For each activity, statistical and morphological features were extracted to create a training dataset. The classifier performed with an accuracy of 82%, and a re-analysis focusing on the five most influential features resulted in 83% accuracy. These accuracies are comparable to similar task classification studies but with the benefit of added explainability, which increases transparency and, thereby, trust in the classifier decisions. The combination of random forest and in-depth feature analysis (SHAP) provided insights into the importance of certain features and the impact of their value on the classification of each task. The insights obtained from these methods can aid in the design of pressure insoles that are optimized for the extraction of impactful features and the prevention of work-related musculoskeletal disorders in Industry 4.0 operators.

Human activity recognition (HAR) plays a crucial role in human-computer interaction, smart home, health monitoring and elderly care. However, existing methods typically utilize camera, radio frequency (RF) signals or wearable devices for activity recognition. Each single-sensor modality has its inherent limitations, like camera-based methods having blind spots, Wi-Fi-based methods depending on the environment and the inconvenience of wearing Inertial Measurement Unit (IMU) devices. In this paper, we propose a HAR system that leverages three types of sensor combinations: Wi-Fi, IMU and a hybrid of Wi-Fi+IMU. We utilize the Channel State Information (CSI) provided by Wi-Fi and the accelerometer and gyroscope data from IMU devices to capture activity characteristics. Then, we employ six machine learning algorithms to recognize eight types of daily activities. These algorithms include Support Vector Machine (SVM), Multi-layer Perceptron (MLP), Decision Tree, Random Forest, Logistic Regression and k-Nearest Neighbors (kNN). Additionally, we investigate the accuracy of hand gesture recognition using different sensor combinations and analyze the calculation speed of each combination. We conduct a survey to collect user feedback on the performance of various sensor combinations in our HAR system. The results show that the combination of CSI+IMU yields the best HAR recognition accuracy, with a accuracy of 89.38%. The SVM algorithm consistently performs well across all systems, especially excelling in the CSI+IMU system supported by energy and average Fast Fourier Transform (FFT) features. However, we also find that the success of sensor fusion depends on specific algorithms and features. Fusion of CSI and IMU does not universally enhance recognition accuracy for all features and algorithms and can, in some cases, actually reduce accuracy.