Search

Over the past few decades, numerous studies have focused on identifying and recognizing human actions using machine learning and computer vision techniques. Video-based human action recognition (HAR) aims to detect actions from video sequences automatically. This can cover simple gestures to complex actions involving multiple people interacting with objects. Actions in team sports exhibit a different nature compared to other sports, since they tend to occur at a faster pace and involve more human-human interactions. As a result, research has typically not focused on the challenges of HAR in team sports. This paper comprehensively summarises HAR-related research and applications with specific focus on team sports such as football (soccer), basketball and Australian rules football. Key datasets used for HAR-related team sports research are explored. Finally, common challenges and future work are discussed, and possible research directions identified. [ABSTRACT FROM AUTHOR]

Location-based social media data provides a new perspective for understanding the relationship between human behavior and urban environments. However, further research is needed to determine the application of cognitive computing in urban environments and physical activities. This study proposes a cognitive computing framework for urban environments and human activities that extracts knowledge from structured and unstructured data through natural language processing and computer vision techniques. This paper utilizes a Naive Bayes Model constructed based on random reviews, as well as semantic segmentation and instant segmentation algorithms based on convolutional neural networks to obtain information about urban environments and human behavior from social media data and other geospatial resources. This study examines the relationship between the urban environment and residents' activity, including spatiotemporal behavior, public sentiment, and physical activity. The study found statistically significant results in subgroup analyses regarding the effects of urban environments on sentiment and physical activity, which also exhibited a strong social gradient consistent with traditional findings. This study validates the feasibility of using cognitive computing based on social media data to explore environmental behaviors, providing technical support for updating health promotion policies. [ABSTRACT FROM AUTHOR]

Human Interaction Recognition (HIR) was one of the challenging issues in computer vision research due to the involvement of multiple individuals and their mutual interactions within video frames generated from their movements. HIR requires more sophisticated analysis than Human Action Recognition (HAR) since HAR focuses solely on individual activities like walking or running, while HIR involves the interactions between people. This research aims to develop a robust system for recognizing five common human interactions, such as hugging, kicking, pushing, pointing, and no interaction, from video sequences using multiple cameras. In this study, a hybrid Deep Learning (DL) and Machine Learning (ML) model was employed to improve classification accuracy and generalizability. The dataset was collected in an indoor environment with four-channel cameras capturing the five types of interactions among 13 participants. The data was processed using a DL model with a fine-tuned ResNet (Residual Networks) architecture based on 2D Convolutional Neural Network (CNN) layers for feature extraction. Subsequently, machine learning models were trained and utilized for interaction classification using six commonly used ML algorithms, including SVM, KNN, RF, DT, NB, and XGBoost. The results demonstrate a high accuracy of 95.45% in classifying human interactions. The hybrid approach enabled effective learning, resulting in highly accurate performance across different interaction types. Future work will explore more complex scenarios involving multiple individuals based on the application of this architecture. [ABSTRACT FROM AUTHOR]

As the proportion of elderly individuals in developed countries continues to rise globally, addressing their healthcare needs, particularly in preserving their autonomy, is of paramount concern. A growing body of research focuses on Ambient Assisted Living (AAL) systems, aimed at alleviating concerns related to the independent living of the elderly. This systematic review examines the literature pertaining to fall detection and Human Activity Recognition (HAR) for the elderly, two critical tasks for ensuring their safety when living alone. Specifically, this review emphasizes the utilization of Deep Learning (DL) approaches on computer vision data, reflecting current trends in the field. A comprehensive search yielded 2,616 works from five distinct sources, spanning the years 2019 to 2023 (inclusive). From this pool, 151 relevant works were selected for detailed analysis. The review scrutinizes the employed DL models, datasets, and hardware configurations, with particular emphasis on aspects such as privacy preservation and real-world deployment. The main contribution of this study lies in the synthesis of recent advancements in DL-based fall detection and HAR for the elderly, providing insights into the state-of-the-art techniques and identifying areas for further improvement. Given the increasing importance of AAL systems in enhancing the quality of life for the elderly, this review serves as a valuable resource for researchers, practitioners, and policymakers involved in developing and implementing such technologies. [ABSTRACT FROM AUTHOR]

Nowadays, video surveillance systems are gaining increasing attention in the computer vision fields due to user demands for security purposes. Observing human movement and predicting such movement perception are promising due to video surveillance systems. In this work, Hierarchical Auto-Associative Polynomial Convolutional Neural Networks (HA-PCNN) with Garran Rufa Fish Optimization (GRFO) is proposed for Human Activity Recognition (HAR) using a video surveillance system. Initially, video surveillance systems-based human activity images are obtained, which are collected by video camera by recording daily human activities. After that, the input images are fed to a pre-processing approach named as Switched Mode Fuzzy Median Filter (SMFM) method. In this, the noise presented in the input images is reduced by applying the SMFM model to normalize the dataset and improve the image quality. After that, the pre-processed images are given to the developed Fast Discrete Curvelet Transform with Wrapping (FDCT-WRP)-based feature extraction method to extract the relevant features. Moreover, the extracted features are given to the HA-PCNN model for HA classification. Generally, the HA-PCNN method does not express any adoption of optimization methods for scaling the ideal parameters and classification. Here, Garra Rufa Fish Optimization (GRFO) is proposed to enhance the parameters of HA-PCNN. Thus, the proposed HA-PCNN-GRFO methodology can classify human activities like standing, sitting, running, walking and sleeping. By then the performance of the proposed HA-PCNN-GRFO methodology is evaluated using Python Platform, and metrics like recall, accuracy, F-measure, precision, and specificity are analyzed. Thus, the proposed HA-PCNN-GRFO approach achieved 97.3% average accuracy rate, 96.5% average recall, 97.2% average precision, 95.9% average specificity, and 96.8% average F1-score for classifying human activities using UCI HAR dataset. Also, the proposed HA-PCNN-GRFO approach is outperformed in HAR than the conventional approaches. [ABSTRACT FROM AUTHOR]

Human activity recognition is a burgeoning field with the aim of observing and understanding human actions, focusing on appearance and movement. The goal of this field is to develop advanced systems for creating lifelike models and facilitating interactions. Computer vision is the foundation, serving surveillance, robot learning, user interfaces, and human–computer interaction. CCTV advancements enable impactful applications like home nursing and elderly care through video surveillance. However, extracting behavior from varied camera sources remains challenging. This paper introduces a proof of concept: a cost-effective fall detection system using a 5MP Pi camera connected via MIPI. A three-stage hybrid Haar Cascade model coupled with video frame sequences, utilizing background subtraction, achieves 89.21% accuracy with 2.5% false positives and 2.0% false negatives, surpassing state-of-the-art methods. The system's core purpose is to accurately monitor human movement, particularly detecting falls, and in emergencies, alert family via SMS or Email for the safety of solitary elderly individuals having age 60 years and above. [ABSTRACT FROM AUTHOR]

Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the past decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of 'fusing' the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR. [ABSTRACT FROM AUTHOR]

Human activity recognition has a wide range of applications in various fields, such as video surveillance, virtual reality and human–computer intelligent interaction. It has emerged as a significant research area in computer vision. GCN (Graph Convolutional networks) have recently been widely used in these fields and have made great performance. However, there are still some challenges including over-smoothing problem caused by stack graph convolutions and deficient semantics correlation to capture the large movements between time sequences. Vision Transformer (ViT) is utilized in many 2D and 3D image fields and has surprised results. In our work, we propose a novel human activity recognition method based on ViT (HAR-ViT). We integrate enhanced AGCL (eAGCL) in 2s-AGCN to ViT to make it process spatio-temporal data (3D skeleton) and make full use of spatial features. The position encoder module orders the non-sequenced information while the transformer encoder efficiently compresses sequence data features to enhance calculation speed. Human activity recognition is accomplished through multi-layer perceptron (MLP) classifier. Experimental results demonstrate that the proposed method achieves SOTA performance on three extensively used datasets, NTU RGB+D 60, NTU RGB+D 120 and Kinetics-Skeleton 400. [ABSTRACT FROM AUTHOR]

Human Activity Recognition (HAR) research has gained significant momentum due to recent technological advancements, artificial intelligence algorithms, the need for smart cities, and socioeconomic transformation. However, existing computer vision and sensor-based HAR solutions have limitations such as privacy issues, memory and power consumption, and discomfort in wearing sensors for which researchers are observing a paradigm shift in HAR research. In response, WiFi-based HAR is gaining popularity due to the availability of more coarse-grained Channel State Information. However, existing WiFi-based HAR approaches are limited to classifying independent and non-concurrent human activities performed within equal time duration. Recent research commonly utilizes a Single Input Multiple Output communication link with a WiFi signal of 5 GHz channel frequency, using two WiFi routers or two Intel 5300 NICs as transmitter-receiver. Our study, on the other hand, utilizes a Multiple Input Multiple Output radio link between a WiFi router and an Intel 5300 NIC, with the time-series Wi-Fi channel state information based on 2.4 GHz channel frequency for mutual human-to-human concurrent interaction recognition. The proposed Self-Attention guided Bidirectional Gated Recurrent Neural Network (Attention-BiGRU) deep learning model can classify 13 mutual interactions with a maximum benchmark accuracy of 94% for a single subject-pair. This has been expanded for ten subject pairs, which secured a benchmark accuracy of 88% with improved classification around the interaction-transition region. An executable graphical user interface (GUI) software has also been developed in this study using the PyQt5 python module to classify, save, and display the overall mutual concurrent human interactions performed within a given time duration. Finally, this article concludes with a discussion of the possible solutions to the observed limitations and identifies areas for further research. Such a Wi-Fi channel perturbation pattern analysis is believed to be an efficient, economical, and privacy-friendly approach to be potentially utilized in mutual human interaction recognition for indoor activity monitoring, surveillance system, smart health monitoring systems, and independent assisted living. [ABSTRACT FROM AUTHOR]

Human pose estimation (HPE) is a technique used in computer vision and artificial intelligence to detect and track human body parts and poses using images or videos. Widely used in augmented reality, animation, fitness applications, and surveillance, HPE methods that employ monocular cameras are highly versatile and applicable to standard videos and CCTV footage. These methods have evolved from two-dimensional (2D) to three-dimensional (3D) pose estimation. However, in real-world environments, current 3D HPE methods trained on laboratory-based motion capture data encounter challenges, such as limited training data, depth ambiguity, left/right switching, and issues with occlusions. In this study, four 3D HPE methods were compared based on their strengths and weaknesses using real-world videos. Joint position correction techniques were proposed to eliminate and correct anomalies such as left/right inversion and false detections of joint positions in daily life motions. Joint angle trajectories were obtained for intuitive and informative human activity recognition using an optimization method based on a 3D humanoid simulator, with the joint position corrected by the proposed technique as the input. The efficacy of the proposed method was verified by applying it to three types of freehand gymnastic exercises and comparing the joint angle trajectories during motion. [ABSTRACT FROM AUTHOR]

Video action recognition based on skeleton nodes is a highlighted issue in the computer vision field. In real application scenarios, the large number of skeleton nodes and behavior occlusion problems between individuals seriously affect recognition speed and accuracy. Therefore, we proposed a lightweight multi-stream feature cross-fusion (L-MSFCF) model to recognize abnormal behaviors such as fighting, vicious kicking, climbing over the wall, et al., which could obviously improve recognition speed based on lightweight skeleton node calculation, and improve recognition accuracy based on occluded skeleton node prediction analysis in order to effectively solve the behavior occlusion problem. The experiments show that our proposed All-MSFCF model has a video action recognition average accuracy rate of 92.7% for eight kinds of abnormal behavior recognition. Although our proposed lightweight L-MSFCF model has an 87.3% average accuracy rate, its average recognition speed is 62.7% higher than the full-skeleton recognition model, which is more suitable for solving real-time tracing problems. Moreover, our proposed Trajectory Prediction Tracking (TPT) model could real-time predict the moving positions based on the dynamically selected core skeleton node calculation, especially for the short-term prediction within 15 frames and 30 frames that have lower average loss errors. [ABSTRACT FROM AUTHOR]

Falling is a significant threat for old people and disabled persons. Therefore, it is necessary to develop an automated fall detection system to make them more independent and to mitigate the physical and psychological consequences of falls. Such a system can lower the death rate among dependent people who live alone. This paper presents a review of vision-based techniques, algorithms and databases used in detecting falls. Recently proposed systems are analyzed and compared. The literature review underscores the promising potential of camera-based fall detection systems and highlights areas for further research and development. [ABSTRACT FROM AUTHOR]

Human action recognition is a challenging task in the field of computer vision, where deep learning-based methods have made significant progress. Existing methods often use uniform or random sampling, which results in behaviorally irrelevant frames that contain redundant irrelevant features, which in turn leads to misclassification and high computational cost. To solve the problem of misclassification caused by unsuitable sampling methods and to reduce the computational cost, we propose a novel framework named Multi-Stream network based on Key Frame Sampling for human action recognition (MS-KFS). Specifically, we first introduce self-attention to associate deep information between different regions. On this basis, a key frame sampling module will be trained based on rewards and pseudo-labels to extract the moments where key actions are performed. Finally, the novel difference feature as well as the appearance and motion feature is developed to enhance feature characteristic in terms of both depth and timing, and thus, the features will be put into classifier and accomplish the task. A series experiment results validate that the MS-KFS outperforms the state-of-art methods. [ABSTRACT FROM AUTHOR]

Inference using skeleton to steer RGB videos is applicable to fine‐grained activities in indoor human action recognition (IHAR). However, existing methods that explore only spatial alignment are prone to bias, resulting in limited performance. The authors propose a Three‐stage Guidance (3sG) framework, leveraging skeleton knowledge to promote RGB in three stages. First, a soft shading image is proposed for alleviating background noise in videos, allowing the network to directly focus more on the motion region. Second, the authors propose to extract RGB frames of interest to reduce the computational effort. Furthermore, to explore more fully the complementary information between skeletons and RGB, the skeleton is coupled to the frame representation in a different spatial–temporal sharing pattern. Third, the global skeleton and skeleton‐guided RGB features are fed into the shared classifiers, which approximate the logit distributions of the two to enhance the performance in RGB unimodal. Finally, a fusion strategy that utilizes two learnable parameters to adaptively integrate the skeleton with the RGB is proposed. 3sG outperforms the state‐of‐the‐art results on the Toyota Smarthome dataset while it is more efficient than similar methods on the NTU RGB+D dataset. [ABSTRACT FROM AUTHOR]

The task of human action recognition (HAR) can be found in many computer vision practical applications. Various data modalities have been considered for solving this task, including joint-based skeletal representations which are suitable for real-time applications on platforms with limited computational resources. We propose a spatio-temporal neural network that uses handcrafted geometric features to classify human actions from video data. The proposed deep neural network architecture combines graph convolutional and temporal convolutional layers. The experiments performed on public HAR datasets show that our model obtains results similar to other state-of-the-art methods but has a lower inference time while offering the possibility to obtain an explanation for the classified action. [ABSTRACT FROM AUTHOR]

Recognition and classification for construction activities help to monitor and manage construction workers. Deep learning and computer vision technologies have addressed many limitations of traditional manual methods in complex construction environments. However, distinguishing different workers and establishing a clear recognition logic remain challenging. To address these issues, we propose a novel construction activity recognition method that integrates multiple deep learning algorithms. To complete this research, we created three datasets: 727 images for construction entities, 2546 for posture and orientation estimation, and 5455 for worker re-identification. First, a YOLO v5-based model is trained for worker posture and orientation detection. A person re-identification algorithm is then introduced to distinguish workers by tracking their coordinates, body and head orientations, and postures over time, then estimating their attention direction. Additionally, a YOLO v5-based object detection model is developed to identify ten common construction entity objects. The worker's activity is determined by combining their attentional orientation, positional information, and interaction with detected construction entities. Ten video clips are selected for testing, and a total of 745 instances of workers are detected, achieving an accuracy rate of 88.5%. With further refinement, this method shows promise for a broader application in construction activity recognition, enhancing site management efficiency. [ABSTRACT FROM AUTHOR]

This document is an editorial for a special issue of the journal Micromachines on the methodology, microfabrication, and applications of advanced sensing and smart systems. The editorial highlights the increasing integration of sensing technology with other cutting-edge technologies, such as machine learning, wearable sensors, and the Internet of Things. The special issue includes articles on topics such as intelligent manufacturing methodologies, fluorescence microscopy image inpainting, MEMS micromirrors, wearable robotics, quantitative analysis of particle size, and medical monitoring systems. The authors express gratitude to the contributors and reviewers and hope that the articles will inspire new innovations in advanced sensing and smart systems. [Extracted from the article]

Human Action Recognition is an essential research area in computer vision due to its automated nature of video monitoring. Human Action Recognition has several applications, including robotics, video monitoring, health care, elderly monitoring, crowd behavior and the detection of aberrant activity. This study seeks to offer the reader an up-to-date overview of intelligent human activity recognition literature and current advancements in this area. This work discusses the recent state-of-the-art research for activity recognition techniques and challenges associated with identifying human activity and discusses publicly available datasets. This work consists of an in-depth survey of numerous works published from 2010 to 2022 focusing on intelligent techniques. This article describes all steps of human action recognition along with their techniques. This study comes with the Datasets for Human Action Recognition, Handcrafted-Feature technique, Machine Learning (ML), Deep-Learning (DL), Hybrid Deep Learning and limitation of this area. This study offers a comparative analysis between ML and DL approaches to show their effectiveness in action recognition. This study examines some unexplored areas in human action recognition that can be unearthed to create a more resilient system in the presence of issues. Previous research has demonstrated that deep learning surpasses standard machine learning for recognizing human activities. This study also emphasizes the most pressing issues and research direction. All relevant datasets are described in detail. Furthermore, our opinions and suggestions for future research have been shared. Compared to past surveys, this study offers a more systematic description of Human Action Recognition methods regarding comparability, problems, and the most recent evaluation technique. [ABSTRACT FROM AUTHOR]

Graph theory originated as a fun way to solve math problems, but it has now evolved into a significant mathematics subject with several applications in computer vision. The human skeleton has garnered much attention in recent years as a simple depiction of human motion. Three-dimensional joints have limitations in terms of dependability, compatibility, and flexibility, notwithstanding the optimistic results of previous investigations. CPG-3D is a new technique for skeleton-based action recognition that focuses solely on 3D skeleton joint points and their placements as the basic expression of human skeletons rather than any additional information. CPG-3D outperforms previous action recognition algorithms in learning how to describe, is more resilient to pose estimation disturbances, and generalises better across data sets. CPG-3D can also manage various alternatives without incurring additional computational expenses, and its capabilities can be readily fused with other modalities during the early phases of fusion, offering a broad design space to increase efficiency further. CPG-3D consistently beats the competition on four complicated and hard datasets when used alone on three-dimensional skeletal joints. [ABSTRACT FROM AUTHOR]

Shadow, a natural phenomenon resulting from the absence of light, plays a pivotal role in agriculture, particularly in processes such as photosynthesis in plants. Despite the availability of generic shadow datasets, many suffer from annotation errors and lack detailed representations of agricultural shadows with possible human activity inside, excluding those derived from satellite or drone views. In this paper, we present an evaluation of a synthetically generated top-down shadow segmentation dataset characterized by photorealistic rendering and accurate shadow masks. We aim to determine its efficacy compared to real-world datasets and assess how factors such as annotation quality and image domain influence neural network model training. To establish a baseline, we trained numerous baseline architectures and subsequently explored transfer learning using various freely available shadow datasets. We further evaluated the out-of-domain performance compared to the training set of other shadow datasets. Our findings suggest that AgroSegNet demonstrates competitive performance and is effective for transfer learning, particularly in domains similar to agriculture. [ABSTRACT FROM AUTHOR]

This article introduces the concept of tinyML, which focuses on implementing machine learning on low-power devices such as microcontrollers and custom-designed circuits. The goal is to enable these devices to perform machine learning tasks while consuming minimal power and lasting for extended periods on a single battery charge. The special issue covers a range of topics related to tinyML, including energy efficiency in computer vision, automatic design and code generation, regression-based profiling, and the application of binary neural networks. The article discusses various techniques and approaches in the field, such as accelerating neural networks on microcontrollers and software frameworks for machine learning libraries. It also explores specific applications, such as real-time video anonymization and predicting gas concentration with low-cost sensors. Overall, the article provides insights into the advancements and challenges in tinyML and offers potential solutions for resource-constrained devices. [Extracted from the article]

Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works. [ABSTRACT FROM AUTHOR]

Multimodal Human Action Recognition (MHAR) is an important research topic in computer vision and event recognition fields. In this work, we address the problem of MHAR by developing a novel audio-image and video fusion-based deep learning framework that we call Multimodal Audio-Image and Video Action Recognizer (MAiVAR). We extract temporal information using image representations of audio signals and spatial information from video modality with the help of Convolutional Neutral Networks (CNN)-based feature extractors and fuse these features to recognize respective action classes. We apply a high-level weights assignment algorithm for improving audio-visual interaction and convergence. This proposed fusion-based framework utilizes the influence of audio and video feature maps and uses them to classify an action. Compared with state-of-the-art audio-visual MHAR techniques, the proposed approach features a simpler yet more accurate and more generalizable architecture, one that performs better with different audio-image representations. The system achieves an accuracy 87.9% and 79.0% on UCF51 and Kinetics Sounds datasets, respectively. All code and models for this paper will be available at https://tinyurl.com/4ps2ux6n. [ABSTRACT FROM AUTHOR]

A comprehensive examination of human action recognition (HAR) methodologies situated at the convergence of deep learning and computer vision is the subject of this article. We examine the progression from handcrafted feature-based approaches to end-to-end learning, with a particular focus on the significance of large-scale datasets. By classifying research paradigms, such as temporal modelling and spatial features, our proposed taxonomy illuminates the merits and drawbacks of each. We specifically present HARNet, an architecture for Multi-Model Deep Learning that integrates recurrent and convolutional neural networks while utilizing attention mechanisms to improve accuracy and robustness. The VideoMAE v2 method (https://github.com/OpenGVLab/VideoMAEv2) has been utilized as a case study to illustrate practical implementations and obstacles. For researchers and practitioners interested in gaining a comprehensive understanding of the most recent advancements in HAR as they relate to computer vision and deep learning, this survey is an invaluable resource. [ABSTRACT FROM AUTHOR]

Aerobics is a form of exercise widely used in health and beauty. Correct posture is the key factor to the effectiveness of aerobics. However, the existing aerobics teaching is often limited by manual judgment and guidance, so a more effective, accurate and real-time posture recognition method is needed. Therefore, this study aims to develop a virtual image technology based on wearable light sensing devices for real-time and accurate aerobics pose recognition. By combining optical principle and image processing technology, we hope to provide an aerobics posture recognition scheme that does not rely on manual judgment. A wearable optical sensor is designed to capture the dynamic posture of aerobics athletes. The device will be able to obtain real-time key point information for athletes. Then, by applying optical principle and image processing technology, the captured node information is converted into virtual image. Finally, computer vision algorithm is used to analyze and compare virtual images to realize posture recognition. After testing and analysis, the system can realize high precision aerobics posture recognition. Compared with the traditional manual judgment, the proposed method has higher accuracy and real-time performance, and can automatically provide feedback and guidance to help athletes improve their posture. [ABSTRACT FROM AUTHOR]

As one of the key research directions in the field of computer vision, human action recognition has a wide range of practical application values and prospects. In the fields of video surveillance, human-computer interaction, sports analysis, and healthcare, human action recognition technology shows a broad application prospect and potential. However, the diversity and complexity of human actions bring many challenges, such as handling complex actions, distinguishing similar actions, coping with changes in viewing angle, and overcoming occlusion problems. To address the challenges, this paper proposes an innovative framework for human action recognition. The framework combines the latest pose estimation algorithms, pre-trained CNN models, and a Vision Transformer to build an efficient system. The first step involves utilizing the latest pose estimation algorithm to accurately extract human pose information from real RGB image frames. Then, a pre-trained CNN model is used to perform feature extraction on the extracted pose information. Finally, the Vision Transformer model is applied for fusion and classification operations on the extracted features. Experimental validation is conducted on two benchmark datasets, UCF 50 and UCF 101, to demonstrate the effectiveness and efficiency of the proposed framework. The applicability and limitations of the framework in different scenarios are further explored through quantitative and qualitative experiments, providing valuable insights and inspiration for future research. [ABSTRACT FROM AUTHOR]

Human behavior recognition has become a popular research topic in the field of computer vision. With the introduction of deep learning and attention mechanisms, this field has been further promoted. However, issues such as dataset acquisition and preprocessing operations on multimodal datasets, modeling of long time information in videos, and fusion of temporal and spatial information still exist. In this paper, we first outline some video action recognition datasets and related preprocessing techniques, including frame extraction, optical flow extraction, and skeletal feature acquisition. Then, the relevant models are classified and parsed according to their characteristics and the types of input data modalities. In addition, we evaluate the performance of the models on several benchmark datasets to gain a deeper understanding of the model development process. Finally, we summarized the current challenges faced in the field of video behavior recognition, including model timeliness, data set subjectivity and effective fusion of multi-modal features, and proposed possible future improvement directions in order to provide more ideas and methods for subsequent research. [ABSTRACT FROM AUTHOR]

Nowadays, for controlling crime, surveillance cameras are typically installed in all public places to ensure urban safety and security. However, automating Human Activity Recognition (HAR) using computer vision techniques faces several challenges such as lowlighting, complex spatiotemporal features, clutter backgrounds, and inefficient utilization of surveillance system resources. Existing attempts in HAR designed straightforward networks by analyzing either spatial or motion patterns resulting in limited performance while the dual streams methods are entirely based on Convolutional Neural Networks (CNN) that are inadequate to learning the long-range temporal information for HAR. To overcome the above-mentioned challenges, this paper proposes an optimized dual stream framework for HAR which mainly consists of three steps. First, a shots segmentation module is introduced in the proposed framework to efficiently utilize the surveillance system resources by enhancing the lowlight video stream and then it detects salient video frames that consist of human. This module is trained on our own challenging Lowlight Human Surveillance Dataset (LHSD) which consists of both normal and different levels of lowlighting data to recognize humans in complex uncertain environments. Next, to learn HAR from both contextual and motion information, a dual stream approach is used in the feature extraction. In the first stream, it freezes the learned weights of the backbone Vision Transformer (ViT) B-16 model to select the discriminative contextual information. In the second stream, ViT features are then fused with the intermediate encoder layers of FlowNet2 model for optical flow to extract a robust motion feature vector. Finally, a two stream Parallel Bidirectional Long Short-Term Memory (PBiLSTM) is proposed for sequence learning to capture the global semantics of activities, followed by Dual Stream Multi-Head Attention (DSMHA) with a late fusion strategy to optimize the huge features vector for accurate HAR. To assess the strength of the proposed framework, extensive empirical results are conducted on real-world surveillance scenarios and various benchmark HAR datasets that achieve 78.6285%, 96.0151%, and 98.875% accuracies on HMDB51, UCF101, and YouTube Action, respectively. Our results show that the proposed strategy outperforms State-of-the-Art (SOTA) methods. The proposed framework gives superior performance in HAR, providing accurate and reliable recognition of human activities in surveillance systems. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) is a focal point of study in the realms of human perception and computer vision due to its widespread applicability in various contexts, such as intelligent video surveillance, ambient assisted living, HCI, HRI, IR, entertainment, and intelligent driving. With the prevalence of deep learning techniques for image classification, researchers have shifted away from the labor-intensive practice of hand-crafting in favor of these methods in HAR. However, Convolutional Neural Networks (CNNs) face challenges such as the receptive field problem and limited sample issues that remain unsolved. This paper introduces a two-branch convolutional neural network for HAR classification, incorporating a polarized full attention method to address the aforementioned issues. The Artificial Butterfly Optimization (ABO) is employed for optimal hyper-parameter tuning. The proposed network utilizes twobranch CNNs to efficiently extract data, simplifying convolutional layers' kernel sizes to enhance network training and suitability for low-data settings. Feature extraction effectiveness is improved by implementing the one-shot assembly method. To amalgamate feature maps and provide global context, an enhanced full attention block called polarized full attention is utilized. Experimental results demonstrate the superiority of the proposed model in detecting human behaviors on the LoDVP Abnormal Behaviors dataset and the UCF50 dataset. Furthermore, the suggested model is adaptable to incorporate new sensor data, making it particularly valuable for real-time human activity identification applications. The Recall is 100 for the 1st dataset, 94 for the 2nd dataset, and 100 for the 3rd dataset, respectively. The F1-Score is 96.61836 for the 1st dataset, 96.90722 for the 2nd dataset, and 98.03922 for the 3rd dataset, respectively. [ABSTRACT FROM AUTHOR]

Current remote sensing technologies employing Unmanned Aerial Vehicles (UAVs) for farm monitoring have shown promise in characterizing the environment through diverse sensor systems, including hyperspectral cameras, LiDAR, thermal cameras, and RGB sensors. However, these solutions often specialize in either activity recognition or crop monitoring, but not both. To address this limitation and enhance efficacy, we propose a multi-vision monitoring (MVM) framework capable of simultaneously recognizing farm activities and assessing crop health. Our approach involves computer vision techniques that transform aerial videos into sequential images to extract essential environmental features. Central to our framework are two pivotal components: the Farmer Activity Recognition (FAR) algorithm and the Crop Image Analysis (CIA). The FAR algorithm introduces a novel feature extraction method capturing motion across various maps, enabling distinct feature sets for each activity. Meanwhile, the CIA component utilizes the normalized Triangular Greenness Index (nTGI) to estimate leave chlorophyll levels, an important indicator for crop health. By unifying these components, we achieve dual functionality—activity recognition and crop health estimation—using identical input data, thereby enhancing efficiency and versatility in farm monitoring. Our framework employs a diverse range of machine learning models, demonstrating the potential of our extracted features to address the defined problem effectively in unison.

Biometrics is a field that has been given importance in recent years and has been extensively studied. Biometrics can use physical and behavioural differences that are unique to individuals to recognize and identify them. Today, biometric information is used in many areas such as computer vision systems, entrance systems, security and recognition. In this study, a new biometrics database containing silhouette, thermal face and skeletal data based on the distance between the joints was created to be used in behavioural and physical biometrics studies. The fact that many cameras were used in previous studies increases both the processing intensity and the material cost. This study aimed to both increase the recognition performance and reduce material costs by adding thermal face data in addition to soft and behavioural biometrics with the optimum camera. The presented data set was created in accordance with both motion recognition and person identification. Various data loss scenarios and multi-biometrics approaches based on data fusion have been tried on the created data sets and the results have been given comparatively. In addition, the correlation coefficient of the motion frames method to obtain energy images from silhouette data was tested on this dataset and yielded high-accuracy results for both motion and person recognition. [ABSTRACT FROM AUTHOR]

Machine vision, an interdisciplinary field that aims to replicate human visual perception in computers, has experienced rapid progress and significant contributions. This paper traces the origins of machine vision, from early image processing algorithms to its convergence with computer science, mathematics, and robotics, resulting in a distinct branch of artificial intelligence. The integration of machine learning techniques, particularly deep learning, has driven its growth and adoption in everyday devices. This study focuses on the objectives of computer vision systems: replicating human visual capabilities including recognition, comprehension, and interpretation. Notably, image classification, object detection, and image segmentation are crucial tasks requiring robust mathematical foundations. Despite the advancements, challenges persist, such as clarifying terminology related to artificial intelligence, machine learning, and deep learning. Precise definitions and interpretations are vital for establishing a solid research foundation. The evolution of machine vision reflects an ambitious journey to emulate human visual perception. Interdisciplinary collaboration and the integration of deep learning techniques have propelled remarkable advancements in emulating human behavior and perception. Through this research, the field of machine vision continues to shape the future of computer systems and artificial intelligence applications. [ABSTRACT FROM AUTHOR]

Audio-image representations for a multimodal human action (MHAiR) dataset contains six different image representations of the audio signals that capture the temporal dynamics of the actions in a very compact and informative way. The dataset was extracted from the audio recordings which were captured from an existing video dataset, i.e., UCF101. Each data sample captured a duration of approximately 10 s long, and the overall dataset was split into 4893 training samples and 1944 testing samples. The resulting feature sequences were then converted into images, which can be used for human action recognition and other related tasks. These images can be used as a benchmark dataset for evaluating the performance of machine learning models for human action recognition and related tasks. These audio-image representations could be suitable for a wide range of applications, such as surveillance, healthcare monitoring, and robotics. The dataset can also be used for transfer learning, where pre-trained models can be fine-tuned on a specific task using specific audio images. Thus, this dataset can facilitate the development of new techniques and approaches for improving the accuracy of human action-related tasks and also serve as a standard benchmark for testing the performance of different machine learning models and algorithms. [ABSTRACT FROM AUTHOR]

Forests are crucial resources for human existence and social progress because they safeguard the earth's natural balance. However, forest fires occur regularly as a result of unmanaged human activities and erratic environmental conditions. These fires are the most destructive natural disasters to forest resources and the human environment. Forest fires have become more common in this scenario as a result of climate change, human activity, and other things. Forest fire detection and monitoring has become a global problem for forest fire prevention organisations. Currently, forest fire detection methods mostly include vigils, observation from watch towers, and, more recently, satellite monitoring. Although observation from watch towers is simple and feasible, there are various obstacles. First and foremost, this strategy requires Keywords — Wireless sensor networks(WSN), Computer vision, aerial vehicles(UAV), YOLO, Internet of things(IOT), Moderate Resolution Imaging Spectroradiometer(MODIS), mAP (mean Average Precision) Many financial and material resources, as well as a skilled labour force. Second, there are numerous issues with fire protection manpower, such as inattention, absenteeism from the post, a lack of real-time monitoring capability, and limited area coverage. A variety of parameters limit the spectrum of use of satellite detection systems, reducing their efficiency in forest fire detection. A satellite monitoring system, for example, has a long scanning period and a low resolution of saturated pixel dots of photographs. Another issue is that cloud layers might obscure photos during the scanning process, and real-time quantitative quantification of fire parameters is extremely difficult to perform. Given the limitations of traditional monitoring, we propose a cloud-based wireless sensor network technology and Explain how it can be used as a monitoring system. This system can monitor real-time related parameters such as temperature and relative humidity and promptly transfer the data to the monitoring center's computer. The computer will assess and manage the collected data. In comparison to standard baroscopic data and fundamental forest resource data, the system can conduct an immediate assessment of a potential fire threat. The analytical results will then be provided to the relevant department as the policy-making foundation on which the department will decide whether to battle fires or prevent fires. [ABSTRACT FROM AUTHOR]

This paper develops a system for determining the psycho-emotional state of the observed people based on the analysis of video surveillance with the application of artificial intelligence technology using hardware and software tools such as PoseNet, PyTorch, SQLite, FastAPI and Flask. In many areas of human endeavor, there is an urgent need for a surveillance system that can reliably function and detect suspicious activities. To solve this problem, this paper proposes a novel framework for a real-time surveillance system that automatically detects abnormal human activities. The system has been tested and validated in real environments. The results of testing artificial intelligence program models showed the best results (f1 score with values of 0.98–0.99). The weighted average value of the f1-score metric was 0.96, which is quite a high value. The use of PoseNet implemented with PyTorch allowed to accurately determine the pose of the person in the video and extract information about the position of different body parts. The peculiarity of this work lies in the development of artificial intelligence models for automatic detection of possible physical aggression in videos, in the methods of forming an optimal set of features for the development of AI models that identify the aggressor and the victim of bullying. The developed system has the potential to be a useful tool in various fields such as psychology, medicine, security and others where it is important to analyze the emotional state of people based on their physical manifestations. The obtained applied results can be used in educational institutions and in spheres where video analysis is necessary. [ABSTRACT FROM AUTHOR]

At present times, indoor surveillance becomes a hot research topic among researchers and business sectors. Human detection is one of the vital areas of focus in the surveillance system owing to its significance in proper person detection, human activity identification, and scene classification. Since the indoor spaces comprise poor lighting, variable illuminations, shadowing, and complex background, the human detection process becomes a tedious task. The advent of computer vision and deep learning (DL) models is commonly employed for human detection. This article presents a new intelligent deep learning model for human detection in indoor surveillance videos (IDL-HDIS). As data augmentation process is one of the most renowned ways to increase the size of the dataset which is highly essential for enhancing the prediction accuracy of the model, the same is carried out as a part of even this research work which includes performing rotation, translation and flipping. The IDL-GDIS model uses Faster Region Convolutional Neural Network (Faster R-CNN) model for human detection. The Faster R-CNN comprises of Fast R-CNN and Region Proposal Network (RPN). The RPN uses Capsule Networks (CapsNet) model as a shared convolution neural network (CNN), which acts as a feature extractor and generates the feature map. Besides, dropout is employed to avoid overfitting problem in the CapsNet architecture, the validation of IDL-HDIS model is done by a comprehensive simulation analysis under different aspects. The validation is supported by the evident results of the IDL-HDIS model which is given in the paper. [ABSTRACT FROM AUTHOR]

Human Action Recognition (HAR) has grown to be the toughest and attractive concern in the domains of computer vision, communication between a person and the surroundings, and video surveillance. In variation to the conventional methods that usually make use of the Long Short Term Memory model (LSTM) for training, this work designed dropout variant Drop-ConvLSTM2D, to provide more effectiveness in regularization for deep Convolution Neural Networks (CNNs). In addition, to speed up the runtime performance of the Deep Learning model, Bayesian Hyper Parameter Optimization (BHPO) is also introduced to autonomously optimize, the hyper parameters of the trained architecture. In this study, a two-stream Bayesian Hyper Parameter optimized Drop-ConvLSTM2D model is designed for HAR to overcome the current research deficiencies. In one stream, an Inception-v3 model extracts the temporal characteristics from the optical frames which are generated through the dense flow process. In another stream, a 3D-CNN involves the mining of the spatial-temporal characteristics from the RGB frames. Finally, the features of Inception-v3 and 3D-CNN are fused using which the Drop-ConvLSTM2D model is trained to recognize human behavior. On perceptive public video datasets UCF-101, HMDB51, the quantitative assessments are conducted on the Drop-ConvLSTM2D BHPO model. For all hyper parameters, the built model explicitly obtains optimized values in this process, which can save time and improve performance. The experimental outcome shows that with a precision of at least 3%, the designed model beats the traditional two-stream model. [ABSTRACT FROM AUTHOR]

Hand pose recognition presents significant challenges that need to be addressed, such as varying lighting conditions or complex backgrounds, which can hinder accurate and robust hand pose estimation. This can be mitigated by employing MediaPipe to facilitate the efficient extraction of representative landmarks from static images combined with the use of Convolutional Neural Networks. Extracting these landmarks from the hands mitigates the impact of lighting variability or the presence of complex backgrounds. However, the variability of the location and size of the hand is still not addressed by this process. Therefore, the use of processing modules to normalize these points regarding the location of the wrist and the zoom of the hands can significantly mitigate the effects of these variabilities. In all the experiments performed in this work based on American Sign Language alphabet datasets of 870, 27,000, and 87,000 images, the application of the proposed normalizations has resulted in significant improvements in the model performance in a resource-limited scenario. Particularly, under conditions of high variability, applying both normalizations resulted in a performance increment of 45.08%, increasing the accuracy from 43.94 ± 0.64% to 89.02 ± 0.40%. [ABSTRACT FROM AUTHOR]

Beyond traditional surveillance applications, sensor-based human action recognition and segmentation responds to a growing demand in the health and safety sector. Recently, skeletal action recognition has largely been dominated by spatio-temporal graph convolutional neural networks (ST-GCN), while video-based action segmentation research successfully employs 3D convolutional neural networks (3D-CNNs), as well as vision transformers. In this paper, we argue that these two inputs are complementary, and we develop an approach that achieves superior performance with a multi-modal ensemble. A multi-task GCN is developed that can predict both frame-wise actions as well as action timestamps, allowing for the use of fine-tuned video classification models to classify action segments and achieve refined predictions. Symmetrically, a multi-task video approach is presented that uses a video action segmentation model to predict framewise labels and timestamps, augmented with a skeletal action classification model. Finally, an ensemble of segmentation methods for each modality (skeletal, RGB, depth, and infrared) is formulated. Experimental results yield 87% accuracy on the PKU-MMD v2 dataset, delivering state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Human action recognition based on still images is one of the most challenging computer vision tasks. In the past decade, convolutional neural networks (CNNs) have developed rapidly and achieved good performance in human action recognition tasks based on still images. Due to the absence of the remote perception ability of CNNs, it is challenging to have a global structural understanding of human behavior and the overall relationship between the behavior and the environment. Recently, transformer-based models have been making a splash in computer vision, even reaching SOTA in several vision tasks. We explore the transformer's capability in human action recognition based on still images and add a simple but effective feature fusion module based on the Swin-Transformer model. More specifically, we propose a new transformer-based model for behavioral feature extraction that uses a pre-trained Swin-Transformer as the backbone network. Swin-Transformer's distinctive hierarchical structure, combined with the feature fusion module, is used to extract and fuse multi-scale behavioral information. Extensive experiments were conducted on five still image-based human action recognition datasets, including the Li's action dataset, the Stanford-40 dataset, the PPMI-24 dataset, the AUC-V1 dataset, and the AUC-V2 dataset. Results indicate that our proposed Swin-Fusion model achieves better behavior recognition than previously improved CNN-based models by sharing and reusing feature maps of different scales at multiple stages, without modifying the original backbone training method and with only increasing training resources by 1.6%. The code and models will be available at https://github.com/cts4444/Swin-Fusion. [ABSTRACT FROM AUTHOR]

Human action recognition (HAR) as the most representative human-centred computer vision task is critical in human resource management (HRM), especially in human resource recruitment, performance appraisal, and employee training. Currently, prevailing approaches to human action recognition primarily emphasize either temporal or spatial features while overlooking the intricate interplay between these two dimensions. This oversight leads to less precise and robust action classification within complex human resource recruitment environments. In this paper, we propose a novel human action recognition methodology for human resource recruitment environments, which aims at symmetrically harnessing temporal and spatial information to enhance the performance of human action recognition. Specifically, we compute Depth Motion Maps (DMM) and Depth Temporal Maps (DTM) from depth video sequences as space and time descriptors, respectively. Subsequently, a novel feature fusion technique named Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to enhance the fusion of space and time features by collaboratively learning the center and boundary information of feature class space. We then introduce a spatio-temporal information filtration module to remove redundant information introduced by spatio-temporal fusion and retain discriminative details. Finally, a Support Vector Machine (SVM) is employed for human action recognition. Extensive experiments demonstrate that the proposed method has the ability to significantly improve human action recognition performance. [ABSTRACT FROM AUTHOR]

As the research community focuses on improving the reliability of deep learning, identifying out-of-distribution (OOD) data has become crucial. Detecting OOD inputs during test/prediction allows the model to account for discriminative features unknown to the model. This capability increases the model's reliability since this model provides a class prediction solely at incoming data similar to the training one. Although OOD detection is well-established in computer vision, it is relatively unexplored in other areas, like time series-based human activity recognition (HAR). Since uncertainty has been a critical driver for OOD in vision-based models, the same component has proven effective in time-series applications. In this work, we propose an ensemble-based temporal learning framework to address the OOD detection problem in HAR with time-series data. First, we define different types of OOD for HAR that arise from realistic scenarios. Then we apply our ensemble-based temporal learning framework incorporating uncertainty to detect OODs for the defined HAR workloads. This particular formulation also allows a novel approach to fall detection. We train our model on non-fall activities and detect falls as OOD. Our method shows state-of-the-art performance in a fall detection task using much lesser data. Furthermore, the ensemble framework outperformed the traditional deep-learning method (our baseline) on the OOD detection task across all the other chosen datasets. [ABSTRACT FROM AUTHOR]

Human action recognition is a computer vision task that identifies how a person or a group acts on a video sequence. Various methods that rely on deep-learning techniques, such as two- or three-dimensional convolutional neural networks (2D-CNNs, 3D-CNNs), recurrent neural networks (RNNs), and vision transformers (ViT), have been proposed to address this problem over the years. Motivated by the fact that most of the used CNNs in human action recognition present high complexity, and the necessity of implementations on mobile platforms that are characterized by restricted computational resources, in this article, we conduct an extensive evaluation protocol over the performance metrics of five lightweight architectures. In particular, we examine how these mobile-oriented CNNs (viz., ShuffleNet-v2, EfficientNet-b0, MobileNet-v3, and GhostNet) execute in spatial analysis compared to a recent tiny ViT, namely EVA-02-Ti, and a higher computational model, ResNet-50. Our models, previously trained on ImageNet and BU101, are measured for their classification accuracy on HMDB51, UCF101, and six classes of the NTU dataset. The average and max scores, as well as the voting approaches, are generated through three and fifteen RGB frames of each video, while two different rates for the dropout layers were assessed during the training. Last, a temporal analysis via multiple types of RNNs that employ features extracted by the trained networks is examined. Our results reveal that EfficientNet-b0 and EVA-02-Ti surpass the other mobile-CNNs, achieving comparable or superior performance to ResNet-50. [ABSTRACT FROM AUTHOR]

Human-human interaction recognition is crucial in computer vision fields like surveillance, human-computer interaction, and social robotics. It enhances systems' ability to interpret and respond to human behavior precisely. This research focuses on recognizing human interaction behaviors using a static image, which is challenging due to the complexity of diverse actions. The overall purpose of this study is to develop a robust and accurate system for human interaction recognition. This research presents a novel image-based human interaction recognition method using a Hidden Markov Model (HMM). The technique employs hue, saturation, and intensity (HSI) color transformation to enhance colors in video frames, making them more vibrant and visually appealing, especially in low-contrast or washed-out scenes. Gaussian filters reduce noise and smooth imperfections followed by silhouette extraction using a statistical method. Feature extraction uses the features from Accelerated Segment Test (FAST), Oriented FAST, and Rotated BRIEF (ORB) techniques. The application of Quadratic Discriminant Analysis (QDA) for feature fusion and discrimination enables high-dimensional data to be effectively analyzed, thus further enhancing the classification process. It ensures that the final features loaded into the HMM classifier accurately represent the relevant human activities. The impressive accuracy rates of 93% and 94.6% achieved in the BIT-Interaction and UT-Interaction datasets respectively, highlight the success and reliability of the proposed technique. The proposed approach addresses challenges in various domains by focusing on frame improvement, silhouette and feature extraction, feature fusion, and HMM classification. This enhances data quality, accuracy, adaptability, reliability, and reduction of errors. [ABSTRACT FROM AUTHOR]