Search

The lack of a standardized sign language, and the inability to communicate with the hearing community through sign language, are the two major issues confronting Pakistan's deaf and dumb society. In this research, we have proposed an approach to help eradicate one of the issues. Now, using the proposed framework, the deaf community can communicate with normal people. The purpose of this work is to reduce the struggles of hearingimpaired people in Pakistan. A Kinect-based Pakistan sign language (PSL) to Urdu language translator is being developed to accomplish this. The system’s dynamic sign language segment works in three phases: acquiring key points from the dataset, training a long short-term memory (LSTM) model, and making real-time predictions using sequences through openCV integrated with the Kinect device. The system’s static sign language segment works in three phases: acquiring an image-based dataset, training a model garden, and making real-time predictions using openCV integrated with the Kinect device. It also allows the hearing user to input Urdu audio to the Kinect microphone. The proposed sign language translator can detect and predict the PSL performed in front of the Kinect device and produce translations in Urdu.

Background: Gait analysis is burdened by time and equipment costs, interpretation, and accessibility of three-dimensional motion analysis systems. Evidence suggests growing adoption of gait testing in the shift toward evidence-based medicine. Further developments addressing these barriers will aid its efficacy in clinical practice. Previous research aiming to develop gait analysis systems for kinetics estimation using the Kinect V2 have provided promising results yet modified approaches using the latest hardware may further aid kinetics estimation accuracy Research question: Can a single Azure Kinect sensor combined with a musculoskeletal modeling approach provide kinetics estimations during gait similar to those obtained from marker-based systems with embedded force platforms? Methods: Ten subjects were recruited to perform three walking trials at their normal speed. Trials were recorded using an eight-camera optoelectronic system with two embedded force plates and a single Azure Kinect sensor. Marker and depth data were both used to drive a musculoskeletal model using the AnyBody Modeling System. Predicted kinetics from the Azure Kinect-driven model, including ground reaction force (GRF) and joint moments, were compared to measured values using root meansquared error (RMSE), normalized RMSE, Pearson correlation, concordance correlation, and statistical parametric mapping Results: High to very high correlations were observed for anteroposterior GRF (ρ = 0.889), vertical GRF (ρ = 0.940), and sagittal hip (ρ = 0.805) and ankle (ρ = 0.876) moments. RMSEs were 1.2 ± 2.2 (%BW), 3.2 ± 5.7 (%BW), 0.7 ± 0.1.3 (%BWH), and 0.6 ± 1.0 (%BWH) Significance: The proposed approach using the Azure Kinect provided higher accuracy compared to previous studies using the Kinect V2 potentially due to improved foot tracking by the Azure Kinect. Future studies should seek to optimize ground contact parameters and focus on regions of error between predicted and measured kinetics highlighted currently for further improvements in kinetic estimations.

There are various methods of rehabilitation to restore disabled gait, one of which is gait analysis. However, some gait analysis systems have shown drawbacks for clinical use, such as system complexity and the high cost of gait equipment. A Kinect sensor could be used as an alternative for gait analysis and is studied in this research. The accuracy of a Kinect in calculation of kinematic gait parameters was computed during normal gait. The gait analysis was processed with Matlab and data was acquired by recording multiple subjects’ gait with different Kinect position. Subjects’ the right knee was calculated and the results shown the best position of Kinect to capture gait is 45º towards motion path with errors of detecting gait parameters about 7%. Hence, Kinect sensor is capable of doing gait analysis for further pre-clinical applications.

Automatic translation from signed to spoken languages is an interdisciplinary research domain, lying on the intersection of computer vision, machine translation and linguistics. Nevertheless, research in this domain is performed mostly by computer scientists in isolation. As the domain is becoming increasingly popular - the majority of scientific papers on the topic of sign language translation have been published in the past three years - we provide an overview of the state of the art as well as some required background in the different related disciplines. We give a high-level introduction to sign language linguistics and machine translation to illustrate the requirements of automatic sign language translation. We present a systematic literature review to illustrate the state of the art in the domain and then, harking back to the requirements, lay out several challenges for future research. We find that significant advances have been made on the shoulders of spoken language machine translation research. However, current approaches are often not linguistically motivated or are not adapted to the different input modality of sign languages. We explore challenges related to the representation of sign language data, the collection of datasets, the need for interdisciplinary research and requirements for moving beyond research, towards applications. Based on our findings, we advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages. Furthermore, the inclusion of deaf and hearing end users of sign language translation applications in use case identification, data collection and evaluation is of the utmost importance in the creation of useful sign language translation models. We recommend iterative, human-in-the-loop, design and development of sign language translation models., Comment: This is the version of the article submitted to peer review to Universal Access in the Information Society. Please refer to "De Coster, M., Shterionov, D., Van Herreweghe, M. et al. Machine translation from signed to spoken languages: state of the art and challenges. Univ Access Inf Soc (2023)." for the published and updated version

The human-machine interaction has evolved significantly in the last years, allowing a new range of opportunities for developing solutions for people with physical limitations. Natural user interfaces (NUI) allow bedridden and/or physically disabled people to perform a set of actions trough gestures thus increasing their quality of life and autonomy. This paper presents a solution based on image processing and computer vision using the Kinect 3D sensor for development of applications that recognize gestures made by the human hand. The gestures are then identified by a software application that triggers a set of actions of upmost importance for the bedridden person, for example, trigger the emergency, switch on/off the TV or control the bed slope. It was used a shape matching technique for six gestures recognition, being the final actions activated by the Arduino platform. The results show a success rate of 96%. This system can improve the quality of life and autonomy of bedridden people, being able to be adapted for the specific necessities of an individual subject.

Face recognition is one of the applications in image processing that recognizes or checks an individual's identity. 2D images are used to identify the face, but the problem is that this kind of image is very sensitive to changes in lighting and various angles of view. The images captured by 3D camera and stereo camera can also be used for recognition, but fairly long processing times is needed. RGB-D images that Kinect produces are used as a new alternative approach to 3D images. Such cameras cost less and can be used in any situation and any environment. This paper shows the face recognition algorithms’ performance using RGB-D images. These algorithms calculate the descriptor which uses RGB and Depth map faces based on local binary pattern. Those images are also tested for the fusion of LBP and DCT methods. The fusion of LBP and DCT approach produces a recognition rate of 97.5% during the experiment.

Depth-based plethysmography (DPG) for the measurement of respiratory parameters is a mobile and cost-effective alternative to spirometry and body plethysmography. In addition, natural breathing can be measured without a mouthpiece, and breathing mechanics can be visualized. This paper aims at showing further improvements for DPG by analyzing recent developments regarding the individual components of a DPG measurement. Starting from the advantages and application scenarios, measurement scenarios and recording devices, selection algorithms and location of a region of interest (ROI) on the upper body, signal processing steps, models for error minimization with a reference measurement device, and final evaluation procedures are presented and discussed. It is shown that ROI selection has an impact on signal quality. Adaptive methods and dynamic referencing of body points to select the ROI can allow more accurate placement and thus lead to better signal quality. Multiple different ROIs can be used to assess breathing mechanics and distinguish patient groups. Signal acquisition can be performed quickly using arithmetic calculations and is not inferior to complex 3D reconstruction algorithms. It is shown that linear models provide a good approximation of the signal. However, further dependencies, such as personal characteristics, may lead to non-linear models in the future. Finally, it is pointed out to focus developments with respect to single-camera systems and to focus on independence from an individual calibration in the evaluation.

The elderly population experiences a decline in upper extremity range of motion (ROM), impairing activities of daily living. The primary mode of quantification is by goniometer measurement. In this cross-sectional observation study, we investigate a sensor-acquired reachable workspace for assessing shoulder ROM decline in an elderly population in comparison to traditional measurements. Sixty-one healthy subjects aged ≥ 65 years were included and compared to a cohort of 39 younger subjects, aged 20 to 64. A sensor acquired reachable workspace using a Kinect motion capture camera measured the maximum reaching ability of both arms while in a seated position, measured in m2 and normalized to arm length to calculate a novel score defined as a relative surface area. This score approximates range of motion in the upper extremity. This measurement was compared to goniometer measurements, including active ROM in shoulder flexion and abduction. Total RSA shows moderate to strong correlation between goniometer in flexion and abduction in the dominant arm (R = 0.790 and R = 0.650, P < .001, respectively) and moderate correlations for the nondominant arm (R = 0.622 and R = 0.615, P < .001). Compared to the younger cohort, the elderly population demonstrated significantly reduced total RSA in the dominant arm (meanelderly = 0.774, SD = 0.09; meanyounger = 0.830, SD = 0.07, P < .001), with significant reductions in the upper lateral quadrant in both arms (dominant: meanelderly = 0.225, SD = 0.04; meanyounger = 0.241, SD = 0.01; P < .001; nondominant: meanelderly = 0.213, SD = 0.03; meanyounger = 0.228, SD = 0.01; P = .004). The test-retest reliability was strong for both dominant and nondominant total RSA (ICC > 0.762). The reachable workspace demonstrates promise as a simple and quick tool for clinicians to assess detailed and quantitative active shoulder ROM decline in the elderly population.

The subject of this work is to evaluate the precision of the Kinect controller operation. Research study was performed and a measuring stand have been prepared. Then an application was created that captures the user's throw gesture and simulates the flight of a virtual ball. Based on this, measurements were made to determine the difference between hand movement and it’s detection by the application, and differences among throw made in real life and in VR. The analysis of these results allowed the accuracy of the controller to be assessed.

Path quality and computational time have formed together a well-known trade-off problem for path planning techniques. Due to this trade-off, contributions were usually considering improving only one of the two aspects, either increasing the swiftness as in real-time robotic path planning algorithms or enhancing the path quality as in shortest path query algorithms. Producing a path planning technique that targets both aspects is a challenging problem for robotic systems. However, this paper proposes a novel path planning framework that controls the motion of robotic systems and aims to overcome this traditional trade-off challenge, by targeting both, decreasing the computational time, and improving the path quality represented by the path length and smoothness. The shortest path is obtained by minimizing a novel objective function inspired by the artificial potential field methodology. To accelerate the execution, the Particle Swarm Optimization (PSO) technique is adopted to obtain the optimal solution in a real-time hop-by-hop manner imitating the procedures performed by computer network routing protocols. Testbed experimental results have proven the effectiveness of the proposed technique and showed superior performance over other meta-heuristic optimization techniques and over classical path planning approaches such as A*, D*, and PRM.

Fil: Souza Sierra, Isabella de. Universidade Federal do Paraná; Brasil. Fil: Chemin Rosenmann, Gabriel. Universidade Federal do Paraná; Brasil. Fil: Aguiomar Foggiatto, José. Universidade Tecnológica Federal do Paraná; Brasil. Fil: Fontana Catapan, Marcio. Universidade Federal do Paraná; Brasil. Fil: Leite Ribeiro Okimoto, Maria Lucia. Universidade Federal do Paraná; Brasil. Este estudio tuvo como objetivo comprender el comportamiento del escaneo 3D mediante luz infrarroja estructurada al escanear objetos que están detrás de materiales transparentes. Se hicieron las siguientes preguntas: ¿Cuáles son los errores encontrados en la superficie generada por el escaneo 3D a través de varios materiales transparentes? ¿Es factible utilizar estos datos para recopilar mediciones? Para responder a estas preguntas, se realizó un experimento en el que se escaneó el mismo objeto a través de diferentes superficies transparentes en diferentes situaciones. Como resultado, se encontraron cuatro tipos de errores: oclusión, calidad de la superficie, cierre y reflejo. Además, también fue posible probar la usabilidad de estas superficies digitalizadas al realizar mediciones antropométricas de partes del cuerpo grandes y medianas según la definición de ISO 20685:2010. This study aimed at understanding the behavior of the 3D scanning by structured infrared light when scanning objects that are behind transparent materials. The following questions were addressed: What are the errors found on the surface generated by 3D scanning through several transparent materials? Is it feasible to use the data gathered to collect anthropometric measurements? To answer these questions an experiment was carried out in which the same object was scanned through different transparent surfaces in different situations. As a result, four types of errors were found in the scan: occlusion, superficial quality, closure and reflection. Furthermore, it was also possible to prove the usability of these surfaces for taking anthropometric measurements of large and medium body parts as defined by ISO 20685:2010. Este estudo objetivou compreender o comportamento do escaneamento 3D por luz infravermelha estruturada, ao digitalizar objetos que estão atrás de materiais transparentes. Problematizam-se as seguintes questões: Quais os erros encontrados em superfície gerada por escaneamento 3D através de diversos materiais transparentes? É viável a utilização desses dados para coleta de medidas? Para responder a essas perguntas realizou-se um experimento no qual se digitalizou um mesmo objeto através de diferentes superfícies transparentes em diferentes situações. Como resultados foram encontrados quatro tipos de erros no escaneamento: oclusão, qualidade superficial, fechamento e reflexo. Além disso também foi possível comprovar a usabilidade dessas superfícies digitalizadas na realização de medidas antropométricas de grandes e médias partes do corpo, conforme definido pela ISO 20685:2010.

In this paper, we focus on the human upper limb rehabilitation scheme that utilizes the concept ofteleoperation. Teleoperation can help the therapist demonstrate different rehab exercises to a different group of people at the same time remotely. Different groups of people from a different place connected to the same network can get therapy from the same therapist at the same time using the telerehabilitation scheme. Here, we presented a humanoid robot NAO that can be operated remotely by a therapist to demonstrate the exercise to a patient. To mimic the movement demonstrated by the therapist, Kinect V2 sensor which is a markerless vision-based motion-tracking device, was used. Modified Denavit-Hartenberg (DH) convention was used for the kinematic modeling of the human upper arm. From the Kinect data, a geometric solution was developed to find a unique inverse kinematic solution of human upper-extremity. Experimental results revealed that NAO could be teleoperated successfully to instruct and demonstrate patients to perform different arm movement exercises in real-time.

Virtual fitting room (VFR) is a technology that replaces conventional fitting rooms. The VFR is not only available in shops, malls, and any shopping center but also in online stores, which makes VFR technology more and more developed, primarily to support online garment sales. VFR become a trending research interest since Microsoft has developed a Kinect tracking system. In this paper, we proposed the interactive 3D virtual fitting room using Microsoft's Kinect tracking and the rigging technique from 3D Modeling Blender and to implement the VFR. VFR manages the progress of virtual fitting that forms the three-dimensional simulations and visualization of garments on virtual counterparts of the real prospective buyer (user). Users can view the clothing animation on the various poses that are following the user body movements. The system can evaluate the user’s match, guiding them to choose the suitable size of the clothes using Euclidean distance.

Background Abnormalities in gait kinetics in patients with Parkinson’s disease (PD) who have suffer from gait impairment have been noted using a conventional inverse dynamic analysis derived by marker-based motion capture system and force plate, which are typically mounted in the laboratory floor. Despite the high accuracy of this approach in tracking markers’ trajectories and acquiring ground reaction forces (GRFs), its dependence on laboratory-mounted equipment restricts its potential use in wider variety of clinical applications. Research question Would a full-body musculoskeletal model driven by a single depth sensor data only produce comparable gait kinetic parameters, including GRFs and lower extremity joints moments, for elderly participants, both healthy and those diagnosed with PD? Methods Nine patients diagnosed with PD and 11 healthy age-matched control participants performed three over-ground gait trials. Full-body kinematic data were collected using a depth sensor and a musculoskeletal model have been constructed using AnyBody musculoskeletal modeling system to predict the three-dimensional GRFs and lower extremity joint moments. Predicted kinetic parameters for both PD and control groups were compared during the braking and propulsive phases of the gait cycle. In addition, ensemble curve analysis with 90% confidence intervals were constructed to compare between group differences across the stance phase of the gait cycle. Results The findings of this study showed that the PD exhibited a significantly lower braking peak vertical GRF and propulsion peak horizontal GRF while no significant between-group differences were found in peak lower extremity joint moments. However, the PD showed significant alterations in lower extremity joint moments during the early and late phases of stance, which indicate a difference in ambulation strategy. Significance The proposed method adopting full-body musculoskeletal model driven by a depth sensor data proves that it has the potential to be a portable and cost-effective gait analysis tool in the clinical setting.

The article deals with the possibilities of interaction in the virtual environment (VE) of the CAVE LIRKIS system. Commonly available peripherals are keyboard, mouse or gamepad. However, these do not provide a sufficient level of immersion, such as hand gestures. The MYO armband has proven to be one of the potential devices for improving interaction. Therefore, a system for editing an object transformation in VE was created using hand gestures and movements. Enhancing the natural user interface (NUI) of the system with additional interaction capabilities has proven the device to be suitable also for further development

In this paper, we demonstrate a perceptual-based 3D skeleton motion data refinement method based on a bidirectional recurrent autoencoder, called BRA-P. Three main technical contributions are made by the proposed network. First, the proposed BRA-P can address noisy data with different noise types and amplitudes using one network, and this attribute makes the approach more suitable for raw motion data with heterogeneous mixed noise. Second, due to the usage of perceptual loss, which measures the difference in high-level features extracted by a pretrained perceptual autoencoder, BRA-P improves the perceptual similarity between refined motion data and clean motion data, especially for the case where the noisy data and target clean data have different topologies. Third, BRA-P further improves the bone-length consistency and smoothness of the refined motion using the perceptual autoencoder as a postprocessing network. Ablation experiments verify the effect of the three technical contributions of our approach. The results of the experiments on synthetic noise data and raw motion data captured by Kinect demonstrate that our method outperforms several state-of-the-art methods in the cleaning of mixed-noise data by one network.

In the field of computer vision, the depth image sequence collected by depth camera is not sensitive to the interference of light, occlusion and background environment. Therefore, in recent years, it is often used to collect behavior data, from which the characteristics of bone joint points are extracted as behavior information. However, it is found that the direct use of joint coordinate information collected by depth camera for behavior recognition is easily affected by individual differences in behavior and changes in shooting distance. Considering the position information of human joints as well as the angle and length information of hidden limbs, based on skeleton data, a behavior feature description method based on the ratio of vector angle and vector mode of human structure is proposed. This method solves the above problems perfectly and obtains ideal results on the self built data set, which is suitable for simple daily behavior recognition.

Current literature lacks a comparative analysis of different motion capture systems for tracking upper limb (UL) movement as individuals perform standard tasks. To better understand the performance of various motion capture systems in quantifying UL movement in the prosthesis user population, this study compares joint angles derived from three systems that vary in cost and motion capture mechanisms: a marker-based system (Vicon), an inertial measurement unit system (Xsens), and a markerless system (Kinect). Ten healthy participants (5F/5M; 29.6 ± 7.1 years) were trained with a TouchBionic i-Limb Ultra myoelectric terminal device mounted on a bypass prosthetic device. Participants were simultaneously recorded with all systems as they performed standardized tasks. Root mean square error and bias values for degrees of freedom in the right elbow, shoulder, neck, and torso were calculated. The IMU system yielded more accurate kinematics for shoulder, neck, and torso angles while the markerless system performed better for the elbow angles. By evaluating the ability of each system to capture kinematic changes of simulated upper limb prosthesis users during a variety of standardized tasks, this study provides insight into the advantages and limitations of using different motion capture technologies for upper limb functional assessment.

Background Motor impairment is widely acknowledged as a core feature in children with autism spectrum disorder (ASD), which can affect adaptive behavior and increase severity of symptoms. Low-cost motion capture and virtual reality (VR) game technologies hold a great deal of promise for providing personalized approaches to motor intervention in ASD. The present study explored the feasibility, acceptability and potential efficacy of a custom-designed VR game-based intervention (GaitWayXR™) for improving gross motor skills in youth with ASD. Methods Ten children and adolescents (10–17 years) completed six, 20-min VR-based motor training sessions over 2 weeks while whole-body movement was tracked with a low-cost motion capture system. We developed a methodology for using motion tracking data to quantify whole-body movement in terms of efficiency, synchrony and symmetry. We then studied the relationships of the above quantities with standardized measures of motor skill and cognitive flexibility. Results Our results supported our presumption that the VR intervention is safe, with no adverse events and very few minor to moderate side-effects, while a large proportion of parents said they would use the VR game at home, the most prohibitive reasons for adopting the system for home therapy were cost and space. Although there was little evidence of any benefits of the GaitWayXR™ intervention in improving gross motor skills, we showed several positive correlations between the standardized measures of gross motor skills in ASD and our measures of efficiency, symmetry and synchrony from low-cost motion capture. Conclusions These findings, though preliminary and limited by small sample size, suggest that low-cost motion capture of children with ASD is feasible with movement exercises in a VR-based game environment. Based on these preliminary findings, we recommend conducting larger-scale studies with methods for improving adherence to VR gaming interventions over longer periods.

[Resumen] En un mundo cada vez más conectado a la “Red” en el que el trato y la forma del proceso de compra-venta ha sufrido un gran cambio en la última década con la irrupción del comercio online, se hace indispensable que el pequeño comercio se reinvente para dar mejor trato y satisfacción a los clientes que pretende fidelizar. En este contexto se enmarca el presente trabajo, que propone el diseño e implementación de un prototipo funcional de escaparate virtual a través de la tecnología de Kinect fusionando informática y técnicas de control. El trabajo y prototipo desarrollado ha sido todo un éxito comercial, obteniendo distintos reconocimientos y premios, siendo además protegido por una patente de invención denominada “Procedimiento para escaparate virtual in situ”, con número de solicitud P201531546 y fecha de prioridad 29 de octubre de 2015. Además, este artículo muestra los resultados comerciales obtenidos de la implantación del prototipo durante 3 meses en la tienda Mikonos de Logroño (La Rioja), haciendo una comparativa de datos con el mismo periodo del año anterior. Como conclusión de este estudio de resultados se puede obtener que la implantación del prototipo de escaparate interactivo virtual ha sido todo un éxito, consiguiendo su objetivo de aumentar el número de personas que han entrado en la tienda en casi un 20% y siendo responsable en parte del aumento de facturación en más de un 10%. [Abstract] In a world increasingly connected to the "Network" in which the treatment and form of the buying and selling process has undergone a great change in the last decade with the irruption of e-commerce, it is essential that small businesses reinvent themselves to give better treatment and satisfaction to the customers they intend to retain. In this context, the present work is framed, which proposes the design and implementation of a functional prototype of a virtual showcase through Kinect technology, merging computer and control techniques. The work and prototype developed has been a commercial success, obtaining different recognitions and awards, being also protected by an invention patent called "On-site virtual showcase procedure", with application number P201531546 and priority date October 29, 2015. In addition, this article shows the commercial results obtained from the implementation of the prototype for 3 months in the Mikonos store in Logroño, making a comparison of data with the same period of the previous year. As a conclusion of this study of results it can be obtained that the implementation of the prototype of virtual interactive showcase has been a success, achieving its objective of increasing the number of people who have entered the store by almost 20% and being responsible in part for the increase in turnover by more than 10%.

KMUTT Research and Development Journal, 45, 3, 341-352

Mild Cognitive Impairment (MCI) in pensioners has become an important concern in the aging population, and there are an increasing number of products, especially virtual reality (VR)-based products, to assist in the identification, intervention, and treatment of MCI older adults. Multiple studies on the efficacy and usability of VR products are also receiving increased attention from designers. However, issues such as the efficacy testing of VR products still face serious challenges. In this study, we evaluated VR products looking at the interactive responsiveness of MCI older adults when using VR products, and analyzed the brain activation status and behavioral conditions of MCI older adults. Multisource data were generated by a functional near-infrared spectroscopy (fNIRS) device with high spatial resolution and a behavioral recording device reflecting motor abilities. Small-wave amplitudes were selected as indicators of brain activation to analyze six brain areas: LPFC, RPFC, LOL, ROL, LMC, and RMC. Eight aspects, such as overall upper-limb speed, upper-limb global acceleration, and median velocity, were selected as indicators for behavioral recording. The differences were observed by comparing the level of completion of interactive responses by MCI older adults between the two groups. The results showed that MCI older adults showed different levels of activation in brain regions when performing VR product-based tasks. The higher the level of cognition, the better the interactive response in the task and the stronger the activation of brain regions. Meanwhile, the level of interaction response had a significant correlation with the motor performance of MCI older adults, with stronger motor functions leading to a more effective interaction response to the product. This study proposes a new method to evaluate the feasibility of monitoring the interaction between the MCI elderly and VR-based products using fNIRS with Kinect, which provides a new way to evaluate the effectiveness of VR-based product-assisted treatment.

Augmented Reality is a technology that allows to expand the traditional learning techniques complementing the perception and interaction with the real world that allows the student to be in real environment with additional information generated by the computational algorithm. However, the knowledge and applicability of this technology in the field of personalized education is not a common practice. In this article, personalized education strategies are applied in the process of developing the application for indoor climbing teaching techniques. The application allows the trainer or climber to select the climbing holds that make up a route and display it by visualization on a projector to customize the training program. The system has the detection algorithm and recognition of climbing holds in real time and visualization of the route to climb. This kind of applications using emergent technologies oriented to personalized training has enormous potential for efficient education.

Purpose For an in-depth analysis of the learning benefits that a stereoscopic view presents during endoscopic training, surgeons required a custom surgical evaluation system enabling simulator independent evaluation of endoscopic skills. Automated surgical skill assessment is in dire need since supervised training sessions and video analysis of recorded endoscope data are very time-consuming. This paper presents a first step towards a multimodal training evaluation system, which is not restricted to certain training setups and fixed evaluation metrics. Methods With our system we performed data fusion of motion and muscle-action measurements during multiple endoscopic exercises. The exercises were performed by medical experts with different surgical skill levels, using either two or three-dimensional endoscopic imaging. Based on the multi-modal measurements, training features were calculated and their significance assessed by distance and variance analysis. Finally, the features were used automatic classification of the used endoscope modes. Results During the study, 324 datasets from 12 participating volunteers were recorded, consisting of spatial information from the participants' joint and right forearm electromyographic information. Feature significance analysis showed distinctive significance differences, with amplitude-related muscle information and velocity information from hand and wrist being among the most significant ones. The analyzed and generated classification models exceeded a correct prediction rate of used endoscope type accuracy rate of 90%. Conclusion The results support the validity of our setup and feature calculation, while their analysis shows significant distinctions and can be used to identify the used endoscopic view mode, something not apparent when analyzing time tables of each exercise attempt. The presented work is therefore a first step toward future developments, with which multivariate feature vectors can be classified automatically in real-time to evaluate endoscopic training and track learning progress., International Journal of Computer Assisted Radiology and Surgery, 17 (9), ISSN:1861-6410, ISSN:1861-6429

Most post-stroke patients experience varying degrees of impairment in upper limb function and fine motor skills. Occupational therapy (OT) with other rehabilitation trainings is beneficial in improving the strength and dexterity of the impaired upper limb. An accurate upper limb assessment should be conducted before prescribing upper limb OT programs. In this paper, we present a novel multisensor method for the assessment of upper limb movements that uses kinematics and physiological sensors to capture the movement of the limbs and the surface electromyogram (sEMG). These sensors are Kinect, inertial measurement unit (IMU), Xsens, and sEMG. The key assessment features of the proposed model are as follows: (1) classification of OT exercises into four classes, (2) evaluation of the quality and completion of the OT exercises, and (3) evaluation of the relationship between upper limb mobility and muscle strength in patients. According to experimental results, the overall accuracy for OT-based motion classification is 82.2%. In addition, the fusing of Kinect and Xsens data reveals that muscle strength is highly correlated with the data with a correlation coefficient (CC) of 0.88. As a result of this research, occupational therapy specialists will be able to provide early support discharge, which could alleviate the problem of the great stress that the healthcare system is experiencing today.

Advancements in motion sensing technology can potentially allow clinicians to make more accurate range-of-motion (ROM) measurements and informed decisions regarding patient management. The aim of this study was to systematically review and appraise the literature on the reliability of the Kinect, inertial sensors, smartphone applications and digital inclinometers/goniometers to measure shoulder ROM. Eleven databases were screened (MEDLINE, EMBASE, EMCARE, CINAHL, SPORTSDiscus, Compendex, IEEE Xplore, Web of Science, Proquest Science and Technology, Scopus, and PubMed). The methodological quality of the studies was assessed using the consensus-based standards for the selection of health Measurement Instruments (COSMIN) checklist. Reliability assessment used intra-class correlation coefficients (ICCs) and the criteria from Swinkels et al. (2005). Thirty-two studies were included. A total of 24 studies scored “adequate” and 2 scored “very good” for the reliability standards. Only one study scored “very good” and just over half of the studies (18/32) scored “adequate” for the measurement error standards. Good intra-rater reliability (ICC > 0.85) and inter-rater reliability (ICC > 0.80) was demonstrated with the Kinect, smartphone applications and digital inclinometers. Overall, the Kinect and ambulatory sensor-based human motion tracking devices demonstrate moderate–good levels of intra- and inter-rater reliability to measure shoulder ROM. Future reliability studies should focus on improving study design with larger sample sizes and recommended time intervals between repeated measurements.

This paper presents a modular approach to generic exergame design that combines custom physical exercises in a meaningful and motivating story. This aims to provide a tool that can be individually tailored and adapted to people with different needs, making it applicable to different diseases and states of disease. The game is based on motion capturing and integrates four example exercises that can be configured via our therapeutic web platform “Blexer-med”. To prove the feasibility for a wide range of different users, evaluation tests were performed on 14 patients with various types and degrees of neuromuscular disorders, classified into three groups based on strength and autonomy. The users were free to choose their schedule and frequency. The game scores and three surveys (before, during, and after the intervention) showed similar experiences for all groups, with the most vulnerable having the most fun and satisfaction. The players were motivated by the story and by achieving high scores. The average usage time was 2.5 times per week, 20 min per session. The pure exercise time was about half of the game time. The concept has proven feasible and forms a reasonable basis for further developments. The full 3D exercise needs further fine-tuning to enhance the fun and motivation.

Hereditary Amyloidosis associated with variant Transthyretin (ATTRv Amyloidosis) is a progressive and highly disabling neurological disorder that affects gait. Quantitative motion analysis is useful for assessing motor function, including gait, in diseases affecting movement. A single markerless RGB-D camera enables 3D full-body motion capture in a less expensive and intrusive, and more portable way than multi-camera marker-based systems. In this study, we examine whether a gait analysis system based on an RGB-D camera can be used to detect significant changes in the gait of ATTRv amyloidosis patients over time, when compared with a 12-camera system. We acquired 3D data provided by both systems from six ATTRv amyloidosis patients, while performing a simple gait task, once (T0) and 18 months later (T1). A direct comparison of systems has already been conducted. In this work, however, for each patient, we investigated if the RGB-D camera system detects statistically significant differences between the two different acquisitions in a similar way to the reference system, and whether it is reliable to use during patients’ follow-up. The obtained results show that the differences detected between T0 and T1 for both systems follow the same tendency for 65% of the spatiotemporal gait parameters, and for 38% of the kinematic parameters (38%). The most reliable parameters were: stride duration/length, gait speed (and its variability), and arm/foot swing velocity, all with an almost perfect strength of agreement.

The multi-fingered dexterous robotic hand is increasingly used to achieve more complex and sophisticated human-like manipulation tasks on various occasions. This paper proposes a hybrid Surface Electromyography (SEMG) and Kinect-based human in-hand motion (HIM) capture system architecture for recognizing complex motions of the humans by observing the state information between an object and the human hand, then transferring the manipulation skills into bionic multi-fingered robotic hand realizing dexterous in-hand manipulation. First, an Adaptive Directed Acyclic Graph (ADAG) algorithm for recognizing HIMs is proposed and optimized based on the comparison of multi-class support vector machines; second, ten representative complex in-hand motions are demonstrated by ten subjects, and SEMG and Kinect signals are obtained based on a multi-modal data acquisition platform; then, combined with the proposed algorithm framework, a series of data preprocessing algorithms are realized. There is statistical symmetry in similar types of SEMG signals and images, and asymmetry in different types of SEMG signals and images. A detailed analysis and an in-depth discussion are given from the results of the ADAG recognizing HIMs, motion recognition rates of different perceptrons, motion recognition rates of different subjects, motion recognition rates of different multi-class SVM methods, and motion recognition rates of different machine learning methods. The results of this experiment confirm the feasibility of the proposed method, with a recognition rate of 95.10%.

Stability during walking is considered a crucial aspect of assessing gait ability. The current study aimed to assess walking stability by applying principal component analysis (PCA) to decompose three-dimensional (3D) whole-body kinematic data of 104 healthy young adults (21.9 ± 3.5 years, 54 females) derived from a depth-sensing camera into a set of movement components/synergies called “principal movements” (PMs), forming together to achieve the task goal. The effect of sex as the focus area was tested on three PCA-based variables computed for each PM: the relative explained variance (rVAR) as a measure of the composition of movement structures; the largest Lyapunov exponent (LyE) as a measure of variability; and the number of zero-crossings (N) as a measure of the tightness of neuromuscular control. The results show that the sex effects appear in the specific PMs. Specifically, in PM1, resembling the swing-phase movement, females have greater LyE (p = 0.013) and N (p = 0.017) values than males. Moreover, in PM3, representing the mid-stance-phase movement, females have smaller rVAR (p = 0.020) but greater N (p = 0.008) values than males. These empirical findings suggest that the inherent sex differences in walking stability should be considered in assessing and training locomotion.

Exercise games (exergames) that combine both exercise and video gaming train people in a fun and competitive manner to lead a healthy lifestyle. Exergames promote more physical exertion and help users exercise more easily and independently in any place. Many studies have been conducted to evaluate the positive effects of exergames. However, in most studies, heart rate was mainly used to measure the effect of exercise. In this study, we evaluate the effects of exercise according to the exercise type (rest, walking, tennis, and running) and gameplay mode (single, competition, and cooperation) of exergaming via quantitative measurements using electrocardiogram (ECG) and Kinect. The multiple comparison results reveal that physical activity measured with Kinect was statistically significant even in exergames that did not show statistically significant differences according to ECG. Running was statistically significant compared to other exercise types, and there was a significant difference in competition compared to other gameplay modes.

Most red-green-blue and depth (RGB-D) motion-recognition technologies employ both depth and RGB cameras to recognize a user's body. However, motion-recognition solutions using a single RGB-D camera struggle with rotation recognition depending on the device-user distance and field-of-view. This paper proposes a near-real-time rotational-coordinate-correction method that rectifies a depth error unique Microsoft Kinect by using the symmetry of the depth coordinates of the human body. The proposed method is most effective within 2 m, a key range in which the unique depth error of Kinect occurs, and is anticipated to be utilized in applications requiring low cost and fast installation. It could also be useful in areas such as media art that involve unspecified users because it does not require a learning phase. Experimental results indicate that the proposed method has an accuracy of 85.38%, which is approximately 12% higher than that of the reference installation method.

Low-cost, portable, and easy-to-use Kinect-based systems achieved great popularity in out-of-the-lab motion analysis. The placement of a Kinect sensor significantly influences the accuracy in measuring kinematic parameters for dynamics tasks. We conducted an experiment to investigate the impact of sensor placement on the accuracy of upper limb kinematics during a typical upper limb functional task, the drinking task. Using a 3D motion capture system as the golden standard, we tested twenty-one Kinect positions with three different distances and seven orientations. Upper limb joint angles, including shoulder flexion/extension, shoulder adduction/abduction, shoulder internal/external rotation, and elbow flexion/extension angles, are calculated via our developed Kinect kinematic model and the UWA kinematic model for both the Kinect-based system and the 3D motion capture system. We extracted the angles at the point of the target achieved (PTA). The mean-absolute-error (MEA) with the standard represents the Kinect-based system’s performance. We conducted a two-way repeated measure ANOVA to explore the impacts of distance and orientation on the MEAs for all upper limb angles. There is a significant main effect for orientation. The main effects for distance and the interaction effects do not reach statistical significance. The post hoc test using LSD test for orientation shows that the effect of orientation is joint-dependent and plane-dependent. For a complex task (e.g., drinking), which involves body occlusions, placing a Kinect sensor right in front of a subject is not a good choice. We suggest that place a Kinect sensor at the contralateral side of a subject with the orientation around 30∘ to 45∘ for upper limb functional tasks. For all kinds of dynamic tasks, we put forward the following recommendations for the placement of a Kinect sensor. First, set an optimal sensor position for capture, making sure that all investigated joints are visible during the whole task. Second, sensor placement should avoid body occlusion at the maximum extension. Third, if an optimal location cannot be achieved in an out-of-the-lab environment, researchers could put the Kinect sensor at an optimal orientation by trading off the factor of distance. Last, for those need to assess functions of both limbs, the users can relocate the sensor and re-evaluate the functions of the other side once they finish evaluating functions of one side of a subject.

Background Automated and accurate assessment for postural abnormalities is necessary to monitor the clinical progress of Parkinson’s disease (PD). The combination of depth camera and machine learning makes this purpose possible. Methods Kinect was used to collect the postural images from 70 PD patients. The collected images were processed to extract three-dimensional body joints, which were then converted to two-dimensional body joints to obtain eight quantified coronal and sagittal features (F1-F8) of the trunk. The decision tree classifier was carried out over a data set established by the collected features and the corresponding doctors’ MDS-UPDRS-III 3.13 (the 13th item of the third part of Movement Disorder Society-Sponsored Revision of the Unified Parkinson’s Disease Rating Scale) scores. An objective function was implanted to further improve the human–machine consistency. Results The automated grading of postural abnormalities for PD patients was realized with only six selected features. The intraclass correlation coefficient (ICC) between the machine’s and doctors’ score was 0.940 (95%CI, 0.905–0.962), meaning the machine was highly consistent with the doctors’ judgement. Besides, the decision tree classifier performed outstandingly, reaching 90.0% of accuracy, 95.7% of specificity and 89.1% of sensitivity in rating postural severity. Conclusions We developed an intelligent evaluation system to provide accurate and automated assessment of trunk postural abnormalities in PD patients. This study demonstrates the practicability of our proposed method in the clinical scenario to help making the medical decision about PD.

In this short survey and case study we want to present our research experience through the project developed by our team, that involves the building of a LEGO MINDSTORMS EV3 robotic arm and tracked robot car which mimics the motion of the human arm and legs. We used 3 interconnected LEGO MINDSTORMS EV3 bricks to reach the desired degrees of freedom. Using a Kinect sensor, the system detects the motion of the human user’s arm and creates the skeletal image of the arm. Coordinate geometry and different approximation methods are used to calculate the rotation angles between the bones connecting the joints. In our project the key is inverse kinematics, whitch makes use of the kinematics equations to determine the joint rotation parameters that provide a desired position for each of the robot’s end-effectors – arms and legs (wheels). The combined motion of the LEGO MINDSTORMS EV3 motors results in a complete robotic forward or backward motion and arm movement which is a perfect mimic of the human arm movement.

Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection. This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation.

Submitted by LEONEL CASEIRO MORGADO (leonel.morgado@uab.pt) on 2019-01-31T17:28:30Z No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Approved for entry into archive by LEONEL CASEIRO MORGADO (leonel.morgado@uab.pt) on 2019-01-31T17:28:53Z (GMT) No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Made available in DSpace on 2019-02-04T11:02:07Z (GMT). No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Previous issue date: 2019 Fundação Calouste Gulbenkian info:eu-repo/semantics/publishedVersion

The quantification of snow transport, both in wind tunnels and the field, apply particle counting methods limited to punctual sampling of relatively small volumes. Particle counting can only capture horizontal mass fluxes, failing to measure snow erosion or deposition. Herein, we present a novel low-cost sensor tool, based on a Microsoft Kinect, adapted to capture snow surface changes during snow drifting at unprecedented spatial and temporal resolutions. In the wind tunnel setting of these experiments we observe a balance between erosion and deposition at low wind speeds, while erosion is dominant at higher wind speeds. Significant differences in power spectral densities of surface mass flux and horizontal particle mass flux are observed. We show that for the saltation-length-scale parameter λ = 1, the integrated particle flux can be used to estimate the total surface mass flux in the wind tunnel. This provides an important basis to interpret mass flux measurements in the field.

Penelitian ini bertujuan untuk melakukan analisa terhadap sistem pemindai 3D untuk komponen mesin peralatan pertanian menggunakan teknologi kinect. Pemanfaat teknologi kinect ini diharapkan mampu memindai dan mengambil gambar komponen-komponen mesin pertanian secara 3D. Analisa yang digunakan adalah untuk mengetahui tingkat akurasi kinect dalam memindai komponen mesin pertanian. Pemindai 3D ini menggunakan teknologi kinect sebagai sesor utama. Kinect terdiri dari sensor infra red, sensor depth image coms dan sensor color image CMOS. Semua sensor kinect memindai obyek secara 3D dan kemudian diproses oleh processor kinect. Hasil proses tersebut dilanjutkan ke perangkat lunak pada pemroses data sehingga mengasilkan file hasil pemindaian dalam bentuk 3D. Hasil pengujian pada komponen dengan ukuran yang berbeda-beda menunjukan bahwa komponen dengan ukuran dibawah diameter 7,5 dan tinggi 10 cm tidak mampu dipindai. Komponen dengan ukuran diatas diameter 10 cm dan tinggi 15 cm dapat dipindai dengan akurasi yang cukup baik.

This paper presented a framework based on fog computing for convenient and efficient Physiological Function Assessment, which consists of three parts: 1)measuring the degree of joint mobility; 2) investigating the abnormality of actions of upper limbs; and 3) abnormal gait detection for lower limbs. Especially, we introduced semi-automatic Rapid Upper Limb Assessment (RULA) using Kinect v2 for the upper limb motion evaluation. Since a specific action can be described by action sequences of different length, we used dynamic time warping (DTW) to find the similarity between action sequences with different length. Traditional DTW algorithm does not work well when the action sequences are long and complex. To address this problem, we improved the DTW method by modifying the mapping relationship and limiting the computation space. Our modified DTW algorithm was evaluated on a standard 3D action dataset (SYSU 3D HOI) and Human Upper Action dataset (HUA), achieving the accuracy of 83.75%, 89.50%, respectively. The result is significantly better than the traditional DTW and the reported methods. In our previous work, we described the framework and how to make physiological function assessment. The goal of this paper is to 1) enrich the experiments of previous work and 2) introduce the framework of using RULA for physiological function assessment. All the tests have been done in this framework based on fog computing.

The virtual rehabilitation system combining virtual rehabilitation environment and upper limb rehabilitation technology is interactive and interesting, which can improve the enthusiasm and initiative of patients for rehabilitation training, improve the efficiency of rehabilitation training and improve the effect of rehabilitation treatment. This paper firstly conducts in-depth research and analysis on the research progress of the upper limb rehabilitation robot system, and deeply studies the principle and rehabilitation principle of the stroke caused by hemiplegic dyskinesia, and summarizes the goals and methods of the upper limb rehabilitation system design. Secondly, the hand motion tracking is realized by Kinect's bone tracking, and the optimal tracking distance is determined experimentally, which verifies the stability and robustness of the tracking. Static gesture recognition adopts two gesture recognition schemes based on Kinect depth image and color space model respectively. Finally, using the rehabilitation robot and Kinect sensor as the hardware platform, the virtual rehabilitation training system experimental platform is constructed, and the horizontal rehabilitation exercise and the three-dimensional space rehabilitation exercise are respectively studied experimentally, and the exercise data obtained by using healthy subjects as the experimental object is analyzed. Based on this, the validity and feasibility of the Kinect-based upper limb exercise rehabilitation interactive system were verified.

trabalho proposto busca realizar uma revisão bibliográfica para levantar as bases de dados e reconhecedores de ações estado da arte na literatura recente, além de selecionar dois reconhecedores para compará-los. De acordo com o estudo realizado, os reconhecedores baseados em esqueletos 3D se mostraram promissores e os dois com maior desempenho foram selecionados: MS-G3D e Shift-GCN. Por outro lado, as bases de dados escolhidas foram a NTU RGB+D 120, com 11480 amostras e 120 classes, e a NTU RGB+D HOME, obtida por um subconjunto da NTU RGB+D 120 de 23784 amostras e 25 classes com o foco em ações realizadas em ambientes domésticos ou relacionadas à saúde. Assim, 12 configurações dos reconhecedores selecionados foram criadas e comparadas nas bases de dados propostas pelos métodos cross-subject e cross-setup, onde todas mostraram desempenhos estado da arte de até 85,80% para a base de dados NTU RGB+D 120 e até 84,32% para a NT RGB+D HOME. Além disso, foi possível observar que os reconhecedores baseados em esqueletos 3D são mais robustos em relação a ambientes e perspectivas diferentes e que as informações das arestas entre as juntas ou as movimentação entre as juntas dos esqueletos 3D em frames adjacentes resultaram em um maior desempenho dos reconhecedores. Por último, é possível observar que a fusão dos resultados de reconhecedores treinados com informações diferentes pode trazer um aumento considerável de seus desempenhos. Espera-se que o estudo realizado neste trabalho pode ser utilizado para guiar outros pesquisadores para a seleção de bases de dados e reconhecedores de ações, além de auxiliar a equipe do programa RoboFEI@HOME a incrementar a plataforma robótica HERA com capacidades de reconhecimento de comportamento humano. The purpose of this project is to review the bibliography in order to understand the characteristics of the state-of-the-art databases and recognition methods for human action and to evaluate two of the studied methods. According to the recent literature, the 3D skeleton based methods have shown promising results, thus the two methods with better performance were selected: MS-G3D and Shift-GCN. On the other hand, to train these methods, the NTU RGB+D 120 database, composed by 114480 samples from 120 classes, and the NTU RGB+D HOME, composed by 23784 samples and 25 classes from a subset of the NTU RGB+D 120 database, where selected. Twelve configurations of the MS-G3D and Shift-GCN where created and evaluated in the proposed databases by the cross-subject and cross-setup methods where state-of-the-art performance was achieved with results up to 85,80% and 84,32% for the NTU RGB+D 120 and NTU RGB+D HOME databases, respectively. Furthermore, the 3D skeleton bases methods achieved a better performance on the cross-setup evaluation where different environments and view points where tested. Also, the configurations trained with bone information and joint motion information presented higher performance. At last, the results indicate that fusion from models trained with different information such as joint, bone, joint motion and bone motion information can achieve a better recognition performance. It is expected that this project may be used by other researchers as a guide in the process of choosing database or methods for human action recognition, and also help the roboFEI@HOME team to increment their robot platform named HERA with human action recognition capabilities.

This paper describes several computer access methods tested by Eva, a woman with choreoathetosic cerebral palsy. This disease prevents her from controlling the peripherals and configurations that normally give access to information and communication technologies, further limiting her independence. To make Eva access a computer, we focused our efforts on the methodologies that Eva could control by just moving her neck and head. These sensors were: Kinect, inertial measurement units (IMU), and video. Kinect, composed of a system of cameras and sensors, gives the option to interact and control the devices contactlessly. The IMU is a device consisting of an accelerometer and a gyroscope that measure velocity, orientation, and gravitational forces. For live image processing, a common webcam was used. During the development of the experiment, Eva must follow a sequence shown on the computer screen that alternates movement of the head with rest. These movements involved moving the head up, down, right, or left. Our results showed that the Kinect system could not be used effectively, while the image-processing algorithm obtained the best performance. Spanish Ministry of Science and Innovation, State Plan 2017–2020: Challenges—R&D&I Projects with grant codes PID2019-104323RB-C32

Com a evolução da tecnologia, monitores de computador e televisores têm-se tornado cada vez mais finos e baratos, o que permitiu a sua colocação em muitos espac¸os públicos tais como aeroportos, halls de entrada de edifícios, etc. para apresentação de informação a quem frequente esses espac¸os. No entanto, a utilização destes ecrãs apresenta alguns desafios sobretudo quanto `a forma como a informação é apresentada e como se pode interagir com a mesma. Em trabalhos anteriores estudámos a possibilidade de utilizar telemóveis como dispositivos de interação com estes ecrãs; neste artigo, apresentamos uma evolução que utiliza o Kinect como dispositivo de entrada permitindo um controlo da informação apresentada no ecrã sem recurso a hardware adicional. São descritas as modificações ao protótipo original (desenhado para telemóveis) e duas propostas de interação diferentes: ''Gesture-Based Interaction'' e ''Pointer-Based Interaction''. Finalmente, apresentam-se os resultados dos testes efectuados para avaliar as escolhas realizadas e testar a viabilidade do sistema para interagir com ecrãs em espaços públicos., 20o EPCG: Encontro Portugues de Computacao Grafica, Artigos longos, 81, 88, Igor Duarte Cardoso, Paulo Dias, and Beatriz Sousa Santos, Kinect, large displays, gesture recognition, natural interaction, user interfaces

Stroke is currently the main cause of death and disability among the elderly in our country. According to medical research, a considerable number of patients with physical disabilities can be recovered through rehabilitation training. However, traditional motor rehabilitation is a continuous, repeated and slow process, and patients tend to feel bored and lose motivation for training.In this paper, we developed a virtual reality rehabilitation system based on Kinect, which is a vision capture sensor, for patients with movement disorders who are at or above Brunnstrom Stage III and have certain motor ability. Through the management platform of the system, physician can obtain the patient's personal information, formulate and adjust the training plan. Patients can control the role in the virtual scene through Kinect sensor, and complete the training action according to the guidance. The system collects the user's motion data in real time and detect the compensation. The system will adaptively evolve to guide the patient to self-correct the compensatory patterns. After the training, the system will evaluate the patients based on the their training performance. Two experiments are also carried out to verify the accuracy of the range of motion and the effectiveness of virtual guidance. It is proved that the virtual reality upper limb rehabilitation training system studied in this paper is reliable, stable, and can guide users to complete the training action and improve the rehabilitation effect.

Using standard digital cameras in combination with deep learning (DL) for pose estimation is promising for the in-home and independent use of exercise games (exergames). We need to investigate to what extent such DL-based systems can provide satisfying accuracy on exergame relevant measures. Our study assesses temporal variation (i.e., variability) in body segment lengths, while using a Deep Learning image processing tool (DeepLabCut, DLC) on two-dimensional (2D) video. This variability is then compared with a gold-standard, marker-based three-dimensional Motion Capturing system (3DMoCap, Qualisys AB), and a 3D RGB-depth camera system (Kinect V2, Microsoft Inc). Simultaneous data were collected from all three systems, while participants (N = 12) played a custom balance training exergame. The pose estimation DLC-model is pre-trained on a large-scale dataset (ImageNet) and optimized with context-specific pose annotated images. Wilcoxon&rsquo, s signed-rank test was performed in order to assess the statistical significance of the differences in variability between systems. The results showed that the DLC method performs comparably to the Kinect and, in some segments, even to the 3DMoCap gold standard system with regard to variability. These results are promising for making exergames more accessible and easier to use, thereby increasing their availability for in-home exercise.

Purpose In endovascular neurosurgery, the operator often acquires three-dimensional (3D) images of the cerebral vessels. Although workstation reoperation is required in some situations during treatment, it leads to time loss because a sterile condition cannot be maintained and treatment must be temporarily interrupted. Therefore, a workstation reoperating system is required while maintaining the desired sterility. Methods A contactless operating interface using Kinect to control 3D images was developed via gesture recognition for endovascular neurosurgery and was applied to a 3D volume rendering technique (VRT) image reconstructed at the workstation. The left-hand movement determines the assigned functions, whereas the right-hand movement is used like a computer mouse to pan and zoom in/out. In addition to the interface, voice commands were used and assigned to digital operations, such as image view changes and mode signal changes. Results This system was used for the actual endovascular treatment of cerebral aneurysms and cerebral arteriovenous malformations. The operator and gesture were recognized without any problems. Using voice operation, it was possible to expeditiously set the VRT image back to the reference angle. Furthermore, it was possible to finely adjust gesture operations, including mouse operation, and treatment was completed while maintaining sterile conditions. Conclusion A contactless operating interface was developed by combining the existing workstation system with Kinect and voice recognition software, allowing surgeons to perform a series of operations, which are normally performed in a console room, while maintaining sterile conditions. Supplementary Information The online version contains supplementary material available at 10.1007/s11548-021-02330-3.

Diante do emergente desenvolvimento de dispositivos inovadores e de baixo custo para jogos, uma recente iniciativa de pesquisa para explorar essa tecnologia de Interface Natural do Usuário (INU) e Realidade Virtual (RV) foi identificada. Nesse contexto, esse trabalho apresenta uma proposta de integração de dois periféricos de entrada natural (Microsoft Kinect e Razer Hydra) com o intuito de melhorar o retorno da interação em ambientes virtuais. O Kinect é um dispositivo não-vestível que oferece rastreamento de corpo livre para uma grande área de aplicações em diversos campos do conhecimento. O controle Hydra, por sua vez, é um sensor relativamente novo, o qual ainda não teve suas inúmeras capacidades suficientemente exploradas. As principais características relacionadas ao design de Interação Natural foram descritas de forma a prover a compreensão e acesso ao estado da arte dessa linha de pesquisa. Um sistema híbrido, composto por um dispositivo de rastreamento do corpo inteiro e por um controle que permite seis graus de liberdade de movimento, foi desenvolvido como prova de conceito a fim de avaliar a proposta e analisar os desafios e possíveis aplicações futuras. Os resultados mostraram que a estratégia de integração endossa a hipótese de ganho de qualidade de interação. Due to the increasing development of innovative and low-cost devices for games, a recent research initiative to explore this technology on Natural User Interface (NUI) and Virtual Reality (VR) applications has been identified. Thus, this work presents an approach to combine two input devices (Microsoft Kinect and Razer Hydra) to enhance the interaction feedback within virtual environments. Microsoft Kinect offers non-wearable tracking with a wide range application in different fields of research. The controller Razer Hydra, on the other hand, is a relatively new sensor which hasn’t had its full capabilities explored yet. The main characteristics related to NUI design were described to comprehend the state-of-art of interaction techniques and related works. A hybrid system composed by a full-body tracking device – Microsoft Kinect – and a hand-held six degrees-of-freedom motion controllers – Razer Hydra – was developed as a proof of concept to evaluate this approach and analyzes its challenges and possible further applications. Results have shown that the proposed integration endorses the hypotheses of interaction improvement. Dissertação (Mestrado)

Smart coaching in martial arts is one of the recent research areas in Human Motion Analysis. Numerous moves are performed incorrectly during the performance. In this paper we offer a system that will record the Players’ movements using IR (Infrared) camera sensor, store the data in a database, pre-process the data, classify the data using F-DTW (Fast Dynamic Time Warping) and then show the users an accurate report that contains every movement the player had done, their mistakes and how to improve their performance the next time. This approach focuses on the first seven movements of Karate Kata 1 (Hein Shodan). The system has reached an accuracy of 91.07% in classifying the moves and one common mistake for each move.