Search

We propose a 6D RGB-D odometry approach that finds the relative camera pose between consecutive RGB-D frames by keypoint extraction and feature matching both on the RGB and depth image planes. Furthermore, we feed the estimated pose to the highly accurate KinectFusion algorithm, which uses a fast ICP (Iterative Closest Point) to fine-tune the frame-to-frame relative pose and fuse the depth data into a global implicit surface. We evaluate our method on a publicly available RGB-D SLAM benchmark dataset by Sturm et al. The experimental results show that our proposed reconstruction method solely based on visual odometry and KinectFusion outperforms the state-of-the-art RGB-D SLAM system accuracy. Moreover, our algorithm outputs a ready-to-use polygon mesh (highly suitable for creating 3D virtual worlds) without any postprocessing steps.

Action recognition has been used in a wide range of applications such as human-computer interaction, intelligent video surveillance systems, video summarization, and robotics. Recognizing action is important for intelligent agents to understand, learn and interact with the environment. The recent technology that allows the acquisition of RGB+D and 3D skeleton data and a deep learning model's development significantly increases the action recognition model's performance. In this research, hierarchical Long Sort-Term Memory is proposed to recognize action based on 3D skeleton joints from Kinect sensor. The model uses the 3D axis of skeleton joints and groups each joint in the axis into parts, namely, spine, left and right arm, left and right hand, and left and right leg. To fit the hierarchically structured layers of LSTM, the parts are concatenated into spine, arms, hands, and legs and then concatenated into the body. The model crosses the body in each axis into a single final body and fed to the final layer to classify the action. The performance is measured using cross-view and cross-subject evaluation and achieves accuracy 0.854 and 0.837, respectively, from the 10 action classes of the NTU RGB+D dataset.

In this paper we evaluate a system based on the Microsoft Kinect™ sensor, aimed at the automatic detection of risk postures during human work activities. We first introduce a pick and place task, where three different lateral standing subjects move light cardboard boxes from the various levels of a bookcase to its top, and then putting them back to their original places. They repeat the task over several work cycles and we capture all their natural movements in a continuous way using Kinect, storing the joint positions and the color images. Secondly, from the joint positions, our system detects specific risk postures following the definitions of the Rapid Upper Limb Assessment (RULA) method. Finally, we compare the posture detections by our system with the baseline detections made by a panel of five experts who used the captured color images. In our study we find that the experts have problems to distinguish among some RULA postures during a work cycle because of the narrow detection margin and the difficulty to perceive if a limb reached a certain position; which is particularly true for the cases of wrist and neck. This leads to a larger false positive rate and to a lower general accuracy, with our system detecting postures that experts do not. After applying a ±1° of relaxation to our system, which in negligible for human perception, we are able to reach an accuracy of 0.93 in the comparison with the baseline. Our results show the suitability of Kinect for lateral risk posture detection in pick and place activities.

Physical exercise has many physical, psychological and social health benefits leading to improved life quality. This paper presents a robotic system developed as a personal coach for older adults aiming to motivate older adults to participate in physical activities. The robot instructs the participants, demonstrates the exercises and provides real-time corrective and positive feedback according to the participant’s performance as monitored by an RGB-D camera. Two robotic systems based on two different humanoid robots (Nao, toy-like and Poppy, mechanical-like) were developed and implemented using the Python programming language. Experimental studies with 32 older adults were conducted, to determine the preferable mode and timing of the feedback provided to the user to accommodate user preferences, motivate the users and improve their interaction with the system. Additionally, user preferences with regards to the two different humanoid robots used were explored. The results revealed that the system motivated the older adults to engage more in physical exercises. The type and timing of feedback influenced this engagement. Most of these older adults also perceived the system as very useful, easy to use, had a positive attitude towards the system and noted their intention to use it. Most users preferred the more mechanical looking robot (Poppy) over the toy-like robot (Nao).

Background Emerging sensing and communication technologies are contributing to the development of many motor rehabilitation programs outside the standard healthcare facilities. Nowadays, motor rehabilitation exercises can be easily performed and monitored even at home by a variety of motion-tracking systems. These are cheap, reliable, easy-to-use, and allow also remote configuration and control of the rehabilitation programs. The two most promising technologies for home-based motor rehabilitation programs are inertial wearable sensors and video-based motion capture systems. Methods In this paper, after a thorough review of the relevant literature, an original experimental analysis is reported for two corresponding commercially available solutions, a wearable inertial measurement unit and the Kinect, respectively. For the former, a number of different algorithms for rigid body pose estimation from sensor data were also tested. Both systems were compared with the measurements obtained with state-of-the-art marker-based stereophotogrammetric motion analysis, taken as a gold-standard, and also evaluated outside the lab in a home environment. Results The results in the laboratory setting showed similarly good performance for the elementary large motion exercises, with both systems having errors in the 3–8 degree range. Usability and other possible limitations were also assessed during utilization at home, which revealed additional advantages and drawbacks for the two systems. Conclusions The two evaluated systems use different technology and algorithms, but have similar performance in terms of human motion tracking. Therefore, both can be adopted for monitoring home-based rehabilitation programs, taking adequate precautions however for operation, user instructions and interpretation of the results.

Virtual fitting room (VFR) is a technology that replaces conventional fitting rooms. The VFR is not only available in shops, malls, and any shopping center but also in online stores, which makes VFR technology more and more developed, primarily to support online garment sales. VFR become a trending research interest since Microsoft has developed a Kinect tracking system. In this paper, we proposed the interactive 3D virtual fitting room using Microsoft's Kinect tracking and the rigging technique from 3D Modeling Blender and to implement the VFR. VFR manages the progress of virtual fitting that forms the three-dimensional simulations and visualization of garments on virtual counterparts of the real prospective buyer (user). Users can view the clothing animation on the various poses that are following the user body movements. The system can evaluate the user’s match, guiding them to choose the suitable size of the clothes using Euclidean distance.

Background Abnormalities in gait kinetics in patients with Parkinson’s disease (PD) who have suffer from gait impairment have been noted using a conventional inverse dynamic analysis derived by marker-based motion capture system and force plate, which are typically mounted in the laboratory floor. Despite the high accuracy of this approach in tracking markers’ trajectories and acquiring ground reaction forces (GRFs), its dependence on laboratory-mounted equipment restricts its potential use in wider variety of clinical applications. Research question Would a full-body musculoskeletal model driven by a single depth sensor data only produce comparable gait kinetic parameters, including GRFs and lower extremity joints moments, for elderly participants, both healthy and those diagnosed with PD? Methods Nine patients diagnosed with PD and 11 healthy age-matched control participants performed three over-ground gait trials. Full-body kinematic data were collected using a depth sensor and a musculoskeletal model have been constructed using AnyBody musculoskeletal modeling system to predict the three-dimensional GRFs and lower extremity joint moments. Predicted kinetic parameters for both PD and control groups were compared during the braking and propulsive phases of the gait cycle. In addition, ensemble curve analysis with 90% confidence intervals were constructed to compare between group differences across the stance phase of the gait cycle. Results The findings of this study showed that the PD exhibited a significantly lower braking peak vertical GRF and propulsion peak horizontal GRF while no significant between-group differences were found in peak lower extremity joint moments. However, the PD showed significant alterations in lower extremity joint moments during the early and late phases of stance, which indicate a difference in ambulation strategy. Significance The proposed method adopting full-body musculoskeletal model driven by a depth sensor data proves that it has the potential to be a portable and cost-effective gait analysis tool in the clinical setting.

The article deals with the possibilities of interaction in the virtual environment (VE) of the CAVE LIRKIS system. Commonly available peripherals are keyboard, mouse or gamepad. However, these do not provide a sufficient level of immersion, such as hand gestures. The MYO armband has proven to be one of the potential devices for improving interaction. Therefore, a system for editing an object transformation in VE was created using hand gestures and movements. Enhancing the natural user interface (NUI) of the system with additional interaction capabilities has proven the device to be suitable also for further development

With the development of depth sensors and skeleton tracking algorithms, many skeleton-based pathological gait classification methods have recently been proposed. However, these methods classify only simple gait patterns, and there is no approach to classify complicated gait patterns. In this paper, we classify 1 normal and 5 pathological gaits (antalgic, stiff-legged, lurching, steppage, and Trendelenburg gaits) by using a gated recurrent unit (GRU)-based classifier and 3D skeleton data. We collected skeleton datasets for 1 normal and 5 pathological gaits by using a multiperspective Kinect system. We developed the GRU classifier to classify the pathological gaits and compared its performance with that of other machine learning-based classifiers. Furthermore, we considered various joint groups to identify important and irrelevant joints for pathological gait classification and to improve the performance of the GRU classifier. When all skeleton data are used, the GRU classifier achieves a classification accuracy of 90.13%. A long short-term memory (LSTM)-based classifier achieves the next highest accuracy of 87.25%. The classification accuracy of the GRU classifier depends on the joint groups considered, and the classification accuracy increases to 93.67% when only leg joints are considered. This study indicates that various pathological gaits can be classified by using skeleton data and the GRU classifier. The proposed method can be used to support medical and clinical decisions. Furthermore, the results for various joint groups can be used to develop other methods in cases where only specific joint data are available because of environmental limitations.

In this paper, we demonstrate a perceptual-based 3D skeleton motion data refinement method based on a bidirectional recurrent autoencoder, called BRA-P. Three main technical contributions are made by the proposed network. First, the proposed BRA-P can address noisy data with different noise types and amplitudes using one network, and this attribute makes the approach more suitable for raw motion data with heterogeneous mixed noise. Second, due to the usage of perceptual loss, which measures the difference in high-level features extracted by a pretrained perceptual autoencoder, BRA-P improves the perceptual similarity between refined motion data and clean motion data, especially for the case where the noisy data and target clean data have different topologies. Third, BRA-P further improves the bone-length consistency and smoothness of the refined motion using the perceptual autoencoder as a postprocessing network. Ablation experiments verify the effect of the three technical contributions of our approach. The results of the experiments on synthetic noise data and raw motion data captured by Kinect demonstrate that our method outperforms several state-of-the-art methods in the cleaning of mixed-noise data by one network.

In the field of computer vision, the depth image sequence collected by depth camera is not sensitive to the interference of light, occlusion and background environment. Therefore, in recent years, it is often used to collect behavior data, from which the characteristics of bone joint points are extracted as behavior information. However, it is found that the direct use of joint coordinate information collected by depth camera for behavior recognition is easily affected by individual differences in behavior and changes in shooting distance. Considering the position information of human joints as well as the angle and length information of hidden limbs, based on skeleton data, a behavior feature description method based on the ratio of vector angle and vector mode of human structure is proposed. This method solves the above problems perfectly and obtains ideal results on the self built data set, which is suitable for simple daily behavior recognition.

Low-cost, portable, and easy-to-use Kinect-based systems achieved great popularity in out-of-the-lab motion analysis. The placement of a Kinect sensor significantly influences the accuracy in measuring kinematic parameters for dynamics tasks. We conducted an experiment to investigate the impact of sensor placement on the accuracy of upper limb kinematics during a typical upper limb functional task, the drinking task. Using a 3D motion capture system as the golden standard, we tested twenty-one Kinect positions with three different distances and seven orientations. Upper limb joint angles, including shoulder flexion/extension, shoulder adduction/abduction, shoulder internal/external rotation, and elbow flexion/extension angles, are calculated via our developed Kinect kinematic model and the UWA kinematic model for both the Kinect-based system and the 3D motion capture system. We extracted the angles at the point of the target achieved (PTA). The mean-absolute-error (MEA) with the standard represents the Kinect-based system’s performance. We conducted a two-way repeated measure ANOVA to explore the impacts of distance and orientation on the MEAs for all upper limb angles. There is a significant main effect for orientation. The main effects for distance and the interaction effects do not reach statistical significance. The post hoc test using LSD test for orientation shows that the effect of orientation is joint-dependent and plane-dependent. For a complex task (e.g., drinking), which involves body occlusions, placing a Kinect sensor right in front of a subject is not a good choice. We suggest that place a Kinect sensor at the contralateral side of a subject with the orientation around 30∘ to 45∘ for upper limb functional tasks. For all kinds of dynamic tasks, we put forward the following recommendations for the placement of a Kinect sensor. First, set an optimal sensor position for capture, making sure that all investigated joints are visible during the whole task. Second, sensor placement should avoid body occlusion at the maximum extension. Third, if an optimal location cannot be achieved in an out-of-the-lab environment, researchers could put the Kinect sensor at an optimal orientation by trading off the factor of distance. Last, for those need to assess functions of both limbs, the users can relocate the sensor and re-evaluate the functions of the other side once they finish evaluating functions of one side of a subject.

The Azure Kinect, the successor of Kinect v1 and Kinect v2, is a depth sensor. In this paper we evaluate the skeleton tracking abilities of the new sensor, namely accuracy and precision (repeatability). Firstly, we state the technical features of all three sensors, since we want to put the new Azure Kinect in the context of its previous versions. Then, we present the experimental results of general accuracy and precision obtained by measuring a plate mounted to a robotic manipulator end effector which was moved along the depth axis of each sensor and compare them. In the second experiment, we mounted a human-sized figurine to the end effector and placed it in the same positions as the test plate. Positions were located 400 mm from each other. In each position, we measured relative accuracy and precision (repeatability) of the detected figurine body joints. We compared the results and concluded that the Azure Kinect surpasses its discontinued predecessors, both in accuracy and precision. It is a suitable sensor for human–robot interaction, body-motion analysis, and other gesture-based applications. Our analysis serves as a pilot study for future HMI (human–machine interaction) designs and applications using the new Kinect Azure and puts it in the context of its successful predecessors.

Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection. This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation.

With Microsoft’s motion sensor Kinect and a customized 3-axis pan-tilt-roll machinery connected together, a new intelligent sensing and tracking system comes into being. In order to simulate human natural visual sensing behavior, this system adopts the sensing function from Kinect placed upon the 3-axis motion machinery, controlled by a proposed expert PID control algorithm based on the adaptive Kalman filter, so as to guarantee automatic real-time visual tracking and to observe human movements and receive his/her position information. Experimental results indicate that this new system is capable of tracking one individual’s real-time movements with relatively high accuracy. Though this system is designed to be applied directly to distance education, it has great potential of functioning as a basic platform where many other human-computer natural interactions can be extended.

Submitted by LEONEL CASEIRO MORGADO (leonel.morgado@uab.pt) on 2019-01-31T17:28:30Z No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Approved for entry into archive by LEONEL CASEIRO MORGADO (leonel.morgado@uab.pt) on 2019-01-31T17:28:53Z (GMT) No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Made available in DSpace on 2019-02-04T11:02:07Z (GMT). No. of bitstreams: 1 Tele-Media-Art_-Feasibility-Tests-of-Web-Based-Dance-Education-for-the-Blind-Using-Kinect-and-Sound-Synthesis-of-Motion.pdf: 1556914 bytes, checksum: 1fb30982f1489eaf88c6355c31604a11 (MD5) Previous issue date: 2019 Fundação Calouste Gulbenkian info:eu-repo/semantics/publishedVersion

We describe the implementation of a bespoke two arm hydraulically actuated robotic platform which is used to semi-autonomously cut approximately 50 mm diameter pipes of three different materials: cardboard, ABS plastic and aluminium. The system is designed to be utilised within radiologically active environments where human access is limited due to dose limits and thus remote operation is greatly beneficial. The remotely located operator selects the object from an image via a bespoke algorithm featuring a COTS 3D vision system, along with the desired positions for gripping with one manipulator, and cutting with the other. A pseudo-Jacobian inverse kinematic technique and a programmable automation controller are used to achieve the appropriate joint positions within the dual arm robotic platform. In this article, we present the latest developments to the system and the lessons learnt from the new cutting experiments with a reciprocating saw. A comparison to tele-operated control and manual cutting is also made, with this technique shown to be slower than manual cutting, but faster than pure tele-operational control, where the requirements for highly trained users and operator fatigue are further deleterious factors.

In this paper, some of the distance sensor, including Kinect, Hokuyo UTM-30LX, and RPLidar were observed experimentally. Strengths and weaknesses of each sensor were reviewed so that it can be used as a reference for selecting a suitable sensor for any particular application. A software application has been developed in C programming language as a platform for gathering information for all tested sensors. According to the experiment results, it showed that Hokuyo UTM-30LX results in random normally distributed error on measuring distance with average error 21.94 mm and variance 32.11. On the other hand, error measurement resulted by Kinect and RPLidar strongly depended on measured distance of the object from the sensors, while measurement error resulted by Kinect had a negative correlation with the measured distance and the error resulted by RPLidar sensor had a positive correlation with the measured distance. The performance of these three sensors for detecting a transparent object shows that the Kinect sensors can detect the transparent object on its effective range measurement, Hokuyo UTM-30LX can detect the transparent object in the distance more than equal to 200 mm, and the RPLidar sensor cannot detect the transparent object at all tested distance. Lastly, the experiment shows that the Hokuyo UTM-30LX has the fastest processing time significantly, and the RPLidar has the slowest processing time significantly, while the processing time of Kinect sensor was in between. These processing times were not significantly affected by various tested distance measurement., 10 pages, 15 figures, published in Journal of Mechatronics, Electrical Power, and Vehicular Technology 8 (2017) 85-94

Depth cameras are developing widely. One of their main virtues is that, based on their data and by applying machine learning algorithms and techniques, it is possible to perform body tracking and make an accurate three-dimensional representation of body movement. Specifically, this paper will use the Kinect v2 device, which incorporates a random forest algorithm for 25 joints detection in the human body. However, although Kinect v2 is a powerful tool, there are circumstances in which the device’s design does not allow the extraction of such data or the accuracy of the data is low, as is usually the case with foot position. We propose a method of acquiring this data in circumstances where the Kinect v2 device does not recognize the body when only the lower limbs are visible, improving the ankle angle’s precision employing projection lines. Using a region-based convolutional neural network (Mask RCNN) for body recognition, raw data extraction for automatic ankle angle measurement has been achieved. All angles have been evaluated by inertial measurement units (IMUs) as gold standard. For the six tests carried out at different fixed distances between 0.5 and 4 m to the Kinect, we have obtained (mean ± SD) a Pearson’s coefficient, r = 0.89 ± 0.04, a Spearman’s coefficient, ρ = 0.83 ± 0.09, a root mean square error, RMSE = 10.7 ± 2.6 deg and a mean absolute error, MAE = 7.5 ± 1.8 deg. For the walking test, or variable distance test, we have obtained a Pearson’s coefficient, r = 0.74, a Spearman’s coefficient, ρ = 0.72, an RMSE = 6.4 deg and an MAE = 4.7 deg. This work has been supported by the Spanish Ministry of Science, Innovation and Universities and European Regional Development Fund (ERDF) across projects RTC-2017-6321-1 AEI/FEDER, UE, PID2019-107270RB-C21 AEI/FEDER, UE and FEDER founds.

[EN] Background Accurate assessment of balance and gait is necessary to monitor the clinical progress of Parkinson's disease (PD). Conventional clinical scales can be biased and have limited accuracy. Novel interactive devices are potentially useful to detect subtle posture or gait-related impairments. Methods Posturographic and single and dual-task gait assessments were performed to 54 individuals with PD and 43 healthy controls with the Wii Balance Board and the Kinect v2 and the, respectively. Individuals with PD were also assessed with the Tinetti Performance Oriented Mobility Assessment, the Functional Gait Assessment and the 10-m Walking Test. The influence of demographic and clinical variables on the performance in the instrumented posturographic and gait tests, the sensitivity of these tests to the clinical condition and phenotypes, and their convergent validity with clinical scales were investigated. Results Individuals with PD in H&Y I and I.5 stages showed similar performance to controls. The greatest differences in posture and gait were found between subjects in H&Y II.5 and H&Y I-I.5 stage, as well as controls. Dual-tasking enhanced the differences among all groups in gait parameters. Akinetic/rigid phenotype showed worse postural control and gait than other phenotypes. High significant correlations were found between the limits of stability and most of gait parameters with the clinical scales. Conclusions Low-cost devices showed potential to objectively quantify posture and gait in established PD (H&Y >= II). Dual-tasking gait evaluation was more sensitive to detect differences among PD stages and compared to controls than free gait. Gait and posture were more impaired in akinetic/rigid PD., This study has been funded by project VALORA, Grant 201701-10 of the Fundacio la Marato de la TV3 (Barcelona, Spain) and the European Union through the Operational Program of the European Regional Development Fund (ERDF) of the Valencian Community 2014-2020 (IDIFEDER/2018/029) to RL, and Alter Laboratories SA to PP.

The Sit-to-Stand (STS) is a widely used test of physical function to screen older people at risk of falls and frailty and is also one of the most important components of standard screening for sarcopenia. There have been many recent studies in which instrumented versions of the STS (iSTS) have been developed to provide additional parameters that could improve the accuracy of the STS test. This systematic review aimed to identify whether an iSTS is a viable alternative to a standard STS to identify older people at risk of falling, frailty, and sarcopenia. A total of 856 articles were found using the search strategy developed, with 12 articles retained in the review after screening based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Six studies evaluated the iSTS in fallers, five studies in frailty and only one study in both fallers and frailty. The results showed that power and velocity parameters extracted from an iSTS have the potential to improve the accuracy of screening when compared to a standard STS. Future work should focus on standardizing the segmentation of the STS into phases to enable comparison between studies and to develop devices integrated into the chair used for the test to improve usability.

Using standard digital cameras in combination with deep learning (DL) for pose estimation is promising for the in-home and independent use of exercise games (exergames). We need to investigate to what extent such DL-based systems can provide satisfying accuracy on exergame relevant measures. Our study assesses temporal variation (i.e., variability) in body segment lengths, while using a Deep Learning image processing tool (DeepLabCut, DLC) on two-dimensional (2D) video. This variability is then compared with a gold-standard, marker-based three-dimensional Motion Capturing system (3DMoCap, Qualisys AB), and a 3D RGB-depth camera system (Kinect V2, Microsoft Inc). Simultaneous data were collected from all three systems, while participants (N = 12) played a custom balance training exergame. The pose estimation DLC-model is pre-trained on a large-scale dataset (ImageNet) and optimized with context-specific pose annotated images. Wilcoxon&rsquo, s signed-rank test was performed in order to assess the statistical significance of the differences in variability between systems. The results showed that the DLC method performs comparably to the Kinect and, in some segments, even to the 3DMoCap gold standard system with regard to variability. These results are promising for making exergames more accessible and easier to use, thereby increasing their availability for in-home exercise.

Purpose In endovascular neurosurgery, the operator often acquires three-dimensional (3D) images of the cerebral vessels. Although workstation reoperation is required in some situations during treatment, it leads to time loss because a sterile condition cannot be maintained and treatment must be temporarily interrupted. Therefore, a workstation reoperating system is required while maintaining the desired sterility. Methods A contactless operating interface using Kinect to control 3D images was developed via gesture recognition for endovascular neurosurgery and was applied to a 3D volume rendering technique (VRT) image reconstructed at the workstation. The left-hand movement determines the assigned functions, whereas the right-hand movement is used like a computer mouse to pan and zoom in/out. In addition to the interface, voice commands were used and assigned to digital operations, such as image view changes and mode signal changes. Results This system was used for the actual endovascular treatment of cerebral aneurysms and cerebral arteriovenous malformations. The operator and gesture were recognized without any problems. Using voice operation, it was possible to expeditiously set the VRT image back to the reference angle. Furthermore, it was possible to finely adjust gesture operations, including mouse operation, and treatment was completed while maintaining sterile conditions. Conclusion A contactless operating interface was developed by combining the existing workstation system with Kinect and voice recognition software, allowing surgeons to perform a series of operations, which are normally performed in a console room, while maintaining sterile conditions. Supplementary Information The online version contains supplementary material available at 10.1007/s11548-021-02330-3.

The paper presents a method for the recognition of human actions based on skeletal data. A novel Bone Pair Descriptor is proposed, which encodes the angular relations between pairs of bones. Its features are combined with Distance Descriptor, previously used for hand posture recognition, which describes relationships between distances of skeletal joints. Five different time series classification methods are tested. The selection of features, input joints, and bones is performed. The experiments are conducted using person-independent validation tests and a challenging, publicly available dataset of human actions. The proposed method is compared with other approaches found in the literature achieving relatively good results.

Smart coaching in martial arts is one of the recent research areas in Human Motion Analysis. Numerous moves are performed incorrectly during the performance. In this paper we offer a system that will record the Players’ movements using IR (Infrared) camera sensor, store the data in a database, pre-process the data, classify the data using F-DTW (Fast Dynamic Time Warping) and then show the users an accurate report that contains every movement the player had done, their mistakes and how to improve their performance the next time. This approach focuses on the first seven movements of Karate Kata 1 (Hein Shodan). The system has reached an accuracy of 91.07% in classifying the moves and one common mistake for each move.

Ataxic syndromes include several rare, inherited and acquired conditions. One of the main issues is the absence of specific, and sensitive automatic evaluation tools and digital outcome measures to obtain a continuous monitoring of subjects' motor ability. Gait evaluation was performed by Kinect v2 in a cohort of young participant affected by ataxia syndrome. The dataset is composed of the spatio-temporal parameters calculated by the skeleton acquired by the Kinect sensor, by the diagnosis of each participant, and by the total score of the clinical scale SARA. These parameters have been previously validated and corrected as requested by the Bland-Altman test.

Background: Recent years have witnessed wide applications of exergames to balance training among the older adults. However, research concerning balance training with the use of Kinect for Xbox has remained scarce. While previous studies have shown the positive effects of exergames on improving balance and preventing falling among the older adults, there has been a paucity of empirical evidence supporting the superiority of Kinect exercise to conventional exercise over balance training among the older adults. Therefore, this study aimed to compare the feasibility, safety, and effectiveness of Kinect exercise against conventional exercise over balance training among the community older adults. Method: A total of 20 participants were randomly assigned to the Kinect Exercise Group (N = 10) or the Conventional Exercise Group (N = 10) for a 5-week balance training (45 minutes a time, 2 times a week). Assessor blinding was employed to assess the participants’ performance before and after the treatment, including 30-Second Chair Stand Test (30-sec CST), Timed Up and Go (TUG), Functional Reach Test (FRT), and One-Leg Stance Test (OLST) respectively with eyes open and closed. Subjective feeling of the intensity of pain and side effects were recorded throughout the investigation period. Nonparametric statistics was used for data analysis. Results: Within-group comparison between the pre-test and post-test indicated that significant differences existed in all of the 5 tests (30-sec CST, TUG, FRT, OLST with eyes open, and OLST with eyes closed) in the Kinect exercise group. To the Conventional exercise group, however, significant differences were only observed in 30-sec CST, FRT and OLST with eyes open. With regard to between-group comparison, significant differences were only found in FRT. Conclusion: Such results indicated that both treatments were helpful in improving the participants’ balance performance, that Kinect exercise was more effective in terms of overall balance ability, and that Kinect exercise was particularly beneficial to functional reach enhancement in comparison with traditional exercise. Kinect exercise could be a feasible, safe, and effective alternative for dynamic balance training among older adults.

The heart girth parameter is an important indicator reflecting the growth and development of pigs that provides critical guidance for the optimization of healthy pig breeding. To overcome the heavy workloads and poor adaptability of traditional measurement methods currently used in pig breeding, this paper proposes an automated pig heart girth measurement method using two Kinect depth sensors. First, a two-view pig depth image acquisition platform is established for data collection, the two-view point clouds after preprocessing are registered and fused by feature-based improved 4-Point Congruent Set (4PCS) method. Second, the fused point cloud is pose-normalized, and the axillary contour is used to automatically extract the heart girth measurement point. Finally, this point is taken as the starting point to intercept the circumferential perpendicular to the ground from the pig point cloud, and the complete heart girth point cloud is obtained by mirror symmetry. The heart girth is measured along this point cloud using the shortest path method. Using the proposed method, experiments were conducted on two-view data from 26 live pigs. The results showed that the heart girth measurement absolute errors were all less than 4.19 cm, and the average relative error was 2.14%, which indicating a high accuracy and efficiency of this method.

Since the upper cervical spine (UCS) has been regarded to be distinct from the lower cervical spine (LCS), joint position error (JPE) needs to be tested separately for both regions. The purpose of this study was to investigate the JPE after cervical protraction/retraction movements, involving opposite movements of extension and flexion for the UCS and LCS. These movements are frequently performed during office work. Cervical JPEs were tracked in thirty healthy office workers while performing four tests of cervical pro-retraction movements with variations in vision and movement direction, and assessed using the Kinect head tracker (Microsoft Corp), placed in front of each participant. The JPE was expressed in constant (CE), absolute (AE) and variable errors (VE). Multilevel linear models evaluated main and interaction effects of vision, movement direction, cervical region and sex. Slightly larger JPEs have been found in the UCS. Vision showed no effect on any outcome variable. No effect exceeded typical measurement errors reported for the Kinect head tracker. This study showed, that JPEs after pro-retraction movements of the head and neck may differ for UCS and LCS. The differences were small and not beyond measurement error reported for the Kinect.

We develop a deep learning refined kinematic model for accurately assessing upper limb joint angles using a single Kinect v2 sensor. We train a long short-term memory recurrent neural network using a supervised machine learning architecture to compensate for the systematic error of the Kinect kinematic model, taking a marker-based three-dimensional motion capture system (3DMC) as the golden standard. A series of upper limb functional task experiments were conducted, namely hand to the contralateral shoulder, hand to mouth or drinking, combing hair, and hand to back pocket. Our deep learning-based model significantly improves the performance of a single Kinect v2 sensor for all investigated upper limb joint angles across all functional tasks. Using a single Kinect v2 sensor, our deep learning-based model could measure shoulder and elbow flexion/extension waveforms with mean CMCs &gt, 0.93 for all tasks, shoulder adduction/abduction, and internal/external rotation waveforms with mean CMCs &gt, 0.8 for most of the tasks. The mean deviations of angles at the point of target achieved and range of motion are under 5&deg, for all investigated joint angles during all functional tasks. Compared with the 3DMC, our presented system is easier to operate and needs less laboratory space.

Image segmentation is one of the most important methods for animal phenome research. Since the advent of deep learning, many researchers have looked at multilayer convolutional neural networks to solve the problems of image segmentation. A network simplifies the task of image segmentation with automatic feature extraction. Many networks struggle to output accurate details when dealing with pixel-level segmentation. In this paper, we propose a new concept: Depth density. Based on a depth image, produced by a Kinect system, we design a new function to calculate the depth density value of each pixel and bring this value back to the result of semantic segmentation for improving the accuracy. In the experiment, we choose Simmental cattle as the target of image segmentation and fully convolutional networks (FCN) as the verification networks. We proved that depth density can improve four metrics of semantic segmentation (pixel accuracy, mean accuracy, mean intersection over union, and frequency weight intersection over union) by 2.9%, 0.3%, 11.4%, and 5.02%, respectively. The result shows that depth information produced by Kinect can improve the accuracy of the semantic segmentation of FCN. This provides a new way of analyzing the phenotype information of animals.

International audience; Background: General Movement Assessment (GMA) is a powerful tool to predict Cerebral Palsy (CP). Yet, GMA requires substantial training challenging its broad implementation in clinical routine. This inspired a world-wide quest for automated GMA. Aim: To test whether a low-cost, marker-less system for three-dimensional motion capture from RGB depth sequences using a whole body infant model may serve as the basis for automated GMA.Study design: Clinical case study at an academic neurodevelopmental outpatient clinic.Subjects: Twenty-nine high risk infants were assessed at their clinical follow-up at 2-4 month corrected age (CA). Their neurodevelopmental outcome was assessed regularly up to 12-31 months CA.Outcome measures: GMA according to Hadders-Algra by a masked GMA-expert of conventional and computed 3D body model (“SMIL motion”) videos of the same GMs. Agreement between both GMAs was tested using dichotomous and graded scaling with Kappa and intraclass correlations, respectively. Sensitivity and specificity to predict CP at ≥12 months CA were assessed.Results: Agreement ofthe two GMA ratings was moderate-good for GM-complexity (κ=0.58; ICC=0.874 [95%CI 0.730;0.941]) and substantial-good for fidgety movements (FMs; Kappa=0.78, ICC=0.926[95%CI 0.843;0.965]). Five children were diagnosed with CP (four bilateral, one unilateral CP). The GMs of the child with unilateral CP were twice rated as mildly abnormal with FMs. GM-complexity and somewhat less FMs, of both conventional and SMIL motion videos predicted bilateral CP comparablyto published literature.Conclusions: Our computed infant 3D full body model is an attractive starting point for automated GMA in infants at risk of CP.

Aim: This study was designed to investigate the effects of virtual reality (VR) through Kinect on both gross and fine motor functions and independence in daily living activities in children with unilateral spastic cerebral palsy (USCP). Materials and Methods: This study was designed as a single-blind, randomized, controlled trial. Sixty children with USCP were randomized and split equally between the VR intervention group (10 females and 20 males with a mean age of 10.5 ± 3.62 years) and the traditional occupational therapy (TOT) intervention group (13 females and 17 males with a mean age of 10.06 ± 3.24 years). Both groups were evaluated in terms of motor functioning via the Bruininks-Oseretsky Test of Motor Proficiency-Short Form (BOTMP-SF) and were assessed in accordance with independence in daily activities via the WeeFunctional Independence Measure (WeeFIM). Interventions were conducted for 8 weeks with the main objective of improving motor functions and independence in daily activities. Results: Total motor functions and total independence in daily lives in both groups improved after 8 weeks of intervention. A comparison between groups revealed significantly greater improvements in both gross and fine motor functions and daily activities in the VR group than in the TOT group (P < 0.001). Conclusion: The Kinect-based VR intervention approach is important to improving motor functions and independence in daily activities of children with USCP.

Deep learning is nowadays a mature technique and it is widely applied to image processing and classification. In recent years, many authors tried to extend this approach also to 3D object classification. However, most of the works in this field refers to complete models, while in many real applications the single acquisition with a vision system may only provide a partial object representation. Thus, the main goal of this work is to study the behaviour of classification neural networks when partial 3D models are considered. In particular, the analysis is focused on the classification reliability using partial point clouds, evaluating the influence of noise level and object scaling on the overall network performance. Tests are carried out both on synthetic point clouds, generated by simulation of common acquisition techniques, and on real clouds acquired by a Kinect device. This pushes towards the development of hybrid solutions, where training is made on simulated clouds and the testing takes place on real scanned objects, providing interesting suggestions for practical applications.

Producción Científica, Driver’s gaze information can be crucial in driving research because of its relation to driver attention. Particularly, the inclusion of gaze data in driving simulators broadens the scope of research studies as they can relate drivers’ gaze patterns to their features and performance. In this paper, we present two gaze region estimation modules integrated in a driving simulator. One uses the 3D Kinect device and another uses the virtual reality Oculus Rift device. The modules are able to detect the region, out of seven in which the driving scene was divided, where a driver is gazing at in every route processed frame. Four methods were implemented and compared for gaze estimation, which learn the relation between gaze displacement and head movement. Two are simpler and based on points that try to capture this relation and two are based on classifiers such as MLP and SVM. Experiments were carried out with 12 users that drove on the same scenario twice, each one with a different visualization display, first with a big screen and later with Oculus Rift. On the whole, Oculus Rift outperformed Kinect as the best hardware for gaze estimation. The Oculus-based gaze region estimation method with the highest performance achieved an accuracy of 97.94%. The information provided by the Oculus Rift module enriches the driving simulator data and makes it possible a multimodal driving performance analysis apart from the immersion and realism obtained with the virtual reality experience provided by Oculus., Dirección General de Tráfico y Ministerio del Interior - (Proyecto SPIP2015-01801)

In this paper, we estimate the accuracy of 3D object reconstruction using multiple Kinect sensors. First, we discuss the calibration of multiple Kinect sensors, and provide an analysis of the accuracy and resolution of the depth data. Next, the precision of coordinate mapping between sensors data for registration of depth and color images is evaluated. We test a proposed system for 3D object reconstruction with four Kinect V2 sensors and present reconstruction accuracy results. Experiments and computer simulation are carried out using Matlab and Kinect V2.

(1) Background: Increasing the amount of therapy time has been shown to improve motor function in stroke survivors. However, it is often not possible to increase the amount of therapy time provided in the current one-on-one therapy models. Rehabilitation-based virtual reality exergame systems, such as Jintronix, can be offered to stroke survivors as an adjunct to traditional therapy. The goal of this study was to examine the safety and feasibility of providing additional therapy using an exergame system and assess its preliminary clinical efficacy. (2) Methods: Stroke survivors receiving outpatient rehabilitation services participated in this pilot randomized control trial in which the intervention group received 4 weeks of exergaming sessions in addition to traditional therapy sessions. (3) Results: Nine subjects in the intervention and nine subjects in the control group completed the study. The intervention group had at least two extra sessions per week, with an average duration of 44 min per session and no serious adverse events (falls, dizziness, or pain). The efficacy measures showed statistically meaningful improvements in the activities of daily living measures (i.e., MAL-QOM (motor activity log-quality of movement) and both mobility and physical domains of the SIS (stroke impact scale) with mean difference of 1.0%, 5.5%, and 6.7% between the intervention and control group, respectively) at post-intervention. (4) Conclusion: Using virtual reality exergaming technology as an adjunct to traditional therapy is feasible and safe in post-stroke rehabilitation and may be beneficial to upper extremity functional recovery.

In this study, we compare and verify data that have been rendered visual by Kinect sensoring with data obtained by conventional devices. This is in order to promote the co-creation of experiential values in the acquisition of a sternal compression technique in basic life support in the context of the improvement of the clinical skills of physicians and healthcare professionals. We find that Kinect sensoring is sufficiently accurate to evaluate measurements of the rate and depth of sternal compression; it is comparable to accelerometers and sternal compression-dedicated sensoring devices. This provides a platform for the co-creation of experiential values for the improvement of clinical skills based on the acquisition of medical techniques using the Kinect sensor, which is low in cost and easy to use. It also provides a platform for the exchange of sensor-captured information between the instructor and trainee. It is hoped that this will lead to the co-creation of values useful for the development of educational services in life-saving medical techniques.

A pure GNSS navigation is often unreliable in urban areas because of the presence of obstructions, thus preventing a correct reception of the satellite signal. The bridging between GNSS outages, as well as the vehicle attitude reconstruction, can be recovered by using complementary information, such as visual data acquired by RGB-D or RGB cameras. In this work, the possibility of integrating low-cost GNSS and visual data by means of an extended Kalman filter has been investigated. The focus is on the comparison between the use of RGB-D or RGB cameras. In particular, a Microsoft Kinect device (second generation) and a mirrorless Canon EOS M RGB camera have been compared. The former is an interesting RGB-D camera because of its low-cost, easiness of use and raw data accessibility. The latter has been selected for the high-quality of the acquired images and for the possibility of mounting fixed focal length lenses with a lower weight and cost with respect to a reflex camera. The designed extended Kalman filter takes as input the GNSS-only trajectory and the relative orientation between subsequent pairs of images. Depending on the visual data acquisition system, the filter is different because RGB-D cameras acquire both RGB and depth data, allowing to solve the scale problem, which is instead typical of image-only solutions. The two systems and filtering approaches were assessed by ad-hoc experimental tests, showing that the use of a Kinect device for supporting a u-blox low-cost receiver led to a trajectory with a decimeter accuracy, that is 15 % better than the one obtained when using the Canon EOS M camera.

Purpose To present and evaluate a straightforward implementation of a marker‐less, respiratory motion‐tracking process utilizing Kinect v2 camera as a gating tool during 4DCT or during radiotherapy treatments. Methods Utilizing the depth sensor on the Kinect as well as author written C# code, respiratory motion of a subject was tracked by recording depth values obtained at user selected points on the subject, with each point representing one pixel on the depth image. As a patient breathes, specific anatomical points on the chest/abdomen will move slightly within the depth image across pixels. By tracking how depth values change for a specific pixel, instead of how the anatomical point moves throughout the image, a respiratory trace can be obtained based on changing depth values of the selected pixel. Tracking these values was implemented via marker‐less setup. Varian's RPM system and the Anzai belt system were used in tandem with the Kinect to compare respiratory traces obtained by each using two different subjects. Results Analysis of the depth information from the Kinect for purposes of phase‐ and amplitude‐based binning correlated well with the RPM and Anzai systems. Interquartile Range (IQR) values were obtained comparing times correlated with specific amplitude and phase percentages against each product. The IQR time spans indicated the Kinect would measure specific percentage values within 0.077 s for Subject 1 and 0.164 s for Subject 2 when compared to values obtained with RPM or Anzai. For 4DCT scans, these times correlate to less than 1 mm of couch movement and would create an offset of 1/2 an acquired slice. Conclusion By tracking depth values of user selected pixels within the depth image, rather than tracking specific anatomical locations, respiratory motion can be tracked and visualized utilizing the Kinect with results comparable to that of the Varian RPM and Anzai belt.

Driver decisions and behaviors regarding the surrounding traffic are critical to traffic safety. It is important for an intelligent vehicle to understand driver behavior and assist in driving tasks according to their status. In this paper, the consumer range camera Kinect is used to monitor drivers and identify driving tasks in a real vehicle. Specifically, seven common tasks performed by multiple drivers during driving are identified in this paper. The tasks include normal driving, left-, right-, and rear-mirror checking, mobile phone answering, texting using a mobile phone with one or both hands, and the setup of in-vehicle video devices. The first four tasks are considered safe driving tasks, while the other three tasks are regarded as dangerous and distracting tasks. The driver behavior signals collected from the Kinect consist of a color and depth image of the driver inside the vehicle cabin. In addition, 3-D head rotation angles and the upper body (hand and arm at both sides) joint positions are recorded. Then, the importance of these features for behavior recognition is evaluated using random forests and maximal information coefficient methods. Next, a feedforward neural network (FFNN) is used to identify the seven tasks. Finally, the model performance for task recognition is evaluated with different features (body only, head only, and combined). The final detection result for the seven driving tasks among five participants achieved an average of greater than 80% accuracy, and the FFNN tasks detector is proved to be an efficient model that can be implemented for real-time driver distraction and dangerous behavior recognition.

Lower limb injuries, especially those related to the knee joint, are some of the most common and severe injuries among sport practitioners. Consequently, a growing interest in the identification of subjects with high risk of injury has emerged during last years. One of the most commonly used injury risk factor is the measurement of joint angles during the execution of dynamic movements. To that end, techniques such as human motion capture and video analysis have been widely used. However, traditional procedures to measure joint angles present certain limitations, which makes this practice not practical in common clinical settings. This work presents SPIRA, a novel 2D video analysis system directed to support practitioners during the evaluation of joint angles in functional tests. The system employs an infrared camera to track retro-reflective markers attached to the patient’s body joints and provide a real-time measurement of the joint angles in a cost-and-time-effective way. The information gathered by the sensor is processed and managed through a computer application that guides the expert during the execution of the tests and expedites the analysis of the results. In order to show the potential of the SPIRA system, a case study has been conducted, performing the analysis with the both the proposed system and a gold-standard in 2D offline video analysis. The results (ICC( $$\rho$$ ) = 0.996) reveal a good agreement between both tools and prove the reliability of SPIRA.

This paper proposes a segmentation scheme based on the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color, geometry and surface orientation information. Normalized cuts spectral clustering is then applied in order to recursively segment the scene in two parts thus obtaining an over-segmentation. This procedure is followed by a recursive merging stage where close segments belonging to the same object are joined together. At each step of both procedures a NURBS model is fitted on the computed segments and the accuracy of the fitting is used as a measure of the plausibility that a segment represents a single surface or object. By comparing the accuracy to the one at the previous step, it is possible to determine if each splitting or merging operation leads to a better scene representation and consequently whether to perform it or not. Experimental results show how the proposed method provides an accurate and reliable segmentation.

This paper introduces a field-programmable gate array (FPGA)-based one-shot depth acquisition system, which is a kind of Kinect-like Depth sensing. The proposed implementation integrates the random binary pattern with an improved matching method, which is able to obtain dense and precise depth maps. Scene images captured by a camera are transferred to FPGA (Xilinx Virtex7 XC7VX690T-2FFG1761C) through a CamerLink interface, and the acquired depth results are sent to host computer through a high-speed USB3.0 interface. An elaborate FPGA architecture is designed to realize efficient and accurate calculations. The proposed system outperforms the existing stereo systems, not only on the quality of depth acquisition, but also on the reconstruction speed for which our system achieves 90 fps at a resolution of $1280\times960$ . Besides, Kinect-like depth sensing has an inherent advantage in deployments, much easier than other kinds of systems. Our proposed system has extensive applications not only in the field of vision processing, but also in the fields where block matching is needed.

Deep Learning has become popular among researchers for a long time, and still, new deep convolution neural networks come into the picture very frequently. However, it is challenging to select the best amongst such networks due to their dependence on the tuning of optimization hyperparameters, which is a trivial task. This situation motivates the current study, in which we perform a systematic evaluation and statistical analysis of pre-trained deep models. It is the first comprehensive analysis of pre-trained deep models, gradient-based optimizers and optimization hyperparameters for static Indian sign language recognition. A three-layered CNN model is also proposed and trained from scratch, which attained the best recognition accuracy of 99.0% and 97.6% on numerals and alphabets of a public ISL dataset. Among pre-trained models, ResNet152V2 performed better than other models with a recognition accuracy of 96.2% on numerals and 90.8% on alphabets of the ISL dataset. Our results reinforce the hypothesis for pre-trained deep models that, in general, a pre-trained deep network adequately tuned can yield results way more than the state-of-the-art machine learning techniques without having to train the whole model but only a few top layers for ISL recognition. The effect of hyperparameters like learning rate, batch size and momentum is also analyzed and presented in the paper.

Physical exercise (PE) has become an essential tool for different rehabilitation programs. High-intensity exercises (HIEs) have been demonstrated to provide better results in general health conditions, compared with low and moderate-intensity exercises. In this context, monitoring of a patients’ condition is essential to avoid extreme fatigue conditions, which may cause physical and physiological complications. Different methods have been proposed for fatigue estimation, such as: monitoring the subject’s physiological parameters and subjective scales. However, there is still a need for practical procedures that provide an objective estimation, especially for HIEs. In this work, considering that the sit-to-stand (STS) exercise is one of the most implemented in physical rehabilitation, a computational model for estimating fatigue during this exercise is proposed. A study with 60 healthy volunteers was carried out to obtain a data set to develop and evaluate the proposed model. According to the literature, this model estimates three fatigue conditions (low, moderate, and high) by monitoring 32 STS kinematic features and the heart rate from a set of ambulatory sensors (Kinect and Zephyr sensors). Results show that a random forest model composed of 60 sub-classifiers presented an accuracy of 82.5% in the classification task. Moreover, results suggest that the movement of the upper body part is the most relevant feature for fatigue estimation. Movements of the lower body and the heart rate also contribute to essential information for identifying the fatigue condition. This work presents a promising tool for physical rehabilitation.

Virtual Reality (VR) has recently experienced rapid development for human–computer interactions. Users wearing VR headsets gain an immersive experience when interacting with a 3-dimensional (3D) world. We utilise a light detection and ranging (LiDAR) sensor to detect a 3D point cloud from the real world. To match the scale between a virtual environment and a user’s real world, this paper develops a boundary wall detection method using the Hough transform algorithm. A connected-component-labelling (CCL) algorithm is applied to classify the Hough space into several distinguishable blocks that are segmented using a threshold. The four largest peaks among the segmented blocks are extracted as the parameters of the wall plane. The virtual environment is scaled to the size of the real environment. In order to synchronise the position of the user and his/her avatar in the virtual world, a wireless Kinect network is proposed for user localisation. Multiple Kinects are mounted in an indoor environment to sense the user’s information from different viewpoints. The proposed method supports the omnidirectional detection of the user’s position and gestures. To verify the performance of our proposed system, we developed a VR game using several Kinects and a Samsung Gear VR device.

This paper describes the results of experiments on detection and recognition of 3D objects in RGB-D images provided by the Microsoft Kinect sensor. While the studies focus on single image use, sequences of frames are also considered and evaluated. Observed objects are categorized based on both geometrical and visual cues, but the emphasis is laid on the performance of the point cloud matching method. To this end, a rarely used approach consisting of independent VFH and CRH descriptors matching, followed by ICP and HV algorithms from the Point Cloud Library is applied. Successfully recognized objects are then subjected to a classical 2D analysis based on color histogram comparison exclusively with objects in the same geometrical category. The proposed two-stage approach allows to distinguish objects of similar geometry and different visual appearance, like soda cans of various brands. By separating geometry and color identification phases, the applied system is still able to categorize objects based on their geometry, even if there is no color match. The recognized objects are then localized in the three-dimensional space and autonomously grasped by a manipulator. To evaluate this approach, a special validation set was created, and additionally a selected scene from the Washington RGB-D Object Dataset was used.

Screen climbing games have made a new category of gaming experience between a human climber and a virtual game projected onto an artificial climbing wall. Here, climbing motion recognition is required to interact with the game. In existing climbing games, motion recognition is based on a simple calculation using the depth difference between the climber’s body area and the climbing wall. However, using the body area in this way is devoid of anatomical information; thus the gaming system cannot recognize which part, or parts, of the climber’s body is in contact with the artificial climbing wall. In this paper, we present a climbing motion recognition method using anatomical information obtained by parsing a climber’s body area into its constituent anatomical parts. In ensuring that game events consider anatomical information, a climbing game can provide a more immersive experience for gamers.

The challenge of describing 3D real scenes is tackled in this paper using qualitative spatial descriptors. A key point to study is which qualitative descriptors to use and how these qualitative descriptors must be organized to produce a suitable cognitive explanation. In order to find answers, a survey test was carried out with human participants which openly described a scene containing some pieces of furniture. The data obtained in this survey are analysed, and taking this into account, the QSn3D computational approach was developed which uses a XBox 360 Kinect to obtain 3D data from a real indoor scene. Object features are computed on these 3D data to identify objects in indoor scenes. The object orientation is computed, and qualitative spatial relations between the objects are extracted. These qualitative spatial relations are the input to a grammar which applies saliency rules obtained from the survey study and generates cognitive natural language descriptions of scenes. Moreover, these qualitative descriptors can be expressed as first-order logical facts in Prolog for further reasoning. Finally, a validation study is carried out to test whether the descriptions provided by QSn3D approach are human readable. The obtained results show that their acceptability is higher than 82%.