Search

The body tracking systems on the current market offer a wide range of options for tracking the movements of objects, people, or extremities. The precision of this technology is often limited and determines its field of application. This work aimed to identify relevant technical and environmental factors that influence the performance of body tracking in industrial environments. The influence of light intensity, range of motion, speed of movement and direction of hand movement was analyzed individually and in combination. The hand movement of a test person was recorded with an Azure Kinect at a distance of 1.3 m. The joints in the center of the hand showed the highest accuracy compared to other joints. The best results were achieved at a luminous intensity of 500 lx, and movements in the x-axis direction were more precise than in the other directions. The greatest inaccuracy was found in the z-axis direction. A larger range of motion resulted in higher inaccuracy, with the lowest data scatter at a 100 mm range of motion. No significant difference was found at hand velocity of 370 mm/s, 670 mm/s and 1140 mm/s. This study emphasizes the potential of RGB-D camera technology for gesture control of industrial robots in industrial environments to increase efficiency and ease of use. [ABSTRACT FROM AUTHOR]

Exercise therapy is an important rehabilitation method for knee osteoarthritis (KOA) patients. This paper developed an exercise game(exergame) system based on Azure Kinect to help KOA patients accurately perform exercise prescriptions and improve their compliance with active exercise. The proposed system digitizes the KOA exercise prescription formulated by the physician and designs a personalized exergame, guiding the KOA patients to perform the correct rehabilitation exercise. Besides, the system employs the Azure Kinect to collect the user's main joint position information and assist in recognizing the user's rehabilitation training actions. We recruited 3 rehabilitation physicians and 30 KOA patients (experimental group: 20, control group: 10) to conduct a clinical trial aimed at evaluating the usability of the system and user satisfaction. Based on the data recorded by the system and evaluations conducted by the physicians, we found that the experimental group demonstrated an impressive accuracy rate of 97.47% in executing actions, which was significantly higher than the control group. These results demonstrate the effectiveness of the system in guiding the rehabilitation training of KOA patients. The system achieved an accuracy of 95.42% for action recognition, demonstrating that the exergame is reliably interactive. According to the analysis of the UEQ-S questionnaire, the average satisfaction score of the experimental group with the system was (1.68±0.893). These results indicate that the system is capable of providing an exceptional user experience, ultimately increasing patients' willingness and compliance to engage in active exercises. The above experiments confirmed that the proposed exergame system has high usability and an ideal user experience and is suitable for home active exercise therapy for KOA patients. [ABSTRACT FROM AUTHOR]

Lower back pain, a common issue among pregnant women, often complicates daily activities like standing up from a chair. Therefore, research into the standing motion of pregnant women is important, and many research studies have already been conducted. However, many of these studies were conducted in highly controlled environments, overlooking everyday scenarios such as using a desk for support when standing up, and their effects have not been adequately tested. To address this gap, we measured multimodal signals for a sit-to-stand (STS) movement with hand assistance and verified the changes using a t-test. To avoid imposing strain on pregnant women, we used 10 non-diseased young adults who wore jackets designed to simulate pregnancy conditions, thus allowing for more comprehensive and rigorous experimentation. We attached surface electromyography (sEMG) sensors to the erector spinae muscles of participants and measured changes in muscle activity, skeletal positioning, and center of pressure both before and after wearing a Maternity-Simulation Jacket. Our analysis showed that the jacket successfully mimicked key aspects of the movement patterns typical in pregnant women. These results highlight the possibility of developing practical strategies that more accurately mirror the real-life scenarios met by pregnant women, enriching the current research on their STS movement. [ABSTRACT FROM AUTHOR]

The development of a cost-effective surface scanning system tailored for live animal image capture can play an important role in biomedical research. The primary aim was to introduce a low-cost system, achieving a surface reconstruction error of less than 2mm, and enabling rapid acquisition speeds of approximately 1 second for a complete 360-degree surface capture. Leveraging a five RGB-D camera configuration, our approach offers a simple, low-cost alternative to conventional lab-based 3D scanning setups. Key to our methodology is a novel calibration strategy aimed at refining intrinsic and extrinsic camera parameters simultaneously for improved accuracy. We introduce a novel 3D calibration object, extending existing techniques employing ArUco markers, and implement a depth correction matrix to enhance depth accuracy. By utilizing Simulated Annealing optimization alongside our custom calibration object, we achieve superior results compared to conventional optimization techniques. Our obtained results show that the proposed depth correction method can reduce the reprojection error from 3.12 to 2.89 pixels. Furthermore, despite the simplicity of our reconstruction method, we observe around a 22% improvement in surface reconstruction compared to factory calibration parameters. Our findings underscore the practicality and efficacy of our proposed system, paving the way for enhanced 3D surface reconstruction for real-world surface capture.

The body tracking systems on the current market offer a wide range of options for tracking the movements of objects, people, or extremities. The precision of this technology is often limited and determines its field of application. This work aimed to identify relevant technical and environmental factors that influence the performance of body tracking in industrial environments. The influence of light intensity, range of motion, speed of movement and direction of hand movement was analyzed individually and in combination. The hand movement of a test person was recorded with an Azure Kinect at a distance of 1.3 m. The joints in the center of the hand showed the highest accuracy compared to other joints. The best results were achieved at a luminous intensity of 500 lx, and movements in the x-axis direction were more precise than in the other directions. The greatest inaccuracy was found in the z-axis direction. A larger range of motion resulted in higher inaccuracy, with the lowest data scatter at a 100 mm range of motion. No significant difference was found at hand velocity of 370 mm/s, 670 mm/s and 1140 mm/s. This study emphasizes the potential of RGB-D camera technology for gesture control of industrial robots in industrial environments to increase efficiency and ease of use.

Lower back pain, a common issue among pregnant women, often complicates daily activities like standing up from a chair. Therefore, research into the standing motion of pregnant women is important, and many research studies have already been conducted. However, many of these studies were conducted in highly controlled environments, overlooking everyday scenarios such as using a desk for support when standing up, and their effects have not been adequately tested. To address this gap, we measured multimodal signals for a sit-to-stand (STS) movement with hand assistance and verified the changes using a t-test. To avoid imposing strain on pregnant women, we used 10 non-diseased young adults who wore jackets designed to simulate pregnancy conditions, thus allowing for more comprehensive and rigorous experimentation. We attached surface electromyography (sEMG) sensors to the erector spinae muscles of participants and measured changes in muscle activity, skeletal positioning, and center of pressure both before and after wearing a Maternity-Simulation Jacket. Our analysis showed that the jacket successfully mimicked key aspects of the movement patterns typical in pregnant women. These results highlight the possibility of developing practical strategies that more accurately mirror the real-life scenarios met by pregnant women, enriching the current research on their STS movement.

Human motion tracking is a valuable task for many medical applications. Marker-based optoelectronic systems are considered the gold standard in human motion tracking. However, their use is not always feasible in clinics and industrial environments. On the other hand, marker-less sensors became valuable tools, as they are inexpensive, noninvasive and easy to use. However, their accuracy can depend on many factors including sensor positioning, light conditions and body occlusions. In this study, following previous works on the feasibility of marker-less systems for human motion monitoring, we investigate the performance of the Microsoft Azure Kinect sensor in computing kinematic and dynamic measurements of static postures and dynamic movements. According to our knowledge, it is the first time that this sensor is compared with a Vicon marker-based system to assess the best camera positioning while observing the upper body part movements of people performing several tasks. Twenty-five healthy volunteers were monitored to evaluate the effects of the several testing conditions, including the Azure Kinect positions, the light conditions, and lower limbs occlusions, on the tracking accuracy of kinematic, dynamic, and motor control parameters. From the statistical analysis of the performed measurements, the camera in the frontal position was the most reliable, the lighting conditions had almost no effects on the tracking accuracy, while the lower limbs occlusion worsened the accuracy of the upper limbs. The assessment of human static postures and dynamic movements based on experimental data proves the feasibility of applying the Azure Kinect to the biomechanical monitoring of human motion in several fields.

Reconstructing a three-dimensional (3D) environment is an indispensable technique to make augmented reality and augmented virtuality feasible. A Kinect device is an efficient tool for reconstructing 3D environments, and using multiple Kinect devices enables the enhancement of reconstruction density and expansion of virtual spaces. To employ multiple devices simultaneously, Kinect devices need to be calibrated with respect to each other. There are several schemes available that calibrate 3D images generated from multiple Kinect devices, including the marker detection method. In this study, we introduce a markerless calibration technique for Azure Kinect devices that avoids the drawbacks of marker detection, which directly affects calibration accuracy; it offers superior user friendliness, efficiency, and accuracy. Further, we applied a joint tracking algorithm to approximate the calibration. Traditional methods require the information of multiple joints for calibration; however, Azure Kinect, the latest version of Kinect, requires the information of only one joint. The obtained result was further refined using the iterative closest point algorithm. We conducted several experimental tests that confirmed the enhanced efficiency and accuracy of the proposed method for multiple Kinect devices when compared to the conventional marker-based calibration. [ABSTRACT FROM AUTHOR]

In the era of expanding manned space missions, understanding the biomechanical impacts of zero gravity on human movement is pivotal. This study introduces a novel and cost-effective framework that demonstrates the application of Microsoft’s Azure Kinect body tracking technology as a motion input generator for subsequent OpenSim simulations in weightlessness. Testing rotations, locomotion, coordination, and martial arts movements, we validate the results’ realism under the constraints of angular and linear momentum conservation. While complex, full-body coordination tasks face limitations in a zero gravity environment, our findings suggest possible approaches to device-free exercise routines for astronauts and reveal insights into the feasibility of hand-to-hand combat in space. However, some challenges remain in distinguishing zero gravity effects in the simulations from discrepancies in the captured motion input or forward dynamics calculations, making a comprehensive validation difficult. The paper concludes by highlighting the framework’s practical potential for the future of space mission planning and related research endeavors, while also providing recommendations for further refinement.

The study of visuomotor adaptation (VMA) capabilities has been encompassed in various experimental protocols aimed at investigating human motor control strategies and/or cognitive functions. VMA-oriented frameworks can have clinical applications, primarily in the investigation and assessment of neuromotor impairments caused by conditions such as Parkinson's disease or post-stroke, which affect the lives of tens of thousands of people worldwide. Therefore, they can enhance the understanding of the specific mechanisms of such neuromotor disorders, thus being a potential biomarker for recovery, with the aim of being integrated with conventional rehabilitative programs. Virtual Reality (VR) can be entailed in a framework targeting VMA since it allows the development of visual perturbations in a more customizable and realistic way. Moreover, as has been demonstrated in previous works, a serious game (SG) can further increase engagement thanks to the use of full-body embodied avatars. Most studies implementing VMA frameworks have focused on upper limb tasks and have utilized a cursor as visual feedback for the user. Hence, there is a paucity in the literature about VMA-oriented frameworks targeting locomotion tasks. In this article, the authors present the design, development, and testing of an SG-based framework that addresses VMA in a locomotion activity by controlling a full-body moving avatar in a custom VR environment. This workflow includes a set of metrics to quantitatively assess the participants' performance. Thirteen healthy children were recruited to evaluate the framework. Several quantitative comparisons and analyses were run to validate the different types of introduced visuomotor perturbations and to evaluate the ability of the proposed metrics to describe the difficulty caused by such perturbations. During the experimental sessions, it emerged that the system is safe, easy to use, and practical in a clinical setting. Despite the limited sample size, which represents the main limitation of the study and can be compensated for with future recruitment, the authors claim the potential of this framework as a useful instrument for quantitatively assessing either motor or cognitive impairments. The proposed feature-based approach gives several objective parameters as additional biomarkers that can integrate the conventional clinical scores. Future studies might investigate the relation between the proposed biomarkers and the clinical scores for specific disorders such as Parkinson's disease and cerebral palsy. [ABSTRACT FROM AUTHOR]

The intelligent training and assessment of gymnastics movements require studying motion trajectory and reconstructing the character animation. Microsoft Kinect has been widely used due to its advantages of low price and high frame rate. However, its optical characteristics are inevitably affected by illumination and occlusion. It is necessary to reduce data noise via specific algorithms. Most of the existing research focuses on local motion but lacks consideration of the whole human skeleton. Based on the analysis of the spatial characteristics of gymnastics and the movement principle of the human body, this paper proposes a dynamic and static two-dimensional regression compensation algorithm. Firstly, the constraint characteristics of human skeleton motion were analyzed, and the maximum constraint table and Mesh Collider were established. Then, the dynamic acceleration of skeleton motion and the spatial characteristics of static limb motion were calculated based on the data of adjacent effective skeleton frames before and after the collision. Finally, using the least squares polynomial fitting to compensate and correct the lost skeleton coordinate data, it realizes the smoothness and rationality of human skeleton animation. The results of two experiments showed that the solution of the skeleton point solved the problem caused by data loss due to the Kinect optical occlusion. The data compensation time of an effective block skeleton point can reach 180 ms, with an average error of about 0.1 mm, which shows a better data compensation effect of motion data acquisition and animation reconstruction.

Introduction: Measuring kinematic behavior during robot-assisted gait therapy requires either laborious set up of a marker-based motion capture system or relies on the internal sensors of devices that may not cover all relevant degrees of freedom. This presents a major barrier for the adoption of kinematic measurements in the normal clinical schedule. However, to advance the field of robot-assisted therapy many insights could be gained from evaluating patient behavior during regular therapies.Methods: For this reason, we recently developed and validated a method for extracting kinematics from recordings of a low-cost RGB-D sensor, which relies on a virtual 3D body model to estimate the patient’s body shape and pose in each frame. The present study aimed to evaluate the robustness of the method to the presence of a lower limb exoskeleton. 10 healthy children without gait impairment walked on a treadmill with and without wearing the exoskeleton to evaluate the estimated body shape, and 8 custom stickers were placed on the body to evaluate the accuracy of estimated poses.Results & Conclusion: We found that the shape is generally robust to wearing the exoskeleton, and systematic pose tracking errors were around 5 mm. Therefore, the method can be a valuable measurement tool for the clinical evaluation, e.g., to measure compensatory movements of the trunk.

The Azure Kinect DK is an RGB-D-camera popular in research and studies with humans. For good scientific practice, it is relevant that Azure Kinect yields consistent and reproducible results. We noticed the yielded results were inconsistent. Therefore, we examined 100 body tracking runs per processing mode provided by the Azure Kinect Body Tracking SDK on two different computers using a prerecorded video. We compared those runs with respect to spatiotemporal progression (spatial distribution of joint positions per processing mode and run), derived parameters (bone length), and differences between the computers. We found a previously undocumented converging behavior of joint positions at the start of the body tracking. Euclidean distances of joint positions varied clinically relevantly with up to 87 mm between runs for CUDA and TensorRT; CPU and DirectML had no differences on the same computer. Additionally, we found noticeable differences between two computers. Therefore, we recommend choosing the processing mode carefully, reporting the processing mode, and performing all analyses on the same computer to ensure reproducible results when using Azure Kinect and its body tracking in research. Consequently, results from previous studies with Azure Kinect should be reevaluated, and until then, their findings should be interpreted with caution. [ABSTRACT FROM AUTHOR]

Time-of-flight cameras are widely adopted in a variety of indoor applications ranging from industrial object measurement to human activity recognition. However, the available products may differ in terms of the quality of the acquired point cloud, and the datasheet provided by the constructors may not be enough to guide researchers in the choice of the perfect device for their application. Hence, this work details the experimental procedure to assess time-of-flight cameras' error sources that should be considered when designing an application involving time-of-flight technology, such as the bias correction and the temperature influence on the point cloud stability. This is the first step towards a standardization of the metrological characterization procedure that could ensure the robustness and comparability of the results among tests and different devices. The procedure was conducted on Kinect Azure, Basler Blaze 101, and Basler ToF 640 cameras. Moreover, we compared the devices in the task of 3D reconstruction following a procedure involving the measure of both an object and a human upper-body-shaped mannequin. The experiment highlighted that, despite the results of the previously conducted metrological characterization, some devices showed evident difficulties in reconstructing the target objects. Thus, we proved that performing a rigorous evaluation procedure similar to the one proposed in this paper is always necessary when choosing the right device. [ABSTRACT FROM AUTHOR]

With the development of 3D vision techniques, RGB-D cameras are increasingly used to allow easier and cheaper access to the third dimension. In this paper, we focus on testing the potential of the Kinect Azure RGB-D camera in the 3D reconstruction of indoor scenes. First, a series of investigations of the hardware was performed to evaluate its accuracy and precision. The results show that the measurements made with the Azure could be exploited for close-range survey applications. Second, we performed a methodological workflow for indoor reconstruction based on the Open3D framework, which was applied to two different indoor scenes. Based on the results, we can state that the quality of 3D reconstruction significantly depends on the architecture of the captured scene. This was supported by a comparison of the point cloud from the Kinect Azure with that from a terrestrial laser scanner and another from a mobile laser scanner. The results show that the average differences do not exceed 8 mm, which confirms that the Kinect Azure can be considered a 3D measurement system at least as reliable as a mobile laser scanner. [ABSTRACT FROM AUTHOR]

In response to the shortage, uneven distribution, and high cost of rehabilitation resources in the context of the COVID-19 pandemic, we developed a low-cost, easy-to-use remote rehabilitation system that allows patients to perform rehabilitation training and receive real-time guidance from doctors at home. The proposed system uses Azure Kinect to capture motions with an error of just 3% compared to professional motion capture systems. In addition, the system provides an automatic evaluation function of rehabilitation training, including evaluation of motion angles and trajectories. After acquiring the user's 3D motions, the system synchronizes the 3D motions to the virtual human body model in Unity with an average error of less than 1%, which gives the user a more intuitive and interactive experience. After a series of evaluation experiments, we verified the usability, convenience, and high accuracy of the system, finally concluding that the system can be used in practical rehabilitation applications. [ABSTRACT FROM AUTHOR]

RGB and depth cameras are extensively used for the 3D tracking of human pose and motion. Typically, these cameras calculate a set of 3D points representing the human body as a skeletal structure. The tracking capabilities of a single camera are often affected by noise and inaccuracies due to occluded body parts. Multiple-camera setups offer a solution to maximize coverage of the captured human body and to minimize occlusions. According to best practices, fusing information across multiple cameras typically requires spatio-temporal calibration. First, the cameras must synchronize their internal clocks. This is typically performed by physically connecting the cameras to each other using an external device or cable. Second, the pose of each camera relative to the other cameras must be calculated (Extrinsic Calibration). The state-of-the-art methods use specialized calibration session and devices such as a checkerboard to perform calibration. In this paper, we introduce an approach to the spatio-temporal calibration of multiple cameras which is designed to run on-the-fly without specialized devices or equipment requiring only the motion of the human body in the scene. As an example, the system is implemented and evaluated using Microsoft Azure Kinect. The study shows that the accuracy and robustness of this approach is on par with the state-of-the-art practices. [ABSTRACT FROM AUTHOR]

Motor impairments are among the most relevant, evident, and disabling symptoms of Parkinson's disease that adversely affect quality of life, resulting in limited autonomy, independence, and safety. Recent studies have demonstrated the benefits of physiotherapy and rehabilitation programs specifically targeted to the needs of Parkinsonian patients in supporting drug treatments and improving motor control and coordination. However, due to the expected increase in patients in the coming years, traditional rehabilitation pathways in healthcare facilities could become unsustainable. Consequently, new strategies are needed, in which technologies play a key role in enabling more frequent, comprehensive, and out-of-hospital follow-up. The paper proposes a vision-based solution using the new Azure Kinect DK sensor to implement an integrated approach for remote assessment, monitoring, and rehabilitation of Parkinsonian patients, exploiting non-invasive 3D tracking of body movements to objectively and automatically characterize both standard evaluative motor tasks and virtual exergames. An experimental test involving 20 parkinsonian subjects and 15 healthy controls was organized. Preliminary results show the system's ability to quantify specific and statistically significant (p < 0.05) features of motor performance, easily monitor changes as the disease progresses over time, and at the same time permit the use of exergames in virtual reality both for training and as a support for motor condition assessment (for example, detecting an average reduction in arm swing asymmetry of about 14% after arm training). The main innovation relies precisely on the integration of evaluative and rehabilitative aspects, which could be used as a closed loop to design new protocols for remote management of patients tailored to their actual conditions. [ABSTRACT FROM AUTHOR]

Accurate 3D tracking of hand and fingers movements poses significant challenges in computer vision. The potential applications span across multiple domains, including human–computer interaction, virtual reality, industry, and medicine. While gesture recognition has achieved remarkable accuracy, quantifying fine movements remains a hurdle, particularly in clinical applications where the assessment of hand dysfunctions and rehabilitation training outcomes necessitate precise measurements. Several novel and lightweight frameworks based on Deep Learning have emerged to address this issue; however, their performance in accurately and reliably measuring finger movements requires validation against well-established gold standard systems. In this paper, the aim is to validate the hand-tracking framework implemented by Google MediaPipe Hand (GMH) and an innovative enhanced version, GMH-D, that exploits the depth estimation of an RGB-Depth camera to achieve more accurate tracking of 3D movements. Three dynamic exercises commonly administered by clinicians to assess hand dysfunctions, namely hand opening–closing, single finger tapping and multiple finger tapping are considered. Results demonstrate high temporal and spectral consistency of both frameworks with the gold standard. However, the enhanced GMH-D framework exhibits superior accuracy in spatial measurements compared to the baseline GMH, for both slow and fast movements. Overall, our study contributes to the advancement of hand tracking technology, and the establishment of a validation procedure as a good-practice to prove efficacy of deep-learning-based hand-tracking. Moreover, it proves that GMH-D is a reliable framework for assessing 3D hand movements in clinical applications. • Hands are vital for carrying out daily-life tasks without external aid. • Measuring objectively hand dexterity can support new digital health treatments. • Deep Learning allows for video-based hand tracking of fine motor tasks. • We validate accuracy of Google Mediapipe Hand and GMH-D against motion capture. • Both frameworks achieve good-to-high spatial and temporal hand tracking performance. [ABSTRACT FROM AUTHOR]

Arm swinging is a typical feature of human walking: Continuous and rhythmic movement of the upper limbs is important to ensure postural stability and walking efficiency. However, several factors can interfere with arm swings, making walking more risky and unstable: These include aging, neurological diseases, hemiplegia, and other comorbidities that affect motor control and coordination. Objective assessment of arm swings during walking could play a role in preventing adverse consequences, allowing appropriate treatments and rehabilitation protocols to be activated for recovery and improvement. This paper presents a system for gait analysis based on Microsoft Azure Kinect DK sensor and its body-tracking algorithm: It allows noninvasive full-body tracking, thus enabling simultaneous analysis of different aspects of walking, including arm swing characteristics. Sixteen subjects with Parkinson's disease and 13 healthy controls were recruited with the aim of evaluating differences in arm swing features and correlating them with traditional gait parameters. Preliminary results show significant differences between the two groups and a strong correlation between the parameters. The study thus highlights the ability of the proposed system to quantify arm swing features, thus offering a simple tool to provide a more comprehensive gait assessment. [ABSTRACT FROM AUTHOR]

The study of visuomotor adaptation (VMA) capabilities has been encompassed in various experimental protocols aimed at investigating human motor control strategies and/or cognitive functions. VMA-oriented frameworks can have clinical applications, primarily in the investigation and assessment of neuromotor impairments caused by conditions such as Parkinson’s disease or post-stroke, which affect the lives of tens of thousands of people worldwide. Therefore, they can enhance the understanding of the specific mechanisms of such neuromotor disorders, thus being a potential biomarker for recovery, with the aim of being integrated with conventional rehabilitative programs. Virtual Reality (VR) can be entailed in a framework targeting VMA since it allows the development of visual perturbations in a more customizable and realistic way. Moreover, as has been demonstrated in previous works, a serious game (SG) can further increase engagement thanks to the use of full-body embodied avatars. Most studies implementing VMA frameworks have focused on upper limb tasks and have utilized a cursor as visual feedback for the user. Hence, there is a paucity in the literature about VMA-oriented frameworks targeting locomotion tasks. In this article, the authors present the design, development, and testing of an SG-based framework that addresses VMA in a locomotion activity by controlling a full-body moving avatar in a custom VR environment. This workflow includes a set of metrics to quantitatively assess the participants’ performance. Thirteen healthy children were recruited to evaluate the framework. Several quantitative comparisons and analyses were run to validate the different types of introduced visuomotor perturbations and to evaluate the ability of the proposed metrics to describe the difficulty caused by such perturbations. During the experimental sessions, it emerged that the system is safe, easy to use, and practical in a clinical setting. Despite the limited sample size, which represents the main limitation of the study and can be compensated for with future recruitment, the authors claim the potential of this framework as a useful instrument for quantitatively assessing either motor or cognitive impairments. The proposed feature-based approach gives several objective parameters as additional biomarkers that can integrate the conventional clinical scores. Future studies might investigate the relation between the proposed biomarkers and the clinical scores for specific disorders such as Parkinson’s disease and cerebral palsy.

Nowadays, the need for reliable and low-cost multi-camera systems is increasing for many potential applications, such as localization and mapping, human activity recognition, hand and gesture analysis, and object detection and localization. However, a precise camera calibration approach is mandatory for enabling further applications that require high precision. This paper analyzes the available two-camera calibration approaches to propose a guideline for calibrating multiple Azure Kinect RGB-D sensors to achieve the best alignment of point clouds in both color and infrared resolutions, and skeletal joints returned by the Microsoft Azure Body Tracking library. Different calibration methodologies using 2D and 3D approaches, all exploiting the functionalities within the Azure Kinect devices, are presented. Experiments demonstrate that the best results are returned by applying 3D calibration procedures, which give an average distance between all couples of corresponding points of point clouds in color or an infrared resolution of 21.426 mm and 9.872 mm for a static experiment and of 20.868 mm and 7.429 mm while framing a dynamic scene. At the same time, the best results in body joint alignment are achieved by three-dimensional procedures on images captured by the infrared sensors, resulting in an average error of 35.410 mm. [ABSTRACT FROM AUTHOR]

Background: Instrumented measurement of spatiotemporal parameters during walking can provide valuable information on an individual's overall function and health. Efficient, inexpensive, and accurate measurement of overground walking spatiotemporal parameters would be a critical component of providing point-of-care assessments of gait function, concussion recovery, fall-risk, and cognitive decline. Depth cameras combined with skeleton pose tracking algorithms, such as the Microsoft Kinect with body tracking software, have been used to measure walking spatiotemporal parameters. However, the ability of the latest generation Microsoft Kinect sensor, the Azure Kinect, to accurately measure overground walking spatiotemporal parameters has not been evaluated in the literature.Research Question: The purpose of this work was to compare overground walking spatiotemporal parameters measurements from a 12 camera Vicon optical motion capture system to measurements of a single Azure Kinect with body tracking SDK (software development kit).Methods: Spatiotemporal parameters of overground walking were simultaneously collected on twenty young healthy participants. Stride length, stride time, step length and step width were derived from ankle joint center locations and measurements from the two instruments were compared using descriptive statistics, scatter plots, Pearson correlation analyses, and Bland-Altman analyses.Results: Pearson correlation coefficients were greater than 0.87 for all spatiotemporal parameters with most parameters demonstrating very strong (> 0.9) agreement. The mean of the differences for stride length between measurements was 35.6 mm for the left limb and 39.1 mm for the right limb, both of which are less than 3% of average stride length. Mean of the differences for step width and stride time were less than 2% and 1% of their averages respectively.Significance: A single Microsoft Azure Kinect with body tracking SDK can provide clinically relevant measurement of walking spatiotemporal parameters, providing accessible and objective measurements that can improve clinical decision making across a variety of patient populations. [ABSTRACT FROM AUTHOR]

Abnormal movement of the head and neck is a typical symptom of Cervical Dystonia (CD). Accurate scoring on the severity scale is of great significance for treatment planning. The traditional scoring method is to use a protractor or contact sensors to calculate the angle of the movement, but this method is time-consuming, and it will interfere with the movement of the patient. In the recent outbreak of the coronavirus disease, the need for remote diagnosis and treatment of CD has become extremely urgent for clinical practice. To solve these problems, we propose a multi-view vision based CD severity scale scoring method, which detects the keypoint positions of the patient from the frontal and lateral images, and finally scores the severity scale by calculating head and neck motion angles. We compared the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS) subscale scores calculated by our vision based method with the scores calculated by a neurologist trained in dyskinesia. An analysis of the correlation coefficient was then conducted. Intra-class correlation (ICC)(3,1) was used to measure absolute accuracy. Our multi-view vision based CD severity scale scoring method demonstrated sufficient validity and reliability. This low-cost and contactless method provides a new potential tool for remote diagnosis and treatment of CD. [ABSTRACT FROM AUTHOR]

Analysing the dynamics in social interactions in indoor spaces entails evaluating spatial–temporal variables from the event, such as location and time. Additionally, social interactions include invisible spaces that we unconsciously acknowledge due to social constraints, e.g., space between people having a conversation with each other. Nevertheless, current sensor arrays focus on detecting the physically occupied spaces from social interactions, i.e., areas inhabited by physically measurable objects. Our goal is to detect the socially occupied spaces, i.e., spaces not physically occupied by subjects and objects but inhabited by the interaction they sustain. We evaluate the social representation of the space structure between two or more active participants, so-called F-Formation for small gatherings. We propose calculating body orientation and location from skeleton joint data sets by integrating depth cameras. The body orientation is derived by integrating the shoulders and spine joint data with head/face rotation data and spatial–temporal information from trajectories. From the physically occupied measurements, we can detect socially occupied spaces. In our user study implementing the system, we compared the capabilities and skeleton tracking datasets from three depth camera sensors, the Kinect v2, Azure Kinect, and Zed 2i. We collected 32 walking patterns for individual and dyad configurations and evaluated the system's accuracy regarding the intended and socially accepted orientations. Experimental results show accuracy above 90% for the Kinect v2, 96% for the Azure Kinect, and 89% for the Zed 2i for assessing socially relevant body orientation. Our algorithm contributes to the anonymous and automated assessment of socially occupied spaces. The depth sensor system is promising in detecting more complex social structures. These findings impact research areas that study group interactions within complex indoor settings. [ABSTRACT FROM AUTHOR]

Background: The sit-to-stand test (STS) is commonly used to evaluate functional capabilities within a variety of clinical populations. Traditionally STS is a timed test, limiting the depth of information which can be gained from its evaluation. The Azure Kinect has the potential to add in-depth analysis to STS. Despite these potential benefits, the recently released (2019) Azure Kinect has yet to be evaluated for its ability to accurately assess STS.Research Questions: Purposes of this work were to compare data captured during STS using both a 12 camera Vicon motion capture system and the Azure Kinect; and to calculate kinematic and spatiotemporal variables related to the four phases of the STS cycle.Methods: Spatiotemporal and kinematic measures for STS were simultaneously collected by both devices for 15 participants. Cycle waveforms were compared for right and left hip and knee flexion/extension angular displacement, right and left hip and knee flexion/extension angular velocity, and knee-to-ankle separation ratio. Evaluated discrete outcome variables included: phase time points, maximum knee extension velocity from phases 3 to 4, medial-lateral pelvic sway range, and total time to completion. Waveform summary data were compared using R, R2, and RMSE. Discrete variables were analyzed using Spearman's Rank correlation coefficient.Results: R and R2 values between the two systems indicated high levels of correlation (all R values > 0.711, all R2 values > 0.660). Although there was an overall high level of agreement between waveform shapes, high RMSE values indicated some minor tracking errors for Kinect within the STS cycle. Spearman's Rank correlation coefficient indicated high levels of correlation between the systems for discrete variables (all R values > 0.89), with the exception of medial-lateral pelvic sway range.Significance: The Azure Kinect provides valuable insight into STS movement strategies allowing for improved precision in clinical decision making across multiple clinical populations. [ABSTRACT FROM AUTHOR]

A single depth camera provides a fast and easy approach to performing biomechanical assessments in a clinical setting; however, there are currently no established methods to reliably determine joint angles from these devices. The primary aim of this study was to compare joint angles as well as the between-day reliability of direct kinematics to model-constrained inverse kinematics recorded using a single markerless depth camera during a range of clinical and athletic movement assessments.A secondary aim was to determine the minimum number of trials required to maximize reliability. Eighteen healthy participants attended two testing sessions one week apart. Tasks included treadmill walking, treadmill running, single-leg squats, single-leg countermovement jumps, bilateral countermovement jumps, and drop vertical jumps. Keypoint data were processed using direct kinematics as well as in OpenSim using a full-body musculoskeletal model and inverse kinematics. Kinematic methods were compared using statistical parametric mapping and between-day reliability was calculated using intraclass correlation coefficients, mean absolute error, and minimal detectable change. Keypoint-derived inverse kinematics resulted in significantly smaller hip flexion (range = −9 to −2°), hip abduction (range = −3 to −2°), knee flexion (range = −5° to −2°), and greater dorsiflexion angles (range = 6–15°) than direct kinematics. Both markerless kinematic methods had high between-day reliability (inverse kinematics ICC 95 %CI = 0.83–0.90; direct kinematics ICC 95 %CI = 0.80–0.93). For certain tasks and joints, keypoint-derived inverse kinematics resulted in greater reliability (up to 0.47 ICC) and smaller minimal detectable changes (up to 13°) than direct kinematics. Performing 2–4 trials was sufficient to maximize reliability for most tasks. A single markerless depth camera can reliably measure lower limb joint angles, and skeletal model-constrained inverse kinematics improves lower limb joint angle reliability for certain tasks and joints. [ABSTRACT FROM AUTHOR]

The Azure Kinect represents the latest generation of Microsoft Kinect depth cameras. Of interest in this article is the depth and spatial accuracy of the Azure Kinect and how it compares to its predecessor, the Kinect v2. In one experiment, the two sensors are used to capture a planar whiteboard at 15 locations in a grid pattern with laser scanner data serving as ground truth. A set of histograms reveals the temporal-based random depth error inherent in each Kinect. Additionally, a two-dimensional cone of accuracy illustrates the systematic spatial error. At distances greater than 2.5 m, we find the Azure Kinect to have improved accuracy in both spatial and temporal domains as compared to the Kinect v2, while for distances less than 2.5 m, the spatial and temporal accuracies were found to be comparable. In another experiment, we compare the distribution of random depth error between each Kinect sensor by capturing a flat wall across the field of view in horizontal and vertical directions. We find the Azure Kinect to have improved temporal accuracy over the Kinect v2 in the range of 2.5 to 3.5 m for measurements close to the optical axis. The results indicate that the Azure Kinect is a suitable substitute for Kinect v2 in 3D scanning applications. [ABSTRACT FROM AUTHOR]

Action Recognition (AR) has become a popular approach to ensure efficient and safe Human-Robot Collaboration. Current research approaches are mostly optimized for specific assembly processes and settings. This paper introduces a novel approach to extend the field of AR to multi-variant assembly processes. The approach is based on generalized action primitives derived from Methods-Time-Measurement (MTM) analysis that are detected by an AR system using skeletal data. Subsequently a search algorithm combines the information from AR and MTM to provide an estimate of the assembly progress. One possible implementation is shown in a proof of concept and results as well as future work are discussed. [ABSTRACT FROM AUTHOR]

The Azure Kinect DK is an RGB-D-camera popular in research and studies with humans. For good scientific practice, it is relevant that Azure Kinect yields consistent and reproducible results. We noticed the yielded results were inconsistent. Therefore, we examined 100 body tracking runs per processing mode provided by the Azure Kinect Body Tracking SDK on two different computers using a prerecorded video. We compared those runs with respect to spatiotemporal progression (spatial distribution of joint positions per processing mode and run), derived parameters (bone length), and differences between the computers. We found a previously undocumented converging behavior of joint positions at the start of the body tracking. Euclidean distances of joint positions varied clinically relevantly with up to 87 mm between runs for CUDA and TensorRT; CPU and DirectML had no differences on the same computer. Additionally, we found noticeable differences between two computers. Therefore, we recommend choosing the processing mode carefully, reporting the processing mode, and performing all analyses on the same computer to ensure reproducible results when using Azure Kinect and its body tracking in research. Consequently, results from previous studies with Azure Kinect should be reevaluated, and until then, their findings should be interpreted with caution.

Time-of-flight cameras are widely adopted in a variety of indoor applications ranging from industrial object measurement to human activity recognition. However, the available products may differ in terms of the quality of the acquired point cloud, and the datasheet provided by the constructors may not be enough to guide researchers in the choice of the perfect device for their application. Hence, this work details the experimental procedure to assess time-of-flight cameras’ error sources that should be considered when designing an application involving time-of-flight technology, such as the bias correction and the temperature influence on the point cloud stability. This is the first step towards a standardization of the metrological characterization procedure that could ensure the robustness and comparability of the results among tests and different devices. The procedure was conducted on Kinect Azure, Basler Blaze 101, and Basler ToF 640 cameras. Moreover, we compared the devices in the task of 3D reconstruction following a procedure involving the measure of both an object and a human upper-body-shaped mannequin. The experiment highlighted that, despite the results of the previously conducted metrological characterization, some devices showed evident difficulties in reconstructing the target objects. Thus, we proved that performing a rigorous evaluation procedure similar to the one proposed in this paper is always necessary when choosing the right device.

Automatic pose estimation has become a valuable tool for the study of human behavior, including dyadic interactions. It allows researchers to analyze the nuanced dynamics of interactions more effectively, and facilitates the integration of behavioral data with other modalities (EEG, etc.). However, many technical difficulties remain. Particularly, for parent-infant interactions, automatic pose estimation for infants is unpredictable; the immature proportions and smaller bodies of children may cause misdetections. OpenPose is one tool that has shown high performance in pose tracking from video, even in infants. However, OpenPose is limited to 2D (i.e., coordinates relative to the image space). This may be undesirable in a multitude of paradigms (e.g., naturalistic settings). We developed a method for expanding the functionality of OpenPose to 3D, tailored to parent-infant interaction paradigms. This method merges the estimations from OpenPose with the depth information from a depth camera to obtain a 3D pose that works even for young infants.•Video recordings of interactions of parents and infants are taken using a dual color-depth camera.•2D-positions of parents and their infants are estimated from the color video.•Using the depth camera, we transform the 2D estimations into real-world 3D positions, allowing movement analysis in full-3D space., Competing Interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (© 2024 The Authors.)

With the development of 3D vision techniques, RGB-D cameras are increasingly used to allow easier and cheaper access to the third dimension. In this paper, we focus on testing the potential of the Kinect Azure RGB-D camera in the 3D reconstruction of indoor scenes. First, a series of investigations of the hardware was performed to evaluate its accuracy and precision. The results show that the measurements made with the Azure could be exploited for close-range survey applications. Second, we performed a methodological workflow for indoor reconstruction based on the Open3D framework, which was applied to two different indoor scenes. Based on the results, we can state that the quality of 3D reconstruction significantly depends on the architecture of the captured scene. This was supported by a comparison of the point cloud from the Kinect Azure with that from a terrestrial laser scanner and another from a mobile laser scanner. The results show that the average differences do not exceed 8 mm, which confirms that the Kinect Azure can be considered a 3D measurement system at least as reliable as a mobile laser scanner.

We have been developing learning support systems for sketching human body. A learning support system compares an artwork by learners with a model of the human body figure in regard to a skeleton or an outline and presents the difference between them and scores. The system in the previous research had the problem that learners could not freely change the poses they wanted to draw. This research proposed a method to change the pose of a virtual humanoid model by 3D pose estimation of a real human using Motion Capture with a depth camera. We developed a prototype system based on this method in order to improve the pose change. In the evaluation experiment, we tested the usefulness of this system by using the improvement values of the scores of the experimental and control groups before and after learning. [ABSTRACT FROM AUTHOR]

Motor impairments are among the most relevant, evident, and disabling symptoms of Parkinson’s disease that adversely affect quality of life, resulting in limited autonomy, independence, and safety. Recent studies have demonstrated the benefits of physiotherapy and rehabilitation programs specifically targeted to the needs of Parkinsonian patients in supporting drug treatments and improving motor control and coordination. However, due to the expected increase in patients in the coming years, traditional rehabilitation pathways in healthcare facilities could become unsustainable. Consequently, new strategies are needed, in which technologies play a key role in enabling more frequent, comprehensive, and out-of-hospital follow-up. The paper proposes a vision-based solution using the new Azure Kinect DK sensor to implement an integrated approach for remote assessment, monitoring, and rehabilitation of Parkinsonian patients, exploiting non-invasive 3D tracking of body movements to objectively and automatically characterize both standard evaluative motor tasks and virtual exergames. An experimental test involving 20 parkinsonian subjects and 15 healthy controls was organized. Preliminary results show the system’s ability to quantify specific and statistically significant (p < 0.05) features of motor performance, easily monitor changes as the disease progresses over time, and at the same time permit the use of exergames in virtual reality both for training and as a support for motor condition assessment (for example, detecting an average reduction in arm swing asymmetry of about 14% after arm training). The main innovation relies precisely on the integration of evaluative and rehabilitative aspects, which could be used as a closed loop to design new protocols for remote management of patients tailored to their actual conditions.

Arm swinging is a typical feature of human walking: Continuous and rhythmic movement of the upper limbs is important to ensure postural stability and walking efficiency. However, several factors can interfere with arm swings, making walking more risky and unstable: These include aging, neurological diseases, hemiplegia, and other comorbidities that affect motor control and coordination. Objective assessment of arm swings during walking could play a role in preventing adverse consequences, allowing appropriate treatments and rehabilitation protocols to be activated for recovery and improvement. This paper presents a system for gait analysis based on Microsoft Azure Kinect DK sensor and its body-tracking algorithm: It allows noninvasive full-body tracking, thus enabling simultaneous analysis of different aspects of walking, including arm swing characteristics. Sixteen subjects with Parkinson’s disease and 13 healthy controls were recruited with the aim of evaluating differences in arm swing features and correlating them with traditional gait parameters. Preliminary results show significant differences between the two groups and a strong correlation between the parameters. The study thus highlights the ability of the proposed system to quantify arm swing features, thus offering a simple tool to provide a more comprehensive gait assessment.

The Azure Kinect, the successor of Kinect v1 and Kinect v2, is a depth sensor. In this paper we evaluate the skeleton tracking abilities of the new sensor, namely accuracy and precision (repeatability). Firstly, we state the technical features of all three sensors, since we want to put the new Azure Kinect in the context of its previous versions. Then, we present the experimental results of general accuracy and precision obtained by measuring a plate mounted to a robotic manipulator end effector which was moved along the depth axis of each sensor and compare them. In the second experiment, we mounted a human-sized figurine to the end effector and placed it in the same positions as the test plate. Positions were located 400 mm from each other. In each position, we measured relative accuracy and precision (repeatability) of the detected figurine body joints. We compared the results and concluded that the Azure Kinect surpasses its discontinued predecessors, both in accuracy and precision. It is a suitable sensor for human–robot interaction, body-motion analysis, and other gesture-based applications. Our analysis serves as a pilot study for future HMI (human–machine interaction) designs and applications using the new Kinect Azure and puts it in the context of its successful predecessors. [ABSTRACT FROM AUTHOR]

Nowadays, the need for reliable and low-cost multi-camera systems is increasing for many potential applications, such as localization and mapping, human activity recognition, hand and gesture analysis, and object detection and localization. However, a precise camera calibration approach is mandatory for enabling further applications that require high precision. This paper analyzes the available two-camera calibration approaches to propose a guideline for calibrating multiple Azure Kinect RGB-D sensors to achieve the best alignment of point clouds in both color and infrared resolutions, and skeletal joints returned by the Microsoft Azure Body Tracking library. Different calibration methodologies using 2D and 3D approaches, all exploiting the functionalities within the Azure Kinect devices, are presented. Experiments demonstrate that the best results are returned by applying 3D calibration procedures, which give an average distance between all couples of corresponding points of point clouds in color or an infrared resolution of 21.426 mm and 9.872 mm for a static experiment and of 20.868 mm and 7.429 mm while framing a dynamic scene. At the same time, the best results in body joint alignment are achieved by three-dimensional procedures on images captured by the infrared sensors, resulting in an average error of 35.410 mm.

Abnormal movement of the head and neck is a typical symptom of Cervical Dystonia (CD). Accurate scoring on the severity scale is of great significance for treatment planning. The traditional scoring method is to use a protractor or contact sensors to calculate the angle of the movement, but this method is time-consuming, and it will interfere with the movement of the patient. In the recent outbreak of the coronavirus disease, the need for remote diagnosis and treatment of CD has become extremely urgent for clinical practice. To solve these problems, we propose a multi-view vision based CD severity scale scoring method, which detects the keypoint positions of the patient from the frontal and lateral images, and finally scores the severity scale by calculating head and neck motion angles. We compared the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS) subscale scores calculated by our vision based method with the scores calculated by a neurologist trained in dyskinesia. An analysis of the correlation coefficient was then conducted. Intra-class correlation (ICC)(3,1) was used to measure absolute accuracy. Our multi-view vision based CD severity scale scoring method demonstrated sufficient validity and reliability. This low-cost and contactless method provides a new potential tool for remote diagnosis and treatment of CD.

Analysing the dynamics in social interactions in indoor spaces entails evaluating spatial–temporal variables from the event, such as location and time. Additionally, social interactions include invisible spaces that we unconsciously acknowledge due to social constraints, e.g., space between people having a conversation with each other. Nevertheless, current sensor arrays focus on detecting the physically occupied spaces from social interactions, i.e., areas inhabited by physically measurable objects. Our goal is to detect the socially occupied spaces, i.e., spaces not physically occupied by subjects and objects but inhabited by the interaction they sustain. We evaluate the social representation of the space structure between two or more active participants, so-called F-Formation for small gatherings. We propose calculating body orientation and location from skeleton joint data sets by integrating depth cameras. The body orientation is derived by integrating the shoulders and spine joint data with head/face rotation data and spatial–temporal information from trajectories. From the physically occupied measurements, we can detect socially occupied spaces. In our user study implementing the system, we compared the capabilities and skeleton tracking datasets from three depth camera sensors, the Kinect v2, Azure Kinect, and Zed 2i. We collected 32 walking patterns for individual and dyad configurations and evaluated the system’s accuracy regarding the intended and socially accepted orientations. Experimental results show accuracy above 90% for the Kinect v2, 96% for the Azure Kinect, and 89% for the Zed 2i for assessing socially relevant body orientation. Our algorithm contributes to the anonymous and automated assessment of socially occupied spaces. The depth sensor system is promising in detecting more complex social structures. These findings impact research areas that study group interactions within complex indoor settings.

The Azure Kinect represents the latest generation of Microsoft Kinect depth cameras. Of interest in this article is the depth and spatial accuracy of the Azure Kinect and how it compares to its predecessor, the Kinect v2. In one experiment, the two sensors are used to capture a planar whiteboard at 15 locations in a grid pattern with laser scanner data serving as ground truth. A set of histograms reveals the temporal-based random depth error inherent in each Kinect. Additionally, a two-dimensional cone of accuracy illustrates the systematic spatial error. At distances greater than 2.5 m, we find the Azure Kinect to have improved accuracy in both spatial and temporal domains as compared to the Kinect v2, while for distances less than 2.5 m, the spatial and temporal accuracies were found to be comparable. In another experiment, we compare the distribution of random depth error between each Kinect sensor by capturing a flat wall across the field of view in horizontal and vertical directions. We find the Azure Kinect to have improved temporal accuracy over the Kinect v2 in the range of 2.5 to 3.5 m for measurements close to the optical axis. The results indicate that the Azure Kinect is a suitable substitute for Kinect v2 in 3D scanning applications.

In the era of expanding manned space missions, understanding the biomechanical impacts of zero gravity on human movement is pivotal. This study introduces a novel and cost-effective framework that demonstrates the application of Microsoft's Azure Kinect body tracking technology as a motion input generator for subsequent OpenSim simulations in weightlessness. Testing rotations, locomotion, coordination, and martial arts movements, we validate the results' realism under the constraints of angular and linear momentum conservation. While complex, full-body coordination tasks face limitations in a zero gravity environment, our findings suggest possible approaches to device-free exercise routines for astronauts and reveal insights into the feasibility of hand-to-hand combat in space. However, some challenges remain in distinguishing zero gravity effects in the simulations from discrepancies in the captured motion input or forward dynamics calculations, making a comprehensive validation difficult. The paper concludes by highlighting the framework's practical potential for the future of space mission planning and related research endeavors, while also providing recommendations for further refinement.

• Colour and depth images provided by an RGB-D Azure Kinect DK camera are used. • Automatic algorithms to estimate size and weight of apples are evaluated. • Allometric models show a high ability to predict the weight of apples. • Different sizing and weighting algorithms can be used for yield prediction. Data acquired using an RGB-D Azure Kinect DK camera were used to assess different automatic algorithms to estimate the size, and predict the weight of non-occluded and occluded apples. The programming of the algorithms included: (i) the extraction of images of regions of interest (ROI) using manual delimitation of bounding boxes or binary masks; (ii) estimating the lengths of the major and minor geometric axes for the purpose of apple sizing; and (iii) predicting the final weight by allometric modelling. In addition to the use of bounding boxes, the algorithms also allowed other post-mask settings (circles, ellipses and rotated rectangles) to be implemented, and different depth options (distance between the RGB-D camera and the fruits detected) for subsequent sizing through the application of the thin lens theory. Both linear and nonlinear allometric models demonstrated the ability to predict apple weight with a high degree of accuracy (R2 greater than 0.942 and RMSE < 16 g). With respect to non-occluded apples, the best weight predictions were achieved using a linear allometric model including both the major and minor axes of the apples as predictors. The mean absolute percentage error (MAPE) ranged from 5.1% to 5.7% with respective RMSE of 11.09 g and 13.02 g, depending to whether circles, ellipses, or bounding boxes were used to adjust fruit shape. The results were therefore promising and open up the possibility of implementing reliable in-field apple measurements in real time. Importantly, final weight prediction error and intermediate size estimation errors (from sizing algorithms) interact but in a way that is not easily quantifiable when weight allometric models with implicit prediction error are used. In addition, allometric models should be reviewed when applied to other apple cultivars, fruit development stages or even for different fruit growth conditions depending on canopy management. [ABSTRACT FROM AUTHOR]

The three-dimensional reconstruction method using RGB-D camera has a good balance in hardware cost and point cloud quality. However, due to the limitation of inherent structure and imaging principle, the acquired point cloud has problems such as a lot of noise and difficult registration. This paper proposes a 3D reconstruction method using Azure Kinect to solve these inherent problems. Shoot color images, depth images and near-infrared images of the target from six perspectives by Azure Kinect sensor with black background. Multiply the binarization result of the 8-bit infrared image with the RGB-D image alignment result provided by Microsoft corporation, which can remove ghosting and most of the background noise. A neighborhood extreme filtering method is proposed to filter out the abrupt points in the depth image, by which the floating noise point and most of the outlier noise will be removed before generating the point cloud, and then using the pass-through filter eliminate rest of the outlier noise. An improved method based on the classic iterative closest point (ICP) algorithm is presented to merge multiple-views point clouds. By continuously reducing both the size of the down-sampling grid and the distance threshold between the corresponding points, the point clouds of each view are continuously registered three times, until get the integral color point cloud. Many experiments on rapeseed plants show that the success rate of cloud registration is 92.5% and the point cloud accuracy obtained by this method is 0.789 mm, the time consuming of a integral scanning is 302 s, and with a good color restoration. Compared with a laser scanner, the proposed method has considerable reconstruction accuracy and a significantly ahead of the reconstruction speed, but the hardware cost is much lower when building a automatic scanning system. This research shows a low-cost, high-precision 3D reconstruction technology, which has the potential to be widely used for non-destructive measurement of rapeseed and other crops phenotype.

The Azure Kinect, the successor of Kinect v1 and Kinect v2, is a depth sensor. In this paper we evaluate the skeleton tracking abilities of the new sensor, namely accuracy and precision (repeatability). Firstly, we state the technical features of all three sensors, since we want to put the new Azure Kinect in the context of its previous versions. Then, we present the experimental results of general accuracy and precision obtained by measuring a plate mounted to a robotic manipulator end effector which was moved along the depth axis of each sensor and compare them. In the second experiment, we mounted a human-sized figurine to the end effector and placed it in the same positions as the test plate. Positions were located 400 mm from each other. In each position, we measured relative accuracy and precision (repeatability) of the detected figurine body joints. We compared the results and concluded that the Azure Kinect surpasses its discontinued predecessors, both in accuracy and precision. It is a suitable sensor for human–robot interaction, body-motion analysis, and other gesture-based applications. Our analysis serves as a pilot study for future HMI (human–machine interaction) designs and applications using the new Kinect Azure and puts it in the context of its successful predecessors.