Search

Previous efforts used digital video to develop computer-generated assessments of surgical hand motion economy and fluidity of motion. This study tests how well previously trained assessment models match expert ratings of suturing and tying video clips recorded in a new operating room (OR) setting.Enabled through computer vision of the hands, this study tests the applicability of assessments born out of benchtop simulations to in vivo suturing and tying tasks recorded in the OR.Compared with expert ratings, computer-generated assessments for fluidity of motion (slope = 0.83, intercept = 1.77, R2 = 0.55) performed better than motion economy (slope = 0.73, intercept = 2.04, R2 = 0.49), although 85% of ratings for both models were within ±2 of the expert response. Neither assessment performed as well in the OR as they did on the training data. Assessments were sensitive to changing hand postures, dropped ligatures, and poor tissue contact-features typically missing from training data. Computer-generated assessment of OR tasks was contingent on a clear, consistent view of both surgeon's hands.Computer-generated assessment may help provide formative feedback during deliberate practice, albeit with greater variability in the OR compared with benchtop simulations. Future work will benefit from expanded available bimanual video records.

We present situated live programming for human-robot collaboration, an approach that enables users with limited programming experience to program collaborative applications for human-robot interaction. Allowing end users, such as shop floor workers, to program collaborative robots themselves would make it easy to "retask" robots from one process to another, facilitating their adoption by small and medium enterprises. Our approach builds on the paradigm of trigger-action programming (TAP) by allowing end users to create rich interactions through simple trigger-action pairings. It enables end users to iteratively create, edit, and refine a reactive robot program while executing partial programs. This live programming approach enables the user to utilize the task space and objects by incrementally specifying situated trigger-action pairs, substantially lowering the barrier to entry for programming or reprogramming robots for collaboration. We instantiate situated live programming in an authoring system where users can create trigger-action programs by annotating an augmented video feed from the robot's perspective and assign robot actions to trigger conditions. We evaluated this system in a study where participants (n = 10) developed robot programs for solving collaborative light-manufacturing tasks. Results showed that users with little programming experience were able to program HRC tasks in an interactive fashion and our situated live programming approach further supported individualized strategies and workflows. We conclude by discussing opportunities and limitations of the proposed approach, our system implementation, and our study and discuss a roadmap for expanding this approach to a broader range of tasks and applications., Comment: Accepted at UIST 2021, October 10-14, 2021, Virtual Event, USA

Many tasks, particularly those involving interaction with the environment, are characterized by high variability, making robotic autonomy difficult. One flexible solution is to introduce the input of a human with superior experience and cognitive abilities as part of a shared autonomy policy. However, current methods for shared autonomy are not designed to address the wide range of necessary corrections (e.g., positions, forces, execution rate, etc.) that the user may need to provide to address task variability. In this paper, we present corrective shared autonomy, where users provide corrections to key robot state variables on top of an otherwise autonomous task model. We provide an instantiation of this shared autonomy paradigm and demonstrate its viability and benefits such as low user effort and physical demand via a system-level user study on three tasks involving variability situated in aircraft manufacturing., Comment: IEEE Robotics and Automation Letters (RA-L)

Worker posture, task time and performance are often affected when one-handed manual dexterous tasks are performed in small overhead spaces under an obscured view. A common method used for supplementing visual feedback in these cases is a hand-held telescopic mirror, but that involves working with both arms extended overhead, and is often accompanied by awkward neck and shoulder postures. A video camera was considered as an alternative to using a mirror for visual feedback and reducing overhead reach. A mirror, a borescope and an omnidirectional camera were evaluated while laboratory participants performed three one-handed simulated manufacturing tasks in a small overhead enclosure. Videos were recorded for quantifying the time that postures were assumed while performing the tasks. The average time that both arms were above mid-shoulder height for the omnidirectional camera was more than 2.5 times less than for the mirror and borescope. The average proportion of neck strain time was 0.01% (or less) for both the omnidirectional camera and the borescope, compared to 83.68% for the mirror. No significant differences were observed in task completion times between the three modalities. Hence, an omnidirectional camera can provide visibility while reducing straining postures for manufacturing operations involving overhead work.

A widely used risk prediction tool, the revised NIOSH lifting equation (RNLE), provides the recommended weight limit (RWL), but is limited by analyst subjectivity, experience, and resources. This paper describes a robust, non-intrusive, straightforward approach to automatically extract spatial and temporal factors necessary for the RNLE using a single video camera in the sagittal plane. The participant's silhouette is segmented by motion information and the novel use of a ghosting effect provides accurate detection of lifting instances, and hand and feet location prediction. Laboratory tests using 6 participants, each performing 36 lifts, showed that a nominal 640 pixel × 480 pixel 2D video, in comparison to 3D motion capture, provided RWL estimations within 0.2 kg (

OBJECTIVE: Computer vision was used to predict expert performance ratings from surgeon hand motions for tying and suturing tasks. SUMMARY BACKGROUND DATA: Existing methods, including the objective structured assessment of technical skills (OSATS) have proven reliable, but do not readily discriminate at the task level. Computer vision may be used for evaluating distinct task performance throughout an operation. METHODS: Open surgeries were videoed and surgeon hands were tracked without using sensors or markers. An expert panel of three attending surgeons rated tying and suturing video clips on continuous scales from 0 to 10 along three task measures adapted from the broader OSATS: motion economy, fluidity of motion, and tissue handling. Empirical models were developed to predict the expert consensus ratings based on the hand kinematic data records. RESULTS: The predicted v. panel ratings for suturing had slopes from 0.73 to 1, and intercepts from 0.36 to 1.54 (Average R(2) = 0.81). Predicted v. panel ratings for tying had slopes from 0.39 to 0.88, and intercepts from 0.79 to 4.36 (Average R(2) = 0.57). The mean square error among predicted and expert ratings were consistently less than the mean squared difference among individual expert ratings and the eventual consensus ratings. CONCLUSIONS: The computer algorithm consistently predicted the panel ratings of individual tasks, and were more objective and reliable than individual assessment by surgical experts.

Repetitive tasks performed in a commercial laboratory potentially expose lab technicians to repetitive-motion stresses and strains. Our goal was to reduce or eliminate the risk of injuries through design strategies, such as automation, changes in work flow, system design improvements, or ergonomic assist tools. This article describes six undergraduate biomedical engineering student design innovations conceived for solving challenging ergonomics problems at Covance Laboratories in Madison, Wisconsin. Their designs tackled laboratory procedures for administering substances, sampling blood, removing container lids, vortex mixing, cage opening, and automated bioanalytical test tube uncapping. The successful designs were built, tested, and put into practice.

As collaborative robots begin to appear on factory floors, there is a need to consider how these robots can best help their human partners. In this paper, we propose an optimization framework that generates task assignments and schedules for a human–robot team with the goal of improving both time and ergonomics and demonstrate its use in six real-world manufacturing processes that are currently performed manually. Using the strain index method to quantify human physical stress, we create a set of solutions with assigned priorities on each goal. The resulting schedules provide engineers with insight into selecting the appropriate level of compromise and integrating the robot in a way that best fits the needs of an individual process. Note to Practitioners —Collaborative robots promise many advantages on the shop and factory floor, including low-cost automation and flexibility in small-batch production. Using this technology requires engineers to redesign tasks that are currently performed by human workers to effectively involve human and robot workers. However, existing quantitative methods for scheduling and allocating tasks to multiple workers do not consider factors, such as differences in skill between human and robot workers or the differential ergonomic impact of tasks on workers. We propose a method to analyze how the inclusion of a collaborative robot in an existing process might affect the makespan of the task and the physical strain the task places on the human worker. The method enables the engineer to prioritize and weigh makespan and worker ergonomics in creating schedules and inspect the resulting task schedules. Using this method, engineers can determine how the addition of a collaborative robot might improve makespan and/or reduce job risk and potential for occupational hazard for human workers, particularly in tasks that involve high physical strain. We apply our method to six real-world tasks from various industries to demonstrate its use and discuss its practical limitations. In our future work, we plan to develop a software tool that will assist engineers in the use of our method.

Corrective Shared Autonomy is a method where human corrections are layered on top of an otherwise autonomous robot behavior. Specifically, a Corrective Shared Autonomy system leverages an external controller to allow corrections across a range of task variables (e.g., spinning speed of a tool, applied force, path) to address the specific needs of a task. However, this inherent flexibility makes the choice of what corrections to allow at any given instant difficult to determine. This choice of corrections includes determining appropriate robot state variables, scaling for these variables, and a way to allow a user to specify the corrections in an intuitive manner. This paper enables efficient Corrective Shared Autonomy by providing an automated solution based on Learning from Demonstration to both extract the nominal behavior and address these core problems. Our evaluation shows that this solution enables users to successfully complete a surface cleaning task, identifies different strategies users employed in applying corrections, and points to future improvements for our solution., Comment: IEEE Robotics and Automation Letters (RA-L)

Remote teleoperation of robots can broaden the reach of domain specialists across a wide range of industries such as home maintenance, health care, light manufacturing, and construction. However, current direct control methods are impractical, and existing tools for programming robot remotely have focused on users with significant robotic experience. Extending robot remote programming to end users, i.e., users who are experts in a domain but novices in robotics, requires tools that balance the rich features necessary for complex teleoperation tasks with ease of use. The primary challenge to usability is that novice users are unable to specify complete and robust task plans to allow a robot to perform duties autonomously, particularly in highly variable environments. Our solution is to allow operators to specify shorter sequences of high-level commands, which we call task-level authoring, to create periods of variable robot autonomy. This approach allows inexperienced users to create robot behaviors in uncertain environments by interleaving exploration, specification of behaviors, and execution as separate steps. End users are able to break down the specification of tasks and adapt to the current needs of the interaction and environments, combining the reactivity of direct control to asynchronous operation. In this paper, we describe a prototype system contextualized in light manufacturing and its empirical validation in a user study where 18 participants with some programming experience were able to perform a variety of complex telemanipulation tasks with little training. Our results show that our approach allowed users to create flexible periods of autonomy and solve rich manipulation tasks. Furthermore, participants significantly preferred our system over comparative more direct interfaces, demonstrating the potential of our approach., Comment: Provisionally accepted in Frontiers in Robotics and AI on the 09/03/2021. Abstract has been shortened due to arxiv abstract requirements

Technicians in a commercial laboratory manually uncap up to 700 sample tubes daily in preparation for bioanalytical testing. Manually twisting off sample tube caps not only is a time-consuming task, but also poses increased risk for muscle fatigue and repetitive-motion injuries. An automated device capable of uncapping sample tubes at a rate faster than the current workflow would be valuable for minimizing strain on technicians' hands and saving time. Although several commercial sample tube-uncapping products exist, they are not always usable for a workload that uses a mix of tube sizes and specific workflow. A functioning uncapping device was developed that can semi-automatically uncap sample tubes with three different heights and diameters and was compatible with the workflow in a commercial laboratory setting. Under limited testing, the average success rate with uncapping each of the three sample tube sizes or a mix of them was 90% or more, more than three times faster than manual uncapping, and met standard acceptance criteria using mass spectrometry. Our device with its current performance is still a prototype, requiring further development. It showed promise for ergonomic benefit to the laboratory technicians, however, reducing the necessity to manually unscrew caps.

Objective Trade-offs between productivity, physical workload (PWL), and mental workload (MWL) were studied when integrating collaborative robots (cobots) into existing manual work by optimizing the allocation of tasks. Background As cobots become more widely introduced in the workplace and their capabilities greatly improved, there is a need to consider how they can best help their human partners. Methods A theoretical data-driven analysis was conducted using the O*NET Content Model to evaluate 16 selected jobs for associated work context, skills, and constraints. Associated work activities were ranked by potential for substitution by a cobot. PWL and MWL were estimated using variables from the O*Net database that represent variables for the Strain Index and NASA-TLX. An algorithm was developed to optimize work activity assignment to cobots and human workers according to their most suited abilities. Results Human workload for some jobs decreased while workload for some jobs increased after cobots were reassigned tasks, and residual human capacity was used to perform job activities designated the most important to increase productivity. The human workload for other jobs remained unchanged. Conclusions The changes in human workload from the introduction of cobots may not always be beneficial for the human worker unless trade-offs are considered. Application: The framework of this study may be applied to existing jobs to identify the relationship between productivity and worker tolerances that integrate cobots into specific tasks.

Historically low, the proportion of female urology residents now exceeds 25% in recent years. Self-assessment is a widely used tool to track progress in medical education. However, the validity of its results and gender differences may influence interpretation. Simulation of surgical skills is increasingly common in modern residency training and standardizes certain objective tasks and skills. The objective of this study was to identify gender differences in self-assessment of surgeons and trainees when using simulation of surgical skills.Medical students, residents, and attending and retired surgeons completed simple interrupted suturing. Assessment was self-rated using previously tested visual analog motion scales. Tasks were video recorded and rated by blinded expert surgeons using identical motion scales. Computer vision motion tracking software was used to objectively analyze the kinematics of surgical tasks.Proportion of female (n = 17) and male (n = 20) participants did not differ significantly by the level of training, P = 0.76. Five expert surgeons evaluated 84 video segments of simple interrupted suturing tasks (mean 3.0 segments per task per participant). Self-assessment correlated well overall with expert rating for motion economy (Pearson correlation coefficient 0.61, P 0.001) and motion fluidity (0.55, P = 0.002). Women underrated their performance in accordance with mean individual difference of self-assessment and expert assessment scores (Δ SAS-EAS) for both economy of motion (mean ± SEM -1.1 ± 0.38, P = 0.01) and fluidity of motion (-1.3 ± 0.39, P 0.01). On the same measures, men tended to rate themselves in accordance with experts (-0.16 ± 0.36, P = 0.63; -0.09 ± 0.41, P = 0.82, respectively). Δ SAS-EAS did not differ significantly on any rating scale across levels of training. Expert ratings did not differ significantly by gender for any domain.Female surgeons and trainees underrate some technical skills on self-assessment when compared with expert ratings, whereas male surgeon and trainee self-ratings and expert ratings were similar. Further work is needed to determine if these differences are accentuated across increasingly difficult tasks.

This study evaluates if hand movements, tracked using digital video, can quantify in-context surgical performance. Participants of varied experience completed simple interrupted suturing and running subcuticular suturing tasks. Marker-less motion tracking software traced the two-dimensional position of a region of the hand for every video frame. Four expert observers rated 219 short video clips of participants performing the task from 0 to 10 along the following visual analog scales: fluidity of motion, motion economy, tissue handling, and coordination. Expert ratings of attending surgeon hand motions (mean=7.5, sd=1.3) were significantly greater ( p

Objective: A method for automatically classifying lifting postures from simple features in video recordings was developed and tested. We explored if an “elastic” rectangular bounding box, drawn tightly around the subject, can be used for classifying standing, stooping, and squatting at the lift origin and destination. Background: Current marker-less video tracking methods depend on a priori skeletal human models, which are prone to error from poor illumination, obstructions, and difficulty placing cameras in the field. Robust computer vision algorithms based on spatiotemporal features were previously applied for evaluating repetitive motion tasks, exertion frequency, and duty cycle. Methods: Mannequin poses were systematically generated using the Michigan 3DSSPP software for a wide range of hand locations and lifting postures. The stature-normalized height and width of a bounding box were measured in the sagittal plane and when rotated horizontally by 30°. After randomly ordering the data, a classification and regression tree algorithm was trained to classify the lifting postures. Results: The resulting tree had four levels and four splits, misclassifying 0.36% training-set cases. The algorithm was tested using 30 video clips of industrial lifting tasks, misclassifying 3.33% test-set cases. The sensitivity and specificity, respectively, were 100.0% and 100.0% for squatting, 90.0% and 100.0% for stooping, and 100.0% and 95.0% for standing. Conclusions: The tree classification algorithm is capable of classifying lifting postures based only on dimensions of bounding boxes. Applications: It is anticipated that this practical algorithm can be implemented on handheld devices such as a smartphone, making it readily accessible to practitioners.

This study examined how individuals with and without neck pain performed exercises under the influence of altered visual feedback in virtual reality. Chronic neck pain (n = 9) and asymptomatic (n = 10) individuals were recruited for this cross-sectional study. Participants performed head rotations while receiving programmatically manipulated visual feedback from a head-mounted virtual reality display. The main outcome measure was the control-display gain (ratio between actual head rotation angle and visual rotation angle displayed) recorded at the just-noticeable difference. Actual head rotation angles were measured for different gains. Detection of the manipulated visual feedback was affected by gain. The just-noticeable gain for asymptomatic individuals, below and above unity gain, was 0.903 and 1.159, respectively. Head rotation angle decreased or increased 5.45° for every 0.1 increase or decrease in gain, respectively. The just-noticeable gain for chronic pain individuals, below unity gain, was 0.950. The head rotation angle increased 4.29° for every 0.1 decrease in gain. On average, chronic pain individuals reported that neck rotation was feasible for 84% of the unity gain trials, 66% of the individual just-noticeable difference trials, and 50% of the “nudged” just-noticeable difference trials. This research demonstrated that virtual reality may be useful for promoting the desired outcome of increased range of motion in neck rehabilitation exercises by altering visual feedback.

Background Often in simulated settings, quantitative analysis of technical skill relies largely on specially tagged instruments or tracers on surgeons’ hands. We investigated a novel, marker-less technique for evaluating technical skill during open operations and for differentiating tasks and surgeon experience level. Methods We recorded the operative field via in-light camera for open operations. Sixteen cases yielded 138 video clips of suturing and tying tasks ≥5 seconds in duration. Video clips were categorized based on surgeon role (attending, resident) and task subtype (suturing tasks: body wall, bowel anastomosis, complex anastomosis; tying tasks: body wall, superficial tying, deep tying). We tracked a region of interest on the hand to generate kinematic data. Nested, multilevel modeling addressed the nonindependence of clips obtained from the same surgeon. Results Interaction effects for suturing tasks were seen between role and task categories for average speed ( P = .04), standard deviation of speed ( P = .05), and average acceleration ( P = .03). There were significant differences across task categories for standard deviation of acceleration ( P = .02). Significant differences for tying tasks across task categories were observed for maximum speed ( P = .02); standard deviation of speed ( P = .04); and average ( P = .02), maximum ( P P = .03) of acceleration. Conclusion We demonstrated the ability to detect kinematic differences in performance using marker-less tracking during open operative cases. Suturing task evaluation was most sensitive to differences in surgeon role and task category and may represent a scalable approach for providing quantitative feedback to surgeons about technical skill.

Background Repetitive hammer use has been associated with strain and musculoskeletal injuries. This study investigated if using a shock-control hammer reduces forearm muscle strain by observing adverse physiological responses (i.e. inflammation and localized edema) after use. Methods Three matched framing hammers were studied, including a wood-handle, steel-handle, and shock-control hammer. Fifty volunteers were randomly assigned to use one of these hammers at a fatiguing pace of one strike every second, to seat 20 nails in a wood beam. Magnetic resonance imaging was used to scan the forearm muscles for inflammation before the task, immediately after hammering, and one to two days after. Electromyogram signals were measured to estimate grip exertions and localized muscle fatigue. High-speed video was used to calculate the energy of nail strikes. Findings While estimated grip force was similar across the three hammers, the shock-control hammer had 40% greater kinetic energy upon impact and markedly less proximal extensor muscle edema than the wood-handle and steel-handle hammers, immediately after use (p Interpretation Less edema observed for the shock-control hammer suggests that isolating handle shock can mitigate strain in proximal forearm extensor muscles.

We previously developed a method for classifying lifting postures using dimensions of a rectangular bounding box drawn tightly around the subject for a single camera view that is more tolerable of conditions encountered in industrial settings than high precision tracking and can be practically implemented on a smart hand-held device. This study explores the use of simple bounding box dimensions to predict trunk angle while lifting. Mannequin poses were generated using the Michigan 3DSSPP software for 105 postures across six anthropometries. A regression model for predicting trunk angle was created (adjusted R2=0.91, p2 = 0.80). This algorithm should be useful for calculating trunk kinematic properties that are associated with increased risks of low-back disorders including trunk speed and acceleration using successive video frames of predicted trunk flexion angles.

Objective: This study explores how common machine learning techniques can predict surgical maneuvers from a continuous video record of surgical benchtop simulations. Background: Automatic computer vision recognition of surgical maneuvers (suturing, tying, and transition) could expedite video review and objective assessment of surgeries. Method: We recorded hand movements of 37 clinicians performing simple and running subcuticular suturing benchtop simulations, and applied three machine learning techniques (decision trees, random forests, and hidden Markov models) to classify surgical maneuvers every 2 s (60 frames) of video. Results: Random forest predictions of surgical video correctly classified 74% of all video segments into suturing, tying, and transition states for a randomly selected test set. Hidden Markov model adjustments improved the random forest predictions to 79% for simple interrupted suturing on a subset of randomly selected participants. Conclusion: Random forest predictions aided by hidden Markov modeling provided the best prediction of surgical maneuvers. Training of models across all users improved prediction accuracy by 10% compared with a random selection of participants. Application: Marker-less video hand tracking can predict surgical maneuvers from a continuous video record with similar accuracy as robot-assisted surgical platforms, and may enable more efficient video review of surgical procedures for training and coaching.

INTRODUCTION AND OBJECTIVES:Historically low, the proportion of female urology residents now exceeds 25% in recent years. Simulation of surgical skills is increasingly common in modern residency tr...

Objective This paper provides a literature review and detailed discussion of surgical skill terminology. Culminating in a novel model that proposes a set of unique definitions, this review is designed to facilitate shared understanding to study and develop metrics quantifying surgical skill. Design Objective surgical skill analysis depends on consistent definitions and shared understanding of terms like performance, expertise, experience, aptitude, ability, competency, and proficiency. Structure Each term is discussed in turn, drawing from existing literature and colloquial uses. Implications A new model of definitions is proposed to cement a common and consistent lexicon for future skills analysis, and to quantitatively describe a surgeon's performance throughout their career.

The proportion of tool operators capable of maintaining published psychophysically derived threaded fastener tool handle deflection limits were predicted using a biodynamic tool operator model, interacting with the tool, task and workstation. Tool parameters, including geometry, speed and torque were obtained from the specifications for 35 tools used in an auto assembly plant. Tool mass moments of inertia were measured for these tools using a novel device that engages the tool in a rotating system of known inertia. Task parameters, including fastener target torque and joint properties (soft, medium or hard), were ascertained from the vehicle design specifications. Workstation parameters, including vertical and horizontal distances from the operator were measured using a laser rangefinder for 69 tool installations in the plant. These parameters were entered into the model and tool handle deflection was predicted for each job. While handle deflection for most jobs did not exceed the capacity of 75% females and 99% males, six jobs exceeded the deflection criterion. Those tool installations were examined and modifications in tool speed and operator position improved those jobs within the deflection limits, as predicted by the model. We conclude that biodynamic tool operator models may be useful for identifying stressful tool installations and interventions that bring them within the capacity of most operators.

This paper compares clinician hand motion for common suturing tasks across a range of experience levels and tissue types. Medical students (32), residents (41), attending surgeons (10), and retirees (2) were recorded on digital video while suturing on one of: foam, pig feet, or porcine bowel. Depending on time in position, each medical student, resident, and attending participant was classified as junior or senior, yielding six experience categories. This work focuses on trends associated with increasing tenure observed from those medical students (10), residents (15), and attendings (10) who sutured on foam, and draws comparison across tissue types where pertinent. Utilizing custom software, the two-dimensional location of each of the participant's hands were automatically recorded in every video frame, producing a rich spatiotemporal feature set. While suturing on foam, increasing clinician experience was associated with conserved path length per cycle of the non-dominant hand, significantly reducing from junior medical students (mean = 73.63 cm, sd = 33.21 cm) to senior residents (mean = 46.16 cm, sd = 14.03 cm, p = 0.015), and again between senior residents and senior attendings (mean = 30.84 cm, sd = 14.51 cm, p = 0.045). Despite similar maneuver rates, attendings also accelerated less with their non-dominant hand (mean = 16.27 cm/s2, sd = 81.12 cm/s2, p = 0.002) than senior residents (mean = 24.84 cm/s2, sd = 68.29 cm/s2, p = 0.002). While tying, medical students moved their dominant hands slower (mean = 4.39 cm/s, sd = 1.73 cm/s, p = 0.033) than senior residents (mean = 6.53 cm/s, sd = 2.52 cm/s). These results suggest that increased psychomotor performance during early training manifest through faster dominant hand function, while later increases are characterized by conserving energy and efficiently distributing work between hands. Incorporating this scalable video-based motion analysis into regular formative assessment routines may enable greater quality and consistency of feedback throughout a surgical career.

A method is introduced for automatically classifying lifting postures using simple features obtained through drawing a rectangular bounding box tightly around the body on the sagittal plane in video recordings. Mannequin postures were generated using the University of Michigan 3DSSPP software encompassing a variety of hand locations and were classified into squatting, stooping, and standing. For each mannequin posture a rectangular bounding box was drawn tightly around the mannequin for views in the sagittal plane and rotated by 30 o horizontally. The bounding box dimensions were measured and normalized based on the standing height of the corresponding mannequin. A classification and regression tree algorithm was trained using the height and width of the bounding box to classify the postures. The resulting algorithm misclassified 0.36% of the training-set cases. The algorithm was tested on 30 lifting postures collected from video recordings a variety of industrial lifting tasks, misclassifying 3.33% of test-set cases. The sensitivity and specificity, respectively were 100.0% and 100.0% for squatting, 90.0% and 100.0% for stooping, and 100.0% and 95.0% for standing. The algorithm was capable of classifying lifting postures based only on dimensions of bounding boxes which are simple features that can be measured automatically and continuously. We have developed computer vision software that continuously tracks the subject’s body and automatically applies the described bounding box.

A novel frame-subsampled, drift-resilient (FSDR) video object tracking (VOT) algorithm is proposed. Two design goals are sought: to improve the accuracy and to reduce the processing time. The drifting problem is mitigated with a drift detector and accompanying drift recovery mechanism. When a drift is detected, the recovery mechanism provides an opportunity to put the tracking back on track. To gather context-dependent statistics required for these procedures, an initial short segment of the video sequence will be used as a training sequence. Thus, this algorithm is more suitable for video sequences much longer than several minutes. To reduce computing time, a novel frame-subsampling strategy is proposed to process the VOT on small subset of frames. The trajectory of the tracked object on frames that are skipped will be estimated via interpolation. Compared with state of art VOT algorithms, dramatic improvement of performance (accuracy) and orders of magnitude computing time reduction are observed.

Performance-cost trade-offs in video object tracking tasks for long video sequences is investigated. A novel frame-subsampled, drift-resilient (FSDR) video object tracking algorithm is presented that would achieve desired tracking accuracy while dramatically reducing computing time by processing only sub-sampled video frames. A new pattern matching score metric is proposed to estimate the probability of drifting. A drift-recovery procedure is developed to enable the algorithm to recover from a drift situation and resume accurate tracking. Compared against state-of-the-art video object tracking algorithms, dramatic performance (accuracy) enhancement and cost (computing time) reduction are observed.

An inertial sensor-embedded virtual reality (VR) head-mounted display, the Oculus Rift (the Rift), monitors head movement so the content displayed can be updated accordingly. While the Rift may have potential use in cervical spine biomechanics studies, its accuracy in terms of cervical spine mobility measurement has not yet been validated. In the current study, a VR environment was designed to guide participants to perform prescribed neck movements. The cervical spine kinematics was measured by both the Rift and a reference motion tracking system. Comparison of the kinematics data between the Rift and the tracking system indicated that the Rift can provide good estimates on full range of motion (from one side to the other side) during the performed task. Because of inertial sensor drifting, the unilateral range of motion (from one side to neutral posture) derived from the Rift is more erroneous. The root-mean-square errors over a 1-min task were within 10° for each rotation axis. The error analysis further indicated that the inertial sensor drifted approximately 6° at the beginning of a trial during the initialization. This needs to be addressed when using the Rift in order to more accurately measure cervical spine kinematics. It is suggested that the front cover of the Rift should be aligned against a vertical plane during its initialization.

Objective: This study was a proof of concept for virtual exertions, a novel method that involves the use of body tracking and electromyography for grasping and moving projections of objects in virtual reality (VR). The user views objects in his or her hands during rehearsed co-contractions of the same agonist-antagonist muscles normally used for the desired activities to suggest exerting forces. Background: Unlike physical objects, virtual objects are images and lack mass. There is currently no practical physically demanding way to interact with virtual objects to simulate strenuous activities. Method: Eleven participants grasped and lifted similar physical and virtual objects of various weights in an immersive 3-D Cave Automatic Virtual Environment. Muscle activity, localized muscle fatigue, ratings of perceived exertions, and NASA Task Load Index were measured. Additionally, the relationship between levels of immersion (2-D vs. 3-D) was studied. Results: Although the overall magnitude of biceps activity and workload were greater in VR, muscle activity trends and fatigue patterns for varying weights within VR and physical conditions were the same. Perceived exertions for varying weights were not significantly different between VR and physical conditions. Conclusions: Perceived exertion levels and muscle activity patterns corresponded to the assigned virtual loads, which supported the hypothesis that the method evoked the perception of physical exertions and showed that the method was promising. Application: Ultimately this approach may offer opportunities for research and training individuals to perform strenuous activities under potentially safer conditions that mimic situations while seeing their own body and hands relative to the scene.

A new equation for predicting the hand activity level (HAL) used in the American Conference for Government Industrial Hygienists threshold limit value®(TLV®) was based on exertion frequency (F) and percentage duty cycle (D). The TLV® includes a table for estimating HAL from F and D originating from data in Latko et al. (Latko WA, Armstrong TJ, Foulke JA, Herrin GD, Rabourn RA, Ulin SS, Development and evaluation of an observational method for assessing repetition in hand tasks. American Industrial Hygiene Association Journal, 58(4):278-285, 1997) and post hoc adjustments that include extrapolations outside of the data range. Multimedia video task analysis determined D for two additional jobs from Latko's study not in the original data-set, and a new nonlinear regression equation was developed to better fit the data and create a more accurate table. The equation, HAL = 6:56 ln D[F(1:31) /1+3:18 F(1:31), generally matches the TLV® HAL lookup table, and is a substantial improvement over the linear model, particularly for F1.25 Hz and D60% jobs. The equation more closely fits the data and applies the TLV® using a continuous function.

Computer vision has already impacted a diverse field of applications, ranging from industrial robotics, intelligent and autonomous vehicles, security surveillance, manufacturing inspection, and human-computer interaction. Furthermore, digital imaging technologies are advancing ever smaller in size, finer in granularity, and faster in processing, while becoming less expensive and thus more accessible to businesses, organizations, and individuals in devices such as smart phones and tablets. Consumer products such as the Kinect™ offer advanced marker-less 3D motion capture capabilities at a low cost. New computer vision methods are now being researched and developed for occupational ergonomics applications. It is anticipated that these new tools will profoundly impact the future of occupational ergonomics and provide a variety of new instruments and techniques for design, analysis and evaluation in the practice of ergonomics. A panel of leading experts will describe some of the cutting edge research they are pursuing utilizing computer vision for occupational ergonomics applications. Radwin, Lee and Li use algorithms that track pixel patterns recorded from conventional video for quantifying repetitive hand motion, manual lifting and whole-body activities. Lee and Li describe the use computer vision tools to predict joint angles for a whole-body link model. Lieblich and Park describe the use of Kinect™ for classifying postures and generating individualized task specific avatars. Each of these approaches has specific advantages and limitations, which will be addressed by the panel. A discussion will follow exploring future research needs as well as engaging in a discussion among panelists and attendees about the needs, limitations, and obstacles that this new technology faces in bringing it into practice.

Objective This research considers how driver movements in video clips of naturalistic driving are related to observer subjective ratings of distraction and engagement behaviors. Background Naturalistic driving video provides a unique window into driver behavior unmatched by crash data, roadside observations, or driving simulator experiments. However, manually coding many thousands of hours of video is impractical. An objective method is needed to identify driver behaviors suggestive of distracted or disengaged driving for automated computer vision analysis to access this rich source of data. Method Visual analog scales ranging from 0 to 10 were created, and observers rated their perception of driver distraction and engagement behaviors from selected naturalistic driving videos. Driver kinematics time series were extracted from frame-by-frame coding of driver motions, including head rotation, head flexion/extension, and hands on/off the steering wheel. Results The ratings were consistent among participants. A statistical model predicting average ratings from the kinematic features accounted for 54% of distraction rating variance and 50% of engagement rating variance. Conclusion Rated distraction behavior was positively related to the magnitude of head rotation and fraction of time the hands were off the wheel. Rated engagement behavior was positively related to the variation of head rotation and negatively related to the fraction of time the hands were off the wheel. Application If automated computer vision can code simple kinematic features, such as driver head and hand movements, then large-volume naturalistic driving videos could be automatically analyzed to identify instances when drivers were distracted or disengaged.

Two computer vision algorithms were developed to automatically estimate exertion time, duty cycle (DC) and hand activity level (HAL) from videos of workers performing 50 industrial tasks. The average DC difference between manual frame-by-frame analysis and the computer vision DC was −5.8% for the Decision Tree (DT) algorithm, and 1.4% for the Feature Vector Training (FVT) algorithm. The average HAL difference was 0.5 for the DT algorithm and 0.3 for the FVT algorithm. A sensitivity analysis, conducted to examine the influence that deviations in DC have on HAL, found it remained unaffected when DC error was less than 5%. Thus, a DC error less than 10% will impact HAL less than 0.5 HAL, which is negligible. Automatic computer vision HAL estimates were therefore comparable to manual frame-by-frame estimates.Practitioner Summary: Computer vision was used to automatically estimate exertion time, duty cycle and hand activity level from videos of workers performing industrial tasks.

We are developing video processing algorithms for automatically measuring the ACGIH TLV® hand activity level (HAL) using marker-less tracking of hand movements. An equation for computing HAL ratings directly from tracked kinematics, rather than using a frequency-duty cycle (DC) look-up table, more readily lends itself to automated processing. Videos from the 33 Latko et al. (1997) jobs were digitized and analyzed using marker-less tracking, and hand root mean square (RMS) speed (S) was measured. A linear regression model was developed for predicting the average observer rated HAL based on the measured hand RMS speed and DC. Since the videos did not contain distance calibration, speed was quantified in pixels/s and normalized by the distance of each worker’s hand breadth, measured in pixels. A Monte Carlo simulation was performed using the US Army (1991) hand anthropometry data to determine how variation is introduced in the equation as hand breadth varies. The resulting equation was HAL= −1.06 + 0.0047 S + 0.053 DC and it predicted HAL ratings within ±1. The development of an accurate equation for estimating HAL ratings should enable use of automated and objective measurement in practice. While expert observer HAL ratings offer speed and efficiency, use of objective measurements based on worker hand kinematics should provide greater reliability, as well as offering specific engineering aspects of the job that may be addressed for reducing exposures and the risk of musculoskeletal disorders. Furthermore, automated videos analysis may help improve the speed and efficiency of making objective measurements in practice.

This paper investigates altering visual feedback during neck movement through control-display (C-D) gain for a head-mounted display, for the purpose of determining the just noticeable difference (JND) for encouraging individuals with kinesiophobia (i.e. fear avoidance of movement due to chronic pain) to effectively perform therapeutic neck exercises. The JND was defined as .25 probability of detecting a difference from unity C-D gain (gain=1). A target-aiming task with two consecutive neck moves per trial was presented; one neck move had varying C-D gain and the other had unity gain. The VR system was able to influence neck moves without changing locations of the target. Participants indicated whether the two neck movements were the same or different. Logistic regression revealed that the JND gains were 0.903 (lower bound) and 1.159 (upper bound) as the participants could not discriminate a 55° turn, ranging from 49.7° to 63.7°. This preliminary study shows that immersive VR with altered visual feedback influenced movement. The feasibility for rehabilitation of individuals with kinesiophobia will next be assessed.

This discussion panel aims to identify ergonomic concerns, solutions and research needs related to outcomes and the health and safety of workers who perform clinical and surgical procedures. This session will begin with formal presentations to demonstrate current ergonomic concerns and research initiatives associated with clinical and surgical procedures to frame the panel discussion for the second part of the session. Discussion of different procedures will help to identify solutions research needs that relate to a broad range of ergonomic problems. Questions will be collected from the attendees and speakers and organized so as to guide the panel discussion and to engage all of the speakers in the discussion to achieve the symposium aims.

Objective: A laboratory study investigated the relationship between power hand tool and task-related factors affecting threaded fastener torque accuracy and associated handle reaction force. Background: We previously developed a biodynamic model to predict handle reaction forces. We hypothesized that torque accuracy was related to the same factors that affect operator capacity to react against impulsive tool forces, as predicted by the model. Method: The independent variables included tool (pistol grip on a vertical surface, right angle on a horizontal surface), fastener torque rate (hard, soft), horizontal distance (30 cm and 60 cm), and vertical distance (80 cm, 110 cm, and 140 cm). Ten participants (five male and five female) fastened 12 similar bolts for each experimental condition. Results: Average torque error (audited − target torque) was affected by fastener torque rate and operator position. Torque error decreased 33% for soft torque rates, whereas handle forces greatly increased (170%). Torque error also decreased for the far horizontal distance 7% to 14%, when vertical distance was in the middle or high, but handle force decreased slightly 3% to 5%. Conclusion: The evidence suggests that although both tool and task factors affect fastening accuracy, they each influence handle reaction forces differently. We conclude that these differences are attributed to different parameters each factor influences affecting the dynamics of threaded faster tool operation. Fastener torque rate affects the tool dynamics, whereas posture affects the spring-mass-damping biodynamic properties of the human operator. Application: The prediction of handle reaction force using an operator biodynamic model may be useful for codifying complex and unobvious relationships between tool and task factors for minimizing torque error while controlling handle force.

This paper introduces the concept of virtual exertions, which utilizes real-time feedback from electromyograms (EMG), combined with tracked body movements, to simulate forceful exertions (e.g. lifting, pushing, and pulling) against projections of virtual reality objects. The user acts as if there is a real object and moves and contracts the same muscles normally used for the desired activities to suggest exerting forces against virtual objects actually viewed in their own hands as they are grasped and moved. In order to create virtual exertions, EMG muscle activity is monitored during rehearsed co-contractions of agonist/antagonist muscles used for specific exertions, and contraction patterns and levels are combined with tracked motion of the user’s body and hands for identifying when the participant is exerting sufficient force to displace the intended object. Continuous 3D visual feedback to the participant displays mechanical work against virtual objects with simulated inertial properties. A pilot study, where four participants performed both actual and virtual dumbbell lifting tasks, observed that ratings of perceived exertions (RPE), biceps EMG recruitment, and localized muscle fatigue (mean power frequency) were consistent with the actual task. Biceps and triceps EMG co-contractions were proportionally greater for the virtual case.

Patterns of physical stress exposure are often difficult to measure, and the metrics of variation and techniques for identifying them is underdeveloped in the practice of occupational ergonomics. Computer vision has previously been used for evaluating repetitive motion tasks for hand activity level (HAL) utilizing conventional 2D videos. The approach was made practical by relaxing the need for high precision, and by adopting a semi-automatic approach for measuring spatiotemporal characteristics of the repetitive task. In this paper, a new method for visualizing task factors, using this computer vision approach, is demonstrated. After videos are made, the analyst selects a region of interest on the hand to track and the hand location and its associated kinematics are measured for every frame. The visualization method spatially deconstructs and displays the frequency, speed and duty cycle components of tasks that are part of the threshold limit value for hand activity for the purpose of identifying patterns of exposure associated with the specific job factors, as well as for suggesting task improvements. The localized variables are plotted as a heat map superimposed over the video, and displayed in the context of the task being performed. Based on the intensity of the specific variables used to calculate HAL, we can determine which task factors most contribute to HAL, and readily identify those work elements in the task that contribute more to increased risk for an injury. Work simulations and actual industrial examples are described. This method should help practitioners more readily measure and interpret temporal exposure patterns and identify potential task improvements.