Order effect when comparing usability of two telehealth remote monitoring technology solutions

An experienced healthcare device evaluator collaborated with a human factors scientist to develop and conduct a usability test of two different telehealth remote monitoring solutions as part of a larger technology evaluation. Twenty-one participants used each of the telehealth platforms to capture and transmit data. Typically users of telehealth platforms capture and send their own data, however, we asked our participants to place a bag of sand on wirelessly connected scales to capture weight information and to place the wireless pulse oximeters on the healthcare device evaluator’s index finger to get blood oxygen saturation measurements. This protocol enabled participants to go through the processes required to capture and share representative data without providing any of their own personal health data. After using each solution to complete the same series of three tasks, participants filled out the System Usability Scale (SUS) questionnaire. We intended to compare SUS scores within-subjects, but discovered that the SUS scores for the second solution were not independent. In particular, the participants who used the system that was harder to use second produced SUS scores for that system that were significantly lower than the participants who rated that system first. We conclude that it may not be appropriate to use a within-subjects design if experimenters plan to compare the SUS scores for different systems.