Search

Two methodological practices are well established in other scientific disciplines yet remain rare in computer-security research: comparative experiments and statistical inferences. Comparative experiments offer the only way to control factors that might vary from one study to the next. Statistical inferences enable a researcher to draw general conclusions from empirical results. Despite their widespread use in other sciences, these practices are haphazardly used in security research. Choosing keystroke dynamics as an example to study, we survey the literature. Of 80 papers wherein these practices would be appropriate, only 43 (53.75%) performed comparative experiments, and only 6 (7.5%) drew statistical inferences. In disciplines such as medicine, comparative experiments and statistical inferences save lives and cut costs. Rigorous methodological standards are required. We see no reason why security research, another discipline where the stakes are critically high, cannot or should not adopt these practices as well. Failure to take a more scientific approach to security research stalls progress and leaves us vulnerable.

Having decided to focus attention on the "weak link" of human fallibility, a growing number of security researchers are discovering the US Government's regulations that govern human subject research. This paper discusses those regulations, their application to research on security and usability, and presents strategies for negotiating the Institutional Review Board (IRB) approval process. It argues that a strict interpretation of regulations has the potential to stymie security research., Presented at Usability, Psychology and Security 2008, UPSEC08, co-located with the 5th USENIX Symposium on Networked Systems Design & Implementation, NSDI '08, in San Francisco, CA on 14 Apr 2008. Published online in proceedings of the same. The original document contains color images.