Tangential-force detection ability of three-axis fingernail-color sensor aided by CNN

We create a new tactile recording system with which we develop a three-axis fingernail-color sensor that can measure a three-dimensional force applied to fingertips by observing the change of the fingernail’s color. Since the color change is complicated, the relationships between images and three-dimensional forces were assessed using convolution neural network (CNN) models. The success of this method depends on the input data size because the CNN model learning requires big data. Thus, to efficiently obtain big data, we developed a novel measuring device, which was composed of an electronic scale and a load cell, to obtain fingernail images with 0 $^\circ$ to 360 $^\circ$ directional tangential force. We performed a series of evaluation experiments to obtain movies of the color changes caused by the three-axis forces and created a data set for the CNN models by transforming the movies to still images. Although we produced a generalized CNN model that can evaluate the images of any person’s fingernails, its root means square error (RMSE) exceeded both the whole and individual models, and the individual models showed the smallest RMSE. Therefore, we adopted the individual models, which precisely evaluated the tangential-force direction of the test data in an $F_x$ - $F_y$ plane within around $\pm$ 2.5 $^\circ$ error at the peak points of the applied force. Although the fingernail-color sensor possessed almost the same level of accuracy as previous sensors for normal-force tests, the present fingernail-color sensor acts as the best tangential sensor because the RMSE obtained from tangential-force tests was around 1/3 that of previous studies.