The effects of output codes on transfer learning in a deep convolutional neural net

The output encodings of neural nets determine the structure of the space in which inference occurs. Yet, they are generally given very little thought. It is common practice for neural nets to use 1-Hot encoding when training to discriminate among many classes. The primary exceptions to this are error correcting output codes, and semantic output encodings. Output encodings based upon semantic descriptors cause a net to learn responses for classes to which it has not been exposed, provided those classes may be characterized by the same semantic descriptors. This raises a number of questions, such as “Can a net implicitly learn encodings for unobserved classes in the absence of a semantic encoding, or any encoding that requires some form of prior knowledge and hand crafting?”. Also, are some output encodings better than others for learning these implicit encodings? In this paper, we will compare how effectively different non-semantic encodings are at causing a neural net to implicitly learn encodings for unobserved classes. Also, while evaluating the efficacy of these implicit encodings, we will look for evidence of a phenomenon akin to over-training. Specifically, as training on the observed classes occurs, we initially see improvement in how well the implicitly learned encodings can be used to differentiate among the classes which are unobserved during the net's training. However, as training continues to improve discrimination among the observed classes, the efficacy of the implicit codes either remains steady, or undergoes a degradation. This degradation is akin to the overtraining that one generally tries to guard against when training a neural net.