Search

Designing a deep neural network model that integrates clinical images with other electronic medical records entails various preprocessing operations. Preprocessing of clinical images often requires trimming of parts of the lesions shown in the images, whereas preprocessing of other electronic medical records requires vectorization of these records; for example, patient age is often converted into a categorical vector of 10-year intervals. Although these preprocessing operations are critical to the performance of the classification model, there is no guarantee that the preprocessing step chosen is appropriate for model training. The ability to integrate these preprocessing operations into a deep neural network model and to train the model, including the preprocessing operations, can help design a multi-modal medical classification model. This study proposes integration layers of preprocessing, both for clinical images and electronic medical records, in deep neural network models. Preprocessing of clinical images is realized by a vision transformer layer that selectively adopts the parts of the images requiring attention. The preprocessing of other medical electrical records is performed by adopting full-connection layers and normalizing these layers. These proposed preprocessing-integrated layers were verified using a posttreatment visual acuity prediction task in ophthalmology as a case study. This prediction task requires clinical images as well as patient profile data corresponding to each patient's posttreatment logMAR visual acuity. The performance of a heuristically designed prediction model was compared with the performance of the prediction model that includes the proposed preprocessing integration layers. The mean square errors between predicted and correct results were 0.051 for the heuristic model and 0.054 for the proposed model. Experimental results showed that the proposed model utilizing preprocessing integration layers achieved nearly the same performance as the heuristically designed model.