The pancreatic cancer is extremely fatal. Due to limitations of anatomic location and condition, physicians are hard to make precise diagnoses of patients from traditional ultrasound (US) or CT images. The purpose of this study is to develop a computer-aided diagnosis (CAD) system for pancreatic tumor by the selected features from CT and US images. In this study, the following steps are included: (1) Segment images by applying GVF SNAKE; (2) Select features by applying t- Test; (3) Identify normal tissues, adenocarcinoma tumors, pseudo tumors, cystic tumors, and pseudo cyst by SVM and SOM for CT and US images, respectively. (4) Finally, totally diagnosed 69 US images and 136 CT images were used to evaluate system performance. In order to improve this system, different numbers of features were selected in three different stages for CT and US images. The results show this CAD system has the best performance to identify all images by applying 2 features (Area, NRL_MA) and 4 features (l_Average, g _Entropy, c_Entropy, Area) in US images and contrast injected CT images, respectively. Moreover, the tumor area is the most important morphological feature for tumor classification in US images and the adenocarcinoma tumor has lower value of “Entropy” in contrast injected CT images. In most cases, the performance (sensitivity, specificity, and accuracy are higher than 0.9) of this developed system is good enough for clinical study. However, US CAD system and CT CAD system have better performances on identifying tiny pancreatitis tumors and cystic tumors, respectively. We suggest physicians to diagnose tumors by the aid of US CAD system, and diagnose cysts by CT CAD system; consequently, reduce cost and improve the diagnostic accuracy.