Background: Colorectal cancer (CRC) is one of the most prevalent malignant diseases worldwide. Risk prediction for tumor recurrence is important for making effective treatment decisions and for the survival outcomes of patients with CRC after surgery. Herein, we aimed to explore a prediction algorithm and the risk factors for postoperative tumor recurrence using a machine learning (ML) approach with standardized pathology reports for patients with stage II and III CRC. Methods: Pertinent clinicopathological features were compiled from medical records and standardized pathology reports of patients with stage II and III CRC. Four ML models based on logistic regression (LR), random forest (RF), classification and regression decision trees (CARTs), and support vector machine (SVM) were applied for the development of the prediction algorithm. The area under the curve (AUC) of the ML models was determined in order to compare the prediction accuracy. Genomic studies were performed using a panel-targeted next-generation sequencing approach. Results: A total of 1073 patients who received curative intent surgery at the National Cheng Kung University Hospital between January 2004 and January 2019 were included. Based on conventional statistical methods, chemotherapy (p = 0.003), endophytic tumor configuration (p = 0.008), TNM stage III disease (p < 0.001), pT4 (p < 0.001), pN2 (p < 0.001), increased numbers of lymph node metastases (p < 0.001), higher lymph node ratios (LNR) (p < 0.001), lymphovascular invasion (p < 0.001), perineural invasion (p < 0.001), tumor budding (p = 0.004), and neoadjuvant chemoradiotherapy (p = 0.025) were found to be correlated with the tumor recurrence of patients with stage II–III CRC. While comparing the performance of different ML models for predicting cancer recurrence, the AUCs for LR, RF, CART, and SVM were found to be 0.678, 0.639, 0.593, and 0.581, respectively. The LR model had a better accuracy value of 0.87 and a specificity value of 1 in the testing set. Two prognostic factors, age and LNR, were selected by multivariable analysis and the four ML models. In terms of age, older patients received fewer cycles of chemotherapy and radiotherapy (p < 0.001). Right-sided colon tumors (p = 0.002), larger tumor sizes (p = 0.008) and tumor volumes (p = 0.049), TNM stage II disease (p < 0.001), and advanced pT3–4 stage diseases (p = 0.04) were found to be correlated with the older age of patients. However, pN2 diseases (p = 0.005), lymph node metastasis number (p = 0.001), LNR (p = 0.004), perineural invasion (p = 0.018), and overall survival rate (p < 0.001) were found to be decreased in older patients. Furthermore, PIK3CA and DNMT3A mutations (p = 0.032 and 0.039, respectively) were more frequently found in older patients with stage II–III CRC compared to their younger counterparts. Conclusions: This study demonstrated that ML models have a comparable predictive power for determining cancer recurrence in patients with stage II–III CRC after surgery. Advanced age and high LNR were significant risk factors for cancer recurrence, as determined by ML algorithms and multivariable analyses. Distinctive genomic profiles may contribute to discrete clinical behaviors and survival outcomes between patients of different age groups. Studies incorporating complete molecular and genomic profiles in cancer prediction models are beneficial for patients with stage II–III CRC.