As living standards have improved, consumer demand for high-quality dried abalone has increased. Traditional abalone grading is achieved through slice analysis (sampling analysis) combined with human experience. However, this method has several issues, including non-uniform grading standards, low detection accuracy, inconsistency between internal and external quality, and high loss rate. Therefore, we propose a deep-learning-aided approach leveraging X-ray images that can achieve efficient and non-destructive internal quality grading of dried abalone. To the best of our knowledge, this is the first work to use X-ray to image the internal structure of dried abalone. The work was divided into three phases. First, a database of X-ray images of dried abalone was constructed, containing 644 samples, and the relationship between the X-ray images and the internal quality of the dried abalone was analyzed. Second, the database was augmented by image rotation, image mirroring, and noise superposition. Subsequently, a model selection evaluation process was carried out. The evaluation results showed that, in a comparison with models such as VGG-16, MobileNet (Version 1.0), AlexNet, and Xception, VGG-19 demonstrated the best performance in the quality grading of dried abalone. Finally, a modified VGG-19 network based on the CBAM was proposed to classify the quality of dried abalone. The results show that the proposed quality grading method for dried abalone was effective, achieving a score of 95.14%, and outperformed the competitors, i.e., VGG-19 alone and VGG-19 with the squeeze-and-excitation block (SE) attention mechanism.