Carbon fiber is composed of crystallite and amorphous carbon. The X-ray diffraction (XRD) and Raman spectroscopy were used to investigate the influencing mechanism of the microstructure of crystallites of polyacrylonitrile (PAN)-based carbon fibers on tensile strength. The results show that the crystallinity, the degree of graphitization, the degree of crystallites structural disorder and the crystallite size have obvious effects on tensile strength. The larger the crystallinity and the degree of graphitization,the larger the tensile strength; the smaller the degree of crystallites structural disorder, the larger the tensile strength; the larger the crystallite size, the smaller the tensile strength. Comparing T300 with T700, the increment of tensile strength, which is caused by the increase of the crystallinity and the degree of graphitization, and the decrease of the degree of crystallites structural disorder, is larger than the decrement of tensile strength, which is caused by the increase of the crystallite size. Therefore, the tensile strength of T700 is larger than that of T300, and the same with that the tensile strength of T800 is larger than that of T700. When M35J is compared with M40J, the increment of tensile strength, which is caused by the increase of the crystallinity and the degree of graphitization, and the decrease of the degree of crystallites structural disorder, is smaller than the decrement of tensile strength, which is caused by the increase of the crystallite size. Therefore, the tensile strength of M40J is smaller than that of M35J, and the same with that the tensile strength of M46J is smaller than that of M40J. In M35J, M40J and M46J, the bigger crystallite size is a crucial factor that affects the tensile strength.