Fruit grading for ripeness and size is an essential process in the supply chain. Incorrect grading can easily lead to spoiled and degraded fruits entering the market, reducing consumers' confidence in purchasing. At the same time, it is easy to cause the fruit supply chain to reduce profits, unreasonable resource allocation, and related practitioners' income. The current mainstream machine vision grading and manual grading in the production line have dilemmas such as susceptibility to environmental interference, inconsistent grading standards, high cost, and labor shortage. To overcome these problems, this study proposes an integrated flexible tactile sensing array (3 × 4) manipulator for efficient, stable, low-cost, and accurate ripeness and size grading of kiwifruit. The flexible sensing manipulator grasps the kiwifruit, detects the hardness of the kiwifruit by relying on tactile sensing, and determines the ripeness level based on the hardness. The size of the kiwifruit is also differentiated according to whether there is a significant change in the resistance of the topmost sensing unit of the flexible pressure sensor array. The 0, 1, 2, 3, 4, and 5 anomalies that may occur in actual production were tested and combined with machine learning KNN, SVM, and RF algorithms for data modeling and grading. The results show that the lowest accuracy of 0, 1, 2, 3, 4, and 5 possible outliers is 86.67% (KNN), 95.83% (SVM), and 92.5% (RF), respectively. KNN has the lowest classification effect, and SVM has the best. This study overcomes the drawbacks of inefficient destructive detection and unstable manual detection and makes up for the vulnerability of single machine vision to interference from environmental factors. This study can alleviate the challenges caused by fruit wastage and promote the sustainable production and consumption of the fruit industry chain.