Rural China faces a shortage of safe drinking water. There are significant challenges associated with small centralized water supply systems and decentralized water supply systems. Many rural residents use point-of-use water treatment systems to ensure the safety of their drinking water. The actual water purification effect and applicability of these water treatment systems in small centralized and decentralized water supply systems need to be determined urgently. In this study, the water quality index (WQI) method was applied for the first time in rural areas to evaluate the effectiveness of point-of-use water treatment systems. A total of 67 reverse osmosis water treatment systems were tested. The rate of compliance with drinking water standards of the decentralized water supply was low (37%). Compared with untreated water, the use of a reverse osmosis water treatment system improved the rate of compliance with drinking water standards, but this effect was not satisfactory (8%). Among the factors potentially affecting the purified water quality of point-of-use water treatment systems (e.g., service time of the water treatment system, service time of the filter element, and type of water source), the service time of the filter element had the most significant influence on the water quality. If the filter element had not been replaced for a long period, the removal effect of the water treatment system on the total hardness, sulfide, copper, and ammonia nitrogen was significantly affected, reducing the quality of the purified water. To ensure the safety of drinking water, it is recommended to replace the filter element of a reverse osmosis water treatment system at least once a year.