The older population faces a high probability of experiencing age-related problems, such as osteoporosis, immobility, gait disturbances, stroke, Parkinson’s disease, and cognitive behavioral functional difficulties. Such problems negatively affect their lives. Thus, access to long-term care is a critical issue for older adults. In response to the aforementioned serious health issues, society must strive to provide a supportive and effective rehabilitation environment for older adults. This study designed an intelligent active and passive limb rehabilitation system to track and quantify the effectiveness of joint movements in patients automatically. The proposed method uses a camera and PoseNet to capture key feature information regarding human skeleton nodes and identify rehabilitation actions through joint movements. In addition, to solve the problem of joint occlusion during joint angle measurement, the designed system is equipped with a self-designed inertial measurement unit with GY-85 nine-axis sensors, which are mounted on the occluding part of the joints. A fuzzy inference system was developed to provide scores, suggestions, and encouragement for each rehabilitation session. This system also provides an interactive interface for users to monitor each rehabilitation session and examine whether rehabilitation is performed accurately.