There is a trend toward formalization of the rehabilitation process for older rehabilitants in a Challenging Rehabilitation Environment (CRE). This concept involves the comprehensive organization of care, support, and environment in rehabilitation wards. So far, literature about the principles of CRE is scarce. This study aims to explore the opinions of rehabilitants and informal caregivers regarding CRE, through a qualitative study between 2019 and 2020. Three telephone interviews were conducted with informal caregivers, and also 3 focus groups with 15 rehabilitants and 3 informal caregivers, all with recent experience in rehabilitation. Nine themes emerged regarding the rehabilitation process: (1) rehabilitant (attention for resilience, motivation, cognitive and emotional aspects), (2) rehabilitant centered (goal setting, physical and cognitive functioning and coping), (3) informal caregivers (involving and attention for resilience and relation), (4) communication (aligning the rehabilitation process), (5) exercise (increasing intensity by using task-oriented exercise, patient-regulated exercise, and group training), (6) peer support (learning experiences and recognition), (7) daily schedule (influence on the planning and activities outside therapy), (8) nutrition (energy for rehabilitation), and (9) eHealth (makes rehabilitation more challenging and fun). Regarding organizational processes, 4 themes were identified: (1) environmental aspects (single bedrooms, shared room for activities and therapy options on the ward), (2) staff aspects (small team with an emphatic supportive and motivating attitude), (3) organizational aspects (organized in an efficient way), and (4) return home (the discharge process should be well prepared for instance with home visits). Organizing excellent rehabilitation care requires a thorough understanding of the concept of CRE, as it is a complex and comprehensive concept that concerns the whole rehabilitation process. Its effectiveness and efficiency should be researched in prospective studies.