Osteoporosis (OP) is a serious metabolic disease that, due to the increased number or function of osteoclasts, results in increased bone brittleness and, therefore, fragile fracture. Some recent studies report the importance of the transforming growth factor β (TGFβ) pathway in bone homeostasis. RepSox is a small molecule inhibitor of TGFβRI that has a wide range of potential application in clinical medicine, except OP. The aim of our study is to evaluate the effects of RepSox on the differentiation and bone resorption of osteoclasts in vitro and in vivo in an ovariectomy (OVX)-induced OP model. An initial analysis showed TGFβRI messenger RNA expression in both bone samples and bone cells. In the in vitro study, RepSox inhibited the receptor activator of nuclear factor κB ligand (RANKL)-induced osteoclast differentiation and bone resorption activity. Real-time polymerase chain reaction (PCR) analysis showed that RepSox suppressed osteoclastic marker gene expression in both dose-dependent and time-dependent manners. In addition, RepSox did not affect osteoblast differentiation, migration or osteoblastic-specific gene expression in vitro. Furthermore, western blot analysis indicated the underlying mechanisms of the RepSox suppression of osteoclastogenesis via the Smad3 and c-Jun N-terminal kinase/activator protein-1 (JNK/AP-1) signaling pathways. Finally, our animal experiments revealed that RepSox prevented OVX-induced bone loss in vivo. Together, our data suggest that RepSox regulates osteoclast differentiation, bone resorption, and OVX-induced OP via the suppression of the Smad3 and JNK/AP-1 pathways.