Background Cancer is a primary public health problem worldwide today. Tetrandrine (Tet) and fangchinoline (Fan) are extracted from the traditional Chinese medicine Stephania tetrandra S., which are two types of bis-benzyl isoquinoline alkaloids with anti-tumor activity. However, Tet and Fan are not suitable for clinical application due to their high effective doses. Results Through chemical modification, 19 novel Tet and Fan derivatives were firstly proposed and synthesized in this work, and they were assessed for the cytotoxicity against six tumor cell lines (H520, H460, HepG-2, HeLa, A549, and MCF-7) and human bronchial epithelial cells (BEAS-2b) by the CCK-8 assay. Compared to the parent Fan, ten derivatives demonstrated better cytotoxic effects, and 4b exhibited the strongest inhibitory activity against A549 cells growth (IC50 = 0.78 µM) and low toxicity against BEAS-2b cells (IC50 = 17.22 µM). Encouragingly, the IC50 value of compound 4b was 0.78 µM, demonstrating 3.53-fold, 10.22-fold and 12.13-fold more anticancer activity than HCPT, Tet and Fan, respectively. Subsequently, the anti-tumor mechanism of compound 4b was studied by hoechst 33258, flow cytometry, cell scratch, western blotting, and JC-1 staining. The results indicated compound 4b remarkably inhibited the cell migration by decreasing MMP-2 and MMP-9 expression and inhibited the proliferation of A549 cells by arresting the G2/M cell cycle. Meanwhile, compound 4b could also induce A549 cell apoptosis by promoting endogenous pathways of mitochondrial regulation. Subsequently, we simulated the drug stability of compound 4b in vivo, compound 4b showed a hydrolysis half-life of over 8 h in simulated gastric and intestinal fluids. In addition, in vivo experiments in nude mice presented that the growth of tumor tissues was markedly inhibited by the consumption of compound 4b in a dose-dependent manner, and the inhibition rate of the high-dose group (40 mg/kg) was greater than 50%. Regarding the anti-tumor mechanism of compound 4b, it was found that compound 4b could inhibit the mTOR/PI3K/AKT pathway in vivo. Furthermore, the binding mode of BCl-2 and Bax with compound 4b was investigated by performing a molecular docking study. Conclusions In summary, our present study unveiled the antitumor activities of 19 novel derivatives against the normal lung epithelial BESA-2b cells and six cancer cell lines in vitro. Furthermore, the following experiments illuminated that compound 4b could exert antitumor effects and induce apoptosis by regulating the PI3K/Akt/mTOR signaling pathway in vivo through triggering the mitochondria-mediated intrinsic pathways. This newly Fan derivative could exert antitumor effects in vivo and in vitro, which implied that compound 4b would be a prospective new anti-lung cancer drug for future development.