Abstract Background Photo-neutrons are produced at the head of the medical linear accelerators (linac) by the interaction of high-energy photons, and patients receive a whole-body-absorbed dose from these neutrons. The current study aimed to find an efficient shielding material for fast neutrons. Methods Nanoparticles (NPs) of Fe3O4 and B4C were applied in a matrix of silicone resin to design a proper shield against fast neutrons produced by the 18 MeV photon beam of a Varian 2100 C/D linac. Neutron macroscopic cross-sections for three types of samples were calculated by the Monte Carlo (MC) method and experimentally measured for neutrons of an Am-Be source. The designed shields in different concentrations were tested by MCNPX MC code, and the proper concentration was chosen for the experimental test. A shield was designed with two layers, including nano-iron oxide and a layer of nano-boron carbide for eliminating fast neutrons. Results MC simulation results with uncertainty less than 1% showed that for discrete energies and 50% nanomaterial concentration, the macroscopic cross-sections for iron oxide and boron carbide at the energy of 1 MeV were 0.36 cm− 1 and 0.32 cm− 1, respectively. For 30% nanomaterial concentration, the calculated macroscopic cross-sections for iron oxide and boron carbide shields for Am-Be spectrum equaled 0.12 cm− 1 and 0.15 cm− 1, respectively, while they are 0.15 cm− 1 and 0.18 cm− 1 for the linac spectrum. In the experiment with the Am-Be spectrum, the macroscopic cross-sections for 30% nanomaterial concentration were 0.17 ± 0.01 cm− 1 for iron oxide and 0.21 ± 0.02 cm− 1 for boron carbide. The measured transmission factors for 30% nanomaterial concentration with the Am-Be spectrum were 0.71 ± 0.01, 0.66 ± 0.02, and 0.62 ± 0.01 for the iron oxide, boron carbide, and double-layer shields, respectively. In addition, these values were 0.74, 0.69, and 0.67, respectively, for MC simulation for the linac spectrum at the same concentration and thickness of 2 cm. Conclusion Results achieved from MC simulation and experimental tests were in a satisfactory agreement. The difference between MC and measurements was in the range of 10%. Our results demonstrated that the designed double-layer shield has a superior macroscopic cross-section compared with two single-layer nanoshields and more efficiently eliminates fast photo-neutrons.