Adding rubber to concrete aims to solve the environmental pollution problem caused by waste rubber and to improve the energy absorption and impact resistance of concrete. In this paper, recycled rubber particles were used to replace fine aggregates in Portland cement concrete to combine the elasticity of rubber with the compression resistance of concrete. Fine aggregates in the concrete mixes were partially replaced with 0%, 20%, 40%, and 60% rubber by volume, and the cement in the concrete mixes was replaced with 0%, 5%, and 10% of silica fume by mass. The properties of the concrete specimens were examined through compressive strength, splitting tensile strength, flexural loading, and rebound tests. Results show that the compressive strength of concrete and the splitting tensile strength decreased to 11.81 and 1.31 MPa after adding silica fume to enhance the strength 37.8% and 23.7%, respectively, and the dosage of rubber was 60%. With the addition of rubber, the impact energy of rubberized concrete was 2.39 times higher than that of ordinary concrete, while its energy absorption capacity was 9.46% higher. The addition of silica fume increased its impact energy by 3.06 times, but the energy absorption capacity did not change significantly. In summary, the RC60SF10 can be used on non-load-bearing structures with high impact resistance requirements. A scanning electron microscope was used to examine and analyze the microstructural properties of rubberized concrete.