Rubberised concrete (RC) is a green and environmentally friendly concrete type that can be used as a sustainable production material. However, investigations on the structural applications of RC are limited due to its increased cost and decrease in strength. In this study, economical mixture designs were formulated with low-level replacements of rubber aggregates from recycled end-of-life tyres, as a prospective approach. The size effect with coarse (15 mm tyre shreds) and fine (2–4 mm tyre crumbs) rubber aggregates on the fresh and mechanical properties of structural concrete, specifically its fracture properties, was investigated, incorporating preliminary testing and constitutive models. The damage evaluation and characteristics of Mode I fracture behaviour of RC were investigated using three-point bending tests on notched beams. Using digital image correlation which possibly can be used to replace the conventional tools, the crack patterns on the beams were detected and crack mouth opening displacements were measured. The rubber replacements led to crack bridging effects. Overall, the results indicate that addition of recycled tyre aggregates at low percentages results in improved ductility and higher energy absorption and thereby improving the toughness of concrete, with marginal impacts on its strength and other properties. Furthermore, the selected mathematical models for compression and fracture can be used to evaluate the performance of RC structural elements with different rubber replacement levels. The modified RC can be potentially applied to structural components for earthquake resistance, integral bridges, or impact resistant road barriers.