Arterial stiffness (arteriosclerosis) has been linked to heightened risks for cognitive decline, and ultimately for Alzheimer's disease and other forms of dementia. Importantly, neurovascular outcomes generally vary according to one's biological sex. Here, capitalizing on a large sample of participants with neuroimaging and behavioral data (N = 203, age range = 18–87 years), we aimed to provide support for a hierarchical model of neurocognitive aging, which links age-related declines in cerebrovascular health to the rate of cognitive decline via a series of intervening variables, such as white matter integrity. By applying a novel piecewise regression approach to our cross-sectional sample to support Granger-like temporal inferences, we show that, on average, a precipitous decline in cerebral arterial elasticity (measured with diffuse optical imaging of the cerebral arterial pulse; pulse-DOT) precedes an acceleration in the development of white matter lesions by nearly a decade, with women protected from these deleterious effects until approximately age 50, the average onset of menopause. By employing multiple-mediator path analyses while controlling for sex, we show that age may impair cognition via the sequential indirect effects of arteriosclerosis and white matter atrophy on fluid, but not crystallized, abilities. Importantly, we replicate these results using pulse pressure, an independent index of arterial health, thereby providing converging evidence for the central role of arteriosclerosis as an accelerating factor in normal and pathological aging and identifying robust sex-related differences in the progression of cerebral arteriosclerosis and white matter degradation.