Continental topography is dominantly controlled by a combination of crustal thickness and density variations. Nevertheless, it is clear that some additional topographic component is supported by the buoyancy structure of the underlying lithospheric and convecting mantle. Isolating these secondary sources is not straightforward, but provides valuable information about mantle dynamics. Here, we estimate and correct for the component of topographic elevation that is crustally supported to obtain residual topographic anomalies for the major continents, excluding Antarctica. Crustal thickness variations are identified by assembling a global inventory of 26,725 continental crustal thickness estimates from local seismological data sets (e.g., wide‐angle/refraction surveys, calibrated reflection profiles, receiver functions). In order to convert crustal seismic velocity into density, we develop a parametrization that is based upon a database of 1,136 laboratory measurements of seismic velocity as a function of density and pressure. In this way, 4,120 new measurements of continental residual topography are obtained. Observed residual topography mostly varies between ±1 and 2 km on wavelengths of 1,000–5,000 km. Our results are generally consistent with the pattern of residual depth anomalies observed throughout the oceanic realm, with long‐wavelength free‐air gravity anomalies, and with the distribution of upper mantle seismic velocity anomalies. They are also corroborated by spot measurements of emergent marine strata and by the global distribution of intraplate magmatism that is younger than 10 Ma. We infer that a significant component of residual topography is generated and maintained by a combination of lithospheric thickness variation and sub‐plate mantle convection. Lithospheric composition could play an important secondary role, especially within cratonic regions. Plain Language Summary: Plate tectonic theory holds that the horizontal movement of tectonic plates drives crustal thickening and the formation of mountain ranges where plates collide, and crustal thinning in rift basins where plates diverge. The dominance of this process, however, obscures a key secondary source of topographic variation caused by density variations within the underlying mantle. In this study, we compile two new databases. The first is a global compilation of crustal thickness and seismic velocity measurements obtained from seismic experiments. The second is a compilation of laboratory analyses of seismic velocity in crustal rocks as a function of temperature, pressure and density. We combine these data sets to accurately map that component of continental topography caused by crustal thickness and density variations and use it to isolate the secondary, residual component supported by the mantle. We find that a substantial proportion of topography is supported by the temperature and chemical structure of the mantle and is manifest as topographic swells and basins that can occur within interiors of tectonic plates, have amplitudes of up to 2 km, and length scales ranging from hundreds to thousands of kilometers. Independent geologic evidence suggests that some of these topographic features developed over million‐year time scales. Key Points: Continental residual topography estimated using 26,725 crustal thickness and 4,067 seismic velocity constraintsCrustal velocity to density conversion scheme developed to correct for crustal density variationsResidual topography varies from ∼1 to 2 km over ∼1,000 km wavelengths, consistent with a range of geological observations [ABSTRACT FROM AUTHOR]