The significant weight of the building sector in global material demand has been addressed in several publications, but a lack of consideration of the energy-materials nexus was identified. To fill this gap, we developed a stock-flow dynamic model from 1950 to 2100, using scenarios of the International Energy Agency and the Shared Socioeconomic Pathways of the academic literature. We find that (i) the increasing stocks in the less developed countries results in a rise of the inflows; (ii) strong inequalities are observed in the in-use stocks per capita of materials in the SSPs, despite optimistic material intensities projection, (iii) a growing materials demand translates into larger final energy demand of material production, which could question the feasibility of some low-energy demand scenarios, and that (iv) the recycled concrete aggregated display a significant potential to enhance the reduction of accumulated concrete stocks in landfills and green concrete could decrease the energy demand of concrete production. We furthermore highlight the crucial temporal aspect of policies to successfully implement these solutions, as long lifetimes are observed in the building sector.