Energy flexibility in district heating systems with the combination of different heating units can balance the energy system and contribute to the sustainability of future energy systems. However, there is lack of literature on the overview of energy flexibility in district heating. To fill the research gap, this paper firstly conducts a review study of district heating system components, markets and the energy flexibility potentials in district heating. The result of the literature review shows that there is no literature on stakeholders in the energy flexibility of smart district heating in relation to the integration of energy flexible buildings and smart technologies. Therefore, a survey is conducted to investigate the stakeholder's perception and motivation on smart district heating grids using energy flexible buildings. Based on the discussion with a modified Smart District Heating Architecture Model framework, our findings reveal that there seems to be a market for intelligent district heating concepts, the renewable energy use in district heating grids is important for stakeholders and there is flexibility potential in utilizing heat sources. However, there are technical, political and economic challenges in utilizing district heating, the energy flexibility market in district heating is not ready, and the social aspects regarding the intelligent use of district heating are not yet fully explored.