The development of a new zero-carbon thermal system is critical for achieving China’s energy transition and carbon neutrality goals. Heat pumps are a key enabling technology in this process, and their successful deployment hinges on two factors: identifying reliable low-temperature heat sources and ensuring that their electricity consumption is supplied by zero-carbon or low-carbon power. Based on China’s specific heating demands and resource endowments, this study proposes a construction pathway for zero-carbon thermal systems. For low-intensity heating demands—such as space heating in southern regions and rural buildings—air-source, ground-source, and surface water-source heat pumps should be employed to efficiently provide heat using electricity. For high-intensity heating demands—such as space heating in northern regions, low-pressure steam and hot water for industry—low-grade waste heat from human activities, including residual heat from nuclear and thermal power plants, industrial processes, and data centers, can be harvested as the low-temperature heat source and upgraded via heat pumps to meet supply needs. Building on this framework, the study presents an integrated roadmap for realizing zero-carbon thermal supply in China. It focuses on the technological architecture required to establish large-scale waste heat sharing networks, and analyses how coordinated operation between heating and power systems can contribute to the development of a new-type power system. Finally, it provides a comprehensive assessment of the investment requirements, economic returns, and carbon reduction potential of zero-carbon thermal infrastructure. This study recommends that the construction of zero-carbon thermal systems be prioritized and accelerated. Efforts should be advanced along two parallel pathways: (1) decentralized heat pumps deployed and retrofitted by end users, and (2) centralized waste heat sharing networks implemented through national-level planning. These should be supported by appropriate enabling policies and safeguards to fully unlock the economic and social benefits of zero-carbon thermal systems, thereby accelerating the low-carbon transformation of the energy system and the realization of carbon neutrality goals.