The new mode for managing construction projects with information technology (IT) has attracted worldwide attention because it can help managers and workers perform tasks and bring potential benefits, such as high-quality products and accident-free production, effectively. However, the application of IT in site have not achieved expected results because it is faced with many constraints caused by internal factors from enterprises and projects and external factors from the government and environment. Although many relevant studies have discussed the constraints of implementing different IT and devices in the construction industry or site, few articles have specifically focused on identifying and analyzing the indicator system. In this work, we took China as the background, scientifically identified 23 influential factors that affect the implementation of IT in construction management through literature review and expert interviews. Subsequently, questionnaires were issued, and Delphi method was used to obtain empirical data that aimed at four different management fields. Then, an efficient and convenient method called DEMATEL was used to deal with these data. Afterward, the factors were divided into four categories, namely, core, diving, independent, and impact factors. Finally, the similarities and differences of the analysis results from the four fields were compared, and the key factors were identified. Results show that the cross-domain talent ability, concept and value cognition, and organization structure are core factors in all management fields that should be managed first along with the IT innovation ability in the enterprise. The formulation of technical standards and related device and training input are also critical in specific fields. Strategic planning plays a role in macro control and promotion. Data management and application, platform construction, solution, and collaboration have direct impacts on information management. The research results provide suggestions not only for the government to formulate effective policies for IT application and promotion in construction industry, but also for enterprises to take measures in improving management efficiency in the construction site and realizing its substantial benefits.