High-precision and efficient structural response prediction is essential for intelligent disaster prevention and mitigation in building structures, including post-earthquake damage assessment, structural health monitoring, and seismic resilience assessment of buildings. To improve the accuracy and efficiency of structural response prediction, this study proposes a novel physics-informed deep-learning-based real-time structural response prediction method that can predict a large number of nodes in a structure through a data-driven training method and an autoregressive training strategy. The proposed method includes a Phy-Seisformer model that incorporates the physical information of the structure into the model, thereby enabling higher-precision predictions. Experiments were conducted on a four-story masonry structure, an eleven-story reinforced concrete irregular structure, and a twenty-one-story reinforced concrete frame structure to verify the accuracy and efficiency of the proposed method. In addition, the effectiveness of the structure in the Phy-Seisformer model was verified using an ablation study. Furthermore, by conducting a comparative experiment, the impact of the range of seismic wave amplitudes on the prediction accuracy was studied. The experimental results show that the method proposed in this paper can achieve very high accuracy and at least 5000 times faster calculation speed than finite element calculations for different types of building structures.