Milling force is key to the understanding of cutting mechanism and the control of machining process. Traditional milling force models have limited prediction accuracy due to their simplified conditions and incomplete knowledge contained for model construction. On the other hand, due to the lack of guidance from physics, the data-driven models lack interpretability, making them challenging to generalize to practical applications. To meet these difficulties, a deep network model guided by milling dynamics is proposed in this study to predict the instantaneous milling force and spindle vibration under varying cutting conditions. The model uses a milling dynamics model to generate data sets to pre-train the deep network and then integrates the experimental data for fine-tuning to improve the model’s generalization and accuracy. Additionally, the vibration equation is incorporated into the loss function as the physical constraint, enhancing the model’s interpretability. A milling experiment is conducted to validate the effectiveness of the proposed model, and the results indicate that the physics incorporated could improve the network learning capability and interpretability. The predicted results are in good agreement with the measured values, with an average error as low as 2.6705%. The prediction accuracy is increased by 24.4367% compared to the pure data-driven model.