As a core infrastructure of high-speed railways, ballast layers constituted by graded crushed stones feature noteworthy particle movement compared with normal railways, which may cause excessive settlement and have detrimental effects on train operation. However, the movement behavior remains ambiguous due to a lack of effective measurement approaches and analytical methods. In this study, an image-aided technique was developed in a full-scale model test using digital cameras and a color-based identification approach. A total of 1274 surface ballast particles were manually dyed by discernible colors to serve as tracers in the test. The movements of the surface ballast particles were tracked using the varied pixels displaying tracers in the photos that were intermittently taken during the test in the perpendicular direction. The movement behavior of ballast particles under different combinations of train speeds and axle loads was quantitatively evaluated. The obtained results indicated that the surface ballast particle movements were slight, mainly concentrated near sleepers under low-speed train loads and greatly amplified and extended to the whole surface when the train speed reached 360 km·h⁻¹. Additionally, the development of ballast particle displacement statistically resembled its rotation. Track vibration contributed to the movements of ballast particles, which specifically were driven by vertical acceleration near the track center and horizontal acceleration at the track edge. Furthermore, the development trends of ballast particle movements and track settlement under long-term train loading were similar, and both stabilized at nearly the same time. The track performance, including the vibration characteristics, accumulated settlement, and sleeper support stiffness, was determined to be closely related to the direction and distribution of ballast particle flow, which partly deteriorated under high-speed train loads.