Machining high-performance engineering materials, faces challenges including low machining efficiency, poor workpiece surface integrity, and rapid tool wear, which restrict high quality and efficient machining. Ultra-high-speed machining (UHSM) has been expected to address these issues. However, the material removal mechanisms involved in UHSM remain unclear and need further exploration. This paper reviews the criteria for evaluating the ductile and brittle behaviors of high-performance materials subjected to machining, as well as the developmental history of the material’s ductile-brittle transition induced by machining, proposing the concept of relativization of ductile-brittle property. Additionally, it further summarizes three typical material removal mechanisms: ductile-mode removal based on shear stress, brittle-mode removal based on tensile stress, and extrusion removal based on compressive stress, clarifying the universality of the brittle-mode removal. On this basis, this paper focuses on the discussion of the material removal mechanisms in UHSM, including high strain-rate-induced material embrittlement, UHSM-induced skin effect of damage, and the thermal effect in UHSM. Furthermore, it provides a detailed description of the typical characteristics of chip morphology in the ductile-brittle transition region (DBTR) under the high strain rate condition and, for the first time, elucidates the material removal mechanisms in the DBTR from a microstructural dislocation perspective, enriching the basic theory of UHSM. In the discussion section, it standardizes the definition for the UHSM, and explores the dislocation movement at high strain rates and the crack propagation in the UHSM. Finally, based on the current status of the UHSM technology, it summarizes the relevant research hotspots. For the first time, this paper brings up the brittle-mode removal mechanism under ultra-high-speed conditions, which is helpful to promote the UHSM for industrial applications.