In wind and solar renewable-dominant hybrid alternating current/direct current (AC/DC) power systems, the active power of high-voltage direct current (HVDC) system is significantly limited by the security and stability events caused by cascading failures. To identify critical lines in cascading failures, a rapid risk assessment method is proposed based on the gradient boosting decision tree (GBDT) and frequent pattern growth (FP-Growth) algorithms. First, security and stability events triggered by cascading failures are analyzed to explain the impact of cascading failures on the maximum DC power. Then, a cascading failure risk index is defined, focusing on the DC power being limited. To handle the strong nonlinear relationship between the maximum DC power and cascading failures, a GBDT with an update strategy is utilized to rapidly predict the maximum DC power under uncertain operating conditions. Finally, the FP-Growth algorithm is improved to mine frequent patterns in cascading failures. The importance index for each fault in a frequent pattern is defined by evaluating its impact on cascading failures, enabling the identification of critical lines. Simulation results of a modified Ningxia–Shandong hybrid AC/DC system in China demonstrate that the proposed method can rapidly assess the risk of cascading failures and effectively identify critical lines.