The rapid growth of mobile Internet technologies has induced a dramatic increase in mobile payments as well as concomitant mobile transaction fraud. As the first step of mobile transactions, bankcard enrollment on mobile devices has become the primary target of fraud attempts. Although no immediate financial loss is incurred after a fraud attempt, subsequent fraudulent transactions can be quickly executed and could easily deceive the fraud detection systems if the fraud attempt succeeds at the bankcard enrollment step. In recent years, financial institutions and service providers have implemented rule-based expert systems and adopted short message service (SMS) user authentication to address this problem. However, the above solution is inadequate to face the challenges of data loss and social engineering. In this study, we introduce several traditional machine learning algorithms and finally choose the improved gradient boosting decision tree (GBDT) algorithm software library for use in a real system, namely, XGBoost. We further expand multiple features based on analysis of the enrollment behavior and plan to add historical transactions in future studies. Subsequently, we use a real card enrollment dataset covering the year 2017, provided by a worldwide payment processor. The results and framework are adopted and absorbed into a new design for a mobile payment fraud detectionsystem within the Chinese payment processor.