Malnutrition in early life increases the risk of osteoporosis, but the association of early-life undernutrition combined with adulthood obesity patterns with low-energy fracture remains unknown. This study included 5323 community-dwelling subjects aged ≥40 years from China. Early-life famine exposure was identified based on the participants’ birth dates. General obesity was assessed using the body mass index (BMI), and abdominal obesity was evaluated with the waist-to-hip ratio (WHR). Low-energy fracture was defined as fracture occurring after the age of 40 typically caused by falls from standing height or lower. Compared to the nonexposed group, the group with fetal, childhood, and adolescence famine exposure was associated with an increased risk of fracture in women with odds ratios (ORs) and 95% confidence intervals (CIs) of 3.55 (1.57–8.05), 3.90 (1.57–9.71), and 3.53 (1.05–11.88), respectively, but not in men. Significant interactions were observed between fetal famine exposure and general obesity with fracture among women (P for interaction = 0.0008). Furthermore, compared with the groups with normal BMI and WHR, the group of women who underwent fetal famine exposure and had both general and abdominal obesity had the highest risk of fracture (OR, 95% CI: 3.32, 1.17–9.40). These results indicate that early-life famine exposure interacts with adulthood general obesity and significantly increases the risk of low-energy fracture later in life in women.