Reducing irrigation water use by improving water use efficiency (WUE) in grain production is critical for the development of sustainable agriculture in the North China Plain (NCP). This article summarizes the research progresses in WUE improvement carried out at the Luancheng station located in the Northern part of NCP for the past three decades. Progresses in four aspects of yield and WUE improvement are presented, including yield and WUE improvement associated with cultivar selection, irrigation management for improving yield and WUE under limited water supply, managing root system for efficient soil water use and reducing soil evaporation by straw mulch. The results showed that annual average increase of 0.014 kg·m for winter wheat and 0.02 kg·m in WUE were observed for the past three decades, and this increase was largely associated with the improvement in harvest index related to cultivar renewal and an increase in chemical fertilizer use and soil fertility. The results also indicated that deficit irrigation for winter wheat could significantly reduce the irrigation water use, whereas the seasonal yield showed a smaller reduction rate and WUE was significantly improved. Straw mulching of summer maize using the straw from winter wheat could reduce seasonal soil evaporation by 30–40 mm. With new cultivars and improved management practices it was possible to further increase grain production without much increase in water use. Future strategies to further improve WUE are also discussed.