Soil contamination with heavy metal(loid)s threatens soil ecological functions, water quality and food safety; the latter is the focus of this review. Cadmium (Cd) and arsenic (As) are the toxic elements of most concern for food safety because they are relatively easily taken up by food crops. Rice is a major contributor of both Cd and As intakes to the Chinese population. Contamination and soil acidification are the main causes of high Cd levels in rice grains produced in some areas of southern China. The risk of Cd and As accumulation in food crops can be mitigated through agronomic practices and crop breeding. Liming is effective and economical at reducing Cd uptake by rice in acid soils. Paddy water management can produce opposite effects on Cd and As accumulation. Many genes controlling Cd and As uptake and translocation have been characterized, paving the way to breeding low accumulating crop cultivars through marker-assisted molecular breeding or genetic engineering. It is important to protect agricultural soils from future contamination. Long-term monitoring of anthropogenic additions and accumulation of heavy metal(loid)s in agricultural soils should be undertaken. Mass-balance models should be constructed to evaluate future trends of metal(loid)s in agricultural soils at a regional scale.