The near infrared (NIR) spectroscopy technique has wide applications in agriculture with the advantages of being nondestructive, sensitive, safe and rapid. However, there are still more than 40 error sources influencing the robustness and accuracy of its calibration and operation. Environmental, sample and instrument factors that influence the analysis are discussed in this review, including temperature, humidity and other factors that introduce uncertainty. Error sources from livestock products, fruit and vegetables, which are the most common objects in the field of NIR analysis, are also emphasized in the second part. In addition, studies utilizing different instruments, spectral pretreatments, variable selection methods, wavelength ranges, detection modes and calibration methods are tabulated to illustrate the complications they introduce and how they influence NIR analysis. It is suggested that large scale of data with abundant varieties can be used to build a more robust calibration model, in order to improve the robustness and accuracy of the NIR analytical model, and overcome problems caused by confining analysis to too many uniform samples.