The contribution level of cross modality to vibration response of structure is governed by damp and modality. The larger the damping and the denser the modality, the higher the contribution. In engineering vibration analysis' a system hypothesis with lightly damping and sparse modality is generally applied to solve approximately vibration response of structure where the contribution of cross modality is neglected due to the contribution small enough. Approximate solution of displacement response can satisfy engineering requirement with certain precision but the error between approximate response and exact one, maybe tiny, can heavily effect the peak frequency and irregular factors of spectral diagram in time domain vibration analysis and change the results of fatigue damage accumulation identification. With a simple supported beam in this paper the contribution of cross modality to fatigue damage accumulation is investigated. Random response is obtained by means of superposition methods and fatigue damage accumulation of structure under random vibration is analysised with Miner's linear damage ruler. Numerical simulation shows that the contribution of cross modality to fatigue damage accumulation of structure can not be neglected even if the structure is characterized with lightly damping and sparse modality.