Typical effects of coarse and fine aggregates on the long-term properties of sea sand recycled aggregate concrete (SSRAC) are analyzed by a series of axial compression tests. Two different types of fine (coarse) aggregates are considered: sea sand and river sand (natural and recycled coarse aggregates). Variations in SSRAC properties at different ages are investigated. A novel test system is developed via axial compression experiments and the digital image correlation method to obtain the deformation field and crack development of concrete. Supportive results show that the compressive strength of SSRAC increase with decreasing recycled coarse aggregate replacement percentage and increasing sea sand chloride ion content. The elastic modulus of SSRAC increases with age. However, the Poisson’s ratio reduces after 2 years. Typical axial stress–strain curves of SSRAC vary with age. Generally, the effect of coarse aggregates on the axial deformation of SSRAC is clear; however, the deformation differences between coarse aggregate and cement mortar reduce by adopting sea sand. The aggregate type changes the crack characteristics and propagation of SSRAC. Finally, an analytical expression is suggested to construct the long-term stress–strain curve of SSRAC.