The primary aim of this study is to correlate the impact of aggregates, if any, on the viscoelastic behavior of rejuvenated asphalt mixtures containing very high amounts of reclaimed asphalt pavement (RAP) (> 50%). First, gradation of 100% RAP was rectified, using a modified Bailey method by adding virgin aggregates to achieve two coarse dense-graded and one fine dense-graded blends. Complex modulus test was then performed from −35 to +35 °C and 0.01–10 Hz. In addition to performance grade (PG) testing, extracted and recovered binders from different asphalt mixtures underwent shear complex modulus test within −8 °C to high temperature PG and frequencies from 0.001 to 30 Hz. Cole−Cole, Black space, complex modulus and phase angle master curves were constructed and Shift-Homothety-Shift in time-Shift (SHStS) transformation was used to compare the linear viscoelastic behavior of asphalt binders and mixtures. The influence of aggregates on the viscoelastic behavior of asphalt mixtures depends on temperature and/or frequency. The role of asphalt binders in the behavior of asphalt mixtures is more pronounced at high temperatures and the effect of the aggregate structure increases as the temperature falls. The maximum difference (60% to 70%) in the viscoelastic behavior of the binder and mixture based on SHStS transformed Cole−Cole curves is within the phase angle of 15°–20°.