Spectral beam split is attracting more attention thanks to the efficient use of whole spectrum solar energy and the cogenerative supply for electricity and heat. Nanofluids can selectively absorb and deliver specific solar spectra, making various nanofluids ideal for potential use in hybrid photovoltaic/thermal (PV/T) systems for solar spectrum separation. Clarifying the effects of design parameters is extremely beneficial for optimal frequency divider design and system performance enhancement. The water-based SiO₂ nanofluid with excellent thermal and absorption properties was proposed as the spectral beam splitter in the present study, to improve the efficiency of a hybrid PV/T system. Moreover, a dual optical path method was applied to get its spectral transimissivity and analyze the impact of its concentration and optical path on its optical properties. Furthermore, a PV and photothermal model of the presented system was built to investigate the system performance. The result indicates that the transimissivity of the nanofluids to solar radiation gradually decreases with increasing SiO₂ nanofluid concentration and optical path. The higher nanofluid concentration leads to a lower electrical conversion efficiency, a higher thermal conversion efficiency, and an overall system efficiency. Considering the overall efficiency and economic cost, the optimal SiO₂ nanofluid concentration is 0.10 wt.% (wt.%, mass fraction). Increasing the optical path (from 0 to 30 mm) results in a 60.43% reduction in electrical conversion efficiency and a 50.84% increase in overall system efficiency. However, the overall system efficiency rises sharply as the optical path increases in the 0–10 mm range, and then slowly at the optical path of 10–30 mm. Additionally, the overall system efficiency increases first and then drops upon increasing the focusing ratio. The maximum efficiency is 51.93% at the focusing ratio of 3.