Passive residual heat removal heat exchanger (PRHR HX), which is a newly designed equipment in the advanced reactors of AP1000 and CAP1400, plays an important role in critical accidental conditions. The primary and secondary side coupling heat transfer characteristics of the passive residual heat removal system (PRHRS) determine the capacity to remove core decay heat during the accidents. Therefore, it is necessary to investigate the heat transfer characteristics and develop applicable heat transfer formulas for optimized design. In the present paper, an overall scaled-down natural circulation loop of PRHRS in AP1000, which comprises a scaled-down in-containment refueling water storage tank (IRWST) and PRHR HX models and a simulator of the reactor core, is built to simulate the natural circulation process in residual heat removal accidents. A series of experiments are conducted to study thermal-hydraulic behaviors in both sides of the miniaturized PRHR HX which is simulated by 12 symmetric arranged C-shape tubes. For the local PRHR HX heat transfer performance, traditional natural convection correlations for both the horizontal and vertical bundles are compared with the experimental data to validate their applicability for the specific heat transfer condition. Moreover, the revised natural convection heat transfer correlations based on the present experimental data are developed for PRHR HX vertical and lower horizontal bundles. This paper provides essential references for the PRHRS operation and further optimized design.