The incorporation of commercial flame retardants into fiber-reinforced polymer (FRP) composites has been proposed as a potential solution to improve the latter’s poor flame resistance. However, this approach often poses a challenge, as it can adversely affect the mechanical properties of the FRP. Thus, balancing the need for improved flame resistance with the preservation of mechanical integrity remains a complex issue in FRP research. Addressing this critical concern, this study introduces a novel additive system featuring a combination of one-dimensional (1D) hollow tubular structured halloysite nanotubes (HNTs) and two-dimensional (2D) polygonal flake-shaped nano kaolinite (NKN). By employing a 1D/2D hybrid kaolinite nanoclay system, this research aims to simultaneously improve the flame retardancy and mechanical properties. This innovative approach offers several advantages. During combustion and pyrolysis processes, the 1D/2D hybrid kaolinite nanoclay system proves effective in reducing heat release and volatile leaching. Furthermore, the system facilitates the formation of reinforcing skeletons through a crosslinking mechanism during pyrolysis, resulting in the development of a compact char layer. This char layer acts as a protective barrier, enhancing the material’s resistance to heat and flames. In terms of mechanical properties, the multilayered polygonal flake-shaped 2D NKN plays a crucial role by impeding the formation of cracks that typically arise from vulnerable areas, such as adhesive phase particles. Simultaneously, the 1D HNT bridges these cracks within the matrix, ensuring the structural integrity of the composite material. In an optimal scenario, the homogeneously distributed 1D/2D hybrid kaolinite nanoclays exhibit remarkable results, with a 51.0% improvement in mode II fracture toughness (G_IIC), indicating increased resistance to crack propagation. In addition, there is a 34.5% reduction in total heat release, signifying improved flame retardancy. This study represents a significant step forward in the field of composite materials. The innovative use of hybrid low-dimensional nanomaterials offers a promising avenue for the development of multifunctional composites. By carefully designing and incorporating these nanoclays, researchers can potentially create a new generation of FRP composites that excel in both flame resistance and mechanical strength.