We report a study of the Fermi surface of the chiral semimetal CoSi and its relationship to a network of multifold topological crossing points, Weyl points, and topological nodal planes in the electronic band structure. Combining quantum oscillations in the Hall resistivity, magnetization, and torque magnetization with ab initio electronic structure calculations, we identify two groups of Fermi surface sheets, one centered at the R point and the other centered at the Γ point. The presence of topological nodal planes at the Brillouin zone boundary enforces topological protectorates on the Fermi surface sheets centered at the R point. In addition, Weyl points exist close to the Fermi surface sheets centered at the R and the Γ point. In contrast, topological crossing points at the R point and the Γ point, which have been advertised to feature exceptionally large Chern numbers, are located at a considerable distance to the Fermi level. Representing a unique example in which the multitude of topological band crossings has been shown to form a complex network, our observations in CoSi highlight the need for detailed numerical calculations of the Berry curvature at the Fermi level, regardless of the putative existence and the possible character of topological band crossings in the band structure.