The evolution of a star is influenced by its internal rotation dynamics through transport and mixing mechanisms, which are poorly understood. Magnetic fields can play a role in transporting angular momentum and chemical elements, but the origin of magnetism in radiative stellar layers is unclear. Using global numerical simulations, we identify a subcritical transition from laminar flow to turbulence caused by the generation of a magnetic dynamo. Our results have many properties of the theoretically proposed Tayler-Spruit dynamo mechanism, which strongly enhances transport of angular momentum in radiative zones. The dynamo generates deep toroidal fields that are screened by the stellar outer layers. This mechanism could produce strong magnetic fields inside radiative stars without an observable field on their surface.