In the context of rough paths theory, we study the regularity of the Itô map with respect to the starting point, the coefficients and perturbation of the driving rough paths. In particular, we show that the Itô map is differentiable with a Hölder or Lipschitz continuous derivatives under general hypotheses. With respect to the current literature on the subject, our proof relies on perturbation of linear Rough Differential Equations, and the Hölder regularity of the Itô map is established.