We consider time-stepping methods for continuous interior penalty (CIP) stabilized finite element approximations of singularly perturbed parabolic problems or hyperbolic problems. We focus on methods for which the linear system obtained after discretization has the same matrix pattern as a standard Galerkin method. We prove that an iterative method using only the standard Galerkin matrix stencil is convergent. We also prove that the combination of the CIP stabilized finite element method with some known A-stable time discretizations leads to unconditionally stable and optimally convergent schemes. In particular, we show that the contribution from the gradient jumps leading to the extended stencil may be extrapolated from previous time steps, and hence handled explicitly without loss of stability and accuracy. With these variants, unconditional stability and optimal accuracy is obtained for first order schemes, whereas for the second order backward differencing scheme a CFL-like condition has to be respected. The CFL condition is related to the size of the stabilization parameter of the stabilized method but is independent of the diffusion coefficient.