We present a set of methods to implement an implicit Finite Element solver on the GPU. In contrast to previous FEM implementations on the GPU which only address explicit time integration, our method allows large time steps for arbitrarily stiff objects. Unlike previous GPU-based sparse solvers, we avoid the assembly of the system matrix, and parallelize the matrix op- erations directly on the original object mesh. This considerably reduces the number of operations required, and more importantly the consumed band- width, enabling the method to be fast enough for highly complex interactive stiff body simulations. The presented methods can be applied in game and visual effects simulations, as well as medical and physics applications, where FEM is well established but currently limited by its computational cost. The core of the method can also be applied to many other scientific appli- cations where a large irregular sparse system of equations is solved using an iterative method.