This work presents a theoretical study of the Kernel Projection Machine, a classification algorithm which is an alternate to the Support Vector Machine (SVM). We propose to replace the Tikhonov regularization in the SVM by a penalized finite-dimensional projection. We show that a penalty term proportional to the dimension is appropriate under a ``gap'' hypothesis on the ditribution of class given observation. We also show how to apply this algorithm in practice and present examples of numerical results.