Along with the sharp increase in visibility of the field, the rate at which new reinforcement learning algorithms are being proposed is at a new peak. While the surge in activity is creating excitement and opportunities, there is a gap in understanding of two basic principles that these algorithms need to satisfy for any successful application. One has to do with guarantees for convergence, and the other concerns the convergence rate. The vast majority of reinforcement learning algorithms belong to a class of learning algorithms known as stochastic approximation (SA). The objective here is to review the foundations of reinforcement learning algorithm design based on recent and ancient results from SA. In particular, it was established in [Borkar and Meyn, 2000] that both stability and convergence of these algorithms are guaranteed by analyzing the stability of two associated ODEs. Moreover, if the linearized ODE passes a simple eigenvalue test, then an optimal rate of convergence is guaranteed. This chapter contains a survey of these concepts, along with a survey of the new class of Zap reinforcement learning algorithms introduced by the authors. These algorithms can achieve convergence almost universally, while also guaranteeing optimal rate of convergence.