The use of reprogrammable hardware devices may lead to efficient, flexible and cheap neural network hardware implementations. Yet the area and connectivity constraints of FPGAs limit their use to rather small and already learned neural networks. A general method to implement both computing and learning of multilayer perceptrons of any size on FPGAs is described. A serial arithmetic, called on-line arithmetic, is used in order to improve parallelism despite the area constraints of the FPGA. A precise analysis of the computations required by the back-propagation algorithm allows us to maximize the parallism of our implementation. Our method is applied to the standard NetTalk benchmark, in order to show how large neural networks may be efficiently implemented on a single FPGA and a single memory chip.