In this paper, we propose a technique for improving the efficiency of hardware accelerators based on timing speculation (overclocking) and fault tolerance. We augment the accelerator with a lightweight error detection mechanism to protect against timing errors, enabling aggressive timing speculation. We demonstrate the validity of our approach for the convolution layers in Convolutional Neural Networks (CNN). We present an implementation of a fault-tolerant CNN accelerator combined with the lightweight error detection for convolution layers. The error detection mechanism we have developed works at the algorithm level, based on algebraic properties of the computation, allowing the full implementation to be realized using High-Level Synthesis tools. We use a set of Zybo boards to experimentally demonstrate that overclocking boosts the frequency by 17-36% with low chances of error, and that the infrequent errors can be detected with a negligible overhead (only 1000 LUTs).