On-line boosting is a recent breakthrough in the machine learning literature that has opened new possibilities in many diverse fields. Instead of generating a static strong classifier off-line, the classifier can be built on-the-fly on incoming samples. This has been succesfully exploited in treating computer vision tasks such as tracking as a classification problem thus providing an intriguing new perspective to an old subject. Known solutions to the on-line boosting problem rely on a fixed number of weak classifiers. The first main contribution of this paper removes this limitation and shows how a dynamic ensemble can better address the tracking problem by providing increased robustness. The second proposed novelty consists in a mechanism for detecting scale changes of tracked targets. Promising results are shown on publicly available and our own video sequences.