We propose a sparse aggregation framework for optical flow estimation to overcome the limitations of variational methods introduced by coarse-to-fine strategies. The idea is to compute parametric motion candidates estimated in overlapping square windows of variable size taken in the semi-local neighborhood of a given point. In the second step, a sparse representation and an optimization procedure in the continuous setting are proposed to compute a motion vector close to motion candidates for each pixel. We demonstrate the feasibility and performance of our two-step approach on image pairs and compare its performances with competitive methods on the Middlebury benchmark.