We present in this paper a way to achieve positioning tasks by visual servoing under complex luminance variations. To do that, we use as visual features the luminance of all pixels in the image as we did in our previous work [Collewet ICRA08]. An important issue of this approach is that it does not rely at all on a any matching nor tracking process, contrary to all the approaches based on geometric visual features (points, straight lines, pose, homography, etc.). However, we consider in this paper a complete illumination model contrary to~ [Collewet ICRA08] where the temporal luminance constancy hypothesis was assumed. The main issue of this paper is thus the analytical computation of the interaction matrix related to the luminance from the phong illumination model. Experimental results on specular objects validate our approach.