Fluorescence lifetime is usually defined as the average nanosecond-scale delay between excitation and emission of fluorescence. It has been established that lifetime measurement yields numerous indications on cellular processes such as inter-protein and intra-protein mechanisms through fluorescent tagging and Förster resonance energy transfer (FRET). In this area, frequency domain fluorescence lifetime imaging microscopy (FD FLIM) is particularly well appropriate to probe a sample non-invasively and quantify these interactions in living cells. The aim is then to measure fluorescence lifetime in the sample at each location in space from fluorescence variations observed in a temporal sequence of images obtained by phase modulation of the detection signal. This leads to a sensitivity of lifetime determination to other sources of fluorescence variations such as intracellular motion. In this paper, we propose a robust statistical method for lifetime estimation on both background and small moving structures with a focus on intracellular vesicle trafficking.