Memory and perception are two tightly interrelated cognitive processes, but the neural level of their interaction remains a matter of debate. Proponents of a late interaction emphasize feedback memory effects on visual processing, whereas others suggest that feed forward processing is affected by memory. In the visual domain, unconscious memory for stable relations among objects is known to influence visually-guided behavior. Recent evidence suggest an early interaction between this form of unconscious memory and visually-driven neural activity: the brain dissociates stable and unstable spatial relations at surprisingly early latencies, within the first 100 ms of sensory processing. The anatomical localization of this early effect however was still uncertain. In this study, we estimated the sources of the early effect in magnetoencephalographic (MEG) recordings, and analyzed intracranial electroencephalographic (iEEG) signal from seven epileptic patients in the modified version of the contextual cueing paradigm we recently developed. In spite of a lack of behavioral effect in the patient population, the striking agreement between the two electrophysiological datasets suggests that memory for spatial relations leads to differential responses in the anterior temporal lobe before 100 ms. The intracranial data further revealed orbitofrontal and more posterior temporal memory related activities around 100 ms. Altogether, the data point toward an early interaction between contextual memories and perceptual processing. The anterior temporal cortex, in particular appears to play a critical role in merging sensory processing with unconscious memory as soon as it gets activated.