For humans to understand the world around them, multimodal integration is essential because it enhances perceptual precision and reduces ambiguity. Computational models repli-cating such human ability may contribute to the practical use of robots in daily human living environments In this paper, we propose Multichannel Recurrent Kernel Machines (MC-RKM) for continuously generating a topological semantic map from multiple sensors. The proposed method consists of two hierarchical memory layers: i) Episodic Memory and ii) Semantic Memory layer. Each layer consists of one or more than one Infinite Echo State Network with a different learning task. The Episodic Memory layer incrementally clusters incoming sensory data as nodes and learns fine-grained spatiotemporal relationships of them. The Episodic Memory layer learning is in an unsupervised manner. The Semantic Memory layer utilizes task-relevant cues to adjust the level of architectural flexibility and generate a topological semantic map that contains more compact episodic representations. The generated topological semantic map represents the memory of the robot in which it is used for robot path planning and navigation.