We present a first step towards anticipatory machine improvisation systems. The proposed system, based on fundamentals of music cognition, is a multi-agent memory-based collaborative and competitive reinforcement learning architecture, capable of live interaction with a musician or a music score. Results demonstrate the ability to model long-term stylistic planning and need for much less training data than reported in previous works.