To reach human performance on complex tasks, akey ability for artificial systems is to understandphysical interactions between objects, and predictfuture outcomes of a situation. This ability, of-ten referred to asintuitive physics, has recentlyreceived attention and several methods were pro-posed to learn these physical rules from video se-quences. Yet, most of these methods are restrictedto the case where no, or only limited, occlusionsoccur. In this work we propose a probabilisticformulation of learning intuitive physics in 3Dscenes with significant inter-object occlusions. Inour formulation, object positions are modelledas latent variables enabling the reconstruction ofthe scene. We then propose a series of approx-imations that make this problem tractable. Ob-ject proposals are linked across frames using acombination of a recurrent interaction network,modeling the physics in object space, and a com-positional renderer, modeling the way in whichobjects project onto pixel space. We demonstratesignificant improvements over state-of-the-art inthe intuitive physics benchmark of Riochet et al.(2018). We apply our method to a second datasetwith increasing levels of occlusions, showing itrealistically predicts segmentation masks up to 30frames in the future. Finally, we also show resultson predicting motion of objects in real videos