In this paper we introduce a probabilistic approach of building extraction in remotely sensed images. To cope with data heterogeneity we construct a flexible hierarchical framework which can create various building appearance models from different elementary feature based modules. A global optimization process attempts to find the optimal configuration of buildings, considering simultaneously the observed data, prior knowledge, and interactions between the neighboring building parts. The proposed method is evaluated on various aerial image sets containing more than 500 buildings, and the results are matched against two state-of-the-art techniques.