In this paper, we propose a rejection strategy for convolutional neural network models. The purpose of this work is to adapt the network's topology in function of the geometrical error. A self-organizing map is used to change the links between the layers leading to a geometric image transformation occurring directly inside the network. Instead of learning all the possible deformation of a pattern, ambiguous patterns are rejected and the network's topology is modified in function of their geometric errors thanks to a specialized self-organizing map. Our objective is to show how an adaptive topology, without a new learning, can improve the recognition of rejected patterns in the case of handwritten digits.