The recent literature on visual recognition and image classification has been mainly focused on Deep Convolutional Neural Networks and their variants, which has resulted in a significant progression of the performance of these algorithms. Nevertheless, these recent advances should not conceal the fact that part-based models are expected to outperform approaches that code images as a whole, because of the flexibility such models offer. Based on this hypothesis, this article introduces a new algorithm for image recognition allowing to model image categories as a collection of distinctive parts, discovered automatically. These parts are matched across images while learning their visual model and are finally pooled to provide images signatures. The so-obtained parts are free of any appearance constraints and are optimized to allow the distinction between the categories to be recognized, in an optimal way. A key ingredient of the approach is a softassign-like matching algorithm that simultaneously learns the model of each part and automatically assigns image regions to the model's parts. Once the model of the category is trained, it can be used to classify new images by finding image's regions similar to the learned parts and encoding them in a single compact signature. The approach is experimentally validated by showing that using neural code as low-level image features allows to go beyond the performance given by Deep Convolutional Neural Networks, hence providing state-of-the art results on several publicly available datasets.