General camera models relax the constraint on central projection and characterize cameras as mappings between each pixel and the corresponding projection rays. This allows to describe most cameras types, including classical pinhole cameras, cameras with various optical distortions, catadioptric cameras and other acquisition devices. We deal with the structure from motion problem for such general models. We first consider an hierarchy of general cameras first introduced in [28] where the cameras are described according to the number of points and lines that have a non-empty intersection with all the projection rays. Then we propose a study of the multi-view geometry of such cameras and a new formulation of multi-view matching tensors working for projection rays crossing the same 3D line, the counterpart of the fundamental matrices and the multifocal tensors of the standard perspective cameras. We also delineate a method to estimate such tensors and recover the motion between the views.