The point vortex system is usually considered as an idealized model where the vorticity of an ideal incompressible two-dimensional fluid is concentrated in a finite number of moving points. In the case of a single vortex in an otherwise irrotational ideal fluid occupying a bounded and simply-connected two-dimensional domain the motion is given by the so-called Kirchhoff-Routh velocity which depends only on the domain.The main result of this paper establishes that this dynamics can also be obtained as the limit of the motion of a rigid body immersed in such a fluid when the body shrinks to a massless point particle with fixed circulation. The rigid body is assumed to be only accelerated by the force exerted by the fluid pressure on its boundary, the fluid velocity and pressure being given by the incompressible Euler equations, with zero vorticity. The circulation of the fluid velocity around the particle is conserved as time proceeds according to Kelvin's theorem and gives the strength of the limit point vortex. We also prove that in the different regime where the body shrinks with a fixed mass the limit dynamics is governed by a second-order differential equation involving a Kutta-Joukowski-type lift force. To prove these results, in a first step we reformulate the dynamics of the body in order to make more explicit different kind of interactions with the fluid. Precisely we establish that the Newton-Euler equations of translational and rotational dynamics of the body can be seen as a 3-dimensional ODE with coefficients solving an auxiliary problem for the fluid. When the circulation around the body is zero, this equation is a geodesic equation for a metric associated with the well-known ``added inertia'' phenomenon; with a nonzero circulation, an additional term similar to the Lorentz force of electromagnetism appears. Then, in the zero-radius limit, surprising relations between leading and subprincipal orders of various terms and modulation variables show up and allow us to establish a normal form with a gyroscopic structure. This leads to uniform estimates on the body's dynamics thanks to a modulated energy, and therefore allows us to describe the transition of the dynamics in the limit.