Viremia rebound is a common but undesirable phenomenon for various diseases, especially for the Porcine Respiratory and Reproductive Syndrome (PRRS), a major concern for the swine industry. What causes some individuals to experience viremia rebound while others manage to steadily clear the virus has however been subject to much speculation. Hypotheses are the emergence of viral escape mutants, re-infection and differences in immune competence. To test this last hypothesis, we developed a mathematical model describing the within-host immune response to PRRS infection. We included the major mechanisms found to influence PRRS infection, as well as their regulations at the between-cell scale. We developed a rigorous ABC-like optimisation method to fit our model to an extensive set of experimental data, consisting of non-rebounder and rebounder viremia profiles. We then compared, between both profiles, the estimated parameter values, the resulting immune dynamics and the efficacy of the underlying immune mechanisms. Confronted to experimental data, our model successfully captured the between-host variations observed in viremia data, including rebounds. Moreover, we found that rebound was promoted by high apoptosis, high cell infection and low cytolysis by cytotoxic lymphocytes, while increasing neutralisation was very efficient to prevent rebounds. These results show that viremia rebound can occur as a result of differences in the immune competence alone and offers, for the first time, insights into potential causative immune mechanisms generating rebound.