Stationary anonymous sequential games with undiscounted rewards are a special class of games that combines features from both population games (infinitely many players) with stochastic games. We extend the theory for these games to the cases of total expected cost as well as to the expected average cost. We show that equilibria in the anonymous sequential game correspond to the limit of equilibria of related finite population games as the number of players grow to infinity. We provide many examples to illustrate our results.