Generative modeling of high-dimensional data is a key problem in machine learn- ing. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models combining the strengths of both models. Our contribution is to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder. In contrast, prior work on such hy- brid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our approach results in models with meaningful latent variable repre- sentations, and which rely on powerful autoregressive decoders to model image details. Our model generates qualitatively convincing samples, and yields state- of-the-art quantitative results.