In this paper we describe a project that adopts a usercentered approach in the design of virtual environments (VEs) with enhanced realism and interactivity, guided by real-world applications in the areas of urban planning/ architecture and cultural heritage education. In what concerns realism, we introduce an image-based 3D capture process, where realistic models are created from photographs and subsequently displayed in a VR system using a high-quality, view-dependent algorithm. The VE is further enhanced using advanced vegetation and shadow display algorithms as well as 3D sound. A high degree of interactivity is added, allowing users to build and manipulate elements of the VEs according to their needs, as specified through a user task analysis and scenario-based approach which is currently being evaluated. This work is developed as part of the EU-funded research project CREATE.