Construction of Mixed Reality Story Environment Based on Real Space Shape

. We propose a system to construct Mixed Reality (MR) environments based on actual environments. Our goal is to develop MR content in which users and characters can coexist in an actual room and experience a story. This study represents a preliminary step in the development of such an MR experience. We proposed a method to construct an MR environment that uses three-dimensional information acquired from the actual environment based on rules. We extract valid plane information from three-dimensional information and use it. Virtual objects used for MR content can be placed in real space automatically. Rules are created by specifying the area and height of the plane of the real space. In this study, we placed seven types of objects in two types of rooms and confirmed that it is possible to construct an environment using a transmission-type head mount-ed display. However, under the current system, users cannot experience a story.


Introduction
Storytelling content allows users to experience various stories through a character's active behaviors [1][2][3]. By matching a real space, the user can feel as if the character is in the same world. If the user can experience an Mixed Reality (MR) story of their room as a character world setting, this may increase the reality of the experience. It is thought that interaction, such as being able to change to the story through room remodeling, is possible. For example, when routes to obtain items differ depending on the shape of the room or the furniture arrangement, it is possible to use furniture to place a bridge (Fig. 1). Everything in the room can be used as a tool to experience the content. This study proposes a method to build an MR environment tailored to the room in preparation for developing a system that can realize an experience in which the user senses that they are living in the same room as the character. (center) Creating a road using furniture.

Related work
One study into the story experience is Steven's AR Façade [4], which explored communicating with characters and experiencing a story in real space using a transparent HMD based on Facade [3]. Façade is the story content in the house where the character lives. The AR Façade reproduces the same placement in real space. Nakevska used the CAVE environment and projected images onto the walls of a room in order to perform the story experience of Alice in Wonderland [5]. In a room, the user can experience a story by acting on objects in the room or projected images; however, such methods cannot be incorporated into a user's own room because such a real environment is special. Another method sets a virtual object that matches a real environment measured using a depth camera to create a character world tailored to an arbitrary room. Misha's Oasis [6] detects a walkable region from three-dimensional information of walls and floors to build a VR space adapted to that region. That is not story content. The proposed method also measures a real environment. With the proposed method, we aim to create MR story experiences in arbitrary environments by setting virtual objects according to measured information.

3
Construction of Mixed Reality space

Overview
We construct an MR space that matches the shape of the three-dimensional data of a real space acquired by scanning. We define the floor, tables, etc. as a "horizontal plane," walls, shelves, etc. as a "vertical plane," and inclined planes as a "slope." We define this classification as a "region." We place a CG object on the surface classified into each region based on the rule. A CG object placed in a certain region is defined as a "stage." In this procedure, three-dimensional data of the physical space are classified into each region, and the stage is arranged in a region according to the rule.

Obtaining surface meshes
Note that the mesh shape of the obtained three-dimensional data is incomplete, and there are extra meshes and holes caused by the ceiling and measurement error. Data resampling is performed because it is difficult to judge the appropriate region. Note that subsequent processes can be simplified by acquiring data as a single plane mesh. We performed data resampling so that we could handle only the confirmed surface from the top of the room model (Fig. 2). First, we sufficiently cover the room model and prepare a fine rectangular mesh relative to the room model mesh. Move from the top of the room model to the bottom and fix each vertex at the point where it hits the mesh of the room model. Thus, it is possible to acquire a mesh of the surface.

Fig. 2.
Retrieving the surface mesh by resampling.

Region classification
To classify the re-sampled three-dimensional data into each region, clustering is performed based on the inner product of the normal vectors of the mesh vertices. The inner product of the normal vectors of adjacent vertices is taken to find the angle between the normal vectors. Clustering performs vertices with angles less than a threshold as the same region. To make vertices classified in each cluster a mesh, a Delaunay triangulation is created from the group of vertices. Thus, it is possible to divide the three-dimensional data of the real space into planes. The plane is estimated using the RANSAC algorithm on the divided planes, and the inclination of the plane is obtained from the estimated plane equations. Each region is classified using the inclination and area of the plane (Table 1). For visibility, we change the color according to the region (Fig. 3).  The placement rule is determined by combining these. When multiple stages satisfy the condition, they are placed in the same region. In addition, the placement position is determined randomly within a range not outside a region but also not in contact with another mesh; however, in the case of a very narrow area, it does not depend on it.

Result of Placement
Stages (Fig. 4) were placed in two real environments (room1 and room2) according to the rules ( Table 2 and Table 3). The scan data are shown in Fig. 5, and the placement results are shown in Fig. 6. Note that they were also confirmed using Microsoft Ho-loLens (Fig. 7).

Discussion and Future Work
The purpose of this study was to experience MR in an arbitrary environment by scanning a real space. Note that we did not achieve completely automatic placement; however, we were able to place objects according to the environment. Note that not all components of this system work automatically. For example, parameters must be changed according to the given environment. Therefore, it is necessary to acquire an appropriate threshold and automatically change the shape and size of the model at the time of placement according to the given environment. In addition, this work represents an incomplete implementation of an environment construction system; thus, this work does not represent a method that can provide a true story experience. Therefore, it is necessary to implement a story system that changes with user interactions and the environment. We think that it is possible to experience an MR story as if the character actually lives with the user by enhancing the completeness of the system.