Interactive Painterly Rendering for Mobile Devices

. Painterly rendering is among the most popular non-photorealistic rendering techniques and has been employed in many applications. Research on painterly rendering mainly focuses on automatically generating artistic results. In this study, we aim to develop an entertaining and interactive application for painterly rendering with touchscreen mobile devices. The proposed application provides user interaction for added enjoyment and ensures high-quality painterly results. We provide a method for ﬁnding the appropriate position of the brush stroke around the points touched by the user and for generating and rendering the brush stroke. With the proposed method, users can quickly generate high-quality painterly results.


Introduction
Non-photorealistic rendering (NPR) [5,11,14] is an area of research in the field of computer graphics concerned with imitating the rules and practices of aesthetics and fine arts. In this field, research has attempted to simulate physical phenomena through the medium of the painting process and to mimic artwork styles. As a result of this research, a number of stylization methods are available that can plausibly imitate artistic styles. Moreover, with the increased use of mobile devices such as smartphones and tablet PCs, users can generate their own artwork using various mobile applications that employ these methods [2,1].
One common approach to NPR requires users manual control of the finished product in order to imitate artwork styles. Because this approach merely simulates physical phenomena [4,3], users must have a well-developed artistic sense or skill for drawing (or painting) akin to artists. However, it is difficult for most users to learn such artistic skills and develop an aesthetic sense.
Another NPR approach is to facilitate the style imitated by generating artistic results without the need for user expertise [7,6]. Most methods that adopt this approach automatically draw according to a particular painting style. Consequently, untrained users can obtain desirable results. However, users are almost completely uninvolved in the drawing process with this approach.
In order to design a popular NPR application, the latter approach is more suitable, insofar as a wider range of users will be able to utilize it. However, if users can only obtain results that are automatically generated, they will quickly lose interest in the application owing to limited user control. Therefore, an NPR approach in which users are involved in drawing process is needed, and this can be accomplished by providing the appropriate level of interaction without requiring pretrained skills.
In this paper, we propose an interactive algorithm for painterly rendering that runs on mobile devices. The proposed algorithm is based on previous methods [12], and it allows users to generate plausible painterly artwork easily. In Section 2, we describe our enhanced algorithm for painterly rendering. We present the results from evaluating our method in Section 3. Finally, we conclude with a summary of our method and discuss the directions for future development in Section 4.

Proposed Algorithm
Our study is based on [12]'s algorithm for painterly rendering. This algorithm automatically generates painterly images using a brush-stroke texture database and a photomosaic method [13,10,9]. However, users cannot participate in the painting process with [12]'s method, apart from choosing the input image and a limited number of rendering settings. To enhance user participation, we propose an interactive algorithm for painterly rendering in order to facilitate the direct involvement of the user during the painting process.

Basic Painterly Rendering algorithm
This study follows the multiple-layered-grids approach proposed by [7]. First, we generate several grid layers. For each grid on each layer, a brush stroke is rendered. Smaller-sized grids correspond to layers at the upper levels. Consequently, only tiny grids are generated for the top layer.
According to the method in [7], a brush stroke is generated for each grid along the gradient of the input image. For our proposed algorithm, however, we employ [12]'s method to determine the most suitable brush stroke from a stroke-texture database and map it to the canvas. First, we sample the color of the input image at a point selected by the user through a touch interface in order to allow the user to choose the color of the brush stroke. Next, we select a brush-stroke texture from the database and map it to the canvas with the sampled color. Then, we resize the size of the texture to 1.5 times that of the grid in order to cover the area of the grid. Finally, we calculate the color distance between the grid area of the canvas and that of the input image. By repeating this process for all of the strokes in the database and selecting the one with the smallest distance, we can find the most suitable brush stroke. We then render this brush stroke to the canvas according to the texture-mapping process described above.

Interactive Painterly Rendering algorithm
By using the method described in the previous section, we can obtain painterly rendering that is similar to the results from previous works [7,6,12]. Rather than performing the process automatically, we offer an interactive approach to the rendering. With our method, users can select the position of each brush stroke. If users are required to select all of the properties for the brush, such as the color, position, size, and shape, the process would be overly tedious for most untrained users. To alleviate such complexity, the size, texture, and color of the brush stroke are adaptively determined according to the position of the stroke selected by the user.
With our method, users input a point on the canvas by using a touch interface. The application then generates a circle with a predefined radius around that point and locates grids whose central points fall within this circle. Then, we eliminate the grids within the brush strokes that have already been generated. The remaining grids are candidates for the brush stroke that will be generated. By following the multilayered approach for rendering brush strokes according the size of the strokes, we select the largest grid among the candidates (Figure 1). If there are other grids that are equivalent in size to the largest grid, they are selected as well. In order to prevent the regularity of the brush location, we obtain the jittered point for the center of the selected grids and generate a brush stroke for that point using the method described in Section 2.1. When multiple grids are selected, we choose the most suitable brush stroke for rendering (i.e., the stroke with the minimum color distance).
If there are no more candidates within the circle generated at the point touched by the user, we then obtain the jittered point from the user-selected point and generate a grid with its center at the jittered point. Then, a brush stroke is generated using the method described above. Thus, the size of the grid is equivalent to the size of the grids at the top layer.
By iteratively performing the above process until the canvas is appropriately covered with a number of brush strokes, painterly results are generated interactively.

Results
Our experiments were conducted on an iPad 3. Figure 2 shows a comparison of the results from an automatic rendering approach, a manual approach, and our interactive approach. Table 1 summarizes the elapsed time for generating the results shown in Figure 2. For the automatic approach, we employed the basic method described in Section 2.1 and applied it in the order of each layer. With the manual approach, a user selected the size and orientation of the brush for each brush-stroke texture and rendered the brush stroke on a user-selected point using the iPads touch interface. For a given image, the user paints brush strokes by touching the screen until he/she wants to finish painting. The experiments demonstrate that the automatic approach rendered the image more quickly to obtain a complete painterly result. The quality of the result appeared to be satisfactory. However, the user does not participate in the rendering process with the automatic approach, apart from selecting a few parameters such as the grid size and the number of layers. In contrast, the manual approach requires a considerable amount of time to complete the painting. Moreover, the quality of the result appeared to be the worst, probably because the user was unfamiliar with the interface and the painting method. Our interactive approach offered a balance between these approaches in terms of the elapsed time. Moreover, it is evident in Figure 2 that the user was able to focus on several interesting points. Because the proposed interactive approach allowed the user to be involved in the painting process, the users preferences and intent were appropriately reflected in the result. Figure 3 shows various results from the proposed interactive method by untrained users.

Conclusion and Future Work
In this paper, we proposed an interactive method for painterly rendering with an enhanced mobile application based on previous methods [12]. To facilitate user interaction, we developed a method that finds the appropriate position of the brush stroke around a point selected by the user, after which the brush stroke is generated, and the image is rendered. With our method, users can generate high-quality painterly results quickly and easily.  Table 1. In future research, we plan to implement various styles of stroke-based rendering [8], such as watercolor, pen and ink, and pastel. To do so, we must reinforce the database to accommodate these various strokes and modify our rendering method to apply them appropriately.