AR/VR-Based Live Manual for User-Centric Smart Factory Services

. Although several physical implementations of the smart factory have been introduced, a viable implementation or platform has not been proposed to make human workers perceive smart factory services more effectively and naturally with respect to task assistance. In this paper, we propose an AR/VR-based Live Manual, a new approach to provide workers with user-centric smart manufacturing services in IoT-enabled smart factory testbeds. The proposed live manual can make both on-site and remote workers do their tasks more effectively by utilizing and synchronizing AR with VR, which can help them to make better decisions. The synchronized dual view between AR and VR can help workers to understand the situation more easily and clearly depending on the worker’s contexts and manufacturing situations.


Introduction
One of the key themes in Industry 4.0 is to build a new manufacturing ecosystem (e.g., smart factory) through the fusion of smart manufacturing systems and information communication technology (ICT). The new manufacturing ecosystem refers to a customeroriented production system that can incorporate ICT technology into the existing manufacturing services to effectively and actively reflect the user's demands and requirements [1,2]. To manufacture products, various production processes such as machining, assembly and inspection should be performed [3,4]. In order to be more productive and safe while performing tasks, the worker must be able to accurately and effectively recognize the manufacturing situation and the operation status of related facilities. The operating conditions of the facilities change in real time due to internal factors such as facility failure and external factors such as production status or customer requirement change. For this reason, it is very difficult for the worker to recognize all working situations and to acquire all related knowledge. Therefore, it is essential to provide relevant and usercentric visual information to workers in a form appropriate to their working situations [5,6].
With the advent of smart augmented reality (AR) and virtual reality (VR) devices, AR and VR have been widely applied to a variety of fields such as entertainment, game and education. In particular, in manufacturing industries, AR and VR are considered to play an important role in providing more user friendly visual information to the worker for task assistance and monitoring [7][8][9]. However, it is almost impossible to provide user-oriented manufacturing services without considering the worker's location, the status of manufacturing facilities, and worker's role and situation [5,6,8].
In this paper, we propose an AR/VR-based Live Manual, a new approach to provide workers with user-centric manufacturing services regarding worker's contexts and situations by combining AR, VR, and deep learning in IoT-enabled smart factory testbeds. The proposed live manual can be executable in various mobile and wearable devices, and has been successfully applied to two smart factory testbeds. Section 2 overviews the proposed approach. Section 3 presents the proposed AR/VR-based Live Manual with some implementation results. Section 4 concludes the paper with some remarks.

Proposed AR/VR-based Live Manual
This paper proposes an AR/VR-based Live Manual to more effectively provide usercentric manufacturing services such as information visualization, monitoring, and sharing for task assistance and maintenance in the IoT-enabled smart factory environment. Fig. 1 presents the research goal and framework of the proposed live manual. We aim to develop an AR/VR-enabled mediator to workers in smart factory testbeds built in two cities such as Ulsan and Gumi, South Korea as shown in Fig. 1. The AR/VR-based Live Manual supports the visual augmentation of manufacturing information acquired from the related manufacturing execution system (MES) in real time into physical artifacts through AR/VR technology depending on the user context or situation. The deep learning technology has also been utilized to find the worker's location and to detect physical facilities in the factory to overcome the limitation of the conventional marker-based AR. The AV/VR Live Manual consists of five main modules: Human-Machine Interface, AR/VR Visualization, Detection and Recommendation, Information View Management, and Interaction with Information modules. Based on the five modules, the AV/VR Live Manual enables to track and recognize physical facilities and products using deep learning-based AR, to analyze the detected results and relevant manufacturing information retrieved from MES and working environment, and to embed the analyzed information into the worker's display of smart devices, rendered in AR and VR.

AR/VR-based Live Manual with Deep Learning-based Object Detection
One of the key features of the AR/VR Live Manual is to make workers do their tasks more effectively by combining and synchronizing AR and VR. Furthermore, the deep learning method is applied to support physical object detection and to suggest recommendation for task assistance by analyzing manufacturing information, user contexts and result of the object detection.
For the traditional marker-based AR, it is essential that the facility or its surrounding environment must have a planar surface to attach AR markers. To make worse, to support different perspective-based recognition, multiple markers must be attached, which is almost impossible in the real environment. For this reason, the object detection without markers is also supported by applying a deep learning method such as YOLO [10] as shown in Fig. 2. This method can overcome the inherent problem of the markerbased AR. Based on the detected facilities and the worker's context, the live manual can retrieve related manufacturing information, filters the relevant information required for the worker, and augments it into corresponding physical facilities or products through smart and wearable devices. For this purpose, another machining learning method with an auto encoder (AE) is used, which can recommend a necessary task or suggest an appropriate information depending on the worker's role [11].
Another key feature of the AR/VR Live Manual is to provide a synchronized dual view in addition to the main AR view. In particular, the dual view can be switched, and the VR can be the main view if the worker needs alternative information than augmented information, which can complement the AR view. Therefore, the worker can more effectively monitor and analyze the manufacturing information depending on the situation of the working environment. Fig. 3 presents the implementation of the AR/VR-based Live Manual in two different modules of the smart factory testbeds. Fig. 3. Implementation of AR/VR-based Live Manual using handheld devices that has been applied to two different smart factory modules; the live manual supports dual views that can complement AR and AR as they are visually synchronized each other.
We have developed the AR/VR-based Live Manual for providing user-centric manufacturing services, which has been applied to the two real smart factory testbeds built in two cities. The two testbeds can produce different types of BLDC motors and handpieces, respectively. In addition to the AR-based task assistance with the dual view and the deep machine learning, the proposed approach can also support different functionalities such as VR-based smart factory simulation with a high immersion using VR HMDs such as HTC Vive TM , world-in-miniature (WIM)-based visualization, mixed reality (MR)-based visualization and monitoring of physical artifacts using wearable glasses such as MS HoloLens TM . In particular, when a remote collaboration is performed, it is possible to interoperate with the on-site worker with the AR module. Using different level of functionalities, on-site workers and remote workers can monitor realtime manufacturing information more easily and perform required tasks more effectively.

Conclusion
In this paper, we proposed the AR/VR-based Live Manual, a unified approach to provide human workers with smart factory services in IoT-enabled testbeds. The proposed live manual system can be executable in various smart and wearable devices by taking advantage of combining and synchronizing AR and VR. In addition, it can make workers do their tasks more effectively as the proposed approach can provide situationadaptive visualization with respect to the worker's surrounding contexts and manufacturing situations. The dual view conversion between AR and VR helps on-site workers to switch the live manual to either VR-or AR-focused mode using smart and wearable devices. We have also developed several AR/VR-based live manual modules using smart glasses. We are currently designing a more concrete platform for supporting various AR/VR-based task assistance. We are also evaluating the usability and usefulness of the proposed approach.