Analyzing Human Robot Collaboration with the Help of 3D Cameras

. Recent developments in robotics allow the design of work systems with enhanced human robot collaboration (HRC) for assembly tasks. Productivity improvements are a common aim for companies that look into the implementation of HRC. To harvest the full productivity potential of these work systems, an analysis of the HRC work processes is essential. However, a dedicated method for the analysis of productivity in HRC is missing. This paper presents an approach using 3D-cameras to observe the employee in HRC. The approach links this information to robot states. The resulting analysis aims at improving the productivity of the work system e.g. by identifying and reducing balancing losses in HRC. The method tracks the movements of the employees in the HRC area and matches the corresponding robot activities.


Introduction
Smart manufacturing has a huge impact to on the design and operation of manufacturing systems and accordingly the work of employees in manufacturing.Romero et al. [1] identify 8 types of technology induced employee augmentation.One aspect of this typology is collaborative automation or "the collaborative operator".Recent developments in robotics allow the design of human robot collaboration work systems with a very deep integration of robot activities in manual operations.As described by Romero et al. [1], HRC can transfer tedious, non-ergonomic and vulnerable tasks to the robot and achieve productivity and quality improvements as well as an increased operator satisfaction.Of the manifold reasons for the implementation of HRC, improvements in productivity are most important for many companies [2].To ensure a high productivity in HRC work systems, an adequate analysis of work processes is essential.Many approaches exist to analyze the productivity of manual work (work sampling, time studies) as well as of machines (overall equipment efficiency) [3,4].A dedicated approach to analyze HRC work systems that supports continuous improvement actions is missing.This paper presents an approach to analyze HRC work systems with the help of 3Dcameras and a robot interface.The camera tracks the position and the movement of the employees and a software matches corresponding robot data to create a digital twin of the HRC work system.The recorded data is analyzed with the help of state type diagrams as proposed by Schweitzer [5].

Collaborative robots and human robot collaboration
When referring to collaborative robots or cobots, many authors describe robots that are capable of working within the same environment as human employees without external safety barriers such as fences [6].In order to do this, these robots usually provide enhanced safety features like rounded edges, reduced weight, limited power and force and protective skins that can detect collisions or the proximity of the employee.In addition to these safety requirements, Romero et al. [1], Maurtua et al. [6] and Wang et al. [7] refer to enhanced interaction mechanisms as necessary feature for HRC.Many authors describe HRC regarding the temporal and spatial relation of collaborating humans and robots.Human robot collaboration is assumed to have at least partially no spatial and no temporal separation of human and robot.This includes simultaneous and supportive work of human and robot on the same workpiece at the same time [2,7,8].In addition to these forms of collaboration, employees and robots may also work independent when completing tasks with spatial separation or with temporal separation.
Deploying HRC in a way that increases productivity is a complex design task [2].While the initial design and task allocation is addressed by some authors (for instance in [9]), a methodology that supports analysis of work in the operation of HRC for continuous improvement is missing.

3
Productivity and work analysis

Productivity and conventional work analysis
In the context of HRC, both the productivity of the employee (labor productivity) and of the robot (machine productivity) are important.Due to the deep integration of human and robot activities, breakdowns or waiting times of either the robot or the employee can lead to losses in the overall productivity of the work system.While measurement of machine productivity can be automated in many cases, the vast majority of existing methods for assessing labor productivity uses observers to analyze the work processes (see [3]).In order to draw conclusions, states of robot and operator have to be captured simultaneously.This usually exceeds the cognitive capabilities of an observer, also because the robot state is not obvious (especially when deploying external equipment) and robot control information is not available for the observer.

Analysis of work processes with 3D cameras
Motion capturing systems track human positions and movements and make this data available for further processing.These systems can be used to analyze work processes without the need of an observer.Both electro-mechanical systems that use special suits to determine human motions and optical systems as 3D cameras can be used to track human movements.3D cameras often emit an infrared light, which is reflected by the human body or special markers and captured by a stereoscopic sensor set.Optical systems require special software to detect human bodies.While electro-mechanical systems usually provide a superior data quality, the additional equipment is often expensive, has to be carefully attached to the employee and may impede the work process.Optical systems often suffer from a lower data accuracy, especially in environments that partly conceal the observed employee [10].We opted for 3D-cameras, because their accuracy is good enough for the purpose of productivity analysis and because they are easy to use.Various approaches have been developed to analyze work processes in regards to productivity with the help of optical systems.Escoria et al. [11] developed a method to detect characteristic work processes on construction sites on the basis of specific body positions.Ying et al. [12] propose an approach that uses convolutional neural networks in order to detect different activities of workers in a scaffolding process with the help of regular cameras.Benter [10] uses the Microsoft Kinect sensor and the corresponding software set to determine stopping points of the whole body or single body joints.Benter uses these stopping points to distinguish different movements during manual processes, which enables him to automatize a detailed productivity analysis of manual assembly processes.
While the existing approaches provide methods for the analysis of manual work processes, they do not integrate machine or robot data in the analysis.

4
Analysis of human robot collaboration work systems

Methodology
To analyze the work processes of HRC work systems, our approach aims at mapping states of employees and robots in the work system on a synchronized timeline with the help of state type diagrams.The methodology aims to reveal situations that will reduce the work systems productivity and result from the collaboration of employee and robot such as:  balancing losses that result from an unbalanced task duration of robot and employee,  breakdowns of the robot or related peripheral systems,  disturbances in manual processes,  speed losses and minor stops that result from the close proximity of employee and robot.
Balancing losses occur if either the robot or the employee needs to wait for the counterpart.Consequently, balancing losses either decrease the robot or employee productivity.
Breakdowns of the robot or a related system will force the employee to repair the disturbed system or to wait for an expert to resolve the issue.Breakdowns therefore cause productivity losses for employees and robots.
Disturbances in manual work processes will force the employee to resolve the disturbances while the robot has to wait for the employee in order to proceed.Thus these situations will cause productivity losses for employee and robots.
Due to safety reasons, collaborative robots will stop if they get in contact with the employee or will work with decreased speed in their close proximity.This is inevitable if robot and employee are about to collaborate.However, these situations may also occur if robot and employee work independently in close proximity.Those situations should be avoided by a better workplace design.

Exemplary work system
In order to illustrate the idea of the analysis, an exemplary HRC work system for research purposes was developed at Garz and Fricke GmbH in Hamburg.The work system consists of a manual work station and a robot work station for collaborative processes and of two handling points where employees take materials for processing or place already processed materials.In this case a Universal Robots UR 10 robot is used.Figure 1 shows the set-up.

Fig. 1. Set-up of the HRC work system
In this exemplary work system, touch display components are glued to plastic frames.This involves the preparation and manual inspection of the frame, the cleaning of the components with ISO-propanol, activation of the frames surface with a plasma activator, application of the glue and the joining and adjustment of the components.The preparation and manual inspection is performed by the employee as this involves the handling of small parts.Due to the higher repetitive accuracy, cleaning, surface activation and the application of glue is performed by the robot.Joining and adjustment of the components is done by employee and robot in a collaborative step.The exemplary set up therefore contains independent, synchronized and collaborative work steps.

Relevant states
Schweitzer [5] suggests an analysis approach for concatenated production lines consisting of states and state diagrams.These approach uses the states regular activity (tact), disturbance, waiting, blocking and set up.These states are also suited for the assessment of HRC, but need some additions.The activity needs to be tracked with greater detail, since robots can perform a wider range of activities.Therefore not only the activity, but also the kind of activity e.g.cleaning, surface activation or glue application has to be detected.In addition to machine warnings or disturbance types, also safety information must be recorded.
The robot states can be applied to the employee with limitations in the states of set up and disturbance.For the robot as well as for the employee it is important to distinguish between different regular activities, since both can perform a variety of tasks.It is also necessary to identify collaborative activities.Employees can also encounter disturbances.Usually these disturbances are not directly related to the employee, but are related to missing material, information and tools, that is required for the task processing.This is the same with set up activities, since employees may be involved in the set up of the robot or of the work station.

Data recording and analysis
We use an optical tracking system based on the Intel RealSense D400 sensor family and the Nuitrack SDK [13] to capture the employee position and motion.A robot TCP socket interface is used to capture robot data.The following section describes the underlying data structure, the state detection approach and the resulting analysis HRC work systems.

Data structure
The robot interface provides basic raw data such as the position of the tool in the robot coordinate system, robot control variables for external devices, robot state messages and safety messages.This data is transformed into robot states in a later step.Based on the infrared depth map of the Intel real sense 3D-camera, the Nuitrack SDK provides information about the position and orientation of 18 human joints of a tracked body.In order to detect different employee states, we intend to define work stations that are linked with activities.A workstation data set consists of the position of the work station and the associated activities.In addition to the work station other areas may be defined in order to detect other states as waiting or disturbances.

State detection
To analyze the work process of the HRC work system, it is necessary to detect the states discussed in the prior section.The state detection is carried out by a self-developed software that stores the detected robot, employee and collaborative states in a spreadsheet.To detect a robot state, the user defines a single parameter or a combination of parameters provided by the robot interface for a certain state.The activity state "apply glue" for instance can be detected if the robot tool position is set in the area of the glue dispenser and the robot control variable that activates the glue dispenser is set to "high".
The employee activity is mainly derived from the employee position, since the employee states are connected to work stations and additional areas.For example the state "prepare frame" in table 1 is associated with the assembly table work station.It is detected if the employee is present at the assembly table and is moving arms and hands.
So far the detection of robot and employee states has not been tested in the exemplary work system.However, the employee state detection was tested in the learning factory of the institute.The robot state detection was tested with the help of an offline simulation software for the UR 10 robot.

Analysis
In order to identify the losses described in section 4.1 and to find ways to prevent these losses, an analysis of the captured state data is necessary.The state type diagram (figure 2) displays the states of the employee and of the robot over time adapting a diagram introduced by Schweitzer [2] for production lines to HRC work systems.2, the diagram reveals balancing losses after cleaning the display (robot) and after preparing the display (employee) and shows the underlying circumstances of these balancing losses.Employee and robot states can also be displayed in pie charts to show the distribution of all states, so that the user can easily identify the percentage of value adding states and of collaborative activities (fig.3).

Summary and outlook
This article presents an approach to analyze the productivity of HRC work systems.The method uses a 3D-camera to track the employee and several types of robot data.The data is used to detect the states of robots and employees and to map them in state type diagrams.This enables the user to identify productivity losses due to an insufficient balancing of employee and robot tasks, disturbances of robot or the employee processes and mutual disturbances of employee and robot.The proposed method supports continuous improvement of productivity in a HRC environment, e.g. the reallocation of tasks between employee and robot in case of balancing losses.However, it is not designed for the use of planning a completely new HRC work system.This requires a more detailed analysis and the integration of other aspects as safety, implementation costs, flexibility of the automation, etc..While a detailed test and evaluation of the presented methodology for the entire work system is still pending, the functionality of single components was tested with positive results.A detailed evaluation in HRC work systems is therefore an important next step.

Fig. 2 .
Fig. 2. Exemplary state type diagram for HRC processesCollaborative states are placed in the center, employee states in the upper, and robot states in the lower half of the diagram.Activity states of employee and robot are placed close to the center in order to better illustrate the workflow.Waiting, disturbances, and safety states are positioned more to the outside of the diagram.In the example of figure

Fig. 3 .
Fig. 3. Exemplary state distribution of employee and robot states

Table 1 .
Robot, employee and collaborative states of the exemplary work system