Using Behavioral Indicators to Assess Competences in a Sustainable Manufacturing Learning Scenario

. This paper introduces a learning scenario created for a serious game to develop competences in the domain of sustainable manufacturing, by applying a Lifecycle Assessment (LCA). A set of behavioral indicators is introduced to assess how particular competences do change while the player is engaged in playing the game scenario. It furthermore presents early evaluation results of the game scenario on a sample of master grade students at the University of Bremen.


Introduction
Manufacturing industries account for a significant part of the world's consumption of resources and generation of waste. Worldwide, the energy consumption of manufacturing industries grew by 61% from 1971 to 2004 and accounts for nearly a third of today's global energy use. Likewise, they are responsible for 36% of global carbon dioxide emissions [1]. Manufacturing industries nevertheless have the potential to become a driving force for the creation of a sustainable society. This requires a shift in the perception and understanding of industrial production and the adoption of a more holistic approach to conducting business [2]. Sustainable manufacturing considers all life-cycle stages, from pre-manufacturing, manufacturing and post-use (holistic view). These stages are spread across the entire supply chain with different partners managing activities at each of these stages. Thus, many players in the manufacturing process must adopt sustainable principles to ensure that higher production standards are met [3]. Therefore, it is quite important to know the environmental impacts of produced products which can be determined by performing a Lifecycle Assessment (LCA). Nevertheless, there are many difficulties associated with the LCA process and traps in its application (conduction and usage), e.g. wrong scoping of the analysis, collecting wrong data, no data available, improper understanding of the production processes, etc. Finally, the success of a LCA analysis depends on the social engineering skills of the analyser to make all responsible managers to support the task.
The training of current and future manufacturing managers needs to achieve two criteria: first, the targeted learning outcomes need to be achieved rapidly, and second, the learners need to be able to apply the learning outcomes into complex, life-like situations and. Competence-based and technology-enhanced learning (TEL) in general and serious games and simulations in particular have recently attracted a great deal of attention as they have the potential to deliver on both accounts [4]. A general introduction to Serious Gaming to support competence development in Sustainable Manufacturing including a requirements analysis has been presented by [5,6]. Serious Gaming has proven to support learners in acquiring new and complex knowledge and is ideally suited to support problem based learning by creating engaging experiences around a contextual problem where users must apply competences to solve these presented challenges [6].
For a comprehensive assessment of the progress of player's competences, behavioral indicators for such a game scenario need to be elaborated. This paper presents the indicators defined for a Sustainable Manufacturing game scenario focusing on carrying out a LCA which includes the competences of 1) information gathering, 2) ability to perform LCA, and 3) decision making.

Scope of the Sustainable Manufacturing Game scenario
The presented challenges in the education and training for sustainable manufacturing are also addressed by the TARGET project which identified sustainable manufacturing as an emerging field where new competences are required to facilitate the new manufacturing paradigms and technologies [6]. The TARGET (Transformative, Adaptive, Responsive and enGaging EnvironmenT) project aims to develop a novel TEL platform that provides learners with a responsive environment that addresses personalized rapid competence development and sharing of experiences in the domain of project management, innovation and sustainable manufacturing. The TARGET environment consists of a learning process supported by a set of components that constitute the TARGET platform. The core component of the TARGET platform is a serious game combined with virtual world technology, which confronts individuals with complex situations in the form of game scenarios. The serious game facilitates situated learning that results in experiences leading to the development of competences, whilst the interaction within a virtual world enables individuals to externalize their tacit knowledge [7,11]. The sustainable manufacturing game scenario reflects the phases an enterprise has to run through when dealing with sustainability issues. Within the game scenario, the player takes over the role of a Sustainability Manager who was recently hired by the Chief Executive Manager (CEO) of a production company. When starting the game scenario, the player finds himself in a meeting with the CEO and the other managers (i.e. Production Manager, Logistics Manager, Human Resources Manager etc.). The CEO introduces the plan that the LCA) should be conducted concerning the production of a specific product and advises the player to do so. He also urges the other managers to support him. The CEO and the other managers are non-player characters (NPCs) who are driven by the game engine. After the meeting finishes, the player starts to execute the relevant steps of the LCA, i.e. 1) setting the objectives, 2) setting the boundaries, 3) selecting the flow chart, 4) selecting inputs and outputs, 5) deciding on the data for inputs and outputs, 6) setting the impact categories.
In the first phase (scoping of the LCA) the player has to define the objectives and boundaries for the LCA. In order to effectively complete his/her tasks the player needs to gather right and relevant information from different NPCs (i.e. CEO, Production Manager, Shift Manager) and furthermore through accessing the Enterprise Resource Planning (ERP) System or directly visiting the shop floor. For instance, when setting the boundaries, the player can discuss the issue with the CEO, who would advise him/her to focus the LCA on the whole life cycle of the product. On the other hand, player can also discuss the issue with the Production Manager, who would advise him/her to focus the LCA solely on the production of the product. The decision made by the player will have impact on costs, time and the final quality of the LCA. All additional LCA steps required follow the similar logic and the player is virtually free to choose from the data that is available to him/her. The final calculation is done by the virtual LCA tool which also reports whether all necessary data has been entered or not. Finally, the final phase is checking the completeness and consistency of all collected data and evaluating the results in terms of impacts per category. The final result is a report to be created with the virtual LCA tool and delivered to the CEO.

Measuring Competence Performance in Serious Games
Competence assessment in the field of TEL is usually carried out by on-line questionnaires or test items provided after the learner consumed a set of learning objects. Serious Games offer the opportunity to assess if the player is able to apply a particular competence while he or she is playing the game. In other words, a game scenario may encompass both, learning and test objects at the same time. However, given this potential of serious games, the challenge is to avoid that the player´s flow experience [8] or feeling of presence [9] is impaired. Thus, a non-invasive or implicit assessment procedure is required. Our implicit and non-invasive assessment procedure is based on the interpretation of the player´s actions and interactions within the virtual environment [10]. These actions and interaction, called Behavioral Indicators (BIs) should be valid clues to distinguish between well and poor performing players. The elaboration of BIs starts with the identification and definition of competences to be assessed (see Table 1). The operationalization of BIs leads to formalized functions consisting of parameters that can be measured while the learner is playing the game (game logs).
The observation and integration of different BIs constitutes the foundation of an online assessment of the level of competence a player has. It is related to conducting and executing the seven key phases of the LCA concerning a specific product, i.e. 1) setting the objectives, 2) setting the boundaries, 3) flow chart definition, 4) inputs and outputs definition, 5) data gathering, 6) choosing impact categories, 7) interpretation of results with recommendations.

C2
Information Gathering Concerns getting the "right" information in adequate quality (completeness and correctness) in adequate time.

C3
Decision Making A very important competence for conducting projects such as the LCA as it requires effective and on-time decisions when 1) setting the objectives; 2) setting the boundaries; 3) flow chart definition; 4) defining inputs / outputs; 5) utilizing gathered data; 6) choosing the impact categories when conducting the LCA.
The operationalization of the competences and the elaboration of BIs listed in Table 1 was built upon existing theories, frameworks and empirical evidence. Before describing the competences and their BIs in more detail, we will briefly outline the theory of information foraging [11] which served as main reference for elaborating BIs for competence C2 (Information Gathering).

Theory of Information Foraging
The theory of information foraging as proposed by [11] aims at describing the strategies that are applied in order to seek for and consume valuable pieces of information (for example, when searching for relevant papers in literature data bases). An ideal information forager gains information from external sources effectively and efficiently. Such external sources encompass a wide range of entities, for example online documents or communication partners. In [11] they are called patches and we consider information sources as specific subset of them. Information sources in the context of TARGET are e.g. an Enterprise-Resource-Planning (ERP) system or NPCs which are also part of the scenario. An efficient and effective information forager maximizes the rate of gaining pieces of valuable information (in our context called information objects) by applying a balanced ratio of explorative and exploitative search activities. These two kinds of activities are mutually exclusive, i.e. the information forager can spend his or her available time on either explorative search behavior (called betweenpatch processing) or exploitative, information consuming behavior (called within-patch processing). The information foraging theory provides a profound set of "success indicators". For example profitability (the ratio of gain per patch to the cost of within-patch processing) or rate of information gain in units of time.

Measuring Performance in Information Gathering
The behavioral indicator used to measure Information Gathering is the number of information objects found during the beginning of the game (t 0 ) and a specific point in time t. Relating this to the total number of information objects contained in the game scenario provides as a performance indicator the percentage of detected information objects. One could also relate that to the time needed to get a more precise performance indicator for that competence, but for our purpose we just consider the ration of found to the total number of information objects.

Boundary
The boundary is necessary to focus the scope of the LCA. It is coded as a sentence of the CEO: "I know that the production manager will say "focus on production" but I suggest to focus on the whole lifecycle".
Flowchart A flow chart describes the production and usage processes defining all the (material) inputs and outputs of each step. When the player has selected the boundary he can get the hint from the CEO and the Production Manager which of the provided flow charts in the LCA tool is the right one.
Inputs/Outputs Inputs and outputs describe the flow of energy materials and parts into and out of a production or usage process. This information object is distributed in sentences of the CEO and the Production Manager. The CEO knows about the sub-parts to be assembled while the Production Manager knows exactly the material and energy inputs and outputs.

Data
Data is a collection of correct values for each of the inputs and outputs. Again, this information object is distributed in the game scenario. Many data can be observed when using the ERP system on of the PCs, other, more precise data is told to the player through the Shift Manager.

Impact Categories
The impact categories are used as indicators to describe the whole life cycle of the product as green.
The information object is distributed in sentences of dialogs of the CEO and the Production Manager.
Within the game scenario information objects are coded in either being hidden in a game object or a sentence of a NPC. The two game objects which provide information to the player is a big wall screen showing production processes and a couple of PCs which are accessible by the player showing ERP related data. The NPCs of the scenario are the CEO, the Production Manger, and a shift manager from the production site. All of them are able to answer questions of the player. Table 2 shows which information objects are hidden in the game scenario.

Evaluation
Evaluation of the Sustainable Manufacturing Scenario has been done 11-13 July 2012 at a laboratory of the University of Bremen. Participants were 24 master students of Management and Industrial Engineering. Evaluation was divided into three steps: 1. All participants filled the first part of a questionnaire with general and scenario related questions to collect demographic data and to assess present understanding of LCA related issues. 2. An instructor introduces the TARGET software and demonstrated how to play it.
The participants played the scenario for 20 minutes. After that all participants reflected on their performance related to the three competences mentioned above. The participants have been asked to do a self assessment for the three competences on a scale of 1-9 (where 1 = very poor and 9 = very good) for the phases of beginning, during, and the end of the gaming session. After that, participants were asked how they could improve their performance. All participants played the game for a second time for 20 minutes trying to improve their performance. At the end of the second playing session participants were asked for another self-assessment. 3. Finally, all participants filled the second part of a questionnaire to gather in-game experience, updates on the scenario understanding and general post game evaluation.
The following results are focusing on the question, whether the participants were able to improve their performance in competence Information Gathering In the beginning of the first gaming round more than the half of all participants assessed themselves to have only marginal performance in Information Gathering (see Fig. 1). 13 of 24 participants (54%) assessed their own competence on a scale of 1-9 as poor (values between 1 and 3), 8 participants (33%) as medium (values between 4 and 6), and only 3 participants (13%) as good (values between 7 and 9). The highest values participants gave was 7 (3 participants). This situation changed by the end of the second gaming round (see Fig. 2). A total of 75% assessed to have medium to good performance in Information Gathering (37.5% with values between 4 and 6, 37.5% with values greater equal 7). Only 25% assessed themselves still with limited performance. The highest values participants gave was 8 (6 participants). The average value of competence Information Gathering grew from the beginning of the first gaming round to the end of the second gaming round from 3.38 to 5.50 (see Fig. 3). The results are based on a self-assessment of the players and not on measures taken during game play. Even when the meaning of the performance indicators has been explained there is still the risk that the participants provided incorrect answers.

Conclusions
This paper introduced a serious game scenario designed to teach the Ability to Perform a Lifecycle Assessment (LCA) and related competences, i.e. Information Gath-