EmotionBike: A Study of Provoking Emotions in Cycling Exergames

,

Abstract. In this work, we investigate the effect of how exercise game design elements generate deliberate real-time sensed emotional responses in gamers. Our experimental setup consists of a cycling game controller, a designed 3D first-person cycling game to provoke emotions, a data recording system, and an emotion analysis system. The physical cycling game controller is an enhanced computer controlled bike-exercise-trainer that enables handle bar steering and sets pedal resistance. Our developed 3D first person cycling game provokes emotions with game elements in different game settings: timed race, parcours traversal, and virtual world exploration. Our recording system synchronously captures video, game controller activity, and game events for emotion analysis. In this case study, we show evidence that crafted computer exergame elements are able to provoke subject emotions displayed in their facial expressions, which can be quantified with our developed analysis method. The game elements selected in the specific gameplay situations follow patterns that give inside and judge of individual players involvement and emotional tension. Our emotion analysis of game events provides insights into player reactions during specific game situations. Our results show that strong differing responses by individuals may be taken into account in the design of game mechanics. For example, the falling event of level 3 showed that two opposing strong reactions could be triggered in players. The emotion analysis methods may be used in other types of games. Hereby we believe that a combination of questionnaires and our in situ emotion analysis provide valuable feedback to aid decision in for game design and game mechanics.
Keywords: Exergame, Affective Gaming, Physical Activity, Cycling Game, Big Five Personality Test, Facial Expression, Emotion Provocation, Emotion Recognition

Introduction
Games, as one form of entertainment, enable the telling of stories by immersion into imaginary worlds. Game content is designed during development to provide rich and unique experiences and engage players. Provoked emotions play a significant role in the design of game scene moods and game character persona. These provoked emotions are either fixed in design or evolve through choices in the game flow. The individual emotional reactions of players during gameplay are a great source to individualize storytelling and provide another level of interaction.
In our work, we introduce the concept of emotion provocation in a physical cycling game. The physicality is hereby important since it requires physical effort in order to progress the game. To showcase the provocation concept we successfully provide our case study to show how game elements are able to provoke emotions. The experiments were conducted with a custom cycling controller, cycling game and emotion analysis. In addition, we investigate the influence of the emotional reaction by consideration of the big five personality traits.

Related Work
Relevant research can be found in the domain of affective computing and affective gaming. Different sensor types have been utilized as sources of affect information. Applied sensors include vision-based sensors that recognize facial expressions, body actions and physiological sensors including ECG, PPG, GSR, Respiration, and HR. A good overview of affective sensors provided in the survey by Kotsia et al. [1]. In this work we focus at this point on facial expression recognition to minimize invasiveness of the subjects. An interesting direction was demonstrated by Bailenson et al. [2], who combines facial feature recognition and physiological responses including electrocardiogram and skin conductance.
One research field using emotional dialogue systems aims to create human or animal like behavior in robots. Human Robot Interaction (HRI) enables bidirectional interaction between robots and humans, where social interacting elements such as Kismet [3] and Flobi [4], virtual avatars [5] and abstract surfaces [6] have been created. Social interaction requires modeling of intrinsic emotions [5] to create believable persona. Dialogue systems are often based on multimodal interaction [7] and offer users a more natural and transparent interaction [8]. Based on existing research we followed the paradigm of natural user interfaces and developed a physical cycling game controller to enhance the accessibility of the system. Tan et al. [9] discovered that physiological data relates to the game experience.
Provoking user emotions is a concept that has been explored previously in different contexts. The most commonly provoked user state is stress, because it is relatively easy to provoke and reliable to measure. It can be sensed with a combination of EEG, Blood Pressure, ECG and GSR [10] [11].
In emotion recognition analysis systems chose predominately continues emotion dimensions (e.g. pleasure, arousal, dominance) or discreet emotion categories (e.g. basic emotions) as the output representation [12] [13]. In our work we chose a discreet emotion category, including joy, disgust, anger, fear, neutral, sad, surprise, and contempt, based on the availability of a recognition system. To obtain emotion measures from facial expressions our work utilizes the automatic FACS [14] coding tool CERT [15].
For the CERT assessment of emotion for computer-mediated tutorial dialogues Grafsgaard et al. [16] reported the importance to perform a bias adjustment, due to inter-person variation. For this study we do not adjust this bias, since we intend to show the ability of our physical exergame to provoke emotions and show differences. In our work, we relate facial expression reactions to game events.
A subset of the research field affective computing, which uses previously discussed techniques to create more entertaining and immersive experiences in games, is called affective gaming. Gilleade et al. [17] defined affective gaming as a form of gameplay where the current emotional state of the player is used to alter the game mechanics. For example, altering the behavior of user input, changing the game environment, artificial intelligence behavior or displaying visual feedback depending on the current emotional state. Currently there are many works which triggers game events in relation to the player's emotional state but there are fewer works that directly link recognized emotions to events that occurred during gameplay such as [18] [19].To take full advantage of interactive entertainment applications it is crucial for game design to analyse the effectiveness of emotions triggered by game events. Parnandi et al. [20] proposed an affective game, analyzing EDA and SCR data to control the game difficulty. They also defined a number of requirements enhancing affective game design and controls. Those requirements are incorporated in the presented cycling controller: be intuitive, engaging, easy to learn, highly dynamic and enabling multiple forms of manipulation, such as steering and speed. Many works in affective gaming focus on horror scenarios, because of the strong influence on stress levels [18] [19]. Another way to enable an emotional dialogue is to apply biofeedback. For example, games including biofeedback [21] and an integrated emotion recognition [22] [23].
For general exercise, Biddle [24] linked physical activity to mood and emotion. Looking at exercise and video games, Warburton et al. [25] showed that interactive games are able to enhance exercise experiences. Specifically for cycling exergames, Hoda et al. [26] demonstrated that combining games with a physical bike increases the speed and the average rpm of participants by motivation, resulting in an improved exercise. These results provide evidence that exergames add entertaining value and motivation for people with basic activity levels to exercise. To our knowledge, our study is the first emotion assessment in a cycling exergame.

Experimental Setup
To conduct our case study we designed an experimental setup for physical cycling game scenarios, capture game and interface controller events as well as emotional responses in facial expression. Our experimental setup, shown in Fig. 1 consisted of four main components: data acquisition, emotion sensors, controller, and visualization. The emotion sensors component generalized input data from cameras and physiological sensors. The emotion provoking game provided scenes on a The data acquisition system records synchronized emotion sensor data, game and controller events in a database. In our case study, the emotion sensors investigated consisted of a Kinect camera that provided HD video at 30 fps and which recorded frontal facial images for offline emotion analysis.

Cycling Game Controller
In our cycling game controller, user inputs are generated by an ergometer, including speed calculated from pedal rpm and pedal resistance that can be software controlled. The exergame controller needs to be intuitive, easy to learn, highly dynamic and enable multiple forms of adaptation. Thus a custom constructed handle bar allowed rotation up to 180°. The cycling game controller is based on a semiprofessional exercise machine 5 , which provides a maximum pedal resistance of 1000W. The ergometer operates on an embedded system and connects with a network interface to read and control parameters such as resistance and speed. Fig. 2 shows the hardware setup consisting of display screen, cycling game controller, face illumination lamp and Kinect 2 camera.

Cycling Game to Provoke Emotions
With our cycling game we aim to provide a unique and engaging natural user interface to participants. The cycling game is realized with the Unity3d game  6 . The resulting in house game is tailored to the needs of the experiment to provide different game styles resulting in five different levels, which provide users with different objectives. The different levels allowed us to explore the effects of different game types in controlled environments for our case study. In particular the investigated provoking game events were separated for easy recognition of emotions in facial expressions.
The overall cycling game can be categorized as a fun racer; it has no ambition to be physically accurate as a real world bicycle. This game concept allows a broad range of gameplay mechanics and game events that would not be appropriate in a realistic physical simulation. The player may physically accelerate and steer. The generated controller input is directly transferred into the game, simulating a virtual bicycle in near real time, as shown in Fig. 4. This is essential for a cycling game since insufficient controls may influence the overall user perception and evoke frustration.

Game Scenarios
The five levels are designed with game elements as stimuli for specific emotions. If the designated game event occurs a message is sent including event type and timestamp. Table 1 lists all levels and events with the intended emotion provocations in players. A training level, not listed in Table 1, allows the user to get familiar with the cycling game controls. This training level presents to the player neither challenge nor obstacle and has the goal to ride from the start to the finish line.

Level 1:
The goal is to race a street track from start to finish. The streets are populated with roaming teddy bears, shown in Fig. 3a. If the player bumps into    Level 3: The level starts on a mountain road. In order to finish this level it is necessary to jump over a giant gap. To succeed, the player is required to cross a booster gate, shown in Fig. 4, in proper alignment with a ramp to catapult the bicycle over the gap. Since the alignment and steady steering is very challenging, player often need multiple attempts to succeed.
Level 4: At the beginning the player climbs up a hill, which drastically increases the physical resistance of the paddles, shown in Fig. 5a. On reaching a plateau filled with dense and tall grass, the player is attacked by spiders that push the virtual bicycle around. Due to the occluded environment these spiders are hard

Participants Profile
In this case study eleven participants are evaluated, three females and eight males aged between 19 and 41 with an average age of 27. At the start of the experiment participants were given a questionnaire including questions concerning personal information, fitness level, and game experience. The personal information part included questions designed by Satow [27], which allows the categorization of the participants according to the big five personality model (openness to experience, conscientiousness, extroversion, agreeableness, and neuroticism). Dividing the participants into small groups with similar personality did decrease the individual differences in the analysis. Preliminary results indicated that people with high extroversion displayed higher emotional reaction. Seven participants stated that they played video games approximately 2-3 hours per week and described themselves as casual gamers. The other four participants declared that they did not play games regularly. In addition, participants were asked about their game controller experience with modern control mechanisms including Wii remote and Kinect. Seven participants confirmed that they had experience. Further all participants stated that they performed sports, but only seven on a regular basis. Nine out of the eleven participants stated that they cycled on a weekly basis, ranging from one to four hours.

Experiment Procedure
All participants started by filling out the questionnaire mentioned earlier. They were informed about their right to abort the experiment at any time. To start the physical experiment they had to mount the exercise trainer and the game was started, beginning with a training level to get familiar with the interface mechanics and the game world. During the gameplay an experimenter guided the player with pre-defined phrases through the game to explain critical parts of a level. For example, in level four the participants had the opportunity to skip to the next level after exceeding a number of trials because of the high difficulty. At the end of the experiment participants were asked to provide a statement about their physical strain perception in the range from 1 to 7, as defined by Borg [28]. The reported mean strain for our participants was 3.9.

Analysis Method
During our experiments, all participants facial expressions were assessed by observers and after completion of each level participants were asked to self assess their emotions. The recorded facial videos are analyzed with the Computer Expression Recognition Toolbox (CERT) [15] providing a probability for all eight basic emotions (joy, disgust, anger, fear, neutral, sad, surprise, and contempt) per video frame. CERT recognition is based on a support vector learning approach of frontal face images and includes pose correction. Grafsgaard et al. [16] suggests that facial expression probabilities may vary significantly between individuals. In order to access this variation phenomenon we compared the machine facial expression analysis with the observer-and self-assessment. The probability values near the provoking events are compared with our target emotions. For each event a time windows of 3 seconds (0.5 seconds before and 2.5 seconds after an event) was evaluated for occurring emotion probability peaks. An example event analysis is shown in Fig. 6. For all subjects each target event maximum and mean basic emotion probability were extracted. Provoked emotion responses are taken into account for the summary results if they exceed a threshold of 0.5. Level 1: Fig. 7a indicates that nine out of eleven participants felt joy during Teddy Hit events. Participant eight did not hit any teddy bears due to successful avoidance and participant two did not recognize the Teddy Hit because the teddy bumped from behind. Participant four had a low mean value but a relatively high max value indicating that he felt joy at least once. In this evaluation ten self and observer assessments supported our analysis results.

Experimental Results
Level 2: Seven participants had each a high joy response connected to the Coin Collected event, at least one time. The low mean value, shown in Fig. 7b indicates that provocation was not maintained for the duration of collection of the 20 coins. Rather than the coin collection itself, circumstantial situations such as near misses in collection and anticipation of finishing the level, provoked a stronger response. An interesting effect was that six participants had high max surprise values. The mean values were low indicating that they displayed surprise by the Smiley Particle Effect, shown in Fig. 8. In this level two events were triggered. The first event occurred while driving through a booster. Eight participants displayed joy at least once. Five participants displayed surprise reactions. Fig. 9a and Fig. 9b present the max and mean probabilities for joy and sadness. Ten participants displayed joy at this event and eleven participants expressed sadness, making this finding interesting. This effect can be explained since falling down the cliff occurs repeatedly. The ambition of most of the participants was very high and both joy and sadness were detected. Our observation, that joy is often detected in frustrating situations as many people smile in natural frustration has been reported by Hoque [29]. Respectively nine self and nine observer assessments labeled frustration. Level 4: Only four participants had a high value for surprise by a Spider Attack. But the high resistance throughout the level led to an often mentioned physical strain by the participants. Level 5: The surprise effect of the Jump Scare event displayed the most varying emotion responses. Emotional facial expression displayed included one for disgust, two for fear, two for surprise and four for joy, shown in Fig. 10. This result neither correlates with the self-assessment nor with the observerassessment, because they unanimously stated that they felt scared. The resulting data show that participants who describe themselves as gamers showed less than maximal surprise or fear. They stated that they expected the events due to their game experience. Some emotions such as surprise and joy decreased in maximum value the more frequently they occurred. Other emotions including anger increased when it occurred more often e.g. during Falling events. The emotion of joy occurred 0.5 seconds earlier than average in level 2, where the main objective was to collect all coins. Thus all participants became more focused and our aim of emotional engagement was reached. The Big Five personality traits showed that people with a high value in extroversion had a higher maximum value in expressing joy. They had a maximum probability of 0.89 against 0.76 for introverted participants.

Conclusion
This work presents a new physical cycling game interface that records and analyses emotions. Our case study demonstrates that tailored game events can provoke specific emotions, as shown in Table 2. In summery, the Teddy Hit and Falling game elements are able to provoke specific emotions for most subjects, while the game elements Coin Collected and Jump Scare provoke a more varied response in the type and probability of emotion.
In this work we present an analysis method to measure emotional reactions to specific game events. The analysis for game events will aid game development and its design process by comparing alternative designs or studying resonance and engagement of players. In addition, this method provides a quantified measure of emotion reactions for a user experience analysis. Emotion response analysis can vary between groups of users and therefore may be correlated with factors including personality traits, gender or age. These factors will enable better specific reaction predictions..

Acknowledgment
We thank Kai von Luck for his support and feedback to this work. Further we express our gratitude to Kai Rosseburg for providing the photos in Fig. 2 and to Wojtek Gozdzielewski, Sobin Ghose, Jonas Hornschuh, Ralf Jettke and Erik Matthiessen for supporting our project. We also thank Enno Putzar for video recording.