Analyzing Brain Waves for Activity Recognition of Learners

,


Introduction
Understanding the states or emotions of learners is useful for improving the quality of the lecture. Learners' states are usually observed visually by the lecturer at the time or estimated by exams conducted after the lecture as a degree of understanding. However, especially in massive or on-line lectures, it is difficult to know the state of each learner at the lecture. Detecting learners' states automatically and in real time is expected to innovate in the current style of learning.
The aim of our research is to detect the states of learners from several kinds of sensor data. On the assumption that some particular activities of a person correspond to the states, we try to recognize activities to understand the states of learners. Activity recognition has been widely explored using video data. Recently, a spread of depth sensors enabled us to analyze the subject in terms of 3-dimensional coordinates [4]. A camera with a depth sensor can recognize some motions, facial expressions, and eye movements with high accuracy. However, the target of our study is learners who are sitting with small moves, and hence we need another kind of sensor data in addition to image data for activity recognition with high accuracy.
We focused on brain waves, electroencephalography (EEG), captured by a simple electroencephalograph for activity recognition of learners as additional data to video data. Generally, it is said that EEG changes by the influence of the emotion of the person. However, usual electroencephalographs have a lot of electrodes and are not suitable for use by learners in a lecture. Moreover, some expert systems are expensive and unsuitable for a large amount of use. Especially for massive lectures, electroencephalographs are required to be easy to use and cheap.
In this paper, we evaluated the validity of using multiple bands of brain wave data captured by a simple electroencephalograph for activity recognition of learners. In usual researches of activity or emotion recognition by EEG data, some particular bands, called α and β waves, are used as features for analysis. We considered other bands of higher and lower frequency in addition to the usual features, α and β waves and their combinations, to improve low accuracy of the activity recognition by simple electroencephalographs. In our experiment, the subjects performed two activities, calculation and meditation, with a simple electroencephalograph which can output a spectrum of brain waves with the 8 kinds of bands. Then, we applied support vector machine (SVM) [5] to the 8-dimensional vectors, and investigated the accuracy of the activity recognition by comparison with scalar data based on the values of one or two bands.
As a result, the accuracy of the activity recognition by the data of the 8 kinds of bands was high compared with considering one or two features. Additionally, we confirmed that the simple electroencephalograph has a possibility of use for the activity recognition of learners.
The rest of this paper is organized as follows. Section 2 surveys related work of activity recognition of learners with EEG data. Section 3 describes the activities, the collected brain wave data, and the data analysis in our experiments. Section 4 reports the results of the experiments. Section 5 shows considerations about the results and future directions of our study.

Related Work
Our study is considering EEG data of learners captured by a simple electroencephalograph for state recognition.
A viewpoint of this survey is EEG data of learners. Some studies indicated a possibility of state or emotion recognition of learners by analyzing their EEG data. Wan et al. [11] investigated the characteristics of EEG data of learners in a Web-based lecture. Chaouachi and Frasson [3] showed a relation between EEG data of learners and the response time to some questions. These studies were using electroencephalographs with many electrodes rather than simple one.
The other viewpoint is EEG data by a simple electroencephalograph. Electroencephalographs with a few (dry) electrodes are less accurate than expert one, but those are easy to use for learners. Ishino and Hagiwara [6] proposed a system which estimates the feeling of subjects by EEG data captured by a simple electroencephalograph with three electrodes. Vourvopoulos and Liarokapis [10] evaluated a usability of a commercial and simple electroencephalograph for braincomputer interfaces. These studies were not necessarily considering learners as the target.
The study by Yoshida et al. [13] is similar to ours in the sense of the two viewpoints. They considered the changes of EEG data to evaluate the states of learners who repeated simple calculations. Then, they reported that the ratio of α wave to β wave was related to the activity in their experiment. Some recent studies investigated more complex relations between emotional states and the range of brain waves rather than the discriminated one such as the relation between relaxation and α wave. For example, Wang et al. [12] classified EEG data into some categories based on emotional states by machine learning. In our experiment, we considered the difference between two kinds of activities and compared the other kinds of waves in EEG data systematically in addition to α and β waves and the combinations of the waves.

Methods
The purpose of this paper is to clarify whether considering multiple bands of brain wave is useful for activity recognition compared with usual features based on a single band or a scalar from a few bands. We collected brain wave data of persons with two activities, and tried to recognize the activities from the data with considering several kinds of features.

Collected Data
We collected brain wave data of five subjects with the two activities: -Calculation of simple additions and -Meditation with the eyes closed.
We used 10 × 10 matrixes with randomly chosen 20 single-digit numbers (Fig. 1) for the activity of calculation. We prepared a sufficient number of matrixes printed on papers, and the subjects tried to fill the matrixes as much as possible in each trial. The time for a single trial of an activity was 2 minutes, and the subjects repeated the two activities alternately 5 times for each with no interval. Therefore, the total time of the experiment for a subject was 20 minutes.
We used a simple electroencephalograph MindWave Mobile (Fig. 2) by Neu-roSky 1 for collecting brain wave data. The simple electroencephalograph has only two electrodes and does not need any gel on the head of a subject for capturing potential differences. One of the electrodes is placed at front polar (the middle point of Fp1 and Fp2 in the international 10-20 system) and the other is placed at left earlobe (A1 in the international 10-20 system). The electroencephalograph (and an attached module) can output a spectrum of brain waves with the following 8 bands:  θ: 3.5-6.75Hz, -Low-α: 7.5-9.75Hz, -High-α: 10-11.75Hz, -Low-β: 13-16.75Hz, -High-β: 18-29.75Hz, -Low-γ: 31-39.75Hz, and -Mid-γ: 41-49.75Hz.
The sampling frequency was 512Hz. The output rate of the spectrum was about 1 for a second. Therefore, we obtained about 6,000 (= 60 × 20 × 5) 8-dimensional vectors with the label for one of the two activities. Table. 1 shows the numbers of the generated vectors for the five subjects and the two activities. We simultaneously took video data of the subject for adjusting the time of each trial to the labels.

Data Analysis
We tried to recognize the two activities from the collected data. First, we considered the following 19 kinds of scalars: -Each element of the 8 bands, -The sum of low-α and high-α, -The sum of low-β and high-β, and -The ratio of β to α for the 9 combinations of (low-α, high-α, and the sum) and (low-β, high-β, and the sum).
Next, we considered the 8-dimensional vectors by the 8 elements.
Additionally, we considered the simple moving averages of the collected data with ranges 10 and 20. The simple moving averages of a time series data v t with a range n is the time series data Then, we obtained two data sets S 10 and S 20 in addition to the collected data set S 1 .
For the 20 cases, we applied SVM with a linear kernel to the three data sets. Concretely, we used the function ksvm in the package kernlab [8] for R 2 with the default values for the parameters. Then, we conducted 5-fold cross-validation [2], where the accuracy of an activity recognition is the ratio of the numbers of the vectors whose label was predicted correctly to the total number of the vectors. We set each test set in the cross-validation to correspond to a single trial of a subject, because vectors in the same trial tend to have similar values. The activity recognition was conducted for each subject separately, and the accuracy was calculated as the mean of the results for the 5 subjects.  Table 2 shows the accuracy of the activity recognition for the 20 cases with the three data sets. The accuracy by considering the 8 bands was higher than other 19 cases for any data set, the accuracy was the highest (86.3%) with S 10 and S 20 . The accuracy by low-γ was the highest in the 8 bands for the three data set. The accuracy by considering the ratio of β to α was not high in any combination even compared with each case of the 8 bands. The graphs for some cases showed 5 peaks which correspond to the change of the activities. Fig. 3 is an example of the graph for the change of brain wave data over time. We can see from the graph that the data of low-and mid-γ divide the two activities.

Major Conclusion
We found that the two activities could be recognized with high accuracy by considering multiple bands in brain wave data compared with usual features. At the same time, we confirmed that brain wave data obtained by the simple electroencephalograph includes useful information for this kind of activity recognition.

Key Findings
The optimal length of the range of the simple moving average was 16. By Table. 2, the accuracy was improved when we considered the average of the data for 10 or 20 seconds for any criterion. We additionally investigated the relation between the accuracy and the range of the simple moving average for the case of the 8 dimensional vectors. Figure 4 shows the accuracy against the range. The accuracy was the highest (86.9%) when the range was 16. Considering the average of data for about 20 seconds seems to be suitable for the activity recognition, which corresponds to the same analysis on another kind of activity recognition with EEG by a simple electroencephalograph [1]. We experimentally conducted personal identification by the collected data. In the experiments, we observed differences of brain waves between the subjects. Table 3 shows the results of the personal identification. We applied SVM with a linear kernel and 5-folds cross-validation as the activity recognition. We could not earn high accuracy for the personal identification with the data. The accuracy by the data for the activity calculation (43.9%) was better than meditation (28.8%).
The results indicate that we need other techniques to utilize the EEG data for personal identification.

Future Directions
The original goal of our research is to recognize the states of learners such as being concentrated or absentminded. In this paper we found that the collected  brain wave data could divide the two activities, calculation and meditation, with a degree of accuracy, however there are some deviations between the activities and the states of the subjects. For example, we can consider the following noises: the brain wave data might have effects of (1) the move of the hands and (2) whether the eyes were closed. Therefore, our future work includes extra experiments with other activities such as (1) Calculation without writing, or moving the hand and (2) Meditation with the eyes opened, or opening and closing the eyes.
We are also planning to use EEG data for activity recognition with data collected by other sensors. Kamizono et al. [7] conducted a similar experiment of activity recognition of learners using video data with depth information. They considered four activities with great moves of the face (meditation, reading texts, looking away, and talking to another person). Moreover, it is possible to capture the heart rate of the subject from video data [9]. Therefore, we can expect improvement of the accuracy of activity recognition by combining these kinds of data with EEG data.

Conclusion
We evaluated the validity of using brain wave data with multiple bands for activity recognition of learners in a lecture. We conducted the experiments of activity recognition for two activities, calculation of simple additions and meditation with the eyes closed, with a simple electroencephalograph. We found that two activities could be recognized with high accuracy by considering multiple bands in the collected brain wave data compared with usual features. Additionally, we confirmed that brain wave data obtained by the simple electroencephalograph includes useful information for this kind of activity recognition.