Evaluation of the Effectiveness of the Use of a Mobile Application on Students’ Study Achievements – A Pilot Study

. At present, mobile learning (m-learning) seems to be a wellestablished methodology at the institutions of higher learning. Most recently, smartphone apps started to be a popular part of m-learning. The purpose of this article is to evaluate mobile application effectiveness on students’ study achievements in the Course of English taught at the Faculty of Informatics and Management in Hradec Kralove, Czech Republic. The results of this study confirm that learning via the smartphone app is effective. However, the content of the mobile app has to be adapted to students’ needs in order to make them motivated to use the app. Therefore, the whole pedagogical process should be well planned, react to students’ immediate needs and implement appropriate teaching and learning methods, which can contribute to the positive learning outcomes.


Introduction
Currently, mobile devices are important part of educational process and they are used in the so-called mobile learning (m-learning), which can be defined as learning across multiple contexts, through social and content interactions, using personal electronic devices [1]. Thus, m-learning can take any form of learning with the help of a mobile device, which can be done at anytime and anywhere when the learner needs it [2]. In this way, mobile devices are revolutionary because they transcend the boundaries of the structural status of classrooms and lecture halls and their associated modes of communication -they do not have to be confined to one particular place and time in order to be effective [3].
Nearly all students nowadays own mobile devices and about half of them own more than one [4]. The most exploited mobile devices among young people seem to be mobile phones, respectively smartphones. In fact, about 90% of young people aged 18-29 years own a smartphone [5]. This has been confirmed also by other research studies [2,6].
M-learning is natural continuation of the well-established eLearning methodology, which has been used at the University of Hradec Kralove (UHK) since 1999. All eLearning courses are created in a virtual learning environment Blackboard Learn. Both methodologies, i.e. m-learning and eLearning, are part of the so-called blended learning (BL). BL can be defined as a combination of traditional, face-to-face teaching and online learning [7], which means that in addition to their face-to-face classes, students are provided with their online complement so that they could revise and practice the material taught at school. Students have no problems with this type of study and they are well equipped with mobile devices (i.e., laptop, smartphone or tablet) as research shows [8][9]. Almost all students own one mobile device and three quarters have even two mobile devices [10]. This is also true for students who attend English language classes at the Faculty of Informatics and Management (FIM) of UHK. Their face-to-face classes are also supported by eLearning courses in which materials from these classes and additional exercises for revising and practicing at home are available. At present, there are about 30 eLearning courses designed and actively used only for English language learning at FIM.
However, as it has been indicated above, students now tend to use their mobile phones, respectively smartphones for learning. Therefore, also English language teaching (ELT) has to face to challenge and in the winter semester of 2017 a pilot course introducing learning with smartphone application (app) was conducted. This was performed within an English language course.
The purpose of this article is to evaluate mobile application effectiveness on students' study achievements in the Course of English.

Participants
Altogether 33 students attended the English language course in the winter term of 2017. They were all full-time students of Management of Tourism in their third year of study at FIM. Their level of English according to CERF (2018) was B2. During the semester, however, two students left the course, and thus, only 31 students completed the whole course. Out of 31 students 19 students, who had access to the app compatible with their smartphone operating system, also used a smartphone with a special mobile app tailored to their needs as a supporting tool to their face-to-face classes. 12 students served as a control group. The course lasted from 26 September till 12 December 2017 and classes were held regularly once a week for 90 minutes.

Study design
The first method used in this study was a needs analysis since the proposed mobile app should have satisfied and enhanced both students' learning and study achievements. The needs analysis was based on the SWOT (strengths, weaknesses, opportunities and threats) self-evaluation and was conducted during the first introductory lesson.
Secondly, on the basis of the data from the needs analysis, a mobile app was designed and gradually developed. The described mobile app consists of two application parts and one server part. The first application part is designed as a web interface for the teacher (Fig. 1) and the second application part is presented with a mobile app for students (Fig. 2). This app is intended now only for the Android operating system and it is available for free at Google Play store. The reason was the ratio of students who use the Android operating system on their smart devices. The app itself then enabled to use its statistics since it can collect all user data and distribute it to the server part for subsequent research and evaluation by the teacher.  Thirdly, methods of analysis and evaluation of the results of students' achievement tests were used, including a statistical analysis. The pass mark for doing the final achievement test was 50%, i.e., 30 points. All the results were recorded and statistically analyzed. A group of 31 students was randomly assigned into two independent groups. The research group, whose size was n1 = 19, used a mobile application in their learning. The second group, the size of which was n2 = 12, did not use the app and was a controlled group. The following hypothesis was set.
H: Students who use mobile apps in their studies have significantly higher learning outcomes than students who do not use this app.

Results
The results of the needs analysis showed that students' biggest weakness was learning and retention of English vocabulary. This finding is also reflected in other research studies [11][12].
Therefore, the newly developed smartphone app was targeted at and tailored to the development and practicing of new English words and phrases. Altogether, there were ten parallel lessons of vocabulary and phrases. The content of the lessons was physically completed with words and phrases by the teacher. The students had to translate the word or the phrase from their native language into English. Each lesson was done as a test and consisted on average of 15 new words and 10 new phrases. The selected words and phrases were those discussed in the face-to-face classes and focused on their field of study, i.e., tourism. The teacher was encouraging students to use the mobile app and practice new words and phrases discussed in class through notifications in their mobile app twice a week (Fig.3).

Fig. 3. Teacher's notifications sent to students via the smartphone app
Students' results of the final achievement test were statistically analyzed at the beginning of the year of 2018. A box diagram of the results of the two groups is shown in Fig. 4 and Table 1 below. It demonstrates that the students of the experimental group achieved the higher results as it was expected.

Fig. 4. A box diagram of students' results
However, this observation was further confirmed by the calculation (Table 1). Due to the small ranges of the two sets in the pilot study, it is not possible to use the parametric t-test to compare the mean values obtained from the tests. Thus a nonparametric Mann-Whitney U test with a p-value estimated from approximation by normal distribution and continuity correction was used. According to the result of this test, the hypothesis H can be assumed at the significance level of 0.01. Specific results of the SPSS calculation can be found in Table 1 below.

Discussion
The results of the statistical analysis confirm that learning via the smartphone app is effective. Therefore, the set hypothesis is true, i.e., students who use mobile apps in their studies have significantly higher learning outcomes than students who do not use this app. Nevertheless, the content of the mobile app has to be adapted to students' needs in order to make them motivated to use the app. This has been confirmed by other research studies on this topic, e.g. [2,[13][14][15][16]. For instance, the findings by Lopuch [15] demonstrate that students who enhanced face-to-face classroom education with an app-based curriculum reached 165% of their expected learning achievement.
Thus, the whole pedagogical process should be well planned, react to students' immediate needs and implement appropriate teaching and learning methods, which can contribute to the positive learning outcomes. Churchill et al. [17] propose a special learning design framework as a key strategy for utilizing multiple affordances of mobile learning technology. This framework should include and integrate at least four core components, which are resources, activity, support and evaluation. Especially, the activity component is very important since it requires students to engage with intellectual and knowledge-based developments. Celik and Yavuz [18] expand that a guided and controlled implementation of smartphones can enhance the effectiveness and quality of language learning process. In addition, they state that mobile apps should correspond to the level of the students' language knowledge and should be designed by experts (cf. [19]).
The limitation of this case study consists in the small subject sample, which is connected with piloting this tailored-made mobile app.

Conclusion
The results of this study indicate that m-learning via a smartphone app can be effective and serve as an appropriate complementary method to other forms of course delivery, especially face-to-face teaching.
Future research should focus on the effectiveness of mobile apps for learning with bigger sample sizes, as well as extending the mobile app for the Apple's platform and iOS.