Abstract : Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes; and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.
https://hal.inria.fr/hal-02111532 Contributor : Hal IfipConnect in order to contact the contributor Submitted on : Friday, April 26, 2019 - 9:59:14 AM Last modification on : Friday, April 26, 2019 - 10:47:29 AM
Siwei Fu, Guifen Chen, Shan Zhao, Enze Xiao. Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province. 11th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Aug 2017, Jilin, China. pp.1-12, ⟨10.1007/978-3-030-06179-1_1⟩. ⟨hal-02111532⟩