Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province

Abstract : Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes; and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.
Document type :
Conference papers
Complete list of metadatas

Cited literature [22 references]  Display  Hide  Download

https://hal.inria.fr/hal-02111532
Contributor : Hal Ifip <>
Submitted on : Friday, April 26, 2019 - 9:59:14 AM
Last modification on : Friday, April 26, 2019 - 10:47:29 AM

File

 Restricted access
To satisfy the distribution rights of the publisher, the document is embargoed until : 2022-01-01

Please log in to resquest access to the document

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Siwei Fu, Guifen Chen, Shan Zhao, Enze Xiao. Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province. 11th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Aug 2017, Jilin, China. pp.1-12, ⟨10.1007/978-3-030-06179-1_1⟩. ⟨hal-02111532⟩

Share

Metrics

Record views

23