Skip to Main content Skip to Navigation
Conference papers

Application and Implementation of Batch File Transfer in Redis Storage

Abstract : In the face of massive information, batch processing of files is an important way of information transmission and storage, and the application is quite common. With the increasing demand for batch files processing reliability and speed, and the problem of low storage efficiency for current batch file processing, the paper proposes a storage method that combine distributed storage system HDFS file storage advantage and Redis cache technology to form a rapid batch merge files. The files that meet the conditions are merged into the Sequence File and stored in the HDFS. The multiple linear regression analysis method is used to determine the load factor, so that the load balancing is adjusted and the Redis cache hash data is used to ensure the efficiency. Through experiments on the corresponding file platform for file upload, query, delete and memory usage, we analysis batch processing method and non-batch method comparatively. It can be concluded that compared with the non-batch direct upload file to HDFS way, improved batch file processing method can process files more faster and ensure the stability and reliability of the file at the same time.
Document type :
Conference papers
Complete list of metadata

Cited literature [6 references]  Display  Hide  Download

https://hal.inria.fr/hal-01820921
Contributor : Hal Ifip <>
Submitted on : Friday, June 22, 2018 - 10:43:50 AM
Last modification on : Friday, June 22, 2018 - 10:51:28 AM
Long-term archiving on: : Monday, September 24, 2018 - 8:27:23 PM

File

978-3-319-68121-4_24_Chapter.p...
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Hu Meng, Yongsheng Pan, Lang Sun. Application and Implementation of Batch File Transfer in Redis Storage. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.228-233, ⟨10.1007/978-3-319-68121-4_24⟩. ⟨hal-01820921⟩

Share

Metrics

Record views

177

Files downloads

104