Aggregation of Network Protocol Data Near Its Source

Abstract : In Network Anomaly and Botnet Detection the main source of input for analysis is the network traffic, which has to be transmitted from its capture source to the analysis system. High-volume data sources often generate traffic volumes prohibiting direct pass-through of bulk data into researchers hands.In this paper we achieve a reduction in volume of transmitted test data from network flow captures by aggregating raw data using extraction of protocol semantics. This is orthogonal to classic bulk compression algorithms. We propose a formalization for this concept called Descriptors and extend it to network flow data.A comparison with common bulk data file compression formats will be given for full Packet Capture (PCAP) files, giving 4 to 5 orders of magnitude in size reduction using Descriptors.Our approach aims to be compatible with Internet Protocol Flow Information Export (IPFIX) and other standardized network flow data formats as possible inputs.