Comparison of Wireless Network Simulators with Multihop Wireless Network Testbed in Corridor Environment

. This paper presents a comparative study between results of a single channel multihop wireless network testbed and the network simulators ns-2 and ns-3. We explore how well these simulators reﬂect reality with their standard empirical radio modeling capabilities. The environment studied is a corridor causing wave-guiding propagation phenomena of radio waves, which challenges the radio models used in the simulators. We ﬁnd that simulations are roughly matching with testbed results for single ﬂows, but clearly deviate from testbed results for concurrent ﬂows. The mismatch between simulations and testbed results is due to imperfect wireless propagation channel modeling. This paper reveals the importance of validating simulation results when studying single channel multi-hop wireless network performance. It further emphasizes the need for validation when using empirical radio modeling for more complex environments such as corridors.


Introduction
Nowadays most of the research in the field of wireless networking is based on network simulators. Simulators are attractive for researching network protocols and mechanisms since they allow creating controlled and reproducible environments. Creating such an environments in real test beds is both expensive and time consuming. Real production networks at the same time often do not allow to obtain repeatable data sets needed for research analysis. Various network settings and large parameter ranges can be tested through simulations at low effort since creation and modification of network scenarios as well as data gathering are easy.
In this paper we study how well simulations reflect the reality with commonly used empirical models of wireless propagation channel that requires little configuration and are memory and computationally sparse. In particular, we explore disparities between simulations in ns2 1 and ns3 2 and testbed results for a multihop wireless network located in a corridor. This environment challenges the wireless propagation channel model present in the network simulators.
The performance of network protocols in a testbed is affected by wireless channel properties that depend on the physical environment, location and mobility of the nodes, and the external interference. Accurate wireless channel modeling for simulations is known to be difficult. The commonly used wireless propagation channel for path loss in simulators is empirically modeled, and path loss is computed depending on distance between transmitter and receiver. Consequently, accumulative interference caused by hidden terminals' concurrent transmissions, and spatial reuse ratios of testbed network may not be correctly represented by the simulators. Due to these reasons simulation results often do not match perfectly with the testbed results [2,5].
The modeling of wave-guiding propagation phenomena of radio waves in corridors as well as modeling of the losses caused by reflections, diffraction and scattering of radio waves are more accurately captured by deterministic channel modeling methods [7]. The deterministic wireless channel modeling, e.g based on ray tracing techniques require the exact knowledge of location, shape, dielectric and conductive properties of all objects in the environments and it also requires extensive computational efforts for accuracy. Thus such models are site specific. In addition these models also considerably increase simulations' run time [12].
The aim of our paper is to make the wireless network researchers aware of the differences of the simulations from real wireless testbed. The differences are mainly caused by empirically modeled wireless propagation channel of the simulators. Empirical wireless propagation channel models of the simulators are simple from implementation perspective but they do not cover all the properties of the wireless propagation channel such as losses due to reflections, diffraction, scattering and penetration of the radio waves. We demonstrate the differences between simulations and testbed for two specific scenarios composed of single and concurrent flows transmissions over a single multihop path.
We find that for single flow transmissions over multiple radio links, ns2 and ns3 simulations roughly match the testbed results. Deviations between simulations and the testbed results are explained by the wireless propagation channel models used in the simulations are not accurately reflecting the accumulative interference caused by hidden terminals' concurrent transmissions and the spatial reuse ratio 3 of the testbed network. This shortcoming of the wireless propagation channel model becomes more evident for simultaneous flows transmissions, which have higher strength of accumulative interference than single flow. Simulations indicate considerably worse matching of the throughput fairness of simultaneous flows with testbed results, which reveals that the wireless propagation channel models of the simulators are not correctly representing the wireless channel properties in the corridors.
The article is organized as follows. Section II presents a brief overview of the background and related works. Section III presents the details and specifications of the testbed network and experiments. Section IV gives the details of the simulation setup and also provides an overview of path loss and multipath fading models. Section V presents the comparison of the experimental results with simulations. Section VI concludes this comparative study.
Network simulators ns-2 and ns-3 are de-facto standard simulation tools in the academic networking research community. Simulations in ns-2 are constructed with C++ code and OTcl scripts; the former provides modeling of applications, simulation nodes, communication channels and other mechanisms involved in networking, while the latter is used to control simulations and define additional features, for example the network topology.
Simulations in ns-3 are fully based on C++, but can also be created with Python. The ns-3 simulator was developed from scratch and cannot directly use the code developed for ns2. Many objects are ported from ns-2 to ns-3 but not all, and hence ns-2 incorporate capabilities not present in ns-3. However, ns-3 has capabilities not implemented in ns-2 such as support for multiple interfaces on nodes, use of IP addressing and closer resemblance with the TCP/IP model, and more detailed 802.11a/b/s models.
The accuracy of wireless channel models for simulations naturally determines the quality of the outcome. It could be expected that the more detailed modeling of the IEEE 802.11 MAC protocol in ns-3 would result in more accurate results for certain network scenarios. Obviously it is expected that the simulations results may deviate from the reality. It is therefore important to understand the degree of reality reflection by the simulators.
Other studies presenting comparisons between simulations of IEEE 802.11 based networks and testbeds include [9], which presents a comparative study between an IEEE 802.11a based testbed and three network simulators (ns-2, QualNet and OPNET). It aims to assess the relevance of these simulators in indoor and outdoor environments. The simulation results match to some extent with the testbed. The authors highlight that tuning of physical layer parameters and selected propagation models have great impact on the results. This study is conducted for a single hop network and no comparison with ns-3 simulations is presented.
In [1] the authors present a validation study of the IEEE 802.11b MAC model in ns-3 by comparing simulations with testbed results. The study shows that ns-3 simulations nearly match with reality after proper tuning of the devices in the testbed. It is also shown that for mismatching between the simulation and testbed results, simulator is not always wrong but specific selection and configuration of the devices in the testbed can be culprit. However, in the testbed wireless channel propagation effects on measurements are ignored because the communication between the devices is via coaxial cables.
The authors in [14] point out the disparities between a wireless network tesbed and ns-2 and Qualnet. The disparities are explored based on antenna diversity, path loss, multihop, transmission rate, interference and routing stability. However, ns-3 simulations have not been taken into account and intra-path interference 4 is discussed only for a single flow traffic over a linear multihop network.

Testbed and experiments description
We built an IEEE802.11b based multihop wireless testbed in an indoor corridor environment. The network consists of eight nodes placed as illustrated in Fig. 1. The logical topology is a chain, the placement of nodes ensures the line-of-sight communications with the immediate neighbors. All nodes are Intel Pentium 4 based desktop PCs with 2.40 GHz processor, cache size 512 KB, RAM memory 256 MB and six USB 1.0 ports supporting the data transfer rate of 12 Mb/s. For wireless connectivity each node is equipped with D-Link DWL-G122 wireless USB adapter 5 with an omni-directional antenna. The operating system is Linux (kernel 2.6.29) and the WLAN driver is p54 6 . At the MAC layer we switched off the options for frame fragmentation, dynamic rate adaption and disabled the RTS/CTS exchange. The transceivers operate on channel 3, the transmit power at each node was set to 18dBm, the physical channel data rate is set to 11Mb/s. We experimented with TCP traffic generated by Iperf (version 2.0.8) traffic generator. We used TCP-Cubic configured with the default settings. The routes were configured statically in order to eliminate the effect of routing protocols on network performance [8].
In the testbed we conducted two types of experiments. The first experiment (further on referred to as experiment-1) was performed with a single flow running over a different number of hops. The second experiment (further on referred to as experiment-2) was conducted with multiple TCP flows running concurrently over different number of (a) Layout of Experiment-1. hops. To obtain the results from both experiments, we used tcpdump (version 3.9.8) to capture all the traffic generated in the network and measure the per-flow throughput.
The setup for experiment-1 is shown in Fig. 2(a). The experiment consists of seven scenarios with different number of wireless hops for the monitored flow. In order to capture the effects of multipath fading and the accumulative interference on the network performance each scenario was repeated six times by changing the position of the source and destination node along the multihop chain for a given number of hops. The duration of each trial is three minutes.
In experiment-2 (see Fig. 2(b) ) the layout of the nodes and the network specification had been kept the same as in experiment-1. The experiment consists of six scenarios. In all scenarios the traffic is alway generated from node 8. We start experimenting with two flows of one and two hops running in parallel. In each subsequent scenario we add flows as shown in the figure. As a result in the sixth scenario we experiment with seven concurrent TCP flows. We perform two trials for each scenario and record an average value of the per-flow throughput. The duration of each experiment is two minutes. The throughput values are used to compute a fairness index as explained below.

Simulation setup
We replicate the testbed experiments in ns-2 and ns-3 simulators. Firstly, we configure the parameters of the simulators with the corresponding values in the testbed. In particular, the transmission power, characteristics of the antenna and the corresponding transmission ranges are set according to the specification of the D-Link DWL-G122 wireless USB adapter. According to the device's data sheet the transmission range is set to 100 m. According to [11] the carrier sensing range in commercial wireless cards is twice or more than the transmission range.
On the transport layer we used TCP-Cubic as in the testbed experiments. Note that while the implementation of TCP-Cubic in ns-2 is a simulator-specific, ns3-links the real implementation from Linux via the Network Simulator Cradle (NCS).
Special attention was paid to the proper configuration of the path loss and multipath fading models in order to reflect the radio environment of the testbed. In the indoor environment, the propagation of radio waves is mainly affected by two types of losses: the path loss and the loss due to small and large scale fading. The small scale fading arises due to the multipath propagation effect and the large scale fading is due to the shadowing effect. The model which closely reflects the path loss in the indoor environment is the log-distance path loss model [10]: In (1) n is the path loss distance exponent, d 0 is the reference distance (1 m), d is the distance in meters between the transmitting and the receiving nodes, L 0 is the reference path loss at the reference distance (dB), L p is the path loss (dB) and X θ is a log-normally distributed random variable (dB) with standard deviation σ and zero mean describing the attenuation caused by the obstacles due to shadowing effect.
Note that the value of n depends on the operating frequency and the characteristics of the propagation environment. In the case of the indoor environment, the type of the construction material and the position of the nodes within the building. In corridors due to wave-guiding propagation phenomena of radio waves, n takes values in the range where λ is the wave length in meters. Note that in our testbed there is no large scale fading due to line-of-sight communication between the adjacent nodes, therefore, X θ = 0 and (1) reduces to However, the small scale fading exists in the corridors due to multipath propagation. In corridors the small scale fading is described by Nakagami distribution [13,15]. The probability density function for the Nakagami m-distribution is In (3) r ≥ 0 is the amplitude of the received signal, Γ (m) is the Euler's Gamma function, ω =r 2 is the mean square received power and m = ω 2 (r 2 −ω) 2 is the fading depth, where m ≥ 1/2. For m = 1, the Nakagami m-distribution becomes Rayleigh distribution.
In ns-3 the log-distance path loss and Nakagami fading models are implemented separately. Both models can be used and configured individually. In ns-2 the log-distance path loss model is not yet implemented, however, the Nakagami fading model is implemented together with three-log-distance path loss model. The three-log-distance path loss model of ns-2 is different from the log-distance path loss model with three distance fields namely near, middle and far. Each field has different path loss exponent. For the three-log-distance model a fourth distance field is also defined from 0 to near distance, however, the loss over this field is zero. The limits of four distance fields along with their corresponding path loss exponents are explained as 0 · · · · · · 0 d 0 · · · · · · n0 d 1 · · · · · · n1 d 2 · · · · · · n2 ∞ So each field starts at the end of the preceding one and hence the resultant three-logdistance path loss model is a continuous function of the distance: In (4) n 0 , n 1 , n 2 are the path loss distance exponents and d 0 , d 1 , d 2 are three distance fields(meter). The Nakagami fading model in ns-2 defines three parameters m for three distance fields as 0 · · · · · · m0 d 1 · · · · · · m1 d 2 · · · · · · m2 ∞ The defaults values of distances and fading parameters m in ns-2 are d 1 = 80 meter, d 2 = 200 meter, m 0 = 1.5 and m 1 = m 2 = 0.75. In the testbed the maximum distance between the adjacent nodes per hop is less than 80 meter, so ns-2 is using log-distance path loss model part of (4).
In order to find a better match of the path loss and multipath fading with the real testbed the simulations are conducted for five combinations of the path loss exponent n = 1.9 and five Nakagami fading parameters m i.e. 1.5, 1.75, 2.0, 2.25 and 2.50. Note that doing simulations with higher values of the fading parameter m is not realistic because the higher m means stronger LOS component of the propagation model. This is not the case with the D-Link devices used in the testbed, which are equipped with omni-directional antennas. The next section report the results of the analysis.   Fig. 3(b) show ns-3 and ns-2 simulations of experiment-1 along with corresponding testbed results of experiment-1. As expected the TCP throughput decreases with increasing number of hops. This is because of the increase in accumulative interference due to hidden terminal problem and the decrease in spatial reuse ratio of the network due to the exposed terminal problem.
In case of testbed results, the most severe effect of accumulative interference and lower spatial reuse ratio on the throughput is observed in the five hops scenario. This severe effect is due to the specific positions of the hidden and exposed terminals in the connected corridors. It is, however, observed that the throughput of the six hops is higher than that of the five hops due to its higher spatial reuse ratio and hence lower effect of the accumulative interference. Similarly the seven hops has higher throughput as compared to both six and five hops because of its further higher spatial reuse ratio. The higher spatial reuse ratios in the six and seven hops scenarios are attained due to the specific positions of the nodes in the corridors.  all other scenarios except for the six hops scenario. There the simulation results of all fading parameters m are almost identical with the testbed results. Fig. 3(b) shows that for the single hop scenario the ns-2 simulation results are almost identical with that of the testbed for fading parameter m = 2.0. The figure shows that ns-2 simulated results in three and four hops scenarios are matching with the testbed for various fading parameters m. Notably, simulations of ns-2 and ns-3 fail to reflect the higher spatial reuse ratio behavior of the six and seven hops than five hops like the testbed.
The simulation results from both ns-2 and ns-3 show that none of the fading parameters m has a persistent match with the testbed results in all multihop scenarios. We also observe that in contrast to ns-3, the ns-2 simulations have a closer match with the testbed results except for six and seven hops cases, where ns-2 results are diverging from the testbed larger than ns-3. It is observed as well that the throughput of a single hop TCP flow is higher in the testbed than the simulated results for certain values of fading parameter m. This is due to wave-guiding signal propagation phenomena not correctly captured by the propagation models in the simulators. It is, therefore, hard to conclude which of the two simulators closer reflect the reality except for stating that both simulators give a rough match of the testbed results.

Experiment-2: testbed vs. simulations
In experiment-2, from the simulations of each simulator, we are getting five plots for the average throughputs (over all scenarios) for five Nakagami parameters. So for sake of simplicity of explanation and limitation of pages of the article, we present comparison of the results with a concise performance metric call throughput fairness index. We find that fairness index behaviours of both the simulators are quite different from the testbed, which implies that average throughputs behaviours of simultaneous flows of simulations are also different from those of experiment-2 of the testbed. The details of the throughput fairness index are given as follows.
In experiment-2 we compared the performance of the simulators with the testbed using Jain fairness index f s (5). The index takes values between 0 and 1. In (5) X (s) i is the network's throughput share obtained by i th flow and s is the number of simultaneous flows.
In order to use the index in the case of flows with unequal characteristics we have to relate the actual measured throughput with a throughput share of the flow under ideal sharing condition [4]. Therefore, X (s) i is computed as in (6). There a As we observe from Fig. 4(a) and Fig. 4(b) the fairness indices obtained from the ns-2 and ns-3 simulators have opposite trends for different values of simultaneous flows and the number of hops. It is observed from ns-3 simulations in the Fig. 4(a) that the fairness index is decreasing in scenarios one to four. The lowest fairness indices are observed in scenarios five, six and seven for certain values of fading parameters m. Like in the testbed, ns3 simulations of scenarios five and greater show an increasing trends in the fairness indices for different fading parameters. Clearly the fairness index obtained from ns3 simulations is lower than the one in the testbed for all values of the fading parameters. It is however worth pointing out that the overall behavior of the index for Nakagami parameters m = 1.5 and m = 1.75 matches the index's behavior in the testbed in scenarios three to seven.
Looking at the results from ns-2 simulations in Fig. 4(b) we observe that the fairness indices increase with increasing number of simultaneous flows and hops. In scenario three the fairness index in simulations matches exactly the values in the testbed. Although the absolute values of the index obtained from ns-2 do not significantly deviate from the measured in the testbed, however the overall development of the index is different from testbed.
It is to be noted that wave-guiding propagation phenomena of radio waves in the testbed are present in both single and simultaneous flow scenarios. However, in the simultaneous flow scenarios the probability of concurrent transmissions along the multihop path is higher than single flow scenarios. This results in higher value of the accumulative interference in experiment-2. We know that wave-guiding propagation phenomena of radio waves reduce the signal strength loss as compared to common radio waves propagation in space. So due to wave-guiding propagation, the accumulative interference has higher range to affect the desired reception of the signal along the multihop path. Hence the deviations of the simulations of simultaneous flow scenarios from testbed are larger than single flow scenarios.  Overall, as in the case with experiment-1 none of the simulators was able to exactly reproduce the performance of the testbed. Partially it depends on simulator specific implementation of protocols on MAC and (or) Transport layers. ns-2, for example, uses own implementation of TCP-Cubic congestion control. We however observe a better match of the network behavior produced by ns-3 simulator which uses native Linux implementation of TCP. The major problem in our opinion comes however from inability of the simulator's propagation models to capture all signal impairments mechanisms in the particular communication environment. The corridor environment of the tesbed exposes strong wave-guiding propagation phenomena, which places a great impact on the accumulative interference and the spatial reuse ratio.

Conclusion
This paper presents a comparative study between testbed and simulations of the network simulators ns-2 and ns-3. The testbed is multihop wireless network deployed in corridors in a non-linear chain topology, which challenges the commonly used empirical models of wireless propagation channel that are currently available in these simulators. The experiments done for this study include single and concurrent flows transmissions over a single multihop path. The goal is to explore how well these simulators reflect the reality represented by this testbed carrying those flows.
Our simulations roughly match with testbed results for single flow transmissions, which cause only limited accumulated interference and allow for good spatial reuse ratio of the network. Simulations deviate however more clearly from testbed results for simultaneous flows transmissions. These transmissions increase the accumulative interference compared to single flow transmissions and thereby decreases the spatial reuse ratio of the network. In particular, for simultaneous flows transmissions simulations indicate considerably worse fairness between flows compared to testbed results. This reveals that the wireless propagation channel models of the simulators are not correctly representing the wireless channel properties in the corridors, especially in scenarios involving accumulated interference in difficult environments such as corridors.
Deterministic wireless channel modeling for example based on ray tracing techniques can better capture reality into simulators, but require exact knowledge of location, shape, dielectric and conductive properties of all objects in the environments and it also requires extensive computational efforts for accuracy. This complexity motivates the use of empirical radio modeling although its shortcomings in correctly modeling complex environments. Our paper emphasizes the need for validation when using such modeling to study single channel multihop wireless network performance for more complex environments.
Future work include to extend this study by exploring how results match between simulations and real multihop wireless networks with linear chain topology for other indoor environments as well as outdoor environments. Thereby we can further characterize possible discrepancies important to be aware of when using simulations to predict performance in real networks.