Are rich Tracing Features available in ns-3 a Strength?

Introduction.

Without any doubt, ns-3 has rich tracing support—it has more tracing features than any other simulator. Most above all, it has an excellent mechanism for grabbing any event from any layers during a network simulation run. So, a researcher will have the ultimate freedom while analyzing any aspect of a network simulation.

So, what is the purpose of discussing the question “Rich Tracing Features available in ns-3 – really a Strength? Or Weakness?” here?

It is somewhat complicated to understand. In fact, it is a feeling that I am having for more than a decade – but I may fail to express it correctly here. Anyway, I am going to try my best to express it.

I may be asking this question because I started learning network simulation from the days of ns-2. So, those who do not know about ns-2 could not understand what I am worrying about here.

So, I am starting with the fact of the old way of doing standard trace analysis with ns-2.

The old ns-2 simply followed a universal trace format. Whoever writes a new routing protocol or mac protocol or transport protocol or any protocol module for ns-2, simply followed simple rules and the standard trace format. For that, they may need to alter several files in the core simulator code.
Even though it is a hard way of doing it, it made it possible to do trace analysis on ns-2 trace files in a universal way. So, by using a single trace file, one can extract, physical or mac, network or transport, or application layer trace statistics. Of course, researchers may also try to add a few additional tracing features in a non-standard way in their core protocol itself.
But still, the standard tracing techniques will work; so that, it will be possible to compute some elementary metrics such as mac load, network load, etc., by only following the universal trace format.
So, most of the standard metrics that will work on the wired networks and wireless networks may also work on the traces generated by an ns-2 module or protocol like Aquasim (UWSN), MannaSim (WSN), etc., Even though this kind of tracing has its own drawbacks, at least, ns-2 tried to do trace analysis in a universal way, So it simply worked in most cases..
Good Old ns-2 Days!

Now, let us try to understand the concern that I am trying to show on “trace analysis on ns-3”

Trace Analysis on ns-3

ns-3 provides several ways of trace analysis for analyzing simulation results. They are:

ASCII Tracing: This method involves writing the simulation results to a text file in ASCII format. This method is simple and lightweight but has some limitations, such as the large size of the resulting trace file and the difficulty in post-processing. Grabbing the required statistics from this fill will be a challenging task and there is no ready-made tutorial for it.
There is a primitive tool called Trace Metrics(separate project). This GUI tool can parse the ASCII trace file and try to get some numeric data from it. But it is an elementary way of doing trace analysis and will not be suitable for serious research work since it can do analysis on only one ASCII trace file at a time and can not be used to do trace analysis on outputs from a batch of simulation runs.
PCAP Tracing: This method involves writing the simulation results to a binary file in PCAP format. This method is more efficient than ASCII tracing and is capable of capturing all packets and events in the simulation. Each node and device in the ns-3 simulation can dump the data in separate pcap format files so that it can be analyzed using conventional tools such as Wireshark. But this kind of analysis will not be convenient/suitable if you need to find the average statistics of all the nodes in the network and the average of a batch of simulation runs. (Refer section 5.3.2 PCAP Tracing )
Using Flow Monitor‘s xml file and analyse it through the readily available python scripts or other language scripts or using your own parsing method.
While doing research work, repeating a research simulation several times by varying some parameters, one may get several rows of data to prepare some line charts from it. Extracting the required statistics from Flow Monitors at the end of the simulation itself. Using Flow Monitor is an efficient way of doing trace analysis on traffic/packet flows. But, some kind of statistics can not be collected through Flow Monitor itself.
Custom Tracing: ns-3 also provides the ability to create custom traces by implementing your own trace sources. This can be useful for capturing and analyzing simulation-specific data.
SQLite Tracing: This method involves writing the simulation results to an SQLite database. This method allows post-processing using SQL queries. Some new ns-3 extensions and modules using SQLite Tracing in addition to the above conventional methods.
Even, there may be other ways of getting required data from a simulation such as ns-3 Data Collection Framework (DCF),…….

Of course, we can grab any statistics from a network simulation by applying the suitable trace analysis technique mentioned above. But the fact is: there is no ‘authentic‘ documentation available on the internet to really teach you the ‘correct or standard or universal way of doing it‘.

For example, if one tried to do some routing protocol evaluation on ns-3, and try to calculate PDF, e2e Dealy, Throughput, Routing overhead, etc., certainly, they will end up with some problems during trace analysis and will search for solutions in the ns-3 user group. But the researcher only will discover the following :

Non-Applicability of Flow Monitor:

Flow Monitor is an efficient way of doing trace analysis on traffic/packet flows but it totally will not work on the DSR protocol.
You can not calculate routing overhead and mac overhead using the flow monitor.
Of course, you can calculate the routing overhead of AODV using Flow Monitor, but, the same way may only partially work on OLSR – but will not work on DSDV.

No ‘Authentic’ Way of Doing Trace Analysis

Is there any simple, obvious way of calculating all the standard routing protocol performance metrics using ASCII trace files? – No. you can not find a single example of a ‘trace analysis script’ on the internet to do this task using ASCII trace files. It is not at all completely discussed in any of the ns-3 documents on the internet – including the official documents of ns-3.
Is there any simple, obvious way of calculating all the standard routing protocol performance metrics using pcap trace files? – No. you can not find a single example of a ‘trace analysis script’ on the internet to do this task using pacap files. It is not at all completely discussed in any of the ns-3 documents on the internet – including the official documents of ns-3.
Is there any document explaining the way of calculating all the standard routing protocol performance metrics using the trace-source/callback mechanism? Of course. you can do anything with the trace-source/callback mechanism – but there is no authentic document that will explain a ‘standard’ way of doing it for calculating the standard routing protocol performance metrics

Found Facts and Evidence:

In a reply to a message on the ns-3 user group, Tomosso mentioned that ASCII trace is not really consistently used in ns-3 (e.g., some devices don’t output drops there).
There is no document that explains the way of finding routing overhead from an ASCII trace file except a few lines of the idea provided by Mr. Tom
FlowMonitor can’t reliably track the flows sent using DSR.
Because of the way that DSR is implemented, the flow monitor can not collect data.
This long discussion explains why FlowMonitor fails to capture DSR
FlowMonitor can’t help on calculating routing overhead.
Flowmonitor discards packets sent to broadcast addresses.
Soulimane Mammar explains the way of finding the routing overhead of OLSR using flow monitor – the same idea will work on AODV – but will not work on DSDV and DSR.
AODV also will not be tracked anymore by FlowMonitor starting from ns-3.20.

If you are one such researcher and trying to do trace analysis in ns-3, then certainly you will end up reading replies from Mr. Tom Henderson, Mr. Tommaso Pecorella, and Mr. Konstantinos. Their replies are somewhat authoritative since they are much involved in the development of ns-3. So, after reading the replies of the experts, still, the big question in the minds of the researcher is: how exactly we should do trace analysis? Where is the document explaining the standard way of doing it with authentic examples?

The Net Result is:

While developing new extensions and nodules for ns-3, no one is following any rules or standards during designing their own tracing techniques. For example, if you check LoRaWAN ns-3 extension, they designed their own metrics and their own way of doing trace analysis. Designing a new set of metrics for new technology is good—but there is no standard way of doing that.

So, if you install similar LPWAN extensions NB-IoT and ns3-Sigfox, there you will find their own way of doing things. The net result is: you can not compare the performance of these three different technologies LoRaWAN, NB-IoT, and ns3-Sigfox on ns-3, even though all of them are available for ns-3.

In addition to that, if someone will write a new state-of-the-art routing protocol for LoRaWAN, sadly, they can not compare it with any of the existing routing protocols of ns-3 because of obvious reasons – they are not at all following any standards in their tracing design.

Conclusion

Of course, ns-3 also tried to mimic the ns-2 way of tracing in its ASCII trace. But, it simply failed to maintain a universal standard in it. ns-3 provides an excellent ‘trace source and callback mechanism ‘, but …

In my earlier blog article “What Makes ns-3 a Complex Thing to Understand and Use?“, I mentioned that “There are so many standards (non-standards) in doing trace analysis in the case of ns-3. These standards (non-standards) are another main obstacle in front of a student or researcher that makes it difficult to understand and do research under ns-3“.

In his reply, Tom said, “We have felt that the ‘non-standards’ was a feature, not an obstacle. There are many choices, some built-in (ascii, pcap, flowmon) and the rest (trace sources, logs) customizable.”

But still, I am not convinced with the customizable trace features since they are simply making a research simulation non-repeatable if the source code of the original simulation is not provided. Even though, as mentioned by Tom, there are new approaches that encourage the publication of source code as in the case of WNS3, still, it is common that researchers could not able ‘publish’ a source code for several other personal reasons. Still, a universal way of doing trace analysis is required.

For example, even though ns-3 has several ways of doing trace analysis, no way is obvious or simple to find ‘classical’ routing overhead. Of course, it may be possible with ASCII and pcap trace, but where is the guide to teach the universal or standard way of doing it? (as in the case of old ns-2)

Overall, ns-3 provides a range of trace analysis methods, each with its own advantages and limitations, along with a lot of confusion. In fact, the variety of features available for trace analysis is giving excellent freedom to researchers. But the same aspect reduces the reproducibility of the exact results of one particular simulation by another researcher since both will try to implement the trace analysis logic completely in a different way.

Proposed Solution for the Future Simulations of ns-3

What should be needed and to be done in this regard?

Anyone of the existing tracing methods should become a standard one and there should be a document strictly defining its standard as in the case of ns-2’s trace format. I feel that the trace source and callback mechanism is a good one to follow as a standard.

So, there should be some fundamental functions implemented in all modules on ns-3 – so that, anyone can use them readily without actually writing their own callback functions. I mean there should be ‘one’ standard, default trace object, from that, most of the basic performance metrics can be attained.

If needed, a user can extend its functionality in their simulation by writing their own additional callbacks by extending the same trace subject of ns-3. So, for example, if the name of that Standard Tracing object is Ns3EventTracer, then by default, it should contain some functions implemented in it:

Ns3::EventTracer.getTotalMacLayerTx()
Ns3::EventTracer.getTotalMacLayerRx()
Ns3::EventTracer.getTotalPhyLayerTx()
Ns3::EventTracer.getTotalPhyLayerRx()
Ns3::EventTracer.getTotalTransportLayerTx()
Ns3::EventTracer.getTotalTransportLayerRx()
Ns3::EventTracer.getTotalApplicationLayerTx()
Ns3::EventTracer.getTotalApplicationLayerRx()

And even we may include object-level functions such as :

Ns3::EventTracer.Node(NodeID).getTotalMacLayerTx()
Ns3::EventTracer.Node(NodeID).getTotalMacLayerRx()
Ns3::EventTracer.Node(NodeID).getTotalTransportLayerTx()
Ns3::EventTracer.Node(NodeID)..getTotalTransportLayerRx()
Ns3::EventTracer.NodeContainer(ContainerID).getTotalTransportLayerRx()

So, now the user can now extend the basic tracer to incorporate his own event of interest as follows:

Ns3::EventTracer.CustomEventTracer.getTotalMacLayerTx()

Of course, these kinds of functions may be already available on ns-3. But the point is: they should be organized and standardized and documented so that, in the future, everyone will do things in the standard way, at least for some basic tracing tasks.

So, in the future, if one does trace analysis using this hypothetical ‘Ns3EventTracer’, then it will be possible to generate some fundamental outputs that will be exactly the same for anyone trying the same simulation. Even, someone may point out that ns-3 is already having this functionality implemented somewhere on ns-3. But what I am telling is: no one is actually following it in a standard way since there is no official document advising it to do so.

There should be a common official document that explains the basic standards one must follow and keep while doing tace analysis on different kinds of simulations (wired network, wireless ad-hoc network, routing protocol, transport protocols, mac protocols etc.,)
There should be a basic guideline for new module developers (researchers and industry people) to restrict them to providing some basic tracing functionality in their module, in a very standard way to fulfill the needs of ‘Ns3EventTracer’, irrespective of the technology they are working on.
I mean, different people or groups who are working on different technologies such as 2G, 3G, 4G, 5G, and 6G, should at least add code for some basic metrics like Network Overhead, Network Load, and Application Layer throughput through the standard ‘Ns3EventTracer’, so that the same trace analysis logic will work on the simulations of all those technologies even if they are developed by different people with different scope.

I think, the existing modules or protocols that are developed for ns-3 were already got complicated too much, and made it impossible to compare one with another with a common metric under a common simulation framework. If the official developers and maintainers of ns-3 will not focus on maintaining a standard for trace analysis, then in the future, every module developed for ns-3 can only be used as a stand-alone piece of software. I mean, if someone will develop a module for a hypothetical 7G technology, then it will become impossible to compare it with the other existing ‘TechnoloGies’ that are already available for ns-3.

Still, we are having time to rectify this. Because there are a lot of fine, open-source ‘third-party extensions or modules available for ns-3 that were not at all merged with official ns-3. If the developers stick to some standards for doing event tracing, then the module developers will follow it and it will make the future ns-3 a much more practical network simulation software for research simulation (that is repeatable by anyone).

Note: My intention in this article is not to criticize ns-3 and the original views and scopes of the developers. I respect their hard work and views. I believe that event tracing and trace analysis is the major obstacle that makes an ns-3 research simulation a non-repeatable one. Even a simple reply from experts will strengthen ns-3 and will help future users of ns-3.