I use a Gigamon Switch to tap network devices. The Gigamon performs it's own time stamping and adds it's own header in each packet, the time is taken remotely from a NTP source (GPS). Connected to the Gigamon is a Wireshark server (tool port definition in Gigamon terms). So data is sent from the network device onto passive tap, then onto the Gigamon (network port in Gigamon terminology), data is sent through the chassis and forwarded on to a Gigamon tool port (Wireshark server in my case). When data is captured I see a major difference is time stamps for each packet (usually 8-10ms). An example 1 packet: Frame # - Arrival Time: Mar 21, 2014 11:05:00.091159120 GMT Standard Time Gigamon Header time stamp (length 16 bytes) - Time Stamp: Mar 21, 2014 11:05:00.108663978 GMT Standard Time; Source: NTP As you can see above the time difference between frame entering the Wire and the Gigagmon time stamp in the hello packet is approx 9ms out. 9ms is a lot, I would expect only µs difference. I logged a support call with Gigamon who said that NTP can be out 10ms + , their suggestion to use their own GPS input antenna on the device. I wouldn't expect that an NTP clock could be out by 10ms. As you can see below the Tool (Wireshark Server) the Primary NTP delay is 1ms, well there is a reason for this delay, we are using a NTP Clock located in another Data Centre, until we buy a new clock. The RTT between Data Centres in 1ms (.5ms is either direction) ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== 9999999.hello.c *--ntp 2 u 15 16 377 1.032 0.162 0.008 +8888888.hello.c ***-110-ntp 2 u - 16 377 1.375 -0.150 0.008 My question is how could there is a 10ms delay between the Frame and Gigamon TS? Can NTP be out by 10MS? Anyone seen this issue before. I work for a low latency financial where accurate timing is imperative. asked 24 Mar '14, 03:55  mortirolo |
One Answer:
There can be a broad range of problems, like:
sure, it can be of > 100ms.
If you mean NTP "accuracy issues": Yes
Then you should be using PTP instead of NTP ;-)
Regards answered 24 Mar '14, 15:03  Kurt Knochner ♦ edited 24 Mar '14, 15:30 I have gone back to Gigamon and they have forwarded my case onto their Lab team. Gigamon 2404 model does not support PTP I've already investigated the possibility of the NTP server been overloaded and that's not case nor is there a NTP varying delay to the NTP Server from a similar local subnet. In regards to PTP been more efficient than NTP, well yes, but you need all the hosts (not heavily populated) in the same subnet and in the same broadcast domain in my view. I have been on a conference regarding PTP vs NTP and you have to consider which is best for your type of environment. PTP is not nessecary better than NTP, here is a good article to read http://www.fsmlabs.com/blog/choosing-between-ptp-and-ntp (25 Mar '14, 02:39) mortirolo Below is the response from Gigamon. The 10ms time difference was a Gigamon problem. Looks like their newer large model HD4 time stamps at the ingress phy to give better accuracy. I'm very disappointed especially when you buy a product that promotes accurate time stamping. "Our development team spent some time looking at the best resolution they could achieve in the lab. The best they could achieve ~10 milliseconds, but this was not guaranteed. The reason is that although the GV 2404 main blade is synchronized with the NTP server, there is some delay between the GigaSMART card and the GV 2404's main board in addition to processing time on the GigaSMART card. This in turn prevents a resolution time better than 10 milliseconds." (04 Apr '14, 00:04) mortirolo As I suspected: a bug in Gigamon.
well, after all it's software and thus prone to errors. Furthermore: How exactly do they define the 'time stamp accuracy' in their product data sheets? (04 Apr '14, 10:04) Kurt Knochner ♦ They don't define times stamp in detail, but the SE tells you differently (data sheet = Add Packet time stamps at line rate for subsequent analysis). At the time there was no alternate product in the Gigamon range, TS was always done in the same way through the GigaSmart module, as mentioned above the new HD4 TS is at the phy with their on board ASIC's which must be much more accurate. I'm testing C-Packets at the moment and the TS between wire and C-Packet TS is approx >1 μs, impressive! We should've carried out more testing at the time with Gigamon TS, in the POC. At the time also they were the only aggregator switch to carry out Masking, which was required in our Switzerland office. (07 Apr '14, 00:30) mortirolo |
Google-ing it seems to indicate an NTP client can easily be out of synch by multiple milliseconds, since your NTP server is not in the same data center. Also, you didn't mention it, but is the Wireshark PC synch'ed to the same NTP server as the Gigamon?
Hi, yes the Wireshark Server is sync'ed with the same NTP Server as the Gigamon
I haven't tested our other Wireshark server and other Gigamon in another Data Centre where the NTP clock is local and I will compare and post the results soon. The Wireshark delay here is 0.171ms
Have you tried sending a packet from the Wireshark PC, such that Gigamon monitors it and copies it back to Wireshark? For example if you have a second "live" interface on the Wireshark PC, use Wireshark to monitor both interfaces, send an ICMP ping out the "live" one, and see when you get the copy from Gigamon - that way you can see both the sent and copied timestamps in Wireshark at the same time for the same packet.