Hi, So we´ve been having this performance issue, and on the capture I can see that maybe 50% of the DC to Core traffic are Out ot Order and TCP Retransmissions. Cisco Nexus device also shows many of these in the Socket Statistics. I do have the same TCP flow going over the same psyical link towards the Checkpoint FW and back to the Core, and then over to the server... but how can I actually be sure if the TCP Retransmissions are not just the members of the same IP flow but detected twice? How does Wireshark detect the Out of order and Retransmission, there is no flag for that, so... some inside intelligence? How do I filter it and compare the packets? asked 21 Mar '15, 00:54  SNArchsCOM |
2 Answers:
With that many Out-of-orders and retransmissions I am pretty sure that you suffer from massive amounts of duplicated packets. This happens if you use a SPAN session with more than one source port or (what my current guess is) when SPANning one or more whole VLANs. What you get is the same packets twice or more when it enters and leaves the switch, which drives the TCP expert crazy. Try deduplicating your capture with editcap. Editcap is a command line tool installed with Wireshark. I recommend running it like this: This will check in a range of 50 packets if there are exact duplicates and remove them. After running this, check the new file with Wireshark again. Otherwise you should read my answer in the other question you commented on: https://ask.wireshark.org/questions/22134/how-does-wireshark-recognize-tcp-retransmission-packets answered 21 Mar '15, 07:12  Jasper ♦♦ edited 21 Mar '15, 07:13 Yes, thanks for your answers, but I actually filtered the capture first, and then analyzed it... and my profe that these packets are not Wiresharks/SPAN fault is that: - I´m doing a SPAN of a single port - On Cisco N5k I can also see the OOO and TCP retransmissions within the TCP statistics - My Checkpoint R76 is also logging many of these OOO and Retransmissions (22 Mar '15, 10:56) SNArchsCOM can you upload a capture file somewhere (google drive, dropbox, cloudshark) and post the link here? If you need/want you can anonymize the file with TraceWrangler @ http://tracewrangler.com . (23 Mar '15, 09:57) Kurt Knochner ♦ |
So here is an update about what happened: I used editcap to split the file, and it turns out that since the traffic flow is using the same physical link to go to the FW in 2 directions (from Client to Nexus to FW and back from FW via Nexus to the Server), it was detecting different flows as retransmissions. The IPs of Source and Destination did match, but the MAC addresses were different, which is why I´m really surprised that WireShark sees this as a re-transmission. Noone had similar experience? PS (@Kurt): I can't just upload a capture to a dropbox, I´m not playing with a Lab here, our client is a financial institution and I´m pretty sure they wouldn't like their data capture to be just... published. answered 01 Jun '15, 07:39 SNArchsCOM 1 See this blog post for an explanation why Wireshark doesn't care about MAC differences: https://blog.packet-foo.com/2015/03/tcp-analysis-and-the-five-tuple/ BTW, if you need to share captures where it all comes down to Ethernet/IP/TCP behavior, just use TraceWrangler with a anonymisation task and cut away everything after layer 4. Then randomize the IP addresses and ports, and you're good to go (you should still do a visual check after sanitization to see if it worked fine). (01 Jun '15, 07:44) Jasper ♦♦ Thanks Jasper, that´s helpful!!! I still don´t get how Cisco Nexus Switches give me the same error in the statistics, I opened the case ... but never got the answer. (01 Jun '15, 08:16) SNArchsCOM |
I would check the IP ID field. That will tell you if you are looking at the same packet more than once or if they are indeed re-transmissions.