Hi, I have two capture files, client facing and server facing. There is a device in the middle that is dropping packets and I have identified the device, however, I require a little more evidence. I would like to know if there is a way to merge these two files together and "see" with a graph or other method, that lines up the missing packets. Ex: Packet 201 (authent packet) is sent from the client, but never "seen" on the Server side. Packet 200 -- sent from client / Packet 200 seen on Server side Packet 201 -- sent from client / ...not seen on Server side Packet 202 -- sent from client / Packet 202 seen on Server side Packet 203 -- sent from client / Packet 203 seen on Server side Packet 204 -- sent from client / ...not seen on Server side My captures occur on each side of the device. I have a suspicion that there are multiple instances of this behavior but I feel that I'm not using the best discovery method to determine which ones, how often, and how many packets are getting dropped. I'm currently working with WS - 1.8.4, thanks, JTech asked 29 Jan '13, 12:12  JTech_17 |
3 Answers:
JTech, go to Riverbed.com and somewhere there, you should be able to download an eval version of Pilot program (Yes, I do work for Riverbed now, but I'm not trying to sell you anything) In Pilot, there is a Multi Segment Analysis that does what you're looking for. It will take two (or more) trace files and will give you a report on packet drops etc. So if you're looking for a one time thing, download the eval, use it, and uninstall it when the timebomb goes off. No harm, no foul. Or as Jasper mentioned, export TCP Sequence numbers or IP ID field. Sometimes IP.ID will make it through LBs etc. If you're lucky, it may be as simple as doing a diff on two columns of IP ID field. BTW, what additional evidence do you need? It seems like you already identified which part of the infrastructure is dropping packets. One word of caution. Make sure it's not the span port that's dropping the packets. If there is packet loss, you have to see retransmissions. So don't get fooled into thinking devices are dropping packets when it's just the span/monitor port that's responsible. answered 29 Jan '13, 19:26  hansangb edited 29 Jan '13, 19:28 The reason I require additional evidence, is, when I showed the device owners the data, they denied it. Yes, I realize how crazy that sounds, but believe me, I was a bit surprised by that reaction myself. I too suspected a span port issue, however, there is only one user traversing through the device and ending up on the other side of this core switch. I plan to place an inline tap between the device and the core switch...simply to rule out a span port issue. If I see the same behavior then I'll know my original findings are valid. thank you, JTech (30 Jan '13, 05:54) JTech_17 The Keep_Alive and Keep_Alive_ACK packets were not seen on both sides. Pilot was able to show this, though it did take some time to learn the navigation within Pilot. we already use Riverbed Steelhead appliances for our WAN links (we just don't have the Pilot app as part of that contract - though req has been submitted). I placed a NetOptics inline tap between the device and the core, to rule out any span port issues, and the results were exactly the same. The device owner is still skeptical, but that is ok - denial is a hard mountain to climb over. thank you, JTech (06 Feb '13, 04:47) JTech_17 |
Dump the two files as text files, the MD5 frame hash only. Corresponding frames should have the same hash in either trace. Or is it a router, NAT or other non-bridge device? 'cat' the text files, 'sort' them based on MD5 hash and see how many 'uniq' entries you have. Hint: the quoted words are handy command line tools for this. answered 29 Jan '13, 12:41  Jaap ♦ |
If there's just a up to a few hundred packets to compare I'd proabably just graph the TCP sequence numbers with the Statistics/Time-Sequence Graph for both locations, color them differently in a drawing/picture editing tool and overlay them with each other. That way it should be easy to spot where a dot is missing - but as I said, this only makes sense if you do not have tons of packets that will make the graph so crowded that it becomes unreadable. Or you could just export all sequence numbers, sort them by order and compare the two sides. If you have sequence numbers twice and if it is seen only once on the other side it is most likely a retransmission. This only makes sense if you filtered the trace to the one tcp connection, and if sender and receiver use quite different sequence number ranges (which they usually do, but the inital sequence is random, so theoretically they could be even identical for both directions). Sequence numbers tend to stay the same even when passing through NAT gateways and other devices that change parts of the packet. Proxies will be a problem though, as they create completely new TCP sessions and you can't match anything for sure except maybe the actual TCP payload (if there is no encryption taking place). answered 29 Jan '13, 17:08  Jasper ♦♦ |
Thank you all for the answers. I will try each one and provide feedback.
JTech
@JTech_17
I've converted your "answer" to a comment as that's how this site works, please see the FAQ for more info.
If any of the answers do solve your issue, for the benefit of other users, please accept the answer by clicking the checkmark icon next to it.