Ok, I'm oficially stumped. I have a corporate campus location with a couple of thousand users and all hell broke lose last week. Multiple applications and multiple users affected and the issue was very sporadic. We could find nothing wrong with the local network or WAN but then realized that none of the Network Engineers were having any issues and the only difference between the Network Engineers other employees is that the Network Engineers have the Cisco VPN client installed with automatically lowers the NIC MTUs to 1300 bytes. On a hunch, we added ip tcp adjust-mss 1300 to the WAN routers and the problems instantly ceased for all users. So now we need to figure out what the hell happened. We have taken a capture from a workstation at an affected site and don't really see too much to worry about. There are some zero windows, but nothing to drastic. So my question is, does anyone have recommandations of what to look for in the capture files? We have already ruled out any WAN issues, so this is looking like a problem in the Data Center. Any thoughts would be greatly apprciated. asked 28 Aug '16, 14:58  Bill7710 |
One Answer:
You won't see anything useful from a single capture file. You need to create, between the server and client machines which suffer from the issue in their regular communication, a TCP session which attempts to make use of the available MTU (like a ftp transfer of a large file) and see it fail after you disable the adjust-mss rule on that path. After that, you'll have to capture in two points, first at the server and at the client, and see the long packet to be sent from one of them but not reach the other. Then, you would keep one capturing point at the source and move the other one to the middle of the path to destination, and repeat the test to see whether the large packet has made it to the middle point or not. By repeating these steps while moving the capture points along the path, you should be able to isolate the section of the path which causes the issue. The issue may be a misconfiguration of the MTU itself or too thorough filtering of icmp, preventing the MSS auto-detection from working by filtering icmp segmentation requests. If the issue appears only sometimes even for file transfers, which normally do make use of the MSS available, there may be some dynamic routing in your network, causing the problematic section to be used in some transfers but not in others. answered 29 Aug '16, 06:04  sindy |
