Hi All. I'm investigating some network slowness. As a part of this I'm checking the pcaps of file transfers around my network. I'm seeing 4/5 instances of packet loss per file transfer (regardless of size??) and in each instance there is a large number of duplicate acks being sent before a retransmission is received. Image: Image In the capture I'm looking at at the moment there are 1763 acks sent before receiving a response. Anyone know if this is normal or what I should focus on next? Thanks! asked 11 Jan '15, 06:17  CarlitoGrey |
One Answer:
Looks like classic buffer bloating to me. The problem appears when you send large amounts of data from a high speed network to a lesser speed network real fast, causing the switch or router buffers to fill up. At that point, packet loss will occur, and the receiver will send duplicate ACKs to notify the sender of the missing segment(s). The problem is: since the full buffer is still constantly slammed with more packets the retransmission can't get through fast but has to "get in line" like all the other packets, which means that it takes a long time to get to the receiver. That's the reason why you see very high numbers of duplicate ACKs for the same missing segment. The only thing you can do is to have the receiver advertise a smaller receive window, to prevent overloading the network. answered 11 Jan '15, 07:27  Jasper ♦♦ edited 11 Jan '15, 07:28 |
Thanks for the answer and link to the article - makes sense to me (I'm not a networking guy, just a security guy fed up with the network speed when I know how decent the back end equipment is).
There's only one switch (well a stack) between the client and the server I'm interpreting that to be that the server is filling the switch stack's buffer at 1Gbps, not knowing the client is connected to the switch at 10Mbps.
I'll pass this on to the 'networking' guys.
Woah, 1Gbps down to 10MBbs? That's a nightmare - factor 100 slowdown, no wonder the buffers are exploding :-)
Actually the virtual hosts have four 1Gbps links teamed together. Am I right in thinking they can in theory transfer data at 4gbps? The clients have a Nortel IP phone between them and the switch, the phone only supports 10mbps :(.
I've enabled discarded packets on PRTG and I can see packets being dropped for the link I'm using.
Would the output buffer be shared between all ports on a switch (cisco 3750's in a stack)? If so would transferring a large amount of data between myself and the server also cause the switch to drop packets for other destinations in the output buffer?
In theory, yes, up to 4GPs, but not for a single TCP connection - for that the maximum is one physical link.
Anyway, you should really come up with a plan to remove the 10MBps point of failure as soon as you can - I know this is often easier said when done, but 10MBps is too slow for current networks.
The buffer architecture depends on the switch model - some have a "big" common buffer for all ports, others have small dedicated buffers per port. AFAIK Cisco has 100KByte buffers per port, so you'll overrun them with your kind of problem almost instantaneously.
It may still affect other ports as well, which is kinda hard to say - I'm not sure how the switch deals with that kind of overload precisely.