How does TCP handle application's small packets? We are currently using an application which is generating small packets all the time. Data length are 98% at 50 Bytes for each packets. On the LAN, it's not too bad, 16 seconds for a single session. However for those remote users(through MPLS WAN), it takes 3 minutes to finish a single session. I am just woundering what I can adjust to let the application get better performance. So far I'v tried: 1. Increased TCP windows size(later I found out it won't work for this case) 2. Reduced MSS/MTU from 1460/1500 to 1340/1380(got better result, but still takes 90 seconds for a single session, should I try reduce further more? But I need a theory to support this idea) Thank all asked 27 May '13, 08:52  Lei |
3 Answers:
Without looking at the pcap file it is hard to give a thorough advice. But most likely you are bit by the amount of application turns and/or the nagle-delayed-ack problem. Since the LAN users are doing fine and the WAN users are not, I suspect it is the amount of application turns that is killing you over the higher-latency WAN. Is your protocol written in a way where each piece of data that is sent is acknowledged at the application level? And there are a lot of pieces of data that need to be exchanged, then this is the problem. Every acknowledgement will cost you a RTT. A rough estimate: Considering a 16 seconds session on the LAN, this could be caused by 8000 application turns with 2 ms RTT. If on the WAN, the RTT is 25 ms, then you will see 8000 x 25 ms = 200 seconds for the same session. TCP works best when data is being streamed instead of being sent one chunk at the time. If this is a database application, then you will rather send a whole table in one go, then send the table row by row or even cell by cell. answered 27 May '13, 09:17  SYN-bit ♦♦ "then send the table row by row" is probably a typo, right? I guess you mean "than", or better "in one go instead of sending...". Not trying to be a wiseguy, just seeing there may be confusion on the other end ;-) (27 May '13, 09:21) Jasper ♦♦ Thank you both of you, Jasper, The latency on the WAN is around 20MS, witch is very high. Vendor just changed recently, from L2 MPLS to L3 MPLS. Before we are running on a LAN extension, now we are on Pseudo Wire, by any chance you know that LAN extension(pure L2) has lower latency then Pseudo Wire(L2 on a L3 technology)? SYN-bit, Since the pcap is caught on the real life, which is with high privacy, I cannot load it, sorry for that. But I see exactly what you said. It is an application that keep exchanging the data, on the LAN, 8000 packets back and forth in one seconds, on the WAN, 370 packets back and forth in one seconds. Is that means nothing we can do? Thanks Lei (27 May '13, 09:34) Lei Thanks for the correction Jasper. Yes, I meant to say "than" :-) (27 May '13, 09:36) SYN-bit ♦♦ Please use "add a comment" to the answer you want to respond to, that is the way this site works best (see the FAQ). I converted the answer to a comment to my answer, to continue the discussion on the application turns, which I believe is the culprit as latency over a WAN is usually something you don't have control over. There is one thing you can do. Talk to the application developers and tell them to test their software over a WAN (simulator) before deploying it live. They can optimize their code to not do so many application turns and so reduce the effect of a higher latency network connection. (27 May '13, 09:39) SYN-bit ♦♦ IMHO the steps forward would be: 1) Analyze the traces thoroughly to pinpoint the cause of the delay (without speculation as we have done till now :-)) If it is indeed application turns, then follow with the next steps: 2) Try to convince the vendor to rewrite their software, but that will be tricky. 3) If you can't reduce the latency, which you normally can't. You can let remote users use a remote desktop session on your central location to use the application as the application is not written for WAN (high latency) access. (29 May '13, 09:18) SYN-bit ♦♦ |
My guess is that the accumulated latency is killing you, because small packets have to run back and forth all the time, which makes the round trip time a rather large factor.
What you need is to reduce the latency, which means that you need to find a way to get lower round trip times. That is hard to achieve; sometimes you can remove slow devices that forward packets with some millisecond delay by ones that can do it in microseconds. Remove slow proxies, reverse proxies, load balancers etc - best would be to only have routers and switches with microsecond latencies, but even then the distance delay can be a problem. I had customers move database servers back to the same switch where the application server was for exactly this reason: low latency. Check your latency in the LAN, and you'll probably see RTT below 1ms. Check the same for the WAN, and it is probably a two digit number of ms. answered 27 May '13, 09:00  Jasper ♦♦ |
can you post the name here?
well... that's absolutely normal today. Some of those 'coding experts' use huge code libraries without thinking or knowing about the internals of those code beasts and definitely without knowing and thinking about the network behavior of their software. The usual test works like this: The developer runs a copy of the web server (or server software) on his laptop. If he/she is able to connect to localhost, the software works in his/her eyes. 'Sometimes' there are tests over the local developer Gigabit LAN (client on laptop 1, server on laptop 2). There are almost no tests at all over any WAN connection. So, your only chance is to search for the product name and 'network problems' on google. If you are lucky, others have found (hidden) options of the software to run it faster on a WAN connection. If your are out of luck, the software will never run fast enough on a WAN connection, due to design errors. The only option then is to use an even faster connection (smaller RTT), or to run a clone of the server system locally (however that is done with that product). Maybe it is also possible to tweak the TCP/IP stack of the server/client, depending on some conditions (as @SYN-bit mentioned - Nagle algorithm together with Delayed ACK). But for any advice in that direction, we would need the capture file, at least up to the TCP header (including flags). You don't have to post any sensitive data, just the communication up to the TCP layer. If you like, you can obfuscate the IP addresses as well, then it's impossible to extract any data from the capture file. See the following artice of @Jasper: http://sharkfest.wireshark.org/sharkfest.11/presentations/A-11_Bongertz-Trace_File_Anonymization.pdf Regards answered 28 May '13, 07:35  Kurt Knochner ♦ edited 28 May '13, 07:45 Hi Kurt, Thanks for the help. How can I load the capture without data part? I am not worry about the IP address, but don't know how to load it here without the data in each packet. (28 May '13, 09:11) Lei You need to strip off the payload. You can do it with editcap.
Then post (28 May '13, 12:41) Kurt Knochner ♦ Hi Kurt, the application is Razor's Edge. (29 May '13, 07:01) Lei I'm sorry, but I can't find any information about that application. Who is the vendor (link to their website)? (29 May '13, 07:36) Kurt Knochner ♦ Is this The Raiser's Edge?
(29 May '13, 08:28) Kurt Knochner ♦ any update? (31 May '13, 03:09) Kurt Knochner ♦ showing 5 of 6 show 1 more comments |
what kind of application is this? Why is it sending so many small packets? Are those request/response cycles (queries to a database with a huge amount of single SELECT statements)?
It's an finance application. Spoke to the vendor, they seemed to be no clue on how their software designed.