hi all, Throughput = TCP maximum receive window size / RTT I have some question regarding this. Throughput = amount of data transferred over a given period of time. so if more data transferred - higher throughput. TCP maximum receive Window Size = what is result of the Bandwidth Delay Product = Bandwith x RTT RTT = time to get to the destination and come back (can use ping). question number 1. when i calculate BDP, Bandwidth will represent the theoretical Throughput of a given link. so if it's a 1 GigEthernet link i take this value convert to bits/bytes x RTT ? question number 2. RTT = yes i can ping the destination host and look at the results BUT isn't it like this that the RTT can fluctuate, i mean when on its way back it can take a different rout for any reason. which value should i put under RTT the Min/Max/Average ? i also found some entries saying to use the One-Way-Delay but on the other hand many people wrote that this is not the correct way to calculate it. PS. i know this are probably very easy topics to most of you guys but i someone could please spent few minutes to answer this , it would be awesome. thank you all and have a great rest of the weekend :) asked 15 Mar '15, 05:11  adasko |
2 Answers:
1) BDP = link speed in byte/s times RTT in seconds. E.g for a 1GBps link with 23ms delay the result is 1,000,000,000 bits per second divided by 8 = 125,000,000 bytes/s. 23ms = 0.023 seconds. 125,000,000 * 0.023 = 2,875,000 bytes, so your optimum Window size is 2.875 MByte, close to 3 MByte. This is only possible by leveraging Window scaling as described in RFC 1323. 2) yes, RTT can fluctuate, but if you determine iRTT by looking at the three way handshake it's almost always the best RTT you can get. It's not a problem if the BDP is a bit larger than you actually need, same as it isn't a problem if its a bit smaller. Keep in mind that BDP is just something that works for a single TCP session - so if your link is used by more than 1 concurrent session the BDP applies to their combined activity. answered 15 Mar '15, 08:43  Jasper ♦♦ wow thank's for the link. that helped me a lot. will definitely get back to it on a regular ! just to understand it completely , it says " Another point is that the handshake packets are handled by the TCP stack of the operating system, so there is no application interference/delay at all." what exactly do they mean by this sentence (that this packets are never gonna make it to the application layers ? ). also " Keep in mind that BDP is just something that works for a single TCP session - so if your link is used by more than 1 concurrent session the BDP applies to their combined activity." so you mean when BDP is calculated it will determine the max WS to completely utilize it for a single connection ( for example backup scenario), yes ? and if i have more thn one open connection i would have to divide the BDP with the number of connections , yes ? (15 Mar '15, 09:06) adasko 1 Yes, by that sentence I mean exactly that (it's my blog, btw ;-)) The three way handshake does not reach the application layer. The application gets notified of a new connection after the three way handshake is complete. And yes, BDP is relevant to a single connection. And yes again, if you have more than one connection they all share the bandwidth - you could divide by the number of connection, but only if they're all doing pretty much the same kind of transfers. (15 Mar '15, 09:24) Jasper ♦♦ cool :-) but BDP will represent the max receiver and sender WS, yes ? also while calculating throughput : TCP-Window-Size-in-bits / Latency-in-seconds = Bits-per-second-throughput where do i get the TCP-Window-Size-in-bits from ? will this be BDP ? and last but not least, will the throughput value (similar as BDP) apply only to 1 connection ? thank you ! (15 Mar '15, 09:37) adasko 1 I guess the window size would come from a sample connection that you've captured already - at least that's the only real number you can have, everything else would be made up. And that window size is BDP if it's from a single active connection that has reached maximum throughput (remember that most TCP stacks start slow and increase the window size as long as no packet loss occurs, so it takes a while until it's at the maximum) (15 Mar '15, 09:42) Jasper ♦♦ i see. so in a scenario i run a backup from host A to host B, the Window Size from advertised by host would be used for the calculation ( the one captured during the 3 way handshake - same where i take the RTT from, yes ? ). can you please answer me only this and i will not bother you anymore :) (15 Mar '15, 10:06) adasko 1 Unfortunately, it's not as simple as looking at the window size of the handshake, because that is usually only a temporary value. You should try to find the maximum window size of the transfer, or, even better, take an average of a couple of the largest values you find. (15 Mar '15, 10:10) Jasper ♦♦ i see, so it's really not that easy. so i run my backup (it's just an example) with wireshark enabled. i try to find the highest WS and try to calculate an average. if i have it and would like to calculate the throughput = WS (what i got from my calculations) x RTT ( actually the iRTT). but again, wouldn't this mean that i'm calculating the throughput same as i would use the the BDP value as the WS ? so to calculate the most accurate throughput between two nodes i think i would have to look at ALL connections and try to compute the WS from all of them or am i wrrong ? (15 Mar '15, 10:44) adasko yes, you'd need to look at all of them. But may I ask why you're all doing this? I only think about BDP when some connection is slower than it should be, and even then, the first thing I do is to check if the window size is the problem. I do that by checking for "Window Full" messages and determine the recovery time from that (basically, the time between the full window and the next segment the server is allowed to send). Only if that happens often and/or accumulates to significant delays (meaning: a delay measured in at least seconds, not milliseconds or less) I'd think about BDP. (15 Mar '15, 12:45) Jasper ♦♦ thank you Jasper. why I'm doing all this? it's very interesting for me and I'm learning / trying to learn all this stuff to get a better understanding of how it works under the hood. many people i met that used to work in IT (even me i have to admit) didn't know that and i think this is totally wrong. maybe because of my lack of experience and expertise i tend to dig to much in something i don't know :/ i also found this : http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ so i started to think about it all. and as i work in support for a backup product i often have to deal with cases where customer complain about the low transfer rate (unfortunately and i have to say it i work with a bunch of idiots that pretend to know everything but they know nothing / nada / nichts - please apologies but it's a fact). so i have cases, for example: dedicated 10 Gig-Ethernet connection, backup going to disk and the customer is unhappy about the transfer rate. what they often do (customers) they do a Windows File copy backup from the source host to destination and compare it with the rate from the backup product. as far i know and understood it right a Windows File Copy is measuring the goodput Edit: i don't demand from people to teach me everything from scratch without moving a**. NO, i spent hours even on weekends to learn, all i need is sometimes to clear my doubts to confirm something. But i think if I'm the Level 1 guy and not even the Level 3 guys can do it hmm, that's not as it supposed to be i guess. (15 Mar '15, 13:05) adasko 1 Okay, I see. So there is a real world problem behind all this. Calculating BDP is a theoretical thing, because it's rare that only a single TCP connection utilizes a link, but it may be an interesting number to know what the maximum Window size is. BDP was much more important with Windows XP and earlier, because starting with Vista, the TCP stack does Window Scaling automatically. With XP, it often happened that the Window size was too small without enabling RFC 1323 options in the registry. So if the OS version involved are Vista/Server 2008 or newer, you can assume that the TCP stack does Window scaling. There are two problems with Window sizes: one is when it's too small, forcing the sender to pause transmission. The other is when it's too big, leading to packet loss because the infrastructure gets overloaded. Both have specific symptoms you can observe in a capture. Window sizes too small result in "Window Full" messages and long delays. Window sizes too big result in massive packet loss and Duplicate ACKs that go up into hundreds or even thousands for the same sequence number. Windows file copies have improved with SMB2 and SMB3, but its hard to compare to another transfer when you don't know how it behaves. If I were you I'd try to get traces of the slow connections when a customer complains. With the actual packets it should be possible to determine where the problem is (Window size too big or too small). With a 10G connection it may also be the backup target, because if that connection is fully utilized you're talking about writing at a maximum speed of 1.3GByte per second to disk. Trouble is, it could be hard to actually capture a full speed 10GBps transaction, because that requires specialized capture hardware. (15 Mar '15, 13:25) Jasper ♦♦ not sure whether I'm allowed to take traces like Wireshark and as you wrote with a 10G network would need to have special hardware . may i ask you if this applies also for other standards like 1 G , FastEthernet is this a Wireshark limitation ( i suspect that the capture hardware need sufficient CPU, RAM etc.) is there any official wireshark KB on this ? "Windows file copies have improved with SMB2 and SMB3, but its hard to compare to another transfer when you don't know how it behaves." so if assume Windows File copy is using SMB2 and i would see based on the trace file that so is doing the backup application could i compare both rates ? where would you place Wireshark in this case ? the sender or receiver or probably on both . . . "With a 10G connection it may also be the backup target, because if that connection is fully utilized you're talking about writing at a maximum speed of 1.3GByte per second to disk." you mean he speed at which we take he data from he buffer and the actual write to the HDD yes ? (15 Mar '15, 14:07) adasko 1 It's not really a Wireshark limitation, it's more like that standard PC hardware is not suitable for high speeds. PC captures with normal NICs can probably do up to 200-300 MBit/s, even with fast SSDs to write to. There are no KB entries as far as I know, but you might want to take a look at some of the Sharkfest presentations on this, e.g. by Chris Greer: https://www.youtube.com/watch?v=_H7PjWqKV0Q If both backup and Windows File Copy use SMB2 you could compare them, yes. Capturing this at both locations would be best, but if you can't do this, do it on either side. And yes, write to HDD. (15 Mar '15, 14:17) Jasper ♦♦ hmm i like the video from the beginning "traffic generator" if i can't get a real trace maybe it will be at least something to run some simulations ;) thank you very very very much Jasper for all your answers . i really appreciate your time and knowledge sharing. looking forward to hear from you soon if new question will come up. btw your web page is already in my favorites :) (15 Mar '15, 14:29) adasko you're welcome :) (15 Mar '15, 14:33) Jasper ♦♦ Jasper, just on more. "BDP was much more important with Windows XP and earlier, because starting with Vista, the TCP stack does Window Scaling automatically. With XP, it often happened that the Window size was too small without enabling RFC 1323 options in the registry. So if the OS version involved are Vista/Server 2008 or newer, you can assume that the TCP stack does Window scaling." so with windows scaling i understand the Window Size can be more than 64 k (65,535 bytes) so why we still have to deal with Window Size to small / to big problems ? what could cause this ? (16 Mar '15, 03:41) adasko 1 oh, lots of causes... e.g. badly written NIC drivers, embedded devices with rudimentary implemented TCP stacks, misconfiguration... most of these do not apply to modern stable desktop operating systems, but the driver issue can be a problem even there. I've had one customer where the 4G modem card driver on Windows would always force a 4MB window, which lead to overload on the infrastructure. (16 Mar '15, 04:46) Jasper ♦♦ hmmm this created even more question in my head. so this is related to what you wrote about earlier Windows Version like XP, W2K3 : the maximum receive window size can be manually configured. The HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCPWindowSize and HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Interface\InterfaceGUID\TCPWindowSize registry values can be set to a maximum of 65,535 bytes (without window scaling) or 1,073,741,823 (with window scaling). Third, the maximum receive window size can use window scaling. You can enable window scaling by setting the HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts registry value to 1 or 3. i cannot find any of those reg keys on Windows 7 and Windows 2008 r2 Server (you also mentioned this, so this is clear for me ). but "To optimize TCP throughput, especially for transmission paths with a high BDP, the Next Generation TCP/IP stack in Windows Vista and Windows Server 2008) supports Receive Window Auto-Tuning. " "The Next Generation TCP/IP stack no longer uses the TCPWindowSize registry value. Receive Window Auto-Tuning has a number of benefits. It automatically determines the optimal receive window size on a per-connection basis. In Windows XP, the TCPWindowSize registry value applies to all connections. Applications no longer need to specify TCP window sizes through Windows Sockets options. And IT administrators no longer need to manually configure a TCP receive window size for specific computers." source: https://technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx so now i'm wondering ... if we are unable to set / modify the window size value on any OS above XP/2003 how the heck we can deal with for example a Window Size to small case. so based on a Wireshark capture i can tell that this is the reason but what about the cause . so like you wrote your example with the 4G modem card forcing a 4 MB Window. also "Applications no longer need to specify TCP window sizes through Windows Sockets options" so if i'm not wrrong we have no option to change the Window Size on the newer MS OS's because all what we could previous adjust manually this is handled right now by the new TCP/IP stack. also are you aware of any type of documentation describing the initial process of detreming the Window Size on both ends ? i found something related to Windows CE (what the heck is this ) but i think it's not valid (16 Mar '15, 13:59) adasko showing 5 of 17 show 12 more comments |
A more complex calculation with the serialisation time and a graphical overview is possible with the java-tool from http://www.aicalc.net (unfortunately the description is partially in German.) answered 15 Sep '15, 09:13  mgoerke |
or do i use 65,535 bytes as the maximum window size when calculating the throughput:
Throughput = TCP maximum receive window size / RTT
also the BDP is the needed Window Size the fill up the pipe, so from my understanding it equals to the maximum TCP Window Size. so if i do max TCP WS x RTT = max throughput when the max WS would be use, right ?
it's this the correct way to calculate it ? the formula i took from :
https://technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx
"The size of the Window field in the TCP header is 16 bits, allowing a TCP peer to advertise a maximum receive window size of 65,535 bytes. You can calculate the approximate throughput for a given TCP window size from the following formula: Throughput = TCP maximum receive windowsize / RTT"
what do they mean by " for a given TCP window size" do they mean the TCP MAX WS ?
Please note that most systems use window scaling these days as a 16 bit value for the WS is not sufficient anymore. This means the WS can be (a lot ) higher than 65535.
SYN-bit as Jasper is away could you please look at me last question in this thread please please :) i will be not able to sleep until i know