slow SMB file transfer

The SMB trace suggests that the client (FreeBSD VM) is slow. Yes, it would be nice to see it with segmentation offload disabled but I'm not sure it matters in this case. The big packets are coming from the Windows machine and the traffic was captured on the Windows machine. The 7 second delay isn't from the big packets it's from the next read request from the client.

The FTP trace is similar. There's a large burst of traffic at the beginning (peaking at 50,000,000 bytes/sec) but then quickly slows down to (and stabilizes) around 4,000,000 bytes/sec.

I don't see much in the way of transport problems (e.g., packet loss). I don't know if the segmentation offload is hiding this--I've never worked much with TSO enabled.

These two traces suggest to me that the (FreeBSD VM) client is slow and I would further theorize that it may be the filesystem that is slow.

You might want to time some (large) file creations on the VM to test the theory. For example:

time dd if=/dev/zero of=/some/file bs=1m count=500

Will create a 500 Mbyte file and tell you how long it took. Do some math and find out what the filesystem write speed is. If you get something on the order of 4 Mbytes/sec (like the FTP transfer) then that's your problem. If it's much faster then my theory's no good.

Note: the above would work on Linux; I think it will work on FreeBSD too.

I had a look at the capture named "2gb".

It is missing a lot of packets because the sniffer dropped them, not because the network dropped them. These occur at very regular intervals - not randomly. At the time period I delved into, for every 35-37ms of data captured there was then 70-74ms of data missing. This would lead me to believe that the capturing mechanism has an inherent flaw, perhaps based on the virtualisation.

Nevertheless, we can find some useful information.

Although we don't see the 3-way handshake for the main SMB connection, other SYN/SYN-ACK setups between the same client and server don't support Window Scaling. Therefore, we can only ever have 64KB in flight (i.e., per round trip). Put another way, we can only do one SMB request/response at a time (because each SMB requests a 64 KB data block).

This may not matter in this case because the "slowness" is entirely due to the client.

We are achieving a throughput of ~500 Mb/s because we are achieving only approximately 64 KB/ms. This is because the client is only making one request per 1 or 2 ms. Each request is serviced very quickly and the 64 KB is delivered from the server across the network very quickly. The client usually ACKs the server data quickly, except for the last few packets which are ACKed by the client's next SMB READ&X request packet.

The time gaps between the client's request packet and the last server packet are consistently just under 0.4ms. The time from the last server packet and the client's next request packet is consistently just under 0.6ms. The time between client request packets is consistently 1ms +/- 0.2ms.

The network RTT seems to be around 0.2ms, so the time gaps between client requests are not related to the network. They are time spent by the client dealing with the received data before bothering to request the next block. The large 14 second time gap is, likewise, due to the client.

So your hunt is to determine why the client waits so long between requests.

• Could it be that it takes it that long to write the received 64 KB away?
• Could the virtualisation be playing a part? Eg, the virtual servers or network components only get a "turn" once per millisecond?

I'd also be interested in why Windows Scaling doesn't appear to be enabled. The client doesn't request it in its SYN packets. It would be interesting to know if enabling it would make a difference.

Happy hunting!