This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Looking for some ideas.

We have a pretty basic network setup in a remote office with very few users. Our ISP provides us with a 100MBit ethernet connection to their equipment, so this office is connected to our main office literally on a 100MBit switched connection. Traffic is passed to and from this office on a VLAN I created and gets passed through a queue in queue tunnel created by our ISP.

At my end, I have a cisco catalyst switch. At the other end they have a 3Com 4400. They have been complaining of very general network speed issues for quite some time. To troubleshoot, I grabbed a guy from our ISP and did a few tests.

  1. We put a laptop on the network in the remote office and downloaded a 300 mb file using ftp. It took about 19 seconds.
  2. We then connected to the same server via smb and tried downloading the same file. The time indicator on the workstation suggested that total download time would be about 30 minutes. Very slow. Same issue with an AFP connection.

I tried another switch at the remote location with the same results. I also have another building set up in an identical fashion. If I try the same file downloads from the same server the files download in seconds.

So I am now capturing packets on that VLAN trying to look for any clues that might indicate a problem from that location. With respect to SMB, what sorts of things would you folks look for that might indicate a bottleneck of some sort?

asked 23 Mar '11, 09:11

tandrews's gravatar image

tandrews
1112
accept rate: 0%


You definitely want to compare the round trip time between the sites.

Imagine that you want to order one extra-large pizza for you and your friends.

FTP is like "bring me one extra-large pizza with my favorite topics". The pizza man will deliver one box that keeps everyone happy. If the shop is in another town you wait a bit longer, but once the delivery is made you dig in and everybody is munching away.

SMB is like ordering one slice at a time. A new order can only be placed, after the last slice has been delivered. Then you pick up the phone, place a new order, wait until the second slice is delivered, then order the next slice and so on.

For the FTP approach you wait only once for the delivery. In the SMB approach the pizza boy has to make several trips to your home. If the pizza shop is far away the driving time will be longer than the time required to prepare the dish. It this large number of round trips that has a big impact on the transfer times.

You might want to visualize the network load from a single transfer by looking at the IO Graph with Statistics -> IO Graphs.

permanent link

answered 23 Mar '11, 16:06

packethunter's gravatar image

packethunter
2.1k71548
accept rate: 8%

Understood. But as I indicated above, using the same laptop downloading the same file from the same server on a different (but identically configured) network, I get a significantly faster download time... seconds as opposed to minutes.

I believe that I can safely eliminate client and/or server configuration issues as the cause given the above.

I really believe that there is some sort of bottleneck between my remote switch and my local switch.

(23 Mar '11, 12:45) tandrews

What do your ping times look like? From the "good" and "bad" site? Are you actually seeing lower throughput, or are there delays in transmission?

(23 Mar '11, 12:56) GeonJay

Check for a duplex mismatch. I'm assuming your latency between the two test beds are similar enough (not 5 ms vs 200ms for example)

(23 Mar '11, 16:27) hansangb
2

Next time I order pizza, I make sure they FTP it to me instead of SMBing it ;-)

(23 Mar '11, 16:37) SYN-bit ♦♦

Hey PH, I'm going to use this analogy from now on, it's a good one!

(24 Mar '11, 05:44) GeonJay

We actually went to our ISP's switch in the CO that feeds our remote office. The problem exists there as well. The switch in question is a fast ethernet switch, the others are tied to Gig switches. So we may be on to something. Thanks for the suggestions.

(24 Mar '11, 06:10) tandrews

I would appreciate if you share the solution with us.

(24 Mar '11, 07:21) packethunter

Gladly. We had to cut our investigation short, but will likely continue it in a week or so. I will update here.

(24 Mar '11, 11:27) tandrews

I had the same issue, but in the same machine, one day in the afternoon I copyied a 2 MB file in 4 seconds without retransmissions, but the next day in the morning it took like 2 minutes and the number of packet utilized in retransmissions mechanisms where like the 30% of the total numbers of the capture.

Regards!

(18 Jan '15, 09:19) ogoname
showing 5 of 9 show 4 more comments

Just a few things to do with Wireshark when analyzing SMB:

  • What is the round trip time between your sites? As SMB uses more TCP round trips the RTT has a much bigger impact on SMB than on FTP.

  • What operating system are you using for client and server? Microsoft introduced a feature called "SMB pipelining" with Windows Vista. When copying large files (like your 300 MB files) the client sends up to 16 parallel requests to the server. A fast server will dump a large number of frames on the network. This can cause quite a traffic jam (or packet drops) at the choke point (or choke points) in your path.

  • If you are using Windows Vista or a newer operating system: Ever since Vista the client advertises at least a WinSize of 64kB - which is often too large. Try to rerun the test after disabling autotuning with netsh interface tcp set global autotuninglevel disabled

  • Check the server response time with Statistics -> Service Response Times -> SMB

  • Check the server load (i. e. verifying the number of concurrent requests) by plotting an advanced IO graph showing LOAD(*) for smb.time

  • Are you hampered by the maximum number of concurrent outstanding commands that a client can send to the server? (This can happen when working with terminal servers)

Something more exotic:

  • Are you using SMB2 (requires Win 7 / Vista / Server 2008)? If yes, check if the server grants as many credits as the client has requested. If the server is not granting credits this could limit the SMB2 throughput.

External references:

Article from the Microsofts Cable Guy on Auto Tuning

Article from Citrix on the maximum number of SMB commands

Video recording of a presentation on SMB 2.1 from Microsoft

Hope that helps a little. Good hunting!

permanent link

answered 23 Mar '11, 09:40

packethunter's gravatar image

packethunter
2.1k71548
accept rate: 8%

Thanks Packethunter... Just to clarify a few things:

  • I am running these tests on a mix of Windows and Mac clients.
  • The server I used for the test is a Windows 2003 server, but it doesn't seem to matter what server I use. We see issues with other sorts of file transfers.
  • I ran this test using the same laptop pulling the file from the same server from two different locations. In the problem location, the large file takes several minutes to download. In the other location (identical network configuration) the download takes seconds. In my mind this would eliminate client and server as part of the problem.
(23 Mar '11, 09:49) tandrews

I'll second packethunter's suggestion of looking at the end-to-end latency. SMB is incredibly latency sensitive. This is one of the most common bottlenecks we face in our corporation. Out of the box SMB on Windows <=2003 experiences performance degradations as latency increases. The way the protocol will work, out of the box, will be to send just two (2) payload bearing packets leave the server, it will then wait until an ACK packet is received from the host before sending the next two packets. This is regardless of the negotiated MSS or RecvWindow size...this is a layer 4/5 bottleneck. You won't notice the problem in a low latency LAN. FTP is a much more efficient protocol for file transfer, but it offers few of the whistles/bells that SMB offers. SMBv2 (specifically that which is included in Windoze 2008R2) is much more efficient. There are several registry tweaks that you can put on the client and server to increase the SMBv1 throughput.

Server 2003 Tuning doc from MS. There are lots of good hints in here, be sure to check out the TCPAckFreq key. http://download.microsoft.com/download/2/8/0/2800a518-7ac6-4aac-bd85-74d2c52e1ec6/tuning.doc

permanent link

answered 23 Mar '11, 12:41

GeonJay's gravatar image

GeonJay
4705922
accept rate: 5%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×55
×52

question asked: 23 Mar '11, 09:11

question was seen: 15,981 times

last updated: 18 Jan '15, 09:29

p​o​w​e​r​e​d by O​S​Q​A