This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

SMB2 Long Deltatime - Applications Crash

0

I am a newbee using Wireshark, so hope for a little help :-) I have an issue with an SQL DB application which craches. I my tracefile I see very high Delta Time on SMB2 and a max SRT(s) of 625.347497 on Notify on SMB2 Service Response Time.

Where should I start to look? Is it the network, server or Client?

===========================================================================
SMB2 Service Response Time Statistics - sql-db-issue_00001_20170103094009_00006_20170103145622_00001_20170104090747:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s)  Sum SRT (s)

SMB2


Close                       6    361     0.000291     0.146918     0.001722     0.621558
Create                      5    371     0.000331     1.787312     0.009587     3.556769
Find                       14    271     0.000600     0.194669     0.005546     1.502843
Flush                       7      1     0.165415     0.165415     0.165415     0.165415
GetInfo                    16     58     0.000330     0.009854     0.002310     0.133973
Ioctl                      11     51     0.000420     0.182926     0.010477     0.534342
Negotiate Protocol          0      3     0.000743     0.000900     0.000814     0.002442
Notify                     15     18     0.000661   625.347497    71.007926  1278.142673
Read                        8    601     0.000328     0.302919     0.028557    17.163042
Session Logoff              2      3     0.000797     0.004081     0.001950     0.005849
Session Setup               1      4     0.000784     0.007014     0.002552     0.010209
SetInfo                    17      9     0.000598     0.000858     0.000745     0.006704
Tree Connect                3      5     0.000346     0.001530     0.000663     0.003313
Tree Disconnect             4      5     0.000816     0.009827     0.004345     0.021723
Write                       9     13     0.000331     0.008499     0.002117     0.027518


13686   4.777025    10.34.80.11 10.34.80.165    SMB2    162 29.962738000        256 Notify Response
Thanks in advance.

asked 04 Jan ‘17, 23:55

bachsec's gravatar image

bachsec
6●1●1●2
accept rate: 0%

edited 07 Jan ‘17, 11:04

grahamb's gravatar image

grahamb ♦
19.8k●3●30●206

Could you explain the topology of the system a little? Is it:

Client PC —- Database Server —- SMB File Server

or

Client PC / Access —- SMB File Server

or

????

(07 Jan ‘17, 10:49) PaulOfford

Hi PaulOfford

It is Client(with developed frontend) —-> SMB Server with SQL Express as DB for the frontend Application

The application connects to the SQL express instance in the SMB server.

(08 Jan ‘17, 00:11) bachsec
1

There are two ways to connect to a Microsoft SQL database; named pipes or TDS over TCP. You could be using named pipes but Find and Notify are SMB2 commands that I wouldn’t expect to see.

By the way, Notify response times might be long as Notify is used to notify a client if a change has been made to a file or directory. See https://wiki.wireshark.org/SMB2/Notify

Can you confirm that the client application connects to the database using named pipes? If you are not sure, check to see if there is any traffic in your trace between the client and port 1433 on your server with the filter expression:

ip.addr==the_client_ip_address and tcp.port=1433

TCP port 1433 is the default port number for Microsoft SQL Server. Although it can be set to a different value and so you might want to check with whoever supports the database.

Sorry to keep asking dumb questions but I don’t want to send you off on a wild goose chase.

(08 Jan ‘17, 01:57) PaulOfford

Hi PaulOfford

In the tracefile I see at lot of traffic between the client and the serve on port 1433. 80% of the packages in the trace is that, when I filter with the filet you wrote.

(08 Jan ‘17, 12:04) bachsec

Does this graph say anything.

(08 Jan ‘17, 12:21) bachsec

no. sorry.

(08 Jan ‘17, 13:16) packethunter

It has answered the question I had. So, although your client application may use SMB file services, the connection to the database is via TDS/TCP.

You say in the title that the application crashes. Do you literally mean that? Does it throw an exception, dump and then exit? Or do you get an error message? If so, what’s the message and have you determined what it means?

(08 Jan ‘17, 14:29) PaulOfford
showing 5 of 7 show 2 more comments

One Answer:

1

Hello bachsec, and welcome to ask.wireshark

Application errors like missing access rights or lock conflicts are easily spotted with this display filter: smb2.flags.response == 1 and not smb2.nt_status == 0

Looking at the Service Response Time Statistics can be helpful. Alas, in this case we only get marginal info.

The most remarkable part in the statistics from your trace file is one instance of a long time for the Create command. The Create function is called to obtain a file handle. This can be used to either create a file or to open an existing file. 1.7 seconds is considered "long" for most cases. However, a a virus scanner checking a 4 GB ISO image in 1.7 seconds would be impressive. So we have a number, but no unit to measure against. The delay could also be caused a TCP retransmissions or other network issues. The long response times for the notify request are usually not a problem, as this an asynchronous message from the server like "Hey client, a time stamp on a file has changed".

A trace file would help a lot. Depending on the configuration (DFS, Branch Cache etc.) a full network trace from the client view would be the best option. Certain problems might require two traces taken simultaneously with client and server.

Please bear in mind, that SMB traffic usually reveals the file contents and (depending on the configuration of workstation and server) could also reveal users passwords. To investigate that problem we need at least the SMB or SMB2 headers with the return codes. I am not aware of a tool that sanitizes SMB traffic as one might desire.

If you can not share the trace file with the general public a network consultant might be your best option.

Good hunting

answered 05 Jan '17, 10:32

packethunter's gravatar image

packethunter
2.1k●7●15●48
accept rate: 8%

Hi Packethunter

With the filter you gave me, I can see something is that errors E.G. Error: STATUS_BUFFER_TOO_SMALL

(08 Jan '17, 00:14) bachsec

Hmmmm. "Buffer too small" is a regular message for certain protocols, that use named pipes. The client sends a request and a default buffer size, say 1kByte. If the server's response exceeds 1 kB the response will have the return code "Buffer too small". Another field will hold the required buffer size for the response.

Here are a few root causes for performance problems, that I have encountered during various analysis projects:

  • Missing index or too many indices (like, 300+ fields for a single row)
  • Disk errors or other hardware problems -> Check the System event log
  • A virus scanner interfering with the application -> Try to put the DB file on an exclusion list. Remove the DB-file(s) from the list if the virus scanner is not involved
  • Bad application design no 1: Too many greedy locks (like exclusive write lock, if a shared read would do)
  • Bad application design no 2: Keep on reading the same data within milliseconds (in the magnitude of 5'000 unnecessary reads per transaction)
  • Bad application design no 3: Using a file share, when a database would be preferable
  • Bad system design no 1: Use RAID-5 storage for files or databases that are written frequently.
  • Bad system design no 2: Too little memory, CPU too weak, (time-) critical data on a USB-2 attached device, other hardware bottleneck
  • Poor system maintenance: Backup application has consumed 12 days of CPU time (guess, when the problem started; then guess who got blamed)
  • Wrong video adapter for the client (on-board Graphics adapter work no so well for CAD applications)
  • Poor network design (round trip times exceeding 10 msec when 100 microseconds are easy to achieve without investment)
  • Half-duplex / full-duplex mismatch (manifests itself in serious retransmissions, easy to spot with wireshark)

After a day at the Sharkfest conference you will certainly have a list that is five times this long. The majority of these issues is best found with diagnostic tools on the server, like the Performance Monitor. Wireshark will help you in many situations to identify the culprit (server, client, network ...). Getting to the root cause requires other tools, once you have arrived at the server.

I am not sure, if I can help you further without a trace file.

Good hunting

(08 Jan '17, 09:16) packethunter