This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

We are investigating an issue which where we have a client application (from Azure) accessing our webservers. The application is experiencing intermitted time-outs when connecting to our server. When we view our logs we do not see those attempts that time-out. We only see succesfull attempts. We have not been able to figure out where these requests fail. Whether it's our Firewall or Load Balancers that is dropping the traffic for whatever reason. I came accross this case: http://serverfault.com/questions/732008/outbound-packets-dropping-timeouts-only-with-azure and also https://blogs.msdn.microsoft.com/mast/2015/07/13/azure-snat/ This looks exaclty like our issue but I need to prove to Azure that this is the case.From the microsoft blog in the example on server side, this is said: The server has already an ESTABLISHED connection, so when it receives a SYN packet from CLIENT VIP:54321 it will ignore and drop it. At this point in time, we’ve ended up with two broken connections: The original that has been idle for 4+ minutes and the new one.
I want to see this happening. What filter would I use?

asked 09 Mar '17, 05:15

Michellangelo's gravatar image

Michellangelo
6113
accept rate: 0%

edited 09 Mar '17, 05:16


The SYN that gets dropped will probably be flagged as an out-of-order packet by wireshark. If you use the filter tcp.flags.syn == 1 to display only SYN packets and SYN/ACK packets, you may be able to find it by scrolling through and looking for any packets flagged as out-of-order (or searching with tcp.analysis.out_of_order) or where there is a SYN without a SYN/ACK response.

It might also get flagged in wireshark as a reused port. You could try doing a search using the filter tcp.analysis.reused_ports.

Of course, if you are behind a firewall, then these SYN packets might not even be making it to your server so you may not see them at all, but might want to check your FW logs.

permanent link

answered 09 Mar '17, 14:24

ryber's gravatar image

ryber
146459
accept rate: 16%

Thanks Ryber.

The difficulty is to find the right place to capture this traffic. To make it more clear, this is the setup (internet)-[router]-[firewall]-[load balancer]-[webserver] Brief explanation. The router only routes the traffic to our firewall. The firewall NATs the traffic to a private ip. The load-balancer decrypts the traffic and forwards it to any available webserver. The time-out requests from the client never reach our load-balancer. So where is the traffic dropped if it even reaches our network. Right now my capture is on the outside interface of our Firewall. Here I see many retransmission and ports reused. I do not see any out-of-order packets. In my firewall logs I do not see any drops at all. Not sure where to look.

(10 Mar '17, 04:16) Michellangelo

The "port reused" are probably the SYN packets form the duplicate 4-Tuple connections. If you are not seeing anything on your FW to indicate they are being dropped (or your load balancer for that matter), your best bet is to take a packet capture on the internal interface of your firewall and/or the external interface of your load balancer. You can then identify the packets that get flagged "port reused" and see if they are present after the firewall. If you see them between the FW and load balancer, the next step would be to take a capture on the link between the load balancer and server(s).

(10 Mar '17, 08:14) ryber

In the Azure Blog about SNAT it states that Azure is using a shared VIP for outgoing connections, reserving 160 ports per VM. As they also say they will keep a connection in TIME_WAIT state for a couple of minutes. Then it says:

The first question you might have already may be What happens if I simultaneously use the 160 ports? Well, if none of them has been freed yet the system will assign more ports on a best-effort basis, but as soon as one becomes free it will available again for use.

So, if you open up enough connections, you can deplete your 160 port block and are then confronted with the best effort service for outgoing connections. At the end of the article it states:

Before we go, we should probably also acknowledge another long-standing problem with this design. Since Azure allocates these outgoing ports in batches of 160, it is possible that the creation of a new batch of 160 may not happen fast enough and an outgoing connection attempt will fail. We typically only see this under very high load (almost always load testing), but if you fall victim to this, the solution is the same – use a PIP.

So basically you won't be able to detect these network timeouts in your network, as they are already dropped within the Azure network. The only way to make sure you don't run into these timeouts seems to be to configure a PIP for each of the Azure VM's

permanent link

answered 12 Mar '17, 15:49

SYN-bit's gravatar image

SYN-bit ♦♦
17.1k957245
accept rate: 20%

edited 12 Mar '17, 15:50

You are correct I wont be able to see when the traffic gets dropped from the remote location (Azure) On the other hand I should be able to see unexpected SYN packets being dropped. These will be dropped when the Server has an ESTABLISHED connection.

The server has already an ESTABLISHED connection, so when it receives a SYN packet from CLIENT

The question is how I will filter these in a capture.

(15 Mar '17, 06:17) Michellangelo

Whenever Wireshark sees a TCP/SYN for an IP/PORT combination that it has already seen, it will mark it as "[TCP ports reused]" (filter: tcp.analysis.reused_ports).

Wireshark does not keep state so it can not (yet) make a distinction between the SYN following a FIN in the previous session and a SYN in the middle of an active session.

You could use some tshark scripting or a LUA script within WIreshark to filter on SYN's that follow a non-closed session on the same IP/PORT combination.

(15 Mar '17, 13:53) SYN-bit ♦♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×248
×55
×23
×5

question asked: 09 Mar '17, 05:15

question was seen: 1,605 times

last updated: 15 Mar '17, 13:53

p​o​w​e​r​e​d by O​S​Q​A