We are investigating an issue which where we have a client application (from Azure) accessing our webservers. The application is experiencing intermitted time-outs when connecting to our server. When we view our logs we do not see those attempts that time-out. We only see succesfull attempts. We have not been able to figure out where these requests fail. Whether it's our Firewall or Load Balancers that is dropping the traffic for whatever reason. I came accross this case: http://serverfault.com/questions/732008/outbound-packets-dropping-timeouts-only-with-azure and also https://blogs.msdn.microsoft.com/mast/2015/07/13/azure-snat/ This looks exaclty like our issue but I need to prove to Azure that this is the case.From the microsoft blog in the example on server side, this is said: The server has already an ESTABLISHED connection, so when it receives a SYN packet from CLIENT VIP:54321 it will ignore and drop it. At this point in time, we’ve ended up with two broken connections: The original that has been idle for 4+ minutes and the new one. asked 09 Mar '17, 05:15  Michellangelo edited 09 Mar '17, 05:16 |
2 Answers:
The SYN that gets dropped will probably be flagged as an out-of-order packet by wireshark. If you use the filter tcp.flags.syn == 1 to display only SYN packets and SYN/ACK packets, you may be able to find it by scrolling through and looking for any packets flagged as out-of-order (or searching with tcp.analysis.out_of_order) or where there is a SYN without a SYN/ACK response. It might also get flagged in wireshark as a reused port. You could try doing a search using the filter tcp.analysis.reused_ports. Of course, if you are behind a firewall, then these SYN packets might not even be making it to your server so you may not see them at all, but might want to check your FW logs. answered 09 Mar '17, 14:24  ryber |
In the Azure Blog about SNAT it states that Azure is using a shared VIP for outgoing connections, reserving 160 ports per VM. As they also say they will keep a connection in TIME_WAIT state for a couple of minutes. Then it says:
So, if you open up enough connections, you can deplete your 160 port block and are then confronted with the best effort service for outgoing connections. At the end of the article it states:
So basically you won't be able to detect these network timeouts in your network, as they are already dropped within the Azure network. The only way to make sure you don't run into these timeouts seems to be to configure a PIP for each of the Azure VM's answered 12 Mar '17, 15:49  SYN-bit ♦♦ edited 12 Mar '17, 15:50 You are correct I wont be able to see when the traffic gets dropped from the remote location (Azure) On the other hand I should be able to see unexpected SYN packets being dropped. These will be dropped when the Server has an ESTABLISHED connection.
The question is how I will filter these in a capture. (15 Mar '17, 06:17) Michellangelo Whenever Wireshark sees a TCP/SYN for an IP/PORT combination that it has already seen, it will mark it as "[TCP ports reused]" (filter: Wireshark does not keep state so it can not (yet) make a distinction between the SYN following a FIN in the previous session and a SYN in the middle of an active session. You could use some tshark scripting or a LUA script within WIreshark to filter on SYN's that follow a non-closed session on the same IP/PORT combination. (15 Mar '17, 13:53) SYN-bit ♦♦ |
Thanks Ryber.
The difficulty is to find the right place to capture this traffic. To make it more clear, this is the setup (internet)-[router]-[firewall]-[load balancer]-[webserver] Brief explanation. The router only routes the traffic to our firewall. The firewall NATs the traffic to a private ip. The load-balancer decrypts the traffic and forwards it to any available webserver. The time-out requests from the client never reach our load-balancer. So where is the traffic dropped if it even reaches our network. Right now my capture is on the outside interface of our Firewall. Here I see many retransmission and ports reused. I do not see any out-of-order packets. In my firewall logs I do not see any drops at all. Not sure where to look.
Could you share us a trace? https://blog.packet-foo.com/2016/11/the-wireshark-qa-trace-file-sharing-tutorial/
The "port reused" are probably the SYN packets form the duplicate 4-Tuple connections. If you are not seeing anything on your FW to indicate they are being dropped (or your load balancer for that matter), your best bet is to take a packet capture on the internal interface of your firewall and/or the external interface of your load balancer. You can then identify the packets that get flagged "port reused" and see if they are present after the firewall. If you see them between the FW and load balancer, the next step would be to take a capture on the link between the load balancer and server(s).