Updated 2015/10/2 - Please see below screen capture showing the high ACK numbers..
Hi Guys, I have been investigating an issue for a month now and still can't figure out what the issue is. Basically we have clients connecting to a server on port 80 but 1% of the connections fail due to a socket error. The packet traverses as below Client - firewall - ipsec tunnel - firewall - loadbalancer - server When we do a TCP dump between the firewall and loadbalancer we can see the successfull connections which follows the normal 3 way handshake as expected but when we look at the failed transactions we are seeing the follwing SYN SEQ(lets say 1000000) [TCP ACKed unseen segment]SYN,ACK SEQ(totally random number 124124) ACK(totally random number + 1 - 124125) [TCP Spurious Retransmission] SYN SEQ(1000000) [TCP ACKed unseen segment]SYN,ACK SEQ(t124124) ACK(124125) [TCP Spurious Retransmission] SYN SEQ(1000000)
The issue is totally random during the day and not a load or busy period issue. From what I can see and read on the internet the LB might be involved with this 3 way handshake even though one would expect this to be between the server and the client. I hope I can get some feedback. asked 03 Dec '15, 05:45  Fronie edited 10 Dec '15, 06:00 showing 5 of 22 show 17 more comments |
This is a static archive of our old Q&A Site.
Please post any new questions and answers at ask.wireshark.org.
And so on until the packet is dropped.. So it seems either the LB or the server is sending the WRONG SYN,ACK SEQ number?? I checked previous TCP streams and also filtered on this wrong SEQ number and can't find it anywhere in my capture so its not something that was stuck in a previous segment or so. I can also confirm I have no packets missing from my wireshark.
Please publish the capture file (anonymized using TraceWrangler if necessary so that the tcp payload and the real IP addresses are not disclosed) somewhere and put a link to it here. Without that, a serious analysis is near to impossible.
And what does the loadbalancer logs have to say??
We can't do a tcp level capture on the LB due to a risk of it falling over. I know if we could have those logs we could determine if it is the LB or the server that is sending the wrong ACK. My question is just why would something reply with the wrong ACK ?
@Jaap was asking about LB logs, not captures. It means files to which the LB application is reporting anomalies in its own run. Usually, you can change the level of detail written to these files on the fly.
As for analysis of this suspicion
you should be able to capture between the server and the LB and between the LB and the client simultaneously. Comparing the traces should tell you two things:
whether the LB terminates a tcp session and opens a new one towards the server (which means that the sequence numbers between the client and the LB are completely independent from sequence numbers between the LB and the server) or whether it mimics L3 transparency in some way
if the LB mimics transparency, whether it fails to do so perfectly and so the suspicion is correct.
We would like to do such a capture but the servers behind the LB are actaully VM's that sits on a different switch which we can't SPAN at this stage. so if the LB terminates the 3 way handshake with the client we should clearly see this in the capture that we are running at the moment ?
From our LB logs we can't see any anomalies.
Just some more info for you we actually had 8 times more failures a few weeks ago but all traffic that reached the LB was NAT'ted we changed this design and everything reaching the LB is now the real IP. So I have a feeling there was a problem with a device which couldn't handle all the translations. Now after we removed the NAT we still have failures but less
Use TraceWrangler to scramble IP addresses and strip tcp payload, don't touch the tcp port numbers, publish the capture somewhere and put a link to it here, and describe the meaning of the important IPs after scrambling.
If there is nothing in LB logs I would think that it is a tcp level issue so the LB application does not know about it (which implies it does not mimic transparency, btw which LB you use?).
Fortibalancer 1000. I am still new to wireshark so I will look into this TraceWrangler. Is it not safe to upload a screen capture even if the IP's are private ?
I still think we should see more info if we could do a tcp debug on the LB but the business don't want to take that risk at this stage.
Uploading a screen capture may be safe but it is definitely useless as it is impossible to use the features of Wireshark for analysis. I forgot to write that the capture should not contain only the failed transactions but also several (at least 15) minutes of capture before them.
At this site, a "capture" is automatically understood as "the .pcap or .pcapng file which can be open in wireshark". To prevent ambiguity, it is better to call pictures of screen contents "screenshots".
I do understand but unfortunately I am not allowed to upload this capture even if I can filter out the payload or sensitive info. I guess there is no way you can direct me to look for something specific without posting the capture online ? I can tell you that the client is connecting to the LB IP which then gets load balanced to the server's ip. So it might be that the 3 way handshake happens on the LB and it keeps track of the sessions between the client and server. Another thing to note is that all of the transactions that fail are failing exactly after 20 seconds. But there is no timer set to that on any of the equipment the packet traverses. on the firewall closest to the LB we can see those failed transactions in the logs which also always close on 69 seconds which is very weird as the actual transaction fails after 20. All the failed transactions in the wireshark always last exactly 9 seconds. So it might be that there is a 60 second timer on the LB or server that close the connection after 60 plus the 9 seconds we see in the wireshark which equals 69 seconds we see on the firewall. I really hope you can assist without needing the capture
Anyway if you guys can have a look at the screenshot I added, This will explain the issue I am having.
No, it does not explain it, it illustrates it, that's a difference.
E.g. I can only guess that the two cases shown do not use the same socket tuple (because the port numbers are missing in the screenshot).
I normally don't believe in homoeopathy and remote-control over noisy channels, but I'd recommend you to:
take the 15 minutes capture,
identify the anomaly you are trying to understand (weird response to a SYN packet) in it, and find the client IP address and tcp port,
set the display filter to
(ip.addr == 'client_ip') and (ip.addr == 'server_ip') and (tcp.port == 'client_port')look at the packets from the previous session using the same socket pair, and/or post a screenshot of the last 10 packets of the previous session and 5 packets of the new one, starting with the one with SYN.
I would guess you shall find there FIN packets sent by the server first, and the SYN for the new session coming earlier that in 120 seconds from the end of the previous session.
But I can't hold myself from saying that being allowed to post a screenshot with visible IP addresses but not being able to post a capture with all IP addresses aliased and all payload removed sounds ridiculous to me.
The screenshot does have port numbers in it (1101 on the failed transaction and 1256 on the working capture.) What is the difference between doing your display filter and actually following the tcp stream ? When I do the filter as you requested I get the very same results as using the tcp.stream eq xxx filter. When you say " look at the packets from the previous session using the same socket pair, and/or post a screenshot of the last 10 packets of the previous session and 5 packets of the new one, starting with the one with SYN " Do you want me to upload a screenshot of a successful transaction tcp stream with all the packets ? Please note we have 15 000 unique clients connecting from XP machines. Which uses source port 1025-5000 for connecting. If you say there might be session that is still open due to the TIME-WAIT on the server that would mean that the next client should connect on the same source ip and port for the server to drop or not accept the connection. But in our case all sources are coming in with unique IP's. Only duplicates there might be is source ports. But I mean each connection falls in its own socket tuple
Also the FIN on successfull transactions ALWAYS comes from the client side and then the server ACK it.
OK. Yes, my idea really was that, although you've mentioned NATs to be removed, that the clients might have a small pool of ports to cycle for new tcp sessions, so run into the TIME_WAIT issue even without NAT en route.
As for display filter for a socket pair vs. tcp stream number: Wireshark will assign different tcp stream numbers to two tcp streams which use the same socket pair and the first one has been properly terminated and the next one properly started, and it will even mark the first SYN packet of the new session with "tcp socket reused". So if filtering by tcp stream number, you would not see the previous tcp session. On the other hand, if something went wrong, the Wireshark detector of sessions may fail and packets from two subsequent sessions using the same socket pair may be shown under same stream number (and a lot of warnings attached to packets of the second one I guess).
Lacking any experience with Fortibalancer, I cannot have any opinion about its behaviour, so at your place I would really seek a possibility to capture simultaneously between the balancer and the client and between the balancer and the servers. I haven't found in what you wrote so far whether the balancer uses the same LAN and ethernet (VLAN) interface to talk to the client side (via firewall) and the server side, but even if it uses two, I hope you are able to SPAN both simultaneously. The goal would be reduced - just to find out whether the servers or the LB are guilty.
As you say there are 15000 unique clients, it should be possible to pair the requests before and after pass through the LB. Doing so should tell you a lot about the internal behaviour of the LB, i.e. whether it mimics transparency or whether it internally terminates (in link sense) the tcp session from client and opens another one towards the server, using only client's source IP but own seq numbers and timestamps.
Hi Sindy,
I managed to capture on both sides at the same time. I can now clearly see how the 3 way handshake happens.
There are 2 different 3 way handshakes before data are transmitted. I looked at a successful transaction to see the traffic flow etc.
First 3 way handshake is between the client and the LB Next 3 way handshake is between the LB and the server - The LB changes the source port to something random and uses its own source IP and a new syn & ack sequence numbers.
Now this makes it difficult to try and see how the packet flows but I looked at the Payload to confirm I am looking at a successful transaction on both captures which it added up.
So I then started to look for failed transactions which I could quickly pick up on the capture between the client and the LB BUT remember here that the connection fails on the 3 way handshake so there is NO payload in the packet. We just see the wrong ACK number as before. I then went to the other capture and looked at the time stamps before and after that wrong ACK was send. There is nothing I can find that indicates that the LB also opened up a 3 way handshake with the server. So this confirms it is not the server that is sending the wrong ACK it is the LB. I also checked the time stamps and can confirm the wrong ACK was sent 0.000026 seconds after receiving the SYN. If I look at a successful transaction I can see that the 3 way handshake first complete between the client and the LB and then it opens a new handshake to the server, which tells me if it was the server that did something wrong we would see that in the packets between the origenal SYN and the 0.000026 seconds later ACK.
There is no way I can locate the 3 way handshake on the capture between the LB and server as the first handshake fails. Also like stated earlier there is no payload so I can't compare other detail.
I guess next step is to try and find out WHY the LB is sending this wrong ACK back...
Let me know if you have any ideas - thanks
The two bits of information from the past,
and
taken together imply the next questions:
do all of the clients only open a single tcp session at a time?
can you have a look what range of its own ports the LB uses to open sessions towards the servers? To do so, use a display filter
tcp.flags.syn == 1 && ip.src == whatever_the_FB1000_IP_is, then go to dissected packet, expand the "tcp" level, right-click on the "source tcp port" line, choose "Apply as Column". In the packet list you'll get a column named "Source Port", and when you click on its header, the packets will get sorted by the port numbers, so if you have enough of them, you should immediately see the range used.Because the fact that you've removed NATs from the way between the clients and the LB may have brought some advantages but the LB itself acts as a NAT (I've assumed that because it is the easiest method of operation but I wasn't sure, in general it can be done several other ways, even preserving the client's own public addresses so that the servers could see them).
So I could imagine that the LB finds out that it is unable to open a session towards the server due to lack of available ports, and tries to do something in the direction towards the client but fails to do it properly. But in such cases I would expect the failures to come in chunks.
From what you wrote I understand that you've used the payload to identify the last successful session pair before and first successful session pair after the failure, and there is nothing resembling a SYN packet between the two sessions at the server-facing side.
As I said I don't know anything about the inside of the FB1000, but if you can configure it with several IP addresses at the server-facing side and make it use them, can you check whether that would decrease the frequency of occurrence of the failure?
it seems I had my stats wrong - there are actually about 15 000 unique clients.
Yes all of them open a single tcp session. Thanks for showing me how to add columns! I can now easy pick up the source port range as well as adding another column on the other capture that includes the ACK numbers.
When sorting the ports I can see the LB uses ports 3000-65300. I have about an hour's logs which have 300 000 filtered packets. I can clearly see it is using the entire range.
I can now also quickly pick up the failed transactions just by sorting the ACK Column. All the high random ACK numbers are the failed transactions.(btw I have ticked the option " Relative Sequence Numbers".
We actually doubled the servers as a test but there was still the same amount of failures which made no sense.
To me the issue points maybe to a bug(which I can't find on the internet)
We are going to see if we can do a debug on the LB as last resort. Maybe we can pick up why it is sending the wrong ACK back.
Cheers
OK, doubling the number of servers is functionally equivalent to doubling the number of LB's server-side IP addresses. So as the ratio of failed sessions was the same regardless the number of servers, I confirm your conclusion that there is a bug in the LB.
I am afraid that at this stage, Wireshark has told you all it could. Just to satisfy my curiosity, can you paste here several absolute values of those crazy ACK numbers? You don't need to switch the display from relative mode to absolute mode to do that, just right-click the ACK number line in the packet dissection pane and choose
Copy -> ...as a Hex Stream, then paste it here.61cd666a , d7946f46 , 2d9944e0 , 4b5b955e , 4864e29b
When I sort the new column I added for ACK number you can see all the successful transactions numbers goes up to 6000 and then the failed transactions starts at 101388467. So one could easy see now the failed transactions. I will add a photo to my first post so you can see the numbers.
The absolute seq values start from an as-if-random value for each session, so in two independent sessions, the same relative number comes from two totally different absolute numbers. So I was wondering whether there wouldn't be some constant absolute value in all the broken ACKs, which would clearly confirm a bug. Unfortunately, it is not the case.
Ok well lets hope I can see something tonight when I do some debugging on the LB