have an app timing out intermittently hitting a database. client is win2k8 server is Oracle Enterprise Linux. No firewall inbetween. To try and troubleshoot, I got a version of the app to run every second. Leaving it for a day I might get 2-3 timeouts. not much, but affects us real bad when it happens in a prod environment. PCAP running the client, tcpdump on the sever. this is what they look like, client sends the START, server gets it, tries to send ACK back but client doesnt get it. You can see the retransmission attempts. the app times out at this point, and starts running fine again. 10gb NICs either side. No server load. No client load. No error messages on either side. What could cause this? CLIENT SERVER asked 10 Sep '14, 04:46  carsmusings |
One Answer:
If there is no firewall in the network between client and server something else must prevent the SYN/ACK to reach the client. Is there any host based firewall running on the client, and if so, can you try to turn it off for a test? Also, some anti virus solutions sometimes mess with the network stack, so maybe disabled that for a test could be an option, too. Both actions (turning off any security software like the firewall or anti virus) is of course a risk. If you can't live with that you might want to try to capture the client traffic right before it gets to the client. E.g configure a monitor port on the switch the client is connected to and capture the client traffic with an additional PC. That way you can determine if the SYN/ACK can be seen on the way to the client, because if you see it there and not on the client itself you know it gets lost in the client itself. answered 10 Sep '14, 05:09  Jasper ♦♦ |
Thinking along the same lines, I do have a firewall for some other apps, I generated a timeout event from a client the other side of it. The client PCAP shows same thing, server shows same thing, but the firewall does not see the return event so this says its either getting lost at the server NIC or switch.
iptables off on the server.
1 very peculiar thing, as Im sure we all know all that TCP SYN/ACK stuff happens in milliseconds. During the troubleshooting when testing with the firewall, if I search the firewall for that port in the time out event, example 42966 in the above dumps, I could see the ACCEPT of the port coming back something like 2 minutes later. The client PCAP doesnt show this but Im guessing at that point the TCP stream is dead from its side.
Delayed ACKs maybe but from reading, that can only happen up to under a second, but 2 minutes?
If the client PCAPs are captured on the client you may not see what's really happening. Local captures are often misleading and do not show what really happened on the network. This is why I recommend to capture on the network, not the nodes involved.
"Delayed ACKs" are not relevant to session setups, they happen when the data flow is already established and an odd number of segments are transmitted.