This is our old Q&A Site. Please post any new questions and answers at

have an app timing out intermittently hitting a database. client is win2k8 server is Oracle Enterprise Linux. No firewall inbetween.

To try and troubleshoot, I got a version of the app to run every second. Leaving it for a day I might get 2-3 timeouts. not much, but affects us real bad when it happens in a prod environment. PCAP running the client, tcpdump on the sever. this is what they look like, client sends the START, server gets it, tries to send ACK back but client doesnt get it. You can see the retransmission attempts. the app times out at this point, and starts running fine again.

10gb NICs either side. No server load. No client load. No error messages on either side.
Ran it to different server. Same problem.
Ran from different client, same problem Wireshark 1.12.0

What could cause this?

CLIENT alt text

SERVER alt text

asked 10 Sep '14, 04:46

carsmusings's gravatar image

accept rate: 0%

If there is no firewall in the network between client and server something else must prevent the SYN/ACK to reach the client. Is there any host based firewall running on the client, and if so, can you try to turn it off for a test? Also, some anti virus solutions sometimes mess with the network stack, so maybe disabled that for a test could be an option, too.

Both actions (turning off any security software like the firewall or anti virus) is of course a risk. If you can't live with that you might want to try to capture the client traffic right before it gets to the client. E.g configure a monitor port on the switch the client is connected to and capture the client traffic with an additional PC. That way you can determine if the SYN/ACK can be seen on the way to the client, because if you see it there and not on the client itself you know it gets lost in the client itself.

permanent link

answered 10 Sep '14, 05:09

Jasper's gravatar image

Jasper ♦♦
accept rate: 18%

Thinking along the same lines, I do have a firewall for some other apps, I generated a timeout event from a client the other side of it. The client PCAP shows same thing, server shows same thing, but the firewall does not see the return event so this says its either getting lost at the server NIC or switch.

iptables off on the server.

1 very peculiar thing, as Im sure we all know all that TCP SYN/ACK stuff happens in milliseconds. During the troubleshooting when testing with the firewall, if I search the firewall for that port in the time out event, example 42966 in the above dumps, I could see the ACCEPT of the port coming back something like 2 minutes later. The client PCAP doesnt show this but Im guessing at that point the TCP stream is dead from its side.

Delayed ACKs maybe but from reading, that can only happen up to under a second, but 2 minutes?

(10 Sep '14, 05:51) carsmusings

If the client PCAPs are captured on the client you may not see what's really happening. Local captures are often misleading and do not show what really happened on the network. This is why I recommend to capture on the network, not the nodes involved.

"Delayed ACKs" are not relevant to session setups, they happen when the data flow is already established and an odd number of segments are transmitted.

(10 Sep '14, 05:58) Jasper ♦♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text]( "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 10 Sep '14, 04:46

question was seen: 4,580 times

last updated: 10 Sep '14, 05:58

p​o​w​e​r​e​d by O​S​Q​A