What does this message mean? Acknowledgment number: Broken TCP. The acknowledge field is nonzero while the ACK flag is not set I see the client attempt to send this after the TCP connection has been torn down by FIN/ACK, ACK+FIN/ACK, ACK. Our firewall blocks the RST packet sent (with the above "broken tcp" message) because at that point the session has already been removed from the firewall because of the four-way FIN teardown. asked 28 Mar '11, 13:55  networkguy09 |
3 Answers:
Basically the message says that whenever a TCP packet does not have the ACK flag set, the acknowledge field must be zero and this packet does not comply to that rule. The reason that you see this message is that the other side has closed it's session (on the Firewall, Loadbalancer or end system) before the last packet comes in, resulting in the RST packet. However, the generation of the RST packet is not valid as the ACK flag is not set and the acknowledge field is non-zero. answered 28 Mar '11, 15:55  SYN-bit ♦♦ showing 5 of 7 show 2 more comments |
Which side of the firewall did you trace at? Between the client and the firewall or between the firewall and the server? If the server is sending the FIN and you traced between the server and the firewall then it is the server that is causing the client to be kicked out. Everything after the FIN is almost moot, right? answered 30 Mar '11, 07:56  motdc We did two traces one directly on the server, the other on the firewall looking before it passes through (so between the client and firewall). The firewall capture you see the RST, but on the server side you do not because after the FIN four way teardown the firewall has removed that session from its table and the RST is dropped at the firewall. The server always sends the first FIN and this is when the users report getting "kicked out" We still don't understand how taking the firewall out of the mix seems to resolve the issue. (30 Mar '11, 09:42) networkguy09 Got it. Is there any other traffic or other connctions that are out of band so to speeak? So say you are tracing just one port only and something on a seperate connection is causing the problem? (30 Mar '11, 10:05) motdc Not sure what your asking. The server for this application only runs this one application, but the client has multiple applications which access other servers via the same firewall, wireless, and wired path ways without issue. Only this one application the users report getting kicked out of. (30 Mar '11, 11:57) networkguy09 |
I ran into this problem with an ASA firewall, windows server 2000, and a VPN connection which wouldn't establish connection. When running traces I got the same error. I uninstalled the VPN software and re-installed it, including re-importing the certificate, and the problem resolved. answered 19 May '11, 05:25  John_Modlin |
What do you mean by "resulting in the RST packet?" I don't understand why the client would send a RST packet after it just finished the session by FIN/ACK, ACK+FIN/ACK, ACK (first FIN/ACK being sent by the server in every case). I expect to not see the RST packet at all and if I understand correctly, the RST packet is a non factor as the session is already terminated.
My problem is the clients report getting "kicked out" of a particular application (which runs over telnet). In every case of the kick out, I see the FIN/ACK, ACK+FIN/ACK, ACK, then the client sends a RST. In troubleshooting we bypassed our ASA firewall for the network in question and found the clients no longer have the issue. I don't understand how the firewall could be causing the issue when we see in the captures the server sends a FIN/ACK to teardown the session. Any suggestions of what to look for in the capture/firewall? Thanks!
(converted your 'answer' to a 'comment' as that's the way this Q&A site works best, see the FAQ for details)
My answer was geared to the original question of what the "Acknowledgment number: Broken TCP. The acknowledge field is nonzero while the ACK flag is not set" message meant.
As far as your "users being kicked off" problem, firewalls have session timeouts so that their session tables don't fill up on sessions that somehow did not terminate. The side effect is that long lived sessions like "Telnet", "SSH", "RDP" and the like will be terminated by the firewall if they are idle for a longer time than the configured timeout on the firewall for that particular service/port/application. This is especially a problem if a custom port is being used.
Check your traces for idle time before the session termination, also check the idle time for your protocol on the firewall.
There is virtually no idle time in the captures for this. They both are sending telnet data and ACK each, then the server sends data along with FIN/PSH/ACK and the session is torn down as mentioned. As for the timeout on the firewall, it is set to 1 hour for established connections. The user reports getting kicked out at random times (could be 2 minutes, 5 minutes, 10 minutes, 2 hours, 4 hours, etc.) The server admin has configured tcp keepalives to send to the clients every 5 minutes and I always see the keepalives sent and ACK'd from captures taken on the server NIC.
Sorry for messing up the earlier post and for asking more than one question in the thread. I suppose I could have mentioned the underlying issue originally. Thanks for all the help because I am stumped!
Hmmm... if the server behaves badly when traffic goes through the firewall, but it behaves properly when the firewall is not in the path, I would make traces on both the client and server in both situations. You then need to compare them meticulously...
If you'd need help with that, feel free to contact me (see my profle for contact details)
Idle timeout is different from the absolute connection timeout...but it doesn't sound like you're hitting either timeout. The ASA has a pretty good builtin debugging system and I'm 99% sure that the log can reveal the firewall's reasoning for terminating the connection. If you have a specific rule/conduit/ACL/etc for allowing the telnet traffic you can enable the log feature for that rule also. Also, don't forget about good ol' Cisco bugs - those can be VERY difficult to troubleshoot.