Losing connection with weird behavior.

Question

My game client loses connection from my game server intermittently. (about 0.2% per minute) When connection is dropped, my client received a error code 10053(connaborted) usually.

The number of user is about 100k, so about 200 users are dropped per minute.

I should find the cause, but can't find the root cause.

Now I have written code to work aroud this problem. When my client detects losing connection, my client tries to connect again.

Although this work around, I want to know the root cause of this problem.

Here is my game C/S's environment.

Client
- works on Windows (XP, Visita, 7)
- uses in-house network library (it's very simple)
Server
- works on Linux (Ubuntu 12.04.1 - kernel 3.2.0-29 / Ubuntu 12.04.2 - kernel 3.5.0-23)
- using iptables or ufw

Here is my approach to solve this problem.

First of all, I doubted my in-house code. I reviewed my codes very carefully but couldn't find any problems.

Second, I tried to capture packets on my server. Before loosing connection, my server can't receve any packets from a client. My server can't receive a ack packet, so my server try retransmission again and again.

Maybe a client also can't receive any packets from a server. (I can't reproduce this problem in my computer, so can't capture packets in client's side.)

The reason I think a client can't receive any packets is that a client is disconnected with a 10053 error code before the server's retransmission timeout. The client doesn't call closesocket explicitly.

I wonder why client and server can't receive any packets? What's wrong? If the reason is congestion, trying reconnect should be fail, isn't it? But almost trying reconnect is success!

I don't know who drops my packets and why?

I suspect the firewall, so I make the firewall disable on the server (just one machine). But, the problem still produces.

Finally, I suspect linux kernel/tcp stack or NIC device driver. But commonsensically the linux kernel/tcp stack is very stable, isn't it?

Do you have any idea? I'll be very appreciate any feedback.

** attach my wireshark's result

port: 4134 is a client
port: 7781 is a server
a client should send a ping msg (which size is 20 bytes) to a server per 3 seconds.
but, there is no client's ping msg from 768 to 786
a server sends a ping msg per 20 seconds. At 786 a server try to send a ping msg. (which size is 30 bytes)
but a server can't receive ack packet from a client.
At 815, a client tries to reconnect to server!! (It's my work around.) But, a server can't receive any FIN or RST.
Maybe a client had sent a RST to a server, but a server can't receive it.

alt text

Answer 1

Assuming that this trace is filtered on the remote client's ip address the client port number 'wraps' from within 4134 to 1737. Very unlikely for a windows client which uses incremental ephemeral ports.
The SYN packet arrives with a reduced MSS of 1414
The client's 'ping' packet was due at 761.64x but was never seen at the server
The server's ping packets don't go through either
815.708 into the trace - 54 seconds after the first missing packet - the client reconnects - immediately after the abort as you say your client is coded.

So, the client entered retransmission and finally aborted the connection when retries were exhausted. The new SYN packet went through immediately, so we can exclude a general IP connectivity problem. I'd say, the problem is with security device in the path that is dropping packets of your TCP session for whatever reason.

Good luck in finding out more using traces at the endpoints ! ;-)

Regards Matthias

Answer 2

I want to know who drops my packets (intermittently) and why? How can I trace this problem and how to find 'who and why?'

Well, that's hard to do, as you won't be able to easily figure that out. A drop of a security device is usually 'silent' meaning, you don't know where is happens.

One option I see is this:

On the server (better on the client as well): If you detect multiple re-transmissions/DUP ACK, etc., you could start a new thread and start sending the last packet (the one you don't get an answer for) with increasing TTL. If you're lucky, you will at least be able to nail down the approx. device that could be dropping the packet, which is the one after which you don't get ICMP time exceeded anymore. This will obviouly only work, if the routers on the way do send the ICMP messages and they are not filtered on the way to your server.

Server -> Router1 --> Router2 --> Firewall/IDS/Whatever --> Router3 --> Router4
TTL:1     # <- ICMP
TTL:2                # <- ICMP
TTL:3                             :: drop
TTL:4                             :: drop
TTL:5                             :: drop

The firewall drops the TCP packet and thus the last station you get an ICMP message from would be Router2.

Regards
Kurt