I am having some issues with a fleet of devices that connects to the Internet through GPRS. These location devices send information about the position of the vehicles on the fleet in a regular basis. The problem is that from time to time, some of the devices enter in a weird TCP loop. They try to establish a TCP connection using the 3-way handshake but this is what I see on the server side: Server <- Client SYN Server -> Client SYN-ACK Server <- Client ACK Server <- Client RST The final RST is sent immediately after the ACK. The network is something like this: Device(GPRS) - NAT Router - Internet - Firewall - NAT Router - Server I cannot see what is really happening in the client side (i.e., the devices). This is turning out into a big problem because all the devices which enter this loop, eat their data communications. Any help regarding these behaviour will be really appreciated. I have a capture if this could provide more information. asked 16 Mar '15, 02:15  Javier Ruiz edited 16 Mar '15, 02:56  grahamb ♦ |
One Answer:
The one thing that seems odd is that the client keeps reusing the same port number. This is not always a problem, but in your case, it is doing it repeatedly within less than 2 seconds after the previous connection. This might confuse the application and lead to a socket close. Can you check if you can force the clients to use different ports each time to see if this helps? Or, if you have access to it, check how the code of the application on the client handles connections that use the exact same socket pair than the previous one? Also, you might want to check what the differences are when the connections are working. Maybe you can see that they use different ports on the client side each time? answered 16 Mar '15, 03:08  Jasper ♦♦ I will change the configuration of the devices to open the port range, as this is the only control I have over them. The provider of the devices told me that they connected to the Internet through a NAT router which obviously changes IP and port to public values. Once I have tested this new configuration I will tell you if it works. Thx (16 Mar '15, 03:18) Javier Ruiz One question about opening the port range. If I do so, assuming there is a NAT router between the devices and the Internet, how these new range will be translated in the NAT router? I mean, if IP 1 port 1 is translated into IP 11 port 11, would be IP 1 port 3 translated into IP 11 port 33, for example? (16 Mar '15, 09:26) Javier Ruiz That depends on the NAT strategy of the router. Some just increment the port for each new connection, others try to keep the original as long as it doesn't colide with an existing port, and then there are some that just randomize ports. Everything is kept in a NAT table, so the router knows what external ip/port pair belongs to which internal pair. (16 Mar '15, 09:34) Jasper ♦♦ Jasper explained the working of a masquerading NAT device, which I assume is used in the client setup. I assume you have control of the server side of things. What is the NAT policy on the Firewall and on the NAT router on the server side? It does not need to translate the source address/port I assume, just the destination IP address. Is it configured to do a static NAT? It would help if you can explain the server side of things a bit more and preferably add traces (as mentioned in a comment to the question). (16 Mar '15, 09:42) SYN-bit ♦♦ Finally this seems to be the answer to my problem. I made the change remotely to the devices and the situation has been stabilized. Thanks everybody for the support. (20 Mar '15, 01:08) Javier Ruiz |
Can you post an example trace, e.g. on www.cloudshark.org? Sanitize it with TraceWrangler (https://www.tracewrangler.com) if you need to - for a problem like this it's probably enough to see the TCP headers.
I have upload the capture to: https://www.cloudshark.org/captures/8f83272d4cb9
Using these filter provides an easy example of what happens to a device: ip.src == 190.239.66.197 || ip.dst == 190.239.66.197 Btw, thx for the quick answer, Jasper.
Can you also make a trace on the Internet side of the FW and between the FW and the NAT router? Is the client really using the same port for each connection or is that caused by the FW of NAT router?
I have no control on the client side more than configuring the devices OTA. I can change the port range using an script but I have no idea how the NAT router translate this afterwards. (Btw, I set a single port because the manufacturer asked me to do so, because my first intention was to set a wide range.)