I have developed an application which reads data over a high volume network (2 sensors, both 1khz). The application reads the packets for a random time, and then crashes due to - what I believe to be - a native buffer overflow. The following image depicts the exacts packet sequence each time the crash occurs - a large Dup ACK block followed by 2 RST flags. Am I correct in thinking that this crash occurs due to the RWIN buffer being too small, and this in turn leads to the sending sockets shutting down. Any advice or suggestions is most appreiciated. Edit - Updated with image of sequence pre DUP-ACK
asked 04 Sep '13, 04:40  Jnanathan edited 04 Sep '13, 05:20 |
2 Answers:
Is that just an application on top of a standard OS or are those 'embedded devices' with a their own TCP/IP stack? That many DUP-ACK looks more than strange. I don't think that any recent OS (Windows, Linux) would send that many DUP-ACK that fast. So, I guess these are either duplicate packets (possibly a routing loop) or another problem in the TCP/IP stack. Can you please add the following information:
BTW: is it possible to post the capture file instead of a screenshot? You can post the capture file on google docs, dropbox or cloudshark. UPDATE To me it looks like the Win7 system (your application) is not able to process the data (fast enough). In both TCP streams, the TCP window size reported by Win7 drops constantly until it reaches 0 (see tcp.stream eq 1 - it shows it pretty nicely).
The embedded system keeps sending ACKs and the Win7 finally (after a long time) closes the connection with a RESET. So, the key to your problem is: Why does the Win7 system lower its TCP window size, i.e. why is it incapable of handling the data (fast enough). As that is your own application, the only way to understand that behavior is by debugging the application. Maybe the application does not receive the data it is expecting and your program logic waits for the correct data, until the TCP receive buffers are full. Regards answered 04 Sep '13, 06:14  Kurt Knochner ♦ edited 06 Sep '13, 02:49 showing 5 of 8 show 3 more comments |
I see only one RST; the others belong to a different connection (with the IP .108 instead of .106). But from that long long long list of Duplicate ACKs I can tell that something is going very wrong, and I guess your application has trouble with packet loss and recovery. There is no way to tell what stopped the sending process because it is not in the screenshot - the screenshot only tells the end of the story. It would be helpful to see how the Duplicate ACK series got started. answered 04 Sep '13, 04:57  Jasper ♦♦ Hi Jasper. Fist of all, I should have noted the 2 IPs of both sensors - being .106 & .108 Secondly, I have re-edited the original with an image of the sequence before the DUP-ACK segment. As you will see there is a "TCP Window full" alert. Any idea what is causing this - might it be the buffer window on the receiving end? (04 Sep '13, 05:10) Jnanathan Window full means that the sender has sent as many bytes as the receiver can buffer, so if you see that message it means that the sender has to pause and wait for the next acknowledge with a new window size to come in. In your case I'm pretty sure that this situation is the problem, because it doesn't seem to recover from that at all. I would expect a "Window Update" message from .2 to .108, but apparently it never arrives, so .108 cannot continue. The stack of .108 is also not behaving like most stacks, because it should do Window Probing instead of ACKing the last segment. (05 Sep '13, 09:27) Jasper ♦♦ |
Hi Kurt, to answer your questions:
I think the link got lost ;-)
Cheers Jasper, it can be found below. I had to break the link into seperate parts to prevent it from being taken as spam.
https://mega.co.nz/#!8M90VKaQ!RnvEnmIMWhm312QLrRy3R0CM6SsLaORN9x6K8rBLLb0
O.K. please tell us the IP address of the Win7 system and the address of the embedded device.
BTW: I 'fixed' the link in your comment.
Thanks Kurt. The address of the Win7 system is 109.168.0.2, and those of the two sensors are .106 & .108.
see the UPDATE in my answer.
Cheers for the break down Kurt. I believe this to be an issue with the hardware that the application is running on. The sensors are sending at a rate of 1khz each, and the host machine gets overloaded after a period of time - In that the CPU cannot process packets from the receive buffer, and this in turn causes the backlog. Thank you for your assistance.
You're welcome.
Hint: If a supplied answer resolves your question can you please "accept" it by clicking the checkmark icon next to it. This highlights good answers for the benefit of subsequent users with the same or similar questions.