Hi guys, I have been using wireshark to troubleshoot this network issue we have been facing. Client iniitates a tcp connection from a server to a third part application and on odd occasions more than one TCP connections are formed between the server and the application. From the TCP traces on wireshark, I can see the handshaking for these connections, but cannot get my head around why is this happening? I can upload the traces to cloudshark. Would really appreciate if someone can help or advise what I should do to troubleshoot? I have exhausted all I know! asked 11 Nov '13, 20:32  Ken15 showing 5 of 9 show 4 more comments |
2 Answers:
"I checked again and the "\001.LOAD\00313240\003Orange News\0031970-01-01T00:00:00.000\004" is the first one sent from the playout client. But also, this data that is sent is received ok by the Captions player successfully! then why would the connection be reset?" The data is NOT received by the Captions player. It has arrived at the windows IP layer and was presented to the TCP layer. The TCP layer however couldn't find a socket for the connections and sends back a RST, which informs the sender that the data has not been delivered to the application. The question "why would the connection be reset?" should be converted to "Why are the sockets no longer there?" and need to be addressed to the vendor. The assumption is that they expected to read data sooner which did not arrive but - without knowing the logic of the program this is just speculation. Filter to produce the Statistics -> Flowgraph: answered 21 Nov '13, 20:57  mrEEde edited 21 Nov '13, 21:00 Hmm. I understand that the data is sent at Windows level as there are no sockets on the TCP layer therefore a RST is sent by the captions player. The part where it tried to form 6 odd connections is very bizzare, mind you only on this occasion. Other times there are 1-7 multiple connections randonmly! Also, the other thing I asked earlier was that after all the resets when the first handshake takes place there are no 'IDLE' messages sent for around 3 seconds could that be the reason for more connections being formed? (21 Nov '13, 21:17) Ken15 https://drive.google.com/file/d/0B6mIG-yxPHOSNXo5LXVlN3hWdFk/edit?usp=sharing Also, whats exactly the difference between this trace and the others. This is an example where only one connection formed! Also, the reason I said that the "Load" was sent and received was because the captions receives it as in the application itself. What the load command does is send all the captions for all the stories in a particular bulletin. Then the other Channel A and B commands sent from the playout client to captions is for the captions for individual stories to playout when the story is played from channel A or B. And the CG player message you were talking about is ignored by the captionscontroller. I promise last question @mrEEde. After this time to contact the vendor! Thansk for all your input and help . Really appreciate it, just this last one. (21 Nov '13, 23:26) Ken15 This trace is showing a new connection setup 3.85 seconds in the trace. On this single session 5 'requests' are sent after a 2.62 seconds delay. You receive responses from the caption controller and then the session sits idle for 17 seconds. From what we've seen so far a subsequent request (after a pause) will possibly get a RST packet causing the client to start 2 sessions which will get RSTs after a pause causing the client to start 3 sessions which will get RSTs after a pause ... maybe ;-) (22 Nov '13, 07:31) mrEEde I can see it start after 3.85 seconds and the request after 2.62 delay. But the delay is seen in all cases even the case with 6 RST. Sorry mate, I can't see the session being idle for 17 seconds?? The may be bit... Also, the case with 6 RST, I am guessing they were all formed at the very beginning and then around 6 seconds the first request was sent by the playout client and then all the RST'S. Sorry, I still don't understand the difference in circumstances between the trace I last posted and the one before that(6 RSTS). (22 Nov '13, 16:59) Ken15 |
Certainly a strange behaviour to start 3 TCP sessions to the same server and sending exactly the same request at the same time answered 13 Nov '13, 03:26 mrEEde Thanks @mrEEde. May I ask what filter did you run to see the above information. The problem is that playout client and the captions controller are from two different vendors but seems like the playout client is openign the connections. The captions controller is a simple application which just listens on a port and accepts data. Both softwares vendors are adamant and blaming each other! Thats why I thought if I can get some information and prove that the fault is with the playout client and there is nothing wrong with the network itself then I can get somewhere. Any suggestions on this? (13 Nov '13, 03:58) Ken15 The "tcp[13]&6 or data.data[0]>0" is filtering all SYN and RST packets and in addition all packets that contain a value greater than 0 in the first byte: Assuming the 0x01 is a 'request' and 0x06 is a 'response' you can create a coloring rule and assign different colors for those events (and name it accordingly) (13 Nov '13, 04:15) mrEEde So, from an external point of view, the first two sessions die (after a certain time of idleness?) because the server doesn't have the sockets anymore to receive the data. Triggered by this event the client opens a new connection (immediately) without sending a request. 2.61 seconds after that sends its first 'IDLE' message for Video player A and immediately starts a new TCP connection (without sending data again) I'd second your assumption: 1. It's the playout client that is issuing new connect() calls - without an obvious need 2. The 'network' is healthy - in fact there is not much of a network between client and server ;-) (13 Nov '13, 04:16) mrEEde @mreeDE - That makes sense. Thanks for filter information. Also, looking at other instances of logs gathered, I have noticed that the first two sessions always die and then multiple connectiosn are open. After the first two session die, I thought the server send a sync request and received a SYN ACK back form the and then sends a ACK back to the controller? The other things I wanted to add is that. The first connection which has the text Wagga News is what is supposed to sent to the controller. The video player A and B shouldn't be opening connection. So May be once "Wagga News" captions are sent to the controller it resets the connection? is that a fair assumption? Also, how to get the text you got above? I can't seem to get it. (13 Nov '13, 18:56) Ken15 @mreeDE- would it be possible to talk to you on msn on skype or something please? I won't take too much of your time. (13 Nov '13, 19:23) Ken15 @mreeDE - I still can't reproduce the screen you have above? (13 Nov '13, 21:53) Ken15 You need to go to Edit -> Preferences -> Protocols -> Data and check the "Show data as text" filed to be able to add 'Text' as a Column (13 Nov '13, 22:29) mrEEde @mrEEde- I did tha but still doesn't work. Even the filter . https://drive.google.com/file/d/0B6mIG-yxPHOSLXBsUzY1cmRtVEU/edit?usp=sharing Also, did you make any more changes to get the above screenshot you posted? If I can get that, it will be valuable in troubleshooting and providing the information to the software company. (14 Nov '13, 16:51) Ken15 1 You are using the default profile. So here is how you can add the text as a column So a good start is to create a new one Shift-Ctrl-A and add an new profile Under the Edit-Preferences [Shift-Ctrl-P] Under "Protocols" Enable the 'Display hidden protocol items' Under "Protocols->Data" Enable 'Display data as text' Apply filter tcp.len > 0 and expand the packet detail 'Data' section RightClick on the Text and Apply as column The screenshot I provided took quite some time to create. Feel free to pass it on to the software vendors or try to build it yourself, this is how you learn to use wireshark. There are some videos and other resources out there that can help you getting there. http://www.youtube.com/results?search_query=profiles+wireshark&sm=3 For coloring rules I want to refer you to https://www.lcuportal2.com/attachments/article/120/WSJ202-Troubleshooting%20with%20Color-100511.pdf I'm not going to walk you through every step in this forum ;-) (14 Nov '13, 22:58) mrEEde Thanks that helped a lot. Just one question, I cant get the packets in order, like above you have the rst flag after the PSH, CK sent from the playout to Captions? I can't ghet th epackets numbers in order like you have? (18 Nov '13, 16:23) Ken15 If you're packets are out of order you must have sorted on a column while processing the trace. So my golden rule #1 is: Never remove the frame.number column from the packet list because otherwise you cannot put the packets in chronological order again. If the "No." column is still there, click on the column title (clicking once sorts in ascending order) (18 Nov '13, 22:06) mrEEde Nah, I haven't sorted anything according to a column. What I meant was that in filtering that you did, the No, shows 1, 2,3 and so on to shwo communication between playout and captions. (18 Nov '13, 22:56) Ken15 Oh, now I see: I filtered on tcp protocol first and File -> Save As -> 'exported specified packets" as cc.tcp.pcapng. (19 Nov '13, 11:27) mrEEde Thanks mate! just another one. When I used the filter you advised, I can't see the reset at all. I even tried tcp.flags.reset present under tcp. My filter looks like this tcp.len >0 and (ip.addr eq 10.62.61.11 and ip.addr eq 10.62.61.105) and tcp.flags.reset. (19 Nov '13, 16:13) Ken15 Hm, I guess this is what you want: "all packets containing data or the RST bit between the two ip-addresses" ? (tcp.len >0 or tcp.flags.reset==1) and ip.addr eq 10.62.61.11 and ip.addr eq 10.62.61.105 (19 Nov '13, 21:37) mrEEde Got it! Cheers. Had another related question. As I mentioned before, I think there has to be something to do with RST being sent back from the captions because, this morning I took another set of logs and can see that the captions controller setn 6 resets! 7 connections were formed from the playout to the captions. Any comment? https://drive.google.com/file/d/0B6mIG-yxPHOSSGU1czZBUy01R0U/edit?usp=sharing Also, could this be due to TCP window size by any chance? (20 Nov '13, 17:28) Ken15 Also, another thing I couldn't understand is for the connections RST is sent I can't see the TCP handshake for any of those? (20 Nov '13, 20:05) Ken15 "... As I mentioned before, I think there has to be something to do with RST being sent back from the captions because, this morning I took another set of logs and can see that the captions controller sent 6 resets! 7 connections were formed from the playout to the captions. Any comment?" 6 sessions (client_port: 4158,4165,4193,4221,4227,4258) are reset by the captions controller when the playout client sends the same "CG Player IDLE" message over all sessions obviously after a certain amount of idle time. So from a distance I'd say the controller wants to receive data periodically, if not, it will silently close its socket without notifying the playout client. "Also, could this be due to TCP window size by any chance?" - No, window sizes are close to a full 64k indicating there is plenty of room in the receivebuffer. (20 Nov '13, 23:10) mrEEde "Also, another thing I couldn't understand is for the connections RST is sent I can't see the TCP handshake for any of those? " You can't see the 3-way handshake because they occured before the trace was started. The first packets for tcp.port==3390 occur 6.372 seconds into the trace, which means the sessions had been active longer than that and the playout client did NOT send any IDLE messages in this period. So the question is, why does the playout client not send heartbeats when the server obviously expects them ... (20 Nov '13, 23:18) mrEEde @mrEEde -Thanks for your reply. I thought that the reset is sent when the "Load" command is sent by the playout client. Also, to understand the last bit you explained the 6 sessions were open before wireshark captured them? Since there was 6.32 or more seconds of idle time, and the fact that there was an idle time to begin with those connections were reset? Also, there is no heartbeat set up coz I thought its TCp and it doesn't require any? I think I know what you mean! I can see that there is atleast 2 seconds difference between the two connections being open. As you said, that could it be that either one of them thinks that the connection is dead and then opens another one? So if I implement a keep alive or heartbeat this might not happen? (21 Nov '13, 05:42) Ken15 There were 6 sessions active and idle (no traffic for at least 6.3 seconds) when the trace started. Then the client sent a 'CG Player IDLE' on EVERY session which cause the TCP stack on the caption controller to send a RST as the sockets are no longer there: 'Somebody' must have closed the sockets in the meantime for whatever reason. The assumption is that this "might" have happened because no traffic was seen (= no data was read from the socket) within certain time interval but only the vendor can tell for sure. TCPKEEPALIVE won't help you here as this would not cause the server to read any data from the socket. Time to talk to the vendors now I'd say... (21 Nov '13, 08:12) mrEEde Thanks again. Sorry to be annoying, I can see the 6 connections, but the first message from the client is "\001.LOAD\00313240\003Orange News\0031970-01-01T00:00:00.000\004" at 6.372728 and then first reset right after that is at 6.372744 where is the "CG player Idle"? Yup, I am going to talk to vendors now. I just want to be sure! (21 Nov '13, 13:11) Ken15 Ok, must have been looking at a previous trace .. Ok, looks like we're done here in the wireshark forum so "accept" the answer please to close this thread ... (21 Nov '13, 14:33) mrEEde ok sure, hopefully I am not missing something in the trace. I checked again and the "\001.LOAD\00313240\003Orange News\0031970-01-01T00:00:00.000\004" is the first one sent fro the playout client. But also, this data that is sent is received ok by th Captions player successfully! then why would the connection be reset? Last one and wil close the thread. (21 Nov '13, 14:43) Ken15 2
if you want to know why something happens you should ask those who know how the software works. As nobody here has access to the code of that software, nobody will be able to answer the why question. As @mrEEde already said: Time to talk to the vendor now ... (23 Nov '13, 09:28) Kurt Knochner ♦ showing 5 of 25 show 20 more comments |
Without more info about the upper layer application protocol we/you have now way of knowing under which circumstances the client will open another TCP session. TCP-wise this is a valid thing to do. If you feel that this is causing a problem in your communication between the "playout client" and the "captions controller" you need to contact the vendor of this software and document the problem.
"I can upload the traces to cloudshark." - Go ahead if you want us to help you understand what's going on ;-)
Hi, my account hasn't been activated on cloudshark, any other I can share the captures?
google drive, dropbox, cloudshark.org (without account), mega.co.nz, or one of the numerous other file sharing services on this planet ;-)
Thanks Kurt,
Here you go https://drive.google.com/file/d/0B6mIG-yxPHOSYWpaSjU4d292TEk/edit?usp=sharing
https://drive.google.com/file/d/0B6mIG-yxPHOSeWNrTWFZZV9ySVk/edit?usp=sharing
Here are links for the captures. The servers Ip is 10.62.61.105 (playct) and the application it is talking to is 10.62.61.11(cc)
Please let me know if you guys require any more information.
There is communication between 10.62.61.105 and 10.62.61.11 in both directions, using the same port 3390. Which one of those connections is the one you don't understand?
So the concern is that you have parallel TCP connections when you expected to see only 1 at a time?
Hi Guys,
Yes mrEEde, thats the issue. The capture I posted is one example, sometimes there are 4 or 5 connections at the same time. I also have instances where there is only one connection.
@Kurt - The cc device uses the same port of course but the playct server uses different ports which I don't understand. In theory, there should only be one connection open.
https://drive.google.com/file/d/0B6mIG-yxPHOSdkNsQVJHRHh1Ulk/edit?usp=sharing
This is another example with the capture from just playct, where you can see that only one connection is established.
https://drive.google.com/file/d/0B6mIG-yxPHOSam12VzhiY2xFa3M/edit?usp=sharing https://drive.google.com/file/d/0B6mIG-yxPHOSRlZiOHVoOUoxLUE/edit?usp=sharing
This is another one where 3 connections are open.
The only difference I have been able to spot is that when multiple connections are open that is the time when the cc send a reset command at the begining! This has been the case in all the captures where multiple connections are open. But I'm not convinced because even when there is a reset, there are multiple connections open after that. I have tried looking at the expert info as well, but couldn't see anything significant.
Why do you think that? What kind of software is this?
Its just a playout client and sends caption (subtitles) for playout to the cc( captions controller). Sorry, should have been clear about this earlier. If you have had a look at the files I uploaded a while back, it opens one connection that time. This is not normal behaviour, I have seen instances where sometimes 7 simultaneous connections are open.