Network guys set up a span port with some traffic to my box. Currently I am observing strange behaviour. Consequently and repeatedly I see "X bytes missing in capture file". I have done such scenario: Open an URL in my browser and tap F5 or ctrl-F5. File is opened correctly. In capture file I see: ET /pc/resources/Ocean/images/logo.gif HTTP/1.1 Host: 10.5.34.230:8080 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive Pragma: no-cache Cache-Control: no-cache [4380 bytes missing in capture file].p .......:..Z..z... ........... ....P..z................; Capture is done by: tcpdump -i any -c 1000 -nn -s 0 -w /tmp/1 host 10.5.34.230 and host 10.5.1.189 This behaviour is observed for this one machine. Net guys say that span port is configured correctly and couple of more span ports are targeted to my box and seems to be fine. I have chcecked this when interface which feeds my box is busy and when its not busy. I have used tcpdump to any interface as well as eth6 (which is the interface receiving traffic). The server is a virual one (vmware). Server guy configured it to use only one interface. This response is always the same. Always in the same place, always 4380 bytes (for this particular object). How to diagnose this problem further? I will ask server admin for access to make capture on his server but I need some advice for network guys or for me to be sure I have set up everything correctly. asked 22 Oct '12, 02:16  ptoki |
3 Answers:
Wireshark will only show that error message, if you follow a TCP stream and wireshark is not able to reassemble all TCP segements, as some packets are missing in the capture file. There are several reasons why that might happen:
Now, the interesting part would be to figure out which one of the above conditions causes the problem. Suggestion: Mirror the traffic to a second port and capture there with a second machine. Then compare the capture files. If they contain the same data, the missing packets are dropped "somewhere" (monitored port, source system, "on the way"). If the capture files are different, the system with less packets in the capture file was not able to write all packets to disk (overload). Regards answered 22 Oct '12, 04:24  Kurt Knochner ♦ edited 22 Oct '12, 04:26 |
I think this is in fact a problem with the SPAN port, and I'm not talking about a wrong configuration, just a wrong approach. If you SPAN one full duplex line to a SPAN port and both receive and transmit channels are 100% busy you have a problem, because the monitor port of the SPAN session will only be able to transmit half of it to the analyzer - which means, 50% of the packets will be dropped at the switch. This gets worse if you add more SPAN source sessions to the same monitor port - as soon as the switch has to transmit more packets out the monitor port than there is bandwidth it will drop the frames. That leads to tcpdump not capturing them, because they never got to the device. This can also happen if not all source ports are busy both ways, it is enough to add more source port traffic to the monitor port than it has bandwidth towards the analyzer/capture machine. My guess is, that when you test your transmission the buffers of the switch get slammed shut, always leading to the dropout of the same packets each single time. How to solve this: make sure that you do not mirror more bits/s to the output port than the line has bandwidth. If you have a really crowded connection and you can't reduce the amount of traffic you should think about using full duplex TAPs instead. answered 22 Oct '12, 08:32  Jasper ♦♦ edited 22 Oct '12, 08:34 |
I have partially solved this issue. Reason to this was problem with cisco nexus device. One nexus had span port configured and sent to another switch. And there was problem with MTU between them. Our support confirmed this problem and suggested changing MTU. This helped a bit but not between nexus and catalyst 6500. So we changed method of collecting traffic. Thank You both for suggestions but it was not the case here. Missing bytes was always in the same place in tcp communication thats why I did not expect to have problem with bandwidth or network card or cpu usage or similar. Problems with bandwidth we have succesfully solved by network teaming (bonding) of interfaces which are fed by span ports. answered 04 Feb '13, 00:57 ptoki |
