We have a server sending a large UDP stream to another server for processing. The server is connected to a Layer3 switch. From there, the packets pass through another internal router to get to the destination server. By the time it gets to the proper network segment, a Wireshark capture shows the UDP stream is truncated by about 100 bytes. It starts off from the sending server at 1558 bytes (this is data only) and arrives at 1458 bytes. Captures at the Layer 3 switch ports involved all show the data at 1558 bytes, but another capture at the incoming port of the router shows the data truncated to 1458 bytes. How can this be? How can the data leaving the switch port disappear at the receiving router port? Or am I misinterpreting the captures maybe? The L3 switch is an old 3Com SSII 3300 with an L3 module. The router is a pfSense Dell SC1425 that supports jumbo frames. I'm inclined to suspect the 3Com switch, but the captures seem to exonerate it. Can anyone shed some light on this? asked 30 Jan '13, 00:01  praevidium |
3 Answers:
As @Kurt already suggested, there is fragmentation. The server that receives the packets over TCP sends them out over UDP. It sends the whole content in one datagram, which needs to be fragmented to fit the MTU. I'm not sure why you don't see both fragments at the pfSense side, but I suspect that you used a capture filter with a udp port, as the second fragment does not contain the udp header, the capture filter will drop it. You can make another trace where you use only ip addresses in the filter on the pfSense. There seemed to have been a problem in pfSense will fragmentation, which version do you use? It would be best if the server that sends the UDP packets will do fragmentation at the UDP layer. But that might not be compatible with the receiving end. Could you make traces (with only an IP cpature filter and full snaplength) on both sides of the pfSense? answered 31 Jan '13, 01:07  SYN-bit ♦♦ showing 5 of 13 show 8 more comments |
did you check if the packets have been fragmented (IP fragmentation)? Another option would be padding bytes in the ethernet frame (1558 Bytes), although 100 bytes of padding are a bit "uncommon". Can you post those two packets in pcap format somewhere? Regards answered 30 Jan '13, 12:59  Kurt Knochner ♦ Thanks for your reply, Kurt. I'll check on the fragmentation. Here's a link with the packets and a little explanation: https://docs.google.com/folder/d/0BxSuHyrWOanUeW0tamg3b21YUWM/edit John (31 Jan '13, 00:14) praevidium |
Well, here's the end of the story: The actual problem turned out not to be a stream truncation at all. A different Wireshark filter showed it had to do with IP fragmentation. The UDP packet was being fragmented and somehow the IP headers were altered and the checksums were incorrect by the time the packets hit the LAN. A packet capture at the LAN nic didn't show any errors, but one at the corresponding switch port did, which was very difficult to figure out. I resolved it by upgrading both the switch firmware and then pfSense (to 2.0.2). It was after the pfSense upgrade that the packets in question finally got to the destination server application. So, the problem really was in pfSense 1.2.2's IP fragmentation. Fortunately, pfSense 2.0.2 does IP fragmentation correctly (at least, so far). I'm relieved. answered 05 Feb '13, 05:28 praevidium So pfSense IP fragmentation was at fault, as suggested by @SYN-bit in his answer. (05 Feb '13, 05:44) grahamb ♦ Indeed. However, to reiterate, pfSense 2.0.2 does not have the same issue. (05 Feb '13, 07:14) praevidium |
You are absolutely correct. I just did another series of captures and filtered them differently and now I see the fragmentation all the way to the receiving server's switch port. However, both the UDP packet and the IP4 fragment show IP checksum errors at both switch ports involved (pfSense LAN and receiving server). Is this a concern or is it a false error due to IP checksum offload? (I'm not sure how to interpret the warning on the IP checksum line.) There are no checksum errors on the pfSense box LAN port, but I'm pretty sure its nic driver supports checksum offloading.
The switch is very old. Could it be that the switch can't handle what the pfSense box is sending it in this case?
(I converted your "answer" to a "comment", please see the FAQ on how to use this site best)
When checksum offloading is enabled you will normally only see bad checksums when capturing on the host itself and only for the outgoing frames.
I assume the checksums on the LAN side of the pfSense are OK. Is that correct? Now on the WAN side you see bad checksums? Also when the server receives the packets? Then it is indeed a pfSense issue. You will find some more info when googling for "pfSense" and "fragment".
OK. That's good to know.
I didn't re-check the WAN-side port of the pfSense box because the packets are moving from the WAN to the LAN and, yes, there are no errors at that LAN-side port. From there, they move through a switch to the receiving server. It is at that switch that I see the checksum errors. So those errors are probably real and the switch is likely the problem?
Is there checksum offloading enabled on the NIC's in your pfSense box? If you don't see errors on the box itself (packets are captured between the IP stack and the NIC driver), but you do see them on the network, then something along the way (only the NIC in your case) has modified the packets.
I just checked on a pfSense box (v1.2.3) in my lab:
(running a ping -l 2024 from 192.168.20.10 to 192.168.21.1, tcpdump output is from 192.168.21.1)
It looks like it is having no problems with the fragments. Which version of pfSense are you running?
'Hardware checksum offloading' is disabled in pfSense. So, the results that I'm talking about are with checksum offloading turned off.
How are you making the traces on the 3Com switch on the lan side, do you use a monitor/mirror/span port? Do you have a TAP at hand to place in between the pfSense and the 3Com switch? Or are you able to switch switches (pun intended)?
I have 'hardware checksum offloading' disabled on my pfSense 1.2.2 box. So, the results I'm talking about are with no checksum offloading.
Sorry, the 'echo' was not intended. A couple of comments got hidden on me.
I'm using the switch's 'Roving Analysis Port' (monitoring one port at another). No TAP at the moment and no switch replacement, either. But, to be clear, there's only the wire between the pfSense LAN port and the switch port at which I captured the packets.
You might want to upgrade pfSense to 1.2.3 (or 2.0.2, but that's a bigger step) as I see a couple of people reporting fragmentation issues when I google for pfSense 1.2.2 and fragmentation.
The 3Com switch will not look at the IP checksum let alone touch it (unless it's natting, which is not the case looking at your earlier traces).
It has been on my mind.
I'll upload those last captures to my Google Drive. The errors are with the IP checksum. But that'll have to wait until later today. I need at least a few hours sleep. It's coming up to 0500 here. Signing off for a bit...
Oops, I meant IP instead of UDP (I just corrected it in my last comment).
Sorry for keeping you awake, it's about 13:00 here :-)
OK. I'm back. I did a capture at the receiving server nic and uploaded both it and the new captures to: https://docs.google.com/folder/d/0BxSuHyrWOanUeW0tamg3b21YUWM/edit?usp=sharing
The stream leaves the pfSense box fragmented but with no checksum errors. All the subsequent captures right up to the receiving server show both packets but with IP checksum errors.
It's only these packets that have any problem. All the other network traffic is pretty normal.
As I understand it, the errors must be introduced somewhere between: LLC-MAC-Physical NIC ---> Physical NIC-MAC-LLC. Could the switch port driver be the culprit?