I'm troubleshooting a problem where wireless clients on certain APs are unable to connect to certain web servers on our wired switched/routed lan. I've traced the client SYN to the web server and verified that the server sent a SYN+ACK in response. On the wireless side the client is receiving the response that appears to have something that looks like a 802.1q header attached except that it has six bytes not four. Obviously the client doesn't process this packet properly and the broswer session times out. The reply packet(s) received at the wireless client look like: [Client MAC] [Router MAC] 81 00 81 00 20 00 [08 00 - and the rest of the IP datagram with IP/port matches the expected SYN+ACK] Wiresharks default filter sees it as a 802.1q packet with an ethertype of 0x2000 and doesn't decode the IP of the datagram. I'm working at the limits of my experience and don't know if this is a malformed 802.1a packet or if I'm looking at some other type of VLAN tag, or something else. At this point, assuming it's a VLAN tag of some kind, I'm thinking somone plugged a trunked port from one switch into a normal port on another switch near the wireless APs and the VLAN tags aren't getting stripped off. I still can't explain the six bytes though and thought I'd see if anyone else has seen anything like this. Unfortunately I don't have a good understanding of the physical layout of the routers/switches in this network, thats another department. I'm limited to sniffing on each end, and trying to give them enough information to locate the problem. I thought someone with more experience than I might have some insight, or see something obvious that I missed. asked 03 Nov '10, 12:51  bkd showing 5 of 8 show 3 more comments |
One Answer:
Have you tried disabling 802.1q on the Netgear AP? If only wireless clients are seeing this issue, I'd start by double-checking the Netgear's configuration against the (Cisco) switch to which it's connected. answered 03 Nov '10, 19:48  wesmorgan1 Today our networking group removed the trunk to an AP, changed it to an access port for one vlan and the problem went away for that AP. We have 4 APs in the same physical location with the issue. Removing the trunks isn't a good solution. Funny thing is that other APs of the same exact model, and supposedly the same exact settings, in the same subnet, with trunked connections to the same model switches are not having this issue. Later today they are supposed to switch a working AP with a problem AP and see if the problem behavior follows the AP, or sticks to the switch/port. (18 Nov '10, 13:45) bkd It sounds like you might have some Q-in-Q going on here. (0x8100 is dot1q and u have two in a row). However, without seeing the exact frame I can't be sure. Is this packet decode coming off of the switch port leading to the AP or taken from the air? (22 Nov '10, 11:21) TaddyOH |
Where are you capturing exactly? Are you capturing on the client itself, or on the AP? Is this a Cisco AP or not? I'm not sure it's a VLAN tagging issue because the frames wouldn't make it to the server if the trunk/crossconnect was configured incorrectly.
I'm capturing the packet above on the wireless client, a windows laptop. The AP is a netgear which is connected into a Cisco layer 2 switched LAN. The web servers and APs are on different subnets/vlans and there is a router between them. I'm not completely sure of the makeup of this network, but there are likely multiple switches on each side of the router and the traffic from server to client likely crosses more than 1 trunked connection.
Why do you say the client didn't process it? Do you see a RST packet or long delay in the tcp handshake? Is there packet trace that you can provide taken at both sides (or even one side?) You can use "editcap -s 96 origfile newfile" since the content is probably not interesting (not yet, anyway)
When I say the client 'didn't process it' I meant the client web browser that is initiating the IP connection never receives the SYN+ACK datagram because the host system doesn't recognize it as an IP packet. The broswer simply says the connection could not be established. The wireless client network stack isn't expecting the vlan tags.
The client browser can successfully access other web sites on the LAN and Internet, and there are no VLAN tags on the responses. In other words, most of the time, everything works and looks normal for the client. Only certain servers on the wired lan are giving problems, and to a wireless user on a web browser, those sites are just down or don't work. These servers are definitely working properly and clients on the wired lan can connect to them fine. All of the affected servers probably share a common switch at some point. Return traffic from those servers have the tag I described above
Suppose two switches in the path are connected with a cable, switch 1 has trunking turned on on it's port, but switch 2 doesn't. Packets from switch 2 would enter switch 1 untagged and, as I understand it would be placed on the default vlan and not be a problem. Reverse traffic from switch 1 to switch 2 would be vlan tagged and since the port on switch 2 isn't configured for trunking switch 2 would accept it and pass it along but wouldn't know to look for or strip off the vlan tag. That might explain a normal vlan tag showing up on reply packets. Thats my working theory anyway.
I wonder if a ping will work. There are trunks that do NOT have to be negotiated - and they are just a bad idea. I also wonder if this could be some kind of LACP misconfig - are you trying to etherchannel somewhere?
Ping replies have the same ethernet vlan header as the TCP/SYN ACK. I'm not a network guy, and I don't have access to the network configuration, only the endpoints. I don't think there are any mutipath connections in this section of the network. I'm waiting on our network hardware group to trace this connection and examine the configuration of all the ports in the path.