I've got a strange problem that manifested with my firewall logging denied connections back to my clients. To investigate it further I tcpdumped and wiresharked the capture:-
So the RST sent by chrome doesn't get received by the server somehow and it continues to send the resend the [FIN ACK] until it's eventually blocked by the firewall. To demonstrate here is the same behaviour with ipv4:-
and here is a iptables firewall log:-
Firefox exhibits no such problems, presumably it doesn't RST the connection. So I'm confused about what the problem is, the ipv4 connection is with nat, other than that same kit, edgerouter lite, linux iptables. asked 20 Nov '16, 09:35  gilesw edited 20 Nov '16, 11:05  sindy showing 5 of 18 show 13 more comments |
This is a static archive of our old Q&A Site.
Please post any new questions and answers at ask.wireshark.org.
Could you share us a trace? https://blog.packet-foo.com/2016/11/the-wireshark-qa-trace-file-sharing-tutorial/
In the IPv6 capture, I can see that both sides have sent their FIN+ACK packets, which may cause the firewall to drop the context of the TCP session, causing it not to forward the RST as it doesn't match any forwarding rule.
But I'm not sure the display filter
tcp.port == 49372you've used is sufficient, so better usetcp.stream == xto show only the stream with the RST.BTW, that's yet another illustration why analysing by screenshots is close to impossible. Publishing the trace according to this tutorial is much more useful.
Thanks both I'm going to get you the full trace. Just trying to get tracewrangler running.
Okay here you go, this is only the ipv6 trace:-
https://www.dropbox.com/s/lp6cu27147q8n27/443-gw_anon.pcapng?dl=0
I'd totally missed that the client had sent the [ FIN ACK ] first.
If you apply a display filter
tcp.stream == 25, you can see only 3 RST packets to be sent, while if you apply a display filtertcp.stream == 23, you can see your issue happening.The difference is that in the "bad" case, the server retransmits frame 7872, causing the client to send a RST back, until the server finally gives up about a minute later.
So I would conclude that as there was FIN, ACK from both directions, the firewall is blocking the subsequent RST packets as pointless, but if it would be the explanation, then tcp stream 25 would have to look the same, which it clearly doesn't. And in such case, I'd also expect that the firewall wouldn't let through the retransmissions coming from the server.
So can you run tcpdump on both interfaces of your firewall simultaneously? It would be fine to see whether the RST is actually forwarded or not and if yes, then how many of them, and whether in the "good" cases, the server isn't retransmitting packets as well but the firewall doesn't let them in in these cases.
Hi Sindy, thanks for taking the time to respond I really appreciate this as I'm on the verge of giving up on using ipv6 which would be sad given the effort I've put in to get it running on this edgerouter lite. I'll give it a go running tcpdump on both interfaces. This is the interface setup:-
[email protected]:~$ show interfaces
eth0 176.25.49.71/22 u/u Internet
eth1 192.168.142.2/24 u/u Local 2a02:c7f:7013:d101::1/64
eth2 192.168.143.1/24 u/D Local 2
lo 127.0.0.1/8 u/u ::1/128
So I was tcpdumping on eth0 originally.
I'm not sure I understand the overall structure of your network as there is just a single interface with IPv6 address assigned in your list, so I can't get how the traffic would be passing through the box. Is the server running at the same box like the firewall? Also, I was looking at the issue as if you were capturing at the client side of the firewall, is that true?
Okay here are 2 new captures
https://www.dropbox.com/s/64z8700khnrsfjc/443-eth0_anon.pcapng?dl=0
https://www.dropbox.com/s/5iqfqc908vab8ta/443-eth1_anon.pcapng?dl=0
and some fw logs
Nov 20 21:30:30 ubnt kernel: [wan_in-6-default-D]IN=eth0 OUT=eth1 MAC=80:2a:a8:8d:3b:44:0c:a4:02:5c:10:01:86:dd SRC=2a03:2880:f01a:0005:face :b00c:0000:0001 DST=2a02:0c7f:7013:d101:6d68:463e:f867:09dd LEN=136 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=TCP SPT=443 DPT=51729 WINDOW=114 RES=0x 00 ACK URGP=0
Nov 20 21:30:31 ubnt kernel: [wan_in-6-default-D]IN=eth0 OUT=eth1 MAC=80:2a:a8:8d:3b:44:0c:a4:02:5c:10:01:86:dd SRC=2a03:2880:f01a:0005:face :b00c:0000:0001 DST=2a02:0c7f:7013:d101:6d68:463e:f867:09dd LEN=136 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=TCP SPT=443 DPT=51728 WINDOW=114 RES=0x 00 ACK URGP=0
err... there is no RST packet in either of the two new captures, what have I missed?
I'm wondering if I didn't leave the captures running for long enough. I was getting a lot of firewall drops though.
The interface setup is odd. I assumed that I would get a ipv6 address assigned to eth0 just like I do with ipv4 but it doesn't work like that. I get a range assigned to eth1 which I then hand out with radvd. I'll run a longer scan now.
Sorry for the delay, tracewrangler is crashing now reading these files so I've put them on this cloud saas service:-
https://www.cloudshark.org/captures/68b9870e8314
Hope that's okay.
I'm going to check vyatta firewall rules I have setup now. I have been through them once but maybe something will jump out.
Might be use if I link my thread from the edgerouter forums. It appears now that I'm not the only person suffering from this.
http://community.ubnt.com/t5/EdgeMAX/Firewall-dropped-443-ipv6-traffic/m-p/1730867/highlight/true#M136015
well... the last capture you've put at Cloudshark is
only from a single interface again,
missing some packets (Wireshark says "ACKed unseen segment" and they are really not there, maybe the network driver uses TCP offloading and so the packets haven't been captured by tcpdump, so please check whether it is the case and how to switch it off),
so it is quite unusable for analysis.
As for the link to ubuntu community: Wireshark will clearly tell you whether the firewall actually steals the RSTs or not, and that's where the role and of both Wireshark and this Q&A site ends.
Apologies I thought both interfaces were accessible from that link. This is the WAN eth0 interface as well.
https://www.cloudshark.org/captures/a6d2b1473dc4
The device does include tcp offloading so I'm going to disable that and recapture, wasn't aware of that.
https://help.ubnt.com/hc/en-us/articles/205204070-EdgeMAX-How-do-I-disable-IP-offload-for-hardware-acceleration-
Could you please paste a log entry of the fw that matches a session of the actual trace?
Sorry for the lack of responses here. I'm afraid I'm snowed under at work so can't get anything done in the eve's.