This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

I've got a strange problem that manifested with my firewall logging denied connections back to my clients. To investigate it further I tcpdumped and wiresharked the capture:-

So the RST sent by chrome doesn't get received by the server somehow and it continues to send the resend the [FIN ACK] until it's eventually blocked by the firewall.

To demonstrate here is the same behaviour with ipv4:-

and here is a iptables firewall log:-

Nov 17 23:34:07 ubnt kernel: [wan_in-6-default-D]IN=eth0 OUT=eth1 MAC=80:2a:a8:8d:3b:44:0c:a4:02:5c:10:01:86:dd src=2a03:2880:f01a:0005:face:b00c:0000:0001 DST=2a02:0c7f:7013:d101:bc7e:76c1:e09e:b2f7 LEN=136 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=TCP SPT=443 DPT=56761 WINDOW=114 RES=0x00 ACK URGP=0

Firefox exhibits no such problems, presumably it doesn't RST the connection.

So I'm confused about what the problem is, the ipv4 connection is with nat, other than that same kit, edgerouter lite, linux iptables.

asked 20 Nov '16, 09:35

gilesw's gravatar image

gilesw
6114
accept rate: 0%

edited 20 Nov '16, 11:05

sindy's gravatar image

sindy
6.0k4851

In the IPv6 capture, I can see that both sides have sent their FIN+ACK packets, which may cause the firewall to drop the context of the TCP session, causing it not to forward the RST as it doesn't match any forwarding rule.

But I'm not sure the display filter tcp.port == 49372 you've used is sufficient, so better use tcp.stream == x to show only the stream with the RST.

BTW, that's yet another illustration why analysing by screenshots is close to impossible. Publishing the trace according to this tutorial is much more useful.

(20 Nov '16, 11:54) sindy

Thanks both I'm going to get you the full trace. Just trying to get tracewrangler running.

(20 Nov '16, 12:00) gilesw

Okay here you go, this is only the ipv6 trace:-

https://www.dropbox.com/s/lp6cu27147q8n27/443-gw_anon.pcapng?dl=0

I'd totally missed that the client had sent the [ FIN ACK ] first.

(20 Nov '16, 12:11) gilesw

If you apply a display filter tcp.stream == 25, you can see only 3 RST packets to be sent, while if you apply a display filter tcp.stream == 23, you can see your issue happening.

The difference is that in the "bad" case, the server retransmits frame 7872, causing the client to send a RST back, until the server finally gives up about a minute later.

So I would conclude that as there was FIN, ACK from both directions, the firewall is blocking the subsequent RST packets as pointless, but if it would be the explanation, then tcp stream 25 would have to look the same, which it clearly doesn't. And in such case, I'd also expect that the firewall wouldn't let through the retransmissions coming from the server.

So can you run tcpdump on both interfaces of your firewall simultaneously? It would be fine to see whether the RST is actually forwarded or not and if yes, then how many of them, and whether in the "good" cases, the server isn't retransmitting packets as well but the firewall doesn't let them in in these cases.

(20 Nov '16, 13:09) sindy

Hi Sindy, thanks for taking the time to respond I really appreciate this as I'm on the verge of giving up on using ipv6 which would be sad given the effort I've put in to get it running on this edgerouter lite. I'll give it a go running tcpdump on both interfaces. This is the interface setup:-

[email protected]:~$ show interfaces

eth0 176.25.49.71/22 u/u Internet

eth1 192.168.142.2/24 u/u Local 2a02:c7f:7013:d101::1/64

eth2 192.168.143.1/24 u/D Local 2

lo 127.0.0.1/8 u/u ::1/128

So I was tcpdumping on eth0 originally.

(20 Nov '16, 13:25) gilesw

I'm not sure I understand the overall structure of your network as there is just a single interface with IPv6 address assigned in your list, so I can't get how the traffic would be passing through the box. Is the server running at the same box like the firewall? Also, I was looking at the issue as if you were capturing at the client side of the firewall, is that true?

(20 Nov '16, 13:33) sindy

and some fw logs

Nov 20 21:30:30 ubnt kernel: [wan_in-6-default-D]IN=eth0 OUT=eth1 MAC=80:2a:a8:8d:3b:44:0c:a4:02:5c:10:01:86:dd SRC=2a03:2880:f01a:0005:face :b00c:0000:0001 DST=2a02:0c7f:7013:d101:6d68:463e:f867:09dd LEN=136 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=TCP SPT=443 DPT=51729 WINDOW=114 RES=0x 00 ACK URGP=0

Nov 20 21:30:31 ubnt kernel: [wan_in-6-default-D]IN=eth0 OUT=eth1 MAC=80:2a:a8:8d:3b:44:0c:a4:02:5c:10:01:86:dd SRC=2a03:2880:f01a:0005:face :b00c:0000:0001 DST=2a02:0c7f:7013:d101:6d68:463e:f867:09dd LEN=136 TC=0 HOPLIMIT=57 FLOWLBL=0 PROTO=TCP SPT=443 DPT=51728 WINDOW=114 RES=0x 00 ACK URGP=0

(20 Nov '16, 13:39) gilesw

err... there is no RST packet in either of the two new captures, what have I missed?

(20 Nov '16, 13:47) sindy

I'm wondering if I didn't leave the captures running for long enough. I was getting a lot of firewall drops though.

The interface setup is odd. I assumed that I would get a ipv6 address assigned to eth0 just like I do with ipv4 but it doesn't work like that. I get a range assigned to eth1 which I then hand out with radvd. I'll run a longer scan now.

(20 Nov '16, 14:07) gilesw

Sorry for the delay, tracewrangler is crashing now reading these files so I've put them on this cloud saas service:-

https://www.cloudshark.org/captures/68b9870e8314

Hope that's okay.

(20 Nov '16, 14:21) gilesw

I'm going to check vyatta firewall rules I have setup now. I have been through them once but maybe something will jump out.

(20 Nov '16, 14:28) gilesw

Might be use if I link my thread from the edgerouter forums. It appears now that I'm not the only person suffering from this.

http://community.ubnt.com/t5/EdgeMAX/Firewall-dropped-443-ipv6-traffic/m-p/1730867/highlight/true#M136015

(21 Nov '16, 02:34) gilesw
1

well... the last capture you've put at Cloudshark is

  • only from a single interface again,

  • missing some packets (Wireshark says "ACKed unseen segment" and they are really not there, maybe the network driver uses TCP offloading and so the packets haven't been captured by tcpdump, so please check whether it is the case and how to switch it off),

so it is quite unusable for analysis.

As for the link to ubuntu community: Wireshark will clearly tell you whether the firewall actually steals the RSTs or not, and that's where the role and of both Wireshark and this Q&A site ends.

(21 Nov '16, 02:45) sindy

Apologies I thought both interfaces were accessible from that link. This is the WAN eth0 interface as well.

https://www.cloudshark.org/captures/a6d2b1473dc4

The device does include tcp offloading so I'm going to disable that and recapture, wasn't aware of that.

https://help.ubnt.com/hc/en-us/articles/205204070-EdgeMAX-How-do-I-disable-IP-offload-for-hardware-acceleration-

(21 Nov '16, 04:45) gilesw

Could you please paste a log entry of the fw that matches a session of the actual trace?

(21 Nov '16, 05:04) Christian_R

Sorry for the lack of responses here. I'm afraid I'm snowed under at work so can't get anything done in the eve's.

(23 Nov '16, 02:13) gilesw
showing 5 of 18 show 13 more comments
Be the first one to answer this question!
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×51
×7

question asked: 20 Nov '16, 09:35

question was seen: 1,131 times

last updated: 23 Nov '16, 02:13

p​o​w​e​r​e​d by O​S​Q​A