dumpcap/tshark loopback capture continual TCP ACKed unseen segment, fine on tcpdump

Question

I'm bumped into a weird bug on a virtualized RHEL/Centos 7.2 where if you capture on the loopback interface with tshark or dumpcap, that I get continual "TCP ACKed unseen segment" errors. But it works fine with tcpdump

In this scenario, I have HAProxy doing SSL termination and then proxying to tomcat running on 8080.

For example:

$tshark -i lo -f 'tcp port 8080'
Capturing on 'Loopback'
  7 3.492486804    127.0.0.1 -> 127.0.0.1    HTTP 436 [TCP ACKed unseen segment] HTTP/1.1 200 OK  (text/html)
  8 3.492498208    127.0.0.1 -> 127.0.0.1    TCP 66 [TCP Previous segment not captured] 41497 > http-alt [ACK] Seq=95 Ack=371 Win=44800 Len=0 TSval=701453508 TSecr=701453508
  9 3.492541125    127.0.0.1 -> 127.0.0.1    HTTP 68 Continuation or non-HTTP traffic
 10 5.007425117    127.0.0.1 -> 127.0.0.1    TCP 74 41499 > http-alt [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701455023 TSecr=0 WS=128
 11 1544792488.772437180    127.0.0.1 -> 127.0.0.1    TCP 74 http-alt > 41499 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701455023 TSecr=701455023 WS=128
 12 5.007450399    127.0.0.1 -> 127.0.0.1    TCP 66 41499 > http-alt [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=701455023 TSecr=701455023
 13 5.007968787    127.0.0.1 -> 127.0.0.1    HTTP 436 [TCP ACKed unseen segment] HTTP/1.1 200 OK  (text/html)
 14 5.007980410    127.0.0.1 -> 127.0.0.1    TCP 66 [TCP Previous segment not captured] 41499 > http-alt [ACK] Seq=95 Ack=371 Win=44800 Len=0 TSval=701455023 TSecr=701455023
 15 5.008025474    127.0.0.1 -> 127.0.0.1    HTTP 68 Continuation or non-HTTP traffic
 16 8.422922387    127.0.0.1 -> 127.0.0.1    TCP 74 41504 > http-alt [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701458438 TSecr=0 WS=128
 17 1544807156.089171770    127.0.0.1 -> 127.0.0.1    TCP 74 http-alt > 41504 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701458438 TSecr=701458438 WS=128
 18 8.422947104    127.0.0.1 -> 127.0.0.1    TCP 66 41504 > http-alt [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=701458438 TSecr=701458438
 19 8.423516653    127.0.0.1 -> 127.0.0.1    HTTP 436 [TCP ACKed unseen segment] HTTP/1.1 200 OK  (text/html)
 20 8.423526522    127.0.0.1 -> 127.0.0.1    TCP 66 [TCP Previous segment not captured] 41504 > http-alt [ACK] Seq=95 Ack=371 Win=44800 Len=0 TSval=701458439 TSecr=701458439
 21 8.423632416    127.0.0.1 -> 127.0.0.1    HTTP 68 Continuation or non-HTTP traffic
 22 8.423633078    127.0.0.1 -> 127.0.0.1    TCP 66 [TCP ACKed unseen segment] http-alt > 41504 [FIN, ACK] Seq=371 Ack=95 Win=43776 Len=0 TSval=701458439 TSecr=701458439
 23 8.423657361    127.0.0.1 -> 127.0.0.1    TCP 54 http-alt > 41504 [RST] Seq=371 Win=0 Len=0

However if I specify any snaplen (including 65535), then it works properly:

$tshark -s 1500 -i lo -f 'tcp port 8080'
Capturing on 'Loopback'
  1 0.000000000    127.0.0.1 -> 127.0.0.1    TCP 74 41926 > http-alt [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701753517 TSecr=0 WS=128
  2 1546074207.537697209    127.0.0.1 -> 127.0.0.1    TCP 74 http-alt > 41926 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701753517 TSecr=701753517 WS=128
  3 0.000023710    127.0.0.1 -> 127.0.0.1    TCP 66 41926 > http-alt [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=701753517 TSecr=701753517
  4 0.000039374    127.0.0.1 -> 127.0.0.1    HTTP 160 GET /opensso/isAlive.jsp HTTP/1.0
  5 0.000050069    127.0.0.1 -> 127.0.0.1    TCP 66 http-alt > 41926 [ACK] Seq=1 Ack=95 Win=43776 Len=0 TSval=701753517 TSecr=701753517
  6 0.000595231    127.0.0.1 -> 127.0.0.1    HTTP 436 HTTP/1.1 200 OK  (text/html)
  7 0.000617025    127.0.0.1 -> 127.0.0.1    TCP 66 41926 > http-alt [ACK] Seq=95 Ack=371 Win=44800 Len=0 TSval=701753517 TSecr=701753517
  8 0.000733316    127.0.0.1 -> 127.0.0.1    HTTP 68 Continuation or non-HTTP traffic
  9 0.000748411    127.0.0.1 -> 127.0.0.1    TCP 66 http-alt > 41926 [FIN, ACK] Seq=371 Ack=95 Win=43776 Len=0 TSval=701753518 TSecr=701753517
 10 0.000770966    127.0.0.1 -> 127.0.0.1    TCP 54 http-alt > 41926 [RST] Seq=371 Win=0 Len=0
 11 0.000777847    127.0.0.1 -> 127.0.0.1    TCP 66 41926 > http-alt [ACK] Seq=97 Ack=372 Win=44800 Len=0 TSval=701753518 TSecr=701753518
 12 0.000784413    127.0.0.1 -> 127.0.0.1    TCP 66 41926 > http-alt [FIN, ACK] Seq=97 Ack=372 Win=44800 Len=0 TSval=701753518 TSecr=701753518
 13 0.000784484    127.0.0.1 -> 127.0.0.1    TCP 54 http-alt > 41926 [RST] Seq=372 Win=0 Len=0
 14 0.000794534    127.0.0.1 -> 127.0.0.1    TCP 54 http-alt > 41926 [RST] Seq=372 Win=0 Len=0
 15 1.412536403    127.0.0.1 -> 127.0.0.1    TCP 74 41928 > http-alt [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701754929 TSecr=0 WS=128
 16 1546080272.032933044    127.0.0.1 -> 127.0.0.1    TCP 74 http-alt > 41928 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=701754929 TSecr=701754929 WS=128
 17 1.412560984    127.0.0.1 -> 127.0.0.1    TCP 66 41928 > http-alt [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=701754929 TSecr=701754929
 18 1.412577999    127.0.0.1 -> 127.0.0.1    HTTP 160 GET /opensso/isAlive.jsp HTTP/1.0

This is strange because the help text says that the default snaplan is "65535", however it's clear from my testing that -s has additional side effects.

$tshark --help
tshark: invalid option -- '-'
TShark 1.10.14 (Git Rev Unknown from unknown)
...
Capture interface:
  -i <interface>           name or idx of interface (def: first non-loopback)
  -f <capture filter>      packet filter in libpcap filter syntax
  -s <snaplen>             packet snapshot length (def: 65535)

I raised a ticket with RedHat support and they have also confirmed with a virtualized RHEL 7.2 this issue is reproduceable, but it is fine on bare-metal.

I have tried this with Ubuntu 16.04 on AWS and it's fine there, so it only seems to affect RHEL/Centos.

I thought tcpdump and dumpcap/tshark used the same underlying capture library, libpcap?

Any ideas what's up with this?

Accepted Answer

Turns out it was an issue with the default capture buffer in RHEL 7, RedHat finally released a fix for it:

https://git.centos.org/commitdiff/rpms!wireshark.git/60db69006fe48d5744fdfa2a1b982dc552b5fd1b

It seems that the actual default snaplen is 262144, which would explain why it was running out of buffer too quickly and why changing it to 65536 helped.

Answer 2

This is not an issue of tshark, it's a side effect. TCPdump doesn't track TCP sequence numbers (meaning, it has no "TCP expert") like tshark does, so it won't complain about "TCP ACKed unseen segment". It has nothing to do with libpcap, but with the analyzing layers on top of it.

"ACKed unknown segment" means that the TCP expert has seen an acknowledge for a packet that it hasn't seen. This is usually a capture performance problem, meaning that your hardware can't capture the packets fast enough (for various possible reasons). Reducing the snaplength helps in this case of course, because you're writing less bytes to disk, allowing the capture to grab more frames in the same timespan.

If you need maximum capture performance, use dumpcap instead of tshark. See this blog post for more details why dumpcap is the better choice, and how to run it.