This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Hi There,

I recently run into the following issue and I hope that someone can shed some light on it.

Our monitoring server sends tens of HTTP requests to our Web server every minute to measure its performance. Occasionally, a couple of those requests fail to establish TCP connections. The captured packets show that the Web server sets win to zero in its syn+ack packet and resets the connection right after the completion of the three-way handshake.

I would love to know

1) What condition(s) would cause such an issue?

2) If the reason is that the kernel run out of some resource, is there a way to monitor its usage & upper bound?

All comments are welcome. Many thanks in advance!

Below packets were captured on the client (the monitoring server)

15:00:56.810144000 [Cli ---> Svr] SYN Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

15:00:56.874678000 [Svr ---> Cli] SYN, ACK Seq=0 Ack=1 Win=0 Len=0 MSS=1380

15:00:56.874740000 [Cli ---> Svr] ACK Seq=1 Ack=1 Win=64860 Len=0

15:00:56.939406000 [Svr ---> Cli] RST Seq=1 Win=0 Len=0

The CPU utilization of the Web server stays below 50% all the time. According to "netstat -s", TCP CurrEstab never exceeds 80 and there are 0 ListenOverflows, 0 TCPBacklogDrop, 0 PruneCalled, 0 TCPMemoryPressures, and 0 TWRecycled.

The output of "sysctl -a |grep net" is as follows:

net.netfilter.nf_log.0 = NONE

net.netfilter.nf_log.1 = NONE

net.netfilter.nf_log.2 = NONE

net.netfilter.nf_log.3 = NONE

net.netfilter.nf_log.4 = NONE

net.netfilter.nf_log.5 = NONE

net.netfilter.nf_log.6 = NONE

net.netfilter.nf_log.7 = NONE

net.netfilter.nf_log.8 = NONE

net.netfilter.nf_log.9 = NONE

net.netfilter.nf_log.10 = NONE

net.netfilter.nf_log.11 = NONE

net.netfilter.nf_log.12 = NONE

net.core.somaxconn = 4096

net.core.xfrm_aevent_etime = 10

net.core.xfrm_aevent_rseqth = 2

net.core.xfrm_larval_drop = 1

net.core.xfrm_acq_expires = 30

net.core.wmem_max = 4194304

net.core.rmem_max = 4194304

net.core.wmem_default = 124928

net.core.rmem_default = 124928

net.core.dev_weight = 64

net.core.netdev_max_backlog = 1000

net.core.message_cost = 5

net.core.message_burst = 10

net.core.optmem_max = 20480

net.core.rps_sock_flow_entries = 0

net.core.busy_poll = 0

net.core.busy_read = 0

net.core.netdev_budget = 300

net.core.warnings = 1

net.ipv4.route.gc_thresh = 524288

net.ipv4.route.max_size = 8388608

net.ipv4.route.gc_min_interval = 0

net.ipv4.route.gc_min_interval_ms = 500

net.ipv4.route.gc_timeout = 300

net.ipv4.route.gc_interval = 60

net.ipv4.route.redirect_load = 20

net.ipv4.route.redirect_number = 9

net.ipv4.route.redirect_silence = 20480

net.ipv4.route.error_cost = 1000

net.ipv4.route.error_burst = 5000

net.ipv4.route.gc_elasticity = 8

net.ipv4.route.mtu_expires = 600

net.ipv4.route.min_pmtu = 552

net.ipv4.route.min_adv_mss = 256

net.ipv4.route.secret_interval = 600

net.ipv4.neigh.default.mcast_solicit = 3

net.ipv4.neigh.default.ucast_solicit = 3

net.ipv4.neigh.default.app_solicit = 0

net.ipv4.neigh.default.retrans_time = 99

net.ipv4.neigh.default.base_reachable_time = 30

net.ipv4.neigh.default.delay_first_probe_time = 5

net.ipv4.neigh.default.gc_stale_time = 60

net.ipv4.neigh.default.unres_qlen = 3

net.ipv4.neigh.default.proxy_qlen = 64

net.ipv4.neigh.default.anycast_delay = 99

net.ipv4.neigh.default.proxy_delay = 79

net.ipv4.neigh.default.locktime = 99

net.ipv4.neigh.default.retrans_time_ms = 1000

net.ipv4.neigh.default.base_reachable_time_ms = 30000

net.ipv4.neigh.default.gc_interval = 30

net.ipv4.neigh.default.gc_thresh1 = 128

net.ipv4.neigh.default.gc_thresh2 = 512

net.ipv4.neigh.default.gc_thresh3 = 1024

net.ipv4.neigh.lo.mcast_solicit = 3

net.ipv4.neigh.lo.ucast_solicit = 3

net.ipv4.neigh.lo.app_solicit = 0

net.ipv4.neigh.lo.retrans_time = 99

net.ipv4.neigh.lo.base_reachable_time = 30

net.ipv4.neigh.lo.delay_first_probe_time = 5

net.ipv4.neigh.lo.gc_stale_time = 60

net.ipv4.neigh.lo.unres_qlen = 3

net.ipv4.neigh.lo.proxy_qlen = 64

net.ipv4.neigh.lo.anycast_delay = 99

net.ipv4.neigh.lo.proxy_delay = 79

net.ipv4.neigh.lo.locktime = 99

net.ipv4.neigh.lo.retrans_time_ms = 1000

net.ipv4.neigh.lo.base_reachable_time_ms = 30000

net.ipv4.neigh.eth0.mcast_solicit = 3

net.ipv4.neigh.eth0.ucast_solicit = 3

net.ipv4.neigh.eth0.app_solicit = 0

net.ipv4.neigh.eth0.retrans_time = 99

net.ipv4.neigh.eth0.base_reachable_time = 30

net.ipv4.neigh.eth0.delay_first_probe_time = 5

net.ipv4.neigh.eth0.gc_stale_time = 60

net.ipv4.neigh.eth0.unres_qlen = 3

net.ipv4.neigh.eth0.proxy_qlen = 64

net.ipv4.neigh.eth0.anycast_delay = 99

net.ipv4.neigh.eth0.proxy_delay = 79

net.ipv4.neigh.eth0.locktime = 99

net.ipv4.neigh.eth0.retrans_time_ms = 1000

net.ipv4.neigh.eth0.base_reachable_time_ms = 30000

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_sack = 1

net.ipv4.tcp_retrans_collapse = 1

net.ipv4.ip_default_ttl = 64

net.ipv4.ip_no_pmtu_disc = 0

net.ipv4.ip_nonlocal_bind = 0

net.ipv4.tcp_syn_retries = 5

net.ipv4.tcp_synack_retries = 5

net.ipv4.tcp_max_orphans = 262144

net.ipv4.tcp_max_tw_buckets = 1440000

net.ipv4.ip_dynaddr = 0

net.ipv4.tcp_keepalive_time = 1800

net.ipv4.tcp_keepalive_probes = 9

net.ipv4.tcp_keepalive_intvl = 75

net.ipv4.tcp_retries1 = 3

net.ipv4.tcp_retries2 = 15

net.ipv4.tcp_fin_timeout = 15

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_recycle = 0

net.ipv4.tcp_abort_on_overflow = 0

net.ipv4.tcp_stdurg = 0

net.ipv4.tcp_rfc1337 = 0

net.ipv4.tcp_max_syn_backlog = 4096

net.ipv4.ip_local_port_range = 16384 61000

net.ipv4.ip_local_reserved_ports =

net.ipv4.igmp_max_memberships = 20

net.ipv4.igmp_max_msf = 10

net.ipv4.inet_peer_threshold = 65664

net.ipv4.inet_peer_minttl = 120

net.ipv4.inet_peer_maxttl = 600

net.ipv4.inet_peer_gc_mintime = 10

net.ipv4.inet_peer_gc_maxtime = 120

net.ipv4.tcp_orphan_retries = 0

net.ipv4.tcp_fack = 1

net.ipv4.tcp_reordering = 3

net.ipv4.tcp_ecn = 2

net.ipv4.tcp_dsack = 1

net.ipv4.tcp_mem = 1141728 1522304 2283456

net.ipv4.tcp_wmem = 4096 16384 4194304

net.ipv4.tcp_rmem = 4096 87380 4194304

net.ipv4.tcp_app_win = 31

net.ipv4.tcp_adv_win_scale = 2

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_frto = 2

net.ipv4.tcp_frto_response = 0

net.ipv4.tcp_low_latency = 0

net.ipv4.tcp_no_metrics_save = 0

net.ipv4.tcp_moderate_rcvbuf = 1

net.ipv4.tcp_tso_win_divisor = 3

net.ipv4.tcp_congestion_control = cubic

net.ipv4.tcp_abc = 0

net.ipv4.tcp_mtu_probing = 0

net.ipv4.tcp_base_mss = 512

net.ipv4.tcp_workaround_signed_windows = 0

net.ipv4.tcp_challenge_ack_limit = 100

net.ipv4.tcp_limit_output_bytes = 131072

net.ipv4.tcp_dma_copybreak = 4096

net.ipv4.tcp_slow_start_after_idle = 1

net.ipv4.cipso_cache_enable = 1

net.ipv4.cipso_cache_bucket_size = 10

net.ipv4.cipso_rbm_optfmt = 0

net.ipv4.cipso_rbm_strictvalid = 1

net.ipv4.tcp_available_congestion_control = cubic reno

net.ipv4.tcp_allowed_congestion_control = cubic reno

net.ipv4.tcp_max_ssthresh = 0

net.ipv4.tcp_thin_linear_timeouts = 0

net.ipv4.tcp_thin_dupack = 0

net.ipv4.tcp_min_tso_segs = 2

net.ipv4.udp_mem = 1141728 1522304 2283456

net.ipv4.udp_rmem_min = 4096

net.ipv4.udp_wmem_min = 4096

net.ipv4.conf.all.forwarding = 0

net.ipv4.conf.all.mc_forwarding = 0

net.ipv4.conf.all.accept_redirects = 0

net.ipv4.conf.all.secure_redirects = 0

net.ipv4.conf.all.shared_media = 1

net.ipv4.conf.all.rp_filter = 0

net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.all.accept_source_route = 0

net.ipv4.conf.all.src_valid_mark = 0

net.ipv4.conf.all.proxy_arp = 0

net.ipv4.conf.all.medium_id = 0

net.ipv4.conf.all.bootp_relay = 0

net.ipv4.conf.all.log_martians = 0

net.ipv4.conf.all.tag = 0

net.ipv4.conf.all.arp_filter = 0

net.ipv4.conf.all.arp_announce = 0

net.ipv4.conf.all.arp_ignore = 0

net.ipv4.conf.all.arp_accept = 0

net.ipv4.conf.all.arp_notify = 0

net.ipv4.conf.all.proxy_arp_pvlan = 0

net.ipv4.conf.all.disable_xfrm = 0

net.ipv4.conf.all.disable_policy = 0

net.ipv4.conf.all.force_igmp_version = 0

net.ipv4.conf.all.promote_secondaries = 0

net.ipv4.conf.all.accept_local = 0

net.ipv4.conf.all.route_localnet = 0

net.ipv4.conf.default.forwarding = 0

net.ipv4.conf.default.mc_forwarding = 0

net.ipv4.conf.default.accept_redirects = 0

net.ipv4.conf.default.secure_redirects = 0

net.ipv4.conf.default.shared_media = 1

net.ipv4.conf.default.rp_filter = 0

net.ipv4.conf.default.send_redirects = 0

net.ipv4.conf.default.accept_source_route = 0

net.ipv4.conf.default.src_valid_mark = 0

net.ipv4.conf.default.proxy_arp = 0

net.ipv4.conf.default.medium_id = 0

net.ipv4.conf.default.bootp_relay = 0

net.ipv4.conf.default.log_martians = 0

net.ipv4.conf.default.tag = 0

net.ipv4.conf.default.arp_filter = 0

net.ipv4.conf.default.arp_announce = 0

net.ipv4.conf.default.arp_ignore = 0

net.ipv4.conf.default.arp_accept = 0

net.ipv4.conf.default.arp_notify = 0

net.ipv4.conf.default.proxy_arp_pvlan = 0

net.ipv4.conf.default.disable_xfrm = 0

net.ipv4.conf.default.disable_policy = 0

net.ipv4.conf.default.force_igmp_version = 0

net.ipv4.conf.default.promote_secondaries = 0

net.ipv4.conf.default.accept_local = 0

net.ipv4.conf.default.route_localnet = 0

net.ipv4.conf.lo.forwarding = 0

net.ipv4.conf.lo.mc_forwarding = 0

net.ipv4.conf.lo.accept_redirects = 0

net.ipv4.conf.lo.secure_redirects = 1

net.ipv4.conf.lo.shared_media = 1

net.ipv4.conf.lo.rp_filter = 1

net.ipv4.conf.lo.send_redirects = 1

net.ipv4.conf.lo.accept_source_route = 0

net.ipv4.conf.lo.src_valid_mark = 0

net.ipv4.conf.lo.proxy_arp = 0

net.ipv4.conf.lo.medium_id = 0

net.ipv4.conf.lo.bootp_relay = 0

net.ipv4.conf.lo.log_martians = 0

net.ipv4.conf.lo.tag = 0

net.ipv4.conf.lo.arp_filter = 0

net.ipv4.conf.lo.arp_announce = 0

net.ipv4.conf.lo.arp_ignore = 0

net.ipv4.conf.lo.arp_accept = 0

net.ipv4.conf.lo.arp_notify = 0

net.ipv4.conf.lo.proxy_arp_pvlan = 0

net.ipv4.conf.lo.disable_xfrm = 1

net.ipv4.conf.lo.disable_policy = 1

net.ipv4.conf.lo.force_igmp_version = 0

net.ipv4.conf.lo.promote_secondaries = 0

net.ipv4.conf.lo.accept_local = 0

net.ipv4.conf.lo.route_localnet = 0

net.ipv4.conf.eth0.forwarding = 0

net.ipv4.conf.eth0.mc_forwarding = 0

net.ipv4.conf.eth0.accept_redirects = 0

net.ipv4.conf.eth0.secure_redirects = 0

net.ipv4.conf.eth0.shared_media = 1

net.ipv4.conf.eth0.rp_filter = 0

net.ipv4.conf.eth0.send_redirects = 0

net.ipv4.conf.eth0.accept_source_route = 0

net.ipv4.conf.eth0.src_valid_mark = 0

net.ipv4.conf.eth0.proxy_arp = 0

net.ipv4.conf.eth0.medium_id = 0

net.ipv4.conf.eth0.bootp_relay = 0

net.ipv4.conf.eth0.log_martians = 0

net.ipv4.conf.eth0.tag = 0

net.ipv4.conf.eth0.arp_filter = 0

net.ipv4.conf.eth0.arp_announce = 0

net.ipv4.conf.eth0.arp_ignore = 0

net.ipv4.conf.eth0.arp_accept = 0

net.ipv4.conf.eth0.arp_notify = 0

net.ipv4.conf.eth0.proxy_arp_pvlan = 0

net.ipv4.conf.eth0.disable_xfrm = 0

net.ipv4.conf.eth0.disable_policy = 0

net.ipv4.conf.eth0.force_igmp_version = 0

net.ipv4.conf.eth0.promote_secondaries = 0

net.ipv4.conf.eth0.accept_local = 0

net.ipv4.conf.eth0.route_localnet = 0

net.ipv4.ip_forward = 0

net.ipv4.xfrm4_gc_thresh = 4194304

net.ipv4.ipfrag_high_thresh = 4194304

net.ipv4.ipfrag_low_thresh = 3145728

net.ipv4.ipfrag_time = 30

net.ipv4.icmp_echo_ignore_all = 0

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.icmp_ignore_bogus_error_responses = 1

net.ipv4.icmp_errors_use_inbound_ifaddr = 0

net.ipv4.icmp_ratelimit = 1000

net.ipv4.icmp_ratemask = 6168

net.ipv4.rt_cache_rebuild_count = 4

net.ipv4.ping_group_range = 1 0

net.ipv4.ipfrag_secret_interval = 600

net.ipv4.ipfrag_max_dist = 64

net.unix.max_dgram_qlen = 10

asked 22 Sep '15, 12:38

kemaru's gravatar image

kemaru
6112
accept rate: 0%


Usually if the kernel hits some resource issue, you would see unanswered SYNs (though that is not always true).

At first glance, it may be that syn cookies are kicking in and the server is not getting a valid response in that ACK packet.

In your netstat -s output, there should be a line item for "invalid SYN cookies received." You may want to check that as the problem occurs to see if that is what you are hitting. If you suspect that, you may want to grab another capture and turn of "Relative Sequence Numbers" to aid in analysis of the client response.

If you suspect that invalid syn cookie responses is the issue, another quick way to check for that issue would be to set net.ipv4.tcp_syncookies = 0. (I don't recall, off-hand, if that requires a system restart)

$0.02

permanent link

answered 24 Sep '15, 15:00

Qwert's gravatar image

Qwert
16226
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×81
×19
×10

question asked: 22 Sep '15, 12:38

question was seen: 2,350 times

last updated: 24 Sep '15, 15:00

p​o​w​e​r​e​d by O​S​Q​A