Hi There, I recently run into the following issue and I hope that someone can shed some light on it. Our monitoring server sends tens of HTTP requests to our Web server every minute to measure its performance. Occasionally, a couple of those requests fail to establish TCP connections. The captured packets show that the Web server sets win to zero in its syn+ack packet and resets the connection right after the completion of the three-way handshake. I would love to know 1) What condition(s) would cause such an issue? 2) If the reason is that the kernel run out of some resource, is there a way to monitor its usage & upper bound? All comments are welcome. Many thanks in advance! Below packets were captured on the client (the monitoring server) 15:00:56.810144000 [Cli ---> Svr] SYN Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 15:00:56.874678000 [Svr ---> Cli] SYN, ACK Seq=0 Ack=1 Win=0 Len=0 MSS=1380 15:00:56.874740000 [Cli ---> Svr] ACK Seq=1 Ack=1 Win=64860 Len=0 15:00:56.939406000 [Svr ---> Cli] RST Seq=1 Win=0 Len=0 The CPU utilization of the Web server stays below 50% all the time. According to "netstat -s", TCP CurrEstab never exceeds 80 and there are 0 ListenOverflows, 0 TCPBacklogDrop, 0 PruneCalled, 0 TCPMemoryPressures, and 0 TWRecycled. The output of "sysctl -a |grep net" is as follows: net.netfilter.nf_log.0 = NONE net.netfilter.nf_log.1 = NONE net.netfilter.nf_log.2 = NONE net.netfilter.nf_log.3 = NONE net.netfilter.nf_log.4 = NONE net.netfilter.nf_log.5 = NONE net.netfilter.nf_log.6 = NONE net.netfilter.nf_log.7 = NONE net.netfilter.nf_log.8 = NONE net.netfilter.nf_log.9 = NONE net.netfilter.nf_log.10 = NONE net.netfilter.nf_log.11 = NONE net.netfilter.nf_log.12 = NONE net.core.somaxconn = 4096 net.core.xfrm_aevent_etime = 10 net.core.xfrm_aevent_rseqth = 2 net.core.xfrm_larval_drop = 1 net.core.xfrm_acq_expires = 30 net.core.wmem_max = 4194304 net.core.rmem_max = 4194304 net.core.wmem_default = 124928 net.core.rmem_default = 124928 net.core.dev_weight = 64 net.core.netdev_max_backlog = 1000 net.core.message_cost = 5 net.core.message_burst = 10 net.core.optmem_max = 20480 net.core.rps_sock_flow_entries = 0 net.core.busy_poll = 0 net.core.busy_read = 0 net.core.netdev_budget = 300 net.core.warnings = 1 net.ipv4.route.gc_thresh = 524288 net.ipv4.route.max_size = 8388608 net.ipv4.route.gc_min_interval = 0 net.ipv4.route.gc_min_interval_ms = 500 net.ipv4.route.gc_timeout = 300 net.ipv4.route.gc_interval = 60 net.ipv4.route.redirect_load = 20 net.ipv4.route.redirect_number = 9 net.ipv4.route.redirect_silence = 20480 net.ipv4.route.error_cost = 1000 net.ipv4.route.error_burst = 5000 net.ipv4.route.gc_elasticity = 8 net.ipv4.route.mtu_expires = 600 net.ipv4.route.min_pmtu = 552 net.ipv4.route.min_adv_mss = 256 net.ipv4.route.secret_interval = 600 net.ipv4.neigh.default.mcast_solicit = 3 net.ipv4.neigh.default.ucast_solicit = 3 net.ipv4.neigh.default.app_solicit = 0 net.ipv4.neigh.default.retrans_time = 99 net.ipv4.neigh.default.base_reachable_time = 30 net.ipv4.neigh.default.delay_first_probe_time = 5 net.ipv4.neigh.default.gc_stale_time = 60 net.ipv4.neigh.default.unres_qlen = 3 net.ipv4.neigh.default.proxy_qlen = 64 net.ipv4.neigh.default.anycast_delay = 99 net.ipv4.neigh.default.proxy_delay = 79 net.ipv4.neigh.default.locktime = 99 net.ipv4.neigh.default.retrans_time_ms = 1000 net.ipv4.neigh.default.base_reachable_time_ms = 30000 net.ipv4.neigh.default.gc_interval = 30 net.ipv4.neigh.default.gc_thresh1 = 128 net.ipv4.neigh.default.gc_thresh2 = 512 net.ipv4.neigh.default.gc_thresh3 = 1024 net.ipv4.neigh.lo.mcast_solicit = 3 net.ipv4.neigh.lo.ucast_solicit = 3 net.ipv4.neigh.lo.app_solicit = 0 net.ipv4.neigh.lo.retrans_time = 99 net.ipv4.neigh.lo.base_reachable_time = 30 net.ipv4.neigh.lo.delay_first_probe_time = 5 net.ipv4.neigh.lo.gc_stale_time = 60 net.ipv4.neigh.lo.unres_qlen = 3 net.ipv4.neigh.lo.proxy_qlen = 64 net.ipv4.neigh.lo.anycast_delay = 99 net.ipv4.neigh.lo.proxy_delay = 79 net.ipv4.neigh.lo.locktime = 99 net.ipv4.neigh.lo.retrans_time_ms = 1000 net.ipv4.neigh.lo.base_reachable_time_ms = 30000 net.ipv4.neigh.eth0.mcast_solicit = 3 net.ipv4.neigh.eth0.ucast_solicit = 3 net.ipv4.neigh.eth0.app_solicit = 0 net.ipv4.neigh.eth0.retrans_time = 99 net.ipv4.neigh.eth0.base_reachable_time = 30 net.ipv4.neigh.eth0.delay_first_probe_time = 5 net.ipv4.neigh.eth0.gc_stale_time = 60 net.ipv4.neigh.eth0.unres_qlen = 3 net.ipv4.neigh.eth0.proxy_qlen = 64 net.ipv4.neigh.eth0.anycast_delay = 99 net.ipv4.neigh.eth0.proxy_delay = 79 net.ipv4.neigh.eth0.locktime = 99 net.ipv4.neigh.eth0.retrans_time_ms = 1000 net.ipv4.neigh.eth0.base_reachable_time_ms = 30000 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_retrans_collapse = 1 net.ipv4.ip_default_ttl = 64 net.ipv4.ip_no_pmtu_disc = 0 net.ipv4.ip_nonlocal_bind = 0 net.ipv4.tcp_syn_retries = 5 net.ipv4.tcp_synack_retries = 5 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.ip_dynaddr = 0 net.ipv4.tcp_keepalive_time = 1800 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_retries1 = 3 net.ipv4.tcp_retries2 = 15 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_abort_on_overflow = 0 net.ipv4.tcp_stdurg = 0 net.ipv4.tcp_rfc1337 = 0 net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.ip_local_port_range = 16384 61000 net.ipv4.ip_local_reserved_ports = net.ipv4.igmp_max_memberships = 20 net.ipv4.igmp_max_msf = 10 net.ipv4.inet_peer_threshold = 65664 net.ipv4.inet_peer_minttl = 120 net.ipv4.inet_peer_maxttl = 600 net.ipv4.inet_peer_gc_mintime = 10 net.ipv4.inet_peer_gc_maxtime = 120 net.ipv4.tcp_orphan_retries = 0 net.ipv4.tcp_fack = 1 net.ipv4.tcp_reordering = 3 net.ipv4.tcp_ecn = 2 net.ipv4.tcp_dsack = 1 net.ipv4.tcp_mem = 1141728 1522304 2283456 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_app_win = 31 net.ipv4.tcp_adv_win_scale = 2 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_frto = 2 net.ipv4.tcp_frto_response = 0 net.ipv4.tcp_low_latency = 0 net.ipv4.tcp_no_metrics_save = 0 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_tso_win_divisor = 3 net.ipv4.tcp_congestion_control = cubic net.ipv4.tcp_abc = 0 net.ipv4.tcp_mtu_probing = 0 net.ipv4.tcp_base_mss = 512 net.ipv4.tcp_workaround_signed_windows = 0 net.ipv4.tcp_challenge_ack_limit = 100 net.ipv4.tcp_limit_output_bytes = 131072 net.ipv4.tcp_dma_copybreak = 4096 net.ipv4.tcp_slow_start_after_idle = 1 net.ipv4.cipso_cache_enable = 1 net.ipv4.cipso_cache_bucket_size = 10 net.ipv4.cipso_rbm_optfmt = 0 net.ipv4.cipso_rbm_strictvalid = 1 net.ipv4.tcp_available_congestion_control = cubic reno net.ipv4.tcp_allowed_congestion_control = cubic reno net.ipv4.tcp_max_ssthresh = 0 net.ipv4.tcp_thin_linear_timeouts = 0 net.ipv4.tcp_thin_dupack = 0 net.ipv4.tcp_min_tso_segs = 2 net.ipv4.udp_mem = 1141728 1522304 2283456 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 net.ipv4.conf.all.forwarding = 0 net.ipv4.conf.all.mc_forwarding = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.all.shared_media = 1 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.src_valid_mark = 0 net.ipv4.conf.all.proxy_arp = 0 net.ipv4.conf.all.medium_id = 0 net.ipv4.conf.all.bootp_relay = 0 net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.all.tag = 0 net.ipv4.conf.all.arp_filter = 0 net.ipv4.conf.all.arp_announce = 0 net.ipv4.conf.all.arp_ignore = 0 net.ipv4.conf.all.arp_accept = 0 net.ipv4.conf.all.arp_notify = 0 net.ipv4.conf.all.proxy_arp_pvlan = 0 net.ipv4.conf.all.disable_xfrm = 0 net.ipv4.conf.all.disable_policy = 0 net.ipv4.conf.all.force_igmp_version = 0 net.ipv4.conf.all.promote_secondaries = 0 net.ipv4.conf.all.accept_local = 0 net.ipv4.conf.all.route_localnet = 0 net.ipv4.conf.default.forwarding = 0 net.ipv4.conf.default.mc_forwarding = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.default.secure_redirects = 0 net.ipv4.conf.default.shared_media = 1 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.src_valid_mark = 0 net.ipv4.conf.default.proxy_arp = 0 net.ipv4.conf.default.medium_id = 0 net.ipv4.conf.default.bootp_relay = 0 net.ipv4.conf.default.log_martians = 0 net.ipv4.conf.default.tag = 0 net.ipv4.conf.default.arp_filter = 0 net.ipv4.conf.default.arp_announce = 0 net.ipv4.conf.default.arp_ignore = 0 net.ipv4.conf.default.arp_accept = 0 net.ipv4.conf.default.arp_notify = 0 net.ipv4.conf.default.proxy_arp_pvlan = 0 net.ipv4.conf.default.disable_xfrm = 0 net.ipv4.conf.default.disable_policy = 0 net.ipv4.conf.default.force_igmp_version = 0 net.ipv4.conf.default.promote_secondaries = 0 net.ipv4.conf.default.accept_local = 0 net.ipv4.conf.default.route_localnet = 0 net.ipv4.conf.lo.forwarding = 0 net.ipv4.conf.lo.mc_forwarding = 0 net.ipv4.conf.lo.accept_redirects = 0 net.ipv4.conf.lo.secure_redirects = 1 net.ipv4.conf.lo.shared_media = 1 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.lo.send_redirects = 1 net.ipv4.conf.lo.accept_source_route = 0 net.ipv4.conf.lo.src_valid_mark = 0 net.ipv4.conf.lo.proxy_arp = 0 net.ipv4.conf.lo.medium_id = 0 net.ipv4.conf.lo.bootp_relay = 0 net.ipv4.conf.lo.log_martians = 0 net.ipv4.conf.lo.tag = 0 net.ipv4.conf.lo.arp_filter = 0 net.ipv4.conf.lo.arp_announce = 0 net.ipv4.conf.lo.arp_ignore = 0 net.ipv4.conf.lo.arp_accept = 0 net.ipv4.conf.lo.arp_notify = 0 net.ipv4.conf.lo.proxy_arp_pvlan = 0 net.ipv4.conf.lo.disable_xfrm = 1 net.ipv4.conf.lo.disable_policy = 1 net.ipv4.conf.lo.force_igmp_version = 0 net.ipv4.conf.lo.promote_secondaries = 0 net.ipv4.conf.lo.accept_local = 0 net.ipv4.conf.lo.route_localnet = 0 net.ipv4.conf.eth0.forwarding = 0 net.ipv4.conf.eth0.mc_forwarding = 0 net.ipv4.conf.eth0.accept_redirects = 0 net.ipv4.conf.eth0.secure_redirects = 0 net.ipv4.conf.eth0.shared_media = 1 net.ipv4.conf.eth0.rp_filter = 0 net.ipv4.conf.eth0.send_redirects = 0 net.ipv4.conf.eth0.accept_source_route = 0 net.ipv4.conf.eth0.src_valid_mark = 0 net.ipv4.conf.eth0.proxy_arp = 0 net.ipv4.conf.eth0.medium_id = 0 net.ipv4.conf.eth0.bootp_relay = 0 net.ipv4.conf.eth0.log_martians = 0 net.ipv4.conf.eth0.tag = 0 net.ipv4.conf.eth0.arp_filter = 0 net.ipv4.conf.eth0.arp_announce = 0 net.ipv4.conf.eth0.arp_ignore = 0 net.ipv4.conf.eth0.arp_accept = 0 net.ipv4.conf.eth0.arp_notify = 0 net.ipv4.conf.eth0.proxy_arp_pvlan = 0 net.ipv4.conf.eth0.disable_xfrm = 0 net.ipv4.conf.eth0.disable_policy = 0 net.ipv4.conf.eth0.force_igmp_version = 0 net.ipv4.conf.eth0.promote_secondaries = 0 net.ipv4.conf.eth0.accept_local = 0 net.ipv4.conf.eth0.route_localnet = 0 net.ipv4.ip_forward = 0 net.ipv4.xfrm4_gc_thresh = 4194304 net.ipv4.ipfrag_high_thresh = 4194304 net.ipv4.ipfrag_low_thresh = 3145728 net.ipv4.ipfrag_time = 30 net.ipv4.icmp_echo_ignore_all = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.icmp_errors_use_inbound_ifaddr = 0 net.ipv4.icmp_ratelimit = 1000 net.ipv4.icmp_ratemask = 6168 net.ipv4.rt_cache_rebuild_count = 4 net.ipv4.ping_group_range = 1 0 net.ipv4.ipfrag_secret_interval = 600 net.ipv4.ipfrag_max_dist = 64 net.unix.max_dgram_qlen = 10 asked 22 Sep '15, 12:38  kemaru |
One Answer:
Usually if the kernel hits some resource issue, you would see unanswered SYNs (though that is not always true). At first glance, it may be that syn cookies are kicking in and the server is not getting a valid response in that ACK packet. In your netstat -s output, there should be a line item for "invalid SYN cookies received." You may want to check that as the problem occurs to see if that is what you are hitting. If you suspect that, you may want to grab another capture and turn of "Relative Sequence Numbers" to aid in analysis of the client response. If you suspect that invalid syn cookie responses is the issue, another quick way to check for that issue would be to set net.ipv4.tcp_syncookies = 0. (I don't recall, off-hand, if that requires a system restart) $0.02 answered 24 Sep '15, 15:00  Qwert |
