Customer seeing lots of latency. Are these tcp.retransmits accurate?

Question

I'm seeing about 6% retransmits when I divide the number of packets displayed with display filter tcp.analysis.retransmission/ total number of captured packets. Am I doing this correctly? I also looked under expert and got the same number of retransmissions. I also was reading that for iSCSI traffic it is recommended to turn off Unicast Storm control. They mention this feature discarding packets when traffic reaches a threshold. What could I look for in the Cisco switch to see if this feature is discarding packets? I see 0 VLAN discard frames on all ports.

Answer 1

0

There is mainly three reasons for a re-transmit

Packet loss (either data or ACK)
The receiver can't handle the data (for whatever reason) and does not ACK it
The sender can't handle the ACK (for whatever reason) and then retransmits

To figure out what causes the retransmits in your case, the capture file would help to get a first glimpse.

If you think strom control is dropping packets, I suggest to check the counters on the switch (and also the logs, if logging is enabled).

Storm control status

show strom-control

this will show "Blocking" (Filter State) if strom control is dropping packets. HOWEVER, you will see this only during the time of blocking.

Switch# show storm-control
Interface Filter State Upper Lower Current
--------- ------------- ----------- ----------- ----------
Fa0/5 Blocking 1m bps 500k bps 2.08m bps

Storm control will block based on bandwith or packet rate. To get an idea if you get close to the numbers configured for strom control, I suggest to look at the IO Graphs during the timeframe you see the retransmits.

Statistics -> IO Graph -> Y Axis -> Unit: Packets/Tick
Statistics -> IO Graph -> Y Axis -> Unit: Bits/Tick

Port/Interface statistics counters

show interface <interface name>

Logs

show log

Regards
Kurt

answered 31 May '12, 02:17

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

edited 31 May '12, 02:32

I've uploaded full trace to ftp://customer:4[email protected]ftp.compellent.com/Gipp/20120530_1.pcap There'e also smaller saves of the same file first one is 10MB next one is 16MB ftp://customer:4[email protected]ftp.compellent.com/Gipp/esxserver3.cap ftp://customer:4[email protected]ftp.compellent.com/Gipp/esxserver4.cap Here's port counter info from Cisco switch for the port with most retrans

GigabitEthernet1/0/18 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 001c.0f72.1912 (bia 001c.0f72.1912)
  Description: Link to ESXs12 Fabric A
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:14, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1351633
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 9418000 bits/sec, 422 packets/sec
  5 minute output rate 1022000 bits/sec, 405 packets/sec
     3512522867 packets input, 11632586449906 bytes, 0 no buffer
     Received 75497 broadcasts (25625 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 25625 multicast, 283 pause input
     0 input packets with dribble condition detected
     5560805283 packets output, 9690588226086 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

(31 May '12, 11:26) gipper

oops. Can someone please move the comment to my answer. I picked the wrong option while I converted the question to an answer...

Done (grahamb) - Thanks (Kurt)

(31 May '12, 13:03) Kurt Knochner ♦

I've uploaded full trace to ftp://customer:4[email protected]ftp.compellent.com/Gipp/20120530_1.pcap

I'm pretty sure, only few people here (if any) are willing to wade through 1,4 GByte of your data. Are you sure you want to disclose that much internal data on the internet!?!

There'e also smaller saves of the same file first one is 10MB next one is 16MB

What did you find in those capture files?

Here's port counter info from Cisco switch for the port with most retrans

can you please format the output, so it's readable? Anyway, I don't see any problem that would explain dropped packets. Did you check the switch logs?

(31 May '12, 13:54) Kurt Knochner ♦

There are some "Lost Segment", "Retransmission", "DUP ACK", etc. That could be a sign for packet loss, however, it could also be a sign that wireshark did not see all packets (switch mirror problem) or it was not able to write all frames to disk (high packet rate and throughput).

What's the status of strom control?
Is it enabled?
If so, how is it configured?
Did you check the switch logs?

(31 May '12, 14:26) Kurt Knochner ♦