Hello, I have been working on a major issue in our enviroment since monday morning. We have a couple of iscsi san's (equallogic) those are connected to a stack of 4 dell switches and then i have a couple of dell server running vmware esx. (i ever have it from windows btw..) Now i generate with IOmeter 64KB read's and i get a latency of +/-270MS. So i opened cases with both vmware and dell. Dell is blaming the switches and vmware is blaming the san's. I would like to know where the latency is generated. I have no idea if that is even possible to find out? But i thought wireshark would be the right tool for that. I made a portgroup on esx with promiscuous mode enabled and installed a virtual machine with wireshark. Beside that i created a spawn port on the switch to mirror all traffic of the switchport where the server is connected. I mirrored the traffic to a physical machine with wireshark installed. Then i created a trace and while doing my IO test. After that i went to statistics and clicked conversations, used my conversation as a filter, went to statistics again, selected service response time and selected scsi. This showed me an average of 0.273 srt. So the same as my iometer results was telling me. I also did exactly the same with a san port (mirrored it to the physical machine.) So i see the same latency on the traces as in iomter. But how can i find out which device is generating this latency? I would be very greatfull if you can take the time to help me. Its a very serieus issue for us with big impact for our business.. Thanks... Regards asked 22 Sep '12, 03:35 Hdejongh edited 22 Sep '12, 03:35 |
Open up the trace in Wireshark and do "tcp.analysis.flags" And check out my Sharkfest 2012's "Deep Dive" presentation. If I had to guess, you are experiencing packet loss and lots of it. good luck.
if i first filter my conversation i have around 15 tcp acked unseen segments... nothing more.
I dont see any dropped frames in ESX or on the switch...
Huh...that's just the span port (mirror port) that's dropping the packet. Any chance you can post a snippet of your trace? in pcap not, txt output.