We have a public facing website that runs on a platform with Apache Web Server + Application Firewall (Imperva) + App Server (Sybase EAServer). This is in production and it's been running fine. But, recently when a developer made some changes, the pages started loading slo...w. First we blamed it on new Bootstrap libraries used. To make matters more puzzling, this page and images loaded fine on cell phones. (Later we realized, it may have to do with file size.. Does cell phones received reduced image size?). I may be wrong, but the slowness seems to be more severe on Windows clients. We were able to reproduce in test and after some testing, I narrowed it down an image and further testing revealed, the slowness was due to any file (JPG, PNG, CSS, JS) above 20 - 30K in size. At this point, I tested the same file with SFTP with the same results. So, the nature of the problem turned out to be, as simple as, Downloading files > 20k was slow and Uploads were fine. One particular image file, 79k in size, would load half way through in browser and stall for up to 2 minutes to complete. Another (71k) would blur for 2 minutes to develop. I put Wireshark on the client side (I don't have rights on server) to see there were a lot of packet Retransmissions and/or DUP ACK in the trace. Few additional tests (excluding Imperva altogether) revealed imperva server is adding that delay. We checled, that there is no Firewall rule causing this, as the file is finally let through. Webserver on Debian Linux, Imperva appliance uses Centos and App server on Win 2k8. I am currently looking at solving it by tuning TCP settings, but I am not all that familiar with it. I can work with the admins to change the settings as needed. I am seeking any advice or tips from the experts here. Thanks in advance. (So far, I've tried, TCP window scaling, sack etc and nothing seem to help). If it will be of any help, I am attaching here, a flow graph export from wireshark, for the download of an image in browser (HTTPS) asked 30 Dec '16, 16:42  kanishqa edited 30 Dec '16, 17:03 |
One Answer:
Hello Kanishqa, and welcome to ask.wireshark.org A first glance reveals a few retransmissions, which most certainly cause delay. The connection(s) start with a round trip time (RTT) of 2 msec. A bit later we see packets marked as "Retransmission" coming in with a delay of 400 msec or more. A good start could be switch counters and port status to identify full duplex / half duplex mismatch (that is, if 100 MBit links are involved). To identify the exact source of the packet loss it might be useful to capture packets either using SPAN ports, maybe even taps. It would be most useful, if you could provide a tracefile. If you worry about private data (even, if the traffic is encrypted), you can clean with the TraceWrangler, which is available for free at https://www.tracewrangler.com answered 31 Dec '16, 09:26  packethunter |
@PacketHunter,
Thanks for taking the time and for the detailed response. That's the best answer I have gotten so far. For the last week or so, we have been running around, blaming everything under the sun :). Thankfully, we have been able to narrow down to download only. Is that what led you to think, it's a duplex issue?
Our network team said, switches were ok, but I will show them your answer next week. Hopefully, they will be able to figure out. To capture packets like you mentioned, can Wireshark do it?
I am a developer who happened to be in this spot to find a solution for the problem that started as a programming issue :). But, I am eager to learn. Is there any link or diagram you can share, to get me started on this topic, so I can be ready to discuss with the network team?
I will post the trace file and any other findings shortly.
Happy New Year!
Sam
Hi Sam / Kanishqa
A good start is the web site https://blog.packet-foo.com. Check out the network capture playbook. By the way, the FDX/HDX issue (that's full duplex / half duplex) would only appear on 10 or 100 MBit Ethernet. Make sure to check both the settings on the end-systems and servers.
Happy new year and good hunting.