I recently started to see a burst of HTTP '400' response codes in our Apache logs running on RHEL5 servers. According to our logs we are serving up a '400' every few seconds. When I manually cut/paste the URL receiving a '400' in a browser it works without any problems. What's odd is that Apache is showing those responses taking 1 second to complete. To debug I ran tshark on one of the RHEL5 web servers. I didn't enter in any abnormal packet capture. Here was my syntax: tshark -i bond0 -f "host port 80" -w ~/test.cap I am unable to find the '400' errors that I see in the Apache logs in the capture. What is odd is there are '400' errors for queries that I am unable to find in the Apache logs, for another URI entirely. The number of '400' errors I see in the capture for URI X and the number I see in the Apache logs for URI Y do not line up directly. We also do not have any sort of rewrite rules or proxies in place that would account for a URL being altered. I ran a wide open packet capture (i.e. not using 'host port 80') on our border router and see the same behavior. What is odd is that I do not see the web server's '400' error ever leaving our network. I have run subsequent captures wide open, capturing all traffic just in case something was out of whack with the filter. I also ran captures against every interface on the servers in case something might be wrong or misconfigured with the bonded interface but that isn't the case. I have always been able to directly correlate entries in wireshark to log entries. This is a first for me. Has anyone run into this problem before? All servers are 64 bit running RHEL5, wireshark version wireshark-1.0.15-1.el5_5.1. I also analyzed the capture on a 64 bit windows laptop running the 64 bit 1.4.1 wireshark version. asked 10 Nov '10, 08:46 devrick0 |
3 Answers:
That capture filter isn't valide - "host port 80." You should have seen an error message though. Can you open the "wide open" traces you captured and apply a display filter for If the 400 errors are being sent and Wireshark can see them, so should you. answered 11 Nov '10, 17:10 lchappell ♦ |
You can use justniffer for logging http traffic answered 15 Sep '11, 05:19 Augustyn |
We had similar symptoms and the problem was that 2 HTTP requests seemed to be intermingled within the same TCP communication left open because of the keep-alive directive. In our case a proxy is talking to the apache, not the browsers directly. Because the queries are intermingled our proxy only detects the 400 response on one of the requests, I assume the other is considered to have timed out. Apache actually doesn't log anything at all in the access_log. answered 13 Sep '12, 10:57 adamloopkin |
Yes, my mistake, it was "tcp port 80" when I tried the filter.
I was able to have the inbound requests that were generating the '400' errors corrected so those are gone. I checked the load balancer and confirmed the errors are not leaving the server. I checked all the interfaces again and I don't see the '400' errors. My next thought is to look at wireshark and see if I can track down inbound requests that didn't receive a response from the web server.
I realize that I am lacking in wireshark skills so I ordered "Wireshark Network Analysis" yesterday on amazon.
One nice feature of Wireshark is Statistics > HTTP > Packet Counter that will list the response types by number group (1xx, 2xx, 3xx, etc.). Check that out as well.