Diagnosing intermittent slowness with HTTPS connections

It turned out to be a duplex mismatch between our firewall's NIC and a router's NIC. The firewall NIC was set to autonegotiate, and the router's NIC was set to full duplex. After setting both to full duplex, the problem went away.

Because one of the key symptoms for diagnosing duplex mismatch is packet loss, it was important to take a capture at both ends of the conversation to see which packets were arriving and which weren't. After analyzing packet captures from both ends of the conversation, I found that the captured behavior matched very closely with Wikipedia's description of the TCP symptoms of duplex mismatch:

The lost packets force the TCP protocol to perform error recovery, but the initial (streamlined) recovery attempts fail because the retransmitted packets are lost in exactly the same way as the original packets. Eventually, the TCP transmission window becomes full and the TCP protocol refuses to transmit any further data until the previously-transmitted data is acknowledged. This, in turn, will quiesce the new traffic over the connection, leaving only the retransmissions and acknowledgments. Since the retransmission timer grows progressively longer between attempts, eventually a retransmission will occur when there is no reverse traffic on the connection, and the acknowledgment are finally received. This will restart the TCP traffic, which in turn immediately causes lost packets as streaming resumes.

As mentioned in the article, a key TCP symptom that indicates duplex mismatch is the TCP connection devolving into almost entirely delayed retransmissions and ACKs, resulting in a TCP connection that is "alive" and "active" but transmitting data at an incredibly slow rate.