Understanding TCP/RST from AD - configured as Directory URL servers in RSA Auth Mgr

Question

Hi there. Am looking to see if the community has had this experience, and if not, maybe some feedback or suggestions:

Environment: RSA Auth Manager 8.1, (1) Primary, (3) Replicas

The Primary has (2) Directory URL servers configured - both Windows 2008 AD VM's

Issue: Whenever the Primary Directory URL server is offline, the Primary RSA instance cannot successfully connect to the 2nd AD/DC. I've also tried using the IP address instead of FQDN, that makes no difference. I've also tried swapping the order of the Directory URL servers, that makes no difference either.

The same results occur, A.M. can always talk to the Primary Directory URL server, no matter which one is configured, as soon as the Primary is turned off, A.M. cant establish successful connection with the Failover URL server, demonstrated by what looks like TCP/RST coming from the AD/DC

Additionally, the FQDN in use is not configured as a DNS Round Robin. Windows Firewall is turned off on both AD/DC's.. and these are vSphere VM's by the way.

from the A.M. Primary I can successfully run "openssl s_client -connect 10.100.0.91:389" to that Failover Directory URL server, and the connection succeeds

I'm providing 4 lines of packet analysis from TCPDump from the A.M. primary..

the A.M. Primary is 10.100.2.167 the Failover URL server is 10.100.0.91

1778 17:17:11.289622    10.100.0.91 10.100.2.167    TCP 74  389→56923 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 TSval=664262070 TSecr=64906072
1779  17:17:11.289669   10.100.2.167    10.100.0.91 TCP 66  56923→389 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=64906072 TSecr=664262070
1780 17:17:11.290053    10.100.2.167    10.100.0.91 TCP 205 56923→389 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=139 TSval=64906072 TSecr=664262070
1781 17:17:11.290351    10.100.0.91 10.100.2.167    TCP 60  389→56923 [RST, ACK] Seq=1 Ack=140 Win=0 Len=0

the sample above will also occur if I move 10.100.0.91 to be the Primary Directory URL server, and then when the same tests are ran, the exact same behavior occurs on the other AD/DC, which was previously able to connect when it was configured as the Primary Directory URL server

If anyone has any ideas, it would be much appreciated. Have spent a LOT of time on this so far, and am running out of ideas. Was thinking next approach is to run Wireshark on the AD/DC side and compare, perhaps something else is happening that we’re missing from the TCPdump on the A.M. appliance

Answer 1

From these 4 packets, one can only assume the following:

The AD server is somehow not happy with the way the LDAP session is being set up. What is the difference in the SSL ClientHello when it is configured as a primary and when it is configured as a secondary? Are you using SNI?
Some device in between the MA and the Primary AD is not happy with the way the LDAP session is being set up. If so, you might see a different IP.TTL for the RST packet.

BTW, why are you using LDAP over SSL on port 389 and not on 636?

Are you able to share the faulty and good session setup on www.cloudshark.org so that we can have a look at the difference? Only the first 6 packets of each stream will suffice for now.