LTLnetworker | IT hálózatok, biztonság, Cisco

               IT networks, security, Cisco

Unreachable network behind TMG

Posted by ltlnetworker on February 3, 2013


I was asked to troubleshoot Active Directory DC synchronization network issues. Two DC’s are behind TMG, the third is in an ASA DMZ so the path is :

                  DC0  —            ASA   —                          TMG — DC73

Subnets:
                  10.0.0.0/24 — ASA — 10.0.203.0/24 — TMG — 10.0.73.0/24

The TMG performs no NAT for these networks so it’s plain routing. TMG has a default gateway set to ASA and ASA has a static route pointing to TMG’s outside address 10.0.203.100 .

Connectivity was completely broken (no ping, AD sync fail). On the ASA we could see half-open TCP connections:

ASA# sh conn det long | i 10.0.73.[91]
TCP sc-dmz:10.0.73.9/135 (10.0.73.9/135) sc-system:10.0.0.6/49198 (10.0.0.6/49198), flags saA, idle 1s, uptime 10s, timeout 30s, bytes 0
TCP sc-dmz:10.0.73.9/135 (10.0.73.9/135) sc-system:10.0.0.6/49197 (10.0.0.6/49197), flags saA, idle 3s, uptime 12s, timeout 30s, bytes 0
UDP sc-dmz:10.0.73.9/53 (10.0.73.9/53) sc-system:10.0.0.6/58639 (10.0.0.6/58639), flags -, idle 5s, uptime 13s, timeout 2m0s, bytes 195
UDP sc-dmz:10.0.73.10/53 (10.0.73.10/53) sc-system:10.0.0.6/58965 (10.0.0.6/58965), flags -, idle 0s, uptime 1s, timeout 2m0s, bytes 7

DC73 keeps initiating TCP connections to DC0 but saA state refers that replies (SYN-ACK) are not arriving.  The other direction was also tested. DC0 kept initiating TCP connections to DC73 but replies (SYN-ACK) were not arriving either. To decide whether TMG drops something and whether servers answer on TCP/135 port I collected packet captures on both ASA interfaces:

SYN-ACKs do arrive from DC0 to the ASA dmz0 interface:

ASA# sh cap CAPdmz0 | i 10.0.73.*64799
 177: 15:48:40.631803 802.1Q vlan#256 P0 10.0.73.10.64799 > 10.0.0.6.135: S 741135103:741135103(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
 178: 15:48:40.632062 802.1Q vlan#256 P0 10.0.0.6.135 > 10.0.73.10.64799: S 1283716945:1283716945(0) ack 741135104 win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
 242: 15:48:43.631726 802.1Q vlan#256 P0 10.0.73.10.64799 > 10.0.0.6.135: S 741135103:741135103(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
 243: 15:48:43.636914 802.1Q vlan#256 P0 10.0.0.6.135 > 10.0.73.10.64799: S 1283716945:1283716945(0) ack 741135104 win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
 285: 15:48:49.627790 802.1Q vlan#256 P0 10.0.73.10.64799 > 10.0.0.6.135: S 741135103:741135103(0) win 8192 <mss 1380,nop,nop,sackOK>
 286: 15:48:49.643643 802.1Q vlan#256 P0 10.0.0.6.135 > 10.0.73.10.64799: S 1283716945:1283716945(0) ack 741135104 win 65535 <mss 1460,nop,nop,sackOK>
 462: 15:49:01.657162 802.1Q vlan#256 P0 10.0.0.6.135 > 10.0.73.10.64799: R 1283716946:1283716946(0) win 0

But the SYN-ACKs are not forwarded to the dmz203 interface:
ASA# sh cap CAPdmz203 | i 10.0.73.*64799
  10: 15:48:40.631726 802.1Q vlan#203 P0 10.0.73.10.64799 > 10.0.0.6.135: S 3888604858:3888604858(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
  20: 15:48:43.631711 802.1Q vlan#203 P0 10.0.73.10.64799 > 10.0.0.6.135: S 3888604858:3888604858(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
  32: 15:48:49.627775 802.1Q vlan#203 P0 10.0.73.10.64799 > 10.0.0.6.135: S 3888604858:3888604858(0) win 8192 <mss 1460,nop,nop,sackOK>

Packets are obviously dropped by ASA, so I collected an ASP-drop capture too but no such SYN-ACKs were recorded in that capture.  Things became a bit mysterious until I checked  the next hop in the ARP table:

ASA# sh arp  |  i 10.0.203.100
ASA#

So the TMG does not use the specified address on its interface. The admins checked it and found that NLB which was supposed to be previously removed still has some effect on the configuration and we need to use another address of the NIC. The solution was to modify the static route to next-hop 10.0.203.1

This kind of error is not something ASA reports you proactively. Layer2 next hop ARP resolution failure does not generate any logs. show arp and debug arp is the only method you can discover such a failure but you are not likely to issue those commands until you suspect something with ARP. And when you already suspect it you are done.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: