A couple of weeks ago I was asked to configure an isolated route between 2 servers over our external PRIVATE network. At first I didn’t think this would take more than an hour to set up, but as I pondered on it some more it became apparent that I was going to need to utilize my skills to achieve the desired result.

Take a look at the diagram:

Traffic from end-to-end normally traverses the INTERNAL network. Static NAT has been configured on the routers to control all traffic destined for the 10.111.0.0/16 network to accommodate the different subnets within the company. Server HQ-HOST is configured with an IP address of 10.30.48.60 and is NAT’d to 10.111.77.60. Server REM-HOST is configured with an IP address of 10.200.208.54 and is NAT’d to 10.111.177.54.

Here’s what the router configuration for HQ-RTR looked like before I made any changes:

host HQ-RTR

int lo 77
ip add 10.111.77.1 255.255.255.0
ip nat inside

int gi 0/0
ip add 10.111.1.1 255.255.255.248
ip nat inside

int gi 0/1
ip add 10.111.1.242 255.255.255.248

int vl 804
ip add 10.111.254.9 255.255.255.248
ip nat outside

router ospf 1
net 10.111.77.1 0.0.0.0 a 0
net 10.111.254.9 0.0.0.0 a 0
net 10.111.1.242 0.0.0.0 a 15
red conn route-map CONN->OSPF sub

route-map CONN->OSPF permit 10
match int gi 0/0

ip nat inside source static 10.30.48.60 10.111.77.60

Here’s what the router configuration for REM-RTR looked like before I made any changes:

host REM-RTR

int lo 177
ip add 10.111.177.1 255.255.255.0
ip nat inside

int gi 0/0
ip add 10.111.1.25 255.255.255.248
ip nat inside

int gi 0/1
ip add 10.111.1.113 255.255.255.248

int vl 804
ip add 10.111.254.10 255.255.255.248
ip nat outside

router ospf 1
net 10.111.177.1 0.0.0.0 a 0
net 10.111.254.10 0.0.0.0 a 0
net 10.111.1.113 0.0.0.0 a 115
red conn route-map CONN->OSPF sub

route-map CONN->OSPF permit 10
match int gi 0/0

ip nat inside source static 10.200.208.54 10.111.177.54

My first objective was to establish the correct path by using static routes on both routers to redirect the traffic out the ASA firewalls on the desired path. Here are the router additions:

host HQ-RTR

ip route 10.111.177.54 255.255.255.255 10.111.1.241

host REM-RTR

ip route 10.111.77.60 255.255.255.255 10.111.1.114

Next, I needed to configure the ASA firewalls for this external private network. Here’s the configuration for HQ-FW:

host HQ-FW

int gi 0/3.15
vlan 999
nameif extprivnet
security-level 50
ip add 10.111.15.2 255.255.255.252

access-l EXTPRIVNET extended permit ip any any
access-gr EXTPRIVNET in interface extprivnet

nat (extprivnet) 315 0.0.0.0 0.0.0.0
static (inside,extprivnet) 10.111.77.60 10.111.77.60 netmask 255.255.255.255

route extprivnet 10.111.177.54 255.255.255.255 10.111.15.1

Here’s the configuration for REM-FW:

host REM-FW

int gi 0/3.15
vlan 999
nameif extprivnet
security-level 50
ip address 10.111.115.2 255.255.255.252

access-list EXTPRIVNET extended permit ip any any
access-group EXTPRIVNET in interface extprivnet

nat (extprivnet) 3105 0.0.0.0 0.0.0.0
static (inside,extprivnet) 10.111.177.54 10.111.177.54 netmask 255.255.255.255

route extprivnet 10.111.77.60 255.255.255.255 10.111.115.1

I verified my configurations by pinging from HQ-HOST to REM-HOST:

[root@HQ-HOST ~]# ping 10.111.177.54
PING 10.255.140.27 (10.255.140.27) 56(84) bytes of data.

None of the ICMP replies were returning so I now I needed to debug each device in my path. On the HQ-RTR router, I added the following:

ip access-l ext TSHOOT
permit ip any any log

int gi 0/0
ip access-gr TSHOOT in

I reran the ping test and and ran ’sh log | i TSHOOT’:

2720322: Jan 18 23:43:29.014 UTC: %SEC-6-IPACCESSLOGDP: list TSHOOT permitted icmp 10.30.48.60 -> 10.111.177.54

The above output tells me that the packet was received by the router. Now I need to verify if the packet was making it to the firewall. In the past, I’ve always had complete control of the routers internally and externally to the firewall so if I needed to troubleshoot anything, I would just verify it on the routers. After a few minutes of digging around on Cisco’s website, I found ‘Document ID: 71871‘ to Troubleshoot Connections Through the PIX and ASA. On HQ-FW, I set up the following:

access-list CAPTURE ext perm ip host 10.111.77.60 any
access-list CAPTURE ext perm ip any host 10.111.177.54

capture CAP access-list CAPTURE in inside
capture CAP access-list CAPTURE in extprivnet

Here’s the output from ’show capture CAP’:

HQ-FW(config)# show capture CAP

4 packets captured
1: 15:19:51.641568 10.30.48.60 > 10.111.177.54: icmp: echo request
2: 15:19:52.641293 10.30.48.60 > 10.111.177.54: icmp: echo request
3: 15:19:53.641019 10.30.48.60 > 10.111.177.54: icmp: echo request
4: 15:19:54.640744 10.30.48.60 > 10.111.177.54: icmp: echo request
4 packets shown

After analyzing the output I could already tell what command I was missing on the router. REM-FW was expecting traffic originating from 10.111.77.60. HQ-FW was seeing the traffic as 10.30.48.60 and not as the NAT’d address. At this point I added a NAT statement on the router:

host HQ-RTR

int vl 8
ip nat outside

Ran the ping test again and didn’t bother check the router. I just checked the firewall:

HQ-FW(config)# show capture CAP
10 packets captured
1: 15:19:51.641568 10.30.48.60 > 10.111.177.54: icmp: echo request
2: 15:19:52.641293 10.30.48.60 > 10.111.177.54: icmp: echo request
3: 15:19:53.641019 10.30.48.60 > 10.111.177.54: icmp: echo request
4: 15:19:54.640744 10.30.48.60 > 10.111.177.54: icmp: echo request
5: 15:31:38.902846 10.111.77.60 > 10.111.177.54: icmp: echo request
6: 15:31:38.902846 802.1Q vlan#999 P0 10.111.77.60 > 10.111.177.54: icmp: echo request
7: 15:31:39.902571 10.111.77.60 > 10.111.177.54: icmp: echo request
8: 15:31:39.902571 802.1Q vlan#999 P0 10.111.77.60 > 10.111.177.54: icmp: echo request
9: 15:31:40.902312 10.111.77.60 > 10.111.177.54: icmp: echo request
10: 15:31:40.902312 802.1Q vlan#999 P0 10.111.77.60 > 10.111.177.54: icmp: echo request
10 packets shown

At this point, I added the reversed configurations on the remote side, verified end-to-end was working properly, and removed the ACLs off the router and firewall.

It’s been a while since I’ve had time to blog. Ever since I became a CCIE, life has been quite busy for me and internal projects have been coming out of the wood work. In the past, I would design a network/security solution, configure it for testing, and implement it into production. Now on top of all that I need to maintain it going forward.

In my mind I’ve always had unique installations where I’m working on my own and can’t really bounce the idea off anyone else. How does one really know whether or not they’ve designed the most optimal solution? The only way to really know is to open up my ideas to everyone. I figure it is time to share what I’ve done with my projects. This can either present ideas for others or help me for future installs. (Plus, the configurations and gotchas will always be available for me to reference).

Also, as a tribute to the upcoming ‘The X Files 2‘ movie later this summer, I’ve decided to label my projects as ‘x-files’ since each solution is its’ own special case. (If you’ve read some of my older posts, you might be able to put 2 and 2 together to figure out which customer I just implemented this solution.)

The project was to replace a Windows 2003 server at a client location that was continuously running a Cisco VPN client to connect to our headquarters. The server sits in the client’s DMZ and the client software running on the box was customized to connect to our ASA every 3600 seconds.

Believe it or not, the above solution worked for our needs but wasn’t as scalable. The demands for this server had changed where monthly trend analysis was necessary, security controlled, and out-of-band management needed to be present in the new solution. I recommended to install our own Cisco switch and router, migrate out the old windows server and replace with a linux appliance, and install a 3rd party console server.

Here’s the diagram outlining the new solution:

Analyzing the diagram, you may notice a few things. The CLI-FW is performing NAT and the router’s external address is 111.91.97.2. For the time being, ignore the PRIVATE network and assume the CLI-FW is allowing all traffic to the router. It is essentially my edge device, so I’ll need to configure NAT and PAT so my server could access the Internet. I decided to take a different approach and configured NAT-on-a-stick to avoid issues with IPSec later. Here’s the configuration I used on CLI-RTR:

int loop 919
ip add 172.16.91.9 255.255.255.248

int gi 0/0
ip add 192.168.91.2 255.255.255.248
ip nat outside

int gi 0/1
ip add 172.16.91.2 255.255.255.248
ip nat inside

ip route 0.0.0.0 0.0.0.0 192.168.91.7

ip nat inside source list CLINAT int gi 0/0 overload

ip nat inside source static tcp 172.16.91.9 22 192.168.91.9 22 extendable
ip nat inside source static tcp 172.16.91.9 23 192.168.91.9 23 extendable
ip nat inside source static tcp 172.16.91.3 20 192.168.91.3 20 extendable
ip nat inside source static tcp 172.16.91.3 21 192.168.91.3 21 extendable
ip nat inside source static tcp 172.16.91.3 22 192.168.91.3 22 extendable

ip access-list extended CLINAT
permit ip 172.16.91.0 0.0.0.7 any

Now I needed to configure an IPSec tunnel back to HQ for our own internal communications. Here are the additional commands on CLI-RTR:

int gi 0/0
cry map SECURE

int gi 0/1
ip policy route-map NONAT

ip access-list extended CLINAT

ip access-list extended NONAT
permit ip 172.16.91.0 0.0.0.7 10.111.0.0 0.0.255.255

route-map NONAT permit 10
match ip address NONAT
set ip next-hop 172.16.91.10

crypto isakmp policy 10
encr aes
authentication pre-share
group 2

crypto isakmp key SECUREKEY address 205.19.73.2

crypto ipsec transform-set CLISET esp-aes esp-sha-hmac

crypto map SECURE 10 ipsec-isakmp
set peer 205.19.73.2
set transform-set CLISET
match address CLI

ip access-list extended CLI
permit ip 172.16.91.0 0.0.0.7 10.111.0.0 0.0.255.255

HQ-FW looks as follows:

interface GigabitEthernet0/0
nameif outside
security-level 0
ip address 205.19.73.2 255.255.255.0
!
interface GigabitEthernet0/1
nameif inside
security-level 100
ip address 10.19.1.241 255.255.255.248

access-list CLI extended permit ip 10.111.0.0 255.255.0.0 172.16.91.0 255.255.255.248

access-list NONAT extended permit ip 10.111.0.0 255.255.0.0 172.16.91.0 255.255.255.248

crypto ipsec transform-set CLISET esp-aes esp-sha-hmac

crypto map secure 10 match address CLI
crypto map secure 10 set peer 111.91.97.2
crypto map secure 10 set transform-set CLISET

crypto isakmp identity address
crypto isakmp enable outside

crypto isakmp policy 10
authentication pre-share
encryption aes
hash sha
group 2
lifetime 86400

tunnel-group 111.91.97.2 type ipsec-l2l
tunnel-group 111.91.97.2 ipsec-attributes
pre-shared-key SECUREKEY

For some odd reason when I configured all of this, I couldn’t continuously ping or copy a large file (i.e. > 1GB) from CLI-to-HQ or HQ-to-CLI. The tunnel never went down, but the sessions within the tunnel kept dropping or timing out. To get around this I had to create a crontab on the linux server to send 3 ICMP packets every 15 minutes.

Everything seemed fine for about a month and then things took a turn for the worse. Files needed to be copied in both directions almost daily, but the sessions were failing once again. After combing through my configurations, I realized I needed to add the following to the HQ-FW:

isakmp nat-traversal

The sessions seemed to be alright if I initiated from CLI-to-HQ, but not in the other direction. Since the tunnel was up since day one, I never thought it was a network problem and focused my time on the linux servers. I eventually drilled down to the crontabs I created a month ago which were restarting sessions within the tunnel. Odd? While I’d like to truly know what is going on, I’ve run out of time.

Knock on wood, the configuration is stable now and my users are happy.