The problem
We’re looking at setting up a management network behind a couple of ASAs.
My requirements and prerequisites are:
- No L2 end to end VLANs through the core. That is bad and wrong.
- A total site failure at one site must not take down hosts at the other site or any services run on the ASAs. This testing won’t get at far as the VPN side of things, today I’m just lookint at routing.
- Routing can be static or dynamic. I’ll use static today because my test switch doesn’t have an OSPF licence and I’m not in a RIP kind of mood.
- The ASAs need to be physically at different sites.
- We can use private fibre.
It will cost about £5K to get all the optics and interface cards we’d need to do proper dual site ASAs, with dual uplinks and HSRP enabled at the other end. I’m looking into an alternate method which only relies on dark fibre connecting the inside network switches and uses a differnt routed connection at each site. One issue is that the ASA configs are synced exactly. Since I want network connectivity to survive a failover and I can’t to send the same network to both sites in a scalable, redundant way, I’ll need to use two ports on each ASA and only connect one at each site. On failover, the main port will be down and the second connection up so I’ll then want the default route to change accordingly.
Summary
What, bored already? Okay my conclusion is that the ASAs can be made to failover to a second routed connection, but it is dog slow.
Network Diagram
Step by step
Set up active / standby
ASA 1
failover failover lan unit primary failover lan interface failover-link Ethernet0/3 failover interface ip failover-link 10.1.1.1 255.255.255.252 standby 10.1.1.2
ASA 2
failover failover lan unit secondary failover lan interface failover-link Ethernet0/3 failover interface ip failover-link 10.1.1.1 255.255.255.252 standby 10.1.1.2
Configure dual uplinks
The config will be replicated across the two ASAs. Site A will have its ‘ISP’ connection on E0/0, Site B will use E0/1.
! interface Ethernet0/0 nameif ISP-10 security-level 0 ip address 192.168.10.2 255.255.255.0 ! interface Ethernet0/1 nameif ISP-20 security-level 0 ip address 192.168.20.2 255.255.255.0 !
Uplink notes
The other end of the ISP links is a 3750 switch. E0/0 on the first ASA is connected to an access port in VLAN 10, E0/1 on the second ASA is connected to an access port in VLAN 20. The SVIs are given 192.168.10.1 and 20.1 repectively.
Static routes and tracking
We will configure static route tracking which allows us to change our default route if the link fails. For a production service we’d also configure the pair to failover on uplink failure.
First we configure the ASAs to keep an eye on their ISP gateways (sla_id 1 and 2):
sla monitor 1 type echo protocol ipIcmpEcho 192.168.10.1 interface ISP-10 sla monitor schedule 1 life forever start-time now sla monitor 2 type echo protocol ipIcmpEcho 192.168.20.1 interface ISP-20 sla monitor schedule 2 life forever start-time now
Now well configure the ASAs to track the sla_ids:
track 1 rtr 1 reachability ! track 2 rtr 2 reachability
Finally we define the static routes, setting them to drop off if the gateway IP should not be reachable and making the main ISP the default. We could have ignored all the setp above and just used the metrics (in bold below, sorry I chose 1 and 2, that is a bit confusing in this context), but then the second route would only be used if the ASA interface went down, which isn’t the only failure scenario.
route ISP-10 0.0.0.0 0.0.0.0 192.168.10.1 1 track 1 route ISP-20 0.0.0.0 0.0.0.0 192.168.20.1 2 track 2
Testing
First lets enable debugging so that we can see exactly what happens:
logging enable logging timestamp logging console debugging sdc-asa# debug track
Tracked IP unreachable tests
I won’t repeat all the debug output here but here are the interesting bits:
sdc-asa# failover active May 24 2011 14:47:56: %ASA-1-104001: (Secondary) Switching to ACTIVE - Set by the config command. sdc-asa# show route <snip> Gateway of last resort is not set C 192.168.10.0 255.255.255.0 is directly connected, ISP-10 C 192.168.20.0 255.255.255.0 is directly connected, ISP-20 C 10.1.1.0 255.255.255.252 is directly connected, failover-link May 24 2011 14:48:41: %ASA-6-622001: Adding tracked route 0.0.0.0 0.0.0.0 192.168.20.1, distance 2, table Default-IP-Routing-Table, on interface ISP-20 sdc-asa# show route <snip> Gateway of last resort is 192.168.20.1 to network 0.0.0.0 C 192.168.10.0 255.255.255.0 is directly connected, ISP-10 C 192.168.20.0 255.255.255.0 is directly connected, ISP-20 C 10.1.1.0 255.255.255.252 is directly connected, failover-link S* 0.0.0.0 0.0.0.0 [2/0] via 192.168.20.1, ISP-20
As you can see, it takes 45 seconds for the alternate default route to appear in the routing table of the second ASA after failover. Lets try failing back over.
sdc-asa# failover active May 24 2011 14:56:35: %ASA-1-104001: (Primary) Switching to ACTIVE - Set by the config command. sdc-asa# show route <snip> Gateway of last resort is not set C 192.168.10.0 255.255.255.0 is directly connected, ISP-10 C 192.168.20.0 255.255.255.0 is directly connected, ISP-20 C 10.1.1.0 255.255.255.252 is directly connected, failover-link Track: 1 Change #11 rtr 1, reachability Down->Up May 24 2011 14:56:59: %ASA-6-622001: Adding tracked route 0.0.0.0 0.0.0.0 192.168.10.1, distance 1, table Default-IP-Routing-Table, on interface ISP-10 show route <snip> Gateway of last resort is 192.168.10.1 to network 0.0.0.0 C 192.168.10.0 255.255.255.0 is directly connected, ISP-10 C 192.168.20.0 255.255.255.0 is directly connected, ISP-20 C 10.1.1.0 255.255.255.252 is directly connected, failover-link S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.10.1, ISP-10
This time it took 24 seconds, which is better but still considerably worse than the subsecond failver time we can acheive with HSRP and cross site, dual uplinks from the ASAs. Repeat testing showed that primary -> secondary was always c. 30 seconds, but secondary -> primary could be much faster:
May 24 2011 15:02:54: %ASA-1-104001: (Primary) Switching to ACTIVE - Set by the config command. May 24 2011 15:02:59: %ASA-6-622001: Adding tracked route 0.0.0.0 0.0.0.0 192.168.10.1, distance 1, table Default-IP-Routing-Table, on interface ISP-10
Repeat test with physical interface failure
Here you can see that this took a full minute to fail over, but did still work. The ASA will track its interfaces by default so no additional config was needed. As you can see, the failover times are rather uninspiring.
! Secondary to active May 24 2011 15:11:13: %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Standby Ready, peer state Failed. Switching to Active May 24 2011 15:11:13: %ASA-1-104001: (Secondary) Switching to ACTIVE - Other unit wants me Active. Primary unit switch reason: Interface check. May 24 2011 15:12:41: %ASA-6-622001: Adding tracked route 0.0.0.0 0.0.0.0 192.168.20.1, distance 2, table Default-IP-Routing-Table, on interface ISP-20 ! Primary returns to active May 24 2011 15:10:58: %ASA-6-721002:(WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Standby Ready, peer state Failed. Switching to Active May 24 2011 15:10:58: %ASA-1-104001: (Primary) Switching to ACTIVE - Other unit wants me Active. Secondary unit switch reason: Interface check. May 24 2011 15:11:59: %ASA-6-622001: Adding tracked route 0.0.0.0 0.0.0.0 192.168.10.1, distance 1, table Default-IP-Routing-Table, on interface ISP-10
Now what?
Next time I’m going to see whether it is possible / desirable to run a VPN on this set up.