We have a customer who uses a Cisco ASA 5505 in transparent mode to protect a small LAN. They did the right thing and took out SmartNet cover, but the reseller botched something and the TAC wouldn’t play with them when they had problems. They gave me a call and the results were interesting enough to prompt this blog.
Problem
After reading the latest Cisco Advisory (worth doing), they had upgraded the software on the ASA from 8.2 to 8.4. However, after doing this DHCP no longer worked on their subnet. The ASA rules needed to get that working were in place. More detail on the DHCP side of things at the bottom of this post.
Cause
When the customer upgraded, they didn’t note the memory requirements needed for version 8.4. They had 256 MB instead of the required 512 MB. It is a Very Good Idea to check this when upgrading the image on any Cisco device, details near the bottom of this post. As we found here, sometimes the device will accept and run code that it shouldn’t. You do get a warning message on boot telling you your device doesn’t have enough memory. In this case, the engineer performing the upgrade didn’t know to look for this.
Impact
Any clients with a static IP were able to access the Internet fine, but no DHCP requests made it through the firewall. The counters on the ACL didn’t even increment. What I find interesting is that the device booted up and sort of ran. Before seeing this I would have assumed a more catastrophic failure. I wonder if a less subtle failure would have been easier to deal with? Since there isn’t always enough flash to store multiple images, not booting at all may not be the best idea. Perhaps booting, not passing any client traffic and filling the logs with memory grumbles is the answer..
Solution
The customer downgraded the image on their ASA and DHCP sprang back into life. They are going to order some more memory before repeating the upgrade. This was a good reminder that an engineer should always check the release notes when upgrading software.
More on memory
Since you may be reading this long after 8.4 is current, and since cisco.com is a complicated beast, I would suggest going to http://www.cisco.com/go/asa (or go/6500 or go/MYDEVICE) and then clicking on ‘Release and General Information’ if something like that still exists. You should then be able to find the release notes for the version of code you wish to install. Any memory requirements are in there.
Additional DHCP mutterings
Although not strictly relevant here, DHCP through a transparent mode ASA is a bit of a pain as you have to explicitly let everything through. I was sidetracked by this at first due to the symptoms the customer experienced. Their ASA was configured correctly as I said. What follows is a run through of their config and the general idea.
The customer uses our central DHCP servers rather than the ASA’s daemon. The gateway for their network is an SVI on a Cisco 6500 with an ip helper-address configured for each DHCP server. A simplified version of what should happen follows:
- The clients broadcast for a DHCP server
- The firewall allows this through
- The gateway proxies the broadcast to the DHCP server
- The DHCP server replies to the gateway
- The gateway sends the reply to the client
- The firewall allows this reply through
There are further messages involved, have a look at RFC 2131 if curious. Since the ASA is in transparent mode, inbound and outbound access-list rules are required for steps 2 and 5 to work. The Cisco config guide doesn’t include example access-lists so I will below.
# Inbound access-list outside_access_in remark Allow DHCP offer access-list outside_access_in extended permit udp host <ip of default gateway> any eq bootpc # Outbound access-list inside_access_in remark Allow DHCP discovery / request access-list inside_access_in extended permit udp host 0.0.0.0 host 255.255.255.255 eq bootps access-list inside_access_in remark Allow DHCP access-list inside_access_in extended permit udp any object-group <group with all dhcp servers> eq bootps