ISC DHCP Failover is just too complex!

We have just been having a conversation in the office about the DHCP failover mechanism that is provided by ISC DHCP. Basically, what started as a really simple way to provide DHCP failover (described in a 14 page Internet draft here), morphed into a highly complex system (described in the revised 133 page draft here).

I can’t help thinking that the failover draft will NEVER become an RFC due to it’s complexity, and we have seen a few issues on our helpdesk now where the complexity actually contributes to problems and can also lead to longer “fix resolution” times.

One of the problems is the sheer number of “states” that either DHCP peer can be in as illustrated in this state table:

The only time when both DHCP peers are operating normally is when they are in the “NORMAL” state. Any other state and something will not be working correctly. Sometimes you can detect that there is a problem because clients are not getting the correct lease time. This is more than likely because they are getting a lease defined by the Maximum Client Lead Time policy (MCLT), which in most cases is around 1 hour.

But we have seen the MCLT used during other operations, for instance if the state changes to “POTENTIAL CONFLICT”. In order to resolve the conflict, neither peer will offer leases during the MCLT, so if you have clients that have previously been issued MCLT leases (maybe because a peer was previously in “COMMUNICATIONS INTERRUPTED” state), all, or many, of your DHCP clients will expire their leases and could fall off the network while the conflict is being resolved.

In simpler DHCP server implementations that use the earlier draft, it is easy to resolve a conflict. Shut down the failover peer, delete the lease file, and restart it. It will obtain a new copy of the leases from the primary peer and then it is fully synched up. It literally takes minutes to do this (rather than 1 hour plus that the MCLT forces you to wait in the latest draft). The earlier draft does not provide load balancing, but we have found very few customers actually need it. So the simpler draft is much easier to manage, maintain, troubleshoot and fix.

We have also seen issues with the load balancing algorithm present in the latest DHCP failover draft. The failover peers are supposed to “re-balance” their lease pools so that they each have approximately 50% of the free leases available. But we have seen issues where the re-balancing doesn’t work, so one server will run out leases and the other peer will not answer because it expects it’s partner to respond to the lease request. Normally the client would just retransmit it’s DHCP lease request and increase the “elapsed time” field in the DHCP packet so that the DHCP servers would see it as a retransmit (the other peer would then respond). But we have seen bugs where the DHCP client would not increment the “elapsed time” field, therefore the DHCP servers would not see it as a retransmit, and the peer with the free leases would still not respond. We also had another customer who only had one of the DHCP servers listed in their ip-helper configuration on their routers, so again only one DHCP server would respond to lease requests, and it would run out of leases when the re-balancing failed to work correctly (for whatever reason).

There is a feeling here in the office that the later draft is way too complicated. The later draft failover protocol appears to have been “enhanced” because people were worried that the original draft did not address the remote possibility that a duplicate IP address could be issued if network connectivity between the two failover peers was interrupted. In reality, most clients will carry on using the IP address they already have, and would just renew accordingly, so the problem only occurs with new clients coming onto the network or clients moving between subnets. In reality, the risk of issuing a duplicate IP address does not justify the complexity of the later drafts, and if there is a problem, Windows can normally detect a duplicate IP address (via an ARP) and request a different IP address.

Even if a duplicate IP address is issued, the quantity is usually very small and it is much more preferable to a customer to have a small number of clients experience issues, rather than have the whole estate affected, due to some weird state that the DHCP servers have got themselves into.

However, to caveat these thoughts, most of the issues we have seen have been to do with a misconfiguration of the customers environment. If everything is set up correctly then there is a good chance this stuff will “just work”. There’s a great post here that explains how DHCP failover works and what happens when things go wrong. Just make sure you understand how it works before you deploy it. 🙂