In most situations when the subject of network availability is raised, the conversation immediately turns to implementing network devices that are highly fault tolerant, and also turns to implementing network designs that have diverse end-to-end paths and which feature fast failover protocols. Those standard network design techniques are all reasonable. However, if IT organizations just implement those techniques they may be missing the bigger picture.

Jim recently consulted with a F200 IT organization and that experience served to highlight the fact that standard network design principles are insufficient to ensure high network availability. The company was looking for a consultant because they had implemented VoIP and their network had crashed twice bringing down a midsized call center. The company hired Jim to analyze their network infrastructure and recommend new technologies and designs that would enhance the reliability of their network.

Thankfully, the IT organization that hired Jim had lots of documentation about their network and how it was designed. Jim soon realized that they had already implemented a highly redundant network. Were there things that could be tweaked? Of course there were, but they were relatively minor.

Shortly after Jim began the project, the company experienced a third outage. Since this outage was recent, Jim was able to do an extensive forensic analysis of what had gone wrong. The company had attempted to update their IP PBX on a Sunday night and the upgrade failed and the system crashed and nobody was notified. To make matters worse, the backup system had been taken offline a week earlier. The bottom line was that the organization’s change management system was determined to be the primary cause of all three of their outages.

In a recent newsletter we discussed a survey that we are currently conducting. One of the questions in that survey is “Do you believe that in your organization ineffective change and configuration management causes the majority of network outages?” We would like to encourage you to participate in the study, you’ll be able to get a copy of the preliminary results as soon as they are available. In addition, we encourage you to write to us and tell us about what steps your organization has taken to get better at change and configuration management.

Check out Network World's Buyer's Guide to compare Forensics Tools.

Steve Taylor is president of Distributed Networking Associates and publisher/editor-in-chief of Webtorials. Jim Metzler is vice president of Ashton, Metzler & Associates.