Showing posts with label availablity. Show all posts
Showing posts with label availablity. Show all posts

Thursday, November 19, 2026

Rule One....

 ...If it ran yesterday, it must run today.  This is the first rule of change management.  I was put in mind of it this week when we suffered an unprecedented number of hours long  system outages with disastrous consequences. Microsoft attributed its outage to a "bug."  My immediate reaction was, no, not a bug, a failure to meet Rule One.  I learned this rule as a young development manager on IBM's Advanced Administrative System, the job that was to determine my sixty year career in what we now call Cyber Security.  

My team had a major change scheduled to go live on Tuesday February second.  We were ready and anxious but our change got put off, I cannot now remember how many times, but it was so many that it became a joke; virtual "February the Tooth."  Every Tuesday morning we were ready. 

When we complained, we were told that the reason was Rule One.  The system was not stable and response time was creeping up.  

My managers were those who had built the American Airlines Sabre System, the legendary reservation system, the grandaddy of all modern day database systems and applications.  While today we can run an airline reservation system on a pair of servers, in those days it took a room full of computers.  Mean time to failure and recovery, system availability, and response time were all specified in the system's requirements, measured, and managed to.

What my mentors and managers taught me was counter-intuitive.  If the system is not operating as intended, if load is high, availability low, if there is any risk in the environment, do not patch.  Institute change control.  The idea was to be sure that any change was more likely to make things better than worse.  

Under change control the rate of change should go down not up.  The rate tolerated was arbitrary but the idea was rooted in bitter experience; availability is inversely proportional to the rate of change.  Therefore all change was submitted to a change control committee that evaluated its risk to the system.  

Now my team was cracker jack. You should all get to manage such a great team.  They had a great track record.  None of our changes had ever brought the system down or failed to work as intended.  However, our change was to logon. If it failed, it would bring down everything.  Needless to say, the committee evaluated it as high risk; our record was not sufficient to overcome the risk. 

One way to meet Rule One was to ensure that the system could always be returned to the last configuration that worked.  Not only did a change come with a plan, the set of procedures by which the change would be made, but a plan for how to undo it, and any all other changes made at the same time, if there was any indication of a problem.  The change control committee evaluated not just the change but these plans.   

No attempt would be made figure out what went wrong or which change caused the problem. If yesterday's configuration worked and today's didn't, then return to yesterday's configuration.  Knowing how to return to a working configuration was essential.  Returning to a working configuration was designed to take minutes, not hours.  No potential improvement could trump Rule One.