critical patch application redux

Steven Chan has an interesting set of observations on patching, taken together they form an interesting argument for regular and scheduled patching.  Steven offers 5 common 'myths' that might influence a conservative patching policy. These are

1. It requires too much downtime
2. Testing is too expensive for end-users
3. It's too complicated
4. We don't have enough staff
5. It ain't broken; why fix it?

 While there undoubtedly is truth in what Steven says I still don't buy them quite yet, or perhaps I do but come to a different conclusion as a practitioner rather than a supplier.

Lets look at them one at a time, with the exception of staffing where it seems to me Steven is right on the money.

Too Much Downtime

Steven raises 3 issues here

Questions to consider if you believe this myth:

• How much downtime is currently caused by unplanned outages due to unpatched stability bugs?
• How much downtime is needed to bounce your application servers due to unpatched JVM memory leaks?
• How much downtime would be required if an attacker takes advantage of an unpatched security risk?

Now we are relatively fortunate to be running 11.5.10 CU1 (plus a number of functional patches). So we are relatively up to date.  A quick review of the last year suggests that in total the downtime due to the first reason was less than 3 hours.  We don't bounce our app servers. So we are pretty much down to the unpatched security risk. This is unfortunately a classic judgement call. At the moment my judgement has been that the most likely breaches of our security would be classic social engineering, misconfiguration and setup error. for an installation like ours, wholly inward facing these are far easier to achieve and far more likely than a technical approach.

My current estimate is that applying CPU patches would definitely cause in the region of 2 days downtime a year. (1/2 a day times 4). That's quite a heavy price to pay for a relatively remote risk.

Interestingly we will be deploying iRecruitment shortly. This entirely changes the risk profile since like Oracle we will be presenting a new attack vector to the outside world. We will therefore be changing our policy, and in fact I'm catching up on two years worth of patches right now!

Too Expensive

• How much productivity is lost for all end-users -- not just the testers -- due to unpatched performance, stability, or security bugs?
• How much could productivity be improved by new features?

In the case of CPU the second does not (or should not) apply. So the question becomes how much productivity do you gain by being on a current CPU level? My guess is that it probably isn't 2 days downtime + 2 man weeks testing (our requirements over a year) per year.

Too Complicated

It is true that keeping up to date reduces pre-requisites, but the truth must surely be that if Oracle apps patches were

• cumulative
• more automated

then we could make considerable inraods into that 2 days a year.

It Ain't Broken; Why Fix It?

Steven's main point here seems to be that if you get very out of date - i.e are on a release out of premier support, then you have a headache with an upgrade to a supported version. That is undoubtedly true, but I do wonder how many customers this applies to. The other discussion missing is the fact that every production dba will have a tale of a patch that itself broke things - this is why we test after all. It also tends to apply to new functionality as the bugs get ironed out. being right up to date used to be called being on the bleeding edge. That was borne out of sometimes bitter experience. One might add to the list of questions

• how much productivity is lost by being on a new and relatively untested patch level.
• how much extra time does problem diagnosis take on a new and relatively untested patch level.

in the above I'm using untested as shorthand for not widely adopted.

Summary

 In summary then, the issues Steven raises are indeed important, and especially where functionality is updated patching is a necessary practice. In the case of security or maintenance patches however the balance of concern for those questions may well lead you to conclude that the patch was not worth it. Especially if like many organisations you don;t have a consistent patchin policy, but you do have an upgrade policy - upgrade to the first service pack of each new release for example.