"Wait for Service Pack 1" – Valid?

The release of Exchange 2007 last week prompted me to re-evaluate, again, a long help mantra in IT – "Wait for Service Pack 1".

The basic premise goes like this: New software, especially software from Microsoft, is normally so buggy on its release that it is far more sensible to wait for Service Pack 1 (SP1). This way others have gone through the pain that’s bound to be there.

But is that really still valid?

Does recent history from Microsoft support the premise? Is there any evidence?

The "Wait for SP1" situation can’t be one that Microsoft wants to persist because it delays that adoption of their newer software and potential stifles a revenue stream for them. But perhaps I’m wrong, perhaps they prefer the damper effect this has on demand. Let’s face it, if everyone upgraded fast they would have a problem.

There are a number of pieces of evidence that are available to us, but do they actually answer the question?

Service Pack History

Does Service Pack history help us out here? Is there evidence that the number of fixes in Service Pack 1 is substantially higher than the number of fixes in later Service Packs? Does the number of fixes in a Service Pack change depending on the maturity or generation of a product? Did Windows 2003 Service Pack 1 have less issues than Windows 2000 Service Pack 1, for instance?

Here are some numbers:

Number of Issues Resolved by Service Pack

	Service Pack 1	Service Pack 2	Service Pack 3	Service Pack 4
Windows 2000	287	470	1014	675
Windows XP	321	826
Windows Server 2003	1012
Exchange 2000	129	25	25
Exchange 2003	40	131

About the only thing that you can say about those numbers is that there is no correlation between the age of a product or the product generation and the number of issues that need to be resolved.

On these number Windows 2000 Service Pack 1 looked like a safe product, and so did Windows XP Service Pack 1, both of which resulted in a huge number of issues later in their life.

I had wondered about whether there might be some stronger correlation between the number of issues resolved and the time between Service Packs, but I don’t have that much time in my life.

These numbers do highlight one significant issue though, and that is that we are trying to make a judgement of quality based on quantity, and that’s normally not a good thing to do. The quantity of fixes probably doesn’t relate to the quality (or impact) of those fixes.

All it takes is one big issue and the quality is a problem. I suppose I could have gone through and tried to measure the number of "major" issues or something like that, but again, I don’t have the time.

There is some indication of quality in the numbers but you need to understand the back story. There are loads of fixes about the time of Exchange 2000 Service Pack 1 and Windows 2000 Service Pack 3 which demonstrate the first awakening to security as an issue within Microsoft, the emergence of SPAM, SASSER, etc.. The high number on Windows XP Service Pack 2 and Exchange 2003 Service Pack 2 demonstrate the second security awakening within Microsoft and the famous Bill Gates announcement. The high number of fixes in Windows Server 2003 demonstrates a shift towards more continuous update and less frequent Service Packs, this resulted in this particular Service Pack being released a long time after the release of Windows Server 2003.

Testing Process

We’ve already seen that we can’t make quality judgements on the basis of quantity. Perhaps, therefore, we need to look at the way that quality is built. In the case of software this is best demonstrated (in my view) by the level of good testing that occurs prior to release.

In the case of Exchange 2007 the numbers of live testers appear to be as follows:

We’ve bet the company on this product. Here at Microsoft, we have over 120,000 mailboxes running in production on Exchange 2007 – exceeding our SLA of 99.95% availability. Likewise, over 200 Technology Adoption Partners and Rapid Deployment Partners have over 55,000 mailboxes in production operating within their enterprise SLA’s.

You Had Me At EHLO…

Yes, I know I’m mixing quality and quantity again, and that’s the problem. Every time I try to assess quality I get to quantity. But taking these quantity numbers on there own, does 175,000 mailboxes account for enough testing?

There are apparently somewhere around 130 million corporate Exchange users. As a representative community 175,000 represents 0.13% of the user population!

Is that really representative, and if it isn’t what would be? The 120,000 internal users are certainly significant if you are going to deliver Exchange service the way that they have, if you aren’t then I’m not sure what it tells you.

I’ve been involved in a few TAP and RDP and the testing hasn’t really been representative of the real requirement. By this I mean that the testing wasn’t really done in a "production" environment and wasn’t really subject to the corporate SLA.

Continuos Update blunting the Bleeding Edge

We have moved a long way towards continuous update these days, this has the tendency to blunt the bleeding edge. Waiting for Service Pack 1 used to mean waiting for the first set of roll-up fixes; today we are used to an almost constant stream of updates. If a major problem is found people can get hold of it very quickly and apply it quickly too.

In the continuous update world it seems anomalous to talk about waiting for Service Pack 1, because that may be some time away.

My Personal Conclusion

My personal point of view is that there is no point in waiting for Service Pack 1 of Exchange 2007 specifically but there is a value in waiting a few months before actual deployment just-in-case. I would extend this view to other software too.

Likewise, there is no safety in staying with current product. The current product may have more uncovered issues that the new product.