- All patches have a probability of introducing new bugs. That probability is always > 0 and <= 1. The probability is never equal to zero. (And for a certain large database vendor, our experience is that the probability of introducing new bugs is very close to one).
- There are many, many bugs that are only relevant under high loads.
- A patch that corrupts data, as in databases or file systems, can be impossible to back out or recover from without irretrievable data loss.
- Building test cases that can put realistic real world loads on test servers is very difficult, very expensive, and may not uncover the new bugs anyway.
- A failed system or application has known, documented consequences. It is not a game of probability or chance. An unpatched security vulnerability is a game of chance where in most cases the odds against you are not known.
But when security people push recommendations out to the world without consideration for availability and/or performance, their recommendations remove value from the Internet community.
Security Researchers add value when
- Uncovering and analyzing vulnerabilities and active exploits. (Research)
- Analyzing probable and improbable attack vectors and calculating and communicating probabilities. (Research)
- Testing and verifying attack vectors. (Research)
- Communicating to the community the relative and absolute risks of vulnerabilities and consequences of exploitation. (Public Service)
- Developing and communicating mitigation options. (Research)
- Making blanket patch advice without consideration for performance or availability. (Operations)
- Complaining about enterprises that do not follow their advice. (Carping)
In that context, when I hear 'patch now' advice, You can bet that I will filter the advice through the prism of availability, performance and operational reality.
- I'll listen to 'patch now no matter what' advice from a security researcher/blogger who has real time operational responsibility for a large customer base, perhaps 100,000 or more customers, and who, if the patch fails, would be responsible for interruption of service for those hundred thousand customers, and who, if the patch fails, could or would be terminated for non-performance.
- I'll listen to 'patch now no matter what' advice from a security researcher/blogger who has had a system with a hundred thousand customers down hard, has escalated to the vendors highest support level, and who has been on a tech support conference call for 13 continuous hours or more.
- I'll listen to 'patch now no matter what' advice from consultants who are putting the reputation and existence of their consultancy on the line every time they give a customer advice.
- I'll listen to 'patch now no matter what' advice from our own security staff, who I know will not point fingers, duck and hide when the patch goes bad and my systems fail.
The bottom line is that unless the people who give the world advice to 'patch now no matter what' are also going to write my e-mail's and presentations explaining why my systems failed, unless they will absorb the inevitable backlash from customers, senior management, governing boards and will stand up in front of representatives from my internal business units and get grilled, castigated, chewed up and spit out for my decision, I don't need them to complain that I am not 'patching now'.
I've been in the 7pm vendor conference call with vendor VP and development supervisor, where our CIO came to the meeting with his/her letter of resignation, to be turned in to our CEO should the vendor fail to deliver performance fixes for the business critical application by 7am the next day.
It was not a fun meeting.
'Patch now' advice must be filtered through the prism of availability, performance and operational reality.