Patch Management: Philosophy and Soliloquy
An introduction to how startups keep third-party software up-to-date and secure.
Patch management is the process of updating software. Software developers write updates, they release patches, you apply those patches, and your software is up-to-date.
Managing updates for the apps on my own personal phone is complicated. There is no company too small to run into overwhelming complexity as they try to manage the patches to all manner of software being used across their environment.
This is the first in a series of essays diving deep on how startups keep their software up-to-date. In this post I’ll talk about my general philosophy toward patch management.
Important, not urgent
Patch management is often a low priority for companies, for very reasonable reasons:
It is a complex problem with a relatively high cost to get right.
The benefit of applying a given patch is usually minimal at best.
The risk of an update breaking important software is greater than zero.
Patch management is a typical example of a problem that is “Important, not urgent.” Those sorts of problems are hard to solve when you have plenty of “Important and urgent” problems on the list.
Maybe a little urgency, as a treat
These are also the sorts of problems I find really interesting, because they are great candidates for setting aside time to “solve it right, once and for all” and then forgetting that other people have such problems.
Patch management gets increasingly difficult to solve as time goes on. Systems become more complex, human behavior becomes canon, and the activation energy required to make changes only grows.
For this reason, I would argue that there is at least some urgency to solving patch management.
Nuanced, but important
Security vulnerabilities are sometimes among the bugs fixed by software updates. Leaving these in place can increase the risk of a successful data breach or other abuse by bad actors.
With that said, the vast majority of vulnerabilities reported in software packages for a given business are not exploitable in any meaningful way. There is an epidemic of noise coming from software composition analysis tools which provide very little actionable data.
Traditionally, you might prioritize patching areas of your IT infrastructure that are more likely to be exploitable, such as “internet-facing” systems. The typical modern cloud-based startup has an architecture that is so far removed from the previous generation of production IT that concepts such as “internet-facing” systems are barely recognizable.
If all of your traffic is run through AWS-managed load balancers to AWS-managed servers running AWS-managed virtual machines and AWS-managed Kubernetes clusters, which is finally running a Docker container with your code, do you have any “internet-facing” systems? Who knows!
Does an outdated version of Vim in the Docker container on the Kubernetes cluster run by the hypervisor, on the server behind the load balancer actually put you at any risk at all?
The best answer I have today is “No, probably not.”
The world is full of nuance though, and these are incredibly complex systems. To completely ignore the problem of patching software in these cases would leave us with no margin for error. No buffer to account for the fact that unexpected things happen all the time.
I’ve been surprised to find vulnerabilities in outdated packages lead to real exploitable issues with minimal required effort. If I had let my overconfidence run wild, I wouldn’t have even checked and the risk of an incident would have been higher.
The current state of vulnerability reporting in patch management programs is flawed. We can accept that and also accept the fact that outdated software does lead to increased risk of exploitable vulnerabilities.
It’s important to stay humble.
Patch ≠ Vulnerability
Patch management as a program is sometimes viewed as a subset of vulnerability management1. In my experience, patch management is categorically different. It has a tangible impact on security, but it does not belong to the security practitioner. It’s a program with responsibility and rewards distributed across departments and individual members of staff.
Security doesn’t get to own the concept of “keeping software up-to-date.”
The process of updating third-party software involves a completely different risk profile, prioritization, different parts of the company, and different action items from vulnerability management.
If you get a penetration test done, and it results in a high severity cross-site scripting finding, you’ll want to ingest that into a ticket workflow, share the context with one or more engineers, help them develop a fix, and then re-test the issue to verify the fix.
You start from a vulnerability, and manage that item through a workflow according to your organization’s SLAs.
The scale and complexity of the updates available across your environment requires a different approach for patch management. You start from the strategic question of how you want to update categories of software, how each can be automated, how different inputs will impact prioritization, how decisions can be delegated, and what the process looks like for handling exceptions.
Patches are cattle, vulnerabilities are pets.
A vulnerability reported in a package that you use is unlikely to affect your security posture and it does not automatically warrant individual attention. The marginal risk of that vulnerability affecting you should be addressed wholesale alongside the many other updates you have to manage.
When you have exceptions to your usual update strategy, you can use your vulnerability management workflow to investigate and address those issues on an individual basis.
Good patch management reduces the chance that a real vulnerability is hidden in the noise. But patches aren’t vulnerabilities and managing them requires a different approach.
A patchwork quilt
Your overall patch management strategy should be composed of individual strategies for different kinds of updates. Some common examples:
Production operating systems
Production operating system packages
Production infrastructure applications
Self-hosted third-party applications
Open source application dependencies
Company-owned devices
Employee-owned devices
Network devices
Desktop application extensions
Third-party CI/CD workflows
I’ll write more about all of these next. For now though, I’ll just reiterate a few main points:
It’s easier to solve patch management the sooner you do it.
Patch management is important, despite the low quality noise endemic to the field.
Patch management and vulnerability management are different problems.