Software composition analysis (SCA) is the capability we’re talking about here. It’s about looking at the open source components used in your application, and analyzing them for security flaws.

Specifically, in this post we’re going to focus on open source libraries, not operating system packages.

In most modern startups, open source libraries are a good place to start your patch management program. You can reliably check compliance boxes using software composition analysis1. Bugs in open source libraries are also more likely to affect your product than bugs in operating system packages running on servers or outdated software running locally on employee workstations.

Identify

The first goal of our software composition analysis program is to inventory all of the open source libraries used in our product development. We want our inventory to include the following context, or have it readily available:

• Packages

• Versions

• Call sites

• Known vulnerabilities

• Available updates

• Dependency hierarchy

• Development or Production

In practice, this is a wishlist and not a checklist. The tools I’ll mention later implement some of these, but aren’t perfect and the relative lift of building something with all of my desired attributes probably just isn’t worth the lift for most companies.

Protect

Let’s now assume that we have a list of potential updates that we could make. This is where reasonable people can decide to take very different paths.

If we automatically update all of these libraries to their latest release, we can declare “Mission Accomplished” on software composition analysis and move on. The reason I’ve never heard of a company doing this is that updates are often liable to introduce breaking changes, leading to severe bugs in your application.

In order to automatically apply updates, we need robust detection of breakage. Strong unit test coverage, integration tests, canary deployments, and automated rollbacks could give us the confidence we need to move forward with this solution. When breakage is detected, an engineer can step in to investigate the cause and remediate.

I suspect that very soon AI tooling will be available to preemptively detect breaking changes and modify our code to prevent issues.

If you think the automated approach might be feasible for your company, I encourage you to try it. I can’t say that I have personal experience with it, and it will almost certainly lead to surprising edge case problems that will require creative solutions. Please do it, solve those problems, and share what you learn. The industry would be much better off if we could just update all of our software without fear.

If you can carve-out a portion of your application dependencies where updates can be safely automated, that will reduce the load on the rest of this program and is still worthwhile. It can be expanded over time and lead you to gradually solve for edge cases instead of taking them on all at once. Start with one package and go from there.

Solving for automated library patching all at once may be too risky, and too much of an investment to be feasible. No problem! We just need a way to prioritize and address our list of available updates.

The following attributes are worth considering in a hypothetical prioritization algorithm:

• Number and severity of known vulnerabilities

• Reachability of known vulnerabilities

• Number of versions between current and latest

• Second-order dependencies updated in recent versions

• Use in development versus production

• Processing of untrusted input

• Exploit prediction score of known vulnerabilities (EPSS)

• Likelihood of breaking changes

• Level of maintainer activity

• Features and improvements added in newer versions

I don’t know of any products on the market that use all of these inputs to help you make decisions. There are a handful that use some of them, and they’re worth looking at to implement your program.

I recommend using a product for both building your inventory and prioritizing updates. Manually implementing this stuff would be a significant lift. Even a product that doesn’t cover everything I’ve mentioned can be great for starting your program.

Note: I don’t have a financial interest in any of these companies, or any hidden economic incentive to recommend them over other tools

I recommend several of them to Observa clients, and want to maintain positive working relationships with these companies, so I won’t share pricing information here. In my experience, none of these are significantly out of sync with the others on pricing, though some may be less expensive for a given customer than the others depending on specifics.

Coana and Semgrep Supply Chain both use static analysis to determine if a known vulnerability is reachable from your application2.

EdgeBit uses runtime analysis to determine reachability. An agent checks to see if specific files are loaded when the application is actually running and uses this context for vulnerability prioritization or suppression. They’re also experimenting with EPSS to include exploit prediction scores as an input to prioritization.

Socket.dev is focused on a slightly different problem3. They do list vulnerabilities, but their primary value proposition is looking for malware in these open source libraries. Takeovers, troll packages, typosquatting, etc. I like this product and it’s a great complement to an SCA tool with reachability analysis and other prioritization features.

A shortcoming of all of these tools is that they are solely focused on the security aspects of patch management. I’d like for a tool to consider other factors, like maintainability and improvements to make this program more holistic.

Bringing this back around to practical advice for your program:

1. Consider how much of this you can automate.

2. Choose an SCA tool that you like, implement it, and check-in on it regularly to apply the highest priority updates.

From there you can work with your engineering team to decide what other factors you’d like to prompt package updates. This will give you a strong foundation and let you move on to other aspects of your patch management program.

Managing updates for the apps on my own personal phone is complicated. There is no company too small to run into overwhelming complexity as they try to manage the patches to all manner of software being used across their environment.

This is the first in a series of essays diving deep on how startups keep their software up-to-date. In this post I’ll talk about my general philosophy toward patch management.

Important, not urgent

Patch management is often a low priority for companies, for very reasonable reasons:

1. It is a complex problem with a relatively high cost to get right.

2. The benefit of applying a given patch is usually minimal at best.

3. The risk of an update breaking important software is greater than zero.

Patch management is a typical example of a problem that is “Important, not urgent.” Those sorts of problems are hard to solve when you have plenty of “Important and urgent” problems on the list.

Maybe a little urgency, as a treat

These are also the sorts of problems I find really interesting, because they are great candidates for setting aside time to “solve it right, once and for all” and then forgetting that other people have such problems.

Patch management gets increasingly difficult to solve as time goes on. Systems become more complex, human behavior becomes canon, and the activation energy required to make changes only grows.

For this reason, I would argue that there is at least some urgency to solving patch management.

Nuanced, but important

Security vulnerabilities are sometimes among the bugs fixed by software updates. Leaving these in place can increase the risk of a successful data breach or other abuse by bad actors.

With that said, the vast majority of vulnerabilities reported in software packages for a given business are not exploitable in any meaningful way. There is an epidemic of noise coming from software composition analysis tools which provide very little actionable data.

Traditionally, you might prioritize patching areas of your IT infrastructure that are more likely to be exploitable, such as “internet-facing” systems. The typical modern cloud-based startup has an architecture that is so far removed from the previous generation of production IT that concepts such as “internet-facing” systems are barely recognizable.

If all of your traffic is run through AWS-managed load balancers to AWS-managed servers running AWS-managed virtual machines and AWS-managed Kubernetes clusters, which is finally running a Docker container with your code, do you have any “internet-facing” systems? Who knows!

Does an outdated version of Vim in the Docker container on the Kubernetes cluster run by the hypervisor, on the server behind the load balancer actually put you at any risk at all?

The best answer I have today is “No, probably not.”

The world is full of nuance though, and these are incredibly complex systems. To completely ignore the problem of patching software in these cases would leave us with no margin for error. No buffer to account for the fact that unexpected things happen all the time.

I’ve been surprised to find vulnerabilities in outdated packages lead to real exploitable issues with minimal required effort. If I had let my overconfidence run wild, I wouldn’t have even checked and the risk of an incident would have been higher.

The current state of vulnerability reporting in patch management programs is flawed. We can accept that and also accept the fact that outdated software does lead to increased risk of exploitable vulnerabilities.

It’s important to stay humble.

Patch ≠ Vulnerability

Patch management as a program is sometimes viewed as a subset of vulnerability management1. In my experience, patch management is categorically different. It has a tangible impact on security, but it does not belong to the security practitioner. It’s a program with responsibility and rewards distributed across departments and individual members of staff.

Security doesn’t get to own the concept of “keeping software up-to-date.”

The process of updating third-party software involves a completely different risk profile, prioritization, different parts of the company, and different action items from vulnerability management.

If you get a penetration test done, and it results in a high severity cross-site scripting finding, you’ll want to ingest that into a ticket workflow, share the context with one or more engineers, help them develop a fix, and then re-test the issue to verify the fix.

You start from a vulnerability, and manage that item through a workflow according to your organization’s SLAs.

The scale and complexity of the updates available across your environment requires a different approach for patch management. You start from the strategic question of how you want to update categories of software, how each can be automated, how different inputs will impact prioritization, how decisions can be delegated, and what the process looks like for handling exceptions.

Patches are cattle, vulnerabilities are pets.

A vulnerability reported in a package that you use is unlikely to affect your security posture and it does not automatically warrant individual attention. The marginal risk of that vulnerability affecting you should be addressed wholesale alongside the many other updates you have to manage.

When you have exceptions to your usual update strategy, you can use your vulnerability management workflow to investigate and address those issues on an individual basis.

Good patch management reduces the chance that a real vulnerability is hidden in the noise. But patches aren’t vulnerabilities and managing them requires a different approach.

A patchwork quilt

Your overall patch management strategy should be composed of individual strategies for different kinds of updates. Some common examples:

• Production operating systems

• Production operating system packages

• Production infrastructure applications

• Self-hosted third-party applications

• Open source application dependencies

• Company-owned devices

• Employee-owned devices

• Network devices

• Desktop application extensions

• Third-party CI/CD workflows

I’ll write more about all of these next. For now though, I’ll just reiterate a few main points:

1. It’s easier to solve patch management the sooner you do it.

2. Patch management is important, despite the low quality noise endemic to the field.

3. Patch management and vulnerability management are different problems.

Risk scenarios

Personal email on work machines

Email was the second most prevalent vector for breaches in the 2023 Verizon DBIR1. It is also ~necessary for everyone to have it on their work machines.

Having personal email on a work machine increases the risk of an email-based attack succeeding against a startup. It’s not obvious why this is the case, but there are some specific risks that apply.

Corporate email systems can implement detection and prevention controls for malicious emails. These same controls can’t be implemented on personal accounts. This means you have to rely on built-in spam filters, which are far from perfect for this use case.

Personal inboxes also receive different types of emails than corporate inboxes on a normal basis. This expands the range of plausible pretexts which can lead to a malicious email being acted upon from a corporate device.

Sometimes attackers will compromise one person’s email account and respond to old conversations in their inbox with attempts to compromise the counterparties. The pretext of the existing thread makes the attempts more convincing. This is called business email compromise (BEC).

Checking personal email from a work device increases the number of existing conversations and counterparties who could be compromised and used as a vector for attacking corporate accounts through BEC.

Work email on personal machines

When someone logs into corporate email accounts from a personal device, they expose the contents of that account to software running on their personal computer.

Malware and malicious browser extensions can compromise sensitive work emails, and this access can even be leveraged to gain access to other systems (e.g. anything using Login with Google).

Enterprise security controls like antimalware, endpoint detection and response, secure DNS, application allowlisting, and more aren’t running on personal computers and will not have any ability to detect or prevent an attack.

People typically have their personal email on their personal computers as well. This presents a direct line of attack to corporate systems which completely bypasses all detective and preventive controls.

Personal GitHub in work organizations

When a new employee joins an organization that uses GitHub, it is common for them to have their personal GitHub account added to the organization’s account. It is less common for them to create a new GitHub account dedicated to that particular company. GitHub recommends that people have one account used for both personal and work contexts2.

Bad passwords and SMS MFA

Any account that requires a password is liable to have an insecure password. Password reuse is common, and old passwords are often leaked in major data breaches3.

Some accounts may not have MFA enabled, and those that do may use less secure methods such as SMS codes.

Insecure passwords and MFA configurations are risks that apply equally to personal GitHub accounts and accounts created specifically for an organization. There doesn’t seem to be anything special about personal accounts that make these risks more pressing.

Old SSH keys

You can also authenticate to GitHub with SSH keys. Each user has the ability to create as many SSH keys as they’d like. It’s common practice to create one per device.

The upshot here is that personal GitHub accounts have many SSH keys, including on old servers and devices that are long forgotten. It can even include servers and devices from previous employers. When this account is granted access to your organization, those keys all have access now too, with no second factor.

Requiring employees to create separate accounts for work ensures that you're starting from a clean slate and you won't have this long tail of SSH keys being granted access to your organization.

Personal access tokens

The third authentication scheme for GitHub is personal access tokens. There are two types: classic and fine-grained. Neither of these requires a second factor to authenticate, regardless of the normal account settings.

Classic personal access tokens grant the holder access to everything the associated user can access. When the user is added to a new organization, existing tokens now have the same permissions. These tokens are often used in scripts running in any number of places (e.g. CI/CD). Requiring new work-only accounts doesn’t carry the risk of old tokens being used to access your organization’s assets.

Fine-grained personal access tokens have the benefit of additional scope limitations and security requirements. They are limited to specific users or organizations, so adding a personal account to your organization won’t result in old fine-grained tokens having access to your organization. Any tokens will have to be created specifically for your organization, and the associated risk applies to both personal accounts and new work accounts.

Dotfiles and Codespaces

If your organization uses GitHub Codespaces, users can choose to automatically install their own dotfiles in Codespaces4. GitHub lets them choose an arbitrary repository where it will look for dotfiles to install whenever a new Codespace is created.

Dotfiles have the ability to run arbitrary scripts. Some users choose a personal repository where they keep their preferred dotfile configurations. An attacker who can modify the personal repository will have the ability to run arbitrary code in your Codespaces. Codespaces often have access to private code, and sometimes live systems. This can be used to advance the attack into the organization.

GitHub only allows selection of a repository owned by the user5. This means that requiring a new work account for your organization reduces the risk of a personal repository with different contributors, access patterns, etc. being used to access your resources.

Some of the controls we will discuss later in this essay apply to organization resources, but not personal repositories. Using dotfiles in this way bridges the gap and allows an attacker to pivot from the latter to the former.

GitHub summary

These risks are more potent when you add existing GitHub accounts versus asking new team members to create new accounts:

• Old SSH keys

• Personal access tokens (classic)

• Malicious dotfiles run in Codespaces

These risks are about the same either way:

• Bad passwords and MFA

• Personal access tokens (fine-grained)

We’ll discuss the controls available through GitHub Enterprise in a bit, which can change the conclusion a bit if that’s an option for you.

Potential controls

SWG and CASB

Secure web gateways (SWG) and cloud access security brokers (CASB) can be used to enforce granular controls on which accounts staff can access from their work machines. They essentially work by proxying all web traffic and blocking attempts to access email or file-sharing accounts that are not explicitly approved.

These tools can prevent access to personal email from work machines, reducing the risk of social engineering or malware attacks which would otherwise be caught by work email filters. They can also block attempts to access malicious websites or certain content categories like illegal streaming or file-sharing sites.

The downside of an SWG is that it will be perceived as heavy-handed and intrusive by some of your colleagues. Some of these tools proxy traffic through a remote server, resulting in increased latency for web traffic.

Conditional access

Conditional access is a way of only allowing access to corporate resources if certain conditions regarding the attempt are true. You can restrict IP addresses, enforce that only corporate devices are used, and even that antivirus software is running or the browser and operating system are up to date.

Conditional access can be used to prevent access to company email accounts from personal devices. This requires a slightly more advanced IDP / SSO / MDM setup than many startups have, but is a great security control.

There are many benefits to conditional access with more or less friction added for end users, but for the purposes of this essay if you have the tools to set it up, it’s low friction for users to require that you come from a corporate device.

You can choose to make exceptions for mobile phones, since they aren’t typically vulnerable to the same kinds of malware as desktop / laptop workstations. With mobile phones the primary risk is that terminated employees may retain access to corporate email.

GitHub

Many of the security controls available to safely implement GitHub’s recommendation of granting personal accounts access to your organization require GitHub Enterprise.

The Enterprise plan costs $21 / month / user versus $4 / month / user on the Team plan at the time of writing6. It includes both the ability to require separate SSO login on a personal account when accessing work repositories, and the ability to enable Enterprise Managed Users, which are fully owned corporate accounts provisioned from the identity provider.

It would be a better fit to have the extra login setting available on the Team plan, and reserve Enterprise Managed Users for the Enterprise plan.

See my essay on this general subject:

Summary of available GitHub controls

• Require SSO to access corporate resources from personal accounts

  • Availability: GitHub Enterprise

  • Gaps: Personal repositories can be used for dotfiles on Codespaces, SSH keys, classic personal access tokens.

• Enterprise Managed Users

  • Availability: GitHub Enterprise

  • Gaps: Stolen cookies from malware, stolen personal access tokens.

• IP Allowlisting

  • Availability: GitHub Enterprise

  • Gaps: Malware accessing GitHub from compromised endpoints (“living off the land”)

• Require MFA to be added to the organization

  • Availability: All organizations

  • Gaps: Insecure SMS MFA, SSH keys, classic personal access tokens

• Restrict access from classic personal access tokens

  • Availability: All organizations

  • Gaps: SSH keys created by classic personal access tokens will still work7. Many organizations use classic personal access tokens out of necessity and can not disable them across the board.

The most secure configuration for an organization with GitHub Enterprise would be to use Enterprise Managed Users and an IP allowlist. This would require any attacker to steal session cookies or personal access tokens and execute the attack from an endpoint within the IP allowlist.

The most secure configuration for an organization without GitHub Enterprise would be:

• Require MFA

• Restrict access from classic personal access tokens if possible

• Ask new team members to create a new account for their work context; or

• Ask new team members to review their account’s SSH keys and classic personal access tokens before being added to the organization and not to use personal dotfiles repositories for Codespaces.

The objection is typically focused on guilting or shaming companies into providing a certain set of features to all of their customers. This ethical frame isn’t an effective tool for persuasion.

B2B software businesses have an interesting problem. It is often difficult to quantify the value they provide to a given customer. 

Some customers get more value from their software than others. This means that they’d be willing to pay more, and the business selling that software would like to charge them more as a result. Fair enough.

Usage-based pricing is one way to solve this, but not every product has a measurable usage metric that directly correlates with value and willingness-to-pay.

Seat-based pricing is a classic solution. The problem is that the number of people logging into the product doesn’t always correlate to value either. Software used exclusively by the CFO might provide a lot more value to a public company than a Series B startup.

You could ask companies how much money they make, and then adjust your price based on the answer. This would be a pretty efficient solution to customer segmentation, but is typically received poorly.

The premise is that companies make money, and the more money they make the more value they get from improvements. Your tool improves them, so they should pay accordingly. This strikes me as true and reasonable.

It also subjectively feels unfair. Possibly because the same formula applied to consumer products wouldn’t support the same conclusion. Elon Musk doesn’t get more value buying chips at the supermarket than me. On the other hand, SpaceX would get more value from most B2B software than my small consulting business.

Software businesses get creative to find metrics that correlate with value provided, and feel subjectively fair to buyers. When done right, it convincingly mimics usage-based pricing even though the metric doesn’t directly measure usage per se.

In the security world, static analysis tools often charge based on some version of “number of people who contributed to any of your git repositories last month.” This feels like usage-based pricing, because git commits tend to get analyzed by static analysis tools and more contributors probably correlates to more commits.

Instead of charging by number of commits, they insert a layer of abstraction. They don’t want to charge a business with one engineer more because that engineer is committing twice as much, and causing them to run twice as many scans. They want to charge more for companies that make more money.

The number of contributors is a proxy for how much money the company makes. It’s a good proxy too, at least for the kinds of companies that buy static analysis tools1. Clever!

What happens if you don’t have a clever proxy that feels fair and correlates to how much money your customer makes?

Alternatively, what happens if you do have a metric like number of users, but it’s only one dimension of the value you provide. Some customers with 100 users actually make more money than other customers with 100 users.

You’ll probably turn to Good, Better, Best pricing. You segment your customers based on how much money they’re willing to pay you, and identify which features are important to each group, and less important to the groups with less buying power.

This is not a very elegant solution. Even so, it can solve the dual problem of correlating to value provided and feeling subjectively fair; the bigger company is paying more, but they have a longer feature list.

The cost of providing that longer feature list to everyone is usually negligible, but it would remove the feeling of fairness that enables you to close the bigger deals. The art is in finding features that smaller customers don’t really care about, so that you can make the bigger customers feel good without degrading your product for everyone else.

There are at least two ways this can go wrong:

1. The most valuable features for customers of any size are intentionally paywalled, meaning everyone but the enterprise customers has an artificially degraded product experience (and knows it).

2. The company is wrong about what smaller customers care about and unintentionally withholds important features, resulting in the same negative outcome for most of their customers.

These failure modes are at the core of the SSO tax debate. Software companies often believe that only enterprise customers care about security features like SSO. Security people at smaller companies believe SSO is important and don’t like important features being artificially withheld2.

These companies aren’t intentionally withholding important security measures from smaller companies. They aren’t saying things like “passwords are only hashed if you’re on the Best plan, don’t forget to upgrade to the salted hash add-on.” There’s a discrepancy between their model of the world and that of the security people.

This isn’t an ethical issue, it’s a matter of recalibrating our models. We can find some security features that are valuable to customers of any size, and some that really only provide value to certain groups.

Keep in mind, the goal here is to make the companies paying more feel like they’re getting a fair shake. The goal is not to generate revenue directly from these features.

Here I’ll provide a bunch of specific examples, and my personal perspective on how they fit into this conversation.

Sign-in With Google

Everyone finds this valuable. It’s amazing for security as well. Google is highly configurable, and even if it’s slightly inconvenient to have some tools on a different provider like Okta and some using Google, it’s a reasonable level of holistic security.

Sign in with Slack, GitHub, etc.

These tools have SSO and similar taxes themselves, so they aren’t as secure as the ability to sign-in with Google. If supported, they should be optional alongside Google.

SAML SSO

Many companies put this behind the Best plan but the customers in Better want it too. It’s reasonable not to include it in the Good tier though.

Configure available login methods

Any time you support multiple login methods, you should give administrators the ability to enable or disable them. Otherwise it negates the benefits of having secure options, because the bad options are still available to attackers. This isn’t a good candidate for gating behind higher tiers.

SCIM Provisioning

Fewer companies in Better want this than SAML SSO. You can probably get away with gating it in the Best tier, and privately granting any Better company’s request for this feature to be turned on. Nothing wrong with providing it to both tiers explicitly though.

Single Logout

This is a feature that isn’t as commonly implemented. Any tier with SAML SSO should include this as well if you do build it. Make sure to list it as a separate, adjacent feature to capitalize on your work.

Audit Trail

Companies in the Better and Best tiers may care about this during purchasing. Companies in the Good tier will care about this if something unusual happens to their account.

You could leave this off of the Good tier, but still record all of the events. You probably want to do this anyway so that someone upgrading can see historical data as well. If a customer suspects an issue and writes in, support should have the ability to enable the feature for a month.

A stronger way to segment within this feature is to increase the retention period for each successive tier. One week for Good, two months for Better, three years for Best. Those numbers are arbitrary, but you get the idea.

Audit Trail Export to SIEM

This is a great candidate for Best-only customers. Customers below Best have a SIEM, but they’re far less likely to be in a position to use the data from all of their SaaS apps. This is just lower on their priority list than other SIEM tasks.

IP Allowlisting

If you offer the ability for customers to set an IP allowlist, it’s possible to gate this behind Better or Best, but it’s another one you should flex on if a company requests it. It does risk generating some negative feelings though, so the safe move would be to offer it to everyone.

Role Based Access Control (RBAC)

Your application should have some concept of roles in all tiers. This depends on the specifics of how people typically use your application. Generally though, not everyone should be able to do everything.

You can allow enterprise customers in the Best tier to build their own roles with customizable permissions.

Passwordless (WebAuthn)

WebAuthn provides phishing-resistant multi-factor authentication. It also lets customers login without passwords (or magic links).

This is just a far better login experience in addition to being much more secure. This should be the standard for all products, everywhere.

Okay, with all of those examples, what do our tiers look like?

Good

• Sign-in With Google, Slack, GitHub

• Configure available login methods

• Audit trail with 7 days of history

• IP allowlisting

• Passwordless WebAuthn

• RBAC with built-in roles

Better

• Everything in Good

• SAML SSO (Single Sign On)

• SAML SLO (Single Logout)

• Audit trail with 60 days of history

Best

• Everything in Better

• SCIM Provisioning

• Audit trail with 3 years of history

• Export audit trail to SIEM

• Customizable RBAC

Reasonable!

If you listen to feedback from customers and adjust over time, you’ll maintain a segmentation that roughly approximates your customers' mental models.

There will always be edge cases. Some small companies care a lot more about security than companies an order of magnitude larger. Since these features are being withheld arbitrarily, it’s understandable that they may be frustrated with you.

You can bundle advanced security features together as a separate line item in your sales process and make them available to anyone who wants it for an additional fee. Then you can offer it as included for the higher tiers.

For example:

Advanced Access Security

• SAML SSO (Single Sign On)

• SAML SLO (Single Logout)

• Audit trail with 60 days of history

• SCIM Provisioning3

Now, the Better and Best plans can have a big line item that says Advanced Access Security is included.

The shelf price of this package should be high enough that Better and Best companies feel like they’re getting a fair deal by having it included. It doesn’t have to be high in practice for the Good companies though.

You can aggressively discount the Advanced Access Security product for smaller companies that care a lot about security, while maintaining a sense of fairness among all of your customers. You could even have a note on your pricing page saying small businesses are eligible for large discounts on the package.

The risk is that some customers will see all of this math as advanced tomfoolery and be wary of your pricing model. You can’t please everyone.