It is in this spirit of collaboration, and with all the respect in the world for the incident responders who have had to deal with Log4Shell, that we want to use this event to reflect a bit on the present and immediate future of security as both an industry and a body of knowledge. In my mind, this event can help us sharpen our understanding of two aspects of security: software supply chain risk and Zero Trust.

Seeking Truth Along the Software Supply Chain

All of the various analyses of this vulnerability and its prevalence agree that cleaning this problem up will take some time, due to the combination of the many different ways that threat actors are weaponizing it and the difficulties in ensuring that all environments are completely clean. Bob Rudis, Chief Security Data Scientist at Rapid7, pointed out that “even organizations that have remediated deployed applications might miss some virtual machine or container images that get spun up regularly or infrequently.”

However, I also think it is vital to conceptually distinguish between the future impact of this specific vulnerability, and the future risk of other but similar vulnerabilities, which is what we’re really talking about when we talk about the software supply chain in general. This vulnerability is already bad and will continue to be bad; that doesn’t inherently mean that we are powerless in the face of our own architectural complexities. It’s true that the growing reticulation of software dependencies has made risk cascade in ways we are not yet habituated to predict (or even inventory—more on that later). It is not necessarily true that all is lost. In the middle is the scope for our agency to mitigate this risk.

Earlier this week, as the incident was raging, I revisited a recently published and exceptionally topical report from our peers at Veracode in collaboration with the Cyentia Institute: the State of Software Security Vol. 11: Open Source Edition. The findings about our ability to control supply chain risk are mixed, but they help quantify a problem and show that we can mitigate it, at least in theory. To cherry pick a few significant findings (page numbers in parentheses):

• On the one hand, 79% of libraries are never updated. (13)
• On the other hand, vulnerable libraries are updated faster than non-vulnerable libraries.(15)
• The most frequent explanation for long update times was lack of developer resources. (23)
• In all code scans, no circular update dependency hell scenarios were found… (31)
• …although 13.9% of scanned applications had update chains that terminated in a flawed version. (31)
• Deep transitive dependencies, such as update chains with more than two steps, always terminated in an unflawed version. In other words, as long as the developers who use those libraries just keep chasing down updates, they will eventually succeed. (32)

Veracode characterizes software dependency risk as largely a problem of information. In both the key insights and the conclusions to the report, they emphasize that “developers who have the information they need” remediate third-party vulnerabilities significantly faster than those who don’t (4, 33). Based on the findings above, however, there is a logistical element here as well, and logistical problems are primarily management problems. The mechanisms for controlling software supply chain risk are understood, and often already in place, but they are starved for resources. This gives me the impression that while supply chain risk is increasing, controlling this risk is not a lost cause—it’s just a task that we haven’t properly scoped and prioritized yet. We don’t need to give up on third-party software. To quote Rudis again, “Use [libraries], but verify, and support!”