Lessons from Solorigate – The need for trustworthy development pipelines

This post was quite some time in the making. However six weeks on, the relevance of the lessons to be learned has not diminished.

The Compromise of SolarWinds’ Orion platform calls into question the integrity and trustworthiness of the development process used. The usual approach of detecting the behaviour of the resulting backdoor and blocking it, does not address the underlying trust and integrity issue of the development pipeline. It is also a very costly approach if every customer of a company such as SolarWinds must implement it. A far more effective approach is to invest in the integrity of the development process, to ensure that unwanted code changes in development are prevented.

What happened

At the beginning of December 2020, a supply chain attack of one attacking many organisations globally, including US government agencies and many international institutions hit mainstream media. It appears that through changes in the SolarWinds Orion product, attackers were able to create a backdoor to the product, enabling attackers to infiltrate environments of SolarWinds customers. It appears that the recent attack on cyber security vendor FireEye, resulting in the theft of many of their hacking tools, is the result of just such an infiltration.

The analyses of both FireEye and of Microsoft of the SolarWinds compromise makes for some interesting reading. In short, attackers were able to place code within the Orion product itself, that then was able to open an attack channel for the attackers to exploit in the target companies’ environments. It appears that this was placed so early in the development process, that it became indistinguishable from the legitimate code of the SolarWinds developers and therefore shipped to SolarWinds’ customers as part of the normal distribution and update processes. A significant amount of effort seems to have been put into the code to prevent it being detected, which is an indication that this might well be a state-sponsored attack. At any rate it looks like the attackers knew what they were doing.

First reactions – block, shutdown, and rebuild

First reactions to this attack were to shut down the vulnerable components in SolarWinds Orion both by implementing patches from SolarWinds and also by disabling either the vulnerable code itself, or the communications channels it uses. An Emergency Directive by the US Cybersecurity and Infrastructure Security Agency (CISA) included the requirement to rebuild the affected systems to prevent the attacks from persisting due to yet undiscovered components.

Preventing future attacks – the usual approach

What then followed was the usual reaction by the security industry, of showing how their specific product can, or would soon be able to, discover the behaviour of such backdoors and prevent them from being exploited in future. And whilst this may detect and prevent same or similar attacks in future, it is just dealing with the symptoms of the attack on SolarWinds, not the root cause. I would venture to say that attackers capable of crafting an attack such as this, are also skilled and innovative enough to create further attacks that again will not be detected by the security products of today.

Addressing the root cause – creating trustworthy development pipelines

The root cause of this attack was that the attackers managed to insert malicious code into a vendor’s product, and that this change went undiscovered for months, if not years. It begs significant questions as to the level of trust we can place in the integrity of the development process employed by SolarWinds. And SolarWinds is not alone in this.

In the last few years, a discussion has been gaining momentum around how we can ensure that hardware and software suppliers are trustworthy. The discussion around companies like Huawei and ZTE are a good example of this, albeit only the tip of the iceberg. All vendors of cyber products will need to demonstrate the trustworthiness of their products and the underlying development, manufacturing, and distribution processes. The question must be “Does our product only do what we intend it to do?”.

Even if developer methods are paying a lot of attention to ensuring that the intended changes are free of error and achieve what they are there for, very few are going that extra step of checking that there are no unintended changes bundled in as well. And such inconspicuous, unintended changes may well be so far away, that they escape even the most diligent peer-review processes. The only way to properly pick these up is by continuous monitoring of the code base using automated tools to support the review and release process.

Development teams update their testing and review processes to look for all changes to the code and configuration that occur, and then to ensure that these are justified and correct in terms of the user story being addressed. If these are correctly documented, we will come a long way towards not only creating more stable products with fewer defects causing “collateral damage”, but also to significantly increasing the trustworthiness of the overall process.

Trustworthiness is the new currency

If Solorigate has taught companies and senior managers one thing, it is to show how complex supply chains have become in this digital day and age. The cry has gone out ensure that this “doesn’t happen again”. And the security industry has answered with their usual “buy are tool and you will be safe” mantra. This may provide them with an increased revenue in the short run but won’t stop the next incident that was outside their direct line of sight from happening. It is a recurring pattern that is becoming all too obvious.

The long-term solution is to get the processes of creating our digital products right and ensuring that they are not circumvented. This is how we (re-)build the trust in our products. Companies like Huawei, Solarwinds, or ZTE are already today experiencing what it means to not have the trust of their customers, and that list is sure to grow. To come out on top, companies will need to put in that little bit of extra effort to ensure the trustworthiness of their development cycle. And the best time for that is now.