• Cloudflare: On February 1, Cloudflare announced it had detected a threat actor on its self-hosted Atlassian server on November 23. Although the primary point of compromise in this incident came through account credentials that Cloudflare failed to rotate after an Okta compromise, the company said the threat actor attempted to gain access to a non-production console server in its São Paulo, Brazil, data center due to a non-enforced access control list. The threat actor was denied access and could not access Cloudflare’s global network.

• First American Financial: On December 29, 2023, First American Financial reported to the US Securities and Exchange Commission (SEC) that it had identified unauthorized activity on certain information technology systems. While providing few details about this incident, First American said it “believes the perpetrator of the activity accessed certain company systems, exfiltrated data, and encrypted data on certain non-production systems.”

• LastPass: On March 21, 2023, LastPass announced the results of its investigation into two major cybersecurity incidents, reporting that an unknown threat actor “exploited a vulnerability in third-party software, bypassed existing controls, and eventually accessed non-production development and backup storage environments.”

Real-world data can be found in non-production systems

One primary risk of insecure production systems is that threat actors can gain access to sensitive data such as encryption and access keys, passwords, knowledge of security controls, or intellectual property that could prove to be a goldmine for further exploitation.

“I think on the CISO and BISO [business information security officer] side of things, there are some fundamental truths that we can acknowledge about these environments that maybe not everyone is willing to admit, which is that oftentimes, development environments include a ton of materially significant intellectual property,” Andrew Krug, head of security advocacy at Datadog Security Labs, tells CSO. “You could have the best development practices and hygiene in the world. Some of your actual real data is going to make it in there at some point.”

Cost savings and complexity often kick in

However, many companies don’t necessarily have the best security practices regarding test environments and other non-production systems, often due to cost-saving measures. With the advent of cloud computing, “A lot of companies broke apart their infrastructure into at least development test production, and then they would have a security account,” Krug says. “Unfortunately, most of the cloud cost models they subscribed to for their vendor management or security platforms didn’t really scale with that segmentation. So, they just opted out of different resources and different things from monitoring” to save money.

“And I don’t just mean security monitoring; I mean all kinds of monitoring,” Krug says. “This is almost like a company culture question more than a legal or regulatory question: How high a value does that company hold for security best practices?”

Staff shortages make securing non-production systems a challenge

Even companies like Microsoft and Cloudflare, which aren’t likely to skimp on security spending, experience challenges in extending robust security measures to their non-production systems. “Cloud environments are getting more and more complex, and it just becomes more and more challenging to have the right governance to observe across all” of the components, Krug says. “We could probably say as we onboard more services and more complexity, it just gets harder and harder to know even what the right things are to observe.”

The lack of available cybersecurity talent only makes analyzing the complexity harder. “We could talk about the cyber skills shortage and that even if companies that are the size of Microsoft and CloudFlare and First American want to hire the right talent, they may not be available,” according to Krug.