As the industry has gotten better at locking down endpoint devices such as laptops, threat actors are finding new ways to infiltrate an organization’s systems—and they’re seeing more and more success.

From the widespread compromise of Ivanti VPNs to a string of cyberattacks involving vulnerabilities in on-premises firewalls, this year has seen threat actors increasingly exploit the very devices that are supposed to be protecting us.

With the industry’s improved proficiency at locking down endpoint devices such as laptops, hackers have turned their attention toward network security systems—whose position as the front door to the IT environment makes them a prized target.

Among other things, the fact that security devices could now be seen as the weak link in cyberdefense is “incredibly ironic,” said Geoffrey Mattson, CEO of Palo Alto, Calif.-based access management vendor Xage Security. “It’s the security devices that are making us less secure. And it’s the access devices providing access to the bad guys.”

At this point, IT and security teams as well as MSPs are all too familiar with the routine: Vulnerability is discovered. Patch now. Repeat.

But for these overwhelmed, understaffed teams, moving to quickly deploy patches has proven to be a massive challenge, sometimes leaving organizations vulnerable to ransomware, data theft or espionage for extended periods of time.

“We’re so far down this road, I think we know what the culture of patching is,” said MacKenzie Brown, vice president of security at Ellicott City, Md.-based managed detection and response provider Blackpoint Cyber. “Understandably, a lot of organizations don’t have the time and resources to patch as quickly as vulnerabilities are disclosed out in the wild.”

And that’s in the best-case scenario, in situations where the fixes are even made available in rapid fashion by vendors. And that hasn’t always been the case lately when it comes to network security devices under active attack.

Meanwhile, hacker targeting of security software and devices has been climbing. Exploits that impacted firewalls and other network infrastructure devices grew to comprise 11 percent of all vulnerability exploits in 2023, nearly quadrupling the percentage from the previous year, according to research from San Jose, Calif.-based connected device security company Forescout.

“Network devices like routers, firewalls, VPNs—those devices and those applications are being targeted across the board from all kinds of attackers,” said Elisa Costante, Forescout’s vice president of research. “We really see a shift toward those as a primary target at the moment.”

Likewise, researchers at Google Cloud-owned cybersecurity firm Mandiant, based in Reston, Va., reported seeing the exploitation of a total of nine vulnerabilities that impacted security products in 2023, nearly doubling from five in 2022. Jurgen Kutscher, vice president at Mandiant Consulting, believes it’s now clear that “more work needs to be done in that space.”

“I think the industry indeed needs to put a lot more focus on making sure that the devices we trust the most—that we need to defend ourselves—are as secure as they possibly can be,” Kutscher said.

And yet, the capabilities do already exist for automated patching of network security devices in the majority of environments, the same way that smart-home devices today will often receive updates automatically. Many vendors and customers, however, are wary of this approach due to concerns about unintended IT disruptions or loss of control, according to industry experts who spoke with CRN.

But at least one major security vendor, Sophos, has broadly implemented auto-updating of its most popular on-premises firewalls in response to significant threats without causing the sort of disruptions that many fear. Some in the security industry say the time has come for this type of approach to be offered more widely.

“Why in the world are we here in 2024, and these devices aren’t auto-patching, aren’t taking care of themselves?” said Kyle Hanslovan, co-founder and CEO at Columbia, Md.-based managed security platform provider Huntress. “Why are we waiting?”

Exploits On The Rise

On Feb. 1, the U.S. Cybersecurity and Infrastructure Security Agency stunned many in the industry with an urgent order to civilian executive branch agencies: Disconnect your Ivanti Connect Secure VPNs within 48 hours.

The extreme measure followed the mass exploitation of three flaws in Connect Secure, following the initial disclosure Jan. 10. Researchers said thousands of Ivanti VPN devices were compromised during the attacks, with CISA itself among the victims.

Others ensnared in the attacks included Mitre, a major provider of federally funded R&D and the promulgator of a cyberattack framework that’s become ubiquitous in the security industry.

While CISA’s order was only meant as a temporary measure, the move to disconnect Connect Secure VPNs underscored the seriousness of the threat posed by a China-linked hacker group believed to be exploiting the vulnerabilities in the devices. It also pointed to the infamous difficulty of deploying security fixes in a timely manner, even in dire situations.

“There’s just limited capacity for an internal security team to apply attention to multiple overlapping issues across many different products,” said Caleb Gross, director of capability development at Tempe, Ariz.-based offensive security company Bishop Fox.

Research from the company over the past year, for instance, has highlighted other cases where hundreds of thousands of Fortinet Fortigate devices and SonicWall next-generation firewall devices remained unpatched against major vulnerabilities, long after fixes were made available by the vendors.

“CISA is recognizing this trend—that [organizations] aren’t able to completely follow up and patch,” Gross said. “So now their strategy is, ‘All right, just pull this stuff off your network. You’ve got 48 hours.’”

Alongside the hacks targeting Ivanti Connect Secure customers, attacks exploiting zero-day vulnerabilities in on-premises firewalls have been a frequent occurrence in recent months.

In February, CISA disclosed that a “critical” vulnerability impacting numerous versions of Fortinet’s FortiOS operating system was seeing exploitation in attacks. Then in mid-April, Palo Alto Networks disclosed a maximum-severity vulnerability affecting several versions of its PAN-OS firewall software, which had already seen exploitation at the time of the disclosure.

Less than a week later, Cisco Systems disclosed a high-severity vulnerability impacting its Integrated Management Controller used by numerous network devices, and code that can be used to exploit the issue was publicly released. A series of other high- and critical-severity vulnerabilities have also been disclosed impacting firewalls from numerous vendors during the first half this year alone.

Without a doubt, “we’re seeing a ton of these vulnerabilities in network edge devices, including many network appliances and security gateways,” said Caitlin Condon, director of vulnerability research and intelligence at cybersecurity company Rapid7.

Increasingly, the vulnerabilities are coming to light as zero-day exploits, previously unknown issues that have already been utilized in attacks. The vulnerabilities are termed “zero days” because vendors have effectively had no time to develop and distribute fixes.

“What stands out to me more than anything is that we’re seeing a high incidence of zero-day exploitation for these devices,” Condon said. “It’s actually been more common over the past year for us to see these vulnerabilities in network edge devices be disclosed as zero days.”

In other words, it’s not just that there is limited time for already overburdened security teams to respond to threats—more and more often, there is no time to react.

All of which makes it even more crucial for vendors to move as quickly as possible to get patches out, according to industry experts. But it hasn’t always worked out that way, perhaps most prominently in the Ivanti Connect Secure attacks.

“[Ivanti’s] response to a new vulnerability being announced was, ‘Hey, there’s a patch coming—not in a matter of days, but in a matter of weeks.’ That’s concerning,’” said Bishop Fox’s Gross. “You want to see more of a sense of urgency.”

And while Ivanti did make mitigations available in the meantime, they were only available to customers registered in their support portal, he said.

“It’s not the kind of thing where they made it freely available,” Gross said. “I think our expectations should be higher for a vendor response. I would have wanted to see them widely, freely distribute the mitigation so that users of that software or device can just mitigate as fast as possible. But that’s not what we saw.”

In a statement provided to CRN, an Ivanti spokesperson said that the security of customers is a top priority and that the company works “quickly and transparently to provide information in their best interests and release fixes when necessary.”

“In this case, we prioritized mitigation releases as patches were being developed, consistent with industry best practices. We are taking proactive measures to improve our overall security posture and practices to combat the increasingly sophisticated threat environment our industry is facing, and are actively engaging with our customers and government and industry partners in this effort,” Ivanti said in the statement. “It is standard industry practice to provide mitigations and patches via a support portal, including for security reasons to prevent threat actors from using the information to reverse-engineer fixes.”

Ivanti released the first patch for some versions of its Connect Secure VPN software on Jan. 31, three weeks after the initial vulnerability disclosure.

Executives at security vendors and solution providers told CRN that the impacts of the Ivanti attacks were widely felt, in some cases prompting hurried migrations to alternative remote access technologies.

At BlueAlly, a Cary, N.C.-based MSSP, one large enterprise customer in the biotech industry was “a big Ivanti shop” prior to the disclosure of the Connect Secure vulnerabilities, said Hooman Mohajeri, vice president of security services for BlueAlly.

“When these vulnerabilities started coming out, I was very surprised at how quickly they ripped [the Ivanti VPNs] out,” said Mohajeri, who noted that BlueAlly hadn’t been involved in the initial deployment of the Ivanti VPNs.

Other makers of network security devices have received high marks for their handling of critical exploited vulnerabilities.

In the case of Palo Alto Networks and the PAN-OS flaw disclosed in April, the vendor responded quickly and has been open with sharing information, Rapid7’s Condon said.

The issue was discovered by researchers at cybersecurity firm Volexity, which also had found evidence of exploitation of the vulnerability. In response, the advisories from Palo Alto Networks “were transparent with that information, even before patches were fully available,” Condon said.

The vendor also provided mitigations in connection with the vulnerability disclosure, she noted—and the message was essentially, “‘Mitigate now, and we’ll get you the patches as soon as we can.’”

Palo Alto Networks then followed through by releasing the first set of patches on a Sunday, just two days after the initial disclosure of the vulnerability. “Clearly they worked through the weekend to make that happen,” Condon said.

As solution providers and customers take stock of the heightened threats against network security devices, it’s valid to assess products based the number of critical vulnerabilities and how vendors handle the discovery of these issues, according to Neil MacDonald, vice president and distinguished analyst at research firm Gartner.

“No one is immune from this problem, but some vendors are better than others,” MacDonald said.

In reality, “all of the vendors are guilty—all of them. Because none of them produce an absolutely secure product,” he said. “So part of the evaluation process of looking at these vendors would be, what’s their track record over the past two or three years? What did they have in terms of critical, exploitable vulnerabilities? Are they going up or are they going down [on critical vulnerabilities], or staying the same? And what is their time to disclose and time to deliver a patch?”

Security experts told CRN there’s no indication that attacks targeting firewalls and VPNs are likely to de-escalate anytime soon.

For one thing, such devices are appealing targets because they must be connected to the internet, said Forescout’s Costante. And for a hacker, “once I am within a firewall or within a router, or within a VPN system, I’m in a very good place to start [an attack],” Costante said.

Another major advantage is that for many externally facing network devices, there are usually administrative privileges on them as well, said Blackpoint Cyber’s Brown. “That’s going to be the goldmine.”

By compromising these devices, hackers “already have privileged access, and they don’t need to do a lot more lateral movement or persistence internally to the environment,” she said. “They can move directly to where they want to go. For ransomware operators, that’s key.”

The Challenges Of A Hyper-Connected World

The rise in zero-day vulnerabilities and exploits related to network security devices is one major piece of evidence that threat actors are favoring these systems for attacks right now, given the amount of effort and expense that goes into development of zero-day exploits. Such vulnerabilities have been known to fetch millions of dollars on darkweb forums, and these price tags can be tough for legitimate bug bounty programs to compete with.

It’s also common for organizations to continue using older, “legacy” network devices and to neglect upkeep of their software, according to security experts.

Many of the devices were not envisioned for the needs of the hyper-connected world we live in today, either, said Deepen Desai, chief security officer and senior vice president of security engineering and research at San Jose, Calif.-based cybersecurity vendor Zscaler, whose technology competes with traditional network security products. “These are devices that were made decades ago,” Desai said.

“It’s served its purpose for sure, whether it’s a VPN or a firewall,” he said. However, “the threat landscape was completely different back then.”

The shift from on-premises network security to newer approaches such as security service edge (SSE) and secure access service edge (SASE) is well underway but is expected to remain a gradual transition, experts said.

Spending on firewalls has significantly decreased in recent years, said Prakash Venkata, a principal focused on cyber risk at PricewaterhouseCoopers. Still, even customers in security-conscious industries such as financial services are saying they are going to be in a hybrid model for quite some time, Venkata said.

While needing to manage on-premises firewalls is an obvious downside for IT and security teams, that’s often outweighed by their familiarity with the systems, he said.

“They already have a systematic way of managing these things,” Venkata said. “They say, ‘Hey, we have the operational excellence here, we can go ahead and take care of it.’”

By contrast, with cloud-based alternatives, there’s still a learning curve to be overcome by many teams, he noted.

On the flip side, “Is it much easier to manage the cloud firewalls? Absolutely,” Venkata said. “If you have the right people and they have established the right rule sets, it’s much easier.”

As a result, it’s just a matter of time until more organizations transition away from on-premises network security, he said. “But at least for the next three years, I do not see that. The demand will be lower [for firewalls], but it won’t be going away.”

While the largest driver for SASE remains the ability to enable distributed workforces, the security benefit of moving away from older, less secure technologies such as VPNs is also a factor for a growing number of customers, solution providers told CRN.

It can require an educational process, and this task often falls to the solution providers that work on the front lines with customers.

“Talking to a customer about that can be difficult because they don’t want to let go of that VPN. They’ve done that for 20 years. They’re very comfortable with it,” said David Gottesman, founding principal and CEO at San Francisco-based Epic Machines. “There are a lot of things that folks just need to be educated on.”

In the interim—with so many organizations continuing to rely on VPN and firewall devices to keep their networks secure—some believe better options from network security vendors are needed when it comes to automatic patching for critical vulnerabilities.

Such capabilities, in fact, have existed for years.

As part of the launch of its XG next-generation firewall in 2016, Sophos included an automatic patching feature that is used in cases of highly severe vulnerabilities, according to Dan Cole, senior vice president of product management at Sophos. The capability is turned on by default, Cole said, and only a “low single-digit” percentage of customers have opted to disable the feature. The rest—the vast majority of customers for the more than 500,000 Sophos XG firewalls deployed today—receive so-called “hotfix” updates for critical bugs about two or three times a year, he said.

Crucially, Sophos reports that it has not seen the sort of disruptions that many in the industry fear from allowing vendors to automatically patch network firewalls.

“Obviously, we test [any patch] pretty significantly in-house before we roll it out,” Cole said. “I think being able to do the hotfixes [automatically] without any type of interruption has really saved our partners and customers a lot of time and effort.”

Like other network security vendors, Sophos has not been spared from the discovery of serious vulnerabilities in recent years, “but the concern dies down pretty quickly” thanks to the auto-update feature, according to Bishop Fox’s Gross.

During its penetration testing work, the team at Bishop Fox has found lower success in exploiting Sophos firewalls than devices from other vendors, he said.

“We’re as opportunistic as possible in trying to—for our customers— exploit as many of these firewalls as we can, as soon as we can. And I’ll say that Sophos is one of the ones we’ve had the hardest time with because of how quickly they actually update,” Gross said. “I think that’s really notable.”

Looking ahead, other network security vendors would be wise to start offering a similar capability, he said. “It takes the dimensions of priority and team capacity mostly out of the equation. It puts a lot more responsibility in the vendor’s hands just to get a patch out as soon as possible. So I would love to see more vendors adopt that feature.”

The bottom line is that without this type of capability, IT and security teams are caught in a cycle of thinking, “‘When the dust clears, then we’ll go back and completely apply patches,’” Gross said. “The problem is that the dust is not clearing.”