Credit: Michael Gaida
Patching security vulnerabilities in industrial control systems (ICS) is useless in most cases and actively harmful in others, ICS security expert and former NSA analyst Robert M. Lee of Dragos told the US Senate in written testimony last Thursday. The "patch, patch, patch" mantra has become a blind tenet of faith in the IT security realm, but has little application to industrial control systems, where legacy equipment is often insecure by design.
The Senate committee hearing highlighted the gulf between information technology (IT) and operational technology (OT) security, and how few of the lessons learned in the IT security space carry over to industrial security. "Operational technology" is a newish term that has emerged to distinguish industrial networks and systems from traditional business-focused information technology.
"There are two different trains of thought," Nick Santora, CEO at Curricula and a former critical infrastructure protection (CIP) cybersecurity specialist at NERC, the North American energy grid regulator, says. "In IT security, it's business critical stuff. On the OT side, you're dealing with mission critical stuff that can't go down. You can't take an outage on a whim, 'Oh, a server went down.'"
Defending critical OT infrastructure, such as the energy grid, requires a different approach, Lee told the Senate. "Our mission is different because it takes on a physical aspect, and therefore focusing on just malware prevention or patching doesn't actually address a human adversary," Lee says. "Malware is not the threat. The human on the other side of the keyboard is the threat."
Everything you thought you knew is wrong
The hard-won lessons of IT security do not apply in the OT space, and trying to manage OT security the "enterprise IT way" is actively harmful, Lee's report shows. Sixty-four percent of all ICS-related patches issued in 2017 don't fully address the risk because the components were designed to be insecure, Dragos concluded in a report submitted to the Senate.
Worse, major vendors have bungled security patches in recent months, the Dragos report says, resulting in outages that have cost companies money. Patching an industrial control system that makes widgets or pumps water is more complicated than rebooting an office desktop PC, and OT networks are a lot less tolerant of downtime.
Nor is patching or basic cyberhygiene sufficient to defend against the nation-state adversaries who daily probe and own critical infrastructure in the US and around the world. Keeping an OT monitoring workstation that may well be running Windows 10 patched and up-to-date will defend against most opportunistic malware, but it is hardly sufficient to prevent intrusion by advanced persistent threats (APTs).
"The industrial threat landscape is largely unknown," Lee says. "The methods from private sector, as well as from government, to target and understand threats in core business networks don't translate into industry."
That's not a recipe for despair, Lee emphasizes. Good cyberhygiene remains a minimum baseline to prevent opportunistic malware infections. "At a minimum, there are 6,000 unique infections in ICS every year," he says, "and each one of those infections can cause operational issues — and in rare cases, safety-related issues."
Securing your IT network and segmenting it from your OT network is a minimum best practice. But how do you secure an OT network that was designed to be insecure in the first place?
"Say you're running a fully patched Windows 10 HMI (human-machine interface — i.e., a control terminal). All ports are turned off. You've followed correct guidelines to ensure that the Windows application is secure. The problem, Thomas VanNorman, director of application engineering at Veracity Industrial Networks, says, "is that the protocols aren't. It's talking ModBus out to a PLC (programmable logic controller, an industrial actuator). I can go connect to that PLC with a rogue computer and modify those registers because it's an unauthenticated protocol. If an attacker has access to that level, it's game over anyway.
Securing industrial control systems
If most of the lessons of IT security don't carry over into the industrial space, how do we secure our critical infrastructure? The answer boils down to threat modeling. Attacking legacy ICS infrastructure is expensive and time-consuming, limiting threat actors to nation-states and organized crime. However, there is no clear path to profit from hacking industrial control systems. A Dr. Evil wannabe who tried to hold a city hostage for one beeeeellion dollars would probably find their hideout reduced to a smoking crater pretty quickly.
That leaves nation-states. In many cases, small private utilities are dealing with nation-state attackers who might even be running training exercises on the utility's networks. Many smaller industrial operators have so little visibility into their own networks that this is the reality we face today, Lee told the Senate.
"Our geopolitical rivals are definitely trying," Bryson Bort, founder and CEO of Scythe and chairman of Grimm, says. "These things take time, patient long-term campaigns to put in place levers or buttons so that when [nation-states] need to rattle the saber these things are in place to do that."
"The true threat," Bort adds, "is the more nuanced, complicated adversary and they're not going to do anything unless there's a nation-state need for them to do that."
In the face of human adversaries actively intruding into industrial control networks in pursuit of political leverage, the only solution, Lee says, is to use humans to hunt humans. No automated, reactive monitoring in the ICS space will ever replace active human defenders seeking out active human adversaries on their networks.
That process, he argues, begins with good threat intelligence: Who are the human adversaries actively breaching industrial control networks? What are their motivations? Who are their targets? What does their tradecraft look like?
Five main activity groups today are targeting ICS networks, including those responsible for the 2015 and 2016 attacks on Ukraine's energy grid, and Trisis, the first ICS malware targeted to take human life, according to Dragos's analysis, submitted to the Senate as part of Lee's written testimony.
One thing is clear: Attacks on industrial control systems are getting worse.
Worsening attacks on ICS
In the beginning, there was Stuxnet. The US-Israeli malware took out the targeted Iranian centrifuges, but collateral damage put the sabotage-ware front and center in the news as the world's first known nation-state attacks against industrial control systems. Other nations have since followed suit.
The 2015 attack on Ukraine's power grid was the first confirmed attack to degrade electricity distribution. It left more than 225,000 people without power. 2016 saw the same activity group launch an even more sophisticated attack on Ukraine's power grid, which the Dragos report called "the first ever malware framework designed and deployed to attack electricity grids." Dubbed Crash Override, this malware framework was only the second piece of malware, after Stuxnet, specifically "designed and deployed for disrupting physical industrial processes," the Dragos report concluded. The Crash Override framework could easily be tweaked to attack power grids in the US or Europe, the report notes.
Things got worse in 2017 with the discovery of Trisis. "The Trisis malware was specifically targeted to kill people," Lee tells CSO. "That is a level beyond anything we've seen before." The Trisis malware targets industrial safety systems, designed to protect human life in the event of an industrial accident.
2017 was a watershed moment for ICS security, and awareness of how vulnerable we are to such attacks is beginning to sink in. That said, Lee cautions against over-hyped movie plot threats. Things are bad, he says, but this is a solvable problem.
Poor media coverage is making things worse
A hacker in a hoodie sends a phishing email to a nuclear power plant. Cut to: nuclear meltdown!
Not gonna happen, experts say. It might make a fun Hollywood movie, though. "It's not doom and gloom," Lee emphasizes. "Our infrastructure is extremely resilient and defensible. We want to move it from defensible to defended."
Communicating the true nature of these threats to the public is important to have an even-keeled conversation about industrial vulnerabilities. Bort calls out coverage of the 2013 breach of the Bowman Avenue Dam north of New York City, which got screaming headlines like "Iranian hackers breached New York dam causing White House alarm."
Just one problem, Bort says: "That dam was an earthen barrier. It had no mechanical parts. Nobody except for the people in the know are aware of that part of the story. Everyone focused on the Hollywood threat."
The attackers managed to breach the Bowman Avenue Dam's monitoring system, but that was all, he says. "That distils the essence of this conversation for the last five years on this topic," Bort adds. "Let's all get very scared and oh, by the way, look at the axis of evil."
Nation-state adversaries do threaten critical infrastructure, both Lee and Bort emphasized, but let's keep things in perspective and do the work, instead of frightening ourselves to death for no good reason. Doing the work means knowing who your adversaries are and hunting them on your own networks.
Getting better threat intel
Private industry has better threat intel than the intelligence community, and pushing OT security teams to get security clearances to view classified threat intel is not super useful, Lee, a former NSA analyst who's worked both sides of the fence, told the Senate committee.
Lee's remarks were a rebuke to the other panelists, including Assistant Secretary of the DoE Bruce Walker, who told the committee to expedite security clearances for ICS workers. OT security teams who at long last get their security clearances are likely to be disappointed when they gain access to classified reports, Lee suggested.
"This focus on intrusion analysis has led the private sector to be able to produce intelligence reports that rival and, in many cases, far exceed similar reporting in classified government settings," Lee's written testimony says. "Simply stated, the best place to collect data relevant to cyber threats is in the networks of the targeted companies."
Lee told the committee that he has better threat intel now that he's in the private sector than he ever did when he at the NSA working on the same problem. "At the NSA, we built the mission to look at nation-states breaking into industrial control systems," Lee tells CSO. "It became very obvious our own collection was extremely limited into that threat landscape."
Despite the government's desire to address the national security concerns posed by vulnerable critical infrastructure, technical experts, including Lee, say that more government intervention is likely to be counterproductive, including the growing drumbeat for stronger regulation.
Compliance considered harmful
Checking boxes to ensure compliance with minimum security baselines might prevent opportunistic malware infections, but it is completely useless against targeted nation-state attacks. Worse, too many compliance measures can be harmful as they create a false sense of security. As a result, Lee and other experts urged a delay in further regulation while industry develops its own best practices.
On the one hand, many ICS networks now fail to meet even minimum standards. "There are industrial sites including those in North America whose internal teams have never even investigated the networks," Lee's written testimony says. "I am aware of small electric co-ops, water utilities, gas pipeline facilities, oil refineries, wind farms, and manufacturing networks where not even the basics of security have been attempted although they are vital for modern civilization."
On the other hand, OT systems face nation-state adversaries, and defending against that threat requires an aggressive monitoring and incident response program far in excess of what any compliance measure could demand. "I have seen first-hand a regulation, check-box, mentality develop at companies subject to strong regulations," Lee told the Senate in written testimony.
Developing a strong OT security program capable of hunting nation-state attackers on an ICS network requires trained cybersecurity professionals, with management support and budget to match. The vast and growing cybersecurity skills shortage, and a lack of understanding at OT management level, make this goal seem distant.
IT, meet OT
IT security continues to struggle to understand the unique challenges facing OT systems, but even more dramatic has been the response of traditional industrial operators to the idea that now, after 40 years of running their plant, magical hackers in hoodies on the other side of the planet could cause an outage.
Santora talks about his time as an auditor at NERC. "We flew around to different companies, and some would literally curse us out," he says, "saying, 'I don't believe in any of this security stuff, it's a waste of time, this stuff isn't real, it isn't going to happen to us.'"
The novel and unintuitive nature of cybersecurity alarms the older generation of industrial workers, he says. "It's an unknown risk people are scared to talk about sometimes."
Hiring cybersecurity experts to augment those decades of experience with a newer perspective would be the logical answer, if only security pros were available to hire. Some estimates put the global shortage of security professionals as high as two million by 2020. Top talent is unlikely to pursue a career in OT due to the much lower salaries that industrial concerns typically pay.
"There are not a lot of people with these skill sets," Lee tells CSO. "Around 500 people in North America have both the ICS and cybersecurity skills."
Solving that problem means scholarships or apprenticeships that put talented graduates in OT security positions in exchange for their schooling. Or maybe just pay them more? Whatever the solution, it seems unrealistic for the federal government to label small utilities a national security concern and then offer no financial support to defend those assets.
"If we're saying it's a national critical infrastructure issue," Bort says, "then you can't depend on some tiny little town to cough up the money to solve that." In the near term, recruiting from IT security may be the answer.
It takes a village?
For the last several years, the ICS Village at Def Con and RSA has offered security folk the chance to get their hands dirty with real world ICS equipment. Debunking myths around OT security and helping IT pros better understand the challenges in OT is why they do it, VanNorman and Bort, the ICS Village organizers, tell CSO. "We allow someone to actually touch and hack the different platforms so they can begin that journey," Bort says.
Everyone has heard tales of nmap taking a gas pipeline offline, but, VanNorman says, OT systems are not as fragile as many think and are far less complex. "It's not going to break if you touch it," VanNorman says. "We show them the difference between what a Raspberry Pi does and a PLC. A Pi does a lot of things. Not a PLC."
The ICS Village works to bridge the gap between IT and OT security, and counts success one hacker at a time. "The most common reaction," Bort says, "is, 'Oh wow, I get it now.'"