Fear and loathing defending ICS security at the Department of Energy's CyberForce Competition

CSO goes gonzo to defend critical infrastructure from hackers as part of a cyber defense competition.

"The HPC is down!"

"But the competition just started!"

Our high-performance computing cluster (HPC) blinked red on the big screen. Minutes ticked by.

"Get it up! Get it up! We're losing points!"

"Working on it!"

Red team had been circling since the day before, hawks swooping and diving. They'd been scanning and probing all day Friday, but weren't allowed to attack until the checkered flag dropped Saturday morning at 8 a.m. We'd hoped to evade their talons, but they wasted no time, and now one of our critical assets blinked out--a meal for a hungry predator.

Our blue team was tasked with defending a mock oil refinery's industrial control system (ICS), the HPC and the integrated back-office IT system—all of it default insecure, some of it insecure by design—and the only real defense active monitoring and split-second eviction before red team could take us down.

"The HPC isn't coming up. What is going on?"

Four Raspberry Pis running the oil refinery and HPC sat on the table in front of us, water pumps clicking on and off. Click-click. Click-click. Click-click. Someone else was in control. Not us.

cyberforce 1 100782993 orig J.M. Porup

Meet Big Oil Logistics and Transportation Corporation (BOLT Corp), its ICS, and an HPC running on four Raspberry Pis.

CyberForce — a competition to defend critical infrastructure

Across the country in seven different US Department of Energy labs, university teams fought alongside us to defend identical infrastructure. The DoE runs its CyberForce competition every year to introduce college cybersecurity students to the challenges of securing critical energy infrastructure, and to recruit the best and brightest. We were the UC Berkeley team representing the Berkeley iSchool's new Masters of Information and Cybersecurity (MICS, pronounced "mikes") program, and I was not only reporting on the event, I was playing to win.

The six of us hunched over our laptops in the competition space at Berkeley Lab, perched cliffside just east of the UC Berkeley campus, too distracted to enjoy the view of the Golden Gate Bridge, San Francisco Bay and the iconic Berkeley Campanile, the picture postcard bell tower. For eight hours we fought to keep our systems up and red team out, until, by the end, the uncontrollable twitch in my right eye told me I had chewed on too much stress for one day.

Teammate Daren tugged at his grey goatee. "Let me check something."

Predator and prey

I stood in the dark under the dripping canopy of the Strada Cafe in Berkeley waiting for my ride to the lab that morning. The piranha in my bowels gnawed away at my spasming entrails. How were we going to defend these completely defenseless industrial control systems? These things were running modbus, insecure by design. We weren't just sitting ducks, we were Peking ducks ready to serve with plum sauce.

This wasn't your normal capture the flag (CTF), where teams competed to break into the systems and steal information. We weren't predators. We were the prey.

I sipped my black coffee and checked my phone: 6:12. Doors opened at 6:30, and the competition began at 8 a.m. Pacific—11 a.m. for the competitors at Brookhaven on Long Island, 10 a.m. at Argonne outside Chicago, and 9 a.m. at Sandia in New Mexico.

It was going to be a long day.

My teammate Josh rocked up with a coffee and we stood watching the rain. The darkness seemed unending. A car sloshed by. Was it—? No. Not our ride. We both pretended to be chill.

"We gonna win today, you think?"

"Gonna try."

Then "The Anvil of Crom" thumping in the car, Nathan our mentor driving, up the rainy black hillside to the unclassified cliffside lab that overlooks campus, to the guard post, documents glanced at, a flicker at us in the back seat, then up, up, up to the SOC we had laid out on two tables the day before.

cyberforce 2 Paul Mueller, photographer

Our ad hoc SOC: six laptops, two big screens, many coffees.

The music still rings in my ear, so wrong in retrospect. We were not warriors going into battle; we were lab mice in a maze with hawks circling, predators ready to crush us with their talons at any moment.

We sat down at our battle stations—two giant big screens propped up on our competition table—and prepared to defend. The piranha gnawed harder.

The dirty half-dozen

"They're all just kids," we muttered, glancing sideways at each other. The other three teams at Berkeley Lab were undergraduates, most of them too young to enjoy a beer after. Our team's average age was 42, and together the six of us have a combined 107 years of experience working in IT, including a couple of folks from the defense sector, a software architect, a systems integrator, an early Facebook employee, and a coder turned wordslinger—me.

We were double their age and had ten times more experience than they did. Somehow that didn't make us feel any more confident about our chances of winning.

cyberforce 3 100782997 orig J.M. Porup

Predator humor

"Having fun?" my teammate Karel asked. I stared at the big screens, looking for rogue logins. The log info came at us in an unending stream. On the CyberForce Slack channel for competitors, threat intel overwhelmed with gossip and meme noise. And the memes! What was with the undergraduate obsession with memes, many of dubious hilarity? Out of place, old and weary, facing an impossible task, a goshawk's meal in waiting.

"Not really," I answered.

Fear and loathing washed over me like a shower of sewage. Were we fools or tools? I shuddered. Not the lesson the DoE wanted to impart when they launched CyberForce, I felt certain.

Red team goes vishing

"Team 69 help desk, can I help you?" Terry clutched the cell phone to his ear in the noisy competition space. Of the 70 teams competing, UC Berkeley had been assigned number 69.

cyberforce 4 100783006 orig J.M. Porup

Team 69, at your service

CyberForce awarded huge points for usability and help-desk support, not just keeping red team out. To secure the ICS modbus service in the handful of weeks we had available, we'd implemented a crude two-step authentication process. Sniffing legitimate LDAP credentials sent in the clear wouldn't be enough to pop our mock oil plant. Red team would also need to know the correct two-step authentication code listed in the user guide.

Terry muted the phone. "Green team wants the two-step auth code."

"It's in the user guide. Tell them to read the user guide."

"Says they don't have access to it."

We had written the user guide from scratch to include an easy look-up table of all the two-step authentication codes. Hacking the green team to steal the user guide was out of scope. Within the constraints of the competition, our solution was as good as it gets.

cyberforce 5 Karel Baloun

Battlestations!

Daren rose to his full height, grey goatee jutting forward, and took the phone. "Hello?" He listened for a moment. "I'm going to have to assume you are red team trying to social engineer us. The auth code you need in the user guide." He hung up and passed the phone back to Terry.

"How did they get the number?"

"It's right there on the login page, call this number if you're having troubles."

Nice try, we all thought. Not gonna get that past this team. We weren't born yesterday. Try to socially engineer us...

Until we read our scoring feedback the day after the competition. It included this nugget: “Tried calling help desk on HMI authentication problem, but another user was logged in and they thought I was from red team.”

Turned out the caller really was green team.

Lab time at the DoE

The competition kicked off with a red-tied Rick Perry beaming on the screen.

"Oh, look it's Trump's Secretary of Energy," I said.

"No time. Less than an hour before things get underway."

A week later, while writing this story, I hunted up Perry's opening remarks. Watching him struggle to read a teleprompter made my eye twitch again.

"Today the digital infrastructure that serves this country is literally under attack," Perry intones. "Protecting our energy infrastructure against those threats is my highest priority as Secretary."

"You are this nation's next generation of innovators, defenders, cyberwarriors"—a twinkle in his eye when he says the sexy cyber word—"We need you to bring your knowledge, passion, competitive spirit to, uh, the job at hand."

The worst part of it is, though, Perry's not wrong. Behind the puzzling upbeat muzak in the video and the "howdy, partner" political happy-clappy lurks a truth to wipe the smile off your face: America's critical infrastructure was never meant to be plugged into the internet. Next door to every spy and gangster on the planet, the energy systems on which our economy—and lives—depend are about as secure as a wet paper bag.

Worse, the massive skills shortage in cybertown means few qualified workers have any interest in building a career in OT/ICS/SCADA security. If Google and Facebook pay top dollar for security talent, a water treatment facility in southwestern Montana pays bottom dollar. The DoE wants to expose cybersecurity students to the problem in the hopes of attracting them to the ICS security space—or at least raising awareness of the issue more broadly among career beginners.

CyberForce—cue flexing muscular men and women on the cover of a vintage Conan the Barbarian pot boiler, oil glistening on scantily-clad physiques, blades flashing, stentorian voice like Zeus announcing their presence—"SIGH BURR FORSS"—launched in 2016, and the December 2018 competition was the fourth so far, and saw double the number of participants as the April competition. The next competition will be held in November 2019.

"The competition is meant for collegiate students to defend and secure an energy-simulated environment," Amanda Joyce, CyberForce Competition Director and Strategic Cybersecurity Analysis and Research Group Lead at Argonne National Laboratory, says in the video. "So every year we change the scenario to be a different energy-based component, this year being oil and high-performance computing. Their job is to take a vulnerable system and to secure it to the best of their ability within realistic environment restrictions."

One surprise gotcha—CyberForce competitors were also lab rats being studied as part of an experiment. I growled at my laptop when I discovered this tidbit buried in a wordy research consent form. Recruiting talent? Cool. Stimulating innovation? Even better. Watching all our VPN traffic and studying us as research subjects? Creepy. Were we the researchers, or the lab mice? Maybe both....

cyberforce 6 Department of Energy

Prey to both red team and researchers.

Out of equal parts alarm and curiosity, I called the Institutional Review Board (IRB) manager at the DoE, a woman with a thick Southern accent by the name of Lindsay Motz. I told her the consent form was opt-out, not opt-in. She seemed genuinely surprised. "You don't have to opt-in in order to compete," she told me over the phone, and encouraged me to reach out to the academic researcher in charge of the project.

"Talk to the pee-AHH," she told me.

I blinked. "The, uh... the who?"

"The pee-AHH."

"I'm sorry, I mean, I, uhh—"

"The principal investigator. The pee-AHH."

"Right, the PI, of course, right... thanks."

The pee-AHH, Benjamin Blakely of Argonne National Lab, sent me a copy of the HRP-503 form, which crisply informs the reader that "gaining an understanding of how to better measure cybersecurity expertise....will help many stakeholders improve training programs, accreditation programs, and workforce frameworks."

Considering the US government's long history of unethical experiments on unwitting human subjects, not to mention totalitarian mass surveillance, giving the DoE the benefit of the doubt was beyond my poor power. The explanations tally, the words all seem correct, but it still felt like they were trying to pull a fast one.

"I don't mind," Karel said. "I've got nothing to hide."

The white-on-black console text blurred together. "I suppose since you've got nothing to say, you don't care about free speech either?"

A shrug. "I think maybe you're reading too much into it."

Then the sCOARboard went down, and a swarm of piranhas feasted on my spleen.

What are the rules again, exactly?

"sCOARboard is down."

"What do you mean sCOARboard is down?"

"It's down. Everyone on Slack is complaining."

"How are we supposed to submit our incident reports?"

Want to check the real-time score updates? Only a refresh-refresh-refresh away—F5 the badly acronymized sCOARboard, the competition's score-tracking system. Teams could also earn points by submitting intrusion reports, or bonus points for solving so-called "anomalies"—discrete security problems like analyzing a pcap file in Wireshark, extracting a message from steganography, etc.

Seventy teams times six players is 420 competitors. Add a couple hundred green teamers playing industrial users plus the red teams and you're well under a thousand users total. The infrastructure couldn't handle all of us trying to access the sCOARboard at the same time, leading to frequent Slack messages to participants to stop all refreshing the sCOARboard at the same time.

The system was down, stayed down, and flickered on and off throughout the day.

"But our incident reports. The anomalies. How are we supposed to compete?"

"I don't know, man. I don't know."

Blood pressures rose. Fine for the undergrads but not great when your average age is 42. Let's not have a heart attack over a simulated hacking scenario, I thought.

"I've got a scaling challenge for you," I joked afterwards. "I want you to scale to three-digit users"—three fingers extended on one hand, thumb and pinky pinched together—"and I want three nines uptime." Three fingers on the other. "Three nines. Can. You. Do. It?"

"Looks like a gang sign," our mentor chuckled, mimicking my double three-finger salute.

Imagine playing baseball, but you don't know how many bases there are, how many strikes make an out, or even whether you are playing with a ball or a hand grenade. What are the rules? Where is the chalk? What is foul and what is fair? Oh, and the lights go out at random intervals and you play in the dark for a while.

Playing CyberForce to win—that's what it felt like.

While it's true the only rule in nation-state hacking is that there are no rules, and defenders in SOC chairs working at oil refineries don't have the luxury of a referee—at least, until the Geneva Convention gets a much-needed upgrade—if you're going to gamify learning, it's best if the game has, you know, like, rules, and those rules are clear and consistent and enforced equally for everybody.

"You think this is part of their research project?" I asked. "Make us squirm, see how we react?"

"No." Daren sat back in his chair, tugged at his grey goatee. "I just think they have no idea what they're doing."

"Incompetence, not malevolence."

"Yeah."

Nothing beyond our egos was riding on the results of the competition, except perhaps Little League bragging rights—and let's be clear, we were playing for Little League stakes—but I had flown out to Berkeley from New York for the competition, and many of the others had traveled long distances within the vast Golden State to be here. It was a point of pride, too. We were the greybeards of CyberForce. Were we really going to let these 19-year-olds beat us?

Two hands tied behind our backs

The backdoors were everywhere.

The Azure virtual machines (VMs) Cyber Force gave us were riddled with pre-installed rootkits and trojans and backdoored binaries, oh my, not to mention unnecessary software and services to purge with extreme prejudice. On some Linux machines, the /usr/sbin/nologin binary had been replaced with /bin/bash, thus giving password-free shell to accounts configured to have no remote access. We found that one before the competition, but what else was there? Did we find them all? Did we get everything? Would only take one oversight and we were screwed. Red team would go down their exploit list and try them all. What if we missed something?

The piranha chewed.

Plus our hands were tied. We weren't allowed to block attacking IP ranges. Doing so would render game day a moot point. If red team couldn't access our infrastructure there'd be no game to play—and in real life an attacker would simply come at you from a different IP range. A fair constraint, but still frustrating.

Nor were we allowed to use certificate-based encryption to protect sensitive data on the wire, even though we were required to ensure uninterrupted HTTP, FTP and SMTP service.

The joker of the group—that would be me, if you hadn't figured it out yet--suggested pulling a Kobayashi Maru. If we pulled it off, we would be legends. If we got caught, we'd be physically expelled from the facility. In retrospect, things were so chaotic we probably could have gotten away with it.

cyberforce 7 Karel Baloun

CTF is over. Time for beer.

Easier to cheat than to solve the real problem: How do you do the impossible? How do you secure the unsecurable?

The fear sputtered out and the loathing kicked in around 2 p.m. Loathing, and peace. I just wanted it to be over, I didn't care if we won, the whole game was chaos, if not out-right a rigged experiment, and all I wanted was a beer or three and a few laughs with my friends before heading home to New York and a long, drowsy slumber in Hell's Kitchen, Manhattan, with the relaxing sound of sirens and car horns outside my window.

We were halfway down the national rankings in the early afternoon, but somehow managed to claw our way back to the top ten before the competition ended. We didn't win CyberForce. Not this year. The UC Berkeley MICS team placed ninth out of 70 nationwide. We didn't even win the local award—the UC Davis team, led by a super smart kid who looked like he didn't have to shave very often, came in fifth nationwide.

cyberforce 8 Karel Baloun

Final results: UC Berkeley came in ninth of 70 teams nationwide.

And that early morning attack? Turned out when CyberForce had rebooted our Pis before the competition, the VNC service didn't auto-start, the way we had configured it the night before.

"Paranoia was our greatest adversary in the early going," Daren Slacked me a week later.

Risk is overrated; ruin is underrated

The day after the competition, I sat under the plane trees at the base of the Berkeley Campanile, drinking black coffee, and basking in the glorious early December sun. A surprise bell tower serenade broke out. My phone sat untouched in my pocket. The idea of plugging my brain once more into the internet.... how can I describe it? Fear and loathing, revulsion, horror. After what we'd been through the day before, just the idea of making myself vulnerable again, simply by turning on my phone, was more than I could bear.

Volunteering to be prey—on purpose, for "fun"—condensed all that emotion into one snarling day of unsuccessful flight, dodging shadows one second too late.

We are all of us now prey, both alone and together, a morsel for those predators—who now live next door to us in the shrinking global village we call home—who seek geopolitical leverage by owning critical infrastructure so unwisely connected to the internet, sabotage halfway around the world a keyboard's tap-tap away.

In cybersecurity we talk about mitigating risk, but rarely about mitigating ruin. Because ruin cannot be mitigated, only prevented, and some failure modes are so unacceptable that they must not be tolerated.

"We can defend this 98 percent of the time!" Karel told me jubilantly at the end of the day.

"What about the other 2 percent of the time?"

"What about then?"

What about then?

I sighed. A cool breeze brushed my cheek. The bell tower fell silent. I reached for my phone.

Show Comments