AUSTIN, TEXAS -- In the ongoing battle against enterprise security threats, Cisco has amassed an army of 500 engineers, researchers and technicians deployed in 11 primary locations worldwide, whose marching orders are to analyse threats and do everything possible to mitigate those threats as quickly as possible.
The nuclei of Cisco's distributed system are its Threat Operations Centers (TOC), one of which is located in a nondescript office building outside of Austin, where Network World recently visited.
The amount of security-related data pouring into the TOC is staggering. "I never wake up in the morning and think I don't have enough access to data. I do wake up frequently in the morning and think 'what are we going to do with all this data?'" says Rush Carskadden, a product line manager in Cisco's security technology business unit.
The task that drives Carskadden and his colleagues is to put all the data in context. Providing context is critical to discovering and thwarting enterprise threats that are becoming increasingly complex and multipronged. Blended threats aren't new, but they're growing in prevalence and severity.
"We're seeing blended threats that act just as intelligently as a very good penetration tester would act," Carskadden says. Meaning, they're patient, thoughtful and persistent. "The real surprise is the degree to which and the sophistication with which these threats are automated."
IN DEPTH: 5 top social media security threats
Night Dragon is a perfect example. First publicized in February, this series of coordinated attacks targeted intellectual property from energy companies. The tools and techniques involved -- social engineering, spear phishing, Windows exploits and Active Directory compromises -- aren't incredibly sophisticated, but the attackers' methods made it difficult to link the malicious actions together and enabled the intrusions to go on for as long as four years.
"It's a very sophisticated threat in the sense that it will actually seek out the Active Directory server, compromise it, use data slurping to grab credentials, and then use those credentials to further compromise the network and gain access to sensitive information," Carskadden says. Beyond an initial SQL injection, the attack consists of activities that would not appear overtly suspicious; the attackers are operating in a manner that doesn't draw attention, surreptitiously looking for valuable information to extract. While not publicly calculated, damages from Night Dragon could potentially be in the hundreds of millions of dollars, Carskadden says.
"If you trace through how this threat works, you will find few better examples of how important it is to tie the intelligence together from the various vectors," Carskadden says.
Tying it all together
Tying together threat intelligence is essentially the mission of Cisco's Security Intelligence Operations (SIO), which provides threat information, vulnerability analysis, and mitigation solutions to enterprise customers. SIO is the command center for Cisco's security services and appliances.
Organisationally, there are three main pillars of SIO. This first is SensorBase, the data repository.
SensorBase collects raw event data from more than 700,000 sensors built into Cisco network security devices deployed worldwide, including intrusion prevention systems, firewalls and web security systems. SensorBase on average processes 2 billion web requests and 13 billion emails daily, resulting in several terabytes of new threat-related data every day.
Recently, Cisco equipped its AnyConnect VPN client to participate, which opens the door to millions of client devices that could also contribute threat intelligence and data back into the SensorBase database. "We have just begun to digest some of the information that we're getting from secure clients," Carskadden says. "It's amazing how much information is out there. It's not all that valuable when you're just looking at data from secure clients, but when you compare it with everything else, you see all kinds of patterns. It's massive."
SensorBase also aggregates data from 600 third-party news and data feeds, such as DNS registry information, public blacklists and whitelists, as well as a global network of spam traps. Cisco also partners with ISPs and hosting companies to gain visibility into domain traffic.
The second pillar of SIO is Cisco's TOC, where the overarching goal is to transform the massive SensorBase threat database into something useful. Information gets pushed to products in the form of automated rules and signatures, and published to customers through security alerts, product advisories and threat mitigation bulletins.
Cisco has automated algorithms to process SensorBase data, and the tools generate about 95% of the rules updates that Cisco's security devices use. People do the rest -- researching threats, publishing alerts, designing mitigation solutions, hand-tuning new rules, and packaging rules for device updates. These people are the heart of the TOC.
The third main component of ISO, dubbed Dynamic Updates, is the communications hub, responsible for streaming information and updates to Cisco devices and customers. Some of the automatic updates for Cisco products occur in real-time: Reputation data used by Cisco security devices to block traffic from known malicious senders is updated continuously, for instance. Other systems, such as Cisco's Intrusion Prevention System (IPS), check for new rules roughly every five minutes.
The Dynamic Updates group is also responsible for distributing all the alert notices, vulnerability synopses and best-practices publications that the analysts and engineers in the TOC produce.
Taken together, Dynamic Updates manages 3- to 5-minute device updates, 3,300 IPS signatures, more than 20 publications, more than 200 tracking parameters, and 8 million rules per day. "The purely automated aspect of this is churning along at a rate that we could not possibly reach with just expertise alone," Carskadden says.
Inside Threat Operations Center
Cisco has invested more than $100 million on research and development within TOC. The specialties of the myriad research teams vary. Some of the engineers are expert in reverse-engineering malware. Others are tasked with infiltrating botnets, performing penetration testing, and helping customers protect their networks against active threats.
The Cisco Applied Security Research (ASR) team, for instance, looks for vulnerabilities in key technology areas and provides current threat indications and analysis. Vulnerability information that's related to Cisco products and networks gets handled by Cisco's Product Security Incident Response Team (PSIRT), which investigates the vulnerabilities and does the associated public reporting.
The Cisco IPS Signature Team researches exploits and writes vulnerability- and exploit-specific signatures that are used by IPS product lines. It's challenging work that requires coding experience, security savvy and what's dubbed "field knowledge" -- which can involve fraternizing with the hackers who make and use the exploits.
When looking for a good signature writer, "I look for curiosity, a desire to solve problems," says Morgan Stonebraker, who manages the signature writing team at Cisco's Texas TOC, which is contained within a cluster of low-rise office buildings north of Austin's city center. "This is like a puzzle. Every day you come in and there's something new, something that someone pretty clever came up with. You have to figure out 'how do I block it? How do I counteract that?'"
Sometimes vendors aren't or can't be candid about a specific vulnerability, which leaves the signature developer to figure out the details. The team might be notified of a serious bug by a vendor, whose only direction is something vague like, "it's an issue with embedded JPEG dimensions." The signature developer then has to compare the patched and vulnerable versions of the products, find the differences, and then attempt to zero in on the vulnerable area by trying to put themselves in the shoes of an attacker.
The IPS signature group has some internal SLAs that govern how fast it generates new or updated signatures. For anything that's related to PSIRT or Microsoft's Patch Tuesday, the team aims to push a signature out to customers within 90 minutes of the time a threat was publicly disclosed, Stonebraker says.
For critical enterprise-level zero day threats, there's a 24-hour turnaround. "That involves a little more work -- sometimes in-depth research, reverse engineering, patch engineering," Stonebraker says. "It can get pretty complex, so we give ourselves a little more time."
On the managed services front, Cisco Remote Management Services (RMS) provides around-the-clock remote monitoring and management services for Cisco security devices deployed at customer sites. In the Texas location, the RMS team's facilities don't look much different from typical office space, except the room is very quiet and fairly dark, and there are monitors aligned below the ceiling so the team can see any trouble spots at a glance. The windows are coated with a film so that no one trying to peer in from outside the building can read anything sensitive or customer-related on any of the screens.
For the IT pros who are responsible for mitigating threats to Cisco products -- enterprise customers and partners -- there's the Applied Intelligence Team. This group provides technical training and consulting services as well as applied mitigation bulletins, tech tips, and instructions to help IT users tackle threat mitigation procedures.
"Their specialty is using the Cisco technologies people already have and getting the most out of them, right down to the command line instruction about how to mitigate a certain type of threat or vulnerability," says Jeff Shipley, who manages Cisco's IntelliShield business and is responsible for coordinating activity across SIO. "It's very technical, very detailed and requires a high level of expertise."
"We can help customers understand that they really have a lot of help at their fingertips," says Randy Ivener, senior security engineer with Cisco's Applied Intelligence team. Network administrators may not be aware of the security capabilities inherent in Cisco's network devices. "They've got a lot of security features and technologies built into the products, and we give them a way to start leveraging that," Ivener says.
Specialized teams are also focused on disseminating security information to enterprise IT customers. At a high level, Cisco's IntelliShield security analysts keep tabs on everything known about existing and emerging vulnerabilities -- not just Cisco-related vulnerabilities, but across the industry. The IntelliShield Alert Manager Service distills all that information and pushes it out to customers, providing threat validation, vulnerability analysis and updates on global security trends. It's filtered based on the customer's environment, so it's easier for network and security teams to prioritize their remediation efforts.
The IntelliShield group also produces the Cyber Risk Report, which addresses top concerns in seven risk-management categories: vulnerability, physical, legal, trust, identity, human and geopolitical. Shipley presides over a weekly meeting among senior analysts from Cisco's security ranks, and the participants bat around ideas -- via teleconference and TelePresence -- about current security issues and how enterprise customers might be impacted.
Network World sat in on the June 23 Cyber Risk Report meeting, where participants discussed current threats, weighed which ones were more significant, and considered the angle that's most relevant for their readership: the network and security professionals who have to protect their environments from dangerous technologies, intrepid hackers and sometimes misguided users.
"As much as we can do with technology, a lot of it boils down to the people who are sitting there, clicking on things," says Shipley, whose experience includes 20 years spent with the U.S. Army in Security and Intelligence, Special Operations, and the National Security Agency. "The teams throughout the Threat Operations Center focus on pushing intelligence not only to the products but also to the customers directly, so they're increasing their awareness."
Information sharing
Committing to the research and development that SIO requires is no small investment. "It's a significant research burden to stay on top of this," Carskadden admits.
The payoff is clear when the disparate technologies and resources from SIO come together. For instance, if Cisco's IPS gear takes advantage of reputation-scoring data from Cisco's web security technologies and filters from Cisco SIO, the effectiveness of IPS goes up significantly. "We've roughly doubled the efficacy of a standalone IPS," Carskadden says.
Adding greater context to threat analysis also pays huge dividends on the timeliness front. In one instance, SIO detected an emerging threat, based on security event data fed to SensorBase, and researchers were able to glean information about how the threat would propagate, based on a characteristic they detected in its random number generator.
"That depth of intelligence enabled us, in a very specific example, to provide an update that would indicate by trajectory, IP block by IP block, who had likely already been infected. We could increase the risk associated with those IP blocks dynamically, as it propagated," Carskadden explains. "That's literally staying ahead of the threat."
Read more about wide area network in Network World's Wide Area Network section.