City lights: Securing critical infrastructure

During the 2007 housing crisis, Columbus, Ohio--like most municipalities--faced significant tax shortfalls and revenue constraints.

That year was also marked by security events--on the physical security side, the Department of Homeland Security completed its Sector-Specific Plans for critical infrastructure protection. On the IT side, public- and private-sector organizations faced phishing attacks focused on stealing sensitive information and intellectual property.

That was the environment when Miki Calero joined the City of Columbus as CSO.

He immediately got to work improving the city's ability to manage risk to physical and IT assets by more tightly integrating security. Early his first morning, as he picked up his ID badge, he spoke to the facilities security manager about which physical access databases could be unified first. Six years in, the implementation of an enterprise security risk management (ESRM) program has improved security across the city, ensuring Columbus complies with seven sets of regulations and streamlining costs by combining existing security and technology investments with increased efficiencies.

[Also read DHS finds critical systems lack basic security--despite years of warnings | Situational awareness: Inside the new World Trade Center]

That's no small accomplishment for a city the size of Columbus. With roughly 790,000 residents, it's the 15th-largest city in the United States, covering 217 square miles and incorporating more than 200 government facilities dealing with permits, taxes, telecommunications and critical infrastructure for IT and utilities.

This is the story of how Calero and his team are pulling it off.

Early Challenges

Before the ESRM program was put in place, IT security was handled by a couple of analysts and by server administrators and network engineers, all of whom had many other responsibilities as well.

As at most organizations back then, and many still today, the work of securing IT systems was getting done, but without unified authority. That likely left gaps in protection that could have proven costly.

The highest reporting level for IT security was a manager, and the analysts "were primarily focused on running antivirus, monitoring and filtering Web content, reviewing requests for system and network accounts, and similar operational responsibilities," Calero said. Each IT group had its own budget, which made it difficult to plan and control security costs.

The Franklin County Government Center along South High Street in downtown Columbus will undergo a complete renovation that could take 10 years and cost $90 million, including significant security enhancements that meld physical- and IT-based defenses.

Physical security needs were defined by individual agencies, each managing their own access control systems and surveillance cameras, using tools and equipment bought at different times from different vendors, and paid for by multiple sources. There were no standards for the tools and equipment, no security project coordination or strategy to converge with IT security.

"I have a strong view that physical and cyber security risk need to be managed holistically," Calero said, so before he would agree to take the position, he made sure the title was CSO, not CISO. He wanted to make sure he could take a convergence approach to security, which involves pursuing a comprehensive security strategy and a supporting project to implement the ESRM program.

Columbus city leadership took its first step toward ESRM by funding the capital improvement project for its implementation. This multiyear endeavor is bolstering security already in place, upgrading capabilities, adding capacity, and laying foundational elements for unifying security for telecommunications, cyber and physical assets, and critical infrastructure and industrial control systems. At the same time, the city formed its official ESRM group, which is in charge of security risk management and regulatory compliance.

Realizing the Vision

Shortly after he was hired, Calero was invited to give a presentation to the city cabinet as a way to build awareness of the new CSO position, as well as to make the leaders aware of security policies and of Calero's plan to handle physical and IT security together.

"Putting a face to the CSO position and sharing the vision with the cabinet was key," he said.

Sharing a vision is one thing. Making it reality is another. As the program was being created, the city was facing a budget shortfall of $115 million, and Ohio's two-year $28.5 billion discretionary budget was projected to come up more than $7.3 billion short.

"While the capital project was already in place, budget shortfalls loomed. Everyone knew we had to do our part to reduce security costs. We focused on what we could consolidate and help others consolidate using early ESRM successes as reference," Calero said.

When the Economic Advisory Committee, which was commissioned by the mayor to review the city's financial health, sought cost savings proposals from city leadership, Calero's included unifying building access control systems and video surveillance, expanding security for the city's extensive fiber optic network, maximizing use of the centralized security command center, and consolidating or outsourcing security functions.

Ultimately, revenue generation stabilized the budget, but the opportunity to promote security convergence was not lost.

"Detailing the efficiencies that would be gained helped grow acceptance of the program," Calero said.

At that point, the CSO had the funding for the project, a strategy for the program and a vision of where to take it. What he needed now was buy-in from stakeholders across the city, he said.

Rather than seeking mandates, Calero chose to create an atmosphere of collaboration through "security cooperatives"--security awareness training in partnership with facilities security--and by dedicating an analyst to physical security alignment.

"If you don't win their hearts and minds, you are not going to get anywhere," he said. (Read effective communication lessons from former CSOs in What I learned when I left security.)

Once that was done, Calero set out to learn about and unite key teams, managers, vendors, and consultants.

"Knowing them, of their construction projects, hearing about planned facility renovations or general security needs is making it possible for me to bring them together, see if they have common security needs, share security assets, or just make them aware of existing assets they did not know existed," he said.

Security effectiveness increased while cost and management burdens decreased. On the IT side, coordinated actions across departments readied city assets for when the Multi-State Information Sharing and Analysis Center would issue early warnings.

The centralized security command center increased building security. The $500,000 command center, finished in 2010, is staffed by up to 20 people. Should an event trigger an alarm, the footage from an appropriate video camera will be displayed on the monitors.

A few years ago, the city's video surveillance system had a couple hundred video cameras, but today that number is 600 and growing.

The importance of unifying the city's building access control system cannot be overstated. Over the years, the city had adopted a hodgepodge of such systems, all of which were implemented at different times, resulting in multiple variations on the same system even where a standard system was used.

Maintaining separate building-access-control systems not only made it much more difficult to manage access properly, but it was also expensive.

According to Calero, consolidating the purchase of equipment related to the city's access-control systems reaped a 15 percent one-time savings, as well as about a 15 percent average annual savings in the cost of recurring maintenance, and an additional 3 percent long-term savings from increased efficiency.

Building Security In

While many organizations strive to incorporate security into an asset's lifecycle, Calero has succeeded in bringing it to buildings and facilities development.

"Building security is IT security, too," said Calero. "Every information system has a physical security requirement. The building itself, the rooms, the network itself--all must be reasonably physically secure, and that includes integrating secure design very early in the phases of construction projects," he said.

[Learn more about CPTED in Security and building design: What changed in this decade?]

This includes the upgrade of the police department headquarters that the city is currently undertaking. To better secure the tax offices, which will be housed there, Calero began working early in the process with the tax agency, building architect and oversight contractor to establish requirements for physical security, including surveillance camera placement and network closet security.

Other agencies are now coming to Calero and his team to ask for security advice when they're planning projects such as renovating recreation centers or building new pool houses.

"[The CSO] being brought to the table is a win for the city. Internal subject-matter expertise is invaluable in reducing the cost of security and increasing security effectiveness," Calero said.

Maintaining separate access-control systems in each building, including the police station, was expensive and made it more difficult to manage access properly. Calero centralized the system, resulting in tighter security and big cost savings.

"Vendors will come in and install security equipment [without] asking to see the most recent risk assessment. If they review it, the assessment will not tell them about available connectivity--280 miles of fiber owned by the city--or the strategy to unify security systems and manage them from the command center. They may propose physical servers be purchased and installed on-site with a battery for backup power, while the city has redundant data centers full of virtualized servers."

Another strategy helped consolidate security efforts and shift costs so it was easier to fund convergence efforts, Calero said. By outsourcing operational IT security functions to a services provider, annual costs were cut from $380,000 to $95,000, and the relationship established the foundation for adding physical security information management (PSIM) to the current security information event management capabilities.

"Encompassing both IT and physical security event correlation will lead to increased situational awareness," Calero said.

While the city has come a long way, Calero said, he also believes there is much more to be done, including further integrating video and access systems, adding more facilities, and exploring industrial control systems monitoring with the outsourced security service provider.

"We are not finished yet. Not by a long shot. But we are moving forward and leveraging wins both big and small."