Complexity is a huge security risk for the enterprise.

While security is always a top priority during the initial build phase of a cloud project, over time security tends to slip. As systems evolve, stacks change, and engineers come and go, it’s very easy to end up with a mash-up of legacy and cloud security policies piled on top of custom code that only a few engineers know how to work with.

The security of your system should not depend on the manual labor — or the memory — of your engineers. They shouldn’t have to remember to close XYZ security loophole when deploying a new environment. They don’t have time to manually ensure that every historical vulnerability is patched on every system across multiple clouds.

Security automation is the only long-term solution.

Automation significantly improves an engineer’s ability to “guarantee” that security policies are not only instituted, but maintained throughout the lifecycle of the infrastructure. Automated security policies encourage the adoption of evolving standards. And as vulnerabilities are exposed, changes can be implemented across hundreds or even thousands of complex systems, often simultaneously.

Why Security Automation?

No one can remember everything. The #1 reason to automate security is that human memory is limited. The bigger the infrastructure, the easier it is to forget to close XYZ security loophole when launching a new environment, or remember to require MFA, etc. Engineers are a smart group, but automation created by expert engineers is smarter.

Code it once and maintain templates, not instances. Manual security work is not only risk, but it is extremely time-consuming. It is much wiser to focus engineering time on building and maintaining the automation scripts that ensure security than it is on the manual work required to hunt down, patch, and upgrade individual components on each of your virtual servers.

Standard naming conventions. Inconsistent or sloppy naming is a bigger security risk than most people think. Imagine an engineer being tasked with opening a port on one of the following security groups, below. It would be fairly easy to mistake one security group for another.

Ensure historical vulnerabilities continue to be patched. When a security vulnerability is identified, engineers must manually patch the vulnerability in the right places, across hundreds or even thousands of separate systems. No human can ensure that the patch is in place across all of these systems.

When something like Heartbleed happens, the engineer can:

1. Update SSL (or affected package) in a single configuration script
2. Use a configuration management tool like Puppet to declaratively update all running and future instances, without human intervention
3. See at first glance which instances are meeting core security objectives
4. Guarantee that any new instances, either created during Auto Scaling event or due to failover, are protected against all historical vulnerabilities

No limited custom configurations. When different environments are built by different engineering teams at different times, manual security configurations often mean custom configurations. This makes it very difficult to gauge the impact of a feature change on security. A single or limited number of custom configurations not only reduces the risk of unexpected security implications, but also means your team is not relying on the memory of the one or two engineers that built the application’s infrastructure.

Our Security Automation Tools

Infrastructure Buildout: AWS CloudFormation

Infrastructure buildout should be the first thing an IT team automates. This includes networking, security groups, subnets, and network ACLs. At Logicworks, we use AWS CloudFormation to create templates of the foundational architecture of an environment.

CloudFormation allows us to spin up completely new environments in hours. This means no manual security group configuration and no AWS Identity and Access Management (IAM) role configuration. Because configuration is consistent across multiple environments, updates / security patches are near-simultaneous. It also ensures that the templated architecture meets compliance standards, which are usually crucial in the enterprise.

There have been a number of tools released in the last year to build out templates of AWS resources. Our opinion is that CloudFormation is the best tool available, despite certain limitations.

Here are a few tasks that CloudFormation performs:

• Build network foundation
• Configure gateways and access points
• Install management services, like Puppet
• Allocate Amazon S3 buckets
• Attach encrypted volumes
• Control and manage access though IAM
• Register DNS names with Amazon Route 53
• Configure log shipping and retention

Configuration Management: Puppet

Boot time is arguably the most crucial part of an instance lifetime. Puppet or another configuration management tool like Chef or Ansible not only simplifies and speeds up the bootstrap process, but for security purposes, continually checks in on instances and rolls back non-authorized changes. Puppet manifests are therefore a living single source of truth on instance configuration across the environment. This means that engineers can ensure that no (permanent) changes are made on an instance level that compromise security.

Puppet is also used to install various security features on an instance, like Identity Detection System agents, log shipping, monitoring software, etc. as well as requiring MFA and binding the instance to central authentication.

If there is more experience on an IT team with tools like Chef or Ansible, these are equally powerful solutions for configuration management.

Iterative Deployment Process: AWS CodeDeploy / Jenkins

Ideally, enterprises want to get to a place where deployment is fully automated. This not only maintains high availability by reducing human error, but it also makes it possible for an organization to respond to security threats quickly.

AWS CodeDeploy is one of the best tools to be able to achieve automated deployments. Unlike Jenkins, which requires a bit more custom configuration, CodeDeploy can be used across multiple environments simultaneously. Any effort that removes custom work is engineering time that can be focused on more important features — whether that’s developing new code or maintaining the automation scripts that make security automation possible.

Monitoring: EM7, Alert Logic, CloudCheckr

By choosing the right 3rd party monitoring tools, you can bake automated security monitoring into every deploy. ScienceLogic’s EM7 is the best tool we’ve found for automated reporting and trend analysis, while Alert Logic provides the most sophisticated intrusion detection tools. CloudCheckr not only provides excellent cost analysis, but it also has introduced governance features that help enterprises stay compliant. Enterprises are usually quite familiar with these tools, and they can function across public clouds and on-premises environments.

Coming soon to the enterprise?

Security automation is not easy.

In fact, for some enterprises, it may be more cost-effective in the short term to configure security manually; CloudFormation and Puppet take several weeks or even months to learn, and it may take a consulting engagement with a 3rd party cloud expert to even understand the foundational security policies in place across different systems.

However, we expect that a manual security approach will be impossible in five years. Enterprises are already spanning on-premises datacenters, on-premises virtualized datacenters, colocation centers, some public clouds, etc. As the enterprise moves towards hybrid cloud on an application-by-application basis, this means even more complexity.

But complexity does not have to mean custom configuration. Security automation tools, combined with tools like containers, mean that engineers can escape manual configuration work on individual servers. As security is abstracted away from the underlying infrastructure, we have the opportunity to improve our overall security posture. This is the next frontier: security as code.

by Phil Christensen, Senior Engineer, DevOps

“Security and Advanced Automation in the Enterprise” was a talk given by Phil Christensen at the NY Summit on July 9, 2015.

Logicworks is an enterprise cloud software and services provider specializing in hybrid and managed AWS solutions for the healthcare, financial, legal and commerce industries. Contact us to learn more about our secure cloud solutions.