Infrastructure as Code has defined the last five years of systems engineering, and will likely define the next five. But as we became better and better at manipulating infrastructure with declarative languages, we started to look beyond JSON to find more full-fledged, dynamic programming languages to create and configure cloud resources.

In many ways, this movement is mimicking the evolution of web development as web “sites” became web “applications” — in a few years, systems engineers will be coding infrastructure applications, not hard-coding declarative templates.  

What is CloudFormation?

Amazon Web Services’ Infrastructure as Code (IaC) offering, CloudFormation, enables engineers to manage infrastructure through the use of declarative templates, utilizing the data format language, JSON. The value of this approach is tremendous. Templates can be checked into a source code management system, linted and validated using tools and IDE’s. Entire environments can be quickly duplicated, modified and deployed in quick time to keep pace with the rapid change ever present in today’s IT world. CloudFormation can also kick off bootstrapping through UserData, creating a seamless bridge into the operating systems where machine images and configuration management can take over. In addition, any infrastructure change can have an audit trail and proper process built around it, automated to reduce human error and disparate configurations.

Despite the significant gains present in using CloudFormation, on its own it has some downsides. At its core, CloudFormation uses JSON to declare resources used for infrastructure build out. JSON is a data format, meant to be read by both machines and humans.  The JSON is interpreted, turned into API calls and run against the infrastructure to create, change and remove various components. Amazon provides excellent documentation on the service, but for any engineer that has to manage large-scale, dynamic and complicated environments using CloudFormation alone, issues soon arise.

Hand-coding JSON is not a pleasant experience. The format does not allow for commenting. Stacks created can quickly become unruly as resource and line limits are hit, and environments grow. Some of this can be resolved by using sub-stacks and splitting the templates into smaller, more manageable chunks, but issues then arise due to a lack of global configuration, so resources’ IDs need to be passed and hard-coded as parameters or mappings in each stack. Stack names and resource names are also immutable. Make a typo? You will be looking at it for a long time to come. Whereas CloudFormation does include some intrinsic functions to map and locate resources, it is by nature not dynamic and that leads to significant hurdles when attempting to deploy and maintain dynamic environments with common code.

At its heart, CloudFormation is simply declarative JSON. And that is a good thing. What is needed is not to make CloudFormation more dynamic, but rather to build tools which can dynamically generate templates and manage stacks. This is happening now to some extent, but the expectation is that this practice will be the future.

CloudFormation and HTML Parallels

In some ways, the state of CloudFormation, and perhaps IaC in general today, is history repeating itself. Parallels can be seen in the long and varied history of web development that give us insight into what the future will bring.

Twenty years ago, developing for the web consisted of a lot of hand-coding HTML, another declarative templating language meant to be interpreted by a machine. As web “sites” evolved into web “applications,” this soon gave way to technologies like CGI, mod_php, JSP and others that provided a means of developing in a full-fledged programming language that could generate the underlying HTML. This gave us easy access to databases, global configurations, sessions, commenting and proper code organization —  benefits that revolutionized and accelerated the pace of the internet. However, even this was not enough and on top of these technologies grew the full fledged web frameworks seen in play today such as Rails, Django, Spring, Laravel and countless others. Yet if you right click and view source on even the most sophisticated web application,  you will see HTML at its core.

For AWS today, CloudFormation’s JSON templates is that underlying core for IaC development. However, I do expect to see many people hand-coding JSON over the next few years directly. This is rapidly being replaced by a variety of tools that can dynamically generate the JSON using a proper programming language. Examples include Troposphere, cfndsl, and many others. A quick search on GitHub will bring up many, many projects that generate CloudFormation in a variety of languages, including ruby, python, javascript, java, and even scala.

Any of these solutions give us the bare basics we need to take IaC within AWS to the next level in much the same way as CGI, apache modules and other technologies did for web development back in the day. It does not stop here though. The next logical step is to develop frameworks that can be used to further abstract and control the stacks created. We’re seeing the beginnings of that with such utilities as SparkleFormation, StackMaster, Cumulus, etc. Other tools such as Terraform take a different approach by accessing the API directly and not using CloudFormation. While that is a wonderful IaC solution for companies managing environments outside of or in addition to AWS, having tools that generate CloudFormation as a base allow you to move between AWS-centric solutions as needed, and keep the underlying work intact.

The Future of IaC

CloudFormation is here to stay and is an excellent service offered by Amazon to implement effective IaC. However, in a few years, we will look back and “remember the time we had to hand-code a ton of JSON.”

Initial tools that allow for dynamic generation of CloudFormation from other programming languages are available in full force. Stack management frameworks are starting to appear in bulk too. As the leaders of those categories rise, Infrastructure as Code in AWS will turn into Infrastructure as an Application. It will achieve the maturity and benefits of a full-fledged development environment much as web development has evolved over the years. And whether it is HTML or JSON, having a proper language at its base will enable multiple technologies to compete and provide choices while keeping a solid underlying base.

By Thomas Rectenwald, Sr. Engineer, DevOps
Logicworks

Logicworks is an enterprise cloud automation and managed services provider with 23+ years of experience transforming enterprise IT. Contact us to learn more about our managed cloud solutions.