Set up your next Cloud project for success: a proven framework for efficiency, scalability, and adaptability
Successful cloud projects are often also the ones that grow over time to serve more users and offer more capabilities. Unfortunately, this capacity for growth can be limited by a lack of flexibility or ability to adapt.
Commonly-used development practices can lead to a sprawling infrastructure that’s hard to manage efficiently, and difficult to update.
In this blog, we’re going to share our proven ‘Blackbird way’ of setting up a cloud project. Built on years of experience with demanding cloud projects, this methodology crystalizes everything we know about making it easy to manage your project over time.
It may cost a little effort to begin with, but once you have this system up and running, you’ll wonder why you ever did it differently.
What problems does this method avoid?
The Blackbird method uses tools and systems you’re probably already familiar with – the difference is how they are used. Crucially, we always use an organized system of smaller Terraform modules, instead of using a single ‘monolithic’ module for the entire deployment.
This system avoids many of the common problems that development teams experience when their project starts to mature and requires frequent updates and changes.
Common problems from Cloud projects that are poorly set-up:
State issues – Terraform includes a few key security measures, including state locking. This prevents multiple developers from working on the same operation at the same time. However, if you’ve put everything in one module, that means only one team member can ever work on it at any given time. This is a huge constraint on productivity. State conflicts can also slow you down.
A monolith of code – It’s easy for a small project to grow over time. Without you realizing, a project that was once easy to manage with a single module can, at some point, become almost unworkable. Even when a project starts out small, it’s still smarter to not lump together all your code into one big ‘monolith’. By doing this, you’re only creating something that will get increasingly harder to read and manage over time: your team cannot coordinate to work on different parts of an application simultaneously, and it’s hard just to find the right bit of code because you need to read through the whole thing just to find it. Monolithic code is also slower, because a single apply requires it to track all of your resources.
Dependencies – If Terraform modules aren’t well organized and utilized, it’s very difficult to manage your dependencies. It can lead to Writing Everything Twice and makes it progressively more challenging to do updates. This can become a huge drain on resources.
Lack of repeatability – You can’t easily repeat important functionality in different workspaces when a project has poor organization. It also results in a lot of extra work, including more WET.
How we build Terraform modules, step by step
Before we get into the organization of modules, let’s start with how we build them.
Step 1:
One of the most irritating things is when you spend many hours building your own module, only to discover a perfectly good ‘prebuilt’ one already exists. So, before you waste time reinventing the wheel, start out by searching for open-source modules on respectable terraform registries. Good sources include Cloud Posse Modules, Terraform AWS Modules, HashiCorp, etc., and these can give you a short-cut without losing quality.
Step 2:
Once you’ve made sure there’s not a ready-to-use module already available, you can write your own. Make sure you pay attention to how you plan to deploy these and always divide functionalities or operations wherever possible (see step 3, below). We’ve created a lot of Terraform modules over the years, and you can find them here.
Step 3:
Save your modules in a logical folder structure that divides key functionalities up across multiple modules. Remember, they’re called modules for a reason, so use them in the most ‘modular’ way you can. Because each module can refer to other modules, the code also becomes lighter and easy to read. It’s also easier to update, because much of the code has been abstracted away into readable references.
Step 4:
Your logical folder system of modules should be used as a template, which can be used to form the basis of new projects or for extending your current one. Thanks to your easy folder structure, it is easy to copy an adapt these as you need. You’ll see the benefits as soon as you need to extend capabilities or create different versions of functionality for new user groups.
An organized system for keeping Infrastructure as Code (IaC) highly adaptable and flexible
Next, let’s take a look at how you can keep your IaC flexible and easy to update. The organized system we’ll outline in a moment will contribute a lot to this, by making it easier to read and update code – but you can gain a lot just by using the right tools too.
Preferred tools
One of the most important of these is Terraform, because it is cloud-agnostic and highly versatile. As well as working seamlessly with major cloud providers like AWS, Google, and Azure, it can also handle a variety of databases like PostreSQL and MySQL.
Terraform, however, also has limitations that make it hard to get the level of modularity and control you need. This is why Terragrunt has high value.
Terragrunt creates a lightweight ‘wrapper’ around Terraform, and this makes it easier to manage in an agile and efficient way. As well as easily referencing other Terraform modules (allowing the same modules to be used for different inputs), Terragrunt also comes with tools that can help you eliminate the need to Write Everything Twice.
You can also easily manage dependencies with Terragrunt, with clear dependency blocks that define which modules are to be deployed, and the deployment order.
By allowing your deployment to be divided into smaller modules, you can avoid the need for a single, cumbersome Terraform deployment. It’s also much easier (and faster) to make updates as you can easily find the right piece of code and multiple people can work different parts of your project at the same time.
Git is another indispensable part of this methodology. As well as making everything as transparent as possible, it enables you to have control over who can make changes and it offers automations for CI/CD.
Logical structure for modules
The second part of keeping your IaC easy-to-maintain relies on a logical folder structure for modules and workspaces. The goal here is to create a single system that makes it easy to find the same analogous code in different workspaces. This makes it clear and avoids confusion.
It makes good sense to start by dividing your project into two workspaces: one for Development, and one for Production.
Within each workspace, you should then have analogous folders for things like VPC, database, and K8s, and keep all the relevant modules in those. It makes them easy to find, read, and update.
You should also separate global options from workspace-dependent ones. This makes sense when you consider the prospect of a simple project growing over time. In this case it will require different configurations for different environments or users.
By using global configurations for things like whitelisted IP addresses, region, and management account id, you can simplify configuration updates and ensure consistency.
You can see an example of our preferred organized folder structure here.
Also, here’s a brief overview of our folder structure:
.github for CI/CD configurations, repos, and automations.
Cloud for putting all the deployments. This is then sub-divided into different account environments like Production and Development, and shared accounts like Security, Networking, and Keys. Deployment folders are numbered, and the terragrunt.hcl (at the root level for all subfolders) details which modules are used in each deployment and the configurations.
Template contains local templates for modules used by all deployments.
global.hcl is in the main folder and contains the global-level configurations for all environments and accounts.
Accelerating your innovation with Blackbird Cloud
As you can see, it’s not the most complicated system in the world – and that’s its strength.
Our system enables an easy to maintain and well-organized system for retaining maximum control over your infrastructure, resources, and dependencies. Because the system is so intuitive, every team member can find the right bit of code very easily. It also means that you can collaborate more effectively to update and extend your project.
By following this strategy, you can accelerate the delivery of new capabilities, which frees up valuable resources for working on other projects too.
Want to discuss your situation in detail? Get in touch.
