What is platform engineering and how can you do it effectively?
Cloud-based infrastructure can support highly complex processes and incredible scalability. However, the rapid growth of cloud technologies has also led to a proliferation of tools and systems.
To wield these successfully requires deep and wide-ranging expertise – and this expertise is scarce. Platform engineering has evolved as a solution to this, because it enables all developers to use a variety of self-service tools so that development, deployment, and maintenance processes follow the same verified path.
In this blog we explore the advantages of platform engineering, highlight suitable use-cases, and explain our preferred methods.
Why is platform engineering so important for the cloud?
Platform engineering is a rising trend, and it’s predicted that 80% of software engineering will involve the use of platform engineering by 2026.
The reasons for this are quite simple: platform engineering ensures the best DevOps practices, toolchains, workflows, feedback, and security processes.
It means that instead of ‘reinventing the wheel’ every time, you create an internal development platform (IDP) that makes software development an easily repeatable process.
This greatly reduces cognitive load placed on developers. They can focus on their projects without needing to worry about the infrastructure. Instead, the platform ensures that development follows an ideal pathway that has been designed with the highest level of DevOps expertise and wisdom.
While the advantages of platform engineering can be very attractive, it’s also important to understand which situations demand this approach, and which do not.
When is platform engineering needed?
Because of the effort needed to set up an internal development platform (IDP), it only makes sense to use platform engineering when there’s a clear return on investment.
There are several scenarios where this can happen, but your team size is often the biggest factor. Once your development team exceeds 15 people, it can be more effective to use platform engineering.
Without an IDP, your infrastructure team will be tied up accommodating infrastructure-related requests from developers. This isn’t a smart way of doing things, because their time is better spent creating and maintaining a platform that automatically applies their expertise in each situation.
With more time available, your infrastructure experts can focus on creating the best possible platform for your needs, equipped with all the right governance structures and guardrails.
Another situation where you may benefit from platform engineering is when you have multiple projects with repetitive and critical (specific) requirements. When a project has highly specialized needs for security, performance, or regulatory compliance, for example, then an IDP mitigates the potential risks by enforcing these during the whole process.
And finally, you may want to use platform engineering when speed is a critical requirement, because it can accelerate your time to market for new products or features. Despite the time required to set up, a platform can still get you to the finish line faster (and reduce time on maintenance) when the product is highly complex and has stringent requirements.
What does platform engineering involve?
Your platform engineering strategy should cover the full development cycle, including ongoing feedback and maintenance, with a continuous stream of toolchains and workflows.
A good place to start is with deployment, because this must be as reproducible and transparent as possible. At Blackbird Cloud, we use GitOps for this by storing cloud manifest on Git in the form of terraform code. This makes it very straightforward to manage versions and automations.
Git also allows developers to add applications and dependencies on a Git branch in the infrastructure repository. They then just need to make a pull request to merge the changes, which triggers the infrastructure team to carefully vet changes. Automated tools can also analyze any changes for potential security issues at the same time.
The platform environment itself relies on a container orchestration system like Kubernetes (K8s) or AWS ECS, which is synchronized with the state in Git using a tool like argoCD or AWS CloudFormation. With this setup, developers have the greatest freedom to use their preferred tools, while ensuring transparency and infrastructure best practices.
Dependencies can be quite variable, depending on your tech stack and changing needs. You can’t really predict this, so the key is to give developers the ability to add or change dependencies over time, with the support of guardrails that ensure it is done the right way. Good examples include AWS Config and Cloud Custodian (c7n).
Finally, you need to ensure there’s a simple mechanism for monitoring and loggin. This is achieved through performance and logging tools like Prometheus, AWS CloudWatch, and Loki. Make sure developers have access to metrics and logs, however – otherwise these will have little value.
Guardrails for security and compliance with platform engineering
Speed and repeatability are two chief benefits of platform engineering, but perhaps the biggest value it can generate is the assurance of security and compliance.
Some of the tools we’ve already discussed above can help ensure your security posture and regulatory compliance is enforced, but there are other guardrails to be aware of too.
Application templates
Application templates are one of your most important tools for secure and compliant applications. Speed is probably the main reason to use templates, but they can also have your best practices ‘baked-in’. Templates can include a secure framework that prevents issues like unauthorized merges with simple measures like branch protection rules. One thing to remember is that these templates should not limit the options too much and be used as a starting point when defining your application, there can be changes/additions on top of the templates itself.
Software bill of materials (SBOM)
An SBOM is recommended as a vital guardrail by the US Cybersecurity and Infrastructure Security Agency (CISA), who call it “a key building block in software security and software supply chain risk management”. Your SBOM makes it crystal-clear which components belong in any application. Any difference between the SBOM and the application itself, then you know there’s a potential vulnerability to supply chain-based attacks, and action is needed.
Automated testing of code
By applying automated code testing at various stages of the application lifecycle, you can continuously ensure your applications meet all required standards for security and compliance.
Multiple checkpoints
Each stage in the pipeline should be safeguarded with security checkpoints. This involves checking code and infrastructure changes with a range of automated tools.
Ready to embark on your platform engineering journey?
Platform engineering is widely seen as the optimal path to achieve highly repeatable and rigorous software deployment for the cloud. It’s a cloud-native approach that lowers the cognitive load placed on developers and increases the velocity of the development process – without sacrificing quality.
However, it’s best to get the advice of experts when you’re starting out – because this can be a significant shortcut that avoids all the potential pitfalls on the way, and helps to keep your costs under control.
Want to discuss your situation in detail? Get in touch.
