Platform engineering seems to be all about Kubernetes. Platform engineering tooling landscapes are awash with tools focused on Kubernetes, and most reference architectures assume that internal developer platforms are only about deploying containerised microservices to Kubernetes clusters.
Kubernetes is undoubtedly a core component of many platform engineering initiatives. According to the Cloud Native Computing Foundation’s 2023 Survey, 66% of organizations are already using Kubernetes in production, with 18% more evaluating it.
But: Organisations that use Kubernetes also have infrastructure outside of Kubernetes. Many even have most of their infrastructure elsewhere. Some don’t use Kubernetes at all.
Which means that there is a need to manage infrastructure outside of Kubernetes. It means managing hybrid infrastructure that is spread across Kubernetes, Linux VMs, Windows servers, serverless infrastructure, bare metal servers and other curiosities.
In this post, I want to talk about the challenges of managing hybrid infrastructure, how they can be solved, and what the benefits are of focusing platform engineering on the entire infrastructure, and not just containers in Kubernetes.
Why there will always be infrastructure outside of Kubernetes
There are some types of workloads that will simply never move to Kubernetes. Why?
- Legacy Architecture – Older applications are not designed for containerization or would need significant refactoring to work in a Kubernetes environment. Often, the cost involved in moving legacy applications to a Kubernetes environment is prohibitive, meaning that they will remain uncontainerised indefinitely.
- Stateful Applications – Stateful apps (like databases) often require complex configurations for persistent storage, making them harder to manage in Kubernetes.
- Performance Concerns – Kubernetes introduces overhead, which can impact performance-sensitive applications that require direct hardware access or low-latency processing.
- Complexity – Kubernetes can be overkill for small, simple applications, leading to increased operational complexity without significant benefits.
- Resource Requirements – Running Kubernetes itself requires resources (control plane, worker nodes) that might be prohibitive in resource-constrained environments.
- Compatibility Issues – Some software may have specific OS dependencies, custom configurations, or legacy tooling that are difficult to containerize and orchestrate with Kubernetes.
- Regulatory or Security Constraints – Some industries have strict requirements that may make containerization challenging or not worth the effort, especially when dealing with sensitive data.
Whether you recognise only one of all of the points above: Infrastructure outside of Kubernetes is here to stay.
Pain Points of Managing Infrastructure Inside and Outside Kubernetes
Managing infrastructure both inside and outside of Kubernetes creates significant challenges:
- Fragmented Visibility: With Kubernetes managing containers and other tools handling legacy systems or VMs, engineers struggle to get a unified view of their infrastructure.
- Inconsistent Developer Experience: Developers often encounter a fragmented experience when accessing both Kubernetes and non-Kubernetes services. They may need different tools, processes, and interfaces depending on the service.
- Complex Networking: Networking between Kubernetes clusters and non-Kubernetes environments (e.g., on-premise systems or cloud VMs) is often complex and requires manual intervention or custom solutions.
- Infrastructure Silos: Different teams may use different infrastructure management tools for Kubernetes and other resources, leading to siloed expertise and tooling. This slows down troubleshooting, upgrades, and general management.
- Security & Compliance Risks: Ensuring consistent security policies and compliance across both Kubernetes-managed and traditional environments can be a major headache, especially in regulated industries.
These are pains that anyone handling infrastructure across different technologies knows. In many organisations, this leads to significant duplication of effort, because many operational capabilities are implemented twice or even more often, separately for different types of infrastructure. This is not only more costly, it also creates incentives for keeping things separate: A lot has already been invested into building up separate capabilities for separate stacks. If you are in this situation, you will probably easily think of one or several important projects or initiatives in your company that are seriously blocked because of this kind of divided infrastructure.
Organisations that take platform engineering seriously (Read more about this topic: Why platform engineering) and want to provide self-service capabilities to software engineers (and other stakeholders) need to think beyond these silos.
Read more about Platform Engineering Challenges here.
The good news? It is possible, not just to unify management of infrastructure and applications across technology boundaries, but also to do so in a stepwise manner that is feasible to start with limited resources.
How to go Bring Together Infrastructure Silos
There is one uncomfortable fact at the center of this: The sooner you make the decision to focus on cross-technology enablement, the cheaper and easier it will be. It’s uncomfortable to say that because for most organisations, a lot of decisions have already been made a long time ago that make it a lot more difficult to switch gears now.
But the truth still holds: The sooner you make this decision, the easier it will be. With each day that passes, more will be invested into siloed tooling that will be harder to harmonise and integrate, with more resistance to it the more has already been invested. So today is as good a day as any to start focusing your platform engineering efforts on the entire infrastructure – better today than tomorrow.
To get things started, you will need a specific pain, problem or requirement that is impossible to fulfil with the current setup. Once you do, the steps to take are simple in theory but hard to do in practice:
- To solve this problem, choose tools that are technology agnostic. You probably already have some. Terraform doesn’t care where it deploys infrastructure to. Obviously, Cloudomation is a good choice for a technology agnostic layer of abstraction and integration 😉.
- Solve the problem you set out to solve. Then use this as an example to show that cross-technology orchestration is possible. If you choose Cloudomation, we will help you every step along the way, from analysing your situation to find the best way forward, implementation and operation, to formulating arguments to showcase the value you created.
- Find the next biggest pain to solve. Rinse and repeat.
It will not be easy. But it is worth it.
Benefits of integrated hybrid infrastructure
What will you gain?
In the short term, mostly trouble 😅.
But in the long term, being able to manage infrastructure across Kubernetes and other technologies will bring you large benefits:
- Improved Visibility & Control: Being able to see and interact with infrastructure and applications across technology boundaries hugely simplifies operations. Centralized monitoring, logging, and policy enforcement help ops teams manage everything more effectively. This makes it easier to detect issues, enforce compliance, and manage cost.
- Better Resource Utilization: One of the biggest cost factors in siloed infrastructure is the duplication of services and functionality for different technology stacks. Creating a unified layer of visibility and control allows organisations to continue to leverage existing investments in non-Kubernetes infrastructure (e.g., Windows servers, legacy apps, databases) while still avoiding duplication of effort.
- Smoother Migrations & Transitions: Supports gradual modernization by allowing legacy and cloud-native systems to coexist. Enables hybrid strategies where workloads move to Kubernetes over time without disruption.
- Unified Developer Experience: Developers get a consistent interface to access services—whether running on Kubernetes, VMs, or elsewhere. This reduces cognitive load and saves a lot of time.
Fortunately, many of those benefits will be tangible quickly. It’s not necessary to have a unified layer of abstraction across everything to start seeing its value. Even if it’s possible to see and manage only e.g. some Windows servers and some Kubernetes services via one interface, this will already reduce complexity for operations teams, and make it tangible to stakeholders why this is useful and valuable. From there, it is possible to integrate additional infrastructure pieces bit by bit.
How to start? Some House Advertising
To successfully implement a unified layer of abstraction across technologies, communication, change management, and organisational clout are essential. But technology also matters. With Cloudomation, you get a battle-tested, powerful platform to integrate and manage infrastructure and application across technology stacks – and a team of experts to support you every step along the way.
From the beginning, our focus at Cloudomation has been to enable infrastructure management outside of, and in addition to, Kubernetes. If you are struggling with managing infrastructure without or in addition to Kubernetes, we should talk.
We don’t promise to make your job easy: Platform engineering is not easy, and it never will be. But we promise to make your job possible: To give you the tools that enable you to build a robust layer of abstraction across whatever infrastructure you have.
