Each IDP is unique, but there are some underlying characteristics that most of them share. Very basically, each IDP consists of approximately three “parts”:
- An IDP frontend, through which developers access the IDP and e.g. read documentation, deploy services or check the status of their build. An IDP frontend can be a graphical user interface (GUI) and/or a command-line interface (CLI) and/or an API (and ideally, all three). IDP frontends are often called portals.
- An IDP backend, which handles integration with other tools, automation, and often configuration management (to some degree). IDP backends are sometimes called platform orchestrators.
- Lots of other tools that are integrated with the IDP. IDPs are typically something companies build which already have a lot of tools and automations for core processes in place – such as CI/CD pipelines – which already tie together other tools such as test automation, build automation, version control systems, infrastructure automation tools, custom scripts etc. IDPs typically don’t replace those existing tools, but provide an integrated layer on top through which developers can access both information as well as functionality from those tools. For example, the IDP might contain information about a component which includes the readme from the source code repository, as well as statistics calculated from past runs of a CI/CD pipeline. One of the main reasons why IDPs are always custom is the fact that they are built on top of and integrated with a varied landscape of existing tools that is different in each company.
If you picture the large and complex toolstack that is used in software development, you can think of the IDP backend as a spider that sits in the middle of all these tools and ties them all together, and the IDP frontend as a shiny layer on top that allows developers to discover and use the underlying tools in a self-service manner.
Architecture
The following architecture diagram from cnoe (Cloud Native Operational Excellence) gives a more detailed overview:
In this diagram, the “Developer Portal” would be the frontend of the Internal Developer Platform. The “Workflow Orchestration” bit would be the backend of the Internal Developer Platform.
Below is a more detailed example architecture featuring our own products and showing which other types of tools and services could be part of an IDP:
Core functionality of an IDP
The main idea of an IDP is to tie together tools, services, configurations and other information and assets in one place. Software Engineers, but also other stakeholders e.g. in operations teams should be able to use the IDP as a single entry point to discover and interact with a company’s applications and infrastructure.
As such, the IDP backend typically needs to be strong in two types of functionality:
#1 Integration
The IDP needs to be able to connect a large number of different “things”, mainly in two categories:
📌 Information
An IDP should be able to get information from different sources, and display it to engineers in one place. This can be information harvested from
- git repositories (like readme files),
- uptime statistics from cloud infrastructure,
- manually maintained documentation that is kept either directly on the IDP, or again collected from elsewhere,
- links to swagger docs for REST APIs,
- information from databases, or any other type of information that is useful to engineers and relevant to the self-service tools provided via the IDP.
Ideally, the IDP backend should be able to integrate with a variety of sources, work with a variety of formats, and transport information both ways: from their original source to the IDP, and from the IDP back to the original source.
🔹For example:
The IDP backend could harvest information from several other systems and transform them into the format used by the Backstage software catalog, if backstage is used as the IDP frontend. Information updated manually in the Backstage software catalog should then be propagated back into integrated systems by the IDP backend.
📌 Functionality
The IDP backend needs to be able to expose functionality independently of underlying technology or tool that provides the functionality, and expose this functionality to engineers in a unified format via the IDP frontend.
As such, the IDP backend should be able to
✅ trigger events / processes in other tools:
- e.g. calling webhooks,
- making API requests,
- executing scripts on remote systems etc. in order to
- e.g. trigger a build or a CI/CD pipeline,
- provision infrastructure via Terraform or directly via a cloud provider’s API etc.
✅ receive events: e.g.
- by offering custom webhooks that can be triggered by other tools,
- exposing custom APIs to allow other tools to notify and trigger custom processes in the IDP backend, in order to e.g. receive a push to a source code repository, or an error in a build pipeline etc.
✅ The IDP backend should then be able to provide functionality in a unified way to engineers, so that engineers can use
- one API or CLI or graphical portal interface to use this functionality and e.g.
- see the status of their pipeline,
- copy an environment, lock an environment,
- create a feature branch system
- or use any other functionality provided via the IDP, without having to worry about the underlying technology and tools.
The functionality provided via the IDP should always follow the same structure, always be usable in the same way, so that engineers only need to learn how to use the IDP once and then have access to the entirety of a company’s infrastructure and applications.
#2 Automation
Since most companies don’t start out with an IDP but rather build one on top of already existing (complex) ecosystems, IDP backends need to be particularly strong in “filling gaps”. Most often, this means tying together and orchestrating existing automation pipelines, e.g. by
✅ Resolving dependencies and applying configurations
✅ Extending existing automation with individual steps or variations for specific use cases
This means that the IDP backend needs to be very flexible and open.
The IDP frontend should be able to expose automation to engineers in ways that best fit their existing workflows. While GUI-portals (like Backstage) are very popular, a strong IDP frontend would feature not just a GUI but
- also allow using functionality via a command-line interface (CLI)
- and/or standard API(s) (e.g. REST).
Sometimes, API and CLI can be exposed by the IDP backend directly, in which case the IDP frontend can be “just a GUI”, which can use the IDP backends APIs to communicate with the IDP backend as well.
IDP as a Layer of Abstraction
A good IDP is a layer of abstraction that allows engineers to interact with a company’s applications and infrastructure in a unified and supported way.
It should
✅ remove complexity from software engineers and other IDP user groups,
✅ while at the same time empowering the platform engineering teams to enforce policies and standards across the organisation
by having one layer (the IDP) through which all (…never happens, sure, but that’s the goal) infrastructure and applications within a company is managed.
Summary
IDPs are by definition unique to each company. But they all aim to be:
🛠 A layer of abstraction
➡️ That allows engineers to discover and interact with a company’s Applications and infrastructure
➡️ In a unified and supported way
They do that by:
✅ Integrating underlying scripts, tools, pipelines, data sources, environments etc.
✅ Exposing information and functionality to engineers in self-service
✅ In one place, with one interface – That always works in the same way, independently of the underlying technologies
To be able to do that, IDPs need:
✔ A backend with broad integration and flexible automation capabilities,
✔ A frontend that provides a unified interface for engineers, ideally allowing engineers to use this interface graphically, via an API, and via a CLI
A few examples:
▶ A test engineer can deploy any application that they need to test in the same way, e.g. via a button in the IDP frontend, or a command in the IDP CLI.
It doesn’t matter
- if the application runs in a K8s cluster, on a linux VM, or on a windows server,
- if the application is written in Java or Python, or if it uses one or another CI/CD pipeline.
➡️ To the test engineer, it looks the same and is usable in the same way.
▶ An SRE has one place where they can get information about
- all currently deployed applications and environments,
- see which version of which software is running on each of them,
- and where to go to find further information, such as links to monitoring tools hooked up to the different environments.
➡️ No matter which environment or application or server or other piece of infrastructure or software the SRE is looking for, they know they can find information about it on the IDP.
In short: A well-built IDP reduces complexity for developers while enabling platform engineering teams to enforce policies and standards centrally. 🚀
