Developing with Kubernetes is hard. Fortunately, a range of tools are available to support developers to work with Kubernetes in development. Here, I want to provide an overview of which tools exist, what they do and how they make development with Kubernetes easier.
Kubernetes Tools: overview
There are a range of different tools available which support different parts of the development workflow:
First, the basics: Docker, kubectl, managed Kubernetes
Developing applications that run in Kubernetes in production requires the use of container technology – usually Docker – and, obviously, Kubernetes. I give a brief overview of Docker and kubectl, since they are the foundations of any development work on Kubernetes-based applications. Theoretically (and also in daily practice by many developers), Docker and kubectl are really all you need to develop Kubernetes-based applications.
For the sake of clarity, I also provide a short list of managed Kubernetes offerings and related tools.
Second: tools to emulate or run Kubernetes locally
These are tools that allow developers to run a Kubernetes cluster, or something that acts like a Kubernetes cluster, locally.
These tools are: minikube and kind
Third: tools to build, deploy and interact with containers in Kubernetes clusters
This set of tools makes it easier for developers to build containers, deploy containers to a (local or remote) Kubernetes cluster and interact with containers once they run in a Kubernetes cluster.
These tools are: Draft, Devspace, Skaffold, Tilt, Garden
Fourth: remote development environments for Kubernetes
These are tools that move the development work away from developers laptops and into the cloud, by providing work environments in the cloud where containers are built, tested, and deployed to a remote cluster.
These tools are: Okteto, Eclipse Che, DevStack
Below, I will give a brief description of what each tool does, how it fits into the development process and give some links to additional information.
Docker
Docker is not really a Kubernetes development tool, but since the majority of developers who work on applications that run in Kubernetes in production use Docker as the main or only runtime in development, it merits a mention here.
What is Docker?
Docker is the containerisation technology that underlies the majority of applications that run in Kubernetes in production. It is not the only container runtime, but by far the most common one.
Applications that run in Kubernetes in production have to be containerised. These containers can be run in a Kubernetes cluster, but they can also be run directly on a laptop or VM using Docker. This is the most common setup that developers use when developing containerised / dockerised applications that run in Kubernetes in production.
Docker is open source.
Where does Docker Desktop fit into the development process?
There are a number of handy tools that can be used together with Docker. Docker Desktop is a bundle of Docker tools, including the Docker engine, as well as a command-line interface (CLI) and a graphical user interface to manage container images, containers and containerised applications on a local computer. With Docker Desktop, you can build containers, publish containers, run containers, start and stop containers etc. It also has a feature that allows you to deploy a single-node Kubernetes cluster locally. Docker Desktop also includes Docker compose, which is a tool to define applications that consist of several containers. In a Docker compose file, developers specify which containers the application consists of and how they communicate with each other.
A very common development workflow is to use Docker compose to run a containerised application locally during development, and then deploy it to Kubernetes in production.
Links
Product website: https://www.docker.com/
Github: https://github.com/docker
Documentation: https://docs.docker.com/
kubectl
Docker is not really a Kubernetes development tool, but since the majority of developers who work on applications that run in Kubernetes in production use Docker as the main or only runtime in development, it merits a mention here.
What is kubectl?
kubectl is the native command line tool for Kubernetes. It exposes all Kubernetes functionality for use via a terminal, which can be used interactively, or scripted. It can be used to configure and manage Kubernetes clusters, manage workloads, interact with containers, deploy to a cluster, debug, ….
Where does kubectl fit into the development process?
kubectl is a powerful tool that allows developers to use the full spectrum of Kubernetes functionality. As such, it is also fairly complex. Many of the other tools described here aim to make Kubernetes handling easier by abstracting or packaging functionality that is also available via kubectl. However since kubectl exposes all Kubernetes functionality, it can (and often is) used on its own as the only tool to manage and interact with a Kubernetes cluster.
Links
Documentation: https://kubernetes.io/docs/reference/kubectl/
K3s, MikroK8s, GKE, EKS, AKS, DOKS, Rancher
All these lovely abbreviations stand for different Kubernetes distributions that are built primarily for running Kubernetes in production. Since they are not development tools, I will only briefly say what each of them is.
K3s is a Kubernetes distribution built for running in resource constrained infrastructure, such as for example IoT / edge locations or other compute infrastructure that doesn’t scale horizontally. For example, with K3s you can run a Kubernetes cluster on a Raspberry Pi.
MicroK8s also allows to run Kubernetes clusters on all kinds of hardware, small and large, in the cloud or the edge or elsewhere.
Rancher is a platform for running and managing self-hosted Kubernetes clusters.
GKE is the managed Kubernetes offering of Google Cloud.
EKS is the managed Kubernetes offering of Amazon Web Services (AWS).
AKS is the managed Kubernetes offering of Azure (Microsoft).
DOKS is the managed Kubernetes offering of Digital Ocean.
minikube
What is minikube?
Minikube is a tool that lets developers run a single-node Kubernetes cluster in a container or a VM on their local computers. The primary purpose of minikube is to make it simple to run a Kubernetes cluster locally. This is supposed to make both experimenting with Kubernetes as well as developing Kubernetes-based applications easier.
minikube clusters have many of the features of a normal Kubernetes cluster, such as load balancing, multi-cluster support, node ports, persistent storage volumes and ingress management. It is compatible with three container runtimes: docker, containerd and cri-o.
minikube is open source.
Where does minikube fit into the development process?
minikube makes it manageable to run a Kubernetes cluster on a laptop or local computer. As such, it allows developers to deploy their applications to that local minikube cluster, inspect and work with their applications in that cluster, and do so in a way that is close to a production environment, where the application would run in a full Kubernetes cluster.
minikube is used together with other Kubernetes development tools such as skaffold, tilt, garden, draft, kubectl or Devspace or DevStack. These other tools make it easier to deploy applications to a cluster, for example a minikube cluster, and provide handy functionality to interact with containers running in a minikube (or other type of) cluster.
Links
Github: https://github.com/kubernetes/minikube?tab=readme-ov-file
Documentation: https://minikube.sigs.k8s.io/docs
kind
What is kind?
kind is a tool for running a Kubernetes cluster locally, using Docker container nodes. Different to minikube, kind does not run in a VM or a Docker container, and it allows to run multi-node clusters. Other than that, it’s use case is very similar to that of minikube: Developers can use it to run their applications locally in a way that is similar to a production environment in Kubernetes.
kind was initially created to make the development of Kubernetes itself easier. Because of this, it supports building Kubernetes releases from the Kubernetes source code.
It’s still a fairly young project but already quite popular with developers working on Kubernetes-based applications.
kind is open source.
Where does kind fit into the development process?
Same as minikube, kind is the place where developers deploy to and run their application containers. It is used together with other Kubernetes development tools such as skaffold, tilt, garden, draft, kubectl or Devspace or DevStack. These other tools make it easier to deploy applications to a cluster, for example a minikube cluster, and provide handy functionality to interact with containers running in a minikube (or other type of) cluster.
Links
Github: https://github.com/kubernetes-sigs/kind
Product website: https://kind.sigs.k8s.io/
Documentation: https://kind.sigs.k8s.io/docs/
Draft
What is Draft?
Draft is an open source tool from Microsoft intended to make it easier to try Kubernetes for the first time. It can create basic Dockerfiles, Kubernetes manifests, Helm charts and other configuration files (including Github Actions) to containerise an application and deploy it to a Kubernetes cluster. To do this, it guides the user through a series of questions and generates configuration files based on the user’s answers. The generated configuration files are minimal and contain only basic configuration options. However it is a handy tool to get a first idea of how a Dockerfile, Kubernetes manifest and Helm chart can look like for your application. It also has a “validate” feature which scans existing Kubernetes manifests and checks if they follow (AKS) best practices.
Draft is open source.
Where does Draft fit into the development process?
Draft is useful when trying Kubernetes for the first time, also with an application that is not yet containerised. By generating minimal configuration files, it can help users understand how these configuration files are structured and get them started on writing their own, or extending the ones generated by Draft. It is less useful when your Docker and Kubernetes configuration already exists and has no features to make day-to-day management of Kubernetes or deployment to a local Kubernetes cluster easier. As such, it is primarily a tool for learning.
Links
Github: https://github.com/Azure/draft
Devspace
What is Devspace?
Devspace is a command-line tool that provides functionality to:
- Build and deploy container images
Watch files and update containers on the go (hot reload) - Stream logs from your containers
- Synchronise files between your containers to your local machine (2-way file sync)
- Open terminals in your containers
- Forward ports from your containers to localhost
- List and watch containers and their status
- Tag container images based on a tag schema
- Push container images to registries
It also has a small UI, which mostly consists of a terminal to run commands. It’s client-only, using kube-context to communicate with Kubernetes clusters (same as kubectl).
Devspace is open source.
Where does Devspace fit into the development process?
Devspace is a tool that makes it easier to work with containers running inside Kubernetes clusters, and to handle container build and deployments. It doesn’t replace the actual Kubernetes cluster but makes it easier working with one. It works with any local or cloud-based Kubernetes cluster: minikube, k3s, MikroK8s, kind, GKE (Google), EKS (AWS), AKS (Azure), DOKS (Digital Ocean), Rancher and other self-managed Kubernetes clusters.
Links
Github: https://github.com/devspace-sh/devspace
Product website: https://www.devspace.sh/
Documentation: https://www.devspace.sh/docs/
Skaffold
What is Skaffold?
Skaffold is a tool for building and deploying containerised applications to a (local or remote) Kubernetes cluster. Triggered by changes in your source code, Skaffold rebuilds your container images, pushes them to a container registry, tags them based on a tagging policy, tests the containers and deploys them to a Kubernetes cluster. (All of this is of course subject to configuration and optional.) It also aims to facilitate the standardisation of Kubernetes development by providing both a yaml-based configuration standard that allows developers to clone a repository and build and run it right away, as well as context and platform aware configuration options that make the same basic configuration work in different environments.
It is a client-only command-line tool.
Skaffold is open source.
Where does Skaffold fit into the development process?
Skaffold makes it easier to continuously build and deploy containers during development. When working with Skaffold, you still need a Kubernetes cluster to deploy to – either locally or remotely, e.g. a minikube cluster on your laptop or a normal cloud Kubernetes cluster. You may also still want to use tools like Devspace that make it easy to then connect to, stream logs from, and generally work with your containers after they’ve been deployed to your Kubernetes cluster.
Links
Product website: https://skaffold.dev/
Github: https://github.com/GoogleContainerTools/skaffold
Documentation: https://skaffold.dev/docs/
Tilt
What is Tilt?
Tilt is a tool for building and deploying containers to a (local or remote) Kubernetes cluster. It watches for code changes, which trigger a rebuild of your containers, which are then automatically deployed to your Kubernetes cluster. It also has features that support developers to debug services that run in Kubernetes. Tilt focuses particularly on applications that consist of many services (i.e. containers) that have complex dependencies that are hard to debug jointly, which Tilt aims to make easier.
Where does Tilt fit into the development process?
Similar to Skaffold, Tilt facilitates building and deploying containers. You still need a Kubernetes cluster to deploy your containers to, for example a local minikube cluster or a Kubernetes cluster in the cloud. Tilt has a stronger focus on providing insights into the running containers in Kubernetes. You might still use other tools like Devspace in addition to make it easier to stream logs or open a terminal within a container, if you need that.
Links
Product website: https://tilt.dev/
Github: https://github.com/tilt-dev/tilt
Documentation: https://docs.tilt.dev/
Garden
What is Garden?
Garden is a devops tool that aims to make it easier to deploy containers to Kubernetes for development, testing and production with the same configuration. This should make it easier to spin up new Kubernetes clusters on demand, reducing the need to share environments and making it easier to test features in an isolated environment. It can watch for changes in the source code and redeploy only changed components to a cluster. Garden also provides functionality to stream logs from containers running in a Kubernetes cluster, and to combine services running in different environments, so that developers can e.g. run one or a few services locally and connect to other services in a remote Kubernetes cluster.
Where does Garden fit into the development process?
Garden deploys containers to existing Kubernetes clusters. Like with other deployment tools, this means that you still need Kubernetes clusters that Garden deploys to. This can be a local minikube cluster, or a normal cloud Kubernetes cluster. Garden then takes care of automatically rebuilding, testing and deploying your containers after a change to the source code. Developers can also use Garden to stream logs from containers.
Links
Product website: https://garden.io/
Github: https://github.com/garden-io/garden
Documentation: https://docs.garden.io/
Okteto
What is Okteto?
Okteto is a Kubernetes development platform. It watches for changes in your source code on your laptop. When there is a change, this is synchronised with a development container in a Kubernetes cluster. In this development container, the code is built (i.e. containers are built) and deployed to the Kubernetes cluster instantly. Its unique feature is the development container where the container build is done, which provides a standardised, controlled environment that can be configured to mirror production build environments and which has more compute resources available than a laptop.
There is an open source version of Okteto available with a minimum feature set, and a commercial offering with extended features.
Where does Okteto fit into the development process?
Okteto makes it easier and faster for developers to test and deploy code changes to a containerised application running in Kubernetes. Its main claim is to save developers a lot of time by automating the build and deployment process, providing something like a hot reload for an application that runs in Kubernetes. As a build and deployment tool, it still needs a Kubernetes cluster to connect to. While it is conceivably possible to use Okteto with a local minikube (or similar) cluster, this is not the idea of Okteto, which specifically aims to provide more remote compute resources to developers to speed up build and deployment. In a normal setup, Okteto would be connected to a cloud Kubernetes cluster.
Links
Product website: https://www.okteto.com/
Github: https://github.com/okteto/okteto
Documentation: https://www.okteto.com/docs/
Eclipse Che
What is Eclipse Che?
Eclipse Che is a cloud development environment running in Kubernetes. Developers work inside a development pod in a Kubernetes cluster, to which they deploy development containers which are their work environments that contain e.g. the source code, language runtimes and an IDE backend, alongside several other application containers. Developers can use an online IDE such as VS Code or run a Jetbrains IDE backend in their development container, which they can connect to with their local Jetbrains IDE (e.g. IntelliJ or PyCharm etc.). Each developer has their own development pod on the container. There, they can deploy several containers alongside their development container, meaning that each developer can have their own full instance of the application running in their own Kubernetes pod.
Eclipse Che is open source. There is a commercial SaaS offering of Ecplise Che available from Red Hat.
Where does Eclipse Che fit into the development process?
Eclipse Che provides functionality for developers to connect to and work inside a development container in a Kubernetes pod on a Kubernetes cluster. Using Eclipse Che, developers can work fully in the browser from any device, or need only an IDE thin client on their laptops. Everything else runs in the Kubernetes cluster. While it is technically possible to use Eclipe Che with a local minikube cluster, this is not the idea: It aims to reduce dependence on laptops and provide more compute resources and standardised environments to developers. So typically, Eclipse Che will run in a cloud Kubernetes cluster. Che also doesn’t provide deployment automation or any of the other handy features (such as log streaming, terminals inside containers etc.) that other Kubernetes tools described here offer, so developers will still use other tools together with Che in their development workflow.
Links
Product website: https://eclipse.dev/che/
Github: https://github.com/eclipse-che/che
Documentation: https://eclipse.dev/che/docs/s
DevStack
What is DevStack?
Cloudomation DevStack is a cloud development environment platform that can be used for developing applications that run in Kubernetes in production, either with a Docker-only development workflow or a Kubernetes-based development workflow. DevStack allows to define development environments that can consist of several containers deployed into a Kubernetes cluster, or on a virtual machine using only Docker. Docker and Kubernetes development tools can be deployed alongside the application, directly on the VM or in a development container in a Kubernetes cluster. DevStack can watch for changes in the source code and rebuild and redeploy containers on the VM, or in a Kubernetes cluster. Definition of development environments is done in Python and can be heavily customised. In addition, DevStack allows automation of container testing and deployment, and can be used to automate the entire CI/CD pipeline.
Where does DevStack fit into the development process?
As a platform for development environments and deployment automation, DevStack makes it fast and easy for developers to set up standardised development environments in which they do their development work. By watching for changes and redeploying code changes into containers, it removes any manual steps and makes the development process a lot faster. Contrary to other development environment platforms, it allows users to choose whether to use a Docker-only development workflow, or to run development environments within a Kubernetes cluster. In both cases, DevStack can either use existing infrastructure (i.e. deploy into an existing Kubernetes cluster) or automate infrastructure creation (i.e. to create VMs on which to deploy the containers). For a Docker-only workflow, DevStack comes with features that allow port forwarding, log tailing and opening a terminal session inside the containers through its CLI. For a Kubernetes-based workflow, additional Kubernetes development tools can be used to interact with the containers within the Kubernetes cluster.
Links
Product website: https://cloudomation.com/devstack/
Documentation: https://docs.cloudomation.com/devstack
Why to use these tools
I feel it’s important to mention that it is not necessary to use a Kubernetes cluster (or something that acts like a Kubernetes cluster) in order to develop and work on applications that run in Kubernetes in production.
Any application that runs in Kubernetes is containerised. These containers can be run in a Kubernetes cluster, but they can also run without one, using e.g. only Docker. The vast majority of developers that work on applications which run in Kubernetes in production do not run a Kubernetes cluster locally and do not use a remote cluster for development. The vast majority of developers simply run their containers directly in Docker on their laptops.
It’s a lot simpler and it’s fine for many cases. However, there are three reasons why it can be necessary to run containers in Kubernetes when developing:
- Containers behave differently when running in Docker than when running in Kubernetes. In some cases, this can lead to issues when developing only with Docker, and then deploying to Kubernetes.
- To develop some features that are supposed to work in Kubernetes, you simply need Kubernetes – for example, to develop Kubernetes network policies.
- For large applications with many components, it can simply be impossible to run them on a laptop because it doesn’t have enough compute resources.
Many companies choose to ignore (1) and deal with issues if they arise, when they arise. (2) is a bit of an edge case for which many companies find workarounds. The main reason why development within Kubernetes is becoming more common is (3): Applications are becoming bigger and more complex, and developers are increasingly unable to deploy them locally, with or without Kubernetes. For this scenario, a local Kubernetes cluster makes the situation even worse since it eats up resources as well.
This means that the only reasons to run a Kubernetes cluster locally is to get an environment that is more similar to the production environment, and/or to develop Kubernetes-specific features and configurations. There is no actual benefit for running Kubernetes locally beyond that. The load balancing and elastic scaling of Kubernetes is pointless locally.
Considering that it is difficult to run a minikube or similar cluster locally, and even more difficult to set up and maintain remote Kubernetes development environments, it is important that you consider whether or not you really need it and if your need is strong enough to warrant the effort.
Because, as many a developer or devops engineer will tell you, Kubernetes is a pain. It is a complex piece of software that requires a lot of knowledge and effort to set up, configure and run.
It is also really cool: It provides elastic scalability that makes it possible to use the flexibility of on-demand compute resources in the cloud. That makes it worth the effort for applications with varying load.
But even if it makes sense to run your application in Kubernetes in production (and in many cases, it doesn’t), that doesn’t mean that it is a requirement to run Kubernetes on every single developer’s laptop.
If you do need to do this, I hope this post gives you some useful pointers to find your way around Kubernetes development tools.
Feedback is welcome to margot@cloudomation.com
