The problem with developing blindly

  • Published

As described here, a common problem when developing multi-component software is that it is increasingly difficult to impossible to run the entire software locally as part of a local development environment. As a consequence, a growing number of software developers are getting used to working blindly: pushing code changes directly to a staging environment without verifying their changes locally. This has three very bad consequences:

  • Slow or missing feedback loop for developers
  • Quality issues in the software
  • Increased dependency on repair mechanisms

Let’s discuss each of these challenges in detail.

Slow or missing feedback loop for developers

The feedback loop for developers is a lot slower and sometimes missing completely. Developers only get results from unit tests right away. Any integration tests, smoke tests, performance tests or other tests run remotely and, depending on the setup available in each company, often bulk test changes of several developers. Developers either get very late feedback that their code change was problematic, or they never get specific feedback related to a specific code change at all. This leads to slower learning and much longer and more costly bug fixing. In addition, developers are likely to push larger changes at once.

When developers are able to build the software locally and if this is working fast, a typical workflow consists of developers making many small changes and building several times in between to verify that their changes work as expected. When they finish their ticket (or feature) and commit their changes (e.g. to staging), they already know quite well that their changes work, and have probably already ironed out a lot of bugs. In addition, such a workflow makes it possible for developers to quickly and easily “learn by doing” by simply making changes, building the software locally and seeing what their changes do.

SDLC with local build
SDLC with local build

If there is no local build available, developers will complete a feature or ticket and make a larger commit to the repository, where a build and tests will run. When there are issues (and there always are: it is impossible to write perfect code), the developers have a much larger chunk of code to go through that could have caused the issue. In addition, the developer will have had a lot less opportunity to learn about the behavior of the code. This is particularly harmful for junior developers, who have a much harder time learning about the software by just reading and writing code without the opportunity to try things out in their own local environment.

SDLC without local build
SDLC without local build

Quality issues in the software

As a direct consequence of the inability to verify their changes locally, many more bugs will make it into staging, and many more bugs will make it into production. The sooner in the software development life cycle (SDLC) a bug is found, the faster and cheaper it is to fix. By removing the earliest stage of verification in a local environment, bugs by definition pop up later and are harder and more costly to fix.

Increased dependency on repair mechanisms

The next direct consequence is that software companies increasingly invest in hardening their CI/CD pipelines and building automated repair mechanisms such as automated rollbacks, staggered releases as well as ever-increasing investment in automated testing.

All of those are great things to have in place. They are, however, post-fact mechanisms that help detect and deal with issues fairly late in the SDLC. Once a bug reaches production and causes an issue that is so severe that a rollback must be done, then it is great if the rollback is automated. However doing a rollback is the most expensive fix for an issue that there is. The bug will still have to be fixed in the software. The rollback and any additional costs such as production outages, penalties to be paid to customers for SLA breaches and loss of trust are all additional costs that have to be borne in addition to the cost of fixing the bug itself.

This goes back to the fact that bugs become ever more expensive the later in the SDLC they are identified. The first stage of validation on a developer’s laptop is typically the one where the largest number of bugs are identified: bugs for which there is never even a ticket created, because developers fix them before they even commit to a shared repository. If that first stage of validation is removed, this increases the number of bugs that make it to all the following steps in the SDLC, increasing the dependency on costly post-fact repair mechanisms.

Solution: Enabling developers with Remote Development Environments (RDEs)

As described in the same article as referenced above, there is a solution for enabling developers with a local build even if the software is too “heavy” to be run locally: Developers can build only one or two (micro-)services locally – the ones they currently work on – and connect them to a shared remote development cluster where other services run. There are some prerequisites and challenges to doing this.

The challenges can be reduced in the same way as a lot of complexity in the SDLC is reduced: through automation. Cloudomation DevStack is a tool that provides automation to deploy RDEs that run a (small) selected set of services in the RDE – the one(s) developers are currently working on – and connect to other services in a shared dev cluster. This enables developers to build and test uncommitted code changes within their RDE in the same way they would locally. 

DevStack

  • deploys an RDE based on a developer’s specifications. Developers choose which components of their software in which version they need on the RDE. It then
  • dynamically generates configuration files to connect the components on the RDE with a shared development cluster,
  • ensures that the components on the developer’s RDE connect to the correct remote cluster or correct remote components and optionally deploys or starts remote components or clusters if they are not available, and
  • exposes all of this to the developer in a simple web-interface and a CLI.
DevStack CDE with shared dev cluster
DevStack CDE with shared dev cluster

Why an RDE? Why not deploy 1-2 services locally?

The biggest problem with automation of local deployment is the high variability in local environments, from the operating system down to specific library versions, which can affect local deployment. It is unfeasible to automate local deployments that account for all the possible variations. That is the reason why developers struggle a lot with local deployments even if the software in principle can run locally: even if a lot has been invested to provide local build scripts, there are always manual steps required, and developers need to be capable to do troubleshooting themselves to iron out issues specific to their setup.

Remote environments are standardized and much easier to automate.

A smaller concern is that local compute resources are limited. Even if only one or two services are deployed as part of the development environment, that can still put a large burden on laptops and provide poor performance for developers.

Remote Development Environments vs. Local Development Environments

In this comparison document, you will learn in detail what the most important differences are between Cloudomation RDEs and Local Development Environments.

Download now

Summary

Being able to build their software locally, developers get the opportunity to

  • learn and experiment,
  • catch and fix bugs early,

which in turn

  • significantly reduces the cost of fixing bugs
  • and increases developer’s productivity.

Software with many (micro-)services is often too heavy-duty to be run locally. To enable developers to still iterate quickly and validate their code changes with local build and testing, developers can deploy only those services they currently work on locally, and connect to other services on a shared development cluster. In this blog post, I describe the prerequisites and challenges of such a setup.

To make such a setup usable for a large number of developers, automation can help to bring down its complexity. Cloudomation DevStack is a tool that provides automation capabilities and templates to deploy remote development environments (RDEs, same as CDEs / cloud development environments) that contain a small subset of services that the developer chooses, and automates the connection to other services in a shared development cluster. The RDE contains all tools required for the developer to work like they would in a fully local development environment on their laptop.

Subscribe to the Cloudomation newsletter

Become a Cloudomation Insider. Always receive new news on “Cloud Development Environments” and “DevOps” at the end of the month. 




    Margot Mückstein

    CEO & co-founder of Cloudomation