The problem with developing blindly

Slow or missing feedback loop for developers

The feedback loop for developers is a lot slower and sometimes missing completely. Developers only get results from unit tests right away. Any integration tests, smoke tests, performance tests or other tests run remotely and, depending on the setup available in each company, often bulk test changes of several developers. Developers either get very late feedback that their code change was problematic, or they never get specific feedback related to a specific code change at all. This leads to slower learning and much longer and more costly bug fixing. In addition, developers are likely to push larger changes at once.

When developers are able to build the software locally and if this is working fast, a typical workflow consists of developers making many small changes and building several times in between to verify that their changes work as expected. When they finish their ticket (or feature) and commit their changes (e.g. to staging), they already know quite well that their changes work, and have probably already ironed out a lot of bugs. In addition, such a workflow makes it possible for developers to quickly and easily “learn by doing” by simply making changes, building the software locally and seeing what their changes do.

If there is no local build available, developers will complete a feature or ticket and make a larger commit to the repository, where a build and tests will run. When there are issues (and there always are: it is impossible to write perfect code), the developers have a much larger chunk of code to go through that could have caused the issue. In addition, the developer will have had a lot less opportunity to learn about the behavior of the code. This is particularly harmful for junior developers, who have a much harder time learning about the software by just reading and writing code without the opportunity to try things out in their own local environment.

Quality issues in the software

As a direct consequence of the inability to verify their changes locally, many more bugs will make it into staging, and many more bugs will make it into production. The sooner in the software development life cycle (SDLC) a bug is found, the faster and cheaper it is to fix. By removing the earliest stage of verification in a local environment, bugs by definition pop up later and are harder and more costly to fix.

Increased dependency on repair mechanisms

The next direct consequence is that software companies increasingly invest in hardening their CI/CD pipelines and building automated repair mechanisms such as automated rollbacks, staggered releases as well as ever-increasing investment in automated testing.

All of those are great things to have in place. They are, however, post-fact mechanisms that help detect and deal with issues fairly late in the SDLC. Once a bug reaches production and causes an issue that is so severe that a rollback must be done, then it is great if the rollback is automated. However doing a rollback is the most expensive fix for an issue that there is. The bug will still have to be fixed in the software. The rollback and any additional costs such as production outages, penalties to be paid to customers for SLA breaches and loss of trust are all additional costs that have to be borne in addition to the cost of fixing the bug itself.

This goes back to the fact that bugs become ever more expensive the later in the SDLC they are identified. The first stage of validation on a developer’s laptop is typically the one where the largest number of bugs are identified: bugs for which there is never even a ticket created, because developers fix them before they even commit to a shared repository. If that first stage of validation is removed, this increases the number of bugs that make it to all the following steps in the SDLC, increasing the dependency on costly post-fact repair mechanisms.