To learn about the market and the needs of our customers, I regularly interview software developers, engineering managers, CTOs, platform engineers and other folks in software development, devops and platform engineering roles. While each company is unique, and each person has a unique perspective on the challenges their company faces, I see some clear patterns. In this blog post, I want to share what I have learned from my interviews.
Disclaimer: This is not scientific research.
The insights I present here are the result of my subjective interpretation of interviews I have conducted myself. I have a questionnaire I use for my interviews, but I frequently deviate from it, and continuously update and change it. As a result, no two interviews are the same.
The people I interviewed are not a representative sample, comprising mostly people who agreed to an interview after I contacted them via LinkedIn, so there is a strong selection bias in the sample.
I conducted the interviews with the aim to learn. While I focused my questions around the topic of development environments, development practices, and challenges in the software development process, I had no clear goal that I wanted to reach in these conversations. As a result, the learnings I present here are a fairly random collection of patterns that I saw after re-reading and analysing my interview notes.
I talked only to B2B software companies.
Here are my main takeaways:
#1 Kubernetes dominates
The vast majority of companies I talked to use Docker and Kubernetes to run their software. I expected to see a bias where younger companies would be more likely to use K8s, but I could not confirm that in my interviews. Companies that don’t use K8s are rare. Reasons for not using K8s I’ve heard are architectures that are simply incompatible (legacy fat clients) or too much operational overhead while not really needing the auto-scaling features anyway.
#2 Local deployment of the software developers work on is a pain
The vast majority of people I’ve talked to admit that running the software they work on locally is either not possible at all, or very painful to do. The main reason is:
#3 Insufficient compute resources on local laptops are a blocker for devs
The dominant pattern is that several (micro or “midi”) services are required to run an application, and they need too much resources. Laptops simply burn up. This leads to it being pretty much standard practice that developers get very big laptops, but many still struggle with the resource requirements of running their applications locally.
#4 More than half of the developers I talked to cannot run their application as part of their development environment at all
About a third of the people who don’t have their applications as part of their development environment say that’s not a problem. ⅔ say that it is a major pain point.
#5 Smaller and younger companies expect more from their developers
The smaller and the younger the company, the more likely they are to:
- expect every developer to be able to deploy and run their app on their own, while at the same time
- not providing a standardised and proven way to do that, leaving each developer to figure out local deployment on their own.
Scripts and configuration files are exchanged between developers, and there are common patterns of how developers run their applications locally, but there very often is no clear and standardised way of setting up a local deployment that works for a majority. Often, there are several very different ways of deploying an application locally that are used by different developers or different teams.
#6 Hybrid setups where developers run only one or two services locally and connect to others in a remote cluster are seen as a gold-star solution
But I’ve heard only one company describe this as the way they currently work. This was a startup and I talked to the CTO, who owned this topic, so I can’t be sure about the perspective of their individual developers, but from the CTO’s telling, this was working very well. However I’ve heard several other companies mention this as a target state, even describing it as “magic, fantastic” (literal quote here). I wrote a blog post series about this, if you’re interested to learn more.
#7 Infrastructure cost is a major worry
Enabling each developer to have their own development deployment of the application is often too expensive. This is again mostly caused by the large resource requirements of the application itself, which often makes it impossible to run locally, and expensive to run remotely. Partial or hybrid deployments where some services are shared between several developers are seen as a potential solution, but are often not possible because of statefulness of services or other architectural peculiarities.
#8 It’s a standard not to standardise dev tools
The majority of companies let developers choose their own tools, from the IDE to debuggers and even the tools they use for local deployment. Most provide tools that developers can use if they want to, e.g. licences for Jetbrains IDEs or other tools, but often do not require developers to use those if they don’t want to.
This refers only to tools used by individual developers within their own development environment (i.e. on their own laptops).
#9 VS Code and Jetbrains are by far the most widely used IDEs
No surprise there.
#10 (Backend) developers do not like browser-only IDEs
Most developers have their preferred IDE and want to stick with it. Most are open to trying others, but have to like it instantly or they won’t switch.
VS code in the browser is a bit of an exception since developers already on VS code mostly said they’d be fine using (or at least trying) the in-browser version.
#11 Releasing new features quickly is always prioritised over technical excellence
This is something mentioned by many developers, and often framed as a complaint.
#12 Technical debt is a fact of live
There is no codebase without technical debt. Many developers chafe at the fact that cleaning up technical debt just for the sake of cleaning up technical debt is never prioritised.
#13 Quality is a problem
Especially large engineering organisations, and in particular ones that have grown quickly, face severe quality issues in the code they produce. Theories about what causes this range from too many juniors, poor onboarding, poor or missing testing, high staff turnover, to simply too much complexity in the application itself. Only one person I talked to said that their company had identified poor developer experience as a root cause: “If testing takes too long, it won’t be done.”
#14 Production outages are common
Most people I talked to admit that production outages of their software happen. In most cases, the impact is negligible – customers don’t even notice or the impact is minimal. I haven’t heard any stories about major outages that led to penalties being paid or companies being sued in my interviews. Nevertheless, software breaking in production and all hands being called on deck to fix it is something that many developers have experienced.
#15 There is no standard deployment model
The large majority of companies that I’ve talked to have some elements of their deployment model that they themselves describe as “non standard”.
The most common non-standard is a high degree of variation in deployments, meaning that each production deployment of the software is different from each other production deployment.
Depending on the overall business model (e.g. predominantly SaaS with some custom deployments, or each customer has a dedicated deployment, or anything in between), this meant dozens or hundreds of different constellations / configurations of the software running in production.
Other non-standard elements have been described to me as “monolith leftovers”, “on-prem leftovers”, having hundreds of microservices, and customer specific components.
#16 Pure SaaS is rare
In the sample of people I’ve talked to, there was only a tiny minority that had a pure SaaS business model, with all their customers consuming their product as SaaS. The majority had either a mixed model, with some customers having dedicated instances and other customers using the SaaS offering. A good number had dedicated instances (often meaning dedicated Kubernetes clusters) for every single customer.
#17 Most companies don’t know how their developers spend their time
One of my standard questions is “How much time do you / the developers in your team spend on setting up, maintaining and troubleshooting their local development environment?”
Most thought about it before they gave an answer, showing that this is not something they had considered before. The answers were all estimates, based on gut feeling. Not a single person was able to give me an answer that was based on actual time tracked for these activities.
When I asked follow-up questions about time tracking, most told me that they either don’t track time at all, or track time only as total time spent working, not broken down on individual tasks. In those few companies where time was tracked on tickets, developers said only the sum is checked and not the time spent on individual tickets. Time spent on administrative tasks, in meetings or on tooling is not tracked separately but tracked on any development ticket the dev has in progress.
#18 Everybody wants great UX
If they don’t love a tool instantly, they won’t use it. “If the developer experience is seamless, it doesn’t matter what’s behind it.”
#19 Cloud development environments (CDEs) are an unknown product category
Most have heard about platform engineering. Few have heard of CDEs.
Conclusion
Taking a look at the picture this paints, some of the points I make are clearly related. Kubernetes is built precisely to enable the running of applications with many services and high overall resource requirements, which are exactly the kind of applications that are difficult to impossible to run on a laptop with limited RAM and CPU. So while K8s has solved the problem of running such applications in production, the problem of running such applications for development remains unsolved at the majority of companies (that I talked to). There are technologies out there that address this, but the open-source portion of those is still immature and not widely adopted, and the commercial portion is even more in its infancy. (See also my blog post about tools for developing K8s-based applications.)
Reading through the list now, I feel like many of the learnings I describe are neither new nor surprising. For me, however, it was important to hear from actual people working in the field about the things they struggle with. The impression I was left with is that software development is in a bit of a wild west state right now. I have heard people claim that the situation has been better, that developer’s lives were easier, in the 2000s and early 2010s when the standard application was a monolith and developers just wrote code, pushed to the repository, and from there, the operations team took over. I can’t confirm this from my own experience (I’m too young) and I’m not sure if it is just an example of humans often believing that things were better in the past. (Generally, they weren’t.) Regardless of how things were in the past, it is clear to me that developers right now face immense complexity in their jobs, and that many struggle with that.
As a company focused on providing automation and tools to make developer’s lives easier, the interviews confirmed that many of the challenges our tools address exist and are real problems for many developers. For our CDE product Cloudomation DevStack, we drew and implemented the following conclusions:
- DevStack must support development of software that runs in K8s in production
- DevStack must support a hybrid setup, where the CDE contains only part of the entire application and connects it to other services in a shared cluster
- DevStack must enable companies to create such a hybrid setup if it doesn’t already exist
- DevStack must enable companies to keep on top of infrastructure cost. We decided to support this by:
- Providing clear overview of current cost
- Simple automatic mechanisms to scale down / hibernate unused resources without interfering with developer’s work
- Provide templates and consulting to support customers in creating resource-optimised, cost-effective development cloud development environments (e.g. via a hybrid setup)
- DevStack must allow developers to continue using their favourite tools, including their favourite offline IDE. We do this by mirroring the source repositories onto the developer’s laptop, leaving the code and IDE local, while only outsourcing the complex and resource intensive development build and deployment of the application to the CDE. As a neat side effect, this also allows developers to work with spotty internet connection or while offline, since they always have access to the source code locally.
- DevStack must support “non-standard” components and deployment models
- DevStack must provide great UX to developers
If you’re interested in finding out more about how we implemented this, or are interested in a trial, get in touch.
