As your company grows, you will need a dedicated team of experts in the field of developer productivity engineering. This discipline has the sole focus of optimizing the effectiveness of the developer toolchain to achieve a high degree of automation, fast feedback cycles and correct, reliable feedback.

The job of Developer Productivity Engineering is to improve productivity through the entire lifecycle from the IDE to production. Too often CI/CD teams are not responsible for developer productivity and make decisions that optimize for other metrics, e.g. auditability without taking adverse effects on developer productivity into account. The priorities and success criteria for this team are primarily based on data that comes from a fully instrumented toolchain. For such a team to be successful it needs a culture where the whole organization commits to improving developer productivity.

There is a significant gap between the actual productivity of software development teams and their full potential. The gap is growing over the lifetime of a project for most organizations but it can be significantly reduced with the practice of developer productivity engineering.

The happiness of your developers depend on their productivity. Most developers want to work in an environment that enables them to work at their full potential. Organizations that can not provide such an environment are seeing a drain of talent leading to even lower productivity.

The economic benefits of not applying this practice are dramatic. Development productivity improvements provide significant leverage for every dollar invested in software development. The amount of features you can ship is heavily affected by it. The productivity gap that comes from not applying the practice of developer productivity engineering is a significant competitive disadvantage. Innovation is at the heart of any business to survive. Most innovations nowadays, for any enterprise, must be funneled through software. That means the productivity of your software development teams is crucial for your business to survive.

Developer Productivity Engineering applies to a wide part of the software development process landscape. It begins with the coding process and applies through to CI/CD and the point where DevOps takes built applications for distribution and scaling.

It is important to say that developer productivity engineering is not about treating developers as factory workers but instead of creative craftspeople. It is about the developer experience and unlocking creative flow and restoring the joy of development as well as the collaboration with the business.

I started the Gradle project out of deep frustration in working in low productivity environments. The amount of time it took to make progress and get feedback on changes in the enterprise environment was incompatible with how I work and was taking the passion and satisfaction away from the craft of software development. Later I was joined by our Gradle co-founder, Adam Murdoch, and many other talented developers. Over the last decade we have worked with hundreds of the top development teams in the world from every industry.

This book represents not just our learnings from that exciting journey but also what we see emerging on the horizon from working with the most respected software development teams in the world, such as LinkedIn and Tableau.

These leading-edge teams make developer productivity engineering an executive-level initiative, realizing that it is critical to lead their respective industry and their developer’s ability to innovate at their full potential. They prioritize developer productivity engineering with a team of experts whose sole purpose is to improve the toolchain, unblock feedback cycles, and fix the developer experience.

In this book we share techniques and stories in how to:

In many industries, there are engineering disciplines dedicated to the practice of making production efficient and effective, such as chemical engineering and industrial or process engineering. They have their own degrees and are much respected within their broader field.

Developer Productivity Engineering is of similar importance when it comes to the production of software. The software industry is at the beginning of this realization. It is a very exciting time to work in this area!

Earlier we talked about how the creative flow depends on the speed of the feedback cycle. Consider a customer example comparing feedback cycles from two teams.

The developers on Team 2 get feedback twice as often as the developers on Team 1. The most likely reason is that their build and tests are faster. Over time, the developers on Team 2 will be much more productive because there are fewer interruptions and they spend less time waiting for each individual build to run.

We see this correlation between build time and the number of builds consistently across organizations, even when the build time is less than a few minutes. The most likely reason for that is that their build and tests are faster. This is obvious when the build times have become painfully long and people explicitly say that they avoid as much as they can to run build and tests. What is interesting though, is that even in environments where building and testing are not perceived or described as slow, the data shows that the number of builds is inversely-proportional to the duration.

First of all, our brains work much better when the disruption to the flow is minimized. Endorphins help us enjoy our work during flow states, motivating us, but in modern offices, we have numerous distractions from meetings and loud coworkers to urgent email messages taking away our focus. With fewer interruptions from builds, we can use our remaining time more effectively.

There is another obvious reason why developers on Team 2 are more productive. A lot of the time waiting for feedback is actually just that, waiting time. When builds and tests run in less than 10 minutes, context switching is often not worth the effort, as a lot of the build and test time is downtime.

The aggregated cost of waiting time is surprisingly high even for very fast builds and even moderate improvements are often worthwhile. If you look at the table below, even though the team with a 1-minute build is doing quite well, reducing the build time by 40% to .6 minutes, saves 44 developer days per year.

On a larger team, this scales exponentially as we see the team of 100 devs nearly halves their build for a savings of 5200 developer days saved. Assuming 220 working days a year per developers, 5200 developer days is roughly 25% of all engineering time. Significant performance improvements like this is a game-changer for productivity.

As build time increases, more and more people switch to do different tasks while the build is running. If the build is the final build to get a feature merged and the build is successful, this is not much of a problem (although longer builds are more likely to lead to merge conflicts). But if the build fails or was necessary to provide intermediate feedback you have to switch back to the previous task and pay the cost of context switching. This often costs 10-20 minutes per switch and is the reason why for builds that are faster than 10 minutes people would rather wait. The cost has to be paid twice, when going back to the previous task and then again when continuing with the new one.

The longer the build takes, the bigger the changesets per build will become. In the case your changes pass the build, the overhead of building and testing is roughly like a flat tax. Whatever the amount of changes is, the tax is the same. So if the tax is high and has to be paid per build run, you are minimizing this tax by not running the build as often. So the amount of coding you do in between the build runs will increase.

Unfortunately, the tax is not flat if the build fails. The more changes you have, the longer on average the triaging will take. This increased debugging cost caused by larger individual changesets has to be paid for failing local builds, failing pull request builds and failing master builds.

Depending on how your CI process is organized, for the master CI builds you might have an additional effect. The fewer master CI builds per day you can run because of their duration, the less merge points you have. Thus the average number of changes included in that CI build is higher. This may significantly increase debugging time for any failed CI build as we are talking possibly about a lot of unrelated changes.

The most extreme cases we have seen are builds of large repositories that take a whole day. If this builds fails possibly 100’s of commits might be responsible for that failure and the triaging is often extremely complicated and time-consuming. But even with much shorter build times, you might have that effect. In general, there is a non-linear relationship between the numbers of changes and debugging time.

Bigger changesets and more pull requests per master CI build both increase the surface area of the changes and thus the likelihood of merge conflicts. There is another effect at play here. Bigger changesets also increase the debugging time and thus the average time of a successful CI master build. This again reduces the frequency of merges and for the reasons discussed this will increase the likelihood of merge conflicts. This is a vicious circle. Merge conflicts are one of the most frustrating and time-consuming things developers have to deal with.

On average 20% of all build and test runs fail. This is a healthy number as it is the job of the build to detect problems with the code. But this number significantly affects the average time until a build is successful and the context switching frequency.

average time for a successful build = Average build time + (failure frequency * average build time) + average debugging time

context switching frequency = failure frequency * number of builds * 2

An unreliable toolchain, for example, one with a lot of flakey tests, can thus both drastically increase the average waiting time as well as the context switching frequency.

Smaller projects will benefit from faster builds because of less waiting time and improved creative flow. They will also be affected by larger change sets as their build time increases. Smaller projects usually don’t suffer from overlapping merges as much as larger projects will suffer from the same issues, as those are less likely to occur with a lower number of developers.

Many larger teams break up their codebase into many different source repositories for that reason. But this introduces other challenges.

The producer of a library, for example, is no longer aware of whether they have broken a consumer after running their build and tests. The consumer will be affected and has to deal with the problem when this happens. The triaging of that is often difficult and very time-consuming.

Fixing problems later take disproportionately more time than fixing them earlier.

There are multiple reasons for this. We already mentioned the non-linear relationship between the size of the changeset and the time required to fix the problem. Context switching also becomes more expensive.

You have a similar effect in a multi-repository environment where the consumer is responsible for figuring out that the consumer contract has been broken and organizing a fix. In this case, the more dependencies that have changed during a build run make it exponentially harder to figure out which dependency and team are responsible.

As build times grow there is increasing friction between getting feedback early and reducing the waiting time before you can push your changes.

This is a terrible situation and there is no good choice. Many organizations go down the route of shifting the feedback later in the development cycle, or ‘to the right’. For example, they weaken the local quality gate by no longer running any tests, and the pull request build will be the first build to run the unit tests.

This is the opposite of a healthy CI/CD pipeline where you want to get feedback as early and as conveniently as possible. All the exponential effects we discussed above will hit you even stronger.

The problems described here will grow as does your number of developers, code bases and repositories.

Even worse is the problem of unhappy developers. Most developers want to be productive and work at their full potential. If they don’t see an effort to turn an unproductive environment into a more productive one, the best ones look for other opportunities.

Let’s say you have built the project and after that you are changing a line of code in the export-api module from the example above. You add an argument to a public method and we assume no other module has a dependency on export-api. With such a change and the build-cache in place you need to run only 16% of the goals or tasks.

How does this work. All of those 25 goals have inputs and outputs. For example the compile goals have the sources and the compile classpath as an input as well as the compiler version. The output is a directory with compiled .class files. The test goal has the test sources and the test runtime classpath as an input, possible also other files. The outputs are the test-results.xml files.

What a build cache does is to hash all those inputs for a particular goal and then calculate a key that uniquely represents those inputs. It then looks in the cache if there is an entry for this key. If so, the entry is copied into the Maven build output directory of this module and the goal is not executed. The state of the Maven output directory will be exactly the same as if the goal had been executed. Copying the output from the cache though is usually much faster than actually executing the goal or task. If an entry for a key is not found in the cache, the goal will be executed and its output will be copied into the cache, associated with the key. In our example, four goals belonging to the export-api module had a new key and needed to be executed. The compile goal of the production code, because one of its inputs, the source directory, has changed. The same is true for checkstyle. The compile goal for the tests has a new key because its compile classpath changed. The compile classpath changed because the production code of export-api was changed and is part of the compile classpath for the tests. The test goal has a new key because its runtime classpath has changed for the same reasons.

Let’s look at another example. Let’s say you have built the project and are then adding a method parameter to a public method of the security module. Webapp has a dependency on security.

Now not just the goals for the module that has changed needs to be re-executed, but also for the module that depends on this module. This can be detected via the inputs of the goals. Because of the change in the code of the security module, the classpath of the webapp compile and test compile goal has changed as well as the runtime classpath for its test goal and thus all those goals need to be executed. Compared to rebuilding everything, you are still only executing 28% of the goals.

Let’s use the same example as before but instead of adding an argument to a public method in the security module, we are changing just something in the method body. A smart build cache can now do further optimizations:

The cache key calculated out of the compile classpath only takes the public API of the classpath items into account. An implementation change will thus not affect that key, reflecting the fact that any such change has no relevance for the Java compiler. Because of that, the execution of three compile goals can be avoided. For the runtime classpath of the test goals, implementation changes in your dependencies are obviously relevant and lead to a new key resulting in executing the test goals for security and webapp. With this optimization only 16% of the goals need to be executed.

Let’s look at another example. Let’s say every other module depends on the core module and you change an implementation of a method in core:

This is a pretty invasive change but even here only 28% of the goals need to be executed. Granted, executing the test goals will probably consume more time than the other goals, it is still a big saving in time and compute resources.

The worst-case change would be adding an argument to a public method in the core module:

Even here to get relevant savings from the cache as only 64% of all the goals need to be executed.

There are two types of build caches. One is a local cache. It is just a directory on the machine that is running a build. Only builds that run on that machine add entries to that cache. A local cache is great for accelerating the local development flow. It makes switching between branches faster and in general, accelerates the builds before you commit and push your code. Many organizations weaken the local quality gate because of the long build times. A local cache is key to strengthen the local quality gate and get more feedback in less time before you push your code.

The other type is a remote cache. A remote cache node shares build output across all builds in the organization.

Local builds write into the local cache. Usually, only CI builds are writing to the remote cache while both, local as well as CI builds are reading from the remote cache. It dramatically speeds up the average CI build time. You often have a pipeline of multiple CI jobs that run different builds against the same set of changes. For example a quick check job and a performance test job. Often those jobs run some of the same goals and a build cache makes this much more efficient.

A build cache does not just improve build and test execution times, it also drastically reduces the amount of compute resources needed for CI. This can be a very significant economic saving. But it also helps with CI build times. Most of the organizations we talk with have a problem with CI queues, meaning new changes are piling up and there are no agents available to start the build. A build cache can significantly improve agent availability. The below diagram is from one of our customers as they started to introduce the cache and optimized its effectiveness.

As already mentioned, if all your code and tests live in a single module you will not get many benefits from a cache. You will get some though. Especially on CI where you often have multiple CI jobs running against the same set of changes. But the majority of the benefits come once you have multiple modules.

Even with a few modules, the benefits will be significant. With many modules, things can become very efficient. For larger multi-module builds often 50% of all the modules are leaf modules, meaning no other module depends on them. If n is the number of modules, rebuilding such a leaf module will only take roughly 1/n of the time compared to building the whole project. A build cache allows you to work much more effectively with larger source repositories if you have some modularization. Investing in further improving the modularization will not reduce the technical debt when it comes to evolving your codebase but will give you immediate benefits when it comes to build performance.

A build cache needs to be continuously monitored when it comes to its effectiveness. One part of this is the caching infrastructure, including things like download speed from the remote cache nodes as well as enough storage space to not lose cache entries that are still in use.

Another important aspect is the reproducibility of your build. Volatile inputs are the enemy of caching. Let’s say you have a goal or task that always add a build.properties file to your jar that contains a timestamp. In such a case the input key for the test action will never get a cache hit as the runtime classpath always changes. You have to be able to effectively detect such issues as they can drastically reduce your cache effectiveness.

Nothing is faster than avoiding unnecessary work. For our own projects, we can see a spectacular on average 90% time saving because of reusing previous build output via caching. We see savings at least over 50% all across the industry once organizations start adopting the build cache. It is must-have technology to address the problems outlined in Why are long feedback cycle times poisonous?

What exactly do we mean with reliability? A toolchain is unreliable whenever:

Performance and reliability are closely related. As discussed in Every second counts, we have the following relationship:

average time for a successful build = Average build time + (failure frequency * average build time) + average debugging time

context switching frequency = failure frequency * number of builds * 2

Obviously non-verification failures add to the failure frequency and thus waiting time and context switching frequency. Additionally, non-verification failures have a much higher average debugging time than verification failures and thus substantially affect the average time until you have a successful build.

The software toolchain is complex machinery that includes the IDE, builds, tests, and CI and many other components. It has complex inputs that are always changing. Every other industry with such complex production environments has invested in instrumenting and actively managing this environment based on the data they are collecting. The data serves two purposes. One is to see quickly trends and pathologies. The second is to effectively figure out the root cause. Let’s take a chemical plant that produces some liquid as an example. One important trend that will be observed is how much of that liquid is streaming out of the vessel in which the liquid is produced. If they see a drop in that number they react by looking at the parameters of the vessel environment, for example, the temperature, pressure, and concentration of certain chemicals. That will usually allow them to figure out or significantly narrow down the root cause for the drop in production. We need similar instruments for optimizing the software production environment.

Toolchain performance and reliability is sensitive to its inputs. That starts with infrastructure like the hard disks in the example above or networking. This also includes the build cache nodes, the CI agents and binary repository manager. For example, dependency download time is often a bottleneck for CI builds. Yet hardly any organization has an idea of how much time they spend on average on downloading dependencies and what the network speed is when this is happening.

The code itself is also a complex input for the toolchain. Developers, for example, add annotation processors, new languages, and new source code generators that might have a significant impact on the toolchain performance (and reliability). And yet I haven’t met a single company who would be able to tell me how much of the compile-time is spent on a particular annotation processor or how much faster on average a compiler for one language is compared to the compiler for another language for their particular codebase. Just changing the version of an already in use compiler or annotation processor might introduce significant regressions.

Other factors are memory settings for the build and test runtimes, code refactorings that move a lot of code between modules, and new office locations. The list goes on and on.

In most organizations, most of the performance regressions go unnoticed. But even if they are noticed they often go unreported. Why? Because reporting them in a way that they become actionable requires a lot of work on the side of the reporter. They need to manually collect a lot of data to let the support engineers reason about the problem. If the issue is flaky they need to try to reproduce it to collect more data. Finally, often reported performance issues do not get addressed which lowers the motivation to report them. Why do they not get addressed? Often the data that is provided as part of the report is not comprehensive enough to detect the root cause, particularly with flaky issues. Furthermore, the overall impact of this problem on the whole organization is hard to determine without comprehensive data. That makes it hard to prioritize any fix.

When a performance regression is really bad, it will eventually get escalated. That usually happens after the regression has already created a lot of downtime and frustration. Because there is no good data available it will also usually take longer than necessary to fix the regression.

All in all, the average build and test time is much higher than necessary and continuously increasing because the toolchain is not instrumented and is thus impossible to be kept in an efficient state.

For reliability issues, the story is a bit different. Failures are always noticed by at least one individual, the developer who is running the build and tests locally or the developers whose changes are causing the CI job to fail. The very first analysis they do is usually to decide whether this is a verification or non-verification failure. This is often not easy. We talk about this in detail in Improving Developer Experience with Proactive Investment in Reliable Builds. Once the developer thinks it is a non-verification failure, they should report it to the people responsible for maintaining the toolchain. But reporting such issues and asking for help is painful for the reporter as well as for the helper for the same reasons it is painful to report and help with performance regressions as described above. But as they often block people they can not be simply ignored. So to avoid reporting them, developers run the build and test again and hope that the issue is flaky. If it still fails they might run it again with all the caches deleted. This is extremely time consuming and often very frustrating. Only when they can not get to a successful build that way, they ask for help informally or by filing an issue. When you talk with the engineers who support developers with those issues as the build and CI experts, they most of the time have no idea about the root cause either. So guess what they often do as the first thing? Run the whole thing multiple times to see if the issues go away.

While you will always have issues that require direct support, one of the most important and game-changing practices all this data will allow you to do, is to actively observe and manage the toolchain to reduce the number of non-verification failures and continuously and pro-actively optimize the performance. Once you have comprehensive data of each and every build and test execution from IDE to local builds to CI, you can see trends and can compare and establish a baseline. Trends and comprehensive data together will enable you to:

We will talk in detail in Improving Developer Experience with Proactive Investment in Reliable Builds how this can be done for non-verification failures.

When I talk with teams that are responsible for the toolchain, I’m often surprised by their approach to performance. They are completely incident driven. If someone complaints they try to improve the problem that was reported. If no one complains, they consider everything to be fine and nothing needs to be done. But they are the experts. They should not wait until people are so miserable that they are complaining about a situation. They should turn into developer productivity engineers who understand how important toolchain performance is for the productivity of the developers. Their focus should be to get the performance to levels beyond what most developers would expect is possible.

Imagine you are in charge of an online shop. Would you wait to improve the page load time until some of your visitors start complaining about it? And after you have improved it, would you do nothing until they complain again? Probably not, as you would lose tons of business along the way. Less will be bought and customers will move to different online-shops. This often happens silently. Similarly, developers will produce fewer features and will look for different job opportunities if you keep them working from their true potential because of a slow toolchain.

The fastest code is the code that doesn’t execute at all. The most efficient, painless support session is the one that doesn’t need to take place at all because proactive investment has made the build highly reliable. To achieve this you need to collect data from all build and test run in the organization, both local and CI, and analyze the data to determine which issues have the greatest impact on developer productivity. To facilitate this type of analysis, we created the Failures Dashboard in Gradle Enterprise, first introduced in version 2019.3.

On one developer productivity team we were very successful using a process like this one to stay on top of reliability in a rapidly-changing build environment:

Ultimately the transparency, predictability, and effectiveness of this process built a great deal of trust between our developer productivity team and our peers working on the product. This credibility manifested as developers coming to us for advice on infrastructure and toolchain proposals, which allowed us to steer them away from anti-patterns and architectural mistakes. Being involved early on in these discussions let us avert many issues before they could ever be committed in the first place, producing a virtuous cycle.

How much time do developers in your organization spend having their thumbs smashed by cursed hammers? How long does it take after an issue affecting developer productivity is detected before it’s fixed? Ultimately only the continuous application of the principles and practices of developer productivity engineering will attain & maintain a great developer experience at scale. If you’re not measuring it, don’t act too surprised when some of the best leave for greener, better-tooled pastures.

Of course, build reliability may is likely not the only pain point in your development toolchain. Flaky tests are another common source of intense developer pain which we will discuss in The Black Pit of Infinite Sorrow: Flaky Tests.

How much pain test flakiness causes an organization is exponential. A little flakiness is barely noticeable. A lot of flakiness and your testing is rendered completely worthless, any actionable signal drowned out in a blood-red sea of noise.

So it follows that the ROI on fixing flakiness is logarithmic: Fixing that most-flaky test reduces pain substantially, but once flakiness is down to acceptable levels it becomes difficult to justify the opportunity cost of investing in flakiness fixing instead of new features. Therefore an ongoing process is required to measure the severity of the problem in a way amenable to correctly prioritizing investments of time & resources.

So how much flakiness can you live with? The amount of flakiness you can tolerate is inversely proportional to the number of tests you have, the rate at which they flake, the number of builds you run, and how many times you retry on failure.

Flaky Failures/day = (Builds/day)(Count of Flaky Tests)(Chance a test flakes)^(1+retries)

Let’s say your organization has 200 flaky tests that run in 500 builds per day and you want no more than 1% of your builds, 5 in this example, to be disrupted by a flaky test failure. With no retries, you need to keep the average rate at which they flake under 0.005%. With a single retry, you can maintain a 1% flaky failure rate with an average test flakiness rate of under 0.7%. This means your tolerance for flakiness goes down as the number of flaky tests you have, or the number of builds you run, increase.

A test can be flaky for an unbounded number of reasons. Here are several that occur across many different kinds of software projects.

Since flaky tests pose a procedural challenge as well as a technical one, procedural adaptations must be part of the solution. Here’s an example process your organization could use you to keep the pain of flakiness tolerable and avoid having to write off whole test suites as a loss:

If performed completely manually, this can be a lot of work! Fortunately, many of these steps are highly susceptible to automation. After the CI jobs are in place, identifying flaky tests and filing bugs accordingly can be automated. Automatically disabling tests is harder, but still achievable with a script that knows how to perform & commit an abstract syntax tree transformation.

Our example team consists of 200 engineers with the following parameters:

The example numbers above and later in the article reflect what we see typically in the wild. But the numbers out there also vary a lot. For some, we have better averages than for others. The example numbers are helpful to get a feeling for the potential magnitude of the hidden costs that come with builds. Your numbers might be similar or very different. You need your own data to get a good understanding of what your situation is and what your priorities should be. The primary purpose of this document is to provide a model with which you can quantify costs based on your own numbers.

The number of builds depends a lot on how your code is structured. If your code is distributed over many source repositories you have more build executions compared to the code being in a single repository which then results in longer build times. But as a rule of thumb, one can say that successful software teams have many builds per day. It is a number that within your organization you want to see go up. As we evolve our model, we want to switch in the future to a lines of code build per day metric.

When developers execute local builds, waiting for the build to finish is pretty much idle time. Especially everything shorter than 10 minutes does not allow for meaningful context switching. That is why we assume WLas 80%. It could even be more than 100%. Let’s say people check and engage on Twitter while the local build is running. That distraction might take longer than the actual build to finish.

Here is the cost for our example team of unproductive waiting time for each minute the local build takes:

BYL * WL * CM = 500000 * 0.8 * $1 = $400,000 per year

Vice versa, every saved minute is worth $400,000. Every saved second $6667. A 17 seconds faster local build gives you additional R&D resources worth one engineer. If you make the build 5 minutes faster, which is often possible for teams of that size, that gives you the resources back of 18 engineers which are 9% of your engineering team or $2,000,000 worth of R&D costs!

The reality for most organizations is that the builds are taking longer and longer as the codebases are growing, builds are not well maintained, outdated build systems are used, and the technology stack becomes more complex. If the local build time grows by a minute per year, our example team needs an additional 4.5 engineers just to maintain your output. Furthermore, talent is hard to come by and anything you can do to make your existing talent more productive is worth gold. If local builds are so expensive, why do them at all ;). Actually, some organizations have come to that conclusion. But without the quality gate of a local build (including pull request builds), the quality of the merged commits drastically deteriorates leading to a debugging and stability nightmare on CI and many other problems for teams that consume output from upstream teams. So you would be kicking the can down the road with even significantly higher costs.

It is our experience that the most successful developer teams build very often, both locally and on CI. The faster your builds are, the more often you can build and the faster you are getting feedback, thus the more often you can release. So you want your developers to build often, and you want to make your build as fast as possible.

The correlation between CI builds and waiting time is more complicated. Depending on how you model your CI process and what type of CI build is running, sometimes you are waiting and sometimes not. We don’t have good data for what the typical numbers are in the wild. But it is usually a significant aspect of the build cost, so it needs to be in the model. For this example we assume W_CI is 20%. The cost of waiting time for developers for 1 minute of CI build time is then:

BYCI * WCI * CM = 500000 * 0.2 * $1 = $100,000 per year

Long CI feedback is very costly beyond the waiting cost:

We are working on quantifying the costs associated with those activities and they will be part of a future version of our cost model. The CI build time is a very important metric to measure and minimize.

One of the biggest time sinks for developers is to figure out why a build is broken (see the challenge of the build engineer for more detail). When we say the build failed, it can mean two things. Something might be wrong with the build itself, e.g., an out of memory exception when running the build. We will talk about those kinds of failures in the next section. In this section, we talk about build failures caused by the build detecting a problem with the code (e.g., a compile, test or code quality failure). We’ve seen roughly these statistics for a team of that size:

Such failure rates for F_L and F_CI come with the territory of changing the codebase and creating new features. If the failure rate is much lower, I would be concerned about low test coverage or low development activity.

For many failed builds the root cause is obvious and does not require any investigation, but there are enough where you need to investigate which is expressed by I_L and I_CI. CI builds usually include changes from multiple sources. They are harder to debug, and multiple people might need to be involved. That is why T_CI is larger than T_L.

If the build itself is faulty, those failures are particularly toxic. Those problems are often very hard to explore and often look to the developer like a problem with the code.

F_L is usually larger than F_CI as the local environment is less controlled and more optimizations are used to reduce the build time like incremental builds. If not properly managed they often introduce some instability. Such problems usually require an investigation which is why the investigation rate is 100%. Such problems are hard to debug, for local builds even more so as most organizations don’t have any durable records for local build execution. So a build engineer needs to work together with a developer trying to reproduce and debug the issue in her environment. For CI builds there is at least some primitive form of durable record that might give you an idea of what happened, like the console output. We have seen organizations which much higher rates for FL and FCI than 0.2% and 0.1%. But as this is currently very hard to measure we don’t have good averages and therefore are conservative with the numbers we assume for the example team.

Often half of the operational DevOps costs are spent on R&D. The CI hardware is a big part of that. For our example team, a typical number would be $200K.

We assume the following average build times for our example team:

This results in the following overall cost:

While this cost will never be zero, for almost every organization it can be significantly improved.

Cutting it in half would give you the R&D worth of 15 engineers for our example team of 200. And keep in mind that if you don’t do anything about it, it will increase year by year as your codebases and the complexity of your software stacks are growing.

There are a lot of other costs that are not quantified in the scenarios above. For example, the frequency of production failures due to ineffective build quality gates or very expensive manual testing for similar reasons. They add to the costs and the potential savings.

I frequently encounter two primary obstacles that prevent organizations from realizing the benefits of investing in this area.

We haven’t yet seen a company where investing in more efficient builds was not leading to a significant return on investment. The most successful software teams on the planet are the ones with an efficient build infrastructure.

The calculator estimates potential savings both in R&D dollars recaptured and developer hours saved at both a team and ‘per developer’ level. There are three sections which correspond to the savings drivers discussed in the previous chapter, including:

The savings figures for each of these drivers total into a summary depicted in the top banner. You set input parameters in each of the sections below to create this overall estimate or calculation for your team.