At West Monroe Partners, one of my responsibilities is helping clients and teams  understand how better DevOps can facilitate greater value and productivity. Recently, I was given the opportunity to assist one of our teams in transforming their DevOps process and correct certain inefficiencies that were holding the teams back in maximizing their efforts.

I joined the team with the aim of assisting its overall development as well as addressing these inefficiencies such as they were.

Phase 0: Gain an Understanding

In any engagement, knowledge is paramount. We must be able to speak with our clients on the level and work to gain their trust so as to better understand their problems. At its heart, DevOps refers more to culture than any specific toolset and within each organization will always be different. This being the case, we must respect these difference and work to create a process that emphasizes what the client needs rather than sticking to a formulaic approach.

In this case, I was joining the team 10 months into the project and thus would have a  limited understanding of the architecture and client relationship. It was paramount that I relax and listen to gain this knowledge and allow for the accurate identification of areas that I could impact to provide maximum value. By doing this, I was aiming to establish a trust with the client rather than becoming a “talking head”. Trust with DevOps is crucial since without it, you cannot expect the organization to take your ideas seriously and thus act on them.

Thus, I spent a large portion of my initial time talking with the team leaders and sitting in on meetings, offering only limited input. This allowed me to gain a better understanding of the overall situation and to better assess the problems areas. Ultimately, I came to four goals that I felt our teams had to address to be put them in a position to be successful:

• Reduce Dev build times from 30+ minutes to more manageable levels allowing developers to use Dev to validate their changes
• Create better consistency by promoting a simpler build and release model for developers to make building applications simpler
• Break apart the large mono repo whose size had become an hindrance on my development teams and clouded clients desired deployment startegy
• Emphasize Infrastructure as Code to enable to quick creation of environments including those that would be ephemeral in nature

The build times were the main issue here as the teams had made the decision to combine the Dev and QA builds into a single serial pipeline. While this did aid in promotion, it bottlenecked the process by holding the Dev artifact until the QA artifact completed, the later had a much longer process.

Phase 0a: Establish What I CAN do

With my goals established, the next step was to talk our client through the process and articulating the perceived value each change would bring. Based on these conversations, I ruled out the later two goals. The end proposal that was accepted basically flowed as such:

• Existing structure of individualized Dev/QA builds would be broken apart and only unit tests would be run to enable greater speed for developer validation. Client also agreed to drop signing, obfuscation, and whitesource scanning requirements from these builds
• Segregated builds would then be merged back into a parallelized build that would create a “package” that could be delivered to the Azure DevOps release pipeline. This would also include a refactoring effort to reduce duplication and promote reusability

With that agreed to, it was time to get started.

Phase 1: Divide the builds (1wk – against active pipeline)

The first phase was accomplished fairly quickly with the main challenge being marking our tests as “unit” to ensure we still could maintain quality without running every test. The results from this were very positive:

• Project 1
  • Average build time went from 35m for Dev to 17m
  • For the release build times remaining unchanged since speeding them up was not the objective
• Project 2
  • Average build time went from 45m for Dev to 20m
  • For the release build times remaining unchanged since speeding them up was not the objective

As you can see, the build times decreased heavily and become more manageable for both teams.

Note:

There is nothing inherently wrong with combining your Dev/QA builds since it does enable easier promotion. However, it is important to be mindful of this choice since there are usually more steps involved in creating the QA/Release build than the CI/Dev build. If you opt for this approach, your builds will take as long as the longest build.

These were solid results so overall people and I was pleased with the progress. But a drawback was the split doubled the number of builds and added extra pressure on the team when creating builds; not something that is ideal, particularly as Project #1 was already a tricky project to build correctly and this move seemed to move the goal posts. However, these problems were expected since the move was more designed as a preparatory move for future work.

With that out of the way, I began preparing to move to Phase 2. Before I could begin, however, I was told to hold off as the client was reconsidering their position on the mono Git repository.

Break apart the Mono Repo

I am not a huge fan of large mono repos as I believe they indirectly communicate a level of monolithic thinking that is progressively finding itself in the minority within our industry. That said, I will point out that there is nothing inherently wrong with using a single repository for multiple applications. In fact, large projects at Microsoft, Google, and other big shops are often housed in single repos and this is true for many legacy applications, particularly those built before the invention of Git.

Within our project, the large repo had consistently been a roadblock as it made it difficult to work due to the sheer size and number of files being tracked; this was pain the development teams on their side and our side both experienced first hand. I think it was this realization that led them to backtrack on the single repo design and explore a more segregated organization for the codebase.

In the end, the agreement was made to split the mono repo out into smaller repos to alleviate the performance problems. Our teams became responsible for setting up and configuring the following repos:

• Repo for Project #1 – containing code that was to be deployed on-prem to clients. It contained the most complex build requirements as it was comprised of 6 separate modules
• Repo for Project #2 – Containing code for cloud hosted applications which lived in Azure.
• Repo for Frontend UIs supporting all Projects – these were written using Angular and supported users interacting with each of the applications
• Repo for Automated UI-Testing for all Projects – these are tests written using Selenium that interacted with the UI of each Project to verify functionality

The most difficult element in this process was identified as the refactoring of code previously shared into a suitable form. It was decided that we would tackle this as it was encountered, opting for either duplication or Nuget packaging as the means to break it out.

I cannot understate how beneficial this break up was if for no other reason than showcasing that Git for Windows (or perhaps NTFS in general) is abysmal at handling large numbers of files. Our speed of interaction with Git went up by at least two orders of magnitude. It contributed immensely to our team reaching a record velocity. For myself, it underscored the importance of critically approaching project organization as a core part of the planning process.

Phase 2: Merge the builds (2wks)

With the code repos split and the teams enjoying better productivity than they had ever before over the course of the project I turned my attention to the final portion of this work: creation of a merged build to facilitate the efficient build of Project #1.

This effort was aimed at refactoring the existing segregated pipelines into a merged version designed to fulfill the following goals:

• Removed duplicative tasks such as restores and builds
• Remove “baggage tasks”, that is tasks that use “baggage” stored in source control and aim to use the standard tasks offered by Azure DevOps.
• Leverage parallelism to maximize the flow of builds and remove any bottlenecks

“baggage tasks” are tasks that are supported by programs or scripts stored in source control. I consider this an anti-pattern as it unnecessarily ties you to your source control and needlessly bloats the size of your code base. Most modern DevOps platforms have built in tasks that can handle these tasks for you while removing its maintenance from the realm of your responsibility. In addition, as you move to a multi repo approach, you end up duplicating these scripts adding another layer of maintenance to your responsibilities.

The goal here is to effectively utilize our build platform and its features to increase simplicity and maintainability while reducing ceremony and “inherent knowledge”. In doing so, teams find the pipelines can more easily be maintained by anyone and problems area easier to discover because everyone understands the process.

Completing the Merged Build: The results

The results were, in a way, phenomenal. What once took 15m to create now took 5m (CI Build). And what once took 35m now took 15m (Release build). All of this resulting from a build script that was cleaner and more efficient than anything that had been there previously.

Further, since everything was under a single build we went from 12 separate build pipelines to 2, further enhancing developer productivity. This was the culmination of the goal we had set out to do.

The lone area I was not able to address was the notion of “true promotion”. That is, developer still had to kick off release builds by specifying the “commit hash”, though it only had to happen once now, instead of 6 times – this let us achieve our goal of a consistent build. This is a shortcoming in the way Azure DevOps approaches the concept of a build pipeline.

 Closing Thoughts

Overall, I am very pleased with how the new process is working and how it enables our teams to perform better. Perhaps more important though is the lessons it has taught us and our leadership just how important is to be clear and committed to DevOps from Day 0 and ensure the patterns and practices for the project are designated up front and are implemented from the get-go.