Back in 2017, I was asked to speak at a conference in the Salt Lake City, Utah area on the topic of Technical Debt. This is an update to the original presentation that was given.
Phase 1 — Initial Feature Development
Let us start with an example of a software development project that will demonstrate how technical debt is created and evolved.
Super-Duper Upper Juice Company — Sales Commissions
Super-Duper Upper Juice Company has developed a new product, and they have made the decision to pay out commissions to all their new salespeople based on some sort of rules.
They have an existing point of sales system in place and it does not currently support custom commissions.
The decision is made to build an extension that allows the sales team to view their commissions inside the existing system.
Sally, the new Software Engineer on the team is assigned to build out this feature. Being new, she is given three months to learn about the system and implement the new feature.
Three months later, Sally is victorious, and on the Salesperson’s Account Page, they can now see their current commission based on their sales. Celebrations ensue, and she has secured herself a spot on the team!
What is Technical Debt?
Technical debt (also known as design debt or code debt) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy solution now instead of using a better approach that would take longer.
Technical debt can be compared to monetary debt. If technical debt is not repaid, it can accumulate ‘interest’, making it harder to implement changes later on. Unaddressed technical debt increases software entropy.
Technical debt is not necessarily a bad thing, and is sometimes required to move projects forward (e.g., as a proof-of-concept). It really depends on the amount and kinds of change the business is dealing with at the time the software is being worked on.
Essentially; Technical Debt = Knowledge Debt
The people, conversations, and decisions that were made along the way are all part of the Technical Debt equation.
Who creates Technical Debt?
Business Stakeholders (Upstream)
Technology Stakeholders (Downstream)
Front End Engineers
Back End Engineers
Business Intelligence Engineers
Pretty much everybody involved in the development process.
Phase 2 — Product Iterations
Okay, some time has passed, and let’s see what is happening in the world of the Super-Duper Upper Juice Company.
Super-Duper Upper Juice Company is so popular now that the demand has become worldwide, and they now need to support international payment processing and sales in their existing system.
Given the complexity of this new set of features, Sally has now been assigned to help build out the international payment system.
Six months later, Sally is victorious again, and everybody is happy. Bonuses, promotions, celebrations ensue again. All is good.
How does Technical Debt get created?
Technical debt is a byproduct of making decisions. Teams have meetings, conversations happen, and decisions are made. The goal is to capture as much of that information as possible, but in reality, it is very difficult. This knowledge is the root of all Technical Debt.
Some situations where Technical Debt is created:
When estimations do not meet reality, decisions must be made to maintain alignment with business needs. If teams are not delivering on schedule, and the business has made commitments, priority changes will happen.
As systems mature, the existing design may not meet the current demands and requires refactoring. Maybe the User traffic has grown 100x and the system is starting to fail regularly, it will need refactoring.
Team resources may need to be reallocated. Perhaps an emergency outage requires key staff members, and other members are moved to fill the gap.
When current needs are not going to fit future needs.
When team members change frequently.
When development teams depend on other teams, and development stalls out. This could be a sign of technical debt that needs to be addressed to allow for fewer inter-team dependencies.
When new tools are introduced into the technology ecosystem. This is a major source of pain as the availability of low-quality tools grows daily.
How to prioritize Technical Debt
Technical debt should be reviewed periodically to ensure it is not getting out of control.
- At the minimum, it should be reviewed Quarterly by all stakeholders, not only engineering
- When system failure is in the near future, not far future. For example, the current system design is near capacity and is preventing system growth.
- When product cycles allow for more open development cycles. Maybe your company has a code freeze during major sales times, take advantage of the spare development cycles.
- When new members join a team, assign them debt problems with senior staff members. This is a great way to solve the technical debt, as you get fresh perspectives combined with a seasoned staff member.
Phase 3 — Feature Enhancement
Back to Super-Duper Upper Juice Company.
Business is still strong, and there is money in the budget for more engineers so Patrick is hired.
The Sales team is happy with Sally’s work but has requested many times that they be able to see their commission on the product Homepage instead of on their Account page.
The decision is made to give this work to Patrick. The feature currently works and he is given three months to learn the system and implement this new feature.
Three months later, Patrick is excited to deploy the new feature to the product homepage. He learned about the system, made a few changes to Sally’s original code and moved it to the Homepage. Sales approved the change and everybody is ready to deploy.
How does Technical Debt fit into the Development Cycle
One thing for certain is that if Technical Debt is left unchecked, the entire business model is at risk of failure. It is important to find a way that will help the teams stay on top of Technical Debt.
In my experience, I have seen these models work for development teams:
- Every 3 sprints, take a technology sprint to catch
- 80% of tasks go to business, 20% go to technical debt
- Create a team that deals with nothing but technical debt
Whether following traditional Project Management methods or Agile methods, there is always some form of a review meeting. In Agile, it is the retrospective.
- Catalog technical debt that was generated during the sprint
- Elevate risk to future development: Low, Medium, High
- Elevate whether to have Technical Debt prioritized in upcoming sprints.
Sources of Technical Debt
There are many sources of Technical Debt, here are some of the more common:
- New tools that go from prototype to production
- Software code that lacks any design patterns
- Software without source control (very painful)
- Manually configured machines — best to automate configuration (DevOps kind of work)
- Stale codebase — longer untouched, the worse it gets
- Team member leaves the team
- New team members
- Software that lacks any visible elements (ex — dashboard, unit tests, documentation, backend services, data stores)
- The Software Industry itself — New languages and frameworks with short shelf lives
Phase 4 — Technical Debt
Let’s wrap up with Super-Duper Upper Juice Company.
A new version of the product is deployed, everybody is so excited to see their commissions amount on the Homepage of the product. The Sales team claims they don’t know how they ever survived without this feature in the past, they are hard sold. Celebrations!!!
Within a week of the new version, there are a lot of small complaints about the performance of the product. People are saying it runs slow, and it can be frustrating to make Sales when the product is running slowly. Not a dealbreaker, but frustration mounts as the team continues to use the product.
What happened? Everybody loved Sally’s work. The feature already worked in the Product (on the Account page) and nothing was changed to the Business Logic.
Now Sally and Patrick are in the hot seat to explain what is happening with the product. They get in and figure out what happened.
- By moving the commissions' calculation to the Homepage, traffic on that call went from 10x/second to 1000x/second.
- The original design required the algorithm that builds the commission amount must run a very complicated query against the Sales database. The queries are proven to be slow and taxing on the data stores. Given the extra traffic, query performance is even worse.
Sally and Patrick present their results to the business. They determine that in order to fix this issue, they will need to refactor the feature to put in a caching component to prevent the commission calculation from happening so frequently.
The business now has to make the decision on whether to give Sally and Patrick the time and resources to implement the system change or live with the performance of the system as is.
Super-Duper Upper Juice Company has so much business going on, they choose to live with the performance of the new feature and move forward with other development features.
Technical Debt ALERT!!
And this is one way it happens.
The Impact of Technical Debt
Some side effects of unmanaged technical debt:
- Systemic Death by a thousand slices
- Unmanagable Knowledge Debt
- Unmodifiable Source Code
- Assets turn to Liabilities
- Team Morale drops
- Velocity slows down
- Business relationships suffer
- Total Cost of Ownership goes up
Technical Debt is a byproduct of change over time. It is inevitable in the development of products and systems. Left unattended, it can have expensive ramifications.
It is my hope that you learned a thing or two about Technical Debt in this article, and helps you make more informed decisions down the road. If nothing else, simply talk about Technical Debt with your team.
Thank you and best to you in your development efforts.