Software development metrics: how to measure and improve the effectiveness of the engineering team

At the time we’re writing this article, it’s the middle of scorching summer so let’s begin with a sunny analogy. Imagine your holidays have just started, and you’re about to hit the road to the dream destination like Rome or Lake Garda. You get behind the wheel, and your journey begins. As you are anything but a reckless tourist, in addition to dreaming about charming Italian landscapes, you are continually monitoring your progress.

You’re keeping track of how full your tank is, how closely you’re following the route drawn by your GPS, and how fast you’re driving. Based on this information, you can assess how well you are doing in reaching your destination and if you need to change anything in the way you are driving. The reason why we’ve listed all the parameters you need to keep track of when taking a trip is simple: they are a more down-to-earth equivalent of software development metrics.

A few words on the importance of measurement

That being said, we could also compare you, an experienced C-level executive, to a thoughtful traveler. How come? You probably get the idea, but let us clarify: measuring progress is critical, whether you embark on a physical or a more metaphorical business journey.

In the case of software development, we have some robust data to prove our point. The good news is that over the last couple of years, project performance and success rate has been steadily rising. In 2018, almost 70% of projects met their original goal, whereas nearly 60% were completed within the original budget.

Nonetheless, IT projects are continuously challenging to manage. The survey results for the 2020 edition of Pulse of the Profession revealed that an average 11.4% of investment is wasted due to poor project performance. The issues include going over budget, exceeding a deadline, or delivering fewer functionalities than agreed on. Luckily, when things do not add up in software development, there’s usually one remedy that works regardless of the scenario: measuring and drawing conclusions.

Keeping track of how your software development team has been doing, you get to enjoy the following benefits:

• improved project management allowing for a precise prediction of both costs and outcomes,
• easier identification of bottlenecks,
• more room for drawing conclusions and data-driven improvement,
• clarification of expectations and performance goals for developers,
• visualization of teams’ accomplishments for clients.

The challenge, however, is not the measurement in itself. The challenge is the choice of relevant metrics that will allow you to assess the performance of your software development team’s performance. Our goal today is to introduce you to some of the most common parameters, explain how they reflect the effectiveness (or lack thereof) of your tech partner’s processes, and show how to improve the productivity of the outsourced team.

What gets measured, gets done: assessing the team’s performance with software development metrics

Software development metrics are measurable pieces of data regarding the way in which IT projects unfold. When recorded and paid attention to, they can become a source of invaluable information about the tech partner’s processes and performance. To continue drawing the travel-related analogies, they can act as a GPS device that shows the team’s current location with respect to the place it should arrive at to reach specific business objectives.

For an IT project to be successful, you need to keep track of both business and software-oriented metrics. According to the Gartner study, the general agreement between CIOs is that the ideal ratio is 56% of data related to business outcomes (e.g., revenue growth or business margins) to 44% of information about IT delivery.

As you may imagine, any data can become a software development metric – especially in this day and age when project management software provides a wide range of analytics and reporting tools. Today, however, we’re going to focus on three types of parameters which we believe to be exceptionally to the point: Agile, operational, and code metrics.

What does the effectiveness of a software development team mean?

First things first, what do we mean by “effectiveness” and “productivity” in software development? The definition will depend on who you’re going to ask. For clients, these terms might refer to one, two, or all of the three following concepts:

• short time to market and delivery time,
• sticking to estimations,
• delivering functionalities of high business value.

For us, effectiveness means a combination of all the factors mentioned above: developing meaningful pieces of software on time and without exceeding the budget. As you may expect, the three aspects of efficiency in software development work a bit like the good/fast/cheap triangle and the pick two rule. The only difference is that with well-thought-out processes in place, we’re able to work out a satisfactory compromise calling for no significant sacrifice.

Agile software development metrics

As stated in the 9th Global Project Management Survey, 71% of organizations reported using Agile approaches for their projects sometimes, often, or always. It shouldn’t come as a surprise that Agile software development metrics are among the most popular IT performance indicators. As observed by Steven A. Lowe, these metrics

don’t measure success or the value-added, and they have nothing to do with the objective quality of the software.

Nonetheless, they are a helpful tool for spotting red flags in your app development lifecycle – before they grow into immovable roadblocks. To be honest, it wouldn’t be an overstatement to say that they’re invaluable in tracking project progress.

• sprint burndown

Application of Agile methodologies assumes dividing project work into smaller chunks called sprints. Their length varies across teams but typically revolves around two weeks. The purpose of a sprint is to facilitate the planning process, make it easier to divide work evenly, and increase the team’s flexibility.

That said, we can proceed to define the first Agile metric in this roundup. Sprint burndown is a visualization of the estimated amount of work that remains to be completed before the end of the sprint. It takes the form of a chart with the number of days in the sprint on the horizontal axis and the time needed to complete all tasks on the vertical axis. Regardless of the project, the goal is for the actual burndown line to reach the bottom right corner of the chart by the end of the sprint.

Burndown chart generated in Jira

The role of a sprint burndown chart is to show the project workflow and reflect the software development team’s productivity in comparison with sprint goals. It also serves to improve project management and enable informed decision-making. For instance, if the sprint burndown chart shows a sudden drop in the actual work remaining to be done, it might mean that the tasks aren’t granular enough or that the work updates take place only at the end of the sprint. If, on the other hand, the actual burndown line suddenly spiked, it might mean that the scope of work for a given sprint has increased. Finally, if the actual line does not reach the zero value on deadline consistently, it most likely means that the team was asked to do more work than they’re capable of.

• team velocity

Velocity is an Agile software development metric that shows the average amount of work (measured, for example, in story points) completed within a single sprint. As a result, it not only serves to assess the last or average efficiency of the project team but also helps to set more realistic expectations and make more probable forecasts.

Velocity chart

The key thing to remember when keeping track of this software development metric is that every team’s velocity is different. Just like there are no two identical projects, there are no two identical engineering squads. On top of that, velocity can change over time, e.g. newly-established teams may see an increase in velocity before they arrive at a perfect synergy in terms of relationships and processes. Each team may have a different approach to estimation so there’s no point in comparing velocities across them.

• lead time and cycle time

Although lead and cycle time are widely used in Kanban, one of the Agile methodologies, they often get confused – which is why we’re going not only to define but also to juxtapose them.

📚 Now you might be interested to learn how Kanban helps to improve the software development process.

Lead time measures how long it takes from the moment the task is created to the point when it is completed. This metric provides insights about the duration of the respective stages of the software development process and indicates practices that need improvement. As far as the reasons for high lead time are concerned, the most common ones include inefficient backlog management and slow approval process.

Cycle time, on the other hand, stands for the period between the commencement and the completion of work. In other words, it indicates how long it takes the team to complete a task. As such, measuring cycle time enables making data-driven estimates regarding the time needed to introduce a specific functionality.

Lead time vs cycle time

Now, how about the difference between these two metrics?  Let’s use food as a means to make a simple comparison. You go to a restaurant to have lunch. The moment you place the order, we begin to measure lead time and it’s only when the cook starts working on it that we start monitoring cycle time. We bet that even the finest dish won’t satisfy you if it takes a considerable amount of time to be served, will it?

The same goes for software development: the longer it takes to assign a task to an engineer, the longer lead time becomes, and the more frustrated users grow. This is why keeping track of both performance metrics time at once seems only reasonable.

• throughput

Another Kanban metric is throughput, a measure of valuable work delivered within a set amount of time. Since it ignores all the tasks whose status is other than “done”, it gives a clear idea of the team’s effectiveness in reaching goals.

As was the case with the above-mentioned software metrics, the team’s throughput should never be considered a single unchangeable value. Thus, when asking the tech partner’s PM about the throughput, expect to hear something like “between 8 and 14 tasks per sprint although we’ve been leaning towards the upper limit recently” rather than, let’s say, “10 tasks a sprint”.

• tools that can help you keep track of Agile metrics

Monitoring the variables mentioned above (or even more metrics) can be an arduous task – so it’s a good idea to invest in tools that will do that for you. At Merixstudio, our software of choice is Jira which, when configured to match your needs, tracks all performance-related metrics and informs us how accurate our predictions were. We also use custom spreadsheets to facilitate risk management and manage project budgets – trust us, there are more formulas in heaven and earth than are dreamt of in your philosophy.

Operational metrics

Let’s state the obvious: nobody likes failures, which is why every software development company strives for perfection. Unfortunately, creating flawless products is exceptionally challenging, if not impossible at all. Applications crash every once in a while and if you want to find out how efficiently the team can address the operational issues, you need a separate set of metrics for that.

Enter mean time between failures and mean time to recover. The former, usually abbreviated to MTBF, measures how much time passes between subsequent breakdowns and can help you assess the performance and safety of a given piece of software. The latter, also referred to as MTTR, refers to the amount of time required to fix an issue. As it monitors repair and testing time, it can provide you with invaluable insights concerning the way your software development team approaches bug fixing.

Although operational metrics do not inform us about the effectiveness of a software development team per se, they provide insights into the quality of a solution and the maturity of a product.

Code metrics

In addition to Agile and operational parameters, there are also code metrics that inform about the complexity of the product and the ease of its maintenance. Let’s focus on the three most important ones.

• code coverage

The first metric in this category shows how many lines of code are being covered by automated tests. As such, it is important for Test Driven Development (TDD) and Continuous Integration (CD). Generally, the higher the value of code coverage the better – although to ensure the high quality of the product, one should keep track of more metrics.

• cyclomatic complexity

Next comes the metric which evaluates the complexity of the code. Broadly speaking, the higher the code complexity, the higher the chance of the performance problems and the longer it takes to fix the bugs in the code or to get acquainted with the code by a new team member.

• code churn

Last but not least comes code churn, a graphic representation of what happens to the code during the development process. It shows the number of lines that a developer added, modified, and deleted in a given period. This metric allows looking for any irregularities and trends in code development.

Not everything that can be counted counts: the issue with software development metrics

We cannot deny that software development metrics are a useful navigational and decision-making tool for both tech partners and their clients. On the one hand, they help stick to the specification, budget, and deadline. On the other, they offer revealing insights about processes and team productivity – not to mention how important they are in identifying areas for improvement.

Nonetheless, as is the case with any other tool, you have to beware of several false assumptions that can be made when choosing metrics for all teams, including the software development team. Here are the mistakes that can impair rather than improve the engineers’ work:

• failing to define goals

Just like you need a destination when planning a trip, you need a business goal when planning the software development process. This concerns not only the functionalities that your product is supposed to be composed of and their quality but also the way they are delivered. To make goals attainable, remember about setting them in line with the SMART model (ensuring your indicators are specific, measurable, attainable, relevant, and time-bound). Guided by a sense of purpose, you’ll find it much easier to choose relevant metrics. And speaking of, the next common problem is…

• poor choice of software development metrics

In the previous section, we introduced ten metrics related, among others, to the productivity of your software development team. This, however, is just the tip of an iceberg. In fact, there are as many metrics and reports you can dream of – so the issue for many entrepreneurs is choosing too many KPIs for the tech partner’s squad in the hope of analyzing all their activities. Of course, you can measure them but this probably won’t give you much advantage.

A good example of a poor choice of metrics is also getting fixated on details such as the number of the lines of code or the number of bugs. The former is a bit like writing a lengthy school essay just for the sake of reaching the desired word count – without conveying any real value. The latter, on the other hand, could lead to a situation in which no bugs would be reported and instead, tasks in Jira would be getting dozens of comments. Can you imagine the chaos that would break out?

Another issue is relying only on the business value which may seem like an objective metric at first glance – after all, it’s determined not by your software development team but by yourself, the client. Nonetheless, the business value is subject to change over time for various reasons. What’s more, if there are a couple of engineering squads working on a project, some may end up being assigned tasks that have greater business value than others. In this case, we can’t talk about objectivity.

• monitoring individual performance

In the title of this article we refer to the effectiveness of a software development team and not a single engineer for a very simple reason: no person involved in the IT project exists in a vacuum. The work of a developer influences the tasks of the designer, which in turn affect the duties of QU specialists – and so on. That’s why, regardless of which performance metrics you choose to monitor, they should always measure the work of the entire project squad.

• making wrong comparisons

When discussing some of the above-mentioned parameters, we mentioned that values will differ across teams. To be honest, that’s the case with every metric you choose. The reason for that is because, as Steven A. Lowe rightly observed,

In software development, everything we build is a snowflake: unique, valuable, and incomparable.

This rule applies to all elements of the app development process during which we craft custom solutions. First of all, even if your product resembles a solution already proposed by the competition, it surely boasts a unique value proposition. Secondly, each specialist assigned to the project has different background knowledge, skillset, and motivation. Finally, even the slightest change in the team composition influences the workflow and relationships.

Let us once again give you a practical example of what this fallacy can lead to. Imagine you start comparing velocities between teams and it shows that one squad is significantly “outperforming” the other. The thing is, however, story points don’t have a fixed value – as they are subjective, each team can assign a different number to seemingly similar tasks. What’s more, comparing values across project teams may lead to a situation whereby the “underperforming” squad will begin to change the way they estimate tasks. Most likely, it won’t be a conscious action – it will all be a result of their desire to perform better.

Summing up, however we decide to measure the performance of a software development team, the only relevant comparison is to the given squad’s previous achievement – although even that may not always be the case, as we already observed above.

• assuming the cause

Let’s say that you’ve selected a few metrics, and you’re monitoring them closely. Suddenly, you notice an alarming drop in the values achieved. The worst thing you can do in such situations is to assume the cause without talking to the team. Of course, there are reasons that correlate with some trends better than others – we’ve mentioned a few ourselves.

At the end of the day, however, only with the help of those involved in the software development process can you explain what accounts for the unexpected change in performance, decide whether there is cause for concern and find out how to fix it. This is why open communication, transparency, and trust should be the building blocks of your relationship with the tech partner.

How to improve the effectiveness of the software development team

So far, we’ve discussed the most common ways to measure your software development team’s productivity and the traps you may fall into when choosing metrics to monitor. Now, we’re going to share a couple of tried and tested tips for improving your IT squad’s effectiveness.

Set explicit goals and communicate them clearly

Most of us are told from a very young age that goal-setting is essential because it helps to prioritize tasks and move forward in a structured way. Mind you; these are not empty words – this claim has been scientifically proved already in the 1960s.

Back in 1968, Dr. Edwin Locke published an article “Toward a Theory of Task Motivation” in which he emphasized the connection between setting clear goals and increasing both motivation and performance. Later in his career, the American psychologist found that over 90% of the time, goals that were specific and challenging (but not unachievable) lead to higher performance than easy or generic goals.

The goal-setting theory has its application in software development as well. The more specific your objectives are, and the more clearly you communicate them, the higher the chance of developers not wasting time working on features that you deem unnecessary later on. Also, when you keep the team in the loop in terms of the purpose of what they’re doing, they are more likely to stay committed to their tasks and the general well-being of the project.

Streamline project work with project management

Another thing worth mentioning when discussing goal-setting is the importance of having a vendor’s Project Manager on a project. Of course, both you and the outsourced engineers are experts in their fields. Nevertheless, having a mediator who’ll translate your business insights into tech terms and vice versa will surely streamline communication and improve the effectiveness of the software development team.

📙 Here you’ll find out more about keeping high standards of communication in remote cooperation with a software house.

Another thing about PMs is that they’re the ones whose job is to keep an eye on the way the project unfolds – and what better way to do this than by paying attention to reliable metrics?

Encourage productive environment

Distraction is the killer of productivity. As Gloria Mark, a professor in the Department of Informatics at the University of California, states:

[When interrupted] you have to completely shift your thinking, it takes you a while to get into it, and it takes you a while to get back and remember where you were… We found about 82% of all interrupted work is resumed on the same day. But here’s the bad news — it takes an average of 23 minutes and 15 seconds to get back to the task.

In 2018, Udemy surveyed a thousand U.S. office workers to collect some solid data on the effect that distractions have on the quality of their work. The results were striking, although not surprising: workplace distractions proved to hinder productivity. Here are the three vital statistics:

How about the top workplace distractors? On top of the list, you’ll find chatty coworkers (80%), noise (70%), and meetings (60%). Many developers treat the latter as a synonym for a waste of time. To increase their productivity, we suggest taking the following steps:

• it’s about the quality not quantity so try to reduce the number of meetings to a bare minimum and set a time limit – trust us; usually a 10-minute daily is much better than an almost hour-long meeting,
• always establish and share the agenda before the meeting – this way you’ll set the conversation path and enable preparation,
• conclude every session with an action plan so that everybody knows what tasks they’re responsible for.

And as far as the office noise is concerned, it also won’t hurt to pay attention to whether your tech partner preserves the culture of mutual respect at the open space.

Give the team a chance to hone skills

Being a C-level executive, you surely know that continuous improvement is key to business success. The same is true for software development – an industry where new frameworks are released every day, and the emergence of new professional specializations is just a matter of time.

Therefore, one of the most effective ways of increasing the software development team’s productivity is pushing them out of their comfort zone and encouraging them to explore cutting-edge technologies. Keeping up to date with the newest trends in the IT world will allow them to suggest optimal solutions – not to mention that acquiring new soft skills will also help them rise to project-related challenges.

And speaking of having an open mind, it’s also worth preventing the team (and yourself) from falling victim to the sunk cost fallacy. Sometimes even the most promising tools or assumptions do not bear fruit. We’re all human and we make mistakes, so try not to get fixated on inefficient solutions only because you’ve already purchased a monthly subscription. Observe what happens in the project, draw conclusions, and don’t be afraid to move on.

Rethink the workflow and processes

One of the reasons why the software development team hasn’t unlocked its full potential yet can be the project management methodology it follows. According to the QSM Associates study, which examined over 8,000 projects, agile teams prove to be 25% more productive and 50% faster to market than their non-agile counterparts.

What can account for such results? Greater flexibility, more granular tasks, the ease of removing blockers on the go, just to name a few reasons. Of course, it doesn’t mean that Agile is the only way to go. After all, there are numerous projects which call for a tailored mixture of methodologies. However, if your tech partner’s team seems to be overwhelmed by the pure Waterfall, maybe it’s high time to find a more agile vendor.

Another way in which Agile fosters productivity is through ceremonies such as retrospective. It’s the meeting held at the end of the sprint to reflect on the work done so far and identify areas for improvement. As such, Agile retrospective is a lessons learned meeting during which the team discusses tech, process, and communication-related issues and attempts to find the best fitting solutions.

Last but not least, when it comes to work organization, it’s a good idea to offer engineers greater flexibility. There are plenty of studies trumpeting the virtue of flexible working, and the benefits are innumerable: greater engagement, effectiveness, happiness… If splitting the traditional workday in half does the trick, why force the development team to rack their brains for 8 hours straight?

Final thoughts on measuring and improving the effectiveness of your team with software development metrics

Unless you measure project progress and team performance, you can’t be sure whether you’re heading in the right direction or not. We admit it: it’s not an easy task, but the effort put into identifying relevant software development metrics and keeping track of them pays off in invaluable insights and learning opportunities. And although there is no magical formula that will work equally well for every business, we hope that our roundup of parameters worth tracking and a few experience-based pieces of advice will help you measure what’s right.

Looking to drive innovation effectively? Check out our services and see how we can help you do that.

‍