#Metrics Archives

Developing a quality software product falls under the realm of software engineering. The ever changing user requirements and harsh project timelines pave way for cost overruns or delays in the delivery schedule. So, bigger the scope of your project, more complex the functionality becomes. With so many factors it is very easy to lose control over the quality processes to develop an excellent product. We know that no engineering is complete without accurate measurements; which further brings us to the point that all measurements are done by applying certain metrics.For this reason we need software metrics for transparent and quality delivery of the product. Because when you measure what you speak about, you have numbers to express your failure or success. You have a calculated expression which shows you where you lacked and places you stood well. In Bob Parsons’ words-‘Anything that is measured and watched, improves.

Now the question arises- which software metrics to use?

One option is to pick a set of predefined metrics. The other one is you can first define the product requirements, discuss with the various stakeholders and then set the required metrics.

For a better understanding, consider an analogy when you are backpacking for an adventure activity. Would you pack a parachute if you are going for kayaking?

Absolutely not.

Instead, you would first decide upon the adventure sport, do a bit of research and accordingly pack your adventure gear. So, if you are going for paragliding, pack your bag with a helmet and a harness.

You can leave the oars behind for the next time.

Defying the law of gravity? It’s okay. But make sure you don’t cause more harm than good in the project. For this reason, it is important to apply the right metrics at the right stage in the product development cycle. As a software development company which builds engaging products.

1. Team Velocity Points

In simple words, team velocity measures the average speed of your team towards achieving its goal during one sprint. The fact that a typical sprint goes on for about 2-4 weeks, calculation of team velocity points proves to be a valid metric to gather the team’s efficiency.

If you want to calculate your team’s velocity, sum up the story points that were completed during that Sprint. To decide the team’s velocity, a three sprint’s average is usually considered.

Why? It’s an industry standard and it determines the project’s pace at which the functionality is being developed. In simple terms, it tracks the velocity of a Scrum team over time. This metric helps in deciding whether the team can take up more work over time or is the team already overloaded and it need to be reduced.

2. Sprint Burndown

Sprint burndown chart is the most effective tracking metric to visualize the health of the project. It helps in reviewing how much work is pending and if the team needs to speed up or slow down. You can also predict the expected completion date of the sprint. A deviation from the expected burndown will send a red flag to the scrum master to take appropriate actions.

For instance, if burndown goes up at any point during the sprint, it reflects an addition to the existing scope. At this time, it’s important for a team to achieve ‘team’s goal’ first before fetching new items from the backlog.

Why? The Burndown chart indicates the progress of work within a sprint. It shows whether amount of work a team is accomplishing every day will lead to a successful sprint completion or not.

3. Release Burnup

Product owners need to evaluate what should be out in the market first based on their discussions with other stakeholders. Release Burnup helps POs to continuously prioritize product backlog based on how the work is progressing.

If the release date is fixed and there has been a few additions to the initial scope, this is an important metric to align scope with the actual release date by either removing low priority backlog items or attempting to increase velocity by taking suitable measures.

Why? The Release Burnup chart indicates whether the sprints are progressing towards a successful project completion in terms of functionality achieved. Also, gives us a projected date of completion of the project.

4. Estimated Cost vs Actual Cost at Completion

Often there are gaps between the estimated cost projection during the start of the project and the actual cost at completion of the project. This metric gives a measurement of what deviations could have led to this difference in the cost.

Although, almost always the project cost goes higher than the expectation (just like we tend to overspend while shopping), this metric helps you understand what could be the possible cause of cost overrun- be it some design error or some changes in the scope of product development.

Why? Helps us understand what cost we had estimated and what cost was incurred. This helps us in improving our planning.

5. Number of major bugs per sprint

The number is an indicator of the overall quality of the product. When you are working in an agile project, it becomes important to put a budget i.e. count on the number of major defects open in a project at any point of time.

How can we stay within the budget? Reduce the count of defects reported.

How can we reduce the number of defects? Do Not Report!

Yes, I’m kidding.

Here is a better solution- focus at the time of planning, look-ahead and clearly define acceptance criteria before jumping on a sprint backlog item.

The priority of resolving any major bug is over and above implementing any new feature. Because if you can’t resolve what your did wrong, you can not pick a new feature.

At the same time, keep a close watch at the number of bugs reported after production release. If you have not been proactive in refactoring and improving quality of your code, chances are you will receive nasty numbers in this metric. This will help you to identify gaps in testing and take corrective measures

When Lyla was asked if she could interview an author on her colleague’s behalf, she jumped at the opportunity. Lyla was interning with an uptown magazine for a role that deemed her ‘unfit’ for any important tasks such as interviewing a popular author. Getting an opportunity to talk to her favorite author was like a jackpot. She thought she is finally gripping onto a childhood dream. She thought she finally can brag about meeting Paula Hawkins. She wanted to be careful about not screwing this golden opportunity, so she double-checked everything on her list.
Notepad, check. Pen, check. Camera, check. Questions, check. Recorder, check! Oh wait! I have to check for a new update, just in case I am outdated.

She updated her app quickly and reached the coffee shop before time. She was excited, she was anxious. It’s an easy-peasy task, you just have to ask questions, record them and pour it out later– she kept repeating this in her mind.
But the moment Paula Hawkins walked in, she started feeling the breathlessness. Part excited, part nervous, she could feel her palms sweating. After exchanging some pleasantries they kicked off with the interview.

“Shall we start?”
“Yes, of course. Let me just..umm… Start this thing”
*she clicked on the record button and started talking, simultaneously taking notes.*

Unaware of what was happening after she clicked on that little red button, she went on with the interview, taking down notes as per her whims. Ignorant of the new updated feature — which now needed ‘long press and hold to start recording’, she finished the interview, smiling and pleased with herself. But as she looked down, in order to ‘STOP’ the recorder, she lost the fanatic feel of doing a good job!

A little hover over the button said “long press and hold to start recording” which was an obvious slip.
Game over. She probably lost her mind or her job. Or both. But that’s none of our concern. Right? She should have been more attentive.

Lyla is a fake person. She did not meet Paula Hawkins ever. But the incident is real. Only, it happened with an acquaintance and I figured out it wouldn’t be fair to name her. This incident forced me to shine some light on the importance of usability testing and why it’s more important to pay attention to user’s behavior while they interact with the product.

They say, never talk to your users, they hardly know what they want. But they missed out on the summary-

“To design the best UX, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior. Users do not know what they want.”

Usability testing- What is it?

Usability testing is like black-box testing of an application to ascertain if the product built is convenient to use and easy to learn.

These are methods of testing and observing the behavior of the users to find out what works and what doesn’t work. Users are given specific tasks to complete and when they are at work, observers watch their body language, facial expressions, emotions and encourage them to “think aloud” i.e. speak up whatever comes to their mind while using the product. Doing this exercise we can get qualitative and quantitative data and figure out usability issues with a product.

So, why usability testing is important?

Doing usability testing the right way, at right time with the right set of people reduces the risk of building the wrong product; thereby saving time, money and other precious resources. In other words, if done at an early stage when the product is at paper prototyping stage, it finds the problems when they are easy and cheap to fix. And when done on a product which has attained maturity it helps to understand user’s success rate and time spent to complete a task. There are hundreds and thousands of cases when usability testing proved to be a good exercise in terms of ROI.

For example, a slight tweak in design suggested by usability testing for Mac’s UI, the company got 90% fewer support calls.

Need more clarity about why usability testing is a good idea? Here you go-
#1. To check if product meet user’s expectations
#2. Matches business decisions to real-world use
#3. Removes flaws in the product
#4. Allows you to see how successful users are with their tasks
#5. Useful for getting user reactions and feedback about the product

What are the types of data that we can get as a result of our analysis?

Two types of data results received are — quantitative and qualitative. Usability testing is largely a qualitative research technique and is not driven by statistics like surveys where lots of people participate. Usability testing is done using a small set of people, usually five to seven.

Qualitative methods are very useful to test the stress response of the users like their body language, movement of hands, expressions on the face and squinting eyes especially doing a test on a mobile device.

The metrics we get after usability testing can be quantitative as well. For example, time spent on doing a task, success and failure rates and also the efforts, like how many clicks a user needs in order to complete a task.

Is there a need to record all the metrics obtained from a usability testing?

Yes, keeping a record of the metrics is very important. Why? Because usability testing is not just for designers to understand how to make better designs but it is also an important tool to influence the rest of the stakeholders like clients, their sales/support team, project managers, developers, other designers etc.

Every stakeholder involved may have a different point of view for a design decision. Being subjective by nature, design decision often leads to long debates among stakeholders. Most often design decisions are influenced by a person who holds the highest position among fellow stakeholder or has superior oratory skills.

In short, metrics help us in iterating and validating design concepts. It gives objectivity to design debates and it helps in taking fact-based design decisions.

At what phase of the design process usability testing is recommended?

When it comes to usability testing there are two terms often referred by big names of the UX industry (like Jacob Nielsen) and these terms are Summative Test and Formative Test.

These tests are done at different stages of the design process. They are as explained below:
Formative tests are low-fidelity tests (to gain quick insights)-
#1. During the very initial development phase using paper prototypes
#2. It can be done anywhere and a formal lab is not required
#3. It can be done just between a moderator and a participant

The results from a formative test may include-
#a. Users’ comments in the form of “Thinking Aloud” narrative i.e. their emotions, confusion sources and their reasons for actions.

Summative tests are high fidelity tests (to capture metrics)-
#1. These are carried out at the end of the development stage
#2. At this stage usability of a product is validated
#3. This gives an answer to the question “How usable the product is?”
#4. This gives a comparison against competitor products
#5. Conducted in usability labs or remotely using many tools available where users can do the test using their computers or mobile phones

The results from summative tests may include-
#a. User’s success rate to achieve a goal
#b. The time spent on completing a task

How many users are required to conduct the testing?

“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests you can afford”
Jacob Nielsen

“It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.”
Laura Faulkner

Who among them is right?

It depends on what type of test we are doing and where we are doing it. For example, if we are doing a low-fidelity formative testing we can do away with a small sample size. But if we are doing a summative testing we need a bigger sample size. In the type of testing where we are comparing our site to competitor’s website by using an online tool which is cheap and fast, we can use a large sample size. But we should keep in mind that these online tools like UserTesting or Loop11, they don’t capture metrics. It’s us who has to be aware of how all the participants did it.

So how to prepare a test plan?

That is certainly a good question. You have an inquisitive mind, I must say. But don’t you think that will be too much to digest in one bite?
Still, if you are really into it, give these things a thought-
-Decide what areas to concentrate on
-Determine potential usability issues
-Determine what tasks you want to test

Tag: #Metrics