Magento 2 is more than a revamped platform – automating testing has also been a significant investment. But is it paying off? Is test automation worth the effort? YES! Here is the data to prove it.

Disclaimer: opinions expressed here are my own and not necessarily that of my employer. Please also note the data shown here has not been verified. Since this is not an official eBay communication I also removed absolute numbers – it’s the trend lines that matter anyway.

Development Practices

In a previous post I described the types of testing being employed on Magento 2. The team has continued to invest in a range of test cases for the different areas. The work is by no means complete or up to what the team believes is a satisfactory level. But the coverage has been improving, enough to see substantive benefits as a result.

Internally development is broken into a series of milestones (or PSI’s if you are into Scaled Agile terminology). Each milestone is 4 sprints and was originally followed by a significant manual regression test cycle (weeks with a large team).  As the milestones progressed, the team has progressively improved the test automation coverage. This was done using practices such as “no new code may be committed without test coverage, even if you are modifying code you did not originally write” and just setting targets for each team and reporting on the results. After a while it has become a part of the culture and you just do it.

Improved Regression Test Cycles

The following graph shows the relative number of issues of different priorities identified during each regression test cycle. Milestone 1 was around mid-2013; milestone 6 completed just the other week. If you look at the graph closely, you can see a steady improvement on the number of P0 (blocker) and P1 (critical) issues identified per regression cycle. P2 issues were given lower priority, but after the P0 and P1 issues were under control more work was invested into the lower priority issues as well. You can now see a steady drop during milestones 5 and 6 for new P2 and P3 issues being identified during regression.

But there is more good news. What is not shown in the graph is the effort required to perform regression testing. This has reduced from many weeks with a large number of QA engineers to a couple of QA engineers finishing the testing within a week (as all the tests went through smoothly).  QA can now spend more time proactively developing tests to stop problems in advance.

Improved Issue Backlog

So less new issues are being found during regression testing. What about the overall issue count trends?

Magento 2 was forked from a version of Magento 1 (unsurprisingly!) and a lot of the Magento 1 known issues (defects/bugs) were brought along with the code base. The QA team also continued to test and raise new Jira issues as they found them while development progressed. During the early milestones of the project a lot of effort was invested around greater visibility in metrics with them being reported openly to upper management. The initial story was not great. The issue count was increasing as product development progressed.  (It was not actually as bad as this sounds, but the back story is too complicated to include here – and the trend was definitely going in the wrong direction regardless of the reasons.)

I will let you decide whether a blog post I made October last year “You Can’t Improve what you Can’t Measure” was at about this time.  (There was a later post on “Feedback Loops and Measurement” if you are interested as well.)

The good news is the Engineering team accepted the challenge. They set targets to decrease the bug counts and improve internal practices. Things steadily improved, but the trend line was still not great. Around the Imagine conference this year (May) an extra effort was put into an internal bug bash. Lots of management was away at Imagine so the teams just went off and killed as many issues as they could. This to me was a significant turning point in the project as from then on the issue count has been steadily dropping. We still get the occasional bumps during regression test phases, but the trend lines are clear.

Continuing at the current velocity the issue backlog should be practically eliminated during H1 next year!

Okay, I actually have a sneaking suspicion we hit some bumps because the issue count arrival rate will probably increase during the developer beta period, but this to me is a fantastic result considering the project started with a trend line that was going up for the first few milestones, not down!

Conclusions

Obviously decreasing the issue counts is good news for Magento 2 – the final deliverable will be more solid from day one. Test automation is also speeding up the ability to add new product features without breaking existing functionality.

But I thought this was also a useful case study for other projects. Getting visibility into your project metrics does help. Show them to upper management. It is scary to start with, but if you are serious about building a quality product and have a mature management structure, the end results can be a success story. It does not happen overnight – it has been a year-long effort of setting targets per sprint and continually and incrementally working at it. But the end result is less new issues are being identified and the effort of regression test cycles is going down. That means more effort is going into product development. Magento 2 has not shipped yet, but multiple metrics are clearly improving as the coverage of automated tests increases.

So from me, my hat goes off to the Magento development team (which includes Product, Engineering, and PMO) for what I think are some spectacular results. Comparing milestone 6 to milestone 1 shows a clear improvement, and every bug that does not exist means saved work to fix it.

Nice!