Improving Test Automation is one of the key goals for Magento 2. Test automation can help improve the quality of code, find introduced bugs earlier, reduce release overheads making more frequent releases possible, and so on. In the modern day world of software development, I doubt any serious project does not include at least some test automation as a part of the overall plan.

This blog summarizes the different types of test automation in Magento 2 and their purpose. There is also testing documentation currently available on the Magento 2 documentation site.

Disclaimer: While I work for eBay, this post contains my own thoughts and opinions – especially in the conclusions section!

Acknowledgements: This post borrows heavily from @max_pronko‘s talk at Imagine 2014. Isn’t plagiarism the sincerest form of flattery? Thanks also to @AntonKril for filling in some gaps.

Unit Tests

Unit tests are designed to test individual classes or functions in isolation. They, along with integration and functional tests, are useful to find logic bugs in code. Because they are highly focused, they make it fast to identify the root cause of problems when a test does fail. They also have a great side effect of making it painful to write highly coupled code.

You can run all tests (php -f dev/tools/tests.php — type::all) or individual test suites (phpunit dev/tests/<test_type>/ testsuite). The unit tests themselves are currently in dev/tests/unit, although these may be moved into the module directories in the future (TBD). That would allow them to be included more easily with each module.

An interesting idea for extensions would be to give a quality score by automatically running all unit tests within the extension and checking the test coverage percentage. One way to reward those extension developers going the extra mile!

Integration Tests

Integration tests are designed to prevent bugs in interactions between components, layers, and the environment. Rather than mock out external classes, an integration test tests multiple real classes in combination. They can be found in dev/tests/integration. Care should be taken to not fall back to integration tests to avoid writing true unit tests, but integration tests definitely add value.

JavaScript Tests

JavaScript in web sites started small, but has grown into large scale libraries. JavaScript is a programming language, and so testing of JavaScript is equally as valid as other tests. Guess where these hide!  Yep, dev/tests/js.

Performance Tests

Internally, the Magento development team run some tests to track and maintain certain performance level per build. This has spotted one or two bits of bad code that almost snuck in. These tests are however not full LnP (Load and Performance) tests. That is a separate dedicated activity that will increase as more areas of code settle. (There is not much point performance testing and optimizing code that is about to undergo major refactoring.) See dev/tests/performance.

Static Code Analysis Tests

These tests are used to verify that the code conforms to coding standards such as PSR-1 etc. They do not run the code – they analyse it. As anti-patterns are discovered, test cases are merged into the static tests to ensure the bad pattern is addressed and eliminated (over time). See dev/tests/static.

Static integrity tests check how the application is linked again without running the code. They perform checks such as checking the correctness of the di.xml file against the code. Static integrity tests are located in dev/tests/static/testsuite/Magento/Test/Integrity.

Integrity Tests

Integration integrity check how an application is linked. They deal with some of the more advanced scenarios that require the application to be run (like merged config validation). The tests are located in dev/tests/integration/testsuite/Magento/Test/Integrity.

Legacy Tests

Legacy tests will help developers port their extensions to Magento 2. They report on usage of methods/classes/constants used by the extension that have been removed/moved and what to use instead.

Functional Tests

Functional tests verify the product features by exercising the system in the same way as a user would interact with it. They are the main test the rendering logic of pages. A new test framework (creatively called the Magento Testing Framework, or MTF for short) is PHP code wrapped around Selenium. If you look at some of the tests in dev/tests/functional you will see code that uses XPath expressions like //tr[normalize-space(td)=”Subtotal”]//span and //a[@class=”rule-param-remove”] to navigate around the returned HTML to extract values to check within the response. (Another advantage of Magento 2 using more class declarations in the HTML, pulling CSS styles out of the code base.) By doing this, the tests are more resilient to changes in the application.

Conclusions

The Magento 2 code base is currently being pushed to GitHub weekly, including tests. The tests are running publicly at https://travis-ci.org/magento/magento2. There has been a lot of interest from parts of the community to see if the test suites provided can be used to help on-site development practices. I think this will be an interesting experiment, but quietly I have some doubts. (Ssh! Don’t tell anyone!) A goal of Magento 2 is to avoid the need for developers to ever have to change the core shipped files (by providing clean extension mechanisms). All the unit tests for example test classes independently. If the core files are not changed, then the unit tests should continue to work as before (because they are purposely cut off from the west of the world).

Functional tests will also be interesting. If a site has local customizations (and what Magento store doesn’t!), it may be correct for some shipped functional tests to fail. The system is behaving differently – as it should! It will take developers effort to determine if the breakages are acceptable or not. The framework may however be a useful tool for developers to build their own tests for extensions they develop.

Tests may also help when developers find bugs in core Magento. Developers can develop a workaround and then test it to make sure there are no unexpected side effects, getting a customer up and going faster. So why don’t I think this is much help? Well, Magento 2 is going to ship without bugs! (pause) Hmmm, maybe they will be useful for a little while after Magento 2 ships…