I have written previously on version numbering for Magento 2 and the relationship between GitHub and Composer – this post provides a level deeper on what we are planning to take effect at Magento 2 GA (“real soon now”). Feedback, especially from extension developers, appreciated.
There are a number of challenges that Magento 2 and the versioning scheme addresses. Let’s review them first.
- A major goal of Magento 2 is to simplify upgrades, including patching. We want to move towards a single central point for releasing all patches, making them available to everyone at the same time. This implies eliminating (or at least reducing) the practice of hot fixes – code sent to specific customers for a reported problem. Magento 2 has numerous platform level changes to address issues that have blocked this effort in Magento 1. We want to make the experience “there is a patch, great I will install it” rather than “I will schedule some time to carefully explore the patch to review its potential impact”.
- Magento 2 has lots of modules (over a hundred) shipped as separate Composer packages. Composer and versioning really works best with one repository per package. We may move to one git repository per module one day, but not for the initial Magento 2 release. This means we need a solution that will support all of CE modules in one git repository. This places restrictions on branching and tagging policies for the repository. (Note: There are advantages in one git repository, such as it makes development easier and test automation easier.)
- Magento 2 is moving to more frequent releases (planned quarterly). This will impact extension developers. How often will an extension developer be required to retest and re-release their extensions to ensure compatibility with the base Magento release? Test automation can help, but reducing the pain here is a Magento 2 goal, which is why we have been moving towards Sematic Versioning of modules.
- What are the rules that extension developers need to follow in terms of declaring dependencies against the Magento platform? These dependencies are important for compatibility verification. How to keep these rules easy to understand, while giving extension developers enough control? For example, one extension developer may prefer to mandate customers use a particular tested version of an extension per CE release, to reduce potential support overheads of using an extension with a future CE release the extension developer has not tried yet. Another extension developer may prefer to not have to re-release their extension as frequently if the Magento 2 platform has not changed recently in areas their extension interacts with. This latter strategy is the goal – Magento 2 needs to demonstrate over time that has got the versioning and backwards compatibility right, so extension developers have the confidence to trust the core Magento team.
- For people providing support, multiple independent version numbers of every module is all well and good, but can greatly increase support complexity. Imagine if there are 5 versions of 100 modules? That is a bucket load of combinations to test for absolute certainty. Instead, by having well known and tested combinations of Magento 2 platform modules this can reduce the number of combinations that are tested. This combinations are then “well known”, also making life easier for support teams.
- If module A depends on a specific version of module B (e.g. 1.2.3) in the composer.json file, a patch to module B will force module A to also be released as a patch in order to update the composer.json file of module A (and no other reason). Of course everything that depends on module A may also then need to be re-released as well. This can trigger large ripples in releasing new modules for a single patch, which is undesirable as there is no actual code change in those other modules – just version number changes in the composer.json file.
Yes, lots of challenges! The good news is we have a solution that addresses all of the above.
The core of the proposed solution is as follows. (EE will follow the same pattern, but I limit the discussion here to CE for simplicitiy.)
- Every CE release such as 2.0, 2.1, or one day 3.0, will be given its own release branch in the git repository.
- As discussed in the past, product versions and module versions are independent. For example, CE will be released as 2.0 then 2.1, but modules will start from 100.0. This separation will help during support conversations, as the product version numbers and module version numbers will drift apart over time. For product version numbers, a marketing person decides when the next release will be Magento 3; for module versions, an engineer decides when behavior of a module has changed to need a major number increment. Modules will get major number increments faster than product number increments. Also, different modules within one CE release may have different major/minor numbers over time.
- The release branch name will use the CE product version number. (As module numbers over time will become out of sync for different modules in CE, module version numbers cannot be used as git tags/branches are global to the whole repository.) So at GA, the release branch will be referred to as something like “CE-2.0”.
- Patches will always be created and released from release branches. Patches only ever update the last digit of the version number of a module. Thus, if a new patch of the Sales module is needed, it will be created on the release branch. The original release will be say 100.0.0, and the first patch 100.0.1. A module on a release branch will never change the major or minor number of that module.
- If a bug fix is needed for multiple previous releases, it will be repeated in each release branch to make sure it is applied to that latest code for each release. This is because the code may be different between releases. Frequently the code will not have changed, making it easy (although boring) to back port a patch to multiple previous release branches.
- Development for the next major release continues in mainline. After a release is made (with a release branch created), all modules in mainline will have the minor number (the second digit) incremented (and the patch number set to zero).
- If during development a backwards incompatible change is made that will affect extension developers, the major number of that module will be incremented and the minor number set back to zero. This is done independently per module, which is why CE 2.1 could end up with some modules with version 100.1 and other modules with version 101.0. This practice is very important for extension developers, as will be expanded upon below.
- All CE modules will declare dependencies on other modules in composer.json files without use of specific version numbers (exceptional circumstances may arise). For example, a dependency of 100.0.* might be used. This avoids the patch release ripple problem mentioned in the problems section.
- Magento will also release a CE Composer metapackage that points to specific versions of all CE modules. Using this metapackage will lock down the exact version of every CE module being used. The metapackage will use the CE product number (plus a patch level) as the version number. So this metapackage at GA will have version 2.0.0. When the first patch is released, a new metapackage version pointing to any patched modules will be released with version 2.0.1. Each metapackage version will represent a supported and tested combination of modules. Most sites will just depend on the CE metapackage, and incorporate any module patches by moving to a newer patch level of the metapackage.
- Most likely, the git repository will be tagged when one or more modules have a patch released (a bug fix might require several modules to be changed, so they would all be released together). The tag will use the CE metapackge version number – something like “CE-2.0.1”. This will make it easy to explore changes made between patch releases in GitHub (you will be able to compare CE-2.0.1 and CE-2.0.2 for example).
Phew! Does the above sound scary or what! Well, the good news is that it is the Magento core team that need to worry about the above. The effort for Merchants and extension developers is pretty simple.
- For Merchant projects, they depend on the CE metapackage (or EE metapackage). They can specify a dependency on a specific version number (such as 2.0.1) or use a pattern to pick up patches automatically (such as 2.0.*).
System integrators can get smarter if they want to, but the above should satisfy most cases.
For Extension Developers
The world is slightly more complex for extension developers.
- Extension developers normally would depend on module major version numbers using version patterns such as 100.*.
- If an extension developer depends on a private API (Tsk! Tsk! …but sometimes it is necessary), they would specify the major and minor version number such as 100.0.* to stop the extension from working with the next CE release. This will force Merchants to upgrade to a newer (tested as compatible) version of the extension when they upgrade the CE release level.
That is it. The idea is extension developers rely on us changing the major version number correctly when we make a change that will impact extension developers. Examples of such changes include public API changes, significant layout file changes, etc.
The above scheme shows how release versioning and semantic versioning fit together. Patches only change the last digit, allowing 100.0.* style patterns to be used across patch levels. Backwards compatible changes to the public API and documented behavior of a module trigger a minor number change, allowing 100.* style patterns to be used when an extension only uses public APIs.
And this is where your voice is useful. If you are an extension developer, can you see any problems with the above? Feedback welcome!
Disclaimer: Even though incredibly detailed information was given above, officially these opinions are my own and not those of my employer and as such are non-binding to Magento.
I just hope this is not a license for Magento developers to go change things regularly at a major version level (which is what is happening right now on github). Because it will kill off all but enterprise SAAS providers who only have 1 extn to worry about. Unit tests or no unit tests.
You need public APIs on every component. You dont have them on many M2 components. So we are all using private, as are you between your components yourselves! Therefore we are always affected by your changes, the versioning is irrelevant.
On the plus side this will allow us to notify merchants that the extn is not compatible more easily than today (where they often just find out when they upgrade). On the downside I think it will create unnecessary support when that major version change doesnt actually impact our extn. They balance each other out.
I think the blog is scary btw, I had to read it 10 times for it to make sense. And I understand semantic versioning.
Sorry and another point. When we sell an extension our hope is that we don’t have to support that person. In M1.x what we find in a lot of cases is that people sit on a version for some time (thus reducing support). If you move to regular releases and easy upgrades (which I totally get from your perspective) you put even more load on extn providers. Given extn providers already will have a 30% squeeze on revenue from the AppStore its just all out of their profits. I think there are easier ways to make money.
I’m grateful we built out ShipperHQ and saw this coming, if I was just doing standalone extns for M2 I’d go find another sector to work in.
Personally I think it is fair to pass costs of improved quality on to merchants. They get better value and cost savings in terms of faster adoption and lower costs due to less problems.
I’d like to add three assertions to Alan’s reply. (1) We’ve made significant investment ourselves in building test architecture and code coverage, which should greatly reduce the burden of testing extensions against new versions (2) SemVer will help vendors know if there are breaking changes in the API. (3) The downward pressure on extension pricing from poor/pilfered extensions should be eliminated by the new Connect. Of course, we’ll know how accurate these assertions are when the pudding is here and we dig in.
Good to hear that the final conclusion is close to what has been voiced in Austin at the beginning of the year, a meta package to reference a set of individual module versions (eventually ending up in separate repositories), decoupled from the need for a product version.
The approach outlined here sounds workable for us. One thing which makes this discussion somewhat abstract: we do not even know how the extension install process is going to work. From what I understand it will be composer based with some Magento sugar on top, however until we get to actually use it, it’s hard to comment on declaring dependencies.
And as you mentioned the crux will really be “The idea is extension developers rely on us changing the major version number correctly” which only time will tell how well this works out. Having SemVer indicating when things break is one thing, the other thing that should go hand in hand with it is to try hardest not to introduce backwards incompatible changes in the first place.
The point that I am going to make is, that so far, I am not seeing any of this in practice and I do not believe something like this can just be switched on for GA. I would really like to see this being put into action now so that issues with rolling this out can be ironed out before GA. For starters I do not want to see the next tag on Github as 1.0.0-betaX as really there were so many backwards incompatible changes since 1.0.0-beta already.
Agreed. And I agree that we have to prove we can do a good job of stability. That is the plan of course. I expect it to be a little rocky then get better – only because that how things normally are after big changes!
Regarding developing an extension I believe the tech writers are working on this now. But basically “build a composer package zip” will be the crux of it.
The team is working hard to get the new Magento connect done at same time as m2 GA (at same time as Company split!). Things a bit tight.
I knocked up a quick post https://alankent.me/2015/10/13/magento-2-extension-experience-quick-note/ to share some information. More is coming, but hopefully you can see at least it is 100% composer based, and its not that complicated.
I am excited to see these changes being put in place. I think being able to depend on specific versions of internal modules is going to be great and I have noticed that many of the public APIs are already in place. There are a few I have not seen available yet (like the ability to get the entity types) but overall, I have been pretty happy with it so far. I think it is going to be a little bit to get used to using the public APIs but having a documented list of them could go a really long time. I plan to post in the devdocs repo about this too.
One can upgrade magento 2 using the two basic methods, one is from admin panel and other is using composer.
One can upgrade magento 2 using the two basic methods, one is from admin panel and other is using composer.