Semantic Versioning and Patch 2.1.2

The recent Magento 2.1.2 patch did not strictly follow the semantic versioning rules for patches. In this case we made a judgment call that the benefit to customers based on feedback we received (especially as we approached holiday peak) outweighed the negatives. The change was new service contracts were added to formalize the API to existing internal functionality. Existing client code was not be impacted (they don’t have to use the new APIs), but true semantic versioning would defer making these APIs available to the next minor release (2.2).

I thought this blog post was a good chance to talk more about what the versioning rules are and explain why Magento knowingly decided to break the rules in this case.

Semantic Versioning

Semantic Versioning (SemVer) describes a way to use version numbers in a consistent and structured way. For example, when you see a version number of 100.1.0 for a Magento module, you know it means major version 100, minor version 1, patch level 0 (that is, not patched yet). Cross module dependencies list compatible version number ranges for that dependency, ensuring you don’t accidentally mix incompatible versions of modules.

A key purpose behind semantic versioning is to allow a patch to be released that can be included into projects with confidence of not breaking other code. For example, if an API has a method removed, that cannot be included in a patch as there may be other code calling that method. Removing the method would break the other code.

In SemVer, patches are not permitted to have a public API change. This allows cross module dependencies to specify a compatible version number ranges of “100.1.*” (meaning any patch level of version 100.1) or even “100.*” (meaning any minor or patch level of major version 100 of a module).

Product vs Module Version Numbers

Product version numbers (such as 2.1.2 for Community Edition) do not follow the SemVer rules, at least for the major and minor version number. Product version numbers are marketing driven. In Magento, SemVer is used at the module level where version numbers start from 100 (to avoid confusion with product version numbers). All cross module dependencies specify module version number ranges, not product version numbers. This is why it is not a problem for product version numbers to not follow SemVer.

In addition, product patches currently lock the major and minor version numbers of all modules in the product. That is, in patch 2.1.2 only the patch level (last digit) of module version numbers is permitted to change. This is the cause of the problem expanded upon below. This rule is going to be reviewed more closely, but for now only module patches can be released in product patches (not new major or minor versions of modules).

What Rule are we Breaking?

Formally, SemVer states that the minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. (See http://semver.org/#spec-item-7.) Why?

Put simply, SemVer says not to introduce new functionality (public APIs) into a patch. Patches are for bug fixes. An example of the complexity that can result from functionality being added to patches is dependency ranges need to specify the minimum patch level that functionality was introduced, rather than the preferred “any patch level of minor version X.Y”. SemVer aims to simplify dependency management by having simple rules.

So What Exactly Changed?

Patch 2.1.2 introduced a new interface to define a standard way to access existing functionality, specifically for capturing payments and shipping an order. (Other interfaces may come in the future for credit memos and emailing a paid invoice.) These were needed to support integrations with 3rd party shipping, ERP, and other solutions. Two new interfaces were exposed as web APIs via new service contracts in the Sales module. Because they have been created as new interfaces this won’t impact any existing customizations or extensions that call the current sales APIs.

The implication is sites or extension developers that would like to use this new functionality will need to update their code to utilize the new interface and specify the minimum patch level. For example, the sales module version number from 2.1.2 or later (not 2.1.0 or 2.1.1). To be precise, it should specify 100.1.2 or later as can be seen on GitHub for 2.1.2 compared to 100.1.1 from the 2.1.1 patch.

Change Details

The introduced APIs are as follows:

  • salesShipOrderV1 interface exposed as a POST web api /V1/order/{orderId}/ship
    • You can post with an empty post body and ship the entire order
    • You can do a partial shipment by specifying the order line item(s) that you’re shipping in the post body
    • If you’ve already done a partial shipment and call again with an empty post body Magento will ship the rest of the order
  • salesInvoiceOrderV1 interface exposed as a POST web api /V1/order/{orderId}/invoice
    • You can post with an empty post body and capture payment for the entire order
    • You can do a partial invoice by specifying the order line item(s) in the post body
    • If you’ve already partially invoiced an order and call again with an empty post body Magento will capture the rest of the order

Why Not Increment the Module Minor Number?

So since there are module version numbers already in Magento 2 (the numbers starting from 100 onwards), why not just increment the minor number for the sales module? Then reference that from the CE/EE patch?

There is a really deep and hard to explain reason why this is a pain in practice for Magento that I will skip for now. Instead I describe the problem it creates for extension developers. If the minor version number is incremented, any extensions that depend on the current minor version number (and excludes the next minor version) may break the next time you run composer. To release a new minor version of a module may require all extensions dependent on sales entities to update their composer dependencies as well. The coordination effort here is significant. That is, it’s not a technical reason why Magento is taking this path – it is a coordination effort.

So a part of the longer term solution is around improving practices (and clarity of those practices) by both Magento and extension developers in terms of version number management of modules. (Magento has a responsibility to clearly communicate the rules to extension developers, and then help them follow the rules.) For example, if extensions who only call the Sales APIs consistently used dependency of 100.* (minor number not specified), it would not have been a problem releasing a new minor version number. Unfortunately, our investigation shows this is not the case today – different extensions use different forms of dependencies without much consistency.

Conclusion

It is understood that adding new functionality (exposed as web APIs) in a patch release is not in strict keeping with semantic versioning.  However, we believe it’s the right thing to do to address the needs of our customers and partners in the most expedient way possible. We have received compelling feedback that the missing functionality is worth adding to 2.1 and not waiting for 2.2 to be released. The change should have no impact on clients of the Sales API as they are completely new interfaces. But it was an interesting example of the trickiness of version management.

Researching into what existing extensions are specifying in terms of dependencies also revealed that Magento needs to put more investment into educating extension developers on how to do Composer based dependencies correctly. Increasing the minor version number, following SemVer strictly, may have been possible in the 2.1.2 patch if this was the case. (Having Magento Marketplace where Magento hosts the code of 3rd party extensions is proving very useful for performing this sort of research – Magento has better insight into the impact of changes across all extension developers.)

Side note: I originally posted a draft of this post publicly by mistake, which some people saw. This is now public and the final form. Sorry for any confusion!

6 comments

  1. Forgive me, but if the marketing version doesn’t follow SemVer, I’m not sure I get the point of claiming SemVer at all. The individual modules of Magento don’t ship individually. If they did, then you could claim SemVer (but it would also make a marketing version irrelevant). They only ship as part of a marketing release. In my opinion, you should have the marketing version follow SemVer strictly. If marketing isn’t willing to yield to that, then product should.

    1. The modules are actually shipped separately via Composer (and Extensions depend on modules, not on CE as a whole), but we found it so confusing to manage due to the number of modules we bound it all together with a metapackage. We have kicked around the idea of putting each module in its own git repo, with its own version history, which would solve some problems. But we have not made the decision to do this yet. (Another idea was to have groups of modules – e.g. all sales related modules – versioned together. That gets you to a better middle ground.)

      1. What I meant is that you would never publish a module independently of a marketing release (eg, you would never ship 100.1.2 on its own. It would *always* coincide with a bump to the marketing version). This process makes it so the only version that matters (again, IMO) is the marketing version.

      2. Extension developers would not need to release per CE minor release if we versions modules independently. So I think your statement is true for SIs doing a site, but not true for extension developers.

  2. TexanHogman · · Reply

    “Formally, SemVer states that the minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards incompatible functionality is introduced to the public API. ( http://semver.org/#spec-item-7.) Why?”

    Didn’t get through the entire entry but saw that the above should say ” ….backwards compatible….”.

    1. Ha! An internal reviewer said I had it backwards so I changed it without thinking it though. Had it right the first time!😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: