The Magento team has been looking at the best way to build and then update sites for Magento 2. The team has committed to using Composer where each module, theme, and language pack will be a Composer package. There were and are some interesting challenges. I thought others might find these interesting as well.

Now I sort of expect someone to jump up and say “didn’t you look at the installation process for product X?” (WordPress being the most likely.) Yes, the team looked around as well. I thought the following discussion was still interesting enough to be worth a blog post and it’s simpler here to not explain what other projects do here to keep the length down.

Disclaimer: These opinions are my own and not necessarily that of my employer. This blog post is also personal observations, not a commitment of strategy of how the final installation process will be done. However, opinions (with reasoning why) always appreciated!

Approach 1 – Build Site with Composer

You can build a full site using Composer alone today. The Magento team leveraged the work of a previous community hackathon project using Composer to build Magento 1.X sites. (Believe it or not, the core team does try to learn from what the community tells us!)

It is worthwhile remembering however that the hackathon project was to get Composer to work on the assumption they could not change the Magento directory structure. The core Magento 2 team does not have this same restriction (other than the effort required to change the directory structure around).  For simplicity in this post, I am ignoring any potential directory structure changes that could simplify the installation problem. The problems I describe here would be reduced, not eliminated with such changes.

Based on the hackathon project using Composer and Magento specific Composer plugins, it is possible to download a series of modules, themes, and language packs and have them installed into the correct directories as required by Magento. Importantly, there is one additional package with Magento 2 which I will refer to as the “skeleton” which creates the overall directory structure. (I am purposely leaving out details here. For example, it might really be a few packages, but that does not matter for the purpose of this blog post.)  So Magento is kind of interesting in that it installs a first “skeleton” directory structure package, then adds modules, themes, and language packs under this directory tree. (Composer normally installs different packages alongside each other under the vendor directory.)

Why is this interesting? Well, it creates interesting situations when you need to update the skeleton Composer package. Normally composer when installing a package removes the old directory tree, then extracts the new package. This is robust because you start from a clean slate each time. The skeleton package is more interesting however as it has other packages sitting within its directory tree. So you cannot just delete the whole tree and start again.

But we believe we have this working with Magento 2 and Composer today. It might not support moving the directory where packages are installed (e.g. if we wanted to rename app/code to app/modules in the future), but it basically works.

Challenge – Merging in Local Changes

A design goal for Magento 2 is to allow all customizations by adding additional packages. You should not directly modify core Magento files or those of extension developers. You should layer your customizations on top using new modules or themes you define and add to the project.

However, while things are better in Magento 2, it is not yet clear this can be relied on 100%. There may be changes to .htaccess files required due to the version of Apache being run, or a hot fix you must get going today to get a customer site running. So while absolutely discouraged, it is advisable to have a development practice that allows such “hacks” to be made. (I will use the word “hack” here to distinguish it from an officially supported override using a module or theme, even though the term “hack” is probably a bit too negative. For example, a hot fix “hack” may be important and necessary.)

So what is the problem?

The simplest (and possibly worst) approach is to build your site with Composer and start changing the files directly. The problem with this is the next time you do a Composer update command to get say a patch for a module, it will discard the changes you made and it will be hard for you to notice. Changes to the skeleton files (non-module and theme files) are particularly tricky here.

Approach 2 – Build Vanilla Site with Composer, and then Apply Local Hacks

A common solution is to have a source set of files that are pristine (untouched), and then apply your local hacks to a copy of those files to build the final site. For simple cases there may never be a hack – it may be all customizable by adding new modules, themes, and language packs.  (Remember, that is the ultimate intended goal.)  But if hacks are needed, then make sure they are applied to the clean set of files to build your final site.

So how to keep the local hacks correct when the Composer update has changed files you have modified? Put simply, there is no way to avoid this being the responsibility of the person making the local hacks. By definition, it is doing things not intended to be supported. All that can be done is to try and make the hacks minimal, and easy to identify and review for correctness the next time you do a Composer update. Developers should also take it as an encouragement to avoid doing hacks and try to make any changes made via a supported mechanism such as defining a module that customizes other modules using supported technologies. It may look a bit more work the first time, but there will be less upgrade work later when you want to fetch that next available patch via Composer.

Approach 2a – Build Site with a Script

One approach to applying local hacks is to keep a directory that is built purely using Composer and downloaded packages. Then a script is used to construct the “real” web server directory from the clean installation, making any desired changes. To be clear, this script might be as simple as copying the original directory tree then copying another set of locally modified files over the top.  (Can anyone say “modman” by the way?)

Note there are a few interesting variations possible here, such as whether to copy files into the final directory tree or use symlinks. Should Magento 2 ‘generated’ code be merged into the main PHP directories and possibly speed up class loading. Also the final directory tree can be quite different to the original if desired, based on your preferences. For example, production may have all the test cases removed, or you might include either the Apache configuration files or Nginx configuration files, or you might put all the web served files under a ‘pub’ directory to have a cleaner final install tree.

Note that a potential side benefit of this approach is the need for Composer plugins goes away. Composer is now only needed to download a compatible set of packages. All the packages (including the skeleton package) can just sit under the ‘vendor’ directory. A separate script is responsible for assembling the real web server directory from the locally available packages. (A script would be supplied with Magento to do this for the standard files. Only local hacks would require an additional script.)

Approach 2b – Use Git Branching and Merging

Another approach to avoid writing a shell script is to take advantage of Git branching and merging. Create a first branch in which all you ever do is Composer update commands. No other local changes are made here. Then create a localized branch from this “vanilla” branch and make your hacks there. Whenever the vanilla branch is updated, merge it back into your customized branch taking advantage of Git’s merging capabilities. When you release to site, you can tag your customized branch so you know exactly what you released. If the vanilla and customized branches have only had minor changes, the merges will be easy; if more extensive changes, merging will be harder.

(I am sure the Git experts can tell me 10 better branching strategies to manage the code base – my point really is to say you can use Git merging to apply local changes instead of writing a shell script to build real directories on disk.)

Approach 3 – Make Magento more Flexible

I feel I should mention the ultimate solution to me is to make Magento more flexible so different parts of the system are less dependent on a particular directory structure.  That would mean the master GitHub repository and a local site would not have to be the same directory structure.  This to me is a longer term goal where incremental progress based on community feedback may be beneficial (possibly up to GA after which we will lock down changes). I can imagine lots of debates about what the ideal directory structure should look like. For example, should there be a ‘pub’ directory, or should index.php be in the root?  There are lots of different scenarios with different requirements (e.g. low end hosting may require an index.php in the root directory.)

Changing the directory structure more radically may open up other approaches to be more Composer friendly, reducing the merging problems described above.

Conclusion

So which approach is best, particularly between 2a and 2b? Actually, I wrote this in part to see what other people think!

• I can see the benefits of having a script apply local hacks in that there is a lot more flexibility if you have preferences around the final directory structure.
• I can see benefits in using Git in that you can use its file merge capabilities when dealing with files that have changed due to composer updates and that were locally hacked.

It is not yet clear if Magento needs to add explicit support for one scheme or the other. To me it is more interesting as approach 2a could mean dropping Composer plugins and might be more robust when updating the skeleton package. But it puts more work on developers – they have to write scripts to apply hacks instead of just using Git.

The ultimate success will be when localized changes are not required. But until then, any thoughts on 2a versus 2b welcome!