In this post I review the typical stages in a Magento 2 tool chain, from development to production. This tool chain is for sites using the Magento “development” and “production” modes, not the simpler “default” mode where you tend to operate directly on the site.

There are variations, but I thought a useful to put all the components onto a single diagram. And if I am missing something, please let me know.

Please note this one is a fairly light weight post. If you understand the following diagram, you probably won’t get much more out of reading the rest of this post.

Project Source Files

While developing a project, I recommend having the source files on your development machine (your laptop or desktop). Some manage their project source files inside a VM or container. I recommend using the native file system directly because it gives better performance. For example, if using PHP Storm and it performs a “reindex”, slower access to the files is noticeable. It also more naturally fits in with native operating system tools.

Version Control System

I recommend using a Version Control System (VCS) such as GIT, Mercurial, or SVN. It is not mandatory, but there are lots of advantages including tracking the history of your project and making collaboration easier between developers.

All project files for Magento sit within a single directory tree. Developers work on and eventually deploy projects. For Magento 2, there are recommendations around what to versus not to commit into the VCS, such as “do not commit the vendor directory”. The vendor directory holds the code that is not specific to your project. Different projects may have some variations, but here is an example that I find a good starting point. But there are other files that are not typically committed including minified CSS files created during the deployment process, and the media directory holding images for test products you created during development. I recommend starting from a “ignore” list from another similar project, and then adapt as you go along.

I normally recommend running the VCS tools natively on your development machine as they typically integrate closely with your IDE or text editor. For example, for Windows “Git Bash” provides a convenient bash shell emulator and Git commands.

Magento Cloud (the Magento hosting solution) uses a variation on this theme, where committing to the VCS is how you perform deployments. Commit your code to the production branch and your code will be deployed automatically for you.

IDE / Text Editor

To edit the source code of Magento you will need an Integrated Development Environment (IDE) or text editor. PHP Storm (a commercial product) is a commonly used IDE with Magento, but other options exist. An IDE differs from a plain text editor in that it has more built in tools that developers commonly use. For example, PHP Storm you can commit changes to your VCS, review code history, run unit tests, refactor code, all from inside the IDE. However, there are users who prefer using simpler text editors. This is purely a matter of choice. For example, I personally frequently use vim from a command prompt for quick edits. And the power of PHP Storm may be overkill if you only edit CSS files.

Magento Marketplace Repository

Magento Marketplace allows you to purchase third party extensions to add to your site. Magento 2 uses Composer to download the code into the project’s “vendor” directory. The same Composer repository also holds all of the Magento CE, EE, and ECE code bases. So this is a useful single location to download all externally developed code.

Private Composer Repository

Modules developed specifically for a project are typically stored under app/code. However, it is also common for a solution partner or merchant with multiple projects to create an internal repository of Composer packages to share across different projects. This “repository” could be as simple as a GIT repo per module, or a full blown Composer repository built with a tool such as Satis. Composer supports multiple ways to create shared resources that Composer can download and add to a project.

PHP Development Environment

A PHP development environment (at least for the purpose of this blog post) consists of a set of tools allowing PHP to be run from the command line or from inside an IDE. It is sufficient in order to run unit tests, but not the full application. For example, it does not include MySQL or the web server. A developer may have a full suite of tools running natively on their laptop, but it is a common modern trend to run the full stack of tools within a VM or container. The PHP development environment available locally is “just enough” to run unit tests.

Note that PHP Storm supports running unit tests remotely (over SSH) via configuration of a “remote interpreter”. This means if the developer is using a VM or container to run the full stack in, the VM/container can also be used to run unit tests, avoiding the need to install a local PHP development environment. This also ensures consistency of environment configurations between developers (they would typically share the VM or container image). This results in one fewer tools to be installed locally, but more importantly consistency across developers.

Full Stack Development Environment

A full stack development environment includes all the tools required to run the Magento 2 application. This includes PHP, the web server (Apache or NginX), MySQL, and so on. It can include Redis, Varnish, RabbitMQ, and more, based on how the developer wants to work. As above, a common trend is to use a VM or container (Vagrant and Docker are two common technologies here) where all the tools are preinstalled. This allows developers on a project to share exactly the same tools and configurations, avoiding time consuming problems where one developer has problem that others do not encounter, due to some custom PHP configuration setting (e.g. a smaller memory limit).

Use of VMs and containers also allow developers more easily switch between different setups suitable for different customer projects. For example, a Magento 1 project may use an older version of PHP. Installing PHP (or MySQL, Redis, Apache, etc) natively can make it harder for a user to switch between projects.

It is worth noting that I do not call out a separate development environment for front end developers. This is because while they may use different tools such as Grunt or Gulp for CSS compilation, frontend developers need most of a full stack to test their changes and so they may as well just use the full stack development environment.

Integration, Staging, and Production Environments

Integration environments in the context of this blog post refers to environments used to test the complete set of code and functionality, such as for for a user acceptance testing. A staging environment is often set up to mimic production as closely as possible (frequently with less hardware to reduce costs) allowing new code drops to be tested before being rolled out into production. By keeping the staging environments as close as possible to production, the risk of production deployment issues is minimized. The production environment is where the live, transacting site runs.

Tools

There are also a number of tools that are worth calling out specifically, related to the blue arrows on the diagram: direct access, Composer, file syncing, and deployment scripts.

Direct access as shown on the diagram simply means the tools can access the project files natively on the local file system.

Composer is a PHP package manager, providing a standard way to download third party code, and performing version compatibility checks.

File system syncing tools are for sharing the project files stored natively on the development machine with the VM or container. Vagrant includes an “rsync-auto” command that watches the local file system for changes, using rsync to then synchronize the local and VM/container directory tree. (See this Vagrantfile for a Magento 2 example.) Docker-sync provides a similar mechanism for use with Docker. Docker 1.12 also provides a file system sharing capability, although there are still teething problems around performance and lack of iNotify support for Windows.

It is worth noting that this is a bi-directional syncing of files. Some changes should be pushed from the project files into the VM/container, such local modules or themes. Other changes are pulled from the VM/container back to the local file system for access by the IDE for debugging (single stepping through code in an IDE requires the IDE to have access to any code generated in the VM/container).

Deployment scripts are also needed to take the set of files within the project, run compilation steps on the project to generate optimized files, and then deploy the combination of local project source files, third party files, generated files from compilation and have them all deployed to an integration or production environment. “Static asset compilation” and “di-compilation” fall into the deployment process.

I often break out “build” as a subset of the deployment process, which is all the activities that can occur without the site being unavailable. There has been a lot of recent effort within Magento to speed up the deployment steps and make them possible to execute on a server without database access (to make them easier to run in a deployment pipeline). Some work has already been released in patches, with more planned for future 2.1 patches and the next 2.2 release. The only reason a site should stay down for a noticeable period during a standard deployment process is to perform any database schema updates.

Conclusion

The purpose of this blog was to provide an overview of the various steps within the Magento tool chain, particularly for new developers. A number of full-stack environments have been created for developing with Magento (I have blogged on a number of these in the past). The pattern of usage is however basically the same. I have also previously blogged on the separation of development environments from deployment processes.

Magento is exploring a “standard” Docker based full-stack development image, as a baseline reference image. I have been saying “coming soon” for a little while now – sorry, we hit some unexpected issues with the Docker 1.12 host OS file sharing feature in the areas of speed for MacOS and iNotify for Windows. But it is coming.

It is possible that more than one such full-stack image will ultimately be supported (e.g. one using Vagrant and one using Docker). This is still to be decided. It is worth noting the official Magento image will not necessarily be “better” than other images already available in the community, just “standard”. For example, if a problem can be easily repeated on the Magento standard image it will make it easier for Magento to repeat reported problems internally without environmental differences. This implies there is no reason not to find an existing container for a development purposes.

It is worth noting that the image will not address deployment. Deployment is specific to your hosting configuration. One goal of sharing the above diagram was to help with any discussions around the stages of Magento 2 software development overall.