A few people asked me recently on progress on “development environments”, which I am currently looking into. This is just a quick update because tweets are too short – especially for me!! There is not much new content here over previous posts.
Let’s first talk about the typical phases in development – the Software Development Lifecycle.
You start a site from the core Magento code base. Not many use the default set of code – you beef up your site via customizations (e.g. designing a local theme), 3rd party extensions, and more. This is what most people call “development” – doing whatever is needed to the code base to create the code base for the site, which is the output of this phase.
For advanced processes, a build phase may occur next. This is where the various assets required to construct a site are collected to create a build artifact. This might just be a copy of the source code for the site, but for a Docker based project it could be the Docker image.
Magento 2 is encouraging an increased usage of test automation. Some tests may be run before the build phase (e.g. typically unit tests are run first); other tests may be run on the build artifact itself, to ensure the build artifact itself is of good quality.
Finally you deploy the build artifact into production, upgrade the database schema, and turn on the upgraded code base.
Magento is used by a wide range of users, from DIY merchants up to the largest system integrators. As such, there is not a single approach that fits all. However there does seem to be a degree of commonality.
- You develop the source code.
- You have the output artifact of the first stage.
- You deploy the artifact.
(Wow Alan! That is just soooo deeeep!)
What I mean by the above is there are several different strategies for developing code. Magento may recommend one approach, but there is no reason to exclude the other approaches. There is also then the “how do I get the code into production”. Some will have a simple approach (zip, ftp) and others will have a sophisticated CICD test and build pipeline, with automated rolling cluster deployments.
There can be mix and match – multiple approaches can be used to develop the site code; when done you pick one approach to get the code into production.
Let’s look at a few of the options here in greater detail.
Let’s start at the middle of the above three. There are several approaches for the code artifact:
- You can make changes live in production. Add a theme, adjust a CSS file, make a configuration file change. Making changes live has higher risk, but very low overheads. This approach is more attractive for sites where the cost of things going wrong is not very high. So the artifact is just the code on production.
- Alternatively, the source code could be bundled up in a ZIP file or similar, to make moving from a development laptop to a production server easier.
- Another common approach is to put the source code into a source code control system, such as Git or SVN.
The deploy-to-production phase has a few options:
- Magento Cloud! (Well, I had to get that in here somewhere don’t I?) You commit the code to Git which triggers an automatic build of an image that is then deployed into an environment based on the Git branch name.
- Advanced solution partners would weave in test automation runner frameworks, like Jenkins or Travis.
- A Docker based deployment strategy would build an image (or images), test them, and roll out into production. The important aspect here is the same Docker image is both tested and deployed.
- Depending again on SDLC maturity levels, the new deployment may be first rolled out to a test or staging environment before the rollout to production. This step is important for sites with low risk tolerance. You want to try the deployment (including code and database upgrades) before attempting them in production, to minimize risk.
Backing up to the first phase again, the development phase.
- For simple sites where the changes are made directly in production, there is no real need for a separate development environment. You just use production as your development environment. Spin up a GoDaddy or similar instance, add a few extensions directly to the live store, do some simple local edits, maybe create a local theme, but no need for Git or similar to manage source code. Version control is to take a backup of the source code periodically, or copy files to “.bak” before you start editing.
If you want to develop a site on your laptop, then push to production hardware later, there are a few options:
- Go native. Get all the required software installed directly on your laptop. Great performance, great control, quite a bit of software to install to make sure you have the right versions of everything.
- Be Vagrant. Be able to move the same set of tools to different environments (nowhere is truly your real home). The benefit of Vagrant is one person picks the set of versions of tools to use, then everyone else can share the setup. Great for standardization, and built in support in PHP Storm (common PHP IDE).
- Mimic Vagrant using Docker. Vagrant creates one VM with everything installed, you can do the same thing with Docker. (Many of my recent personal projects have been doing this.)
- Real Docker. The idea behind Docker is to have standard, small, containers that you can mix and match. For example, you can have an NginX or Apache container, and use the one you want. Someone works out a good Redis configuration – everyone else then just uses it. The theory is also you can use the same image in development as production, but a “good” image for production locks down the image and has it rather skinny; this is not good for development as it means less tools are installed. I am a little less convinced in this approach as a “good” approach for being developer friendly for the masses – I found it rather complicated with less good developer tools being available. But there is the definite advantage that getting from dev to prod is typically smoother (you can use the same image on dev and prod, just with different resources allocated).
- In the cloud. There are cloud based development environments, such as CloudAnywhere I have mentioned in other posts. Avoids the need for local setup.
So what have I been up to, the real topic for this quick note. I have been trying several of the above approaches, with current thoughts and feelings.
- Native – too painful for the masses.
- Vagrant – I think a sensible default, just want to find a good box to use.
- Docker mimicking Vagrant – has some pros and cons compared to Vagrant. I think in a few months’ time with the release of the next version of Docker, Docker is going to be able to do what Vagrant can but faster. There is more momentum in Docker from what I see.
- Docker with many containers – seems great for production, but overly complex for development if you don’t need the “I want to use production images on my development box, for extra testing”.
I have been building up a few options to get firsthand experience. Vagrant has work by Joshua Warren as well as the Magento team Vagrant box; I have built a few mimic-Vagrant-using-Docker images (I like them personally); then there are a few multiple container Docker based Magento deployments out there.
The problem I keep coming back to is file system access. I want access to files from PHP Storm running natively on my laptop. I want easy to use, fast, with Grunt/Gulp support so saves to disk of Less files are detected automatically to trigger CSS compilation with auto-reloading in the web browser. The next Docker release should make this very slick on Mac’s. Vagrant I have been using rsync-auto so I save on my Windows laptop and it auto-pushes into the container. On Docker I either use an rsync like approach, or use Samba so I can mount the drive directly and see all generated files.
So what is next? I am interested to collect a bit more feedback from developers. I also want to build a real (mini) project to experience the tools first hand. I think Vagrant has mindshare today, but Docker will catch up quickly once the next release is out. I want to play a bit more to really use the different tools. For example, with Docker I find the network goes crazy every so often requiring a reboot; with Vagrant I find aborting a “vagrant rsync-auto” command sometimes leaves it running in the background. None of the tools are prefect.
So no answer from me yet – but probably should have one before the end of May. This post was just a brain dump / update. Separating the development environment from the source code artifact does have the advantage that swapping later is not difficult. Right now I am leaning Vagrant because of the greater share today. But really it does not matter that much if the Git repo structure can be standardized, because the development environment is unrelated to the deployment pipeline.
Thanks for pointing out the different approaches. We’re fiddeling around with more or less all of them to find out the best deployment strategy.
Most of our developers go for the native way. We rarely had problems with the dev environment beeing different to the production. So sometime I just don’t get why everyone is praising docker for creating the same env on all devices, if there isn’t really a problem. I find myself often developing with a simple MAMP installed on my mac. PHPStorm is running the grunt tasks on startup, so I can dive into coding without any hassle.
The main issue I see is in the deployment itself. From a high level view we’re trying the following right now:
– Magento is in a git repo. Only composer.json, composer.lock and individual stuff.
– The production env is running on docker containers.
– Jenkins fetches git updates and rebuilds the PHP-FPM Container with the code updates.
– The newly built container is pushed to the docker repo and the production env is updated.
Where I see the biggest problems is within the building process of Magento. In a perfect world I would like to do these steps in the build process:
1. fetch git repo
2. composer install
3. setting file permission
4. set deploy mode to production, including compiling and generating static content
5. building the docker image
It really doesn’t matter if you go with a container structure or if you deploy the artifacts by capistrano, FTP or whatever. But the fourth step in the building process leads to bigger problems. First a database is needed and Magento has to be installed on the build server to be able to compile the code. On another post you said, that this will be handled soon. Yeah!
If you did the composer install with the no-dev option, it will not be possible to compile Magento at all, because of missing packages. So you have to put the dev resources in your live build…
After you pushed all the artifacts from the build-server to the live server, you will have to run another script to check whether magento is already installed. If not, you’ll have to install it and then have to rerun compilation etc. That makes no sense. So is there any way to use the build artifacts and just run a bin/magento setup:db-schema:upgrade ?
Coming to an end here, I just want to say that development environment and deployment are strongly related. Reading all the blogs about Magento and also the Magento docs, they only talk about setting up production once and leaving it like that. So I think that we should think and talk about that process more often from now on to find out an ideal deployment process.
There is a fifth way, that sit just after the native one: use something like Valet (https://laravel.com/docs/5.2/valet). It’s only for one platform (OSX) and rely on local deps (php and mysql) but installed with a specific package manager (Homebrew), so enough standardized that new “instances” can be spin up without headache and the configuration could be easly shared with teammates via source control.
I think that the Laravel team is doing a great job trying to make the development experience the more painless possible, with tools like Valet and Homestead (an “official” Vagrant box packed up with all development tools needed for a tipical Laravel project), and Magento should draw inspiration.
So far we are using Vagrant for development of out M1 and M2 project, but the performance are the biggest problem. We looking to Docker but really is too much complex and is risky to leave it alone with a developer without devops skills.
So I’m tempted to do a try with Valet, I’ll try to make it working with M2 in the next days.