Switching Distros on a Running Site

As Mårten explained in an earlier post we decided to redo our installation profile into a more diversified model with a common core set of modules and with a differentiated set of modules for every site that will use the common core.

Splitting the previous Yle – profile that we used as a base for our installation from the beginning was easy enough. Just create two new installation profiles – Syndprofile and Fyndprofile that used most parts of the earlier Yleprofile and the parts that were specific for the two installations into their own sets of repositories, ie synd_modules and fynd_modules and so on. Starting development on the fynd-platform was also very successful and has since launced two successful projects: Kuningaskuluttaja and MOT.

However, on the svenska.yle.fi – platform, the change posed a lot of challenges to overcome, since we had to change installation profiles on a running site. And since a lot of the path settings to modules and themes are set in the database at installation time a lot of file paths would have to be changed. So when we first tried the easiest approach, just replacing the profiles/yleprofile installation folder with a profile/syndprofile we were left with a severely broken site, that couldn’t find any modules or themes, no matter how many times you tried to clear the cache. So it became clear that we had to do the changes directly into the production database.

So what we did was to dump the database into an sql file and doing a search and replace on every occurence of yleprofile to the new synprofile. We opted to slightly change the name to synprofile since there are a lot of serialized data that contains the path in the database as well.

We have around 500 000 nodes in our database at the moment, and a pretty large index, but using sed to do the search and replace the operation only took a few minutes to do, even though the sql file itself is getting close to 5 Gigabytes in size.

$ sed ‘s/yleprofile/synprofile/g’ < svenskaylefi.sql > svenskaylefi_synprofile.sql

This, however, was not enough in our case. We also had a couple of modules and our own theme settings and features that was not going to be used in the other distributions so we had to manually change the location of these. Fortunately most of the file path settings are stored in just a few tables.

system
registry and
registry_file

The paths are also stored in the cache tables so it is advisable to truncate them at the same time, especially

cache
cache_bootstrap and
cache_path

It took a few tries to get every step in this workflow to work out without problems, but when we finally figured out how to do it the migration process went pretty smoothly. Just one final hurdle gave us a bit of cold sweat when the site didn’t even bootstrap even though the migration process had otherwise gone smoothly. But, a manual flush of the memcache server solved that too.

 

The making of a Yle Drupal Distro (YDD)

From the start of the project (Swedish Yle’s New Drupal (SYND)) we had a goal to build it as a distro. What is better than that someone else can use the same code base? 🙂 With YDD we reached the internal distro milestone, an important step if we also some day want to be able to make it a public distro.

During January we spent two weeks making our own code more abstract (for some reason there is always something that gets hard coded), making new features that were wanted for the PoC (Proof of concept) of FYND (Finnish Yle’s New Drupal), setting up a new dev environment and splitting up the install profiles to support YDD, SYND and FYND.

The structure we decided to use is as follows: YDD contains the basic functionality that all sites should have. SYND and FYND each contain their separate modifications. Example; recipes, maktbasen and some Svenska Yle specific styling is found only in SYND. The themes in fynd_themes and synd_themes are subthemes of the theme in yle_themes.

The GIT structure and install profile logic:

FYND

YDD

SYND
fynd_modules
fynd_features
fynd_themes
fynd_profile
yle_modules
yle_features
yle_themes
synd_modules
synd_features
synd_themes
synd_profile

With this structure we are able to use a common core, and if needed add more functionality, translations or styling per site. Issues we ran into include views that were created when the system was in Swedish. It does not work to then switch the system to English before exporting, the view will be in Swedish.

Another issue was that a view that limited content based on a taxonomy was exported with the VID, and not the machine name of the taxonomy. Solved by changing the view so that the machine name was used in the filter.

We also had some displacements of fields and taxonomies. They had been placed in the wrong feature, so they were not available when some features needed by SYND were not enabled in FYND.

Moving theming to the features and modules was also something that we now needed to do. Not all the views that had styling was going to be used in both SYND and FYND.

The RSS feed generated by Views is one open question. It would be good to share them between the install profiles, but settings like Site name and headlines are hard coded in the view.

Being more thorough in with what is placed where and making sure that the language always was English would have made the process even smoother.

One thing we should have done in the early phase is to change the name on the live production install profile. That we tried to do it at the same time as we had moved some of the modules from yle_modules to synd_modules caused problems when migrating the production database.

The process itself turned out to be a good way to also do a review of the code. Even though we have had two external reviews (thanks Bala, @dasphere) it sometimes takes deep knowledge of a site to realize that something has unnecessary overhead. Having to split the site forces you to look at parts of the code that otherwise does not get looked at that often. We ended up merging and/or removing four features/modules.

Working with Jari Lana who does Drupal for Ylex, Oppiminen & co was also a good way to get a fourth review, from someone who also could make improvements directly in the code.

Even if FYND is never launched, the work will not go to waste. SYND has reached a new abstraction level, got new features and improvements. If the need arises, a new site can be up and running in no time.

Currently svenska.yle.fi is running on the new YDD + SYND setup.