Address feedback on the "Upgrading from Drupal 7" documentation

[klonos]

Current status:

Three pages as separated out from the original upgrade instructions:

• Planning
• Preparation
• Upgrade

Three pages based on @BWPanda's blog post:

• Planning
• Preparation
• Upgrade

---

Original issue:

https://docs.backdropcms.org/documentation/upgrading-from-drupal-7-steps

Feedback in Zulip:

@bobchristenson:

I'm attempting my first D7 -> Backdrop conversion now and following the instructions here: https://docs.backdropcms.org/documentation/upgrading-from-drupal-7-steps and, even as a very experienced Drupal dev I'm unclear on the instructions provided. A few notes that are causing a little confusion:

1. During "prepare your Drupal site for upgrade" (step 2) I'm unclear as to which modules I should disable at htis point, which ones should be completely uninstalled. My experience tells me that disabling anything isn't going to cause problems, but what about uninstalling? Not exactly clear on what to leave enabled vs. disable vs uninstall.
2. During step 3 "Upgrade your drupal site to backdrop cms" It starts by sayign "Follow steps 1 and 2" from the install page. Step 1 is basically download the files. STep 2 says "enter the mysql database information". But enter it where? Does this just mean create a new database and have (and empty database) ready for step 3? Or am I actually putting in db user/password info somewhere?

Thats confusing because step 3 is to run the installer....and that is where you actually enter database credentials...but I'm not supposed to run step 3 according to the upgrade instructions. So...where do I enter database details in step 2? Not sure if start the upgrade with an empty database? Or I run the installer and have a Backdrop database without certain info entered? Again, even as an experienced dev, I'm a bit confused.
(and this is basically resolved in a later step when you import your d7 database into an empty backdrop database...but its still confusing up to that point)

@BWPanda

Not sure if this'll help or add more confusion, but here're the instructions I've written up and use for myself: https://packweb.io/blog/migrating-drupal-7-backdrop-cms

@bobchristenson

The "migrating..." blog post from @BWPanda is really good...much better than the current official documentation. The most important part, IMO, is the breakdown about status of modules from D7->Backdrop and then, crucially, what to uninstall, disable, remove, etc. I'd suggest somehow moving some of this info (or this whole process) over to the official documentation. I think its more helpful in the 'real world' than the current docs.

[klonos]

I have been tripped by this when attempting a test upgrade (see discussion in Zulip for details/backstory: https://backdrop.zulipchat.com/#narrow/stream/218635-Backdrop/topic/D7.20upgrades/near/233986096)

I started with a blank Backdrop setup, but without actually running the installer. So this bit in the documentation:

Start by following steps 1 & 2 from the Installing Backdrop page.

I translated as:

1. Download the Backdrop codebase
2. Adjust settings.php (add db credentials and set $settings['update_free_access'] = TRUE;)
3. Import the D7 database export.
4. Run update.php

After troubleshooting this, and discussing it in Zulip, @BWPanda provided the key bit I was missing:

Upgrade notes say to setup new Backdrop site first, then import D7 DB & files. So that way the config would be populated with default values. Are you not doing that?

🤦🏼

[klonos]

Here's what I've done to improve things:

1. I've updated the change record in https://docs.backdropcms.org/change-records/variable_get-variable_set-and-variable_del-deprecated-by-cmi to explain what CMI is and differences between D7/Backdrop.
2. I've changed Start by following steps 1 & 2 from the Installing Backdrop page. to this instead:

1. You will need to setup a new Backdrop site first, to ensure that the configuration is populated with default values (see the Installing Backdrop page for detailed steps):
   1. Download Backdrop and extract the files
   2. Enter the MySQL database information in settings.php
   3. Run the installer

[klonos]

The main sections in https://docs.backdropcms.org/documentation/upgrading-from-drupal-7-steps are currently as follows:

• Step 1: Research what will be necessary to upgrade to Backdrop CMS
• Step 2: Prepare your Drupal site for upgrade
• Step 3: Upgrade your Drupal site to Backdrop CMS

To me, it seems that the first point in step 3 does not belong, and should be separate. So it should be:

• Step 1: Research what will be necessary to upgrade to Backdrop CMS
• Step 2: Prepare your Drupal site for upgrade
• Step 3: Install a fresh Backdrop site
• Step 4: Upgrade your Drupal site to Backdrop CMS

@BWPanda and @bobchristenson what do you think? ... @jenlampton? ... @stpaultim? ...@olafgrabienski? ...others?

[ghost]

Sounds good

[jenlampton]

Step 3: Install a fresh Backdrop site

No install is necessary if you are upgrading... Maybe my objection is to the word "Install", since that means running the install script (and populating a database) and none of that happens in an upgrade. Maybe "Set up" Is better?

[klonos]

@jenlampton but that specific documentation states this (in step 3):

We recommend setting up a second site, so that the Drupal version and the Backdrop version can be compared side-by-side. The instructions below follow that process.

[klonos]

...and none of that happens in an upgrade.

Umm. It obviously needs that though. @jenlampton please see the discussion in https://backdrop.zulipchat.com/#narrow/stream/218635-Backdrop/topic/D7.20upgrades/near/233986096 where I was running into weird errors, until @BWPanda mentioned:

Upgrade notes say to setup new Backdrop site first, then import D7 DB & files. So that way the config would be populated with default values. Are you not doing that?

We need to agree on which of the two is valid, and then fix our documentation and installer/upgrade logic 🤔

[jenlampton]

Upgrading from Drupal to Backdrop is exactly the same as upgrading from Drupal 5 to 6, and Drupal 6 to 7. In fact, it's the exact same process as updating Backdrop core now. Delete the old code, add the new code, and run the update script. A fresh install of the destination version is never necessary.

We recommend setting up a second site, so that the Drupal version and the Backdrop version can be compared side-by-side. The instructions below follow that process.

Yes, this is correct.

When doing a Backdrop core update, you usually do that on the same site -- so you don't have two copies.

When upgrading from Drupal to Backdrop, the "Extra step" is to make a second copy of the Drupal 7 site, so you can refer to that later for comparison.

Umm. It obviously needs that though. Please see the discussion in backdrop.zulipchat.com/#narrow/stream/218635-Backdrop/topic/D7.20upgrades/near/233986096 where I was running into weird errors

Ahh. I think we should address the weird errors. It's definitely not because you didn't have a fresh install of Backdrop before you started -- that would cause all kinds of other issues.

All the config files that are needed will be created when you run the update. The config needs to match the database exactly. There shouldn't be any config that wasn't generated from the upgrade, or it won't be an exact match.

[jenlampton]

I've updated the documentation on upgrading from Drupal 7 to more clearly indicate that installing is not needed in the "Set up" process.

1. I added an intro section at the top that reads "Upgrading from Drupal to Backdrop is exactly the same process as upgrading from Drupal 6 to Drupal 7. In short: Delete the old code, drop in the new code, and run the update script."

2. In the "set up" step I added "Do NOT install Backdrop." for those who skip the intro and jump straight to the steps :)

I believe I also addressed @bobchristenson's question #2, but for #1 I think we should add a more detailed page with information about each Backdrop core module that was removed so that people will have an easier time with this. I'm starting on that now.

[jenlampton]

Okay, here's a more detailed page for what to do if you are using Drupal 7 core modules that have been removed from Backdrop: https://docs.backdropcms.org/documentation/upgrading-drupal-7-modules-that-have-been-removed-from-core.

[jenlampton]

I've also moved the "Categorize D7 modules" section from @BWPanda's blog post to the "Research" page: https://docs.backdropcms.org/documentation/step-1-research.

I think that addresses all the requests in this issue?

[bugfolder]

In reading through the discussion above (before today's comments), the current Upgrading documentation, and BWPanda's checklist, I'd come to some suggestions about the structure of the book pages in that section and ideas for new pages. First, structure:

Old:

• Upgrading from Drupal [currently empty]
  • Introduction
    • Features added to core
    • Features removed from core
    • Top 100 Drupal 7 modules
  • Upgrading from D7 [currently empty]
    • Overview
    • Upgrade steps
    • Converting Drupal code
      • Converting modules
      • Converting themes

Suggested changes:

• Flatten by 1 level for greater visibility
• Eliminate empty pages (put useful information on all pages)
• Incorporate process from https://packweb.io/blog/migrating-drupal-7-backdrop-cms; merge with process on https://docs.backdropcms.org/documentation/upgrading-from-drupal-7-steps
• Add initial "Planning" step

New:

• Upgrading from Drupal
  • Overview
    • Features Added to Core
    • Features Removed from Core
    • Top 100 Drupal 7 Modules
  • Upgrade Steps
    • Planning [new]
    • Preparation [new]
    • Upgrade [new]
  • Converting Drupal Code
    • Converting Modules
    • Converting Themes

I envisioned the new Upgrade Steps page and its sub-pages as being a merge of the existing instructions, BWPanda's checklist, and some additional material. Turns out the "Planning" page seems to be roughly equivalent to the new Research page just added. (GMTA)

In addition to merging the existing instructions with BWPanda's, I've also added some bits that were important in my own experience. I"ve taken the liberty of creating pages for the three new pages on the docs site, but haven't put them in the menu (so haven't changed what people see as the current documentation).

Would folks take a look and let me know (a) if you agree with the overall menu reordering, (b) if you like the three new pages?

Planning
Preparation
Upgrade

If you like them, I can merge them with today's additions. If not, they're deletable.

[jenlampton]

@bugfolder I think we've been doing the same thing this morning. I've already flattened the Upgrading section one level. I think I prefer your structure though, with "converting" moving one more step closer to the top.

I've also separated the "Upgrade steps" page into three separate pages (Research / Prep Drupal site / Upgrade) but again, I prefer your terminology :)

I'm not done reading through your three new pages, but I'm instantly nervous about the "Upgrade" page starting "Install Backdrop". This is an area that seems to be confusing everyone already, since backdrop does NOT need to be installed before updating.

I read those steps in @BWPanda's blog post, and it makes sense there since he both installed, then later deleted everything that was created during the install. But I don't think we should include it in our official instructions since it can be confusing.

[bugfolder]

I'm not done reading through your three new pages, but I'm instantly nervous about the "Upgrade" page starting "Install Backdrop". This is an area that seems to be confusing everyone already, since backdrop does NOT need to be installed before you begin.

Yes, I wrote that before reading your further discussion. I'm in complete agreement that it should be rewritten to make prominent that there's a way of doing the upgrade without needed to do an installation. (I'd wavered on that; the alternative is to just make the changes directly to settings.php.)

If you're favorable to these pages as a starting point, I'll merge in your recent additions, and make the 'no need to install" more prominent.

[jenlampton]

I'd wavered on that; the alternative is to just make the changes directly to settings.php.

I did include the exact changes that are needed for to the default settings.php file provided with Backdrop on my new "Step 3" page, in an attempt to make that part clearer too.

I've got a meeting now but can look at the 3 new pages when I get done.

[bugfolder]

I did update step 1 of Upgrade.

[jenlampton]

I did update step 1 of Upgrade.

This still isn't right, it still says to run the installer under "If you are doing a side-by-side upgrade".

There should be no mention of running the installer for either option. Running the installer was something Peter did to make sure that his local enviornment was set up correctly, it has nothing to do with the Backdrop update so doesn't belong in the update instructions.

[jenlampton]

Can you take a look at the updated instructions on https://docs.backdropcms.org/documentation/step-3-upgrade-the-drupal-site, and see if it makes more sense now that it's been updated? I spent a lot of time on it this morning.

Not only did I clarify that the installer does not need to be run, but I changed the order of the steps so that updating settings.php happens after importing the Drupal database.

[bugfolder]

It does make sense (especially the do not install Backdrop ;o)), but there are points that I tried to address in my version that aren't addressed yet. For example, in step 5, there's a 3rd option for the files directory, which is to move them to some other place than those two (e.g., below the webroot), for which there's valid use cases that should at least be acknowledged.

Also, that description doesn't allow for the possibility of an in-place upgrade, for which there's valid reasons to do (even if it's not preferred).

A few other minor points: the old D7 site doesn't need to be "web-accessible," it could be a local site.

The three pages I put together somewhat work together: the different module categorization in Planning are then used in Preparation and Upgrade (as they were in BWPanda's list). So I'd suggest that either your Step 1/Step 2/Step 3 or my Planning/Preparation/Upgrade pages stay together. I am (obviously) biased in favor of mine (except for the wrong bits, of course, which should be changed), but I also suspect that (like Peter's original list) the more detailed discussion might be more helpful to the newbie (which I was not too long ago). Perhaps one of us could merge description from the one into the other. What do you think?

(BTW, having a Markdown text format on the docs site would be really nice.)

[jenlampton]

For example, in step 5, there's a 3rd option for the files directory, which is to move them to some other place than those two (e.g., below the webroot), for which there's valid use cases that should at least be acknowledged.

The "two options" we mention in step 5 are, are for how to move the files, not where to move them. We only mentioned one possible location -- the one backdrop core uses -- but there's no reason any other location wouldn't also work with these instructions.

I've changed the first line of the recommended approach to say Recommended: Move the files from where they were in Drupal, to where you would prefer to have them so that it's clear the location is up to you.

Also, that description doesn't allow for the possibility of an in-place upgrade, for which there's valid reasons to do (even if it's not preferred).

I've been thinking about these two approaches, but I'm not sure we need instructions for both. I think having a single set of instructions will be clearer.

Having two sites side-by-side is the same thing as starting with 2 copies of the Drupal 7 site and then do "an in-place upgrade" on the 2nd site (but it skips a few steps on the in-place upgrade, like having the Drupal 7 code there to start).

What if we re-phrased the step 1/2/3 instructions to focus on the in-place upgrade instead? We could recommend setting up "a Drupal 7 backup site" rather than focusing on the nuances of the side-by-side set-up.

A few other minor points: the old D7 site doesn't need to be "web-accessible," it could be a local site.

Hm, I've been writing these docs as though both these sites were local. Can you think of another way to say "You can visit them in your browser"? (that's what I meant by web-accessable... but you're right, that's the wrong word.)

So I'd suggest that either your Step 1/Step 2/Step 3 or my Planning/Preparation/Upgrade pages stay together.

I am hoping that we will end up with a single set of these 3 pages that contains the best from both sets.

I am (obviously) biased in favor of mine

I'm most comfortable changing my own words (which is why I've been making edits there), but I do also prefer much of what is in your pages.

I have re-worked all the sections from my "Planning" page to match yours, for example. (I've left out the directory structure bit since I believe we've covered that elsewhere in the documentation, and would prefer to link to it rather than have two pages to update if/when we make changes.)

I'd appreciate your help in determining what from your pages is still missing from mine, and how to get it all into one set in a way that makes the most sense :)

the different module categorization in Planning are then used in Preparation and Upgrade (as they were in BWPanda's list)

Yeah, I really loved that from his article! I've incorporated those categories into both my Planning and Preparation pages, but I renamed the categories so they are all "To X", for consistency: to uninstall, to replace, to port, to install.

I've also tried to address a concern I had about instructing people to uninstall modules labeled to replace because often that data is needed by the replacement module. I'm not sure I've done this in a way that's clear and concise though.

Could you give it another read-through?

[bugfolder]

Could you give it another read-through?

Will do, starting now...

The main page Upgrading from Drupal just contains links to subsections (which are already visible in the menu block to the right). It would be nice UX if when a user clicked on that page, they immediately were presented with useful information, rather than "oh, I have to click through to another page to get to anything." Perhaps consider moving some (or all?) of the Introduction onto that page?

And then, it seems like "Features added to core", "Features removed from core", and "Top 100 D7 modules" are pages one would need in the planning process, so perhaps they could be sub-pages of [Step 1: Planning]? (Yes, I know this is different from what I'd suggested earlier.)

On Upgrading from Drupal 7, since step 1's page is now titled "Planning", perhaps the corresponding heading on the page could use the same verb, e.g., instead of "Step 1: Research what will be necessary to upgrade to Backdrop CMS", use "Plan your upgrade to Backdrop CMS." Also, the planning step includes more than just "research": it also involves decision-making on things like file organization, where config will be live (and be called), version control, etc.

I'm now on Step 1: Planning, and like it very much. (There's something weird going on with the hyperlink to hook_update_N() at the bottom.)

Will continue review in subsequent comments.

[bugfolder]

On Step 2: Preparation, it would be nice to have a little overview/big-picture explanation at the top of the page, to help the user understand "what are all these steps trying to accomplish?" So, perhaps add something like the following:

To prepare your Drupal site for the upgrade, you'll need to make some changes to the Drupal site that will alter the database that Backdrop uses for its conversion process. In this step, we'll make those changes, making some backups along the way.

Step 4: perhaps the title should be "Disable and uninstall (some) core modules" (like steps 5 & 6)?

The collapsing of the different categories into steps 5 and 6 is very nice: clearer and more streamlined that what I'd had.

Step 7: this saving of the Views is an important step that's easy to miss (and beginners might be entirely unaware of the possibility of Views in code). Because it's an easy thing to accomplish in drush, I'd like to suggest including here the drush command to do this so people can just copy paste (assuming they have drush, of course):

drush ev 'foreach (views_get_all_views() as $view) {$view->save();}'

Do you want to add a comment at the bottom about the scriptability, or just assume people know that? I'd said:

Note that all of these steps are scriptable with shell scripts and drush. With a complex site, you may be making multiple upgrade runs on a local development copy of the site; putting all this is a script could be a big time-saver.

[bugfolder]

On Step 3: Upgrade:

Instead of "web-accessible", how about "browser-accessible"?

In step 2, "create a new empty database", this is assuming a "side-by-side" (versus "in-place") upgrade. Your earlier comments suggested we describe only one for simplicity, which makes good sense. I wonder, though, if there might be a place somewhere for a brief comment about how things would differ for an "in-place" upgrade, like:

If you are doing an in-place upgrade, the process is mostly the same, with these changes:

• In step 2, instead of creating a new database, you will use the upgrade-ready version of the Drupal database;
• In step 2, you will completely replace the Drupal codebase with the Backdrop codebase (but make sure you save a copy of the files directory for use in step 5);
• In step 7, edit the database settings in the settings.php file to point to the upgrade-read version of the Drupal database.

Speaking of step 7, there may be one more thing that needs to be done to settings.php. When I do a from-scratch Backdrop installation, the installer adds the line $database_charset = 'utf8mb4'; to the end of settings.php. Is this something that the user should do as part of the upgrade process?

In the last section, "You're done! Now test everything", I think the odds are good that for a lot of developers, they're not necessarily done; there's still the modules that were in the "To port" category. While some of those could be ported using a vanilla Backdrop site as the testbed, for many, it's going to be the case that they need their mostly-upgraded site to be used as the testbed for the remaining porting. It would be nice to have some discussion of this at the end of this page; that also then provides a nice segue into the next section, "Converting Drupal Code."