forked from MetaTunes/ProcessDbMigrate
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathhelp.html
666 lines (477 loc) · 72.1 KB
/
help.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
/* Responsive images */
img {
max-width: 100%;
height: auto;
}
/* Style the sidenav */
.sidenav {
height: 100%;
width: 20%;
position: fixed;
z-index: 1;
left: 0;
background-color: blue;
padding-top: 20px;
float: left;
overflow-y: scroll;
top: 0;
bottom: 0;
}
.sidenav a {
padding: 6px 8px 6px 16px;
text-decoration: none;
color: white;
display: block;
}
.sidenav a:hover {
color: #f1f1f1;
}
.sidenav ul {
margin-left: -30px;
list-style-type: none;
}
/* Style the main content */
.main {
margin-left: 20%; /* Same as the width of the sidenav */
width: 80%;
padding: 0px 10px;
float: left;
}
@media screen and (max-height: 450px) {
.sidenav {padding-top: 15px;}
}
</style>
<!-- Additional scripts & styles for AdminInModal --><script src="/wire/modules/Jquery/JqueryCore/JqueryCore.js"></script><script src="/wire/modules/Jquery/JqueryMagnific/JqueryMagnific.js"></script><link rel="stylesheet" href="/wire/modules/Jquery/JqueryMagnific/JqueryMagnific.css"><script src="/site/modules/AdminInModal/AdminInModal.js"></script><link rel="stylesheet" href="/site/modules/AdminInModal/AdminInModal.css"></head>
<body>
<div class="sidenav">
<ul class='uk-nav uk-nav-default'><li><a style='padding-left: 0px' href='#summary'>Summary</a></li><li><a style='padding-left: 0px' href='#background'>Background</a></li><li><a style='padding-left: 0px' href='#concept'>Concept</a></li><li><a style='padding-left: 0px' href='#uses'>Uses</a></li><li><a style='padding-left: 0px' href='#design'>Design</a></li><li><a style='padding-left: 0px' href='#health-warnings-and-known-limitations'>Health warnings and known limitations</a></li><li><a style='padding-left: 0px' href='#installation'>Installation</a></li><li><a style='padding-left: 10px' href='#dependencies'>Dependencies</a></li><li><a style='padding-left: 0px' href='#upgrading'>Upgrading</a></li><li><a style='padding-left: 0px' href='#uninstalling'>Uninstalling</a></li><li><a style='padding-left: 0px' href='#overview'>Overview</a></li><li><a style='padding-left: 10px' href='#documenting-the-migration'>Documenting the migration</a></li><li><a style='padding-left: 10px' href='#populating-the-migration-page'>Populating the migration page</a></li><li><a style='padding-left: 20px' href='#snippets'>Snippets</a></li><li><a style='padding-left: 10px' href='#exporting-the-migration'>Exporting the migration</a></li><li><a style='padding-left: 10px' href='#installing-the-migration'>Installing the migration</a></li><li><a style='padding-left: 10px' href='#locking'>Locking</a></li><li><a style='padding-left: 10px' href='#database-comparisons'>Database comparisons</a></li><li><a style='padding-left: 20px' href='#creating-a-migration-from-a-comparison'>Creating a migration from a comparison</a></li><li><a style='padding-left: 10px' href='#hooks'>Hooks</a></li><li><a style='padding-left: 20px' href='#available-hooks'>Available hooks</a></li><li><a style='padding-left: 20px' href='#placement'>Placement</a></li><li><a style='padding-left: 20px' href='#usage'>Usage</a></li><li><a style='padding-left: 0px' href='#troubleshooting-technical-notes'>Troubleshooting / Technical notes</a></li><li><a style='padding-left: 10px' href='#guide-to-the-files'>Guide to the files</a></li><li><a style='padding-left: 10px' href='#issues-with-change-logging'>Issues with change logging</a></li><li><a style='padding-left: 10px' href='#issues-with-page-migrations'>Issues with page migrations</a></li><li><a style='padding-left: 10px' href='#partial-unsuccessful-migrations'>Partial / Unsuccessful migrations</a></li><li><a style='padding-left: 10px' href='#field-types'>Field types</a></li><li><a style='padding-left: 10px' href='#subsequent-installations'>Subsequent installations</a></li><li><a style='padding-left: 10px' href='#rescue-mode'>Rescue mode</a></li><li><a style='padding-left: 10px' href='#system-and-repeater-templates-fields-and-flags'>System and repeater templates/fields and flags</a></li></ul></div>
<div class="main">
<div edit="dbMigrateAdditionalDetails">
<h1>DB Migrations module - "ProcessDbMigrate"</h1>
<h2><a href="#" name="summary">Summary</a></h2>
<p>This is a ProcessWire module to enable database changes in a development environment to be migrated to a test or live environment, without the need for any coding, and for all such migrations to be fully documented and capable of reversal.</p>
<p><strong>The module also has automated tracking of changes</strong> - your migration specification is built for you as you make changes in the development system (from v 1.0.0). In this help file, instructions relating to manual creation of migrations are displayed in <span style="color:#7f8c8d;">grey text</span> and may be ignored if you are using change tracking. This version (2.0.0+) inludes enhanced sorting of the migration sequence and persistent installation to deal with cyclical dependencies. There is no guarantee that major version changes (x.0.0) will be compatible with earlier versions, so great care needs to be taken if you happen to have tried a previous version and want to upgrade it.</p>
<p>The following documentation is pretty comprehensive, for which I make no apologies (in the hope that users will read it before asking questions). So:</p>
<p>TLDR:</p>
<ul>
<li>Install the module in each database environment and enter the required settings</li>
<li>Create the migration:
<ul>
<li><em>For automated specifications</em>: Create a migration page and turn on 'log changes'
<ul>
<li>Make your database changes in the development environment</li>
</ul>
</li>
<li><span style="color:#7f8c8d;"><em>For manual migrations</em>: Specify your migration on the migration page in the source database.</span>
<ul>
<li><span style="color:#7f8c8d;">Optionally use 'sort on save' to reorder your migration items to take automatically account of dependencies</span></li>
</ul>
</li>
</ul>
</li>
<li>Review your migration and, if happy, “export” it</li>
<li>Sync your target (test or live) code environment with the development environment</li>
<li>Install the migration in your target environment</li>
</ul>
<h2><a href="#" name="background">Background</a></h2>
<p>ProcessWire is an outstanding content management system/framework owing to its flexibility, ease of use and extensive API. However, it suffers from a common problem in database-oriented CMSs in that business logic may be explicitly or implicitly stored in the database rather than in code. This is particularly the case where it is used for a full-blooded app rather than just a simple CMS for a website. This is a shame because it is such a brilliant app-building tool.</p>
<p>The problem arises where the developer wishes to develop and test new or changed features in a development/test environment and then port those to the live environment. If the business logic is purely in the code and there are no database changes, then no issue arises, but if the changes involve database changes then typically the solution might be to test them, then replicate them manually (and hopefully accurately) in the live system. For large changes, this can be time-consuming and possibly error-prone. If a test environment is used as well as a development and live environment, then the problem is doubled. The 'RockMigrations' module circumvents this problem by putting the database definitions in code. ProcessDbMigrate allows the developer to use the PW GUI and not to have to write any additional code to define database changes. The choice of approach depends on user preferences.</p>
<p>The ProcessDbMigrate module is designed to make updating the test and/or live system as quick, easy and error-free as possible. However, it should be appreciated that deriving migrations from UI interactions is complex, so the user is advised to take back-ups before any critical steps and to test migrations before applying them in a live environment.</p>
<h2><a href="#" name="concept">Concept</a></h2>
<p>Purely code-based migration approaches effectively eschew the use of PW’s GUI development environment. This is technically perfectly sensible but restricts usage to those who are happy not to use the GUI. PW is a great tool for enabling less experienced developers to build great apps. This module is aimed at that group and others who wish to use the PW GUI but still be able to do controlled and accurate migrations.</p>
<p>The concept of this module is therefore to achieve the following:</p>
<ol>
<li>To allow development to take place (in a separate environment on a copy of the live database, or on a database with the same structure) using the Admin UI, with all changes logged, resulting in a <ins>declarative </ins>specification of the migration.</li>
<li>To allow testing of the migration in a test environment on a copy of the live database.</li>
<li>To allow roll-back of a migration if installation causes problems (ideally while testing rather than after implementation!).</li>
<li>To provide a record of changes applied.</li>
<li>Optionally, if changes are made directly on the live system (presumably simple, low-risk mods – although not best practice), to allow reverse migration to the development system in a similar fashion.</li>
<li>To provide a database comparison tool which can (if wished) be used to create a draft migration for the differences between the databases (health warning: unscoped comparison of entire databases, including all pages, is likely to result in an exception).</li>
</ol>
<h2><a href="#" name="uses">Uses</a></h2>
<p>The module has quite a wide range of applications. The original intention was just to handle the first of the examples below, but it has actually proved more useful than intended!</p>
<ol>
<li>Developments where database changes are involved (as well as possibly code changes). The changes can be fully tested in a dev environment. Installation of the migration can then easily be checked in a test environment (on a copy of the live database). Installation in the live environment is then very quick (sync code changes and migration files then one click to install the migration), resulting in little or no down time.</li>
<li>Updating of admin pages (not editable by general users), such as settings pages and help pages in the dev environment, then installing as a batch.</li>
<li>Updating of language pages (e.g. by a simple migration of pages with 'template=language') – the associated files will be updated along with the pages.</li>
<li>The selective reversion of parts of the database (by exporting migration data from a backup copy) for example where users have created numerous errors in the database.</li>
<li>Comparison of two databases (within a limited defined scope of fields, templates and pages) and optional creation of migration to align them.</li>
<li>Creation of a ‘blank canvas’ database from an existing working version. In other words, migrate the structure and all the necessary settings pages to a clean database.</li>
<li>Installation of another module which requires a number of module-specific database elements (requires coding of the 'install()' method in the module).</li>
</ol>
<h2><a href="#" name="design">Design</a></h2>
<p>The module has the following principal components all within the ProcessDbMigrate folder:</p>
<ul>
<li>A PW module “ProcessDbMigrate” - the main autoload module which also provides the setup pages;</li>
<li>A Page Class “DbMigrationPage” - the class for the DbMigration template which defines the migration pages;</li>
<li>A bootstrap migration;</li>
<li>A Page Class “DbComparisonPage” which extends DbMigrationPage (see separate section on comparisons for further information about this);</li>
<li>A folder “RuntimeFields” containing php files dbMigrateActions.php, dbMigrateControl.php and dbMigrateReady.php along with javascript files dbMigrateActions.js, dbMigrateControl.js and dbMigrateReady.js. These runtime fields are used in migration pages to display status and provide controls;</li>
<li>A module FieldtypeDbMigrateRuntime to implement the above runtime fields (pre v0.1.0 versions required the FieldtypeRuntimeOnly module).</li>
<li>This help documentation.</li>
</ul>
<p>Migration definitions are held in .json files in the site/templates/DbMigrate /migrations/{migration name} directory. This directory contains up to 3 sub-directories - “new”, “old” and “archive”. The first two may each contain a file called a migration.json file, which defines the scope of the migration, and - once the migration been exported (for ‘new’) or installed (for ‘old’) – a file called data.json. The data.json file contains the data specifying the details of the installation (or uninstallation, in the case of the ‘old’ file). In addition, the “new” and “old” directories can contain “files” directories which hold the files associated with pages in the migration. The “old” directory may also contain an “orig-new-data.json” file for comparison purposes. The “archive” directory contains earlier versions of the “old” directory if migration definitions have changed. There may also be a file ‘lockfile.txt’ if the migration has been locked, which just holds a date & time stamp of when it was locked.</p>
<p>The migration files described above are mirrored by pages of template "DbMigration" under a parent /{admin name}/dbmigrations/, with template "DbMigrations". The mirroring happens in two ways:</p>
<ol>
<li>If a new migration is created in the module (from the Setup -> Database Migration menu – see below, re installation), then initially no json files exist. The json files are created in the "new" directory, after the scope of the migration is defined on the page, by running "Export Data" from the eponymous button.</li>
<li>If (new) json files exist, but there is no matching migration page and the migration was created in a database with a different name from the current one, then a page is created by the module on accessing the Database Migration admin setup page. In this case, we are in the "target" database so there is no "Export Data" button, but instead "Install" and/or "Uninstall" buttons.</li>
</ol>
<p>Migrations therefore either view the current database environment as a “source” or a “target”. Technically, this is determined by whether the $page->meta(‘installable’) for the page is set or not. (The terms ‘installable’ and ‘exportable’ are used in this help file to differentiate the two types). Thus, if required, a knowledgeable superuser can change the type of a migration by adding or removing this meta item (e.g. in the Tracy console), but this is deliberately not made easy. (See further notes below on source and target databases). Also, the source database name is documented in the migration.json file.</p>
<p>Migration items may be either fields, templates or pages.</p>
<h2><a href="#" name="health-warnings-and-known-limitations">Health warnings and known limitations</a></h2>
<p>This module alters files and database elements. It is intended for superuser use only. <strong>Always take a backup before installing, updating or uninstalling it. Also take a backup before installing or uninstalling any migration.</strong></p>
<p><strong>Use at your own risk and test before implementing on the live system. Always have the same version of the module in source and target databases.</strong></p>
<p>The module may not handle all field types properly - particularly some pro fields and third party fieldtypes. Please refer to the section on fieldtypes and report any additional needs via the forum support thread.</p>
<p>When migrating pages, the sort order of children is ignored (it is also ignored in change tracking). So if you need to change it, then either do it manually or programmatically (possibly via a hook on the migration to run after installation).</p>
<h2><a href="#" name="installation">Installation</a></h2>
<p>Initially install the module in your dev (source) environment. Backup your database first.</p>
<ol>
<li>
<p>Place the whole folder in your site/modules/ directory.</p>
</li>
<li>
<p>Install ProcessDbMigrate.</p>
</li>
<li>
<p>Installing the module runs a ‘bootstrap’ migration which creates (system) templates called DbMigration, DbComparison and DbMigrations, and parent pages in the admin named ‘dbmigrations’ and ‘dbcomparisons’, so make sure you are not using those already and rename if required. It also creates some (system) fields which include “dbMigrate” in their name. All templates and fields have the ‘dbMigrate’ tag and are set as ‘system’ (i.e. flags=8) so they do not muddy the standard menus. The bootstrap migration files are copied by the installation process to the site/templates/DbMigrate directory, which will be created if it does not exist.</p>
</li>
<li>
<p>Configure the settings. Note that the settings are specific to the current database.</p>
<ul>
<li><em>Database name</em>: You can give the (current) database a name – in fact this is very strongly recommended. If you do so, this name will be included as an item (‘sourceDb’) in the migration.json of any migration you create from this database. Any database with the same name will treat this migration as exportable rather than installable. The best way to differentiate between a source (development) database and a target (production/live) database is to use the same database name and append the environment to it (see below). However, you can can also use custom names in each environment - in which case, if you copy a production database to use in the development environment, you will need to rename it to be the same as your original development database, so that any migrations sourced from that development database will be shown as exportable, not installable.</li>
<li><em>Append environment to database name: </em>This will automatically append the environment name to the database name. For this to operate, you need to set <code>$config->dbMigrateEnv</code> in your config.php file according to the detected environment there. This value will be appended to the database name by the module when determining the database in use. For example, in your config.php file:
<pre>
<code>$config->dbMigrateEnv = '_live';
//Below is for dev site
if($config->paths->root === '/var/www/html/') {
$config->ENV = 'DEV';
$config->dbMigrateEnv = '_dev';</code></pre>
</li>
<li><em>Show database name in notice</em>: The current database name, including the environment if appended, is notified (as a PW message) in every admin page (in case you forget which environment you are in!) – this will only be shown to superusers or users with the admin-dbMigrate permission.</li>
<li>
<p><del><em>Change tracking: </em>From version 1.0.0, the module incorporates automated change tracking, which eliminates the hardest part of the previous versions - namely the specification of the migration scope. In this section you just set selectors defining the scope of tracking and any changes within that scope will be tracked. For fields and templates, you might wish to use <code>id>0</code> to track all changes. For pages, you will probably just want to track 'settings' pages which are not routinely amended by users - typically this will be done by specifying the template(s), e.g. <code>template=MotifLayout.</code> All fields, templates and pages related to the module itself will be excluded automatically. <strong>If you do not enter a selector then no changes will be tracked</strong>. </del>From version 2.0.0, this is specific to each migration, not set in the module config.</p>
</li>
<li><em>Exclude field types, fields, attributes</em>: You can exclude any fields or fieldtypes from page migrations that might cause problems and which are not needed (you will need to do this in each database). DbMigrateRuntime, RuntimeMarkup and RuntimeOnly fields are always excluded and do not need to be specified here. Similarly, you can exclude attributes from field and template migrations. Excluded types will also be excluded from change tracking.</li>
<li><em>Auto-install bootstrap</em>: If selected (the default), the bootstrap will be automatically installed on an upgrade.</li>
<li><em>Prevent conflicting saves</em>: It is strongly recommended that you do not make any direct changes in the target database to any objects which are the subject of an "active" migration as this can cause problems if you need to uninstall the migration (the stored original state will not reflect the changes you made). This option (selected by default) prevents any such object changes in the target which may otherwise happen inadvertently. Even if this option is not selected, a warning will be given when editing the an object which is a target of an "active" migration. (Note that an "active" migration is defined as an unlocked installable migration after first installation).</li>
<li><em>Number of installation attempts</em>: Multiple installation attempts may be needed if there are circular references between migration items. Enter the number of repeated installation attempts to make before giving up (min 1, default 3, max 5)..</li>
</ul>
</li>
<li>
<p>Open the admin page “Setup -> Database Migration” to create your first migration. You will see that one (“bootstrap”) is already installed and cannot be modified.</p>
</li>
</ol>
<h3><a href="#" name="dependencies">Dependencies</a></h3>
<p>Note that this module has only been fully tested on ProcessWire>=3.0.206. At least 3.0.210 is recommended. It may work on previous versions, but certainly no earlier than 3.0.172.</p>
<h2><a href="#" name="upgrading">Upgrading</a></h2>
<p>Place the code in the modules directory, replacing the existing files, then refresh modules in the database. Check whether the bootstrap is still showing as ‘installed’. It should have been installed automatically; if not, then 'Refresh migrations' on the <em>Setup->Database Migrations</em> page.</p>
<p>If you are upgrading from a version earlier than 0.1.0, then the RuntimeOnly module will no longer be required for this module so, unless you use it elsewhere, you can uninstall it after the upgrade.</p>
<p>Before any upgrade, ensure that all current migrations are completed and locked, and back up the database.</p>
<h2><a href="#" name="uninstalling">Uninstalling</a></h2>
<p>Before uninstalling the module, all migration pages (other than the bootstrap) must be removed manually (and the trash emptied). Then, uninstalling the module uninstalls the bootstrap migration automatically; if that fails then the error will be reported, so that it can be fixed manually, before attempting to uninstall again.</p>
<h2><a href="#" name="overview">Overview</a></h2>
<p>The pic below illustrates the DB Migrations page in the 'source' environment. Please note that some pictures may be from earlier versions and may not show all features to be found in later versions.</p>
<p><img alt="DB Migrations page in the 'source' environment" src="help/Migrations_table.png" /></p>
<p>The status of a migration (as a source page) can be 'pending' or 'exported'. 'Pending' means either that the migration data have not yet been exported or that the current export files differ from the source database.</p>
<p>To install on the live (or a separate test) environment, sync your source and target code environments. Then install the module in your target environment (from step 2 above). After installation of the module, on opening the admin page "Database Migrations", the individual Migration pages are created from the definitions in their respective /new/migration.json file.</p>
<p>If you do not have a separate test environment, one approach is to backup the dev database and restore a copy of the live (or test) database to the dev environment. Then install the module on the restored database (from step 2) - you will have to assign a different database name in the module config. However, a separate test environment is better in that it more accurately mimics the live one and is less likely to cause confusion.</p>
<p>Your new migration should be listed (as 'indeterminate' status) in the Database Migration admin page.</p>
<p>The pic below illustrates the Database Migrations page in the target environment.</p>
<p><img alt="DB Migrations page in the 'target' environment" src="help/Migrations_table_target.png" />Using the module</p>
<h3><a href="#" name="documenting-the-migration">Documenting the migration</a></h3>
<p>When carrying out development work, first create a migration page and turn on change tracking - 'log changes'.<span style="color:#7f8c8d;"> If you do not turn on change tracking, you will need to keep a note of what fields, templates and pages you have added, changed or removed (create and update a migration page as you go along, rather than keep a separate note of changed components).</span></p>
<p>Note that the module does not handle components such as Hanna codes and Formbuilder forms. These come equipped with their own export/import functions, so use those (see "Snippets").</p>
<p>The migration page also allows you to document the migration using a rich text box, and it is recommended that you use this to add notes about the migration, including any pre- or post-migration checks and procedures to be carried out. (You may carry out some procedures automatically – see the section on hooks).</p>
<h3><a href="#" name="populating-the-migration-page">Populating the migration page</a></h3>
<p>See example below (note that this example has been created manually, not with change tracking, and is from a previous version):</p>
<p><img alt="Example source migration (previous version)" src="help/Migration_page_source_full_v4.jpg" /></p>
<p>The next example illustrates a migration created by logging changes:</p>
<p><img alt="Log changes example source migration" src="help/Migration_source_example.png" /></p>
<p><em>Change tracking: </em>From version 1.0.0, the module incorporates automated change tracking, which eliminates the hardest part of the previous versions - namely the specification of the migration scope. From version 2.0.0, the scope of this tracking is included in each migration page, rather than globally. You just set selectors defining the scope of tracking and any changes within that scope will be tracked. For fields and templates, you might wish to use <code>id>0</code> to track all changes. For pages, you will probably just want to track 'settings' pages which are not routinely amended by users - typically this will be done by specifying the template(s), e.g. <code>template=MotifLayout.</code> I<strong>f you use the PageAutosave module then make sure it is turned off</strong>, at least for the pages that you want to change, as otherwise changes will be saved without hooks operating and DbMigrate will not know about them. All fields, templates and pages related to the module itself will be excluded automatically. <strong>If you do not enter a selector then no changes will be tracked</strong>.</p>
<p>Based on the change tracking scope you have defined, on your migration page, the module will have entered the names of new/changed/removed fields and templates, and the paths of pages (as /path/to/pagename/) as you modified the database through the back-end UI - you may need to reload the page to see this.</p>
<p><em>Alternatively you can choose to add migration items manually</em>. Selectors may be used instead of paths or names (see box below) - this has to be done manually. Note that entry of names and paths is in text fields, not using dropdowns, as names, paths and ids may be different in the source and target systems – there is limited real-time checking of these.</p>
<p>If an item has a different name in the target database then it is provided in the 'old name' box, otherwise it is left blank (with change tracking on, the old name will be the name before any changes made).</p>
<p>If you are using change tracking then any pages which are deleted in the UI will be shown as having been moved to the trash. They will not be shown as removed until they are deleted from trash. If you make a number of changes to an object but end up where you started then it will (should) be automatically removed from the migration (including if you create a new object then delete it).</p>
<p>Note that <strong>the sequence is very important</strong> – if items are dependent on other items, they must be declared in the correct order for installation (when uninstalling, the order is automatically reversed and 'new' and 'removed' are swapped). Note particularly for Repeater and PageTable field types, you need to define the components in the right order – e.g. the template before the field that uses it, in the case of new/changed components. <strong>With change tracking on, the order should automatically be correct, </strong>but it is wise to review it in any case.<span style="color:#7f8c8d;"> If you have entered the items manually, you can use the 'sort on save' option to sort the items for you, but again, check that it makes sense as it may not be perfect. It is possible (especially with Page Table fields) for items to have circular dependencies. Error messgaes will be given about this - see the troubleshooting section on cyclical dependencies for more information. Usually the installation will proceed satisfactorily even if the dependencies are not correct, but it helps to get them right.</span></p>
<p><span style="color:#7f8c8d;">If you define a page item which has a rich text editor field that includes an image on <strong>another page</strong>, you will also need to include that other page in the scope (as 'changed' even if it has not changed). Otherwise you do not need to declare components that are unchanged.</span> Change tracking should automatically add pages with the source images used in a RTE field but, again, it is wise to check. Note that if you subsequently change the 'image source' page and revert the principal page so that the latter is removed from the migration, then the 'image source' page should also be removed from the migration unless it is within the defined tracking scope in its own right.</p>
<hr />
<p><span style="color:#7f8c8d;"><strong><em>Selectors (only applicable if creating migration manually)</em></strong>:</span></p>
<p><span style="color:#7f8c8d;"><em>Only one object name is permitted per item, however objects may be selected by using a selector rather than individual names or paths. Selectors operate as follows:</em></span></p>
<ol>
<li><span style="color:#7f8c8d;"><em>For ‘new’ and ‘changed’ objects, the selector identifies the objects in the</em> <strong><em>source environment only</em></strong>. <em>If these objects also exist in the target environment, they will be changed, otherwise they will be created (but avoid ambiguity – see 4 below). There is no possibility of matching objects whose name (or path) has changed.</em></span></li>
<li><span style="color:#7f8c8d;"><em>For ‘removed’ objects, the selector identifies objects in the <strong>target environment only</strong>.</em></span></li>
<li><span style="color:#7f8c8d;"><em>The use of “sort=path” is permitted in page selectors, even though this normally has no effect. If it is used, the pages will be sorted in parent-child order, using the ‘natural’ sort order (e.g. XYZ100 is greater than XYZ20). This means that parents will be installed before children. For ‘removed’ pages, the order is automatically reversed so that children are deleted before parents (there is no need to use the inverse: sort=-path).</em></span></li>
<li><span style="color:#7f8c8d;"><strong><em>Important</em></strong>: <em>Make sure that the scope of a selector works in both the source and target environments, i.e.:</em></span></li>
</ol>
<ul>
<li><span style="color:#7f8c8d;"><em>a. If the action is ‘changed’ then it must be unambiguous – i.e. all the names/paths of the selection of objects must be the same in both environments. Do not use ‘changed’ with an ambiguous selector because, although it may appear to install correctly, it will not be possible to uninstall it - any moved or removed pages will not be removed and uninstallation will fail.</em></span></li>
<li><span style="color:#7f8c8d;"><em>b. If the action is ‘new’ then the selection of objects should only exist in the source environment;</em></span></li>
<li><span style="color:#7f8c8d;"><em>c. If the action is ‘removed’ then the selection of objects must only exist in the target environment.</em></span></li>
</ul>
<p><span style="color:#7f8c8d;"><em>Do make sure that your selector works in the source and/or target environments, as appropriate, before implementing it (TracyDebugger is great for this).</em> <strong><em>A typical error is forgetting to add “include=…”.</em></strong></span></p>
<p><span style="color:#7f8c8d;"><strong><em>Note</em></strong> <em>that if your selectors encompass a large number of objects, processing time may be extended.</em></span></p>
<hr />
<p>You can limit the scope of changes to <ins>pages </ins>by restricting the fields to those specified in the "Restrict Fields" box. This restriction will apply to all pages within the scope of this migration, but only this migration; if you wish to exclude certain fields or fieldtypes globally, enter these in the module configuration. It will also restrict the logging of changes to pages so that only pages which have changes to the specified fields will be logged as changed.</p>
<p>You can preview the changes at any time, even before you export them, this will help you ensure that the data you are proposing to migrate is correct. You can also test as you go along, if you wish, and add to the migration in stages.</p>
<p>Note that migration pages just define the scope of the migration. It is entirely feasible for other parts of the dev database to be changed which are outside this scope and which will therefore not be migrated. <strong>However, do not do this with 'log changes' turned on, or you will have to remove unwanted items manually</strong>. <span style="color:#7f8c8d;">If you are doing this intentionally, be careful to ensure that the scope of your migrations do not overlap, otherwise you may inadvertently make changes that you do not wish to do yet.</span></p>
<p>When saving a migration page in the source database, the system will warn you if the current migration scope overlaps with other exportable migrations. <strong>Do not proceed to install such overlapping migrations</strong>. They will interfere with each other, even if they are not making conflicting changes – once one has been installed, installing the other will create an ‘old’ json file that reflects changes made by the first, so (for example) attempting to uninstall this second installation will appear not to be successful as it will not be able uninstall the changes made by the first. If two migrations necessarily overlap, then the correct process is to install the first one and lock it before installing the second migration (see 'Locking' below).</p>
<h4><a href="#" name="snippets">Snippets</a></h4>
<p>On the migration page, you can add any number of "snippets". These snippets do not do anything, but can be a convenient place to store (for example) Hanna code exports for pasting into the target environment and they help to make the page a comprehensive record of the migration.</p>
<p>You can also use snippets for php code to run before or after installation (or uninstallation). The code can then be copied into Tracy’s console log and run from there. Use the description box in the snippet to indicate when the code should be run. For example, say you have changed an address field to split out the post code into a separate field. The migration definition will specify the new field and the changed template. The code snippet (to be run after installation) will then include the code to extract postcodes and place them in the new field.</p>
<p>Alternatively, if you want code to run automatically, you can use hooks – see the separate section on this.</p>
<h3><a href="#" name="exporting-the-migration">Exporting the migration</a></h3>
<p>When ready to test (or migrate to live), click the "Export Data" button on the migration page. Some informative messages should result. Your codebase will now contain the json files in site/templates/DbMigrate/migration/{your migration page name} – you can inspect them if you wish in your IDE.</p>
<p>Sync the code files with your test/live environment (or restore the test database to your dev environment, <em>making sure you back up the dev database first</em>). In the test/live database, install the module as described above, if necessary, and go to the Database Migration admin page. You should see your new migration listed.</p>
<h3><a href="#" name="installing-the-migration">Installing the migration</a></h3>
<p>Go to the migration page. Before installing, you can use the "preview" button to see what changes will be implemented. If you are happy, click "Install". This should result in various messages and maybe some errors, if the migration was not able to complete fully (see section below for how to deal with partial migrations). See example of the migration page in 'installation' mode below:</p>
<p><img alt="Example installable migration" src="help/Installable_migration.png" /></p>
<p>and an example preview:</p>
<p><img alt="Preview differences" src="help/Preview_install_data.png" /></p>
<p>Note that you will now have the following files:</p>
<ul>
<li>A folder “old” in site/templates/DbMigrate/migration/{your migration page name} – this contains (as file data.json) the definition of the database (within the migration scope) <ins>before </ins>the migration was installed and is used for roll-back (uninstall). It also has the migration page definition that was used for the installation (as /old/migration.json) and the data.json file that was used for the installation (as old/orig-new-data.json) – these are used to detect whether the migration scope definition has changed since installation. The ‘old’ folder is used for uninstalling. If the migration scope is changed (in the source environment) then it cannot be installed in the target environment without first uninstalling the previous version of the migration scope and then installing the new version – in this way the ‘old’ files will reflect the revised scope.</li>
<li>A folder site/assets/cache/dbMigrations has the json files defining the current state of the database (within the scope of the migration)</li>
</ul>
<p>To uninstall a migration, click the "Uninstall" button (again, you can preview beforehand). This should revert the database to the initial state, but again there may be reasons why this cannot complete fully – see the notes below. Please note that uninstalling a migration is NOT the same as restoring the database. After uninstalling a migration, some of the object ids might be different from their original values. To fully revert the database, then restore the original backup (you did take one, didn't you?), but you will also need to remove the 'old' folder.</p>
<p>N.B. When re-installing migrations, if the migration definition has changed, the system will require you to uninstall first - otherwise the “old” data.json will not properly reflect the new scope, affecting any future uninstallation. In these circumstances, a backup copy of the “old” directory is created in the archive directory.</p>
<h3><a href="#" name="locking">Locking</a></h3>
<p>When an installation is complete, then you should lock it so that it is clear that it is complete and so that it will not be referred to by the system when checking for conflicts etc.</p>
<p>Migrations can be locked in the source system, the target, or both, in the following ways:</p>
<ul>
<li>Click on the 'Lock' button. This creates a lock file in site/templates/DbMigrate/migration/{your migration page name} which is time-stamped. You can do this independently in the source and target system(s).</li>
<li>Sync a lock file between the environments. This will lock the migration in the sync'd environment.</li>
</ul>
<p>This allows for some flexibility of use, e.g.</p>
<ul>
<li>Installing a migration on one target and locking it there before installing it on other targets</li>
<li>Installing a migration on all targets then remotely locking by sync'ing the files.</li>
<li>Temporarily locking a migration in the source (development) environment when the migration has 'log changes' turned on, so that you can experiment with some changes without affecting the migration.</li>
</ul>
<p><strong>It is stongly advised that you install and lock a migration before making subsequent changes to the source and building further migrations</strong>. Otherwise earlier migrations may attempt to install later features, if they overlap, and may fail because dependent items are not present. This might be resolved by installing both migrations and then attempting to re-install the first, but is not guaranteed.</p>
<h3><a href="#" name="database-comparisons">Database comparisons</a></h3>
<p>Comparisons work in a similar way to migrations. First, make sure you are in the database which you wish to be the ‘source’ of the comparison (usually the development database – this will be assumed for the rest of this narrative). Also, make sure that your database is named (on the module settings page).</p>
<p>On the Database Migrations page, select the “Add New DbComparison” button. On the DbComparison page you can add a summary and a number of ‘comparison items’. Note that comparison items just comprise an object type (field/template/page) and a name/selector. These items define the scope of the comparison. In theory, you can compare entire databases by using a selector “id>0” for each of pages, fields and templates. However, this is likely to be quite resource-hungry if the database is large and may cause your system to hang, so it is better to use a scope that is more selective. In any case, you will probably want to exclude migration and database comparisons from the scope, together with the related repeater pages, otherwise it can be a bit self-referential (but shouldn’t crash for this reason alone). The example below compares all templates and fields (but not pages):</p>
<p><img alt="Comparison - source" src="help/Comparison_source.jpg" /></p>
<p>After saving the page, click ‘export data’ (or preview first) and sync the templates/DbMigrate/Comparison/{name}/ directory to your target environment. Then go to the target database and open the comparison page (you may need to refresh the comparisons summary page first). Here you will see two ‘migrate actions’:</p>
<ul>
<li>click ‘Compare Database’ to see the differences between the current database and the source;</li>
<li>click ‘Create a Draft Migration For This Comparison’ to do exactly that (see below).</li>
</ul>
<h4><a href="#" name="creating-a-migration-from-a-comparison">Creating a migration from a comparison</a></h4>
<p>This process is only semi-automated. The system will work out the scope of the required migration (i.e. what fields, templates and pages require adding/changing/removing) but will not identify any dependencies, so the sequence of migration items might be wrong. Also, it will not identify any name changes – if names are different then this will result in a ‘new’ and a ‘removed’ item.</p>
<p>Therefore, clicking the ‘create draft migration’ button does only create a draft. You need to</p>
<ul>
<li>sync the migration files to the source environment;</li>
<li>review the migration page in the source database (<strong>turn off 'log changes'</strong>), resequencing the items as necessary (or use the 'sort on save' feature).</li>
<li>For name changes, you can alter the ‘new’ item to be ‘changed’ and set the ‘old name’ to be the name of the ‘removed’ item; then delete the removed item (although the migration should still work without doing this, it is neater).</li>
<li>then export the migration from the source environment and install it in the usual way.</li>
</ul>
<p>If you have selected the whole database for comparison, then rather than create a migration, you are probably better just to do a backup and restore!</p>
<p><em>Subsequent migrations</em>: Up until the point when migration created from a comparison has a /new/data.json file exported from the source system, it is considered to be ‘draft’ (signified by a meta(‘draft’) element). While in this draft state, it is replaced by any new draft migration created from the same comparison. After it is no longer draft, creating a draft migration from a comparison will result in a <ins>new </ins>draft migration, leaving the original in place.</p>
<p>In this way, a comparison page can be left in place and might used to create multiple migrations over time (provided, of course, that any new changes to the data are exported from the comparison page in the source database and sync’d to the target). But note that this will only work if the scope of the generated migration (which may be tighter than the scope of the comparison) covers all the changes. If it does not pick up all the changes you will have to create a new comparison.</p>
<h3><a href="#" name="hooks">Hooks</a></h3>
<h4><a href="#" name="available-hooks">Available hooks</a></h4>
<p>Many of the ProcessDbMigrate methods are hookable:</p>
<ul>
<li>___execute() – Display the Database Migrations setup page</li>
<li>___newPage($template, $parent, $title, $values) – Create new migration or comparison (depending on $template - DbMigration or DbComparison)</li>
<li>___executeGetMigrations() – refreshes all migration pages</li>
<li>___exportData($migrationPage) – creates the .json files for a migration</li>
<li>___removeFiles($migrationPage, $oldOnly=false) – removes the .json files for a migration or comparison ($oldOnly=true will only remove the /old/ directory)</li>
<li>___installMigration($migrationPage) – installs a migration in the target</li>
<li>___uninstallMigration ($migrationPage) - uninstalls a migration in the target</li>
<li>___lockMigration($migrationPage, $migrationFolder) – locks a migration (so, for example, you could automatically lock a migration in the target on successful installation)</li>
<li>___unlockMigration($migrationPage, $migrationFolder) – unlocks a migration</li>
<li>___previewDiffs($migrationPage, $comparisonType) – previews migration changes where $comparisonType is one of: export, install, uninstall, review</li>
</ul>
<p>In addition, DbMigrationPage::array_compare is hookable. This means that the comparison can be altered (with an 'after') hook to affect whether or not the installation is considered to be complete.</p>
<h4><a href="#" name="placement">Placement</a></h4>
<p>You can place your hooks in site/ready.php. In this case you will need to check the name of the migration page before running – e.g.</p>
<pre>
wire()->addHookAfter('ProcessDbMigrate::installMigration', function ($event) {
$migrationPage = $event->arguments(0);
if ($migrationPage->name == 'my-migration') {
///code to run
}
});
</pre>
<p>and then you use $migrationPage to further reference the migration page. This approach keeps all your hooks together and you have to remember to sync the site/ready.php as well as your migration files.</p>
<p>Alternatively (<strong>the recommended approach</strong>), you can place a file called ready.php in the site/template/DbMigrate/migrations/{my-migration}/ directory, in which case your script would be</p>
<pre>
$this->addHookAfter('ProcessDbMigrate::installMigration', function ($event) {
// code to run
});</pre>
<p>and then you can use $this to reference the migration page. This approach keeps all your migration-related data & code together and you only have to sync the migration folder. It also means that your migration-specific ready.php code will be displayed at the bottom of the migration page. Also, if you ‘remove migration files’ it will remove your custom code as well (usually you will only be doing this as preparation to delete the migration, so that is what you want). With the first approach, your hook will remain in site/ready.php.</p>
<h4><a href="#" name="usage">Usage</a></h4>
<p>Of the available hooks, installMigration and uninstallMigration are likely to be the most useful. For example, a hook after ProcessDbMigrate::installMigration could be used to carry out database-wide changes following the amendment of the structure. Using the example described under ‘Snippets’ earlier, say you have changed an address field to split out the post code into a separate field. The migration definition will specify the new field and the changed template. The hook could then include the code to extract postcodes and place them in the new field. You could place code to undo this as a hook before ProcessDbMigrate::uninstallMigration, so that executing the uninstall exactly reverses the install.</p>
<p>In some circumstances you may not wish the code in your hook to run if (for example) the installation was not completely successful. You can test the status of a migration as follows (<em>use $migrationPage = $event->arguments(0); instead of $this if your code is in site/ready.php</em>) by checking if <em>$this->meta('installedStatus')['status']</em> is 'installed' or 'uninstalled'.</p>
<p>Note that $this->meta(‘installedStatus’) is an array as follows:</p>
<ul>
<li>'status' => (string) one of <em>pending</em> (not yet exported or different from exported data), <em>exported</em>, <em>indeterminate</em> (not installed or uninstalled – usually means not yet installed and no ‘old’ files yet exist to provide uninstall data), <em>installed</em>, <em>uninstalled</em>, <em>superseded</em> (locked migration which would otherwise be shown as pending or indeterminate), <em>void</em> (both installed and uninstalled, so the migration achieves nothing),</li>
<li>'scopeChange' => <em>true</em> if the proposed migration and old/orig-new-data.json refer to different database elements (so the ‘old’ definition would no longer be a suitable basis for an uninstall action)**,</li>
<li>‘scopeDiffs’ => the differences between the new and previous migration, where there is a scope change</li>
<li>'installed' => <em>true</em> if installedData and installedMigration are true,</li>
<li>'uninstalled' => <em>true</em> if uninstalledData and uninstalledMigration are true,</li>
<li>'installedData' => <em>true</em> if installedDataDiffs is empty,</li>
<li>'uninstalledData' => <em>true</em> if uninstalledDataDiffs is empty,</li>
<li>'installedDataDiffs' => (array) reportable* differences between the current state and the installation data (new/data.json),</li>
<li>'uninstalledDataDiffs' => (array) reportable* differences between the current state and the uninstallation data (old/data.json),</li>
<li>'installedMigration' => <em>true</em> if installedMigrationDiffs is empty,</li>
<li>'installedMigrationDiffs' => (array) reportable* differences between the current migration definition and the installation migration definition (new/migration.json),</li>
<li>'uninstalledMigration' => <em>true</em> if uninstalledMigrationDiffs is empty,</li>
<li>'uninstalledMigrationDiffs' => (array) reportable* differences between the current migration definition and the uninstallation migration definition (old/migration.json) – it will be empty if the migration has not yet been installed and so no ‘old’ files yet exist,</li>
<li>'uninstalledMigrationKey' => <em>true</em> if uninstalledMigrationKeyDiffs is empty,</li>
<li>'uninstalledMigrationKeyDiffs' => (array) any 'key' differences between the current and uninstallation migration definitions; 'key' differences are those which affect the database (migration items and 'restrict fields')</li>
<li>'reviewedDataDiffs' => (array) reportable* differences between the installation data (new/data.json) and the uninstallation data (old/data.json) – it will be empty if the migration has not yet been installed and so no ‘old’ files yet exist,</li>
<li>‘timestamp’ => just that – the time the installedStatus meta was created</li>
</ul>
<p>*Reportable differences exclude any differences in excluded fields and also any differences that are purely caused by pages having different ids in the source and target databases. The array is multidimensional of varying depths with the bottom elements being 2-element arrays containing the differing values.</p>
<p>**Note that it is perfectly possible for there to be a scope change even if there are no differences in the migration definition. For example, if a page selector is used and changes to the source database mean that there is a change in the pages found by the selector.</p>
<p>Use the Tracy console - <em>d($page->meta('installedStatus'));</em> - to inspect the installed status for any migration page. Call the method exportData() - <em>$page->exportData('compare');</em> - first to update the meta if required.</p>
<p>There are also a number if other 'meta' items set which may be of assistance in hooks:</p>
<ul>
<li>idMap: A map of id's in the source database to those in the target - i.e. an array of [oldId => newId, ...] where oldId is from the sourceand newId is in the target.</li>
<li>dataArray: A php array corresponding to the 'new/data' json</li>
<li>migrationArray: A php array corresponding to the 'new/migration' json</li>
<li>installFields: A php array of fields to be installed.</li>
<li>installPages: A php array of pages to be installed.</li>
<li>installTemplates: A php array of templates to be installed.</li>
<li>ignoreDiffs: A php array. Intended to hold arrays [path, field name] so that it can be set in a hook and then referenced in a further hook (on array_compare) to ignore differences in page fields with the given path and field name. However, it is not set by the DbMigrate code at all (except to reset it at the start of a migration) so it could actually be set to contain anything for any purpose in hooks. An example of the intended use is:
<blockquote>
<pre>
$this->addHookAfter(('DbMigrationPage::array_compare'), function($event) {
$R = $event->return;
if($this->meta('ignoreDiffs') && is_array($this->meta('ignoreDiffs'))) {
foreach($this->meta('ignoreDiffs') as $diff) {
$R = removeNode($R, $diff[1], $diff[0]);
}
$event->return = $R;
}
});
function removeNode($R, $f, $p, $keys = '') {
foreach ($R as $key => &$value) {
if (is_array($value)) {
if (count($value) == 2 && $key == $f && strpos($keys, $p) !== false) {
unset($R[$f]);
} else {
$value = removeNode($value, $f, $p, $keys . $key);
}
} else {
if ($key == $f && strpos($keys, $p) !== false) {
unset($R[$f]);
}
}
}
return $R;
}</pre>
</blockquote>
</li>
</ul>
<h2><a href="#" name="troubleshooting-technical-notes">Troubleshooting / Technical notes</a></h2>
<h3><a href="#" name="guide-to-the-files">Guide to the files</a></h3>
<p>The following files are created:</p>
<ol>
<li>Migrations files (template/DbMigrate/migrations) hold all details of migrations in directories named after the migration name. The bootstrap migration is created on installation.</li>
<li>Within each migration directory, the following may be found:
<ul>
<li>a. A ‘new’ directory which holds the exported details of a migration. This contains migration.json (the definition of the scope of the migration, derived from the source migration page), data.json (the definition of all the elements within the scope, derived from the source database) and a 'files' directory containing directories for any files/images associated with migrated pages. To install a migration in a target environment, this ‘new’ directory needs to be copied there (in its parent migration directory).</li>
<li>b. An ‘old’ directory (created on installation of a migration in the target environment). This has a similar structure to ‘new’. Note that the migration.json file is a copy of the file that scoped the installation. The data.json file defines the pre-installation state as a ‘reverse migration’ (i.e. it reverses the actions of the data.json file that was used for the installation). The orig-new-data.json file stores a copy of the new/data.json file that was used to install the migration and create this old directory – it is used to check for scope changes. Note that subsequent installations do not change this directory unless the scope of the migration has been changed and the old installation is uninstalled (see c below).</li>
<li>c. An archive directory holding previous ‘old’ directories when migrations have changed.</li>
<li>d. A ready.php file if the user created it.</li>
<li>e. A lock file – lockfile.txt – if the migration is locked.</li>
</ul>
</li>
<li>Temporary files are created in assets/cache/dbMigrate, related to the current (or most recently viewed) migration page:
<ul>
<li>a. migration.json is the current migration scope</li>
<li>b. new-data.json is the current state for comparison with new/data.json and</li>
<li>c. old-data.json is the current state for comparison with old/data.json.</li>
</ul>
</li>
<li>Comparison files (template/DbMigrate/comparisons) hold all details of comparisons in directories named after the comparison name. Theses directories have a 'new' directory with a similar structure to migrations.</li>
</ol>
<h3><a href="#" name="issues-with-change-logging">Issues with change logging</a></h3>
<p>The 'log changes' method is designed to faithfully record objects (<strong>which are within the scope of change tracking defiend in the migration</strong>) which have changed and put them into the correct dependency sequence. It is still wise to review the result to see if it makes sense. Sometimes it might appear not to: for example, if you have created and then deleted a repeater field, the repeater_fieldname template item will still appear in the migration. That is because deleting a repeater does not delete its associated template - the module has worked correctly because there is still a new template - see <a href="#system-and-repeater-templates-fields-and-flags">System and repeater templates/fields and flags</a>.</p>
<p>With a really complex migration, it is possible that the migations item cannot be sorted properly as a "cyclical dependency" has arisen. This can occur with either 'log changes' or 'sort on save' selected. If this occurs with change logging on, you should be alerted immediately after the action which created the dependency. Assuming it is not a spurious dependency (maybe the system is being over-zealous), in which case you can try and manually sort the items, the most rigorous solution is to back out the last change, migrate what you have (or a sensible chunk of it) and create a new migration as a second stage. If you just did 'sort on save' then the error message will arise when you save the migration page and you will need to review the changes to detemine where the cyclical dependency might arise.</p>
<p>If you do end up with a migration which contains cyclical dependencies, it is probably still possible to install the migation (and it is probably best to tolerate it at this time, rather than trying to split it retrospectively, because you may end up with a migration which depends on items in a later migration - see section on 'locking' above). On installing the migration, the system will attempt up to 3 successive installations in order to get the required data, although you will get error messages implying it is not fully installed (arising from the earlier attempts). If it is still not installed, then you can try clicking 'install' again, but it is likely that something is missing in your migration, so look at the preview to determine what it is.</p>
<h3><a href="#" name="issues-with-page-migrations">Issues with page migrations</a></h3>
<p>Issues may arise with migrations of pages, which are not possible to foresee fully. This is particularly the case if the user migrates pages with multiple complex field types and custom processing. The module was designed for migrating developments, not for mass updating of user pages. It is therefore assumed that any pages being migrated are of a ‘site-settings’ nature, not user pages. That said, it is possible to use the module more generally (e.g. in ‘rescue’ mode) but test carefully first. In particular, the host PW application may make use of page hooks. All page actions in the module allow hooks to run. To enable users to modify this behaviour, session variables are set for the duration of the following methods:</p>
<ul>
<li>installPages() – new/changed pages – ‘dbMigrate_installPages’ is set to true</li>
<li>removeItems() – all item removals – ‘dbMigrate_removeItems’ is set to true</li>
</ul>
<p>These can then be referenced in the user's application code as required.</p>
<h3><a href="#" name="partial-unsuccessful-migrations">Partial / Unsuccessful migrations</a></h3>
<p>In some circumstances, migrations may not complete properly, even after the maximum of 3 attempts that the system will execute. This may result in error messages when the migration is installed. Also, the page will show that it is not fully installed and will display a button to preview the remaining differences. Some typical examples and possible solutions are set out below.</p>
<p>Hopefully, these will only tend to occur when you have created the migration manually, rather than using 'log changes', but nothing is perfect...</p>
<p>Occasionally, re-clicking 'install' (or 'uninstall' if that's what you were attempting) may resolve this, but it is unlikely given that the system will have already tried 3 times. Note that you cannot change a migration definition in the target environment, so any changes need to be done in the source environment and re-exported (and installed after uninstalling the previous version).</p>
<p><em>Fields not fully installed</em>: This may be because there are fieldtypes which cannot currently be handled by the module. It could also be caused by dependencies on pages (e.g. for page reference fields) where the pages have not been included in the migration ahead of the field, although those should have been handled by the triple-install approach; more likely you have omitted to include a dependentn item in the migration, particularly if it was created manually.</p>
<p><em>Templates not fully installed</em>: You may not have included new fields, used by a template, within your migration page scope, ahead of including the template. You will need to go back to the dev system and do this (re-export and re-install)</p>
<p><em>Pages not fully installed</em>: This may be because you have not previously defined new templates or fields used by a page, in which case you will need to go back to the dev system and do this (re-export and re-install). It might also be because certain fieldtypes cannot be sync'd but do not need to be and should be excluded. To exclude field types from the pages scope, go to the module config page. If there are field types that do not sync correctly, but which you do not wish to exclude, then you will need to update the target system manually (the preview should give you the details).</p>
<p><em>Components not removed</em>: This is most likely because you have misnamed the component or because it does not exist in the target database. Double check the naming and go back to the dev system to change as required.</p>
<p>Use the preview button to see what changes have not been fully implemented. You can also inspect and compare the json files to identify the issue (your IDE is probably the best place to do this). For example, compare templates/…/new/data.json (the required state) with assets/cache/dbMigrations/new-data.json (the current state). (Similarly, for uninstallation problems, compare ../old/data.json with assets/cache/dbMigrations/old-data.json).</p>
<p>If you are unable to resolve the issue then it may be a simple bug or that the situation is complex and not envisaged in the design of the module. In any case, the ultimate fix is to review the stated differences in the 'preview' and make the required changes manually (uncheck 'Prevent conflicting saves?' in the module configuration first, or lock the migration, so that saves are not blocked). If you unchecked the 'Prevent conflicting saves' option then, when you are satisfied that everything is correct, lock the migration and re-check the option.</p>
<h3><a href="#" name="field-types">Field types</a></h3>
<p>The module handles (I think) the following field types:</p>
<ol>
<li>
<p>Core fieldtypes. These have not all been tested, but most are straightforward:</p>
<ul>
<li>Select Page has been tested and should work – however there may be problems if a parent page name/path has changed, in which case the migration might need to be split into two successive installations.</li>
<li>RTE (Rich text editor) fields are fine, but a complication arises when they include a link to an image (or file), because the link will reference the files directory with the page id in the source environment. The module allows for different page ids in source and target systems.</li>
</ul>
</li>
<li>
<p>FieldtypeRepeater (and FieldtypeRepeaterMatrix). This is more complicated because the field is actually linked to a template and multiple repeater pages.</p>
<p>When defining a migration, the normal sequence is to define fields before templates which might use them. However, repeater fields are linked to their own templates, so the repeater_fieldName template needs to be included before fieldName.<br />
The system is designed to handle nested repeaters on a page.<br />
When using 'log changes' all required repeater templates and dependencies should be handled automatically. Note however, that if you delete a repeater field, PW <strong>does not delete the corresponding template.</strong> So if, for example, you have 'log changes' on, create a new repeater field then delete it, the migration page will still show the new repeater template (but not the field) - which is exactly right, but might not be what you intended.</p>
</li>
<li>
<p>FieldtypePageTable. This has a PageArray so requires special processing somewhat similar to repeaters.</p>
</li>
<li>
<p>FieldtypeStreetAddress. This is an object and requires special processing analagous to PageTable.</p>
</li>
<li>
<p>Images and files should be migrated along with the page that holds them. Any Rich Text Editor (textarea) fields with embedded images or linked files should migrate satisfactorily <strong>provided</strong> the pages which hold the images/files are included in the migration (this should be automatic with 'log changes'). If image/file fields use custom fields (defined in the template "field-xxxx" where xxxx is the image/file field name), then it is important that any new or changed field-xxxx template is specified in the migration before the page that uses the related image/file field. if the image/file field itself is unchanged, it does not need to be specified.</p>
</li>
</ol>
<p>ProFields other than RepeaterMatrix have not been tested, so may or may not work.</p>
<p>Note that, when the migration installation saves pages, any hooks related to the page save are suppressed. This is to prevent spurious errors (e.g. if ConnectPageFields module is installed). It is possible that it might cause some incomplete actions for some field types, in which case it should be resolvable by saving all the affected pages (possibly in a migration ready.php file). Installed pages can be found in the meta('installedPages') array - see the section on hooks above. Since version 2.0.14 all migration saves are with noHooks.</p>
<h3><a href="#" name="subsequent-installations">Subsequent installations</a></h3>
<p>After an initial export from a source and installation in a target, the question arises as to how to deal with revised or additional migrations. If a migration is revised in the source, then it can be installed in the target once the previous version has been uninstalled. New migrations exported from the source are handled in the same way as initial migrations.</p>
<p>However, the user may wish to use a (migrated) test or live database as a new source for subsequent development (assuming the migrations so far are bug-free). In this case, use one (or both) of the following strategies:</p>
<ul>
<li>Use database naming (in the module settings page) to either:
<ul>
<li>'append environment to database name'; or</li>
<li>rename the imported database back to the name you were using for the development database.</li>
</ul>
</li>
</ul>
<p><cite><small>Note that the 'append environment' method does not work if you are using the same environment for a development and test database - you will need to use the custom name method for this.</small></cite></p>
<ul>
<li>Lock the completed migrations (by clicking on the 'lock' icon on the migration page) before importing the database. They will then be there as a record only – the database can be copied to the development environment (after locking) and new migrations can be developed for export, knowing that the base system is in sync (except for user changes to the live system, which will normally only be to pages, not fields or templates).</li>
</ul>
<p>The use of names for items rather than ids should mean that most user changes will not disrupt future migrations. Installation 'previews' should highlight any difficulties. Overlapping scope detection on migrations only looks at unlocked migrations.</p>
<p>It is also possible, in theory (not advised, but maybe necessary if the development environment is inaccessible) to make changes directly to the live database and 'export' them for installation on the development system. These should be locked after installing them. Database naming is strongly recommended if you use this strategy.</p>
<h3><a href="#" name="rescue-mode">Rescue mode</a></h3>
<p>An additional use of the module is ‘rescue’ mode. If a number of erroneous changes have occurred to a live database which need to be reverted, then a suitable backup copy can be restored to the development environment (after backing up the dev database!), given a unique name, and the relevant migration exported for installation in the live environment. After successful installation, the development database can be restored to the development environment – the ‘rescue’ migration will then be loaded automatically as a (‘installable’ – i.e. non-editable or exportable) migration and can be installed if it makes sense to do so.</p>
<h3><a href="#" name="system-and-repeater-templates-fields-and-flags">System and repeater templates/fields and flags</a></h3>
<p>All repeater_.... templates are system templates. If you delete a repeater field the template will still remain. If you want to delete the template as well, you will need to unset the system flag first, as described below. If you are using 'log changes' on a migration page and (for example) you created a new repeater field then deleted it, the migration item for the field will be removed, but the migration item for the repeater_.... template will remain (because the template is still there). If you want to remove it, either delete the template as described below (probably the best approach) or switch the migration to 'manual', delete the item, the switch back to 'log changes'.</p>
<p>In addition to the PW-created system templates/fields, note that all dbMigrate templates and fields are also designated as ‘system’</p>
<p>In system fields the 'flags' is set to 8. This means that they are hidden from the setup dropdown and cannot be used in the front end. To unset the flags it is necessary to first set the override, viz:</p>
<pre>
$t->flags = Template::flagSystemOverride;
$t->flags = 0;
$t->save();</pre>
<p>(Substitute Field for Template if resetting a field. For pages you need <em>$p->status = Page::statusSystemOverride</em> and <em>$p->removeStatus(Page::statusSystem)</em>).</p> </div>
</div>
<script type='text/javascript' src='/site/modules/MarkupCookieConsent/MarkupCookieConsent.js'></script><form id='mCCForm' class='mCCF mCCF--bottom mCCF--dark' action='./?accept=cookies' method='post'><button id='mCCButton' class='mCCF__accept' name='action' value='acceptCookies'>Accept cookies</button><p class='mCCF__message'>This website uses cookies to ensure you get the best experience<a class='mCCF__link' href='/privacy-policy/' target='_self'>View our privacy policy</a></p></form></body>
</html>