sync failure from 3.2.6-2+deb11u2 to 3.6 and 3.8.3 (last one + last patches)

[jcdelepine]

From time to time my 3.2.6-2+deb11u2 production servers can't synchronise to their 3.8.3 replicant, with segfault.
Getting a fresh new empty replicant make the synchro working again.
Moving the "bad" user on an other production server make the mailbox synchronise on the new replicant, and the initial server can finish its synchronisation list.

Here's an actual folder with the problem :

root@cyrus-backend-pers-1-03:~# cyrus sync_client -n replica -v -m "user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24"
MAILBOXES user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24
Segmentation fault

root@cyrus-backend-pers-1-03:~# cyrus mbexamine "user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24" 
Examining user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24... Mailbox Header Info:
  Path to mailbox: /var/spool/cyrus/mail/s/user/xxxx/FORMULAIRE FR /MOIS /JANV 24/22-01-24
  Mailbox ACL: xxxx	lrswipcda	anyone	p	
  Unique ID: 9h8r5xxbt2x333ga0umyfc14
  User Flags: NonJunk 

 Index Header Info:
  Generation Number: 1
  Minor Version: 16
  Header Size: 160 bytes  Record Size: 112 bytes
  Number of Messages: 0  Mailbox Size: 0 bytes  Annotations Size: 0 bytes
  Last Append Date: (1706168349) Thu Jan 25 08:39:09 2024
  UIDValidity: 1705907263  Last UID: 9
  Deleted: 0  Answered: 0  Flagged: 0
  Mailbox Options: POP3_NEW_UIDL
  Last POP3 Login: (0) Thu Jan  1 01:00:00 1970
  Highest Mod Sequence: 507261

 Message Info:

On the replicant :

root@cyrus-backend-pers-2-03:/var/lib/cyrus/log/admin# cat syncserver-4185553 
---------- admin Wed Jul 10 18:21:21 2024

<1720628481<COMPRESS DEFLATE
>1720628481>OK DEFLATE active
<1720628481<GET MAILBOXES ("user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24")
>1720628481>* MAILBOX %(UNIQUEID 9h8r5xxbt2x333ga0umyfc14 MBOXNAME "user.xxxx.FORMULAIRE FR .MOIS .JANV 24.22-01-24" MBOXTYPE ei SYNC_CRC 0 LAST_UID 0 HIGHESTMODSEQ 0 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1705907263 PARTITION NIL ACL "" OPTIONS "" CREATEDMODSEQ 473437 FOLDERMODSEQ 507184)
OK success

How can I help you find the problem ?

[jcdelepine]

Other server, other mailbox :
mbexamine :

Examining user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE... Mailbox Header Info:
  Path to mailbox: /var/spool/cyrus/mail/x/user/xxxx/01 B ANNEE 2024/LAON CALENDRIER ADMINISTRATIF/PHILO 2ND DEGRE
  Mailbox ACL: xxxx   lrswipcda       anyone  p       
  Unique ID: tuyzf1ssoizh6t4sne0rd91r
  User Flags: NonJunk $Forwarded 

 Index Header Info:
  Generation Number: 1
  Minor Version: 16
  Header Size: 160 bytes  Record Size: 112 bytes
  Number of Messages: 49  Mailbox Size: 17293968 bytes  Annotations Size: 0 bytes
  Last Append Date: (1714725930) Fri May  3 10:45:30 2024
  UIDValidity: 1686068638  Last UID: 65
  Deleted: 0  Answered: 9  Flagged: 0
  Mailbox Options: POP3_NEW_UIDL
  Last POP3 Login: (0) Thu Jan  1 01:00:00 1970
  Highest Mod Sequence: 239311

 Message Info:
000001> UID:00000001   INT_DATE:1688377161 SENTDATE:1688335200 SAVEDATE:1688377535 SIZE:251905
[...]

root@cyrus-backend-pers-1-05:~# cyrus sync_client -m  -n replica 'user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE'
Segmentation fault

root@cyrus-backend-pers-2-05:~# cyrus mbexamine 'user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE'
user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE: Mailbox does not exist

<1720955516<COMPRESS DEFLATE
>1720955516>OK DEFLATE active
<1720955516<GET MAILBOXES ("user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE")
>1720955516>* MAILBOX %(UNIQUEID tuyzf1ssoizh6t4sne0rd91r MBOXNAME "user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE" MBOXTYPE ei SYNC_CRC 0 LAST_UID 0 HIGHESTMODSEQ 0 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1686068638 PARTITION NIL ACL "" OPTIONS "" CREATEDMODSEQ 184144 FOLDERMODSEQ 227403)
OK success
<1720955516<APPLY RESERVE %(PARTITION default MBOXNAME ("user.xxxx.01 B ANNEE 2024.LAON CALENDRIER ADMINISTRATIF.PHILO 2ND DEGRE") GUID (33f03fdd57e7a418a2221b3bec2edef86e019c8e c7338c0cb251436b00114ed9031435f5b91bfa52 a2493999c579f7f1e88ff50e4d3308dff5b7c174 44df68edeb2196ba8107d18678bfe952dc1c00ea 45bdeceb6c94b14fcbd8ac6442fa52b5903eb214 a8a8acb2d3a4e58e2c67f0b6c64b65ba97a83a24 a0552e71d7da7ee9b70c281af0a93278858981bb 10c02e1170c2f31afa1730c0a5b929cc2e639a41 d65fbde990bc3d07215c90f0df7de6fdc805fd79 22598cb0e0c96a86b8ae1c376d235a62b47e6285 afaa644e48d6a381f766254347808cddbd3f154a cead1043ad847a37d7a4ab370eb0dd44bdcdaca9 c6e9fba42c6717596e847b6fc5b769ac8201bd72 7a0384194caa27ad982468235f2e22e00832a0cf ba6d26673733b1bed56fdbfacd5623e7f39c811f 402c4bcb4953201e6fee6fb05cd7611ab306681f 410e2524d6e7e014976024492d8f975fe0571dc3 1183f974fe1738ea36bea9343acf7956384cf7c7 48a62f31ed4e85950609c440e9ffc8abe8520f61 387c8b3c2696473d99f4af7653a0748cd7b01de7 d24f9fa27e6e65237e1bb9057e4f9fa1687f1d7e 5ab45ffb63d1ea25eb073817c83145cb0f37a924 fc2b00052dc592d5e6afccae2f76d68c4c4c1361 ab8c5ae510c772804c6092fab8ddc8c108d0ca80 b069559b630a196c0ab18ad361d12d5680986457 b5ea3d785e9b354b4a8f47364edbde70da1df049 f8c080a981a799c60733d7e022c5d5c85db9234c 501b36ff80bee083e6759704cf48d6c9f44b459b 7f25861dd0301b0d63e09f66e27fc1f25c5aa397 b9145d42fa7b546ecc52dad95c7b7a73f171cea9 8877f8abd3f31e9f0d0c386f26030af42aed0a4e 4df3eb458f63cd87f30e72c6dc60e1fb256bdcc0 3b407ab155ef1b446360560c238e32f9f39e955a f7b0400bfdc5d228e65bc9fa9446f5211a45658a 6330fd6a21a646183e7608b9a40f0cb8c3ce28e4 7e5e36af8230d76721dc7bee87640885f92f4794 80fab9ecf21de18396b4c335e0fe62df05fc641a 88d6b9880e5f833fd6a767c795cdb4df2038a8c3 57440e8c12d0f5727c37368d91b3b8971c2a65f9 af58c995c95dc6be6a542d7cfd4f8dd2539b02bf c502c081bff56ca5ad7485e25386576823c24a23 488886b0da949ca0247963934aec851ec55001bb e3cbc7d2585004dfc898ddbd0c941aeaad67b04d dcb5107ee95916a72bef99ace31034faa0fe2f96 e9f3bc5c3d16d585710ad2b33f969cbdb5f65eae e7131ea9cce87076f6286cf0c0c5dae24abe0f70 ea41a7fea0dd09581296a181d643494becc7c91b fc180487311e383d30988f62e5393144e2323b5b))
>1720955516>* MISSING (33f03fdd57e7a418a2221b3bec2edef86e019c8e c7338c0cb251436b00114ed9031435f5b91bfa52 a2493999c579f7f1e88ff50e4d3308dff5b7c174 44df68edeb2196ba8107d18678bfe952dc1c00ea 45bdeceb6c94b14fcbd8ac6442fa52b5903eb214 a8a8acb2d3a4e58e2c67f0b6c64b65ba97a83a24 a0552e71d7da7ee9b70c281af0a93278858981bb 10c02e1170c2f31afa1730c0a5b929cc2e639a41 d65fbde990bc3d07215c90f0df7de6fdc805fd79 22598cb0e0c96a86b8ae1c376d235a62b47e6285 afaa644e48d6a381f766254347808cddbd3f154a cead1043ad847a37d7a4ab370eb0dd44bdcdaca9 c6e9fba42c6717596e847b6fc5b769ac8201bd72 7a0384194caa27ad982468235f2e22e00832a0cf ba6d26673733b1bed56fdbfacd5623e7f39c811f 402c4bcb4953201e6fee6fb05cd7611ab306681f 410e2524d6e7e014976024492d8f975fe0571dc3 1183f974fe1738ea36bea9343acf7956384cf7c7 48a62f31ed4e85950609c440e9ffc8abe8520f61 387c8b3c2696473d99f4af7653a0748cd7b01de7 d24f9fa27e6e65237e1bb9057e4f9fa1687f1d7e 5ab45ffb63d1ea25eb073817c83145cb0f37a924 fc2b00052dc592d5e6afccae2f76d68c4c4c1361 ab8c5ae510c772804c6092fab8ddc8c108d0ca80 b069559b630a196c0ab18ad361d12d5680986457 b5ea3d785e9b354b4a8f47364edbde70da1df049 f8c080a981a799c60733d7e022c5d5c85db9234c 501b36ff80bee083e6759704cf48d6c9f44b459b 7f25861dd0301b0d63e09f66e27fc1f25c5aa397 b9145d42fa7b546ecc52dad95c7b7a73f171cea9 8877f8abd3f31e9f0d0c386f26030af42aed0a4e 4df3eb458f63cd87f30e72c6dc60e1fb256bdcc0 3b407ab155ef1b446360560c238e32f9f39e955a f7b0400bfdc5d228e65bc9fa9446f5211a45658a 6330fd6a21a646183e7608b9a40f0cb8c3ce28e4 7e5e36af8230d76721dc7bee87640885f92f4794 80fab9ecf21de18396b4c335e0fe62df05fc641a 88d6b9880e5f833fd6a767c795cdb4df2038a8c3 57440e8c12d0f5727c37368d91b3b8971c2a65f9 af58c995c95dc6be6a542d7cfd4f8dd2539b02bf c502c081bff56ca5ad7485e25386576823c24a23 488886b0da949ca0247963934aec851ec55001bb e3cbc7d2585004dfc898ddbd0c941aeaad67b04d dcb5107ee95916a72bef99ace31034faa0fe2f96 e9f3bc5c3d16d585710ad2b33f969cbdb5f65eae e7131ea9cce87076f6286cf0c0c5dae24abe0f70 ea41a7fea0dd09581296a181d643494becc7c91b fc180487311e383d30988f62e5393144e2323b5b)
OK success

We still have this "MBOXTYPE ei", MBTYPE_EMAIL and MBTYPE_INTERMEDIATE.
Indeed, on the replicant server 'PHILO 2ND DEGRE' doesn't exist yet but 'PHILO 2ND DEGRE.EMPLOIS DU TEMPS' and several others do exist.

Similarly, in the previous report, "FORMULAIRE FR .MOIS .JANV 24.22-01-24" doesn't exist yet but "FORMULAIRE FR .MOIS .JANV 24.22-01-24.FAIT " already exists.

Sincerly,
Jean Charles Delépine

[elliefm]

From time to time my 3.2.6-2+deb11u2 production servers can't synchronise to their 3.8.3 replicant, with segfault.
Getting a fresh new empty replicant make the synchro working again.

This isn't really surprising. The first replication works, because that's important when using replication as part of an upgrade process. But the newer replica will be storing different data than the older primary, and is likely to be confused when it receives incremental updates that it disagrees with. When this happens, the right thing to do is remove the data from the replica and do a fresh full replication.

You generally shouldn't run mismatched primary/replica versions on an ongoing basis like this, just temporarily during an upgrade.

We still have this "MBOXTYPE ei", MBTYPE_EMAIL and MBTYPE_INTERMEDIATE.
Indeed, on the replicant server 'PHILO 2ND DEGRE' doesn't exist yet but 'PHILO 2ND DEGRE.EMPLOIS DU TEMPS' and several others do exist.

Similarly, in the previous report, "FORMULAIRE FR .MOIS .JANV 24.22-01-24" doesn't exist yet but "FORMULAIRE FR .MOIS .JANV 24.22-01-24.FAIT " already exists.

In this particular case, I think the problem will go away if you just create real mailboxes for the MBTYPE_INTERMEDIATE mailboxes on the primary. You could do this using IMAP CREATE or "createmailbox" in cyradm. In 3.6 and later, all mailboxes path components are real mailboxes (no more MBTYPE_INTERMEDIATE), so the replication is probably disagreeing about the MBTYPE_INTERMEDIATE on the primary vs the real mailbox on the replica. In a one-shot upgrade replication this difference wouldn't be a problem, but it is for ongoing incremental updates since the updates don't match.