[Question]: time required for a complete Gmail synchronization cycle

[gioxx]

What is your question?

As reported here: #207 (comment)
I would like to know how long I need to wait for the software to complete a full synchronization and indexing cycle, as it is still running (we are talking about a mailbox with approximately 170,000 emails and 13GB of disk space).

I am joining this thread because it partly deals with the same topic/problem. I confirm that Gmail's limitations definitely slow down the complete backup process (you need to be very patient), but I also note that the issue of folder ownership takes just as long.

I will only quote an excerpt:

11/29/2025, 07:05:22 PMinfo: MailArchiver.Services.EmailService[0]
11/29/2025, 07:05:22 PM Updated folder for existing email: Example 0 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:05:09 PMinfo: MailArchiver.Services.EmailService[0]
11/29/2025, 07:05:09 PM Updated folder for existing email: Example 1 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:05:01 PMinfo: MailArchiver.Services.EmailService[0]
11/29/2025, 07:05:01 PM Updated folder for existing email: Example 2 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:04:48 PMinfo: MailArchiver.Services.EmailService[0]
11/29/2025, 07:04:48 PM Updated folder for existing email: Example 3 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:04:42 PMinfo: MailArchiver.Services.EmailService[0]
11/29/2025, 07:04:42 PM Updated folder for existing email: Example 4 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:04:40 PM Updated folder for existing email: Example 5 from '[Gmail]/Important' to 'Call log'
11/29/2025, 07:04:40 PMinfo: MailArchiver.Services.EmailService[0]

The question is: how long does this "second part of the process" last? It runs day and night, even though the underlying machine is not very busy. The process keeps the NAS disks (where I put the data volume) “hostage” without ever allowing them a moment's rest. Nothing tragic, of course, but at least I'd like to understand how long it might go on for. I've been using GYB for years and I'm certainly not used to all this “busyness.” 😄

BTW: thanks for your project, I really appreciate it!

Many thanks!
Giovanni.

Application Version

2511.3

[gioxx]

I would like to add one more observation: every time I update the base image, it restarts checking all emails, which means it will take a long time to verify that everything is backed up and then move on to indexing again.

[MelanieTanaka]

I think I'm in a similar situation. My gmail inbox has about 450k items. My first sync as expected took a very long time, almost a week. I let it run and it completed some time yesterday.
I expected the next sync to just transfer over the difference of what hasn't synced yet but that sync job has been running almost 24h now and hasn't added any new mail to the archive so it feels like it is going over every single item again.

Is it going to do this every time?

[s1t5]

By default, sync only retrieves emails since the last completed sync, not always the entire server inventory. However, the problem with Gmail is that Google limits IMAP retrieval to a maximum of 2500MB per day. This means that the initial sync is problematic for large mailboxes in Gmail and can take a long time

[s1t5]

@gioxx Your observation seems to be based on the fact that a complete sync has not yet been performed at runtime.

[gioxx]

@s1t5 That's right. Between system updates, image updates, and the large amount of emails to back up, I can confirm that the first truly complete synchronization was never completed. We're only talking about my main personal Gmail account, because I've already set up other accounts that worked perfectly (but are much smaller).

I have temporarily blocked the process (disabled that mailbox sync) while waiting to discuss it here and figure out how best to proceed.

[Cantello]

Same here, I stop all containers during the night to perform backups via backrest and that's why mail-archiver did not yet complete a single backup for over three weeks. Have now disabled stopping the container at night and see how this works out.

[luison]

Not directly my case but also stuck with two Google accounts for 2 weeks now. Please let me know if I should move this to a new thread.

The 2 accounts with error show as last sync at "01/01/1970, 01:00 AM" so never gets completed.
I have the impression it starts from the beginning, all the account every time:

Wondering how to debug this and also if it might be related with any of the extended settings expressed in the documentation like:

 # MailSync Settings
      - MailSync__IntervalMinutes=15
      - MailSync__TimeoutMinutes=60
      - MailSync__ConnectionTimeoutSeconds=180
      - MailSync__CommandTimeoutSeconds=300
      - MailSync__AlwaysForceFullSync=false
      - MailSync__IgnoreSelfSignedCert=false

Are there any "ideal values" for these. In our case (currently in default) we plan to execute it every 6–8 hours as we have concerns to overflow the connections of "real" imap users that are working at office hours.

[s1t5]

That's correct, if there is an error during synchronization, a complete synchronization will start again at the next interval. This will continue until an error-free synchronization has taken place (or you manually set a last sync timestamp in the account details). After that, only the changes since the last sync will be retrieved. The IMAP limit of 2500 MB per day will probably throw a spanner in the works here, as it cannot be deactivated on the Google side. One idea here might be to use Google Data Takeout to import all the emails as a file and then manually set the sync time so that only new items are retrieved via IMAP, but the initial batch of emails has been imported manually.

[gioxx]

At this point, I wonder if it might actually make sense to "take on the dragon one head at a time", perhaps with a filter that allows me to filter only a specific period of time (which could even be one year at a time).

By doing so, would it be possible to establish a sort of "cut-off" that would allow me to complete one synchronization and then tackle the next (perhaps the following year)? This would avoid having to start from scratch every time and make it much easier to synchronize larger mailboxes.