Google Drive sync

[MadLittleMods]

Setup

Configure

First setup the rclone config and add your Google Drive account as a remote. The docs are easy to follow but I also included the output from when I set things up below.

• https://rclone.org/drive/
  • https://rclone.org/drive/#making-your-own-client-id

rclone config setup

$ rclone config
2023/05/08 13:26:18 NOTICE: Config file "/home/eric/.config/rclone/rclone.conf" not found - using defaults
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> google-drive-remote

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[...]
18 / Google Drive
   \ (drive)
[...]
Storage> 18

Option client_id.
Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a value. Press Enter to leave empty.
client_id> xxx

Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret> xxx

Option scope.
Scope that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Full access all files, excluding Application Data Folder.
   \ (drive)
 2 / Read-only access to file metadata and file contents.
   \ (drive.readonly)
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ (drive.file)
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ (drive.appfolder)
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ (drive.metadata.readonly)
scope> 1

Option service_account_file.
Service Account Credentials JSON file path.
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a value. Press Enter to leave empty.
service_account_file> 

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.

y) Yes (default)
n) No
y/n> y

2023/05/08 13:33:30 NOTICE: Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.
2023/05/08 13:33:30 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxx
2023/05/08 13:33:30 NOTICE: Log in and authorize rclone for access
2023/05/08 13:33:30 NOTICE: Waiting for code...
2023/05/08 13:35:14 NOTICE: Got code
Configure this as a Shared Drive (Team Drive)?

y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: drive
- client_id: xxx
- client_secret: xxx
- scope: drive
- token: {"access_token":"xxx","token_type":"Bearer","refresh_token":"xxx","expiry":"2023-05-08T14:35:13.748643158-05:00"}
- team_drive: 
Keep this "google-drive-remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:

Name                 Type
====                 ====
google-drive-remote  drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Setup bisync

Then setup rclone bisync between your Google drive remote and a local folder and do an initial sync with the --resync flag to get things started.

• https://rclone.org/bisync/

A word of warning from the docs:

rclone bisync remote1:path1 remote2:path2 --resync

Warning: Path1 files may overwrite Path2 versions.
Consider using --verbose or --dry-run first.

This will effectively make both Path1 and Path2 filesystems contain a matching superset of all files. Path2 files that do not exist in Path1 will be copied to Path1, and the process will then sync the Path1 tree to Path2.

In terms of how to refer to your remote, this notice explained it pretty well:

NOTICE: "google-drive-remote" refers to a local folder, use "google-drive-remote:" to refer to your remote or "./google-drive-remote" to hide this warning

With all that being said, we can use the following command:

$ rclone bisync google-drive-remote: ~/google-drive/ --resync --drive-acknowledge-abuse
# Wait for it to finish...
2023/05/08 14:24:54 INFO  : Bisync successful
2023/05/08 14:24:54 INFO  : 
Transferred:   	   31.500 KiB / 31.500 KiB, 100%, 117 B/s, ETA 0s
Checks:              6411 / 6411, 100%
Transferred:            1 / 1, 100%
Elapsed time:      6m39.8s


# If it fails, ----------------------------------------------------------------------------------------
# we can double-check what it will do on resync with a dry-run and then do a `--resync` again
$ rclone bisync google-drive-remote: ~/google-drive/ --resync --drive-acknowledge-abuse --dry-run --verbose
# Inspect the output to make sure it's doing sane things
$ rclone bisync google-drive-remote: ~/google-drive/ --resync --drive-acknowledge-abuse
# Wait for it to finish...

Automatically run the sync periodically/regularly

Then we need to setup a scheduled task to run the sync periodically. We're going to be using a systemd timer instead of a cron for a variety of benefits.

1. First create the service and timer unit files:
   /etc/systemd/system/eric-rclone-bisync.service

   [Unit]
   Description=Run rclone bisync to sync Eric's Google Drive
   
   [Service]
   User=eric
   Group=eric
   ExecStart=rclone bisync google-drive-remote: ~/google-drive/ --drive-acknowledge-abuse
   

   /etc/systemd/system/eric-rclone-bisync.timer

   [Unit]
   Description=rclone bisync service timer
   
   [Timer]
   OnBootSec=0min
   # Run every 6 hours
   OnCalendar=*-*-* */6:00:00
   Unit=eric-rclone-bisync.service
   
   [Install]
   WantedBy=multi-user.target
   

2. Run sudo systemctl daemon-reload to make systemd aware of the new files (this makes systemd reload all files and re-consider their dependencies)

3. Enable the timer unit so the timer is enabled at boot every time: sudo systemctl enable eric-rclone-bisync.timer

4. To start the the timer unit immediately (without booting): sudo systemctl start eric-rclone-bisync.timer

5. To see the status of the timer and service: sudo systemctl status eric-rclone-bisync.timer eric-rclone-bisync.service

---

Other references:

• https://www.reddit.com/r/rclone/comments/v93mxu/comment/j6v3amf/

Problems

Unable to bisync Google Docs

When trying to sync after --resync, a bunch of Ignoring incorrect line for every file that then turns into Path2 file not found in Path1 later.

See rclone/rclone#5696