backend: Add Apple iCloud Drive backend #7717
Conversation
|
Nice work :-) Some quick questions
Thank you |
not sure what client_id you are referring to? and no I havent run any integration test yet. |
|
fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering So if a file exists I move it to iCloud trash. And upload a new version of the file. |
|
@ncw how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent. Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching) cachedir doesn't seem to refresh after invalidation, see: rclone/backend/icloud/icloud.go Line 424 in 37816e9 When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated. |
This one at the top of the source! clientId = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"
Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done
Seems OK!
Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.
Not sure! |
etags are needed for file operations unfortunately, but I can just refresh it when it got stale. the clientid is a static that icloud.com uses. So I reuse it to mimic the browser. |
|
@ncw small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0 But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty) Not sure howto handle this. |
That sounds correct.
Are you using So you are seeing a zero length file in the finder? I'm not sure what you mean by a partial file? Can you reproduce this without the |
|
Its slow but finished alle interfaces and all integration test are passing 🎉 Only documentation left before its ready for review. |
Well done! You are running the full suite from
:-) |
|
@lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files. |
bug reports are welcome. Will fix with the following pass, tnx |
|
@epetousis this has been fixed |
|
crap i broke my icloud. Too many trash items and now the endpoint is broken lol the icloud web api is a mess |
lostb1t
changed the title
|
Fix works great. Much appreciated! |
|
Hi, thanks. |
@cyayon Could you post a log from the command with the -vv flag (e.g. output from rclone -vv ls |
|
Great ! Another thing, I found the download very slow (about 1MB/s) for a large 3GB file. When using Finder, the download speed is much higher (about 20MB/s). Is there some reason ? Thank you very much. |
if you say finder so you mean apples native drive integration? |
|
Yes on macOS (and files.app on iOS). |
|
different apis. Apple probaply using a private api we dont have access to. This uses the web api. So if you want to compare use icloud.com, those speeds should |
|
I understand. |
|
With the current HEAD commit, I did When I try to copy a non-zero-bytes file into it, when mounted, it says ENV: Xubuntu 24.04 (6.8.0-31-generic) |
|
@hurryman2212 > do you maybe have advanced data protection enabled? thats currently not supported and need to be disabled. |
It has been almost a year since they introduced it and I enabled it, I think, so I have completely forgotten about it.. Thank you! Indeed, it was due to ADP. Now, it is working fine, although performance-wise, there are some issues as @cyayon reported. On my environment, |
|
Hi, Is it normal ? I thank that it will be done without human interaction. Thanks. |
|
@cyayon it should selfrenew. Could you send me the log when this occurs again? |
Hi, Here is a log : |
|
Hi, Same issue today. I have to refresh token and 2FA again... |
|
Hi @lostb1t I was trying to test it and got this error
How can I debug it? And how can I help in the project overall? |
did you run the config succesfully? (2fa) |
This happened when trying to create the remote. Below is the attempt to create the remote. It never asked for the 2fa and no code was sent to my device. |
|
Make sure 2fa is enabled on your account. I made the password required just now, could you pull the changes and try again? |
Seems something not going right when reauthenticating. Not sure why on first glance. Cookies are |
Yes, I had 2fa setup in my account. And yes, the change made it work, it was a couple of things that made it not work.
I successfully mounted the drive and now will use it an make some tests. If anything goes wrong, I'll report it back. Thank you very much for this rClone backend! I am recently migrating to Linux after the hole Microsoft Recall fiasco, and I rely some of my daily workflow on iCloud, this was the missing piece for the migration. PS: Do you have GitHub sponsor, ko-fi, ...? |
|
k this sucks but it seems the trust token is only valid for 30 days. After that it has to be renewed with an MFA code.... not ... ideal... im open to suggestions on howto handle this the least intrusive way |
|
Hi, It seems that the other GitHub project icloud_photos_downloader have also the same issue. I am using this while waiting for native rclone support for iCloud photo (backup). And each 30 days (I think) I have to renew with MFA (for me, my wife and son...) I will ask ask directly on this other GitHub project if there is a solution. I will post here the issue link. |
|
Someone already ask for the expiration of the cookie : icloud-photos-downloader/icloud_photos_downloader#803 (comment) I ask icloud-photos-downloader/icloud_photos_downloader#803 (comment) |
|
An idea here (not great) : boredazfcuk/docker-icloudpd#524 Another solution could be to add WebUI for password/MFA : icloud-photos-downloader/icloud_photos_downloader#805 to be able to renew directly on rclone without the need of an apple device... |
|
I spent a few minutes debugging the "iCloud Drive" app for Windows, and it looks like they're using gRPC & protobuf in those. Unfortunately my reverse engineering skills are pretty limited, but maybe it's worth pursuing this in the future to improve the performance and speed here. |
Apple uses SSL pinning so its unfortunately impossible to reverse engineer it with a MITM. The only way this was possible |
I think by using something like IDA to inspect actual control flow it might be possible to reverse engineer the protocol from the binary, but it's very tedious and definitely above my experience with this. Just leaving it for inspiration. |
|
Hi, it's me again. I'm trying this integration, and it is in a good working condition with minor hiccups. I don't know if i did something wrong, but now I'm getting errors when trying to sync. Here is the log rclone-iCloudDrive.log One specific folder is not able to send files, but all my iCloud is not seeing changes that I make on other locations. What can I do to help debug this? |
|
@fcrespo82 can you post your mount command |
What is the purpose of this change?
Add support for a new backend, iCloud Drive
Was the change discussed in an issue or in the forum before?
#1778
Checklist
Notes
This feature branch is in a working state, feedback on finished implementations is appreciated. Ill keep it in draft untill everything is implemented and ill update the TODO accordingly
This PR is only for iCloud Drive. Ill continue work on Photos once this PR is merged.
Todo
Rclone Interfaces
Fs
Object
Mover
DirMove
Copier