Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rclone copy creating empty directories it didn't create before #7689

Closed
YukiUnHappy opened this issue Mar 25, 2024 · 18 comments
Closed

rclone copy creating empty directories it didn't create before #7689

YukiUnHappy opened this issue Mar 25, 2024 · 18 comments

Comments

@YukiUnHappy
Copy link
Contributor

What is the problem you are having with rclone?

Rclone will now sync directory modification times if the backend supports it
That's a cool new feature, but I think it shouldn't be enabled by default, or it should ignore folders by default where no files need to be transferred.
In the previous version, I could simply copy to the remote within the folder where I needed to upload files, but now I have to remove the other folders in that folder that have no files first, otherwise these empty folders will also be created on the remote due to the sync modification time.
Conversely, if I only need to copy specific files from the remote to the local (using --include), but other irrelevant folders under that path on the remote will also be created on my local machine with the original structure. This did not exist in previous versions and is a bit annoying.

What is your rclone version (output from rclone version)

rclone v1.66.0

  • os/version: debian 10.0 (64 bit)
  • os/kernel: 5.14.9 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

The information has already been mentioned above.

Which cloud storage system are you using? (e.g. Google Drive)

This is not related to a specific backend.

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone copy -vv Source Remote --include E.txt

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

root@localhost:~/TEST# tree
.
└── Source
    ├── A
    │   └── D
    ├── B
    ├── C
    └── E.txt

5 directories, 1 file
root@localhost:~/TEST# rclone copy -vv Source Remote --include E.txt
2024/03/24 23:51:52 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "-vv" "Source" "Remote" "--include" "E.txt"]
2024/03/24 23:51:52 DEBUG : Creating backend with remote "Source"
2024/03/24 23:51:52 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/24 23:51:52 DEBUG : fs cache: renaming cache item "Source" to be canonical "/root/TEST/Source"
2024/03/24 23:51:52 DEBUG : Creating backend with remote "Remote"
2024/03/24 23:51:52 DEBUG : fs cache: renaming cache item "Remote" to be canonical "/root/TEST/Remote"
2024/03/24 23:51:52 DEBUG : A: Making directory with metadata
2024/03/24 23:51:52 INFO  : A: Made directory with metadata (mtime=2024-03-24T23:50:07.378243503-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "A", newDst=A
2024/03/24 23:51:52 DEBUG : B: Making directory with metadata
2024/03/24 23:51:52 INFO  : B: Made directory with metadata (mtime=2024-03-24T23:50:01.04917523-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "B", newDst=B
2024/03/24 23:51:52 DEBUG : C: Making directory with metadata
2024/03/24 23:51:52 INFO  : C: Made directory with metadata (mtime=2024-03-24T23:50:03.246198928-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "C", newDst=C
2024/03/24 23:51:52 DEBUG : E.txt: Need to transfer - File not found at Destination
2024/03/24 23:51:52 DEBUG : A/D: Making directory with metadata
2024/03/24 23:51:52 INFO  : A/D: Made directory with metadata (mtime=2024-03-24T23:50:07.378243503-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "A/D", newDst=A/D
2024/03/24 23:51:52 DEBUG : Local file system at /root/TEST/Remote: Waiting for checks to finish
2024/03/24 23:51:52 DEBUG : E.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2024/03/24 23:51:52 DEBUG : E.txt.cihekil4.partial: renamed to: E.txt
2024/03/24 23:51:52 INFO  : E.txt: Copied (new)
2024/03/24 23:51:52 DEBUG : Local file system at /root/TEST/Remote: Waiting for transfers to finish
2024/03/24 23:51:52 INFO  : A/D: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : C: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : A: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : B: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  :
Transferred:              5 B / 5 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2024/03/24 23:51:52 DEBUG : 7 go routines active
root@localhost:~/TEST# tree
.
├── Remote
│   ├── A
│   │   └── D
│   ├── B
│   ├── C
│   └── E.txt
└── Source
    ├── A
    │   └── D
    ├── B
    ├── C
    └── E.txt

10 directories, 2 files

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@cau118
Copy link

cau118 commented Mar 26, 2024

I'm on macOS 14.3.1 running rclone v1.66.0 and I am experiencing the same issue. Folders are now being copied/synchronized, and at the destination, they are empty. They are empty because the folders do not contain files match the extension I have specified in the include flag on the rclone command line.

rclone v1.66.0

  • os/version: darwin 14.3.1 (64 bit)
  • os/kernel: 23.3.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.22.1
  • go/linking: dynamic
  • go/tags: none

OS used: macOS 14.3.1.

Remote is SSH/SFTP.

Command line used:
rclone copy . pi:testfolder -P --include="*.txt" --dry-run -vv

Results of the command with dry-run:

2024/03/26 09:05:39 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "." "pi:testfolder" "-P" "--include=*.txt" "--dry-run" "-vv"]
2024/03/26 09:05:39 DEBUG : Creating backend with remote "."
2024/03/26 09:05:39 DEBUG : Using config file from "/Users/cmlkh/.config/rclone/rclone.conf"
2024/03/26 09:05:39 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/cmlkh/Downloads/testfolder"
2024/03/26 09:05:39 DEBUG : Creating backend with remote "pi:testfolder"
2024/03/26 09:05:39 DEBUG : sftp://[email protected]:22/testfolder: New connection 172.16.0.227:51097->172.16.0.12:22 to "SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3"
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Shell type "unix" from config
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Relative path resolved to "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Using root directory "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : 13.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 19.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 9.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 NOTICE: 9.txt: Skipped copy as --dry-run is set (size 1.022Ki)
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder01", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder02", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder03", newDst=<nil>
2024/03/26 09:05:40 DEBUG : scratch.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: scratch.txt: Skipped copy as --dry-run is set (size 8.616Ki)
2024/03/26 09:05:40 NOTICE: 19.txt: Skipped copy as --dry-run is set (size 1.653Ki)
2024/03/26 09:05:40 NOTICE: 13.txt: Skipped copy as --dry-run is set (size 1.737Ki)
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Waiting for checks to finish
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Waiting for transfers to finish
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped set directory modification time as --dry-run is set
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s
2024/03/26 09:05:40 NOTICE:
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s

2024/03/26 09:05:40 DEBUG : 15 go routines active
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Closing 1 unused connections
2024/03/26 09:01:30 DEBUG : 15 go routines active
2024/03/26 09:01:30 DEBUG : sftp://[email protected]:22/testfolder: Closing 1 unused connections

@sabitm
Copy link

sabitm commented Mar 28, 2024

I'm not 100% sure if this is related, but bisync also has a weird behavior on 1.66.0. It keep producing a file conflict (which I have to resync to resolve it) between local and remote storage. I just assume that is because of experimental nature of bisync, hence I'm not opening an issue and will happily use the previous version for time being.

@nielash
Copy link
Collaborator

nielash commented Mar 29, 2024

Hi @sabitm, bisync maintainer here. bisync received a very large update in v1.66, and handling of sync conflicts is indeed one of the things that changed. So it may be that the new behavior you're seeing is intentional, but if there's a bug then I'd definitely like to know about it. Please feel free to open an issue or post in the forum with more details!

As for the original issue -- FYI there is an ongoing discussion about it in the forum:
https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/9

@sabitm
Copy link

sabitm commented Apr 2, 2024

Hi @nielash, thanks for maintaining bisync!

I'm not entirely sure how to debug though. It happens randomly enough. I'll try to find a way to reproduce it and will opening an issue if I found something interesting.

@ncw ncw changed the title 1.66.0 has broken the default behavior from before rclone copy creating empty directories it didn't create before Apr 2, 2024
@ncw
Copy link
Member

ncw commented Apr 2, 2024

There are several threads about this on the forum

For rclone v1.66.1 I am going to make sure it obeys the --create-empty-src-dirs flag.

What the defaults for this for rclone sync, rclone copy and rclone move I have yet to decide. Currently they are all false. There is an argument for making this true for rclone sync but that might be a step too far for a point release.

@ncw ncw added the bug label Apr 2, 2024
@ncw ncw added this to the v1.67 milestone Apr 2, 2024
@YukiUnHappy
Copy link
Contributor Author

Thank you @ncw for confirming the bug. Additionally, I have a small issue that I won't open a separate issue for.
The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

root@localhost:~/TEST# tree
.
├── ABC
└── D EF

2 directories, 0 files
root@localhost:~/TEST# rclone copy D
D

If we use the old version of rclone to create a completion script and apply it, we can also complete paths normally in 1.66.

root@localhost:~/TEST# rclone-v1.65.2-linux-amd64/rclone completion bash
root@localhost:~/TEST# . /etc/bash_completion
root@localhost:~/TEST# rclone version
rclone v1.66.0
- os/version: debian 10.0 (64 bit)
- os/kernel: 5.14.9 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none
root@localhost:~/TEST# rclone copy D\ EF/

@ncw
Copy link
Member

ncw commented Apr 3, 2024

@YukiUnHappy

Additionally, I have a small issue that I won't open a separate issue for. The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

I can confirm that. Can you open a new issue about this please? So we don't mix the two up - thank you. (BTW I think this is a bug in cobra and here is the fix spf13/cobra#2126 - Note also that if you type a " first eg "D<TAB> you'll get the correct completions.)

ncw added a commit that referenced this issue Apr 4, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
ncw added a commit that referenced this issue Apr 5, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
@ncw
Copy link
Member

ncw commented Apr 5, 2024

Here is a first attempt at a fix - this obeys --create-empty-src-dirs in the same way that 1.65.2 does

v1.67.0-beta.7854.8d02ad3ba.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing. Running with -golden sorted out the linux tests but not Windows or macOS. Can you help? Also any code review gratefully received :-)

@nielash
Copy link
Collaborator

nielash commented Apr 5, 2024

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing.

It looks to me like the issue is that Size() for the dir is returning an unexpected value. It is expecting 4Ki but getting 96 on mac_amd64:

2024/04/05 15:31:30 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:31:30 | @@ -139 +139 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)
2024/04/05 15:31:30 | @@ -141 +141 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)

and 0 on windows:

2024/04/05 15:30:53 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:30:53 | @@ -139 +139 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)
2024/04/05 15:30:53 | @@ -141 +141 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)

This comment seems potentially relevant:

rclone/backend/local/local.go

Lines 1494 to 1497 in 05e5712

// The value in info.Size() is not always correct
// - Windows links read as 0 size
// - Some virtual filesystems (such ash LucidLink) links read as 0 size
// - Android - some versions the links are larger than readlink suggests

I can take a closer look later!

ncw added a commit that referenced this issue Apr 6, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
@nielash
Copy link
Collaborator

nielash commented Apr 6, 2024

@ncw I'm doing the closer look at fix-7689-empty-dirs now and I think there's an issue with the --create-empty-src-dirs=false case. Here's a test that demonstrates it: nielash@16d856a

The same test would have passed previously.

@nielash
Copy link
Collaborator

nielash commented Apr 6, 2024

@ncw added a few other inline comments:
dd463b2
13c72a9

and a test that (I think) catches another regression: nielash@cd0891d

ncw added a commit that referenced this issue Apr 16, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
@ncw
Copy link
Member

ncw commented Apr 16, 2024

I finally got round to fixing up this code. Can you take a quick look @nielash ?

I think it is doing the right thing now. I left the fixes unsquashed so you can see them and I put your tests in (though I took one line out as I don't think it was correct any more).

v1.67.0-beta.7901.7042810e8.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

Thank you :-)

@nielash
Copy link
Collaborator

nielash commented Apr 17, 2024

@ncw I think it mostly looks good, but the issue I mentioned here is still not fixed.

Here's a test that demonstrates it: nielash@d670d07 (same test would have passed previously)

...which passes once this is reverted: nielash@ff4198d

Otherwise I think it looks good!

ncw added a commit that referenced this issue Apr 17, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
@ncw
Copy link
Member

ncw commented Apr 17, 2024

Thanks for the review @nielash

This feels like the never ending bug!

I've fixed the issue that you pointed out (I realize I misunderstood the purpose of s.modifiedDirs it is to keep the modified status of the directories themselves, not the parents like we need for delayed modtime setting).

Please take another look!

I also fixed a bug pointed out in the forum https://forum.rclone.org/t/empty-dirs-not-wanted/45059/14

Hopefully if this checks out I'll squash it down, neaten it up and then that is the last bit needed for v1.66.1

v1.67.0-beta.7906.91d78fd38.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

@nielash
Copy link
Collaborator

nielash commented Apr 17, 2024

Added a comment for you inline. This is a tricky one indeed!

@ncw
Copy link
Member

ncw commented Apr 22, 2024

@nielash I can't find your comment! I thought I saw a notification but it has gone! Please post a link - thank you :-)

@nielash
Copy link
Collaborator

nielash commented Apr 22, 2024

@ncw here you go! 91d78fd#r141149941

ncw added a commit that referenced this issue Apr 23, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
ncw added a commit that referenced this issue May 3, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes #7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
@ncw ncw closed this as completed in 10eb474 May 3, 2024
@ncw
Copy link
Member

ncw commented May 3, 2024

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.66.1

Thanks for all your help @nielash :-)

Fornax96 pushed a commit to Fornaxian/rclone that referenced this issue Jul 30, 2024
…false

In v1.66.0 the changes to enable metadata preservation on directories
introduced a regression, namely that empty directories were created
despite the state of the --create-empty-src-dirs flag.

This patch fixes the problem by letting the normal rclone directory
creation create the directories and fixing up their timestamps and
metadata afterwards if --create-empty-src-dirs=false.

Fixes rclone#7689
See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants