Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ACQ ID package_images.py, package_rsync.py, and transfer_rsync.py #56

Open
wants to merge 35 commits into
base: main
Choose a base branch
from

Conversation

dreyjo
Copy link
Contributor

@dreyjo dreyjo commented Jun 4, 2024

No description provided.

@dreyjo dreyjo requested a review from arhonemus June 4, 2024 15:48
@dreyjo
Copy link
Contributor Author

dreyjo commented Jun 6, 2024

new streams folder in destination not being created and stream files not being moved.
Screen Shot 2024-06-06 at 4 38 06 PM

return parser.parse_args()


def run_rsync(source: Path, dest: Path, quiet: bool = None) -> None:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently rsync seems to be unable to transfer some files. This will throw an error but won't indicate what files aren't transferred or why. There may be additional rsync flags to add to this function.

args = parse_args()

base_dir = pb.create_package_dir(args.dest, args.carrierid)
pb.create_bag_in_objects(args.payload, base_dir, args.log, "rsync")
Copy link
Contributor Author

@dreyjo dreyjo Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the script creates a packages with no metadata folder, an object folder with an empty data folder. payload is not being moved but manifest is generated. Note in testing I used the rsync log generated by transfer_rsync.py.

Issues may be connected to an OS Error thrown by os_rename in move_files in move_data_files in package_base.py, this error is thrown because of an .fseventssd file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the script creates a packages with no metadata folder, an object folder with an empty data folder. payload is not being moved but manifest is generated. Note in testing I used the rsync log generated by transfer_rsync.py.

Issues may be connected to an OS Error thrown by os_rename in move_files in move_data_files in package_base.py, this error is thrown because of an .fseventssd file.
Current error:

% poetry run python3 src/digarch_scripts/package/package_filetransfer.py --payload '/Volumes/BJF MEMOS' --carrierid ACQ_5181_541623 --dest '/Volumes/DigArchDiskStation/Staging/ingest/fileTransfers' --log '/Users/djonathan/Desktop/ACQ_5181_541623_rsync.
log'
/Volumes/BJF MEMOS/.fseventsd /Volumes/DigArchDiskStation/Staging/ingest/fileTransfers/ACQ_5181/ACQ_5181_541623/objects/data/.fseventsd
Traceback (most recent call last):
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_filetransfer.py", line 34, in
main()
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_filetransfer.py", line 28, in main
pb.create_bag_in_objects(args.payload, base_dir, args.log, "rsync")
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 260, in create_bag_in_objects
create_bag_in_dir(
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 236, in create_bag_in_dir
move_data_files(paths, bag_dir)
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 214, in move_data_files
return move_files(data_paths, pkg_dir, "data")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 197, in move_files
move_file(file_path, pkg_dir, dest)
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 190, in move_file
file_path.rename(new_file_path)
File "/usr/local/Cellar/[email protected]/3.12.5/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1363, in rename
os.rename(self, target)
OSError: [Errno 18] Cross-device link: '/Volumes/BJF MEMOS/.fseventsd' -> '/Volumes/DigArchDiskStation/Staging/ingest/fileTransfers/ACQ_5181/ACQ_5181_541623/objects/data/.fseventsd'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants