Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor fixes to standalone GC documentation #8321

Merged
merged 2 commits into from
Oct 30, 2024
Merged

Minor fixes to standalone GC documentation #8321

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5a3006a
minor fixes to standalone GC documentation
yonipeleg33 Oct 29, 2024
96be01f
wording
yonipeleg33 Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/howto/garbage-collection/standalone-gc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,10 @@ The minimal required permissions on LakeFS are:
"statement": [
{
"action": [
"fs:ReadConfig",
"fs:ReadRepository",
"retention:PrepareGarbageCollectionCommits",
"retention:PrepareGarbageCollectionUncommitted",
"fs:ReadConfig",
"fs:ReadRepository",
"fs:ListObjects",
"fs:ReadConfig"
],
Expand Down Expand Up @@ -212,8 +212,8 @@ Usage: \
`lakefs-sgc run <repository>`

Flags:
- `--cache-dir`: directory to cache read files and metadataDir (default is $HOME/.lakefs-sgc/data/)
- `--parallelism`: number of parallel downloads for metadataDir (default 10)
- `--cache-dir`: directory to cache read files (default is `$HOME/.lakefs-sgc/data/`)
- `--parallelism`: number of parallel downloads for metadata files (default 10)
- `--presign`: use pre-signed URLs when downloading/uploading data (recommended) (default true)

### How to Run Standalone GC
Expand All @@ -235,7 +235,7 @@ treeverse/lakefs-sgc:<tag> run <repository>

#### Mounting the `~/.aws` directory

When working with S3-compatible clients, it's often more convenient to mount the ~/.aws` file and pass in the desired profile.
When working with S3-compatible clients, it's often more convenient to mount the `~/.aws` directory and pass in the desired profile.

First, change the permissions for `~/.aws/*` to allow the docker container to read this directory:
```bash
Expand Down Expand Up @@ -298,6 +298,6 @@ run_id=<GC run id>
# Download the CSV file
aws s3 cp "$storage_ns/_lakefs/retention/gc/reports/$run_id/deleted.csv" "./run_id-$run_id.csv"

# Move all addresses to the output bucket under the run_id prefix
cat run_id-$run_id.csv | tail -n +2 | head -n 10 | xargs -I {} aws s3 mv "$storage_ns/{}" "$output_bucket/run_id=$run_id/"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head -n 10 was there for my local testing, it's taking only the first 10 addresses.

# Move all addresses to the output bucket under the "run_id=$run_id" prefix
cat run_id-$run_id.csv | tail -n +2 | xargs -I {} aws s3 mv "$storage_ns/{}" "$output_bucket/run_id=$run_id/"
```
Loading