Skip to content

Commit

Permalink
- Add option --skip-projections to create, upload, restore co…
Browse files Browse the repository at this point in the history 
…mmands, with table pattern to allow make backup

  without projection, restore supported only in `clickhouse-server` 24.3+, fix #861

Signed-off-by: Slach <[email protected]>
  • Loading branch information
@Slach
Slach committed Jan 25, 2025
1 parent 1eb6e93 commit 88d2649
Show file tree
Hide file tree
Showing 8 changed files with 177 additions and 154 deletions.
3 changes: 2 additions & 1 deletion ChangeLog.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
IMPROVEMENTS

- Add option `--skip-projections` to `create`, `upload`, `restore` commands, with table pattern to allow make backup
without projection fix [861](https://github.com/Altinity/clickhouse-backup/issues/861)
without projection, restore supported only in `clickhouse-server` 24.3+,
fix [861](https://github.com/Altinity/clickhouse-backup/issues/861)
- remove `S3_PART_SIZE` and `AZBLOB_BUFFER_SIZE` parameter from configuration and significant decrease memory usage
during upload and download, fix [854](https://github.com/Altinity/clickhouse-backup/issues/854)
- add `--configs-only` and `--rbac-only` options to `upload` and `download` command,
Expand Down
86 changes: 43 additions & 43 deletions Manual.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Look at the system.parts partition and partition_id fields for details https://c
--rbac-only Backup RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Backup 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections Skip make hardlinks to *.proj/* files during backup creation
--skip-projections db_pattern.table_pattern:projections_pattern Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true
```
Expand All @@ -69,17 +69,17 @@ If PARTITION BY clause returns tuple with multiple fields, then use --partitions
If you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*
Values depends on field types in your table, use single quotes for String and Date/DateTime related types
Look at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/
--diff-from value Local backup name which used to upload current backup as incremental
--diff-from-remote value Remote backup name which used to upload current backup as incremental
--schema, -s Backup and upload metadata schema only, will skip data backup
--rbac, --backup-rbac, --do-backup-rbac Backup and upload RBAC related objects
--configs, --backup-configs, --do-backup-configs Backup and upload 'clickhouse-server' configuration files
--rbac-only Backup RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Backup 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignore when 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections Skip make and upload hardlinks to *.proj/* files during backup creation
--delete, --delete-source, --delete-local explicitly delete local backup during upload
--diff-from value Local backup name which used to upload current backup as incremental
--diff-from-remote value Remote backup name which used to upload current backup as incremental
--schema, -s Backup and upload metadata schema only, will skip data backup
--rbac, --backup-rbac, --do-backup-rbac Backup and upload RBAC related objects
--configs, --backup-configs, --do-backup-configs Backup and upload 'clickhouse-server' configuration files
--rbac-only Backup RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Backup 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignore when 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections db_pattern.table_pattern:projections_pattern Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--delete, --delete-source, --delete-local explicitly delete local backup during upload
```
### CLI command - upload
Expand All @@ -103,12 +103,12 @@ If PARTITION BY clause returns tuple with multiple fields, then use --partitions
If you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*
Values depends on field types in your table, use single quotes for String and Date/DateTime related types
Look at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/
--schema, -s Upload schemas only
--rbac-only, --rbac Upload RBAC related objects only, will skip upload data, will backup schema only if --schema added
--configs-only, --configs Upload 'clickhouse-server' configuration files only, will skip upload data, will backup schema only if --schema added
--skip-projections Skip make and upload hardlinks to *.proj/* files during backup creation
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--delete, --delete-source, --delete-local explicitly delete local backup during upload
--schema, -s Upload schemas only
--rbac-only, --rbac Upload RBAC related objects only, will skip upload data, will backup schema only if --schema added
--configs-only, --configs Upload 'clickhouse-server' configuration files only, will skip upload data, will backup schema only if --schema added
--skip-projections db_pattern.table_pattern:projections_pattern Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume, --resumable Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--delete, --delete-source, --delete-local explicitly delete local backup during upload
```
### CLI command - list
Expand Down Expand Up @@ -170,16 +170,16 @@ If PARTITION BY clause returns tuple with multiple fields, then use --partitions
If you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*
Values depends on field types in your table, use single quotes for String and Date/DateTime related types
Look at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/
--schema, -s Restore schema only
--data, -d Restore data only
--rm, --drop Drop exists schema objects before restore
-i, --ignore-dependencies Ignore dependencies when drop exists schema objects
--rbac, --restore-rbac, --do-restore-rbac Restore RBAC related objects
--configs, --restore-configs, --do-restore-configs Restore 'clickhouse-server' CONFIG related files
--rbac-only Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--skip-projections Skip make hardlinks to *.proj/* files during backup restoring
--resume, --resumable Will resume download for object disk data
--schema, -s Restore schema only
--data, -d Restore data only
--rm, --drop Drop exists schema objects before restore
-i, --ignore-dependencies Ignore dependencies when drop exists schema objects
--rbac, --restore-rbac, --do-restore-rbac Restore RBAC related objects
--configs, --restore-configs, --do-restore-configs Restore 'clickhouse-server' CONFIG related files
--rbac-only Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--skip-projections db_pattern.table_pattern:projections_pattern Skip make hardlinks to *.proj/* files during backup restoring, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume, --resumable Will resume download for object disk data
```
### CLI command - restore_remote
Expand All @@ -203,16 +203,16 @@ If PARTITION BY clause returns tuple with multiple fields, then use --partitions
If you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*
Values depends on field types in your table, use single quotes for String and Date/DateTime related types
Look at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/
--schema, -s Download and Restore schema only
--data, -d Download and Restore data only
--rm, --drop Drop schema objects before restore
-i, --ignore-dependencies Ignore dependencies when drop exists schema objects
--rbac, --restore-rbac, --do-restore-rbac Download and Restore RBAC related objects
--configs, --restore-configs, --do-restore-configs Download and Restore 'clickhouse-server' CONFIG related files
--rbac-only Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--skip-projections Skip make hardlinks to *.proj/* files during backup restoring
--resume, --resumable Save intermediate download state and resume download if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
--schema, -s Download and Restore schema only
--data, -d Download and Restore data only
--rm, --drop Drop schema objects before restore
-i, --ignore-dependencies Ignore dependencies when drop exists schema objects
--rbac, --restore-rbac, --do-restore-rbac Download and Restore RBAC related objects
--configs, --restore-configs, --do-restore-configs Download and Restore 'clickhouse-server' CONFIG related files
--rbac-only Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added
--configs-only Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added
--skip-projections db_pattern.table_pattern:projections_pattern Skip make hardlinks to *.proj/* files during backup restoring, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
--resume, --resumable Save intermediate download state and resume download if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'
```
### CLI command - delete
Expand Down Expand Up @@ -305,11 +305,11 @@ If PARTITION BY clause returns tuple with multiple fields, then use --partitions
If you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*
Values depends on field types in your table, use single quotes for String and Date/DateTime related types
Look at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/
--schema, -s Schemas only
--rbac, --backup-rbac, --do-backup-rbac Backup RBAC related objects only
--configs, --backup-configs, --do-backup-configs Backup `clickhouse-server' configuration files only
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections Skip make and upload hardlinks to *.proj/* files during backup creation
--schema, -s Schemas only
--rbac, --backup-rbac, --do-backup-rbac Backup RBAC related objects only
--configs, --backup-configs, --do-backup-configs Backup `clickhouse-server' configuration files only
--skip-check-parts-columns Skip check system.parts_columns to allow backup inconsistent column types for data parts
--skip-projections db_pattern.table_pattern:projections_pattern Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax
```
### CLI command - server
Expand Down
Loading

0 comments on commit 88d2649

Please sign in to comment.