files cache with size #7846
Merged
files cache with size #7846
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is trying to prepare a bigger change in how borg works: giving up with precise refcounting and removing chunk size information from the chunks index/cache (so borg could just work with the set of existing chunks with no further information about them in the chunks index/cache). A first step into that direction is adding the chunk sizes to the files cache (it currently only has the chunk ids). If borg create detects a file is unchanged, we need to have that information at hand to create the item's chunk list without reading/chunking the file. AdHocWithFilesCache tries to be similar to AdHocCache, but additionally has the files cache, so it is super fast for unchanged files. AdHocCache is DEPRECATED now. |
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
3 times, most recently
from
527b612
to
aef7d7a
Compare
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #7846 +/- ##
==========================================
- Coverage 83.76% 81.61% -2.16%
==========================================
Files 67 67
Lines 12074 12156 +82
Branches 2193 2192 -1
==========================================
- Hits 10114 9921 -193
- Misses 1367 1649 +282
+ Partials 593 586 -7 ☔ View full report in Codecov by Sentry. |
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
5 times, most recently
from
cdf705b
to
f404810
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
7490f37
to
c5e5ce6
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
c5e5ce6
to
5c9dcc0
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
5c9dcc0
to
02184e0
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
2 times, most recently
from
dc25787
to
1389bd1
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
2 times, most recently
from
4fd25be
to
2b27872
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
2b27872
to
6fae834
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
2 times, most recently
from
2f78282
to
cce4fcc
Compare
the files cache used to have only the chunk ids, so it had to rely on the chunks index having the size information - which is problematic with e.g. the AdhocCache (has size==0 for all not new chunks) and blocked using the files cache there.
incref: returns (id, size), so it needs the size if it can't get it from the chunks index. also needed for updating stats. decref: caller does not always have the chunk size (e.g. for metadata chunks), as we consider 0 to be an invalid size, we call with size == 1 in that case. thus, stats might be slightly off.
when given, force using the AdHocCache.
if a chunk is missing in repo, it will also be missing in a ad-hoc built chunks index.
thus: - no cache.path - skip on-disk cache corruption tests for AdHocCache
Also: - move common code to ChunksMixin - always use ._txn_active (not .txn_active) Some tests are still failing.
if we use AdHocCache or NewCache, we do not have precise refcounting. thus, we do not delete repo objects as their refcount does not go to zero. check --repair will just remove the orphans.
Only LocalCache implements these.
NewCache does not do precise refcounting, thus chunks won't be deleted from the repo at "borg delete" time. "borg check --repair" would remove such chunks IF they are orphans.
NewCache does not do precise refcounting, thus it does not know about unique chunks. For now, let's just test num_chunks, but not unique_chunks.
NewCache and AdHocCache do not have a persistent chunks index, so both check_cache and test_check_cache are pointless.
We can not use unique_chunks counter with NewCache, thus we use a simpler (and weaker) assertion.
removed some code borg had for backwards compatibility with old borg versions (that had previous_location only in the cache). now the repo location is only checked against the location file in the security dir, simplifying the code and also fixing a related test failure with NewCache. also improved test_repository_move to test for aborting in case the repo location changed unexpectedly.
removed some code borg had for backwards compatibility with old borg versions (that had key_type only in the cache). now the repo key_type is only checked against the key-type file in the security dir, simplifying the code.
removed some code borg had for backwards compatibility with old borg versions (that had timestamp only in the cache). now the manifest timestamp is only checked against the manifest-timestamp file in the security dir, simplifying the code.
Add new borg create option '--prefer-adhoc-cache' to prefer the AdHocCache over the NewCache implementation. Adjust a test to match the previous default behaviour (== use the AdHocCache) with --no-cache-sync.
- skip test_cache_chunks if there is no persistent chunks cache file - init self.chunks for AdHocCache - remove warning output from AdHocCache.__init__, it gets mixed with JSON output and fails the JSON decoder.
BORG_CACHE_IMPL allows users to choose the client-side cache implementation from 'local', 'newcache' and 'adhoc'.
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
c72b244
to
78954b6
Compare
Also: support a "cli" env var value, that does not determine the implementation from the env var, but rather from cli options (similar to as it was before adding BORG_CACHE_IMPL).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.