Conversation
abhijat
force-pushed
the
abhijat/hack/defrag-2
branch
25 times, most recently
from
0183083 to
05d88dc
Compare
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
abhijat
force-pushed
the
abhijat/hack/defrag-2
branch
from
05d88dc to
f326f23
Compare
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
abhijat
force-pushed
the
abhijat/hack/defrag-2
branch
from
a7f5a82 to
3bce4c0
Compare
Contributor
Author
|
afl error is from build failure, not fuzzer https://github.com/dragonflydb/dragonfly/actions/runs/25446794970/job/74652993045 |
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
abhijat
force-pushed
the
abhijat/hack/defrag-2
branch
from
3bce4c0 to
76db21e
Compare
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
This comment was marked as resolved.
This comment was marked as resolved.
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Introduce two experiment scripts for measuring and comparing defrag algorithm performance: - defrag_run.sh: end-to-end single experiment driver. Starts a CPU-pinned single-shard Dragonfly, creates heap fragmentation via defrag_baseline.py, then iterates MEMORY DEFRAGMENT until arena waste falls below a target or a cycle cap is reached. Results written as JSONL. Accepts phased|legacy mode and exposes all key params as CLI flags (-m, -c, -w, -p, -t, -s, -o, -b) with env-var fallback. - defrag_sweep.sh: batch sweep over MUL × mode combinations. Calls defrag_run.sh for every combination, captures per-run stdout to .log and JSONL files under runs/sweep/. MUL series, cycle count, mode subset, and output dir are all configurable via flags or env vars. Extra flags can be forwarded to defrag_run.sh after --. Signed-off-by: Gil Levkovich <69595609+glevkovich@users.noreply.github.com>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
ClassifyForTarget() decides whether a census page is a defrag target. It covered three cases (no observed blocks, already empty, stale observation) but fell through to kKeep when observed_movable_blocks < used_blocks, i.e. when mimalloc reports more live blocks than the census ever saw as movable. Those extra blocks are immovable (internal metadata, objects arriving after the census walk). Selecting such a page as a target causes EVACUATE to spin trying to drain blocks it can never move. Add the missing branch to return kHasImmovableData, making kKeep reachable only when observed == used (all live blocks were observed as movable). Fixes TargetPlanTest.AppliesFilterClassification. Signed-off-by: Gil Levkovich <69595609+glevkovich@users.noreply.github.com>
The phased defragmenter (CENSUS + EVACUATE) can now optionally defrag key allocations (it->first) in addition to values, controlled by --defrag_keys (default: false). - Add ShouldDefragKeys() virtual to PageUsage; CensusTaker and Evacuator override it to read the flag - Wire key defrag into RunPrimeTableSlice and RunPrimeTableHinted, guarded by defrag_keys in both paths - Track per-key/value reallocation counts and bytes moved in DbSliceResult, DefragCycleStats, and EVACUATE log lines - Add KeyDefragIntegrity test: verifies no corruption and that key/value realloc stats are non-zero after defrag - Add -- passthrough support to defrag_run.sh for forwarding extra flags to the dragonfly binary The flag defaults to false. Benchmarks at MUL=15 show key defrag hurts throughput for short-key/large-value workloads: key moves consume quota without emptying pages. Enable for workloads where keys are a significant fraction of object memory (long metric names, search indexes, time-series). Signed-off-by: Gil Levkovich <69595609+glevkovich@users.noreply.github.com>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.