Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16876 dtx: misc fixes for DTX related logic #15942

Merged
merged 1 commit into from
Feb 21, 2025
Merged

DAOS-16876 dtx: misc fixes for DTX related logic #15942

merged 1 commit into from
Feb 21, 2025

Conversation

Nasf-Fan
Copy link
Contributor

  1. Do not clear dae_need_release flag on DTX entry in DRAM until related local transaction has been successfully committed. Otherwise, it will skip subsequent DTX commit or abort this entry.

  2. Do not allow the case that some DTX entry does not exist in DRAM but related modification target still references it. Add assertion check during vos_dtx_check_availability() and vos_dtx_deregister_record().

  3. Replace daos_gettime_coarse() with daos_wallclock_secs() for DTX time. Then DTX cleanup and DTX aggregation logic can use them after restart.

  4. Bypass MRU operation during DTX LRU array lookup, that is unnecessary and may hide some potential bugs.

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@github-actions GitHub Actions
Copy link

Ticket title is 'LRZ: m02r01s07dao engine coredumps with vos EMRG src/vos/ilog.c:411 ilog_open() Assertion'
Status is 'In Progress'
Labels: 'ALCF,LRZ,alcf_track,google-cloud-daos,scrubbed_2.8'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-16876

@github-actions github-actions bot added the priority Ticket has high priority (automatically managed) label Feb 19, 2025
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15942/2/execution/node/1558/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15942/2/execution/node/1537/log

@Nasf-Fan
DAOS-16876 dtx: misc fixes for DTX related logic
fc052c9 
1. Do not clear dae_need_release flag on DTX entry in DRAM until related
   local transaction has been successfully committed. Otherwise, it will
   skip subsequent DTX commit or abort this entry.

2. Do not allow the case that some DTX entry does not exist in DRAM but
   related modification target still references it. Add assertion check
   during vos_dtx_check_availability() and vos_dtx_deregister_record().

3. Replace daos_gettime_coarse() with daos_wallclock_secs() for DTX time.
   Then DTX cleanup and DTX aggregation logic can use them after restart.

4. Bypass MRU operation during DTX LRU array lookup, that is unnecessary
   and may hide some potential bugs.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/DAOS-16876_5 branch from a645830 to fc052c9 Compare February 20, 2025 06:42
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15942/4/execution/node/1541/log

@Nasf-Fan
Copy link
Contributor Author

https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15942/4/execution/node/1541/log

Hit some hardware issue on wolf-216:
wolf-216 storage: code = 304 description = "NVMe SSDs [0000:65:00.0 0000:a5:00.0] not found"

@Nasf-Fan
Copy link
Contributor Author

pool-create test failure because of out of space on wolf-215:
2025-02-20 16:01:57,071 dmg_utils L0615 ERROR| pool create failed: DER_NOSPACE(-1007): No space on storage target

@Nasf-Fan
Copy link
Contributor Author

Restart CI test for the hardware related issues.

NiuYawei
NiuYawei reviewed Feb 21, 2025
View reviewed changes
src/vos/vos_dtx.c
@@ -825,9 +826,6 @@ dtx_rec_release(struct vos_container *cont, struct vos_dtx_act_ent *dae, bool ab
UMOFF_P(dbd_off), DP_UUID(cont->vc_id));
}

if (rc == 0)
dae->dae_need_release = 0;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the key change? The dae_need_release should be cleared after local tx committed instead of here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, originally we cleared such flag too early.

src/vos/vos_dtx.c
return ALB_AVAILABLE_CLEAN;
}
D_ASSERTF(found, "Non-committed DTX must be in active table: lid=%x, epoch="
DF_U64 ", boundary " DF_U64 "\n", entry, epoch, cont->vc_solo_dtx_epoch);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I'm wondering why we need to tolerate this before? Is it because of the bug you fixed in this PR or other reason?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may because we wanted to make the system to be more robust at that time or related optimized/tolerated case has been removed. Anyway, consider current implementation, add assertion here will help us to detect corrupted DTX entry more early.

src/vos/vos_dtx.c
found = lrua_lookupx(vos_hdl2cont(coh)->vc_dtx_array,
(entry & DTX_LID_SOLO_MASK) - DTX_LID_RESERVED, epoch, &dae);
D_ASSERTF(found, "Could not find active DTX record for lid=%x, epoch=" DF_U64 "\n",
entry, epoch);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question here, was there any other consideration on tolerating "!found" before? I'm worried that if deregister or check availability could be called before the in-memory tree is fully setup? (though that should not be tolerated as well).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we do not allow to access the DTX LRU array before its fully setup. So it is safe from this perspective.

@Nasf-Fan Nasf-Fan marked this pull request as ready for review February 21, 2025 03:40
@Nasf-Fan Nasf-Fan requested review from a team as code owners February 21, 2025 03:40
@gnailzenh gnailzenh merged commit 238cbb5 into master Feb 21, 2025
55 of 57 checks passed
@gnailzenh gnailzenh deleted the Nasf-Fan/DAOS-16876_5 branch February 21, 2025 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NiuYawei NiuYawei NiuYawei approved these changes

@gnailzenh gnailzenh gnailzenh approved these changes

@janekmi janekmi Awaiting requested review from janekmi

@jolivier23 jolivier23 Awaiting requested review from jolivier23

Assignees
No one assigned
Labels
priority Ticket has high priority (automatically managed)
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Nasf-Fan @daosbuild1 @NiuYawei @gnailzenh