Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix use-after-free of zfsvfs after unmount #288

Conversation

EchterAgo
Copy link

@EchterAgo EchterAgo commented Oct 13, 2023

During unmount zfsvfs is destroyed and the pointer is zeroed in the VCB. There is however still a copy of the pointer in the DCB.

Windows can still call into zfs_AcquireForLazyWrite, through CcMgr after unmount and this would use the already freed zfsvfs pointer.

To fix this we set the pointer to zero in the DCB and add a zero check in zfs_AcquireForLazyWrite, zfs_AcquireForReadAhead and fastio_acquire_for_mod_write.

fixes #282

@EchterAgo EchterAgo marked this pull request as ready for review October 13, 2023 11:51
@EchterAgo
Copy link
Author

With this the tests.py with no delay finally passes consistently.

@EchterAgo
Copy link
Author

Should we include removing the delays from the tests in this PR or in #286?

@EchterAgo
Copy link
Author

EchterAgo commented Oct 13, 2023

This test can be used to easily reproduce the error:

fsutil file createnew C:\testfolder\poolfile.bin 200000000

:hans
zpool create tank \\?\C:\testfolder\poolfile.bin
zpool destroy tank
goto hans

del C:\testfolder\poolfile.bin

@EchterAgo EchterAgo changed the title fix use-after-free after unmount Fix use-after-free after unmount Oct 13, 2023
@andrewc12
Copy link

I don't think it matters as long as they get merged with or after this one gets merged

@EchterAgo
Copy link
Author

Yay, test5_py passed!

@EchterAgo
Copy link
Author

EchterAgo commented Oct 13, 2023

Hmmm, while unlikely to occur, I think there is an issue with this, zfs_AcquireForLazyWrite could still be called between freeing the VCB zfsvfs and setting it to zero in DCB. The mutex does not guard against this because it is in zfsvfs. Maybe we should set a flag in mount?

@EchterAgo
Copy link
Author

We also can't just vfs_setfsprivate(zmo_dcb, NULL); before zfs_vfs_unmount because it could fail. It also looks like if zfs_vfs_unmount fails things are already in a bad state. We could check zmo_dcb->vpb->Flags & VPB_MOUNTED in zfs_AcquireForLazyWrite

@EchterAgo
Copy link
Author

Updated to add VPB_MOUNTED check, I ran tests.py in a loop for a while and didn't see any problems.

@EchterAgo EchterAgo force-pushed the fix_unmount_use_after_free branch 4 times, most recently from cd5f081 to 73b95fb Compare October 13, 2023 15:52
@EchterAgo EchterAgo marked this pull request as draft October 13, 2023 18:32
@EchterAgo
Copy link
Author

EchterAgo commented Oct 13, 2023

Sorry, converting to draft because I think my last change is wrong, checking the VPB_MOUNTED flag will likely prevent data from flushing in zfs_vfs_unmount.

@EchterAgo
Copy link
Author

We should also have a test for flushing on unmount, I'll try to add one.

@EchterAgo
Copy link
Author

Ideally we'd set both zmo and zmo_dcbs fsprivate to zero before freeing it, but at the same location, at the end of zfs_vfs_unmount. I'll push such a change soon.

@EchterAgo
Copy link
Author

Updated, I think that is the best I can come up with :\

@EchterAgo EchterAgo marked this pull request as ready for review October 13, 2023 19:45
@EchterAgo

This comment was marked as off-topic.

@lundman
Copy link

lundman commented Oct 13, 2023

04 ffffd086dde6e8e0 fffff80021f1e6e8 nt!ObReferenceObjectByPointer+0x21bec3
05 ffffd086dde6e920 fffff80022220a3e OpenZFS!vflush+0x168 [H:\dev\openzfs\module\os\windows\spl\spl-vnode.c @ 1604]

This is unrelated, and has been a thorn for quite some time.

@EchterAgo EchterAgo changed the title Fix use-after-free after unmount Fix use-after-free of zfsvfs after unmount Oct 14, 2023
@EchterAgo
Copy link
Author

This is unrelated, and has been a thorn for quite some time.

Yea, I found #95 which seems to be the same. I added my info there.

@EchterAgo
Copy link
Author

I think there are a bunch more locations that might need a NULL check, I have a change for that too, but haven't tested it enough yet.

Better to have a NULL dereference than having to chase a use-after-free though.

@lundman
Copy link

lundman commented Oct 14, 2023

Generally, all requests to ZFS come in via dispatcher(). There are 3 exceptions, guess which ones they are :)

@lundman
Copy link

lundman commented Oct 14, 2023

./module/os/windows/zfs/zfs_vnops_windows.c: 7965: continuation should be indented 4 spaces

During unmount `zfsvfs` is destroyed and the pointer is zeroed in
the VCB. There is however still a copy of the pointer in the DCB.

Windows can still call into `zfs_AcquireForLazyWrite`, through `CcMgr`
after unmount and this would use the already freed `zfsvfs` pointer.

To fix this we set the pointer to zero in the DCB and add a zero check
in `zfs_AcquireForLazyWrite`, `zfs_AcquireForReadAhead` and
`fastio_acquire_for_mod_write`.

Signed-off-by: Axel Gembe <[email protected]>
@EchterAgo
Copy link
Author

Fix for checkstyle

@lundman lundman merged commit 51b8af1 into openzfsonwindows:zfs-Windows-2.2.0-release Oct 14, 2023
26 of 29 checks passed
@EchterAgo EchterAgo deleted the fix_unmount_use_after_free branch October 14, 2023 08:25
lundman pushed a commit that referenced this pull request Oct 18, 2023
During unmount `zfsvfs` is destroyed and the pointer is zeroed in
the VCB. There is however still a copy of the pointer in the DCB.

Windows can still call into `zfs_AcquireForLazyWrite`, through `CcMgr`
after unmount and this would use the already freed `zfsvfs` pointer.

To fix this we set the pointer to zero in the DCB and add a zero check
in `zfs_AcquireForLazyWrite`, `zfs_AcquireForReadAhead` and
`fastio_acquire_for_mod_write`.

Signed-off-by: Axel Gembe <[email protected]>
lundman pushed a commit that referenced this pull request Dec 11, 2023
During unmount `zfsvfs` is destroyed and the pointer is zeroed in
the VCB. There is however still a copy of the pointer in the DCB.

Windows can still call into `zfs_AcquireForLazyWrite`, through `CcMgr`
after unmount and this would use the already freed `zfsvfs` pointer.

To fix this we set the pointer to zero in the DCB and add a zero check
in `zfs_AcquireForLazyWrite`, `zfs_AcquireForReadAhead` and
`fastio_acquire_for_mod_write`.

Signed-off-by: Axel Gembe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants