nlog #18
Draft
nlog #18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implemented the 32-bit next-gen pager, ensuring all fast tests are now passing. - Migrated Voron internals to utilize the next-gen pager for improved performance and maintainability. - Updated the server to align with the new pager infrastructure - Big hammer approach to getting the tests to compile - Removed unused code to streamline the codebase
- Corrected the usage of EnsureMapped to ensure proper memory mapping. - Updated the crypto pager to explicitly call raw pointers for better control. - Ensured that the global state is not modified unexpectedly when using the crypto pager.
…Gen Pagers - Move temporary buffers (compression, decompression, recovery) to new gen pagers - Moved memory locking to new gen pagers - Refactoring how we deal with scratch files pager to next gen - Removing the need to have BreakLargeAllocationToSeparatePages
- Moving transaction state to the current record, removing dead code - Fixing transaction id on new database - Simplify how we register scratch pager states in the transaction. - Refactoring naming. - Removed remnants of lazy transaction. - Fix race between flushing & committing a transaction updating the current state record. - Store a pointer to the pager directly in the PageFromScratchBuffer - Avoid nullable for PageFromScratchBuffer, to reduce padding requirements.
…tables from Journals to the transaction. - DirectWrite now works in 32 bits mode
All the page translation table is handled at the level of the EnvironmentStateRecord now.
…know when to stop using them
…ions - Moved all scratch state to the PageFromScratchBuffer to simplify state management. - PageFromScratchBuffer is now a truly immutable record for improved reliability. - Flush Handling: Added handling for flushing when there are no active transactions. - Optimized the process of getting pages in write transactions. - Modified the behavior to publish the modified scratch table immediately on flush, without waiting for the next write transaction. - Ensured that a freed scratch page is held until all transactions are completed, eliminating the need for additional filtering. - Properly reset the state of a transaction on rollback to maintain consistency. - Reduced the number of allocations by not tracking freed pages that were not being used. - Ensured that empty pages are not written to disk. - Added checks to ensure that pages from scratch are not freed if they were already freed in a previously flushed transaction. - Do not attempt to validate page checksum from the scratches.
- Modified the flush process to iterate over all complete transactions and free their scratch buffers, not just the latest one. - Removed the TransactionId argument from the pager as it was deemed unnecessary. - Eliminated the concept of journal snapshots to streamline the code and reduce complexity.
…andling - Removed StorageEnvironmentState and migrated its logic to EnvironmentStateRecord for streamlined state management. - Adjusted the logic for removing journals to retain them for a bit longer, resulting in much simpler and more maintainable code. - Updated tests to reflect the changes in journal removal timing and ensure proper functionality.
- Fixed the method for identifying the longest tests to improve test coverage and reliability. - Transitioned to using an immutable dictionary along with a builder pattern for the writer to enhance performance and maintainability. (200x gain).
…, fixed conditional
… Updating Dependencies - Introduced the notion of client state in the EnvironmentStateRecord - an immutable record attached to a particular transaction that gets published atomically with the transaction commit - Updated DocmentStorage to use the client state - Updated indexing to use the client state
- Introduced IndexStateRecord and moved all the cache / records of the index to the client state of the transaction RavenDB-22457 - Fixing name of method to convey the real intent
- Migrate from AfterTransactionCommit in Rachis - Changed the way we manage state for Rachis, this is now married to the transactional state and published atomically with the transaction commit. - Fixing up references to the Rachis Engine state to use the transactional version instead - Make sure that the journal application happens before the transaction commit is published - Removed JournalSnapshot entirely, no longer needed - A journal is now considered unused only when the currently flushing transaction was flushed to a *later* journal - Added debug code to detect a potential race condition
- Simplifying Sync Behavior - Remove sync behavior per physical drive in favor of global sync behavior, multiple physical drives are rare enough that it doesn't matter in production deployments. Removed physical drive throttling for sync, specific to Windows and now largely irrelevant. - Started to implement next gen pagers: - Introduced basic data pager functions, including initialization, closure, and expansion, transitioning to native functions for improved performance. - Memory Management: Transferred memory locking and 32-bit memory mapping logic to native C code. - Read-Only Memory Mapping: Completed read-only memory-mapped files for data pagers, with scratch files continuing to use read/write pagers. - Journal Application: Shifted journal application to direct file I/O for better efficiency and performance. - Code Refinements: Implemented various minor changed to ensure proper pager operations, including avoiding AsSpan() on pages to handle overflow pages correctly. - Synchronous Flushing: Enabled synchronous file flushing to ensure data integrity. - Ensure that we can allocate pages in crypto mode using raw reads - Ensure that an async commit will first get the immutable view of the scratch pages *before* we move to the next transaction, to avoid leaking pages between transaction using the pooled scratch table builder pooled instance. - Implemented measures to detect leaking pages by ensuring transactional invariants are valid. - Adding debug checks to verify that we are doing the proper thing on allocation of files - Refactoring ScratchBufferPool.Free to split the freeing of a page and the recycling of a file
- Avoid AfterCommitWhenNewTransactionsPrevented in tests - Remove AfterCommitWhenNewTransactionsPrevented from ClusterStateMachine - Ensuring that we'll keep the reference for the old state of the compression buffer for the duration of PrepareToWriteToJournal. This is required because are making *two* calls to EnsureContinious, and may create a new mapping because of that, and the old state (and mapping) may be cleared away by the finalizer *while the method is running.
- Moving to a zig based cross build system - Removing win7 support - Adding rvn_map_memory for linux, removing dead code - Temporary file handling (delete on last close) on Linux - Fixing Unicode filenames on Windows
- Removing NativeMemory.CurrentThreadStats from NewLowLevelTransaction, that alone took 20% of the transaction initialization - Moved that to just hold the managed thread id and do a lookup as needed. - May cause us to miss thread names, but they are mostly debug information - Removed StackTrace capture code that was used only by tests
- Moving Lucene state to the transaction ClientState directly - Moving suggestions state to the transaction as well - Removing LuceneIndexSearcherHolder - Removing usage of AfterCommitWhenNewTransactionsPrevented - Removed another lock from creating the transaction! - Removing the PreventNewTransaction support, AfterCommitWhenNewTransactionsPrevented event
- Fixing incremental backups support to next gen pager - Do not try to increase the file size if we opened it in readonly mode - Zeroing a buffer requires using the raw memory - Wiring up encryption events properly again - Crypto pager now expects to get the data directly from the scratch files. - PagerTransactionState events are raised once per transaction - _freePagesBySizeAvailableImmediately was removed from ScratchBufferFile - only useful for rollback, and they are rare enough that we shouldn't worry about them - Ensure that code that modify the memory map directly has access to do so (test only)
- Fixing how we use cached NumberOfConflicts - Fixing wiring of suggestions index searcher using new IndexStateRecord - Making sure we set LastFlushState *after* we call AddJournalToDelete - Make sure that only a transaction commit will update the record state, so there is no contention there at all
- Moved the flushing logic that runs under the transaction lock to commit stage 1, because it *must* run before creating the next async transaction so it will keep consistent state - Fixing allocation of a new page using encrypted storage - Fixing name of CurrentStateRecord - When flushing, we need to flush the *raw* page data (encrypted if encrypted) - We only register *once* for transaction commit or dispose - Ensure that when we rollback an async commit, it's base state is the previously committed transaction - Removing all temp Not Implemented markers - Fixing double dispose in 32 bits mode - Fixing not respecting the number of pages for new pages in 32 bits mode - Removing DecompressionBuffersPool which used a global pager in favor of allocating memory directly from the current transaction allocator - Removing tests that checked DecompressionBuffersPool state - Ensuring we are copying the _raw_ data from the data pager - Removing test for removed functionality - Move crypto memory usage computation to the PagerTransactionState - Dispose the data pager's state on StorageEnvironment failure - Fixing failing tests on database recovery
- Ensure that we can only discard memory that was allocated in this transaction - Fixing crypto pager to write the encrypted data (instead of all zeroes) to the data file upon database recovery - We need to ensure that we increase the file size on database recovery - Fixing missing rnv_munmap_memory() in Posix - Add notice on how to install Zig if missing
- If an async commit tx didn't change anything, don't update the transaction id for the _next_ Transaction - Properly propogate the prefixes states for output collections - Reading from the right index to count the number of unique document conflicts - Fixing a single shared instance used for multiple purposes - Moving LIBRVNPAL and LIBZSTD to use an internal nuget package
- Small optimization - skip doing hash table lookup if the table is known to be empty - PAL build will now generate (and register) a new nuget package automatically - Ensure that getting disk space is working on Windows (use the Unicode variant) - Removing test that no longer make sense, we don't keep track of page state in this manner any longer - Fixing 16105 test because now we don't have break apart large allocations - Properly handle mapping 0 length read-only files by now actually doing the mapping - PAL nuget package is set on the Sparrow.Server project now - We need to dispose the old data pager even if we don't dispose the options, since we need to discard any potentially old states. - Removed outdated comment - We need to keep a file handle on PureMemoryStorageEnvironmentOptions so DeleteOnClose will not be triggered early - Update the conflict count live during the transaction (as well as on commit) so we can detect new conflicts as part of the current transaction and not just after commit
- Moving the current position in the journal to the Environment's state, so it will be pushed only on commit - Adding RavenFact to new tests & Better indication on what test exactly is missing RavenFact - Full journals should also be removed - Fixing test so it will verify that rollbacked pages are reused after we commit the _next_ transaction - PAL should have pdbs - Mark zero length READ_ONLY files as DO_NOT_MAP so we won't fail on dispose - Fixing PAL build to ensure that we aren't using the old binaries - Fixing test to use predictable output - Ensure that we use the _current_ journal file position, not just the committed one - Fixing last file flush computation
6.1 to 6.2 merge
6.1 to 6.2 merge
Merge v6.1 to v6.2
6.1 to 6.1 merge
6.1 to 6.2 merge
- Restore StreamWithTimeout to previous implementation - Remove usage of StreamWithTimeout from RequestHandler
…reads & writes), now will only do the CancelAfter when being actively to.
…ve the oldest active transaction
… write transaction
6.1 to 6.2 merge
6.1 to 6.2 merge
ppekrol
force-pushed
the
v6.2
branch
2 times, most recently
from
3acdf0c
to
72d9f13
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.