Continous scan #767
Open
Continous scan #767
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Supporting higher parallelism in scans A configuration variable that can disable or enable this feature.
…us scan Supporting higher parallelism in scans The feature is only supported for ReadCommitted and when the configuration supports it. This means that a flag is needed in both SCAN_FRAGREQ and SCAN_TABREQ that specifies whether it is used or not. Adapted also signal printers.
Supporting higher parallelism in scans DBTC needs to maintain four times as many receiver id's as the parallelism. Thus the parallelism is a fourth of the number of receiver objects. We have to track which receiver object to use for the moment and store the receiver objects used by the various fragment scans.
Supporting higher parallelism in scans We support being one scan ahead for continous scans. This means that the NDB API can receive the first batch, the second batch and even a third batch and still immediately respond to DBTC for continued scans. But if three batches have been requested we have to wait until DBLQH signals the first batch (through DBTC) completed until we send the fourth batch. The reason is that DBLQH can work on the fourth batch if three batches have been requested. To support this DBLQH needs access to all four receiver objects, it also need a new state tracking where in the continous scan process it is. DBLQH keeps track of which receiver object to use it, thus synchronisation between NDB API and DBLQH is algorithmic and not using any special communication. The main benefit for DBLQH scans is that when it returns SCAN_FRAGCONF it previously had to wait for SCAN_NEXTREQ until it could continue scanning. Now it can continue scanning the next batch immediately. Thus we only need to stop scanning in DBLQH if the return of SCAN_NEXTREQ happens after completed the next batch. Thus normally we would get continous scanning in DBLQH, thus the name continous scanning.
mronstro
force-pushed
the
continous_scan
branch
from
b02a328 to
24db7a5
Compare
Supporting higher parallelism in scans The most complex part of supporting continous scans is for sorted scans on ordered indexes. This requires a fairly complex state machine tracking the state of all concurrent fragment scans and ensuring that we enter wait states when appropriate. It is important to get this part right since much of the performance improvements comes from avoiding too many wait states and do as much batching as possible without overusing memory. A lot of new debugging is added to support this. The NdbReceiver added a new variable with its index in the array of NdbReceiver to make it easy to correlate to its state. A new state for NdbReceiver objects was introduced with 7 different states. Some minor changes to handling SCAN_TABCONF was required. The handling of nextResult for continous scans with sorting is a completely new method to avoid complicating the existing code.
mronstro
force-pushed
the
continous_scan
branch
from
24db7a5 to
4120233
Compare
Added 3 functions where the major work is done: get_first_sorted_batch get_next_sorted_row get_next_sorted_batch This makes the flow easier to understand hopefully and should make it easier follow the flow.
mronstro
force-pushed
the
continous_scan
branch
from
3f71157 to
ef5f33c
Compare
…om different programs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.