even though the trainer streams samples to the WordCounter; the total training set size is limited by the memory pressure of the WordCounter.
If the WordCounter supported a max_words, with eviction, larger sample sets could be trained on.
There's a churn/generation question to consider. If we simply evict the lowest count new words on overflow, then we're very likely to evict the new new word on the next overflow. If we set max_words to be significantly larger than vocab size, this may not matter; but there are tricks we could pull.
We could say that there's a nursery of new words of some size (K).
When we add a new word, we add it to the nursery.
When the nursery fills up, we fold in the contents.
We then ensure that the counter has at least (K) empty slots.
This would require a fair shuffle of the input samples to use effectively.
even though the trainer streams samples to the WordCounter; the total training set size is limited by the memory pressure of the WordCounter.
If the WordCounter supported a max_words, with eviction, larger sample sets could be trained on.
There's a churn/generation question to consider. If we simply evict the lowest count new words on overflow, then we're very likely to evict the new new word on the next overflow. If we set max_words to be significantly larger than vocab size, this may not matter; but there are tricks we could pull.
We could say that there's a nursery of new words of some size (K).
When we add a new word, we add it to the nursery.
When the nursery fills up, we fold in the contents.
We then ensure that the counter has at least (K) empty slots.
This would require a fair shuffle of the input samples to use effectively.