-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] fix: updates the training sampling strategy to complete the last batch #538
base: main
Are you sure you want to change the base?
Conversation
Changed Files
|
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #538 +/- ##
==========================================
- Coverage 76.34% 75.72% -0.62%
==========================================
Files 47 48 +1
Lines 3483 3522 +39
Branches 479 486 +7
==========================================
+ Hits 2659 2667 +8
- Misses 721 752 +31
Partials 103 103 ☔ View full report in Codecov by Sentry. |
I have a bunch of experiments setup using the latest version of EV and this PR. I am unable to get this PR to start training. I originaly setup an experiment using the "z" character / removed from training / added and I was having a hard to even confirming that this bug exist since / was able to synth the z sound... I was in the process of setting up an other experiment using the "J" character till I was noticing that none of the training using this PR was able to successfully train . I tried using LJ and other datasets with no modifications . I always get the error below when I use this PR to train. I am not convinced this PR is ready to be release. See attached logs :
|
ad8cd7f
to
c3b6c71
Compare
Fixes #438
PR Goal?
Updates sampling strategy in training and complete last batches with random samples from other batches instead of dropping last batches.
Fixes?
Fixes #438
Feedback sought?
If a model works with this new sampler. If it produces results that are better or at least not worse after training.
Priority?
Low
Tests added?
No tests added, but it would be good to have some testing of this sampler.
How to test?
Place a breakpoint and inspect the composition of a last batch in an epoch. Check that the number of batches correspond to expectations during training. Train a model in a scenario when difference between dropping and keeping last batches is noticeable (e.g. very small dataset or a dataset where samples in last batch have unique phonemes).
Confidence?
Low. This code wasn't properly tested.
Version change?
No. Can be a part of a larger update.
Related PRs?
No.