Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add estimate of remaining time for indexer #99

Open
nrejac opened this issue Jul 28, 2020 · 2 comments
Open

Add estimate of remaining time for indexer #99

nrejac opened this issue Jul 28, 2020 · 2 comments

Comments

@nrejac
Copy link
Contributor

nrejac commented Jul 28, 2020

It would be useful to have an estimate of time remaining when running the indexer. Currently only the rate of indexing (records per second) is reported, but if possible, the estimated remaining time based on rate * total number of records to index would be helpful.

@danstoner
Copy link
Contributor

We don't have a "count" anywhere.

For performance we are fetching from cursors and iterators which don't know the size until they get to the end (I think).

We could probably add an SQL count before we begin work but in postgres count is notoriously slow.

@danstoner
Copy link
Contributor

postgres keeps table statistics (for query planner) that have numbers that are generally going to be on the same order of magnitude as the table count.... so that might work for a time estimate.

Actually, with newer postgres features such as Parallel Seq Scan, the time to compute count(*) is much faster than before, definitely fast enough to work for this use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants