Parallelize indexer by uploading the marshal files in a batch and pug…

[segiddins]

…ing fastly as a fan-in step

[codecov]

Codecov Report

Merging #3548 (5d1942e) into master (08bcab5) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3548      +/-   ##
==========================================
- Coverage   97.51%   97.51%   -0.01%     
==========================================
  Files         186      186              
  Lines        4472     4471       -1     
==========================================
- Hits         4361     4360       -1     
  Misses        111      111              

Impacted Files	Coverage Δ
app/jobs/indexer.rb	100.00% <100.00%> (ø)
app/models/deletion.rb	100.00% <100.00%> (ø)
app/models/pusher.rb	94.91% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[indirect]

it does seem like splitting it into three parallel jobs is going to make it faster, let's give it a shot

[simi]

@segiddins is this still worth it?

[segiddins]

yes still worth it