Low frequency samples ingestion

[bodulos]

Hi,

I'm migrating my current prometheus long-term storage setup (promscale+timescaleDB) to Mimir and have some challenges with ingesting low frequency samples. Because of the nature of the datasources we need to collect some offline generated timeseries data (jobs, timeseries file dumps etc.) with sample frequency of 1 hour, 1 day etc. Those samples have their embedded timestamps which we need to honor.
This is not an issue for current setup, promscale accepts older timestamps in samples and handles older chunk decompression before ingestion.

Because of the default restriction in Mimir samples older than 1h cannot be ingested. I am aware that this could be tuned but not sure this is the right approach.

From what I have read, ingesting samples more than 24 hours old samples falls in domain of data backfill. Idea is to use the backfill method to ingest older samples - create prometheus format chunks from timeseries data and upload them with mimirtool.
Hourly samples are border case. Changing the ingestion window from 1h to 2h in would solve problem for them. But could be handled in the same way as daily samples, with backfill.

Before I proceed in this direction I have few questions to confirm it:

1. Do you see a better solution than the backfill solution (I haven't found any after extensive search)
2. Is backfill with mimir tool considered stable enough - the plan is to use it as ingestion option which is not it's primary purpose
3. Should some special considerations be taken into account? E.g. for backfill chunk upload, can we do concurrent chunk uploads - multiple interdependent processes upload the chunks. What if chunks that were uploaded first are being processed by the compactor and new ones start arriving?
4. What would be the preferred way for making uploaded and processed chunks visible ASAP? Or we always need to wait for "querier.query-store-after" to get the uploaded blocks visible? This is not an issue for daily samples, but is for hourly ones, as they also hit the "less than 1h old timestamp" limit.

Any thoughts on the subject would be highly appreciated.