estimate hot/warm/cold data access ratios within archive or even per dandiset

[yarikoptic]

cold - over a week I guess

[waxlamp]

Could you say more about this, @yarikoptic? Perhaps give some examples of what you expect to see?

[yarikoptic]

I believe this request was in relationship to questionnairs from protocollabs while investigating data access schemes if data gets deposited to IPFS. It relates to typical scenarios on how people access data -- are the same files get re-accessed immediately, or typically just once and then not again for a while. I do not truly remember hot/warm separation idea here, but we can make it following that "Cold" - prior to first access in current period (day? week?) it was not accessed for a week. "Warm" - was accessed within last day, "Hot" - was accessed within last hour.

So e.g. for overall transfer of 1TB of data, we could have Cold: 900G, Warm: 90G, Hot: 10G thus showing that primarily data is accessed once and only small portion (1%) could benefit from some kind of faster caching layer to speed up access. But we could also see Cold: 10G, Warm: 90G, Hot: 900G suggesting that having some cache over overall slow storage backend could be of great benefit since most of the time data is immediately reaccessed.

Makes sense?