Skip to content

Is there a way to keep track of cache hits vs actual bytes requested? #1527

Answered by betolink
betolink asked this question in Q&A
Discussion options

You must be logged in to vote

Hi @martindurant, I did a first pass and seems to be working as expected, by adding counters in the base class and updating the debug message accordingly I'm now getting these logs:

<File-like object S3FileSystem, file.h5> read: 432000000 - 440000000, total: 276824064, cache hits: 34, cache misses: 33

total is the total requested bytes, and cache hits vs misses in blockcache is a bit tricky because unless we keep reading from the same block over and over we'll have 1 miss + 1 hits on each pass. I added some unit tests for these numbers.

https://github.com/betolink/filesystem_spec/blob/master/fsspec/tests/test_caches.py

I also consolidated the __repr__ and I feel like there is room for so…

Replies: 1 comment 4 replies

Comment options

You must be logged in to vote
4 replies
@martindurant
Comment options

@betolink
Comment options

@martindurant
Comment options

@betolink
Comment options

Answer selected by betolink
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants