set expiration date of individual items

[sqlalchemy-bot]

Migrated issue, originally created by Dave Reeves ()

I'm coming from redis-py where set("author","Mike",ex=60) would set the ttl to 60 seconds. How do I achieve this behaviour with dogpile?

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

The idea is that if a certain class of items wants a different expiration time you make a separate region for that. The dogpile.cache model does not store expiration times with the cached value, only the time that it was cached. Expiration times are OTOH local to the in-memory region so the architecture does not support per-item expiration times, only per-region. The redis backend itself has a fixed "redis_expiration_time" that is sent along with each value.

Short answer if you are using redis, and you don't want to change how you are doing things, you can get at the backend directly, region.backend.client.setex(key, expiration_time, pickle.dumps(value)).

[sqlalchemy-bot]

Changes by Michael Bayer (zzzeek):

• removed labels: low priority

[sqlalchemy-bot]

Changes by Michael Bayer (zzzeek):

• removed labels: task
• added labels: feature

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

similarly to #37, #62, this request bundles in with the idea that there could be more exposure of how the CacheValue object is created and accessed, such that a custom version of it that stores an expiration time could be produced that would also be able to interact with the expiration mechanics.

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

note also that region.get() does accept an expiration_time that overrides the default: http://dogpilecache.readthedocs.io/en/latest/api.html#dogpile.cache.region.CacheRegion.get

[sqlalchemy-bot]

Dave Reeves () wrote:

Is "redis_expiration_time" set globally for the entire region or would it set it for each item? If I had: redis_expiration_time = seconds_left_in_month() would it call the function every time I put an item in the cache or would it just be called once?

What I'm trying to achieve is to make all items expire at the start of a new month.

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

I think a pretty easy way to get that is to use a custom invalidation strategy, should look like this:

#!python

from dogpile.cache.region import DefaultInvalidationStrategy
import datetime

region = CacheRegion()

class EOMInvalidation(DefaultInvalidationStrategy):
    def is_invalidated(self, timestamp):
        previous_eom = <date logic to get the end of month timestamp>
        if timestamp < previous_eom:
            return True
        else:
            return super(EOMInvalidation, self).is_invalidated(timestamp)

region = region.configure(region_invalidator=EOMInvalidation())

[sqlalchemy-bot]

jvanasco (jvanasco) wrote:

i never noticed this before, but it looks like there's a deficiency in dogpile's docs or features...

region.set and region.set_multi don't accept a kwarg for expiration_time, however these do:

• region.cache_on_arguments
• region.cache_multi_on_arguments
• region.get
• region.get_multi
• region.get_or_create

I'm not sure if the expiration_time above applies to reading (expiring a cache hit) or creation (setting a new value on cache miss or hit+expiry).

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

so....that's the issue here. there's no server side expiry. so passing it to set() would have no effect.

[sqlalchemy-bot]

jvanasco (jvanasco) wrote:

Ah, I missed this line and totally forgot about this behavior. Sorry for the noise!

'''
The dogpile.cache model does not store expiration times with the cached value, only the time that it was cached.
'''

As a frequent abuser of this library, I have a few suggestions:

1. A slightly modified version of Michael's serverside expiry above:

   region.set(key, value)
   region.actual_backend.client.expire(expiration_time) # new api to get around proxies, set expiry in redis
   region.get(key, value, -1) # get key with no expiration time

2. You can create a custom backend that inherits from your current backend, and override the serialization / deserialization functions inset/get and set_multi/get_multi. This allows a developer to overwrite the internal invalidation timestamp during the serialization process, and substitute it with a value that always passes, fails or is a cache miss.

I actually do this often. If your backend is redis, I released a (fully tested, in production) backend that accepts custom dumps/loads functions: https://github.com/jvanasco/dogpile_backend_redis_advanced

3. ProxyBackends

Another method is the ProxyBackend. The values in get/set/multi are the CachedValue tuple with the expected value in the .payload attribute and a .metadata attribute with some specific information in it. I previously used these to alter the metadata expiration timestamp (as in #2), but switched to serialization function as I already moved to that concept.

I recommend against dealing with the raw Redis connection, as it can be a bit weird to properly correlate the values, serializers, etc. This gist I wrote the other day gives an overview of ways you can leverage the dogpile library into raw redis sets:

https://gist.github.com/jvanasco/7ee03c118885bd06e25b651e3139f606