option to drop module/function from key_generator functions in util.py

[sqlalchemy-bot]

Migrated issue, originally created by jvanasco (jvanasco)

I tracked down a performance issue to usage of dogpile.cache.region:Region.cache_on_arguments in a few places.

The default key generation is provided key generator functions in dogpile.cache.util, with this relevant code shared by all:

    if namespace is None:
        namespace = '%s:%s' % (fn.__module__, fn.__name__)
    else:
        namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)

My issue was the "base" of the key('%s:%s' % (fn.__module__, fn.__name__)) was incredibly long in a handful of the most-used places, and the keys themselves were taking up a significant chunk of the memory allocated to Redis. Dropping the key length reclaimed a lot of space.

While a key_mangler would solve this, I needed to keep the keys unmangled. The easiest solution was to just reimplement the stock key generator to just use the namespace argument as the key.

This seems like a relatively useful approach for many people, so I wanted to suggest porting it upstream:

I think the relevant changes would be:

1. The key_generators have a new kwarg to omit the module+name prefix (like to_str)
2. cache_on_arguments(_multi) have a new kwarg for the same, which is just passed to the key_generator (again, like to_str)

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

this is...for convenience vs. making your own function_key_generator? the issue of "long keys" is indented to be solved by key_mangler. You can make a key_mangler that just truncates the module/name part, if you dont want to go with making a GUID (I assume that's where your concern wtih mangling is)..

that is, you can get what you want with your own fn_key_geneartor, you can get what you want with a key mangler . where is this lacking?

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

how about this:

def function_key_generator(namespace, fn, to_str=compat.string_type, namespace_key_template="%(module)s:%(fn_name)s|%(namespace)s", key_template="%(module)s:%(fn_name)s"):

?

[sqlalchemy-bot]

jvanasco (jvanasco) wrote:

re: custom key_generator.
This is for convenience. I fixed my deployment in 5 minutes with a custom key_generator, but this seems like it's something other people would want/encounter .

re: mangled guid
Yes, having a GUID/hash isn't an option. There are a bunch of tools that dump keys and pull metrics off them.

re: mangled 'other'
this particular deployment (web app) already has a key
mangler on all regions to handle 'invalid' values. adding splits/regex to the existing region would not work - dogpile is heavily leveraged on some pages (100-1000 hits). splitting it into another "region" with the same settings but a different key_mangler is an option, but would require a lot of work due to how regions are setup, managed, and tested in this app.

the idea for a key_template is much better.

[sqlalchemy-bot]

Michael Bayer (zzzeek) wrote:

makes sense but the whole function should be templated then