Reuse KV cache of prefixes

[tohtana]

This PR implements reusing of KV cache across multiple requests. You can set enable_prefix_cache to True in RaggedInferenceEngineConfig to enable this feature.

config = RaggedInferenceEngineConfig(enable_prefix_cache=True)

This feature keeps KV cache blocks as long as we have free space. When a new request has a prefix that matches the KV cache blocks, FastGen reuses them. The blocks can also be reused by multiple requests. This drastically reuses the computation for prompt and memory usages for KV cache when many requests have common prefixes.
Note that looking up the cache has some overhead. You can disable this feature when prompts don't have much overlap.

Here is a benchmark result using this feature. We used prompts that have the same prefix when using this feature.

When the prompts are short and generation are long, the benefit will be smaller.