By default cloud_io::remote::list_objects is greedy. In this case, we want to
control paging in the garbage collector.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

The control loop doesn't need to know about the rest of the list_objects
response on the happy path, except if an error occurred.

This will make it easier to change the source of those bucket items when/if
we add some caching.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Includes plumbing self node ID & members_table down into level_zero_gc

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Gives the range of random prefixes assigned to this node by dividing prefix
space up according to the total number of shards in the cluster and assigning
a range to each shard based on index from lowest node ID value.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Probably could be more efficient, factored slightly differently.

e.g. instead of building a vector of prefix strings, could just modify the
trie structure inline so that it represents the minimal set of prefixes, then
some mechanism to iterate over stored prefixes

Return a fully qualified object key fragment.

Provide a stream of object_key prefixes to cover some prefix_range.

- list_objects
- filter on assigned prefix range
- report back to caller whether we've gone past our assigned max prefix

The general idea here is to hide the details of prefix filtering from the
control loop. This is where, ultimately, we'll want to wire in some kind
of local cache for known outstanding garbage.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

In a loop:
  - try to get the next page
  - if an error occurs, bail
  - if the result is non-empty, return the objects
  - otherwise do_next_page reports whether or not we have passed the
    maximum assigned prefix for this shard
    - if so, stop iterating, there are definitely no more objects we care about
      in the bucket
    - if not, we have _not yet_ reached the minimum assigned prefix, so we
      should keep paging through the bucket

This is not very efficient if the number of objects in the bucket grows very
large. Alternatives include:

- caching bucket contents
- listing on assigned prefixes only

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Needs cleanup, but basically simulates multi-shard prefix space partitioning.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>