Refactor Redpanda Migrator components

[mihaitodor]

I hijacked this PR to address several issues:

Added

• New redpanda_migrator_offsets input.
• Fields offset_topic, offset_group, offset_partition, offset_commit_timestamp and offset_metadata added to the redpanda_migrator_offsets output.
• Fields kafka_key and max_in_flight for the redpanda_migrator_offsets output are now deprecated.
• Fields batching for the redpanda_migrator output is now deprecated.
• Field topic_lag_refresh_period added to the redpanda and redpanda_common inputs.
• Metric redpanda_lag now emitted by the redpanda and redpanda_common inputs.
• Metadata kafka_lag now emitted by the redpanda and redpanda_common inputs.

Fixed

• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured.
• The redpanda_migrator output no longer rejects messages if it can't perform schema ID translation.
• The redpanda_migrator input no longer converts the kafka key to string.

Changed

• The kafka_key and max_in_flight fields of the redpanda_migrator_offsets output are now deprecated.
• Fields batch_size and multi_header for the redpanda_migrator input are now deprecated.
• • The redpanda_migrator_bundle input and output now set labels for their subcomponents.
• The redpanda_migrator input no longer emits tombstone messages.

Redpanda Migrator offset metadata

One quick way to test this is via the following config. Note how I overwrite kafka_offset_metadata to foobar in a mapping processor.

input:
  redpanda_migrator_bundle:
    redpanda_migrator:
      seed_brokers: [ "localhost:9092" ]
      topics:
        - '^[^_]' # Skip internal topics which start with `_`
      regexp_topics: true
      consumer_group: migrator_bundle
      start_from_oldest: true
      replication_factor_override: true
      replication_factor: -1

    schema_registry:
      url: http://localhost:8081
      include_deleted: true
      subject_filter: ""

output:
  processors:
    - switch:
        - check: metadata("input_label") == "redpanda_migrator_offsets_input"
          processors:
            - mapping: |
                meta kafka_offset_metadata = "foobar"
  redpanda_migrator_bundle:
    redpanda_migrator:
      seed_brokers: [ "localhost:9093" ]
      max_in_flight: 1
      replication_factor_override: true
      replication_factor: -1

    schema_registry:
      url: http://localhost:8082

[mihaitodor]

When I added the redpanda_migrator input, I had both this gauge and the kafka_lag metadata field. I don't know if we want any of these available by default. Also, should this gauge name be somehow derived from the actual input type (redpanda, redpanda_common, redpanda_migrator, redpanda_migrator_offsets)? It does get the label of the input if set, so maybe that's sufficient.

[mihaitodor]

This constructor got a bit messy to use... It can be hard to tell which of these funcs is set to nil at the call site and one can easily mix them up. I'm thinking to maybe introduce functional options for it or maybe a struct which contain all the parameters. WDYT?