Rework Kafka health check

[nasark]

• The push/pop approach for Kafka health checks is not ideal because this can lead to timing issues when dealing with multiple consumers (multiple appliances). Therefore using a simple call to retrieve metadata from the Kafka server, this will fail if the server is down which is ideal for a health check
• Kafka client connections should be closed to prevent issues with multiple consumers on the same topic

Depends on:

• Fix Kafka Client List Topics Method manageiq-messaging#88
• Rework Kafka health check in manageiq-messaging-ready manageiq-appliance#384
• Close Kafka admin client connection manageiq-messaging#89

@miq-bot assign @agrare
@miq-bot add_reviewer @agrare
@miq-bot add_labels enhancement, bug

[nasark]

We will need to bump manageiq-messaging once ManageIQ/manageiq-messaging#88 is merged and a new version is released

[nasark]

@agrare Should we log on success here? The push/pop approach had logging baked in by the messaging client

[agrare]

I think we're okay to just log on failure

[agrare]

You need to handle the case where broker is nil here because ensure will still trigger on the return up here https://github.com/ManageIQ/manageiq/pull/23020/files#diff-0c71d22135646ed908fa6a66bec47aa6024935e7d9ed66e4a99c64f9355b3307R628

The other option is to have an explicit begin/rescue/ensure like

def messaging_health_check
  broker =
  return if broker.nil?

  begin
    ...
  rescue => err
    ...
  ensure
    broker.close
  end
end

[agrare]

@nasark manageiq-messaging v1.4.2 released with ManageIQ/manageiq-messaging#88

[miq-bot]

Checked commits nasark/manageiq@38debb3~...8229620 with ruby 2.7.8, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
2 files checked, 0 offenses detected
Everything looks fine. 🍰

[Fryguy]

Backported to radjabov in commit 29a2b3c.

commit 29a2b3cc9da76d0f21cf2f597dd4dfc14e8e613b
Author: Adam Grare <[email protected]>
Date:   Fri May 3 16:30:12 2024 -0400

    Merge pull request #23020 from nasark/get_metadata_for_kafka_health_check
    
    Rework Kafka health check
    
    (cherry picked from commit 6351fd6e13bd3b36f64088e7da6571c4f2e624b3)