Embedded Kafka does not work with Spark 3.5

[adeshrd]

I have been using Embedded Kafka in my batch job to test if my output is correct.
It works correctly upto Spark 3.4.3 but fails at Spark 3.5.0.
I have tried different versions of Embedded Kafka as well upto the latest version.

This is the exception I get:

io.github.embeddedkafka.KafkaUnavailableException was thrown.
io.github.embeddedkafka.KafkaUnavailableException
at io.github.embeddedkafka.ops.ProducerOps.publishToKafka(ProducerOps.scala:219)
at io.github.embeddedkafka.ops.ProducerOps.publishToKafka(ProducerOps.scala:82)
at io.github.embeddedkafka.ops.ProducerOps.publishToKafka$(ProducerOps.scala:72)
at MySpec.publishToKafka(MySpec.scala:5)

This is my test code:

import io.github.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig}
import org.scalatest.matchers.should.Matchers // Corrected import
import org.scalatest.wordspec.AnyWordSpecLike

class MySpec extends AnyWordSpecLike with Matchers with EmbeddedKafka {

  "runs with embedded kafka on arbitrary available ports" should {

    "work" in {
      val userDefinedConfig = EmbeddedKafkaConfig(kafkaPort = 0, zooKeeperPort = 0)

      withRunningKafkaOnFoundPort(userDefinedConfig) { implicit actualConfig =>
        // A Kafka broker is listening on actualConfig.kafkaPort
        publishStringMessageToKafka("topic", "message")
        consumeFirstStringMessageFrom("topic") shouldBe "message"
      }
    }
  }
}

These are the relevant libraries:

<spark.version>3.5.0</spark.version>
<scala.version>2.12.15</scala.version>
<embedded.kafka.version>3.8.0</embedded.kafka.version>

I have reproduced this by just including Spark Core and Embedded Kafka as dependencies.

Please help. Thank you