[Bug] Every retry against Livy spins up a new Spark session

[mdrakiburrahman]

Is this a new bug?

• I believe this is a new bug
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Adapter retries against Livy like this:

Each of these spins up a new Spark Cluster:

The adapter goes off and uses the final one, and the others run idle.

Expected Behavior

Ask for a new Livy session and await for that specific one to turn green.

Steps To Reproduce

Fire the adapter and look at monitoring hub.

Relevant log output

See screenshot above with retries.

Environment

Irrelevant

Additional Context

I'll have a fix in PR

[mdrakiburrahman]

This commit solves the problem:

mdrakiburrahman@1a9afdc

One session requested:

Code awaits:

Session ready:

Code moves on:

[billy-moore-98]

I am currently coming across this issue and it's blocking our dev, so the solution looks promising. When can we expect it to be released?

[mdrakiburrahman]

Hey @billy-moore-98,

I have to get my PRs merged first 🙂 I need to work with the maintainer @prdpsvs to do so.
My branch above has a lot of bugfixes and improvements, so I'll probably need to break it apart before it merges etc.
The Unit Tests in the repo are broken too, so I need to fix those first.

In the meantime, if you take my branch above, you can see I have a little README under contrib on how to build a whl with uv. You can use that in your dev environment (this is how I unblocked myself too by using my private build).

#56

[Jwolinda]

@prdpsvs and @mdrakiburrahman we're running into this issue as well. Anything we can do to get it patched would be greatly appreciated

[mdrakiburrahman]

Thanks @Jwolinda, I'm looking into becoming a maintainer of this repo with @prdpsvs / @pradeepsrikakolapu 's help soon. I'll aim to have a bump of the version in PyPi after that.

In the meantime, please feel free to use the branch above and stash the whl in a blob storage, you can use pip install against the blob to unblock yourself.

[billy-moore-98]

Thanks @mdrakiburrahman, I have cloned your branch and works well locally!

We are working on deploying our dbt projects as durable functions. Due to write permissions we redirect any file output to a ~/tmp/ dir (target or log files). Testing this I noticed the below logged error:

23:46:54  Microsoft Fabric-Spark adapter: Error writing session ID to file: [Errno 13] Permission denied: '/home/site/wwwroot/livy-session-id.txt'

If I understand correctly this prevents any session reuse since livy-session-id.txt cannot be created or read?

If so, is it possible to make the session ID file location configurable instead of writing to the working directory, or even storing the ID in memory/state?

[mdrakiburrahman]

Hey @billy-moore-98

Good stuff!

Yea in my branch there's a session_id_file param you can pass it to make it configurable, see line 9:

https://github.com/mdrakiburrahman/spark-sandbox/blob/be630e934ea63f239e330f36962dc2ceee22cc0a/projects/spark-dbt/dbt-adventureworks/profiles.yml#L9

even storing the ID in memory/state?

I don't think this will work, because when you invoke multiple dbt commands like run and test, the dbt CLI exits, so the in-mem state would be lost.

I'm assuming durable functions should give you some HDD scratch space, popping the livy session ID there should work out, this is how I've been testing multiple dbt projects concurrently on the same machine without noisy neighbor.

[billy-moore-98]

Awesome, thanks @mdrakiburrahman !

[Kha7734]

It's really interesting, thanks @mdrakiburrahman

[billy-moore-98]

Any progress on getting this patched?

[mdrakiburrahman]

Thanks for the reminder, @billy-moore-98. I'm going to try to have a sync with @prdpsvs about seeing what his thoughts are on ownership of this repo.

[mdrakiburrahman]

I'll share a tangible update by end of this week, please tag me here if I forget for some reason :)