replace lakefs_client with lakefs-sdk in Spark integration test harness

[mahdikhashan]

Closes #8391

Change Description

Background

Replace lakefs_client (deprecated python client) with the lakefs-sdk (1.43.0).

Bug Fix

Since in the Spark Integration tests the old deprecated client was used, this PR introduces the new python client (LakeFS_sdk).

New Feature

No new feature was added.

Testing Details

Will be added later, right now I'm struggling with running the docker-compose which contains the testing environment on my local machine.

Breaking Change?

No Breaking changes.

Additional info

Logs, outputs, screenshots of changes if applicable (CLI / GUI changes)

Contact Details

You can contact me here or by myemail: [email protected]

[arielshaqed]

Hi @mahdikhashan ,
Thanks for your contribution! This is very much a needed change. I would like to explain why it may take a while to run ci on this.
There were security risks on GitHub with running CI on PRs that did but originate from the project group. As a result, it doesn't run automatically. I will make an effort to get it to run promptly on your changes.
I suspect we may need additional changes to get the Spark client integration test to run; I'll let you know. In the meantime I'm not requesting changes but also but approving; sorry for keeping you hanging. We should make progress on this PR by Monday.

[arielshaqed]

Running CI on #8405; will push that but easier to run CI there.

[arielshaqed]

Okay, I can run Esti (system tests, including the Spark test affected by this PR). First failure is due to using repositories not repositories_api on the lakeFS Client object.

A guide to running system tests locally is here. This is roughly what you'd need in order to run the Spark test locally - but unfortunately not exactly.

[mahdikhashan]

Okay, I can run Esti (system tests, including the Spark test affected by this PR). First failure is due to using repositories not repositories_api on the lakeFS Client object.

A guide to running system tests locally is here. This is roughly what you'd need in order to run the Spark test locally - but unfortunately not exactly.

thanks @arielshaqed, I'll setup the local test env and fix the failing ones asap.

[arielshaqed]

[...]

thanks @arielshaqed, I'll setup the local test env and fix the failing ones asap.

Hi @mahdikhashan ,
Worth mentioning - I believe I am not currently blocking you here. If I am, please do let me know!

[mahdikhashan]

[...]

thanks @arielshaqed, I'll setup the local test env and fix the failing ones asap.

Hi @mahdikhashan , Worth mentioning - I believe I am not currently blocking you here. If I am, please do let me know!

Thanks for remaining me of this, I got busy with university coursework. I'll dedicate some time on it today (end of the day, vienna time) and move it forward - hopefully I deliver my changes by tomorrow.

[mahdikhashan]

hey @arielshaqed, as you mentioned above, the doc to run the spark tests are poorly written. I have a few questions now:

to run the spark tests, I need to have working version of local development setup if lakefs with spark, its worker and postgress in a docker env.

I have tried two docker-compose I found in the repo,

1. docker-compose at test/spark path

for this one I receive:

2024-12-06T12:42:27.527090888Z time="2024-12-06T12:42:27Z" level=fatal msg="Invalid config" func=cmd/lakefs/cmd.initConfig file="cmd/root.go:156" error="bad configuration: missing required keys: [blockstore.type]" fields.file=/home/lakefs/.lakefs.yaml file="cmd/root.go:156" phase=startup

which the config is:

credentials:
  access_key_id: AKIAIOSFODNN7EXAMPLE
  secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
server:
  endpoint_url: http://lakefs:8000

could you help me figure out why this config is invalid?

I also tried the following config without any success:

database:
  type: "local"
  local:
    path: /tmp/lakefs/metadata
auth:
  encrypt:
    secret_key: "some random value that should be replaced for production environments"

blockstore:
  type: local
  local:
    path: /tmp/lakefs/data

logging:
  format: text
  level: trace

2. docker-compose at etsi/ops

for this one I receive:

✘ lakefs Error failed to resolve reference "docker.io/treeverse/lakefs:dev": docker.io/treeverse/lakefs:dev: not found 1.1s
✘ esti Error context canceled

it seems that in set_env_vars.sh, I may need to define some vars, in this case would you be happy to share if any exists?

looking forward to hearing from you

[mahdikhashan]

otherwise, i fixed this attribute issue: AttributeError: 'LakeFSClient' object has no attribute 'repositories'. Did you mean: 'repositories_api'?

[arielshaqed]

Hi @mahdikhashan ,
This line:

error="bad configuration: missing required keys: [blockstore.type]"
indicates that your configuration is missing a blockstore type. Indeed, you need to configure a blockstore there.

Your second issue is easier to fix. You need to build an appropriate docker image, since you are in dev. Can you try to make build-docker? That will build a local Docker image that Esti can use.

[mahdikhashan]

Hi @mahdikhashan , This line:

error="bad configuration: missing required keys: [blockstore.type]"
indicates that your configuration is missing a blockstore type. Indeed, you need to configure a blockstore there.

Your second issue is easier to fix. You need to build an appropriate docker image, since you are in dev. Can you try to make build-docker? That will build a local Docker image that Esti can use.

ok, thanks. I'll follow up from here. will update you.