[WIP] Add Local Dashboard for Profiler #2000
Conversation
|
✅ 27/27 passed, 2 flaky, 1m39s total Flaky tests:
Running from acceptance #2236 |
goodwillpunning
temporarily deployed
to
tool
GitHub Actions
Inactive
| dashboard_str = dashboard_str.replace("`PROFILER_SCHEMA`", f"`{schema_name}`") | ||
|
|
||
| # TODO: check if the dashboard exists and unpublish it if it does | ||
| # TODO: create a warehouse ID |
There was a problem hiding this comment.
This should be taken care during deployment this part assumes that necessary infra is setup already
| logging.basicConfig(level=logging.INFO) | ||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
There was a problem hiding this comment.
Add a Comment to say why we are deviating from LSQL deployment, so other reviewers don't block this PR.
|
|
||
| def __init__(self, ws: WorkspaceClient, current_user: User, is_debug: bool = False): | ||
| self._ws = ws | ||
| self._current_user = current_user |
There was a problem hiding this comment.
I think you don't need this attribute, you can retrive form workspace client
| def __init__(self, ws: WorkspaceClient, current_user: User, is_debug: bool = False): | ||
| self._ws = ws | ||
| self._current_user = current_user | ||
| self._dashboard_location = f"/Workspace/Users/{self._current_user}/Lakebridge/Dashboards" |
There was a problem hiding this comment.
We should rely on the blueprint to save the file workspace. It should be inside the .lakebridge folder
| "tasks": [ | ||
| NotebookTask( | ||
| notebook_path=f"/Workspace/{databricks_user}/Lakebridge/profiler/load_extracted_tables.py", | ||
| base_parameters={ |
There was a problem hiding this comment.
There are 2 ways we can implement this, have the ingestion job as python package and use a wheel task
Or have the notebook upload and then run the jobs.
I prefer option 1.
| assessment.run() | ||
|
|
||
|
|
||
| @lakebridge.command(is_unauthenticated=False) |
There was a problem hiding this comment.
| @lakebridge.command(is_unauthenticated=False) | |
| @lakebridge.command() |
Changes
What does this PR do?
Relevant implementation details
Caveats/things to watch out for when reviewing:
Linked issues
Functionality
databricks labs lakebridge ...Tests