Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Explorers #237

Open
wants to merge 76 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
0a798b5
added config for Explorer results
NiciXitri Sep 5, 2024
3e2c340
added explorer model and endpoints
NiciXitri Sep 5, 2024
bfd070f
added explorer Job
NiciXitri Sep 5, 2024
7f2ef23
added validations and retrieval of results
NiciXitri Sep 5, 2024
185e442
Merge remote-tracking branch 'origin/develop' into project/explorer
NiciXitri Sep 24, 2024
14c2a37
validation changes to explorers
NiciXitri Sep 24, 2024
d04dcbb
added explorer creation, launch and selection to frontend
NiciXitri Sep 24, 2024
ee1ea1c
fix typing error
NiciXitri Sep 24, 2024
f8fdc8d
fix typing errors
NiciXitri Sep 24, 2024
13161c9
added visualization for explorations results
NiciXitri Sep 26, 2024
c4fe4c7
modifications to metadata and results getter
NiciXitri Sep 26, 2024
8d25e82
added explorations > explorers on backend
NiciXitri Oct 10, 2024
414c000
added explorations > explorers on frontend
NiciXitri Oct 10, 2024
8569469
(explorers) removed deprecated files
NiciXitri Oct 10, 2024
9911785
fix pre-commit errors
NiciXitri Oct 10, 2024
4b7bf63
added shared components
NiciXitri Oct 24, 2024
74fd36c
added prepare dataset to explorer
NiciXitri Oct 24, 2024
bea941b
modified schema and parameter preparation
NiciXitri Oct 24, 2024
f3951f6
added scatter plot explorer
NiciXitri Oct 24, 2024
1d7ebb3
take split in explorer
NiciXitri Oct 24, 2024
d1f8113
modified actions to use TooltipedCellItem
NiciXitri Oct 24, 2024
47807ca
explorer module context modification
NiciXitri Oct 24, 2024
03aace9
small fix to get_sample
NiciXitri Oct 29, 2024
d5e7a1c
added visualizers
NiciXitri Oct 29, 2024
e615cea
optimization, documentation and details
NiciXitri Oct 29, 2024
795285c
feat: exploration runner & results visulization
NiciXitri Oct 29, 2024
b7e9d43
precommit prettier fix
NiciXitri Oct 29, 2024
d219a9d
fix: Fixed typing issues
NiciXitri Oct 29, 2024
1f2849c
added wordcloud dependency
NiciXitri Nov 6, 2024
04db4b8
front: added image visualizer
NiciXitri Nov 6, 2024
82e446f
back added wordcloud explorer
NiciXitri Nov 6, 2024
cd894d6
removed unused prop
NiciXitri Nov 6, 2024
e377b02
column flex on tabular results
NiciXitri Nov 13, 2024
8b27216
added row explorer
NiciXitri Nov 13, 2024
eb60c1c
added display name to explorers
NiciXitri Nov 16, 2024
b542cec
precommit fixes
NiciXitri Nov 16, 2024
5dfff7f
added ordered selection to columns
NiciXitri Nov 19, 2024
2c3672b
added path sanitization
NiciXitri Nov 19, 2024
2810aef
updated deletion logic
NiciXitri Nov 19, 2024
51a5ec9
updated column validation
NiciXitri Nov 19, 2024
af7f7ad
fix: added column order to automatic selection
NiciXitri Nov 19, 2024
3302483
fix: pre commit issues
NiciXitri Nov 19, 2024
6155e4f
added basic box_plot
NiciXitri Nov 19, 2024
fed0761
finished box plot explorers
NiciXitri Nov 19, 2024
7c4de3b
pre commit fixes
NiciXitri Nov 19, 2024
80f8bfa
fix: typechecking errors
NiciXitri Nov 19, 2024
e46cfe2
explorer tooltips now only take the description and disabled reason
NiciXitri Dec 17, 2024
23d7a8e
disabled filter and sorting model
NiciXitri Dec 17, 2024
079436c
added dataset columns refresh to context
NiciXitri Dec 17, 2024
f0487e0
added snackbar messages and modified buttons
NiciXitri Dec 17, 2024
f7b9beb
added tooltip to values on index column
NiciXitri Dec 17, 2024
0271b17
centered and added icons to detail tabs
NiciXitri Dec 17, 2024
76c2d1c
modified default viewMode and order
NiciXitri Dec 17, 2024
08fcc37
added optional closeOnBackdropClick property
NiciXitri Dec 17, 2024
0ed7695
skip column validation on patch methods
NiciXitri Dec 17, 2024
58726ad
Merge branch 'develop' into project/explorer
NiciXitri Dec 17, 2024
3e54783
tooltip does not follow cursor
NiciXitri Dec 17, 2024
3d16455
sort explorer options
NiciXitri Dec 17, 2024
7874f6b
added correlation and covariance matrix explorers
NiciXitri Dec 19, 2024
d43d178
small optimization
NiciXitri Dec 19, 2024
13b835d
display exploration type on result
NiciXitri Dec 22, 2024
b4bf240
small fixes and used plotly io module instead of pickle
NiciXitri Dec 22, 2024
9690a10
added more explorers
NiciXitri Dec 22, 2024
5b7d805
Merge branch 'develop' into project/explorer
NiciXitri Jan 17, 2025
7d58d51
added typeguard to explorerJob
NiciXitri Jan 29, 2025
53a58ea
Merge remote-tracking branch 'origin/develop' into project/explorer
NiciXitri Jan 29, 2025
4f14bc5
Merge remote-tracking branch 'origin/develop' into project/explorer
NiciXitri Feb 23, 2025
18b2d37
Merge remote-tracking branch 'origin/feat/predict-models-datasets' in…
NiciXitri Feb 23, 2025
ffe12a2
fix: updated explorers to use DashAIDataset
NiciXitri Feb 23, 2025
19e0992
fix: pre-commit issues
NiciXitri Feb 23, 2025
83b80ed
Merge remote-tracking branch 'origin/develop' into project/explorer
NiciXitri Mar 8, 2025
5f9d59e
compiled requirements with wordcloud
NiciXitri Mar 8, 2025
1674ec0
fix: fix endpoint format for jobs on frontend
NiciXitri Mar 8, 2025
d2072cc
fix: pip-compile adding pywin32 to requirements.txt
NiciXitri Mar 9, 2025
6134c8b
Merge remote-tracking branch 'origin/develop' into project/explorer
NiciXitri Mar 20, 2025
318d9b0
Merge branch 'develop' into project/explorer
Felipedino Mar 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions DashAI/back/api/api_v1/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
datasets,
experiments,
explainers,
explorations,
explorers,
jobs,
predict,
runs,
Expand All @@ -15,6 +17,8 @@
api_router_v1.include_router(datasets.router, prefix="/dataset")
api_router_v1.include_router(experiments.router, prefix="/experiment")
api_router_v1.include_router(explainers.router, prefix="/explainer")
api_router_v1.include_router(explorations.router, prefix="/exploration")
api_router_v1.include_router(explorers.router, prefix="/explorer")
api_router_v1.include_router(jobs.router, prefix="/job")
api_router_v1.include_router(runs.router, prefix="/run")
api_router_v1.include_router(predict.router, prefix="/predict")
206 changes: 206 additions & 0 deletions DashAI/back/api/api_v1/endpoints/explorations.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
import logging
import os
import pathlib

from beartype.typing import Any, Dict, List, Union
from fastapi import APIRouter, Depends, Response, status
from fastapi.exceptions import HTTPException
from kink import di, inject
from sqlalchemy.orm import Session, sessionmaker

from DashAI.back.api.api_v1.schemas import exploration_params as schemas
from DashAI.back.dependencies.database.models import Dataset, Exploration, Explorer

logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(__name__)

router = APIRouter()


# validations
def validate_exploration_params(
params: schemas.ExplorationCreate,
session: Session,
):
"""
Function to validate exploration parameters.
It validates:
- The `dataset_id` and `columns` against the dataset.
"""
dataset = session.query(Dataset).get(params.dataset_id)
if dataset is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Dataset not found",
)

return True


@router.get("/", response_model=List[schemas.Exploration])
@inject
async def get_explorations(
dataset_id: Union[int, None] = None,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
skip: int = 0,
limit: int = 0,
):
"""
Get all explorations.
"""
db: Session
with session_factory() as db:
if dataset_id is not None:
explorations = db.query(Exploration).filter(
Exploration.dataset_id == dataset_id
)
else:
explorations = db.query(Exploration)

if skip > 0:
explorations = explorations.offset(skip)
if limit > 0:
explorations = explorations.limit(limit)

return explorations.all()


@router.get("/{exploration_id}/", response_model=schemas.Exploration)
async def get_exploration_by_id(
exploration_id: int,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
):
"""
Get an exploration by id.
"""
db: Session
with session_factory() as db:
exploration = db.query(Exploration).get(exploration_id)
if exploration is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Exploration not found",
)
return exploration


@router.get("/dataset/{dataset_id}/", response_model=List[schemas.Exploration])
async def get_explorations_by_dataset_id(
dataset_id: int,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
skip: int = 0,
limit: int = 0,
):
"""
Get all explorations by dataset.
"""
db: Session
with session_factory() as db:
explorations = db.query(Exploration).filter(
Exploration.dataset_id == dataset_id
)

if skip > 0:
explorations = explorations.offset(skip)
if limit > 0:
explorations = explorations.limit(limit)

return explorations.all()


@router.post(
"/", response_model=schemas.Exploration, status_code=status.HTTP_201_CREATED
)
@inject
async def create_exploration(
params: schemas.ExplorationCreate,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
):
"""
Create an exploration.
"""
db: Session
with session_factory() as db:
exploration = Exploration(**params.model_dump())
validate_exploration_params(exploration, db)

db.add(exploration)
db.commit()
db.refresh(exploration)
return exploration


@router.patch("/{exploration_id}/", response_model=schemas.Exploration)
@inject
async def update_exploration(
exploration_id: int,
params: schemas.ExplorationBase,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
):
"""
Update an exploration by id.
"""
db: Session
with session_factory() as db:
exploration = db.query(Exploration).get(exploration_id)
if exploration is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Exploration not found",
)

params_dict = params.model_dump()
for key, value in params_dict.items():
setattr(exploration, key, value)

db.commit()
db.refresh(exploration)
return exploration


@router.delete("/{exploration_id}/", status_code=status.HTTP_204_NO_CONTENT)
@inject
async def delete_exploration(
exploration_id: int,
session_factory: sessionmaker = Depends(lambda: di["session_factory"]),
config: Dict[str, Any] = Depends(lambda: di["config"]),
):
"""
Delete an exploration by id.
"""
db: Session
with session_factory() as db:
exploration = db.query(Exploration).get(exploration_id)
if exploration is None:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Exploration not found",
)

# delete explorers and their results
explorers = db.query(Explorer).filter(Explorer.exploration_id == exploration_id)
for explorer in explorers:
db.delete(explorer)
explorer.delete_result()

db.delete(exploration)

dir_path = pathlib.Path(
os.path.join(
config["EXPLORATIONS_PATH"],
(f"{exploration.id}"),
)
)

if dir_path.exists() and dir_path.is_dir():
# remove the directory if it is empty
if len(list(dir_path.iterdir())) == 0:
dir_path.rmdir()
else:
log.warning(
f"Exploration Directory {dir_path} is not empty. "
"It will not be removed."
)

db.commit()
return Response(status_code=status.HTTP_204_NO_CONTENT)
Loading