Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private link error handling #3689

Merged
merged 6 commits into from
Oct 30, 2024
Merged

Private link error handling #3689

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d85df82
throw permission error if swallowed in mlflow exception stacktrace
nancyhung Oct 29, 2024
d3cf117
error message
nancyhung Oct 29, 2024
acddb15
Merge branch 'main' into nancy/pl-error
nancyhung Oct 29, 2024
ba1e244
Update composer/utils/object_store/mlflow_object_store.py
nancyhung Oct 29, 2024
b74a761
ok well we retry on 500 i guess
nancyhung Oct 29, 2024
3d308f8
linter and pr feedback
nancyhung Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions composer/utils/object_store/mlflow_object_store.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ def _wrap_mlflow_exceptions(uri: str, e: Exception):
retryable_server_codes = [
ErrorCode.Name(code) for code in [
DATA_LOSS,
INTERNAL_ERROR,
nancyhung marked this conversation as resolved.
Show resolved Hide resolved
INVALID_STATE,
TEMPORARILY_UNAVAILABLE,
DEADLINE_EXCEEDED,
Expand All @@ -62,9 +61,19 @@ def _wrap_mlflow_exceptions(uri: str, e: Exception):
retryable_client_codes = [ErrorCode.Name(code) for code in [ABORTED, REQUEST_LIMIT_EXCEEDED, RESOURCE_EXHAUSTED]]
not_found_codes = [ErrorCode.Name(code) for code in [RESOURCE_DOES_NOT_EXIST, NOT_FOUND, ENDPOINT_NOT_FOUND]]

# MLflow wraps Azure data exceptions as INTERNAL_ERROR. Need to unwrap and check msg for the specific error.
non_retryable_internal_error_codes = [
'401',
'403',
]

if isinstance(e, MlflowException):
error_code = e.error_code # pyright: ignore
if error_code in retryable_server_codes or error_code in retryable_client_codes:
if error_code == ErrorCode.Name(INTERNAL_ERROR):
error_message = e.message # pyright: ignore
if any(code in error_message for code in non_retryable_internal_error_codes):
nancyhung marked this conversation as resolved.
Show resolved Hide resolved
raise PermissionError(f'Permission denied for object {uri} from the data provider. Details: {error_message}') from e
nancyhung marked this conversation as resolved.
Show resolved Hide resolved
elif error_code in retryable_server_codes or error_code in retryable_client_codes:
raise ObjectStoreTransientError(error_code) from e
elif error_code in not_found_codes:
raise FileNotFoundError(f'Object {uri} not found') from e
Expand Down
Loading