Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve camouflage startup issues #1436

Merged

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Dec 14, 2023

Description

This fixes three issues:

  1. Camouflage not starting up within the 10s timeout when the CI system is under high load. This was fixed by simply increasing the timeout to 30s.
  2. Camouflage startup errors (command not fount, out of memory etc...) not being reported since stdout/stderr were being sent to /dev/null. This was fixed by redirecting stdout/stderr to a file, and displaying the content on failure.

This is a work-around for not being able to control the log level of camouflage's console logger:
testinggospels/camouflage#244

We can't redirect stdout/stderr to a PIPE since on successful startup the PIPE would fill and block camouflage.

Closes #1417

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

Improve camouflage error reporting by redirecting stdout and stderr to a file.
This is a work-around for not being able to control the log level of camouflage's console logger
testinggospels/camouflage#244

We can't redirect stdout/stderr to a PIPE since on successful startup the PIPE would fill and block
camouflage. On the other hand sending stdout/stderr to /dev/null prevents users from seeing errors
from the shell/OS regarding camouflage failing to start.
@dagardner-nv dagardner-nv added bug Something isn't working non-breaking Non-breaking change labels Dec 14, 2023
@dagardner-nv dagardner-nv self-assigned this Dec 14, 2023
@dagardner-nv dagardner-nv requested a review from a team as a code owner December 14, 2023 16:33
@dagardner-nv dagardner-nv marked this pull request as draft December 14, 2023 18:00
@dagardner-nv dagardner-nv marked this pull request as ready for review December 14, 2023 18:13
@dagardner-nv dagardner-nv marked this pull request as draft December 14, 2023 21:26
@dagardner-nv dagardner-nv marked this pull request as ready for review December 18, 2023 19:28
@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit fd96b68 into nv-morpheus:branch-24.03 Dec 20, 2023
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[BUG]: Launching camouflage intermittently fails failing in CI
2 participants