scylla_image_setup: avoid script error when perftune.py failed #787
Conversation
|
@syuu1228 please rebase to latest |
|
|
||
| LOGGER = logging.getLogger(__name__) | ||
|
|
||
| def disable_perftune(): |
There was a problem hiding this comment.
@syuu1228 don't we need perftune running ? what is the impact of disabling it?
|
@syuu1228 please rebase , it seems you 2 additional commits that don't suppose to be here |
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
from
7b3c918 to
c780e71
Compare
|
Rebased with latest next |
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
2 times, most recently
from
78ebc60 to
7dba512
Compare
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
2 times, most recently
from
adbb50a to
ab18dcf
Compare
|
Added code to handle special exit code, which is added scylla_sysconfig_setup on scylladb/scylladb#26344 |
|
next-machine-image passed: https://jenkins.scylladb.com/view/master/job/scylla-master/job/releng-testing/job/next-machine-image/544/ |
|
@avikivity Please review this as well, this is scylla-machine-image part of scylladb/seastar#2956 |
@avikivity ping review |
| except subprocess.CalledProcessError as e: | ||
| if e.returncode == 3: | ||
| disable_perftune() | ||
| LOGGER.warning('Failed to enable perftune.py, continue without using it') |
There was a problem hiding this comment.
@yaronkaikov one problem this change may cause is that our tests will miss these failures.
We may need to introduce a specific check that this warning isn't there.
There was a problem hiding this comment.
I have talked with @syuu1228 about this, we should probably raise this message only for the known instance types with the issues we know of, other wise when we hit a regression we would never find it
There was a problem hiding this comment.
Implemented the limitation. Now it avoid backtrace only on im4gn.8xlarge and im4gn.16xlarge, which are the target of this patch. On other instance type or on other IaaS, it just causes backtrace.
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
2 times, most recently
from
ec7b1dc to
14dd0ae
Compare
| except subprocess.CalledProcessError as e: | ||
| if e.returncode == 3: | ||
| disable_perftune() | ||
| LOGGER.warning('Failed to enable perftune.py, continue without using it') |
There was a problem hiding this comment.
I have talked with @syuu1228 about this, we should probably raise this message only for the known instance types with the issues we know of, other wise when we hit a regression we would never find it
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
3 times, most recently
from
0feeab4 to
43b1037
Compare
|
next-machine-image passed: https://jenkins.scylladb.com/view/master/job/scylla-master/job/releng-testing/job/next-machine-image/577/ |
|
@gmizrahi Please test this PR, here's AMI ID and version tag: |
|
@yaronkaikov please review updated patch |
yaronkaikov
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
from
43b1037 to
498c324
Compare
yaronkaikov
dismissed
their stale review
The merge-base changed after approval.
Switch from print() to logging API on scylla_image_setup.
On EC2, some environments cause errors on perftune.py due to corrupted NUMA topology information. Even in such an environment, scylla_image_setup should not cause a script error. It should handle the error, print a warning message, and then continue the startup process. Related scylladb/seastar#2925
syuu1228
force-pushed
the
scylla_image_setup_avoid_traceback_on_perftune_error
branch
from
498c324 to
ba96671
Compare
@avikivity ping |
|
longevity failed, but the error is known issue so we can ignore this (described at #712 (comment)): https://jenkins.scylladb.com/view/master/job/scylla-master/job/releng-testing/job/longevity/job/longevity-100gb-4h-test/110/ |
|
@yaronkaikov all tests passed, I think we can merge this |
5 participants
On EC2, some environments cause errors on perftune.py due to corrupted
NUMA topology information.
Even in such an environment, scylla_image_setup should not cause a script error.
It should handle the error, print a warning message, and then continue the startup process.
Related scylladb/seastar#2925
This is scylla-machine-image part of the fix for scylladb/seastar#2925