Skip to content

Conversation

rockwotj
Copy link
Contributor

@rockwotj rockwotj commented Sep 24, 2025

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.2.x
  • v25.1.x
  • v24.3.x

Release Notes

  • none

@Copilot Copilot AI review requested due to automatic review settings September 24, 2025 01:46
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enables hardened libc++ for non-release builds to improve safety and debugging capabilities while maintaining maximum performance in production releases.

  • Adds extensive hardening mode for debug and fastbuild configurations
  • Explicitly disables hardening for release builds to prioritize performance
  • Applies hardening settings to both regular and host compilation flags

@Copilot Copilot AI review requested due to automatic review settings September 24, 2025 01:57
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

.bazelrc Outdated
# the fast mode in production as well.
#
# https://libcxx.llvm.org/Hardening.html
build --cxxopt=-D_LIBCPP_ASSERTION_SEMANTIC=_LIBCPP_ASSERTION_SEMANTIC_ENFORCE
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ASSERTION macro is only in LLVM 21...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh well we can go to clang 21 its at its second point release now :P

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building LLVM 21, at least to test this CI run

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Sep 24, 2025

CI test results

test results on build#72801
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
DisablingPartitionsTest test_disable null integration https://buildkite.com/redpanda/redpanda/builds/72801#019979bc-3159-46b2-8715-9bc33b3638e6 FLAKY 14/21 upstream reliability is '87.87878787878788'. current run reliability is '66.66666666666666'. drift is 21.21212 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DisablingPartitionsTest&test_method=test_disable
WriteCachingFailureInjectionTest test_unavoidable_data_loss null integration https://buildkite.com/redpanda/redpanda/builds/72801#019979bc-3159-46b2-8715-9bc33b3638e6 FLAKY 18/21 upstream reliability is '95.0909090909091'. current run reliability is '85.71428571428571'. drift is 9.37662 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteCachingFailureInjectionTest&test_method=test_unavoidable_data_loss
src/v/cloud_storage/tests/cloud_storage_e2e_test src/v/cloud_storage/tests/cloud_storage_e2e_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cloud_storage/tests/delete_records_e2e_test src/v/cloud_storage/tests/delete_records_e2e_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cloud_storage/tests/read_replica_test src/v/cloud_storage/tests/read_replica_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cloud_storage/tests/topic_namespace_override_recovery_test src/v/cloud_storage/tests/topic_namespace_override_recovery_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/archival/tests/ntp_archiver_reupload_test src/v/cluster/archival/tests/ntp_archiver_reupload_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/archival/tests/ntp_archiver_test src/v/cluster/archival/tests/ntp_archiver_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/cloud_metadata/tests/cluster_recovery_backend_test src/v/cluster/cloud_metadata/tests/cluster_recovery_backend_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/data_migration_table_test src/v/cluster/tests/data_migration_table_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/partition_balancer_planner_test src/v/cluster/tests/partition_balancer_planner_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/partition_balancer_rpbench_test src/v/cluster/tests/partition_balancer_rpbench_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/partition_balancer_simulator_test src/v/cluster/tests/partition_balancer_simulator_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/partition_leaders_table_test src/v/cluster/tests/partition_leaders_table_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/plugin_frontend_validation_test src/v/cluster/tests/plugin_frontend_validation_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/rm_stm_test src/v/cluster/tests/rm_stm_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/topic_table_partition_generator_test src/v/cluster/tests/topic_table_partition_generator_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/topic_table_test src/v/cluster/tests/topic_table_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/cluster/tests/topic_updates_dispatcher_test src/v/cluster/tests/topic_updates_dispatcher_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/config/tests/bounded_property_test src/v/config/tests/bounded_property_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/config/tests/scoped_config_test src/v/config/tests/scoped_config_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/datalake/coordinator/tests/coordinator_test src/v/datalake/coordinator/tests/coordinator_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/datalake/coordinator/tests/state_machine_test src/v/datalake/coordinator/tests/state_machine_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/datalake/tests/translator_fixture_test src/v/datalake/tests/translator_fixture_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/iceberg/tests/values_json_test src/v/iceberg/tests/values_json_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/raft/tests/group_configuration_test src/v/raft/tests/group_configuration_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
src/v/storage/tests/timequery_test src/v/storage/tests/timequery_test unit https://buildkite.com/redpanda/redpanda/builds/72801#01997971-783d-462b-b358-aa51cfe6a109 FAIL 0/1
test results on build#72865
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
debug_bundle_service_started_fixture restart_unsuccessful_run unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
EndToEndCloudTopicsTest test_write null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-6794-49f4-b179-3e2d9a567337 FLAKY 16/21 upstream reliability is '92.58793969849246'. current run reliability is '76.19047619047619'. drift is 16.39746 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsTest&test_method=test_write
EndToEndCloudTopicsTxTest test_write null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-678a-4682-8ba5-6bea1c343c88 FLAKY 10/21 upstream reliability is '82.81938325991189'. current run reliability is '47.61904761904761'. drift is 35.20034 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsTxTest&test_method=test_write
ClusterRateQuotaTest test_client_group_consume_rate_throttle_mechanism null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-678c-4464-9e4a-87a7dabcab63 FLAKY 12/21 upstream reliability is '82.3905109489051'. current run reliability is '57.14285714285714'. drift is 25.24765 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ClusterRateQuotaTest&test_method=test_client_group_consume_rate_throttle_mechanism
ClusterRateQuotaTest test_client_response_throttle_mechanism_applies_to_next_request null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-6791-4889-a96c-d241dbcace46 FLAKY 18/21 upstream reliability is '84.04255319148936'. current run reliability is '85.71428571428571'. drift is -1.67173 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ClusterRateQuotaTest&test_method=test_client_response_throttle_mechanism_applies_to_next_request
DisablingPartitionsTest test_disable null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-6794-49f4-b179-3e2d9a567337 FLAKY 16/21 upstream reliability is '99.33184855233853'. current run reliability is '76.19047619047619'. drift is 23.14137 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DisablingPartitionsTest&test_method=test_disable
RecoveryModeTest test_rolling_restart null integration https://buildkite.com/redpanda/redpanda/builds/72865#01997cec-678c-4464-9e4a-87a7dabcab63 FLAKY 16/21 upstream reliability is '94.54225352112677'. current run reliability is '76.19047619047619'. drift is 18.35178 and the allowed drift is set to 50. The test should PASS https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=RecoveryModeTest&test_method=test_rolling_restart
src/v/cloud_storage/tests/cloud_storage_e2e_test src/v/cloud_storage/tests/cloud_storage_e2e_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cloud_storage/tests/delete_records_e2e_test src/v/cloud_storage/tests/delete_records_e2e_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cloud_storage/tests/read_replica_test src/v/cloud_storage/tests/read_replica_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cloud_storage/tests/topic_namespace_override_recovery_test src/v/cloud_storage/tests/topic_namespace_override_recovery_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/archival/tests/ntp_archiver_reupload_test src/v/cluster/archival/tests/ntp_archiver_reupload_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/archival/tests/ntp_archiver_test src/v/cluster/archival/tests/ntp_archiver_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/cloud_metadata/tests/cluster_recovery_backend_test src/v/cluster/cloud_metadata/tests/cluster_recovery_backend_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/data_migration_table_test src/v/cluster/tests/data_migration_table_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/partition_balancer_planner_test src/v/cluster/tests/partition_balancer_planner_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/partition_balancer_rpbench_test src/v/cluster/tests/partition_balancer_rpbench_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/partition_balancer_simulator_test src/v/cluster/tests/partition_balancer_simulator_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/partition_leaders_table_test src/v/cluster/tests/partition_leaders_table_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/plugin_frontend_validation_test src/v/cluster/tests/plugin_frontend_validation_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/rm_stm_test src/v/cluster/tests/rm_stm_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/topic_table_partition_generator_test src/v/cluster/tests/topic_table_partition_generator_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/topic_table_test src/v/cluster/tests/topic_table_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/cluster/tests/topic_updates_dispatcher_test src/v/cluster/tests/topic_updates_dispatcher_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/config/tests/bounded_property_test src/v/config/tests/bounded_property_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/config/tests/scoped_config_test src/v/config/tests/scoped_config_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/datalake/coordinator/tests/coordinator_test src/v/datalake/coordinator/tests/coordinator_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/datalake/coordinator/tests/state_machine_test src/v/datalake/coordinator/tests/state_machine_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/datalake/tests/translator_fixture_test src/v/datalake/tests/translator_fixture_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/raft/tests/group_configuration_test src/v/raft/tests/group_configuration_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1
src/v/storage/tests/timequery_test src/v/storage/tests/timequery_test unit https://buildkite.com/redpanda/redpanda/builds/72865#01997ca1-665f-4564-ae11-87a0a2f1e780 FAIL 0/1

@dotnwat
Copy link
Member

dotnwat commented Sep 25, 2025

that's cool. quite a few test failures to sort through

@dotnwat
Copy link
Member

dotnwat commented Sep 25, 2025

happy to help if they look like valid failures

@rockwotj
Copy link
Contributor Author

happy to help if they look like valid failures

They are certainly UB (like where front was called on an empty span), but I have just started going through them. I have a patch locally that gives better errors and stacktraces (by default they use sigill and by default seastar blocks that signal). I should just triage them and put them in jira.

decimal_value{absl::int128{std::numeric_limits<long>::max()}},
decimal_type{21, 0},
R"("9223372036854775807.")",
R"("9223372036854775807")",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

em? the test explicitly states it tests with trailing decimal

other tests too, they explicitly don't have integral parts

# https://libcxx.llvm.org/Hardening.html
build --cxxopt=-D_LIBCPP_ASSERTION_SEMANTIC=_LIBCPP_ASSERTION_SEMANTIC_ENFORCE
build --cxxopt=-D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE
build --host_cxxopt=-D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not _LIBCPP_HARDENING_MODE_DEBUG?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extensive mode, which contains all the checks from fast mode and some additional checks for undefined behavior that incur relatively little overhead but aren’t security-critical. Production builds requiring a broader set of checks than fast mode should consider enabling extensive mode. The additional rigour impacts performance more than fast mode: we recommend benchmarking to determine if that is acceptable for your program.

Debug mode, which enables all the available checks in the library, including heuristic checks that might have significant performance overhead as well as internal library assertions. This mode should be used in non-production environments (such as test suites, CI, or local development). We do not commit to a particular level of performance in this mode. In particular, this mode is not intended to be used in production.

https://libcxx.llvm.org/Hardening.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants