Logging from tiered storage produces 5k log messages per day

[vsarunas]

Version & Environment

v24.3.1

What went wrong?

Dec  9 23:48:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:48:36,215 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:49:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:49:06,288 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:49:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:49:06,288 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:49:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:49:36,362 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:49:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:49:36,363 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:50:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:50:06,434 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:50:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:50:06,434 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:50:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:50:36,505 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:50:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:50:36,506 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:51:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:51:06,577 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:51:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:51:06,577 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:51:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:51:36,648 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:51:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:51:36,648 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:52:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:52:06,721 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:52:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:52:06,721 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:52:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:52:36,792 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:52:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:52:36,793 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:53:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:53:06,871 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:53:06 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:53:06,871 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:53:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:53:36,962 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:53:36 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:53:36,963 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:54:07 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:54:07,035 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:54:07 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:54:07,035 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:54:37 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:54:37,107 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes
Dec  9 23:54:37 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:54:37,108 [shard 0:main] resource_mgmt - storage.cc:641 - No tiered storage partitions were found, unable to reclaim
Dec  9 23:55:07 demo-redpanda-1 rpk[515778]: INFO  2024-12-09 23:55:07,179 [shard 0:main] resource_mgmt - storage.cc:596 - Log storage usage 68.574GiB > target size 59.971GiB by 8.603GiB (adjusted 17.206GiB). Garbage collection expected to remove 0.000bytes. Space management of tiered storage topics to remove 17.206GiB. Total estimated available to remove 0.000bytes

This logs over 5k logs per day:

grep rpk /var/log/syslog | grep "storage.cc" | grep "Dec  9" | grep "tiered storage" | wc -l
5744

What should have happened instead?

No tiered storage partitions were found, unable to reclaim seems should be logged only if cloud storage is enabled?

[dotnwat]

hey @vsarunas thanks for the report.

you should be able to disable this warning by setting space_management_enable which doesn't have any affect when there are no tiered storage topics.

we'll fix this in an upcoming patch release.