Repository navigation
Migrating Vroom data into Seaweed should not interrupt service uptime #4263
Description
Self-Hosted Version
25.5.1
CPU Architecture
x86_64
Docker Version
29.3.1
Docker Compose Version
5.1.1
Machine Specification
- My system meets the minimum system requirements of Sentry
Installation Type
Upgrade from 25.5.1 to 26.3.1
Steps to Reproduce
- run install.sh
Expected Result
Sentry is successfully upgraded
Actual Result
Upgrade fails. Last lines of output before the failure:
upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' (14659 bytes in 0.0 seconds, 4.71 MB/s) [6097783 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' (211 bytes in 0.0 seconds, 81.24 KB/s) [6097784 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53' (22012 bytes in 0.0 seconds, 6.63 MB/s) [6097785 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' (211 bytes in 0.0 seconds, 73.90 KB/s) [6097786 of 6097786]
Done. Uploaded 94406925034 bytes in 29262.6 seconds, 3.08 MB/s.
Migration completed.
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 15 sec...
ERROR: S3 Temporary Error: Request failed for: /?lifecycle. Please try again later.
Error in install/bootstrap-s3-profiles.sh:112.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' setlifecycle /tmp/profiles-lifecycle-policy.xml s3://profiles' exited with status 75
-> ./install.sh:main:43
--> install/bootstrap-s3-profiles.sh:source:112
Clearly, trying to migrate ~ 6.1 million files into s3 (took roughly 12 hours) and then expecting to apply lifecycle policy within a few minutes was quite naive.
Install script should probably stop executing and offer a proper solution when so many files found in vroom volume. Or at least continue the upgrade after s3cmd failed to apply lifecycle policy, or even do it in the background without waiting for the result.
Event ID
No response
Metadata
Metadata
Assignees
Projects
- Status
- Status
Activity
aldy505 commentedon Apr 7, 2026
I'll rename your issue to fit the problem instead of making it an "upgrade issue".
[-]Upgrade 25.5.1 -> 26.3.1 failed[/-][+]Migrating Vroom data into Seaweed should not interrupt service uptime[/+]afaianswq commentedon Apr 7, 2026
Sure, the goal is to make sure the upgrade will not fail because of this problem.
Meanwhile, is it safe to simply delete
sentry-self-hosted_sentry-vroomvolume before the upgrade to skip this migration entirely? AFAIK, this volume contains not important runtime data.aldy505 commentedon Apr 8, 2026
No you cannot, it will result in an invalid Docker volume state.
26 remaining items
afaianswq commentedon May 15, 2026
Makes sense. Maybe you can make it optional? Installation script can explain that migration may take 20+ hours and offer to skip it entirely if loosing current profiling data is acceptable.
dlouzan commentedon May 16, 2026
@aldy505 @afaianswq The problem is that currently, we can't even apply the lifecycle policy on the existing storage. We have no indication of how long it would take nor how much memory it would need. For my tests on our stage instance, it meant having the whole host down for 1 hour and the linux oom manager started killing things, so weedfs would need even more memory.
It seems the lifecycle did not apply cleanly when we migrated to seaweedfs, and now there's too much data to be able to apply the lifecycle. The implication "too-much-data-hence-can't-apply-the-policy" is of course pure speculation on my end, I do not know the internals of seaweedfs.
Since it's a stage instance we wouldn't have an issue making another test and letting it run let's say 24h, but for production this is obviously not ok, as it means full downtime. With the current approach, the only option is to lose all existing data, which is also not ok.
aldy505 commentedon May 17, 2026
I've pushed some changes onto #4268, feel free to take a look & comments if you think the approach is awful.
gstorme commentedon May 17, 2026
Maybe test it on seaweedfs 4.24, they have done changes to how lifecycle commands (such as
PutBucketLifecycleConfigurationHandler) are processed.More info on seaweedfs/seaweedfs#9346
aldy505 commentedon May 22, 2026
@gstorme doing it here: #4345