Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure prune_logs can be used more efficiently when dealing with resource limits #45

Open
alextreme opened this issue Oct 31, 2024 · 0 comments

Comments

@alextreme
Copy link
Member

We have our prune_logs task here:

https://github.com/maykinmedia/django-log-outgoing-requests/blob/main/log_outgoing_requests/tasks.py

This works if you have unlimited resources, however when running this for a very large database (eg the task hasn't been run before) and in a resource-restricted environment (k8s resource limits) the following can occur:

In [1]: from log_outgoing_requests.models import *                                                                                                                                                                           

In [2]: OutgoingRequestsLog.objects.first()                                                                                                                                                                                  
Out[2]: <OutgoingRequestsLog: api.twitter.com at 2024-02-07 12:10:00.254395+00:00>

In [3]: OutgoingRequestsLog.objects.count()                                                                                                                                                                                  
Out[3]: 697150

In [4]: from datetime import timedelta                                                                                                                            

In [5]: now = timezone.now()                                                                                                                                      

In [6]: settings.LOG_OUTGOING_REQUESTS_MAX_AGE                                                                                                                    
Out[6]: 30

In [7]: OutgoingRequestsLog.objects.filter(timestamp__lt=now - timedelta(180)).delete()                                                                           
Geëlimineerd

I'd recommend to perform this delete in batches (it wasn't a problem when I did this in batches of ~60000 logs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant