-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance Testing #2
Comments
Hi @rodpayne, Given your PR, I was wondering if you'd be interested in testing this fork as it has a new implementation that /might/ improve efficiency with your use case (but also might not). I'll note that this fork is probably best considered as alpha so if you'd like to wait till there's some more documentation that's also fine.
I was also wondering if you'd be willing to share some information about how you end up with such a low message count but very high report count. |
I gave it a try and haven't yet figured out how to build
I updated the
|
As you've noticed the repository layout has changed I suspect that you should instead use this copy: Though also I'll note, I've not been using docker/hatch to develop, I've just been using a virtual environment + |
Thank you, that change worked. I am running a test with a day's worth of input emails. While running for 11 hours so far, it has restarted from the beginning three times, with logs like the following:
While troubleshooting, I found it interesting that there are at least a couple of messages that seem to have the same UID:
|
I haven't made it all of the way through the benchmarks yet, but I have identified a couple of fixes to make it run faster. I will open a pull request so that you can take a look. Comparing partial runs, which could have some observation problems, 232 email messages took 11 hours and 29 minutes with the |
Ah, something I should add as well is the new application is The config format has changed as well, you'd want something like the following: parser:
nameservers:
- 1.1.1.1
- 8.8.8.8
sources:
rods-email:
class: .email:MicrosoftGraph
auth_method: UsernamePassword
client_id: rodsid
clinet_secret: rodssecret
sinks:
elasticsearch:
class: .elasticsearch:Elasticsearch
client:
hosts: localhost:9200
|
I think that I am going to have to wait for some documentation on the new config file format. I've gotten lost in the weeds after switching from |
Yeah that's fair enough, I'll let you know when we've got better documentation up. |
I was getting What is your preferred python version? |
That is somewhat surprising. I've run it on a mix, though I can say the I have it reliably running on 3.11 (it is too late at night for me to look into my other devices). That said, the tests are running in 3.8 so I would have though it got caught. It's possible that it's from a file that is missing a |
Here is some detail:
|
To close the loop on the above thread of comments - this was fixed in #30 |
It would be good for some "heavy" users of parsedmarc to start testing the fork.
Starting this issue as a place to aggregate the data / find out a bit more of the use-cases.
The text was updated successfully, but these errors were encountered: