Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

robotsMap memory continues to grow #815

Open
ZigHuang opened this issue May 13, 2024 · 1 comment
Open

robotsMap memory continues to grow #815

ZigHuang opened this issue May 13, 2024 · 1 comment

Comments

@ZigHuang
Copy link

ZigHuang commented May 13, 2024

Hello, I am using colly to visit some websites and set c.IgnoreRobotsTxt = false.

As it runs, you will observe that the memory continues to grow over a relatively long period of time.

This growth rate is difficult to observe by using pprof.

As a control experiment, I set up a set of colly with the same configuration, the only difference was setting c.IgnoreRobotsTxt = true.

After running for a period of time, the memory of the latter continues to be stable within 1G, but the memory of the former continues to increase.

截屏2024-05-13 16 24 46

I can't find any other way to reset this robotsMap if c.IgnoreRobotsTxt = false is set other than reinitializing via colly.NewCollector()

@ZigHuang
Copy link
Author

ZigHuang commented May 13, 2024

I can raise a PR to set the size or other limits of robotsMap to avoid continuous increase in memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant