Skip to content

Conversation

@chrisfu
Copy link

@chrisfu chrisfu commented Jun 19, 2014

No description provided.

chrisfu added 3 commits June 19, 2014 11:44
This crawler is supposed to be respectful. Unfortunately it ties Magento's core_url_rewrite table in knots by hammering non-existent links with non-existant parameters. More to the point, this crawler exists only so that Majestic SEO can sell the data it generates. It'll never be of any use to anyone that doesn't hire them, so blocking it is fine.
I waited 3 weeks for AhrefsBot to honor a robots.txt. Gave up and blocked it.

I removed the carat from MJ12Bot. I mean, who *else* would use that in their user agent string?
@einfallstoll
Copy link

I'd really like to see this merged.

@chrisfu
Copy link
Author

chrisfu commented Jan 12, 2016

The project appears to be dead, which is a shame. Need folk to decide which fork to choose to resume development and ask folk to resubmit pull requests to that fork.

@stevetoza
Copy link

I dont mind creating a fork if people want to push there and we can resume ?

@einfallstoll
Copy link

I suggest to use the fork of @chrisfu and maybe merge the changes by @kijin

@chrisfu
Copy link
Author

chrisfu commented Jan 18, 2016

That's fine with me @einfallstoll. If you sent me a pull request @kijin I'd be happy to merge your changes.

@Firesphere
Copy link
Contributor

Also an option, is add collaborators, so more people can merge pull-requests?

@einfallstoll
Copy link

@Firesphere Yes, that'd be possible as well, but it seems like @bluedragonz abandoned his account.

@chrisfu No need for a PR. Just cherry pick / rebase / merge them.

@chrisfu
Copy link
Author

chrisfu commented Jan 27, 2016

@einfallstoll Sounds good to me.

@Firesphere I can add a couple of collaborators to my fork, no problem. Any takers?

@Firesphere
Copy link
Contributor

I'm okay with keeping an eye on pull requests and manage/merge them every once in a while.

@einfallstoll
Copy link

I could do this as well.

I'm okay with keeping an eye on pull requests and manage/merge them every once in a while.


Reply to this email directly or view it on GitHub.

@chrisfu
Copy link
Author

chrisfu commented Jan 31, 2016

I've added you both as collaborators now. :)

@Firesphere
Copy link
Contributor

Yep, got the message.
It seems you already included the pull-requests here. I'll keep an eye out and check if the bots are indeed malicious, and possibly update from here to your repo.
Thanks for the effort!
Right now, I'm quite short on time, since I'm migrating from NL to NZ (only one letter difference, and 40k km apart :P ) but when I arrive and settled, I'll be more active again :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants