This repository will move to @wdes #224
Replies: 7 comments 10 replies
-
|
Dear users, I ping you on this thread to have some feedback on some of the questions above.
Org followers, should follow @wdes instead:
Other links: |
Beta Was this translation helpful? Give feedback.
-
|
i don't mind any migration, in fact I'm glad i was made aware. |
Beta Was this translation helpful? Give feedback.
-
|
So, as promised, here my 2¢: First of all, congrats for using crowdsec. I haven't gotten around to getting started with it, at the time I started there was only fail2ban, which is set up and fine tuned pretty well in the meantime. I'll look into using crowdsec more when time allows. Looks like a nifty project, only I'm not sure it's worth using it when the good bits are behind a paywall. Fail2ban is FOSS, I already made a couple contributions to it. YMMV though. Using the repo itself for storing the generated files is not a good idea ever — see https://github.com/StevenBlack/hosts/ (which is also worth looking into from your perspective) that has also grown over gigabytes of repo size with time — but better if you don't have anything else as a start. You don't need a CDN though, just put the generated artifacts somewhere on one of your servers, when you succeed to rearrange the repo. You know what they say, the cloud is just someone else's computer :) Moreover, I wouldn't keep the project on github. They aren't exactly pro-free-speech, and while this may not be a concern of yours for the time being, the time will come when it will. If you don't want to set up a gitea/gitlab/whateveryouprefer like I did, just use codeberg; I wouldn't recommend using gitlab at this point either. In the end it's your call though. Since as you mentioned, you have your own servers, it shouldn't be a hassle to use your own pipeline for the builds, I use buildbot for many of my projects, it's also FOSS. Adding a free account with an API key (that will always stay free) for downloading the artifacts is not a bad idea, as long as you keep it free. Maxmind GeoIP and AbuseIPDB and the likes are already practicing this, probably for the same reason you intend to: to have a sane impediment for exhausting your bandwidth. |
Beta Was this translation helpful? Give feedback.
-
|
Some updates:
Now I have big questions:
|
Beta Was this translation helpful? Give feedback.
-
|
Hey, instead of using gitea, try forgejo. My experience showed that gitea lacks functionality: for example, getting emails about new registrations who are often just spammers. Forgejo can do that, gitea can't. For the time being, forgejo is a drop-in replacement, this will change on the long run since they did a hard fork. As a donation, I'm willing to contribute the $5/5€ on a yearly basis (we can talk about wiring that to you directly). Just put out a donation link so it's not begging and voluntary. People who will make use of it, will donate. AbuseIPDB has a subscriber model with a free tier, I don't know how that works out for them, for me (my size of a company) the paid tiers are just too expensive. Or if you're uncomfortable going with that, use the FUTO license. You might even apply for a grant with them, why not? About contributing a server: I can rent a VPS for doing scanning pretty cheaply (~30€ a year), but the question is, did you ever face backlash in the form of bans or service refusal for mass reverse IP scans? |
Beta Was this translation helpful? Give feedback.
-
|
Hello
Okay, well I will not move since I will not accept users to register. And some of my clients pay for Gitea hosting on this instance.
That's nice, consider picking what fits you the best.
I host my own servers, it is best that you do not add costs on your end.
Nope, I round robin DNS resolvers. |
Beta Was this translation helpful? Give feedback.
-
Do as you wish, but it will be more painful later on when more hindrances of gitea will come to light to you :) Forgejo to me seems to be a more active community in terms of actual development. Gitea in my opinion, moves way too slow. They can't even seem to be able to address their spammer problem, not even with temporary workaround, until a real solution is figured out. I'd wire you the money directly (to an IBAN), so that way you wouldn't to get the paypal fees deducted amount (which is horrendous imo). You can issue an invoice to my German company, in which case no VAT applies, and just to be fair, let's make it 40€ then, to cover more of your spent costs and time. Please email me at [email protected] for the details :) I have a bare metal FreeBSD box and a Linux VPS rented, but I'm kinda reluctant to start scanning from those since both act as a nameserver. If any providers will start refusing service because of the mass scans, I might run into trouble servicing my clients. Hence why I asked if you had any problems before. Knowing MS (with their new stretchoid ranges), they might refuse service to our IPs entirely over time. Even if you don't have problems now, they might arise, and so it's a good idea to have dedicated IPs doing this stuff. Also, another idea would be to do something similar that AbuseIPDB does. They have distributed action files in fail2ban (and probably other clients I don't use), that contribute reports to their API, and they do a weighed analysis on which IPs to list. Keep that in mind as a long-term improvement. |
Beta Was this translation helpful? Give feedback.
-
I guess there is enough public DNS resolvers on the planet so we can round robin them and get our DNS response. And MS can not block us since we ask the resolver to get us the reply. |
Beta Was this translation helpful? Give feedback.
-
|
Using open resolvers (if you mean 8.8.8.8, 8.8.4.4, 9.9.9.9 and the likes) can get you rate limited over time imo. Don't know, never really used them. I use a locally installed unbound as a recursive resolver, and bind as an authoritative, serving local zones. I once used my provider's nameserver (Hetzner provides a couple ones), but they got rate limited with using RBLs for spam filtering. I decided then to use my locally installed nameserver, which works ever since, but being recursive means, your IP will go to their nameserver for querying PTRs. |
Beta Was this translation helpful? Give feedback.
-
|
I am using 8. 1. and 9. since months to operate this project and had no issues. That said I noticed slowness on some of them. When it occurs I just round robin to another one. Cheap and effective solution. It cools down and another request can use the resolver. Resolving options can be configured at a code level on https://github.com/wdes/dns-ptr-resolver |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good. If you can dockerize this, I can set up a resolver in a container on my VPS, using open resolvers. |
Beta Was this translation helpful? Give feedback.
-
|
New URLs 🎉
Be aware that some params will be required in the future to authentify the request. |
Beta Was this translation helpful? Give feedback.
-
|
And:
|
Beta Was this translation helpful? Give feedback.
-
|
Sure, I would host a node. Support can be achieved by maybe offering a premium experience which I would say is your API which could be higher request limits, time between service requests, like IPinfo.io, and 10$ at least. Lower costs, remove abstractions, gain fidelity
|
Beta Was this translation helpful? Give feedback.
-
For now in my infrastructure Crowdsec work great and avoids me a bunch of attacks, so I will keep it.
I am aware of the current drama around their commercial practices, and moved out the registar all my domains.
Thank you for all the light on Gitea, for now I can not start another migration. It feels like an endless fight. |
Beta Was this translation helpful? Give feedback.
-
Thank you @wravoc and @karolyi for proposing to host a node, I will split out my code to have a simple binary that does the work and reports it back to the requester. And let you know how to join the project runner group. One small detail I need to address is: how do I trust the scan results ? |
Beta Was this translation helpful? Give feedback.
-
|
DNSSEC is not built for this. I think you'll have to build out a trust based relationship here. I don't have a better idea. |
Beta Was this translation helpful? Give feedback.
-
Q: What is @wdes ?
A: A French software/server&infrastructure company (https://wdes.fr/en/) that I run on my own. I have 3-4 servers and mange servers for my clients too.
That means that I need to protect services. I found some very efficient ways to protect from nasty bots:
Q: What was @datacenters-network and why was it not in @wdes first ?
A: Because I was thinking that @datacenters-network was a good "sub" brand name for my company. But I did not realize that this was a bit like X vs Twitter. De-structuring the company/brand name of "Wdes" with another one was a terrible idea. This repo is the last step in my migration to move all services back to "wdes.eu". This domain and TLD is used for all services hosted by the company.
Q: What will change
A: The data has to be updated in a very regular way, that means that there is a lot of commits to store.
The repository grows quite a lot. Using a separate branch is my current migration idea.
Q: Contributions or second reviewers to each data update
A: I can accept to have a second review of data updates if users think that would be more secure.
I do not want that users download raw files from GitHub, In my opinion this could lead to some rate limiting and users would be unprotected and scripts would fail. Until now this works great for me, but when this repo moves I think security.wdes.fr or security.wdes.eu (what is the best tld to use as an end user perception ?) will be used to provide files. CloudFlare will be the CDN to be used.
Users still can continue to download files from GitHub but as soon as the website will be up, only the website URLs will be guaranteed stable.
Questions for my current users:
?username=or?email=while fetching the block lists. I honestly think that consuming a block list without any way to be notified about changes or breaking changes is quite risky.Beta Was this translation helpful? Give feedback.
All reactions