-
Notifications
You must be signed in to change notification settings - Fork 233
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Public IPv6 IPs Pool with major cloud providers like AWS, GCP, Azure, etc. #218
Comments
Adding references: @fabienvauchelles, I would greatly appreciate your insights on this matter. |
Hi,
Can you details the usecases ? Because most users need only IPv4. |
Most IPv6 proxy providers won't give you a direct IPv6 endpoint, what they will do is use some sort of 6to4 method, you will connect to the proxy via IPv4 and get an outgoing IPv6 address from the provider. So I don't see a need to add full IPv6 support yet, but I guess it can be done in the future as Cloud Services are moving away from free IPv4 addresses. |
To address the growing costs associated with public IPv4 IP addresses, as cloud providers have started charging for them, we are in the process of transitioning to IPv6 for our AWS cloud infrastructure. During local testing, we deactivated outbound IPv4 traffic on ports 80 and 443 for our Scrapoxy instances. However, we've encountered an issue where requests are halting during the TLS handshake phase. This problem arises despite configuring our proxy agent and master to utilize IPv6 addresses as hostnames. Below are the logs from a test run:
Curl command output showing attempt to connect through proxy to www.google.com, indicating a successful connection to the proxy but stalling at the TLS handshake phase. We would appreciate any insights or suggestions on troubleshooting this issue further. |
Hi,
|
Please find the details below -
Here are the modifications we've implemented:
Following these adjustments, we've successfully enabled the use of IPV6 addresses for Scrapoxy workers during requests. Although IPV6 was verified to be used within the proxyOptions, requests directed to google.com defaulted to using the public IPV4 address of Google, despite the presence of IPV6 addresses for both the requesting and target entities. Upon manually accessing one of our Scrapoxy workers, we observed that the traffic from the worker to the target persisted in routing through the IPV4 interface, even though the request from the master to the worker was confirmed to travel over the IPV6 interface. This observation was made after analyzing packet exchanges via TCP dump. @fabienvauchelles We greatly appreciate your assistance with this issue, as we're struggling to find a solution. Addressing this is a top priority for us, especially since the cost of AWS IPV4 IPs is significantly impacting our budget. |
Hi @RajatThukral-Draup , As I understand that you made a lot of custom code on 3.1.1, I need to understand for a proper integration on the V4. Can you share the custom code with me? (can be a private repository). Thanks. |
Certainly! I've shared the link to our custom Scrapoxy code repository below. An invitation link has also been sent to you. Here's the link: https://github.com/Draup/scrapoxy. Incorporating this feature into Scrapoxy version 3.1.1 would be greatly beneficial for us, as upgrading to Scrapoxy version 4 is expected to require a considerable amount of time for us. |
Just wanted to check in and see if you got a chance to peek at the Scrapoxy code repo link I sent over. Here it is again just in case: https://github.com/Draup/scrapoxy. We're really keen on getting that feature rolled into Scrapoxy v3.1.1 since jumping to v4 is a bit of a stretch for us right now. Let me know your thoughts, or if there's anything you're wondering about it. Hope to catch up soon! Thanks |
Hi @RajatThukral-Draup , I fetch the repository, thanks. I need to explore it now. Let me sometime to explore it and understand how it can smartly integrated. I will have some further questions for you to correctly write the requirements. |
Awesome, glad to hear you've got the repo! Take all the time you need to dive into it. I'm here to help answer any questions or clarify anything that might help you in understanding how we can best integrate this feature. Just hit me up whenever you're ready or need some info. Looking forward to your insights and the questions you'll have! Thanks |
Hi @RajatThukral-Draup , I've reviewed the code, and it is an excellent work! I truly appreciate the enhancements you've made to version 3, particularly regarding spot instances, Prometheus integration, and the introduction of new metrics. I have some initial inquiries:
|
How do you go about building the image and ensuring AWS employs IPv6? (I couldn't find any references to IPv6 during instance creation)
Have you made any updates to the proxy.js file (located at tools/install/proxy.js)? If so, would you mind sharing the code with me?
Can you confirm whether you utilize a subnet to prevent IPs from being publicly accessible on the internet?
How many instances/proxies do you use on AWS? By region?
What's the purpose behind the "multi-region" settings?
Additionally, I'm interested in integrating the newly added metrics. Could you highlight which ones you consider most important and provide insight into how you utilize them?
Thanks |
I would appreciate hearing from you on this matter. We've upgraded our code to support IPv6, yet our requests continue to default to IPv4. Is there a preference for IPv4 over IPv6 in the Node.js library? Any advice or insights you could provide on this issue would be very helpful. Additionally, when executing a curl request to google.com from the same system, it appears to utilize an IPv6 address. I'm uncertain about the precise cause of this behavior and how to resolve it. Our Node version: v18.18.0 Few references on this: Thanks |
Hi, On Add the To get the list of network interface, you can use this function Can you keep me updated if this upgrade works ? |
Any luck with your transition to v6? Do you have your version available anywhere? Can't access the https://github.com/Draup/scrapoxy anymore. Best regards |
Hello,
Thank you for the fantastic work on this project.
Given that major cloud providers are now charging substantially for public IPv4 addresses, it would be highly beneficial to incorporate IPv6 Pool support into this project. This would entail modifications to the provider API SDK code and the sections where requests are actually initiated.
The text was updated successfully, but these errors were encountered: