CloudProxySharp.Exceptions.CloudProxyException: The cookies provided by CloudProxy are not valid

[ChristianKernDev]

I try to access: https://www.spigotmc.org/resources/hubkick.2/download?version=203285

Console output of CloudProxy:

2020-11-03T22:17:45.832Z INFO REQ-5 Incoming request: POST /v1
2020-11-03T22:17:45.832Z INFO REQ-5 Params: {"cmd":"request.get","url":"https://www.spigotmc.org/?version=203285","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0","maxTimeout":60000,"headers":null,"cookies":null}
2020-11-03T22:17:45.833Z DEBUG REQ-5 Launching headless browser...
2020-11-03T22:17:46.225Z DEBUG REQ-5 Using custom UA: Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0
2020-11-03T22:17:46.226Z DEBUG REQ-5 Adding custom headers: {}
2020-11-03T22:17:46.226Z DEBUG REQ-5 { headers: [Function] }
2020-11-03T22:17:46.228Z DEBUG REQ-5 Navegating to... https://www.spigotmc.org/?version=203285
2020-11-03T22:17:46.242Z DEBUG REQ-5 {
headers: {
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'
}
}
2020-11-03T22:17:47.218Z INFO REQ-5 Cloudflare detected
2020-11-03T22:17:47.258Z INFO REQ-5 Successful response in 1.426 s

The code i use:

public async Task<Stream> GetPluginAsync(string url)
{
    ClearanceHandler handler = new ClearanceHandler("http://localhost:8191/")
    {
        UserAgent = "Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0",
        MaxTimeout = 60000
    };
    HttpClient client = new HttpClient(handler);
    Uri uri = new Uri(url);
    return await client.GetStreamAsync(uri);
}

The only thing i noticed is that the url i pass to the function "https://www.spigotmc.org/resources/hubkick.2/download?version=203285" is not the one that is shown in the logs of CloudProxy.

Can you tell me if this is an error with your c# lib or with CloudProxy?

[JBou]

There is a know workaround for spigotmc! I will provide more info tonight and can implement you a workaround in the c# lib until there is a proper solution in CloudSharp for file downloads. I'm already looking into a fix for CloudSharp to support all websites including downloads.

The url you pass to the function and the url on the logs for cloudproxy are different, because I already use a workaround for downloads: Cloudproxy can't bypass the protection on downloads (as written above), so I solve it on the homepage and remove the path from the url, because the homepage doesn't invoke a download. But on spigotmc, the homepage is not cloudflare challenge protected, so it fails to get the correct cookies there, only the downloads are protected. Once we have the cookies, they are valid for the whole page.

[ChristianKernDev]

Ah i see. Thanks for the detailed info.

If you could implement the workaround it would help a ton. Thank you

[JBou]

I started with it but need some more time. Thought it was easier, but there are some redirects that make it harder. There is still a problem with it, I just need to find a good implementation or a fix in cloudproxy itself.

[ChristianKernDev]

Ok don't worry i'll just use it once it works 😃

I would help you but i don't have enough experience with this kind of stuff so I'll let you figure it out. If i can help with anything let me know

[ChristianKernDev]

Any update?

[JBou]

Yeah, i got it working!

Started on the workaround a week ago, but haven't pushed it yet, because there were some things missing.
Instead of working on on the workaround, today I decided to fix this on CloudProxy itself, just created a PR.

After it gets merged, I will update my library to work with it.

[ChristianKernDev]

Nice!

Thank you very much for your effort.

[JBou]

I published a pre-release so you can test it out (if you want):
https://www.nuget.org/packages/CloudProxySharp/1.0.0-justcookies

You have to use it with the following docker container: jbouhd/cloudproxy:feature-justcookies

Example:
docker pull jbouhd/cloudproxy:feature-justcookies

Hope the Pull request gets merged soon so it will be officially supported by CloudProxy and usable for end users.
If you have any questions in the meantime, feel free to contact me.

[ChristianKernDev]

Thank you very much for your effort!

[JBou]

Just published the new version of the library to nuget: https://www.nuget.org/packages/CloudProxySharp/1.0.1

Should be working now, you have to use noahcardoza/cloudproxy version v1.1.0 or above to use it (It has been released a day ago).

Could you test it out and reply so we can close the issue if it's working?

[ChristianKernDev]

Ok i just tried it and it still does not work, but maybe I'm doing something wrong.
I upgraded the Nuget package to version 1.0.1 and installed the latest CloudProxy version.

Here is a patebin of the console: https://pastebin.com/12LB6WEY

[JBou]

Hmm will have a look tonight. I added a test for spigot wich worked fine, it's even the same url:

CloudProxySharp/test/CloudProxySharp.Tests/ClearanceHandlerTests.cs

Lines 64 to 75 in a112345

	public async Task SolveOkCloudflareDownload2()
	{
	var handler = new ClearanceHandler(Settings.CloudProxyApiUrl)
	{
	UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36",
	MaxTimeout = 60000,
	};
	var client = new HttpClient(handler);
	var response = await client.GetAsync(_protectedDownloadUri2);
	Assert.AreEqual(MediaTypeHeaderValue.Parse("application/octet-stream"), response.Content.Headers.ContentType);
	}

One thing that could make a difference is using another userAgent... Try the one i am using in the test... When I first worked on this issue I was struggling for 3 days to find out that the useragent I have used wasn't working correctly, it never returned valid cookies.

[ChristianKernDev]

Yes that was it. Ok i think this issue can be closed, as my initial problem was solved.
Thanks a lot again

[ChristianKernDev]

One more thing: you should change the User Agent in the readme.md

[JBou]

Thanks for the information, just fixed it 👍