Added request.cookies command to fix file downloads
#37
Conversation
|
This is also related to #20 and should make life easier for @lululombard. We are intercepting the response here (using Chrome DevTools Fetch API before the response-content is send to the browser, but after the cookies/response-headers have been received. |
|
I'll take a look at this on Monday, it's currently Saturday for me and I'm adding the finishing touches to a hackathon project 💀 |
JBou
force-pushed
the
feature-justcookies
branch
from
a62d486
to
57cca35
Compare
…ponse/content, fixes file downloads using external libraries
JBou
force-pushed
the
feature-justcookies
branch
from
57cca35
to
b0cd8bc
Compare
| // TODO: find out why these pages hang sometimes | ||
| while (Date.now() - ctx.startTimestamp < maxTimeout) { | ||
| await page.waitFor(1000) | ||
| try { | ||
| // catch exception timeout in waitForNavigation | ||
| await page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: 5000 }) | ||
| response = await page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: 5000 }) |
There was a problem hiding this comment.
Without this assignment the response status and the response headers when using request.get are not correct. They remain 503 (assigned here). The page content, instead, is returned correctly, because it doesn't depend on the response variable, but is being retrieved from the page directly
| @@ -11,7 +11,8 @@ | |||
| "es2015", "dom" | |||
| ], | |||
| "module": "commonjs", | |||
| "outDir": "dist" | |||
| "outDir": "dist", | |||
| "sourceMap": true | |||
There was a problem hiding this comment.
I've added this line to enable break points in the source files (*.ts) as described here, to make debugging work
| let interceptingResult: ChallengeResolutionT; | ||
| if (returnOnlyCookies) { //If we just want to get the cookies, intercept the response before we get the content/body (just cookies and headers) | ||
| await interceptResponse(page, async function(payload){ | ||
| interceptingResult = payload; | ||
| }); | ||
| } | ||
|
|
||
| // submit captcha response | ||
| challengeForm.evaluate((e: HTMLFormElement) => e.submit()) | ||
| response = await page.waitForNavigation({ waitUntil: 'domcontentloaded' }) | ||
|
|
||
| if (returnOnlyCookies && interceptingResult) { | ||
| await page.close(); | ||
| return interceptingResult; | ||
| } |
There was a problem hiding this comment.
I've added the same logic as above to the captcha response, it should work here too, but wasn't able to test it because I'm not using captcha-harvester. Needs to be confirmed to work.
|
Re: #31 (comment) Since I don't have much time but wan't want to be the bottleneck, I'm just going to merge this. I just went over it and it seems all good so let's hope for the best. I'll take a more in-depth look this weekend hopefully. |
|
Thanks a lot! The only thing left is to add documentation about this endpoint in the readme. |
Added
request.cookiescommand to only fetch the cookies without response/content.This fixes file downloads using external libraries (for example CloudProxySharp).
It's working fine as of now.
The only thing missing is handling captchas, haven't tested it yet and don't know how it behaves.
Would like this PR to get merged, maybe someone else could try out with captcha-harvester. I'm not using it.