Skip to content

feat: Replace agentkeepalive with native Node.js HTTP agents for proxy support #788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 20, 2025
Merged

feat: Replace agentkeepalive with native Node.js HTTP agents for proxy support #788

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c424e90
feat: Replace agentkeepalive with native Node.js HTTP agents for prox…
tducret Nov 19, 2025
4fd4403
feat: Enhance HTTP agent with agentkeepalive-inspired optimizations
tducret Nov 20, 2025
f986aa4
chore: Exclude node:http and node:https from browser bundle
tducret Nov 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 1 addition & 22 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@
"@apify/log": "^2.2.6",
"@apify/utilities": "^2.23.2",
"@crawlee/types": "^3.3.0",
"agentkeepalive": "^4.2.1",
"ansi-colors": "^4.1.1",
"async-retry": "^1.3.3",
"axios": "^1.6.7",
Expand Down
15 changes: 9 additions & 6 deletions src/http_client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import http from 'node:http';
import https from 'node:https';
Comment on lines +1 to +2
Copy link
Member

@barjin barjin Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add both these modules as externals to rsbuild.config.ts (link)? They are not used outside of Node, so we don't need them in the browser bundle. Together, they add around 250kB to the bundle size.

Copy link
Contributor Author

@tducret tducret Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing, done in commit f986aa4

Copy link
Member

@barjin barjin Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

import os from 'node:os';

import KeepAliveAgent from 'agentkeepalive';
import type { RetryFunction } from 'async-retry';
import retry from 'async-retry';
import type { AxiosError, AxiosInstance, AxiosRequestConfig, AxiosResponse, InternalAxiosRequestConfig } from 'axios';
Expand Down Expand Up @@ -32,9 +33,9 @@ export class HttpClient {

timeoutMillis: number;

httpAgent?: KeepAliveAgent;
httpAgent?: http.Agent;

httpsAgent?: KeepAliveAgent.HttpsAgent;
httpsAgent?: https.Agent;

axios: AxiosInstance;

Expand All @@ -58,11 +59,13 @@ export class HttpClient {
// is for inactive sockets, sometimes the platform would take
// long to process requests and the socket would time-out
// while waiting for the response.
const agentOpts = {
const agentOptions: http.AgentOptions = {
keepAlive: true,
Copy link
Contributor

@janbuchar janbuchar Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the only thing that agentkeepalive adds.

Image

...is it safe to ignore the rest of the changes they mention?

Copy link
Contributor Author

@tducret tducret Nov 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I've just added commit c424e90 with these optimizations.

timeout: this.timeoutMillis,
};
this.httpAgent = new KeepAliveAgent(agentOpts);
this.httpsAgent = new KeepAliveAgent.HttpsAgent(agentOpts);

this.httpAgent = new http.Agent(agentOptions);
this.httpsAgent = new https.Agent(agentOptions);
}

this.axios = axios.create({
Expand Down
Loading