fix(seeder): TLS proxy CONNECT — fixes FRED fetch failures (FSI, yield curve, macro)#2538
fix(seeder): TLS proxy CONNECT — fixes FRED fetch failures (FSI, yield curve, macro)#2538
Conversation
Decodo gate.decodo.com:10001 requires TLS. Previous code used http.request (plain TCP) which received SOCKS5 rejection bytes instead of HTTP 200. Two issues fixed: 1. Replace http.request CONNECT with tls.connect + manual CONNECT handshake. Node.js http.request also auto-sets Host to the proxy hostname; Decodo rejects this and responds with SOCKS5 bytes (0x05 0xff). Manual CONNECT over a raw TLS socket avoids both issues. 2. Handle https:// and plain "user:pass@host:port" proxy URL formats — always uses TLS regardless of PROXY_URL prefix. _proxy-utils.cjs: resolveProxyStringConnect now preserves https:// prefix from PROXY_URL so callers can detect TLS proxies explicitly. All 24 FRED series (BAMLH0A0HYM2, FEDFUNDS, DGS10, etc.) confirmed working locally via gate.decodo.com:10001.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR fixes a broken FRED proxy tunnel by replacing an Key changes:
Minor cleanup items found:
Confidence Score: 5/5Safe to merge — the fix is targeted and well-reasoned; all remaining findings are P2 style/cleanup items. The core change is correct: TLS-to-proxy + manual CONNECT + TLS-in-TLS is exactly the right approach for a TLS-only CONNECT proxy, and local validation confirms it restores FRED fetches. No P0 or P1 issues were found. The four P2 items (unused import, stale comment, single-chunk CONNECT read, redundant resume) are minor and do not affect correctness in the current Decodo deployment context. No files require special attention; scripts/_seed-utils.mjs has the unused http import worth cleaning up. Important Files Changed
Sequence DiagramsequenceDiagram
participant S as _seed-utils.mjs
participant P as Decodo Proxy (gate:10001)
participant T as api.stlouisfed.org:443
S->>P: tls.connect (Step 1 — TLS to proxy)
Note over S,P: TLS handshake over TCP
S->>P: CONNECT api.stlouisfed.org:443 HTTP/1.1
P-->>S: HTTP/1.1 200 Connection established
Note over S: once('data') checks status, then pause()
S->>T: tls.connect({socket: proxySock}) — TLS-in-TLS
Note over S,T: TLS handshake through tunnel
T-->>S: secureConnect
S->>T: https.request GET /fred/series/... (createConnection: tlsSock)
T-->>S: JSON response (gzip-decoded)
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
scripts/_seed-utils.mjs
Outdated
| // Step 3: Read CONNECT response (first data chunk contains the status line). | ||
| await new Promise((resolve, reject) => { | ||
| proxySock.once('data', (chunk) => { | ||
| const resp = chunk.toString('ascii'); | ||
| if (!resp.startsWith('HTTP/1.1 200') && !resp.startsWith('HTTP/1.0 200')) { | ||
| proxySock.destroy(); | ||
| return reject(Object.assign(new Error(`Proxy CONNECT: ${resp.split('\r\n')[0]}`), { status: parseInt(resp.split(' ')[1]) || 0 })); | ||
| } | ||
| resolve({ socket }); | ||
| }).on('error', reject).end(); | ||
| proxySock.pause(); | ||
| resolve(); | ||
| }); | ||
| proxySock.on('error', reject); | ||
| }); |
There was a problem hiding this comment.
data chunk
proxySock.once('data', …) captures only the first TCP segment. In practice this is fine — Decodo sends a short HTTP/1.1 200 Connection established\r\n\r\n that fits in one segment, and the server cannot send TLS data until we emit the ClientHello (which happens after step 3). However, if the proxy ever sends response headers across more than one chunk (e.g. a Proxy-Agent header arrives in a second segment) the unread tail will be handed to the TLS layer as garbage, silently breaking the handshake.
A minimal guard is to buffer until \r\n\r\n is seen before resolving:
await new Promise((resolve, reject) => {
let buf = '';
const onData = (chunk) => {
buf += chunk.toString('ascii');
if (!buf.includes('\r\n\r\n')) return;
proxySock.off('data', onData);
const statusLine = buf.split('\r\n')[0];
if (!statusLine.startsWith('HTTP/1.1 200') && !statusLine.startsWith('HTTP/1.0 200')) {
proxySock.destroy();
return reject(Object.assign(new Error(`Proxy CONNECT: ${statusLine}`), { status: parseInt(statusLine.split(' ')[1]) || 0 }));
}
proxySock.pause();
resolve();
};
proxySock.on('data', onData);
proxySock.on('error', reject);
});| tlsSock.on('secureConnect', resolve); | ||
| tlsSock.on('error', reject); | ||
| }); | ||
| proxySock.resume(); |
There was a problem hiding this comment.
proxySock.resume() after TLS upgrade
Node.js's TLSSocket constructor internally calls socket.resume() on the wrapped parent socket so that TLS data can flow during the handshake. By the time secureConnect fires the socket is already resumed; this explicit call is a no-op. It can be removed to avoid any confusion about why the underlying socket is being manipulated after the TLS layer has taken over.
| * When PROXY_URL uses https:// (TLS proxy), returns "https://user:pass@host:port" so | ||
| * httpsProxyFetchJson can detect and use https.request instead of http.request. | ||
| * Returns empty string if no proxy configured. |
There was a problem hiding this comment.
https.request instead of tls.connect
The comment says httpsProxyFetchJson will "detect and use https.request instead of http.request", but the new implementation unconditionally uses tls.connect for the proxy leg regardless of the https:// prefix. The prefix is now purely a normalisation marker that is immediately stripped inside httpsProxyFetchJson, not a dispatch signal. The JSDoc should be updated to reflect that the actual connection always goes through tls.connect.
Two protocol-correctness fixes:
1. http:// proxies used plain TCP before; always-TLS regressed them.
Now: bare/undeclared format → TLS (Decodo requires it), explicit
http:// → plain net.connect, explicit https:// → TLS.
2. CONNECT response buffered until \r\n\r\n instead of acting on the
first data chunk. Fragmented proxy responses (headers split across
packets) could corrupt the TLS handshake by leaving header bytes
on the wire when tls.connect() was called too early.
Verified locally: BAMLH0A0HYM2 → { date: 2026-03-26, value: 3.21 }
- Drop `import * as http from 'node:http'` — no longer used after replacing http.request CONNECT with tls.connect + manual handshake - Update resolveProxyStringConnect() JSDoc: https.request → tls.connect
1 participant
Problem
All 24 FRED series have been failing since at least 01:53 UTC (2026-03-30), causing FSI data unavailable, yield curve blanks, and any FRED-dependent panel going stale.
Two root causes found by local validation:
1. Wrong connection type to proxy
httpsProxyFetchJsonusedhttp.request(plain TCP) to connect togate.decodo.com:10001. The proxy requires TLS. Plain TCP received SOCKS5 rejection bytes (\x05\xff) from the proxy, causing Node.js to throwParse Error: Expected HTTP/, RTSP/ or ICE/.2. Node.js
http.requestsets wrongHostheader for CONNECThttp.requestauto-setsHost: gate.decodo.com:10001(the proxy host). For HTTP CONNECT theHostmust be the target (api.stlouisfed.org:443). Decodo rejects the wrongHostwith SOCKS5 bytes.Fix
Replace
http.requestCONNECT withtls.connect+ manual CONNECT handshake:tls.connectto proxy (always TLS — no dependence on PROXY_URL format)CONNECT target:443 HTTP/1.1\r\nHost: target:443manually over the TLS socketHTTP/1.1 200 OKresponsetls.connectover the tunnel for the target (TLS-in-TLS)Handles both
user:pass@host:portandhttps://user:pass@host:portPROXY_URL formats.Validation
Tested locally —
BAMLH0A0HYM2(HY spread, the FSI input that was null):Both plain and
https://proxy URL formats confirmed working.