Axios can't access static.etherpad.org but Docker Container can

[Viajaz]

Describe the bug
Node app (Axios) can't access online network resources (static.etherpad.org) despite other tools (eg: curl) being able to inside same container.

To Reproduce
docker-compose.yml

services:
  etherpad:
    container_name: etherpad
    image: etherpad/etherpad
    ports:
      - 127.0.0.1:9001:9001
    environment:
      - DB_TYPE=postgres
      - DB_HOST=etherpad-db
      - DB_PORT=5432
      - DB_NAME=etherpad
      - DB_USER=etherpad
      - DB_PASS=REDACTED
      - TRUST_PROXY=true
      - ADMIN_PASSWORD=REDACTED
    networks:
      - etherpad
    depends_on:
      - etherpad-db
  etherpad-db:
    image: postgres:latest
    container_name: etherpad-db
    restart: unless-stopped
    volumes:
      - ./etherpad-db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=etherpad
      - POSTGRES_USER=etherpad
      - POSTGRES_PASSWORD=REDACTED
    networks:
      - etherpad
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 1s
      timeout: 5s
      retries: 10

networks:
  etherpad:

Expected behavior
Functioning network access.

Screenshots

> [email protected] prod /opt/etherpad-lite
> pnpm --filter ep_etherpad-lite run prod


> [email protected] prod /opt/etherpad-lite/src
> node --require tsx/cjs node/server.ts

[2024-06-27T04:00:22.183] [INFO] settings - All relative paths will be interpreted relative to the identified Etherpad base dir: /opt/etherpad-lite
[2024-06-27T04:00:22.209] [INFO] settings - settings loaded from: /opt/etherpad-lite/settings.json
[2024-06-27T04:00:22.211] [INFO] settings - No credentials file found in /opt/etherpad-lite/credentials.json. Ignoring.
[2024-06-27T04:00:22.212] [INFO] settings - Using skin "colibris" in dir: /opt/etherpad-lite/src/static/skins/colibris
[2024-06-27T04:00:22.212] [INFO] settings - Random string used for versioning assets: c76cda27
[2024-06-27T04:00:23.200] [INFO] server - Starting Etherpad...
[2024-06-27T04:00:23.296] [INFO] plugins - pnpm --version: 9.0.4
[2024-06-27T04:00:23.337] [INFO] plugins - check installed plugins for migration
[2024-06-27T04:00:23.338] [INFO] plugins - start migration of plugins in node_modules
[2024-06-27T04:00:23.573] [ERROR] settings - Can not perform Etherpad update check: AggregateError
[2024-06-27T04:00:24.103] [INFO] plugins - Loading plugin ep_etherpad-lite...
[2024-06-27T04:00:24.106] [INFO] plugins - Loaded 1 plugins
[2024-06-27T04:00:25.908] [WARN] settings - oidc-provider WARNING: Unsupported runtime. Use Node.js v18.x LTS, or a later LTS release.
[2024-06-27T04:00:26.728] [INFO] server - Installed plugins:
[2024-06-27T04:00:26.729] [INFO] settings - Report bugs at https://github.com/ether/etherpad-lite/issues
[2024-06-27T04:00:26.730] [INFO] settings - Your Etherpad version is 2.1.0 (c45b7a3)
[2024-06-27T04:00:26.734] [ERROR] settings - (node:29) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[2024-06-27T04:00:28.008] [INFO] http - HTTP server listening for connections
[2024-06-27T04:00:28.009] [INFO] settings - You can access your Etherpad instance at http://0.0.0.0:9001/
[2024-06-27T04:00:28.009] [INFO] settings - The plugin admin page is at http://0.0.0.0:9001/admin/plugins
[2024-06-27T04:00:28.009] [INFO] server - Etherpad is running
[2024-06-27T04:00:32.116] [ERROR] settings - AxiosError [AggregateError]
    at Function.AxiosError.from (/opt/etherpad-lite/node_modules/.pnpm/[email protected]/node_modules/axios/lib/core/AxiosError.js:89:14)
    at RedirectableRequest.handleRequestError (/opt/etherpad-lite/node_modules/.pnpm/[email protected]/node_modules/axios/lib/adapters/http.js:610:25)
    at RedirectableRequest.emit (node:events:520:28)
    at eventHandlers.<computed> (/opt/etherpad-lite/node_modules/.pnpm/[email protected]/node_modules/follow-redirects/index.js:38:24)
    at ClientRequest.emit (node:events:520:28)
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:520:28)
    at emitErrorNT (node:internal/streams/destroy:170:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at Axios.request (/opt/etherpad-lite/node_modules/.pnpm/[email protected]/node_modules/axios/lib/core/Axios.js:45:41)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at <anonymous> (/opt/etherpad-lite/src/static/js/pluginfw/installer.ts:174:5) {
  code: 'ETIMEDOUT',
  errors: [
    Error: connect ETIMEDOUT 5.75.249.160:443
        at createConnectionError (node:net:1647:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1706:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '5.75.249.160',
      port: 443
    },
    Error: connect ENETUNREACH 2a01:4f8:c17:ba99::1:443 - Local (:::0)
        at internalConnectMultiple (node:net:1181:16)
        at Timeout.internalConnectMultipleTimeout (node:net:1711:5)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2a01:4f8:c17:ba99::1',
      port: 443
    }
  ],
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http', 'fetch' ],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: [class Blob] },
    validateStatus: [Function: validateStatus],
    headers: Object [AxiosHeaders] {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': undefined,
      'User-Agent': 'Etherpad/2.1.0',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    method: 'get',
    url: 'https://static.etherpad.org/plugins.json',
    data: undefined
  },
  request: <ref *1> Writable {
    _events: {
      close: undefined,
      error: [Function: handleRequestError],
      prefinish: undefined,
      finish: undefined,
      drain: undefined,
      response: [Function: handleResponse],
      socket: [Function: handleRequestSocket]
    },
    _writableState: WritableState {
      highWaterMark: 65536,
      length: 0,
      corked: 0,
      onwrite: [Function: bound onwrite],
      writelen: 0,
      bufferedIndex: 0,
      pendingcb: 0,
      [Symbol(kState)]: 17580812,
      [Symbol(kBufferedValue)]: null
    },
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: Infinity,
      protocol: 'https:',
      path: '/plugins.json',
      method: 'GET',
      headers: [Object: null prototype],
      agents: [Object],
      auth: undefined,
      family: undefined,
      beforeRedirect: [Function: dispatchBeforeRedirect],
      beforeRedirects: [Object],
      hostname: 'static.etherpad.org',
      port: '',
      agent: undefined,
      nativeProtocols: [Object],
      pathname: '/plugins.json'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _eventsCount: 3,
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      _header: 'GET /plugins.json HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: Etherpad/2.1.0\r\n' +
        'Accept-Encoding: gzip, compress, deflate, br\r\n' +
        'Host: static.etherpad.org\r\n' +
        'Connection: keep-alive\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/plugins.json',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'static.etherpad.org',
      protocol: 'https:',
      _redirectable: [Circular *1],
      [Symbol(shapeMode)]: false,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kChunkedBuffer)]: [],
      [Symbol(kChunkedLength)]: 0,
      [Symbol(kSocket)]: [TLSSocket],
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 65536,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    _currentUrl: 'https://static.etherpad.org/plugins.json',
    [Symbol(shapeMode)]: true,
    [Symbol(kCapture)]: false
  },
  cause: AggregateError [ETIMEDOUT]:
      at internalConnectMultiple (node:net:1117:18)
      at internalConnectMultiple (node:net:1185:5)
      at Timeout.internalConnectMultipleTimeout (node:net:1711:5)
      at listOnTimeout (node:internal/timers:575:11)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'ETIMEDOUT',
    [errors]: [ [Error], [Error] ]
  }
}

However, I can go into the docker container and run curl just fine.

$ docker exec -it etherpad curl https://static.etherpad.org/plugins.json
{"ep_mobile_eruda":{"name": ....

[heldersepu]

This is 100% on your side...
It could be a firewall or some other network setting on your side

[Viajaz]

I would have thought so but I can't think of what networking or firewall configuration would impact the node app but not impact curl when they're both running inside the same container. Can you?

[heldersepu]

I'm not a network expert ... all I can suggest is try a different machine on a different network.
If you truly think this is an Axios problem I suggest to reduce the problem to a minimal example and report it to them, maybe they can help: https://github.com/axios/axios/issues

[SamTV12345]

I would have thought so but I can't think of what networking or firewall configuration would impact the node app but not impact curl when they're both running inside the same container. Can you?

Did you find the solution?

[Viajaz]

Did you find the solution?

No, Docker is doing it's own management of IPTables, which hasn't introduced any rules that I could see would interfere, and nothing between the Docker host and the internet is interfering (Eg: IPS). Whilst it hasn't impacted core functionality, it's a bit frustrating. We're considering moving to cryptpad instead because it has native support for SSO (with a plugin at least) which fits our use-case better anyway. I guess if no one else has this issue, one could close it.

[eternalphane]

It seems that axios is always connecting over both ipv4 and ipv6, and the network on your side does not support ipv6. You can try the following patch:

diff --git a/src/node/server.ts b/src/node/server.ts
index f96db3ab..edb687d7 100755
--- a/src/node/server.ts
+++ b/src/node/server.ts
@@ -29,6 +29,13 @@ import pkg from '../package.json';
 import {checkForMigration} from "../static/js/pluginfw/installer";
 import axios from "axios";

+axios.head('https://static.etherpad.org/plugins.json').catch(() => {
+  const http = require('http');
+  const https = require('https');
+  axios.defaults.httpAgent = new http.Agent({ family: 4 });
+  axios.defaults.httpsAgent = new https.Agent({ family: 4 });
+});
+
 const settings = require('./utils/Settings');

 let wtfnode: any;

[Viajaz]

@eternalphane You were able to reproduce this issue? I wonder why others have not encountered this problem. Should this perhaps be spun out into a toggle/environmental variable?

[SamTV12345]

I'm wondering why this should be the case. I'll try that patch. Maybe you can too @Viajaz if you are still considering Etherpad.

[eternalphane]

@Viajaz @SamTV12345 I just happened to come across this issue when I deployed etherpad in a dev cluster which I forgot to enable IPv6 support

[Viajaz]

I have tried the fix from @eternalphane and it has worked for me.
I take that back, for thoroughness I retested the original problem without this change, it appears the functionality now works regardless of the patch? Container is running etherpad/etherpad:latest (sha256:179f1645617665cdcd9d8f465316873c2e479f53867e56fb68b9a5bfba166459). We haven't changed any configuration, other than normal software updates, since issue was submitted.

[eternalphane]

Maybe your environment now supports IPv6? You can test it by running curl -iIL6 https://static.etherpad.org/plugins.json in the container.

[Viajaz]

Our environment hasn't and doesn't support IPv6 and it's routing.

[eternalphane]

Our environment hasn't and doesn't support IPv6 and it's routing.

Then it may be that your DNS is not returning the IPv6 address of static.etherpad.org? Anyway I'm still getting this error even with the latest release (v2.2.2).

[SamTV12345]

Fixed you can now add your own server like https://etherpad.org/ep_infos