Incompatible dynamic import in npm:* commonjs context

[mizchi]

Version: Deno 1.44.4

Description

The result of dynamic import to type:commonjs is different between Node and Deno.

It may cause commonjs context's dynamic import.

The following issue is similar, but since this is a user-controlled code called next.config.js, it may be controversial, but since npm: protocol and below should be required to be compatible with node, this should be a Deno incompatibility.

#21149

Reproduce

This is langchain.js sample code.

import { CheerioWebBaseLoader } from "npm:@langchain/community/document_loaders/web/cheerio";

const loader = new CheerioWebBaseLoader(
  "https://docs.smith.langchain.com/user_guide"
);
const rawDocs = await loader.load();

Error Log

error: Uncaught (in promise) TypeError: load is not a function
    at Function._scrape (file:///Users/kotaro.chikuba/mizchi/misc/playground/node_modules/.deno/@[email protected]_1/node_modules/@langchain/community/dist/document_loaders/web/cheerio.js:82:16)
    at eventLoopTick (ext:core/01_core.js:168:7)
    at async CheerioWebBaseLoader.load (file:///Users/kotaro.chikuba/mizchi/misc/playground/node_modules/.deno/@[email protected]_1/node_modules/@langchain/community/dist/document_loaders/web/cheerio.js:100:19)
    at async file:///Users/kotaro.chikuba/mizchi/misc/playground/02.ts:9:17

Error Reason

https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/src/document_loaders/web/cheerio.ts#L158

    static async imports() {
        try {
            const { load } = await import("cheerio");
            return { load };
        }
        catch (e) {
            console.error(e);
            throw new Error("Please install cheerio as a dependency with, e.g. `yarn add cheerio`");
        }
    }

Deno's result.

$ deno eval 'import("npm:cheerio").then(console.log)'
[Module: null prototype] {
  default: [Function (anonymous)] {
    load: [Function (anonymous)],
    html: [Function (anonymous)],
    xml: [Function (anonymous)],
    text: [Function (anonymous)],
    parseHTML: [Function (anonymous)],
    root: [Function (anonymous)],
    contains: [Function (anonymous)],
    version: "0.22.0"
  },
  version: "0.22.0"
}

Node CommonJS's result

$ node                    
Welcome to Node.js v20.8.1.
Type ".help" for more information.
> const m = await import('cheerio')
undefined
> m
[Module: null prototype] {
  contains: [Function: contains],
  default: [Function: initialize] {
    contains: [Function: contains],
    html: [Function: html],
    merge: [Function: merge],
    parseHTML: [Function: parseHTML],
    root: [Function: root],
    text: [Function: text],
    xml: [Function: xml],
    load: [Function: load],
    _root: Document {
      parent: null,
      prev: null,
      next: null,
      startIndex: null,
      endIndex: null,
      children: [],
      type: 'root'
    },
    _options: { xml: false, decodeEntities: true },
    fn: Cheerio {}
  },
  html: [Function: html],
  load: [Function: load],
  merge: [Function: merge],
  parseHTML: [Function: parseHTML],
  root: [Function: root],
  text: [Function: text],
  xml: [Function: xml]
}

Node ESM's dynamic import

// foo.mjs
import("cheerio").then(console.log)
/*
[Module: null prototype] {
  default: [Function (anonymous)] {
    load: [Function (anonymous)],
    html: [Function (anonymous)],
    xml: [Function (anonymous)],
    text: [Function (anonymous)],
    parseHTML: [Function (anonymous)],
    root: [Function (anonymous)],
    contains: [Function (anonymous)],
    version: '0.22.0'
  },
  version: '0.22.0'
}
*/

cheerio's package.json

https://github.com/cheeriojs/cheerio/blob/main/package.json#L29-L40

  "main": "lib/index.js",
  "types": "lib/index.d.ts",
  "module": "lib/esm/index.js",
  "type": "commonjs",
  "exports": {
    ".": {
      "require": "./lib/index.js",
      "import": {
        "node": "./lib/esm/batteries.js",
        "default": "./lib/esm/index.js"
      }
    },

[kt3k]

I couldn't reproduce the issue with the latest Deno (1.44.4) and cheerio (1.0.0-rc.12)

$ deno eval --reload 'import("npm:cheerio").then(console.log)' 

[Module: null prototype] {
  contains: [Function: contains],
  default: [Function: initialize] {
    contains: [Function: contains],
    html: [Function: html],
    merge: [Function: merge],
    parseHTML: [Function: parseHTML],
    root: [Function: root],
    text: [Function: text],
    xml: [Function: xml],
    load: [Function: load],
    _root: Document {
      parent: null,
      prev: null,
      next: null,
      startIndex: null,
      endIndex: null,
      children: [],
      type: "root"
    },
    _options: { xml: false, decodeEntities: true },
    fn: Cheerio {}
  },
  html: [Function: html],
  load: [Function: load],
  merge: [Function: merge],
  parseHTML: [Function: parseHTML],
  root: [Function: root],
  text: [Function: text],
  xml: [Function: xml]
}

Also your langchain example worked on my end

langchain.js

import { CheerioWebBaseLoader } from "npm:@langchain/community/document_loaders/web/cheerio";

const loader = new CheerioWebBaseLoader(
  "https://docs.smith.langchain.com/user_guide"
);
const rawDocs = await loader.load();
console.log(rawDocs);

$ deno run -A langchain.js
[
  Document {
    pageContent: "\n" +
      '!function(){function e(e){document.documentElement.setAttribute("data-theme",e)}var t=function(){var e=null;try{e=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(e){}return e}()||function(){var e=null;try{e=localStorage.getItem("theme")}catch(e){}return e}();null!==t?e(t):window.matchMedia("(prefers-color-scheme: dark)").matches?e("dark"):(window.matchMedia("(prefers-color-scheme: light)").matches,e("light"))}()\n' +
...

What happens if you clear node_modules and reinstall it?

What package manager do you use for downloading npm packages?

[dsherret]

The exact same issue happens when using cheerio 0.22.0 with Node ESM:

> node main.mjs
file:///V:/scratch/node_modules/.deno/@[email protected]/node_modules/@langchain/community/dist/document_loaders/web/cheerio.js:79
        return load(html, cheerioOptions);
               ^

TypeError: load is not a function
    at CheerioWebBaseLoader._scrape (file:///V:/scratch/node_modules/.deno/@[email protected]/node_modules/@langchain/community/dist/document_loaders/web/cheerio.js:79:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CheerioWebBaseLoader.load (file:///V:/scratch/node_modules/.deno/@[email protected]/node_modules/@langchain/community/dist/document_loaders/web/cheerio.js:97:19)
    at async file:///V:/scratch/main.mjs:6:17

Node.js v22.3.0

To fix this issue, specify the 1.0.0-rc.12 version of cherio in a package.json (the latest tag, what langchain has a peer dep on) or add the following import:

import "npm:[email protected]";
import { CheerioWebBaseLoader } from "npm:@langchain/[email protected]/document_loaders/web/cheerio";

const loader = new CheerioWebBaseLoader(
  "https://docs.smith.langchain.com/user_guide"
);
const rawDocs = await loader.load();

It would be nice if Deno would error for conflicting peer dependencies.