Upgrade from v6 to v7 sends digested assets to outside packs folder

[chuttam]

Note: Shakapacker v7.2.3 upgrade attempt from shakapacker v6.5.2 (currently working fine in production)

A discussion had previously been opened describing the same issue, but it was closed after discussion went into an irrelevant tangent around root cause, with no solutions offered.

Root Question

Why does running /bin/shakapacker with assets from npm modules create node_modules outside packs in v7, but creates node_modules folder within packs in v6?

---

Expected behavior:

When running bin/shakapacker, some 3rd-party assets (.eot, .woff, .woff2 font assets from font-awesome) are digested and placed in the following folder:

public/packs/node_modules/font-awesome  # <-- node_modules folder is a sub-folder in the packs folder.

This is the current, working behaviour with shakapacker v6.5.2. Browser access to https://my.asset.host/packs/node_modules/@fortawesome/fontawesome-pro/webfonts/fa-brands-400-2ac0cbf0ae3aa5e15f77.eot for example, resolves fine.

These assets are imported via the following SCSS file and appear to be picked up fine by the loader, and generate the pack manifest.

// Import Font Awesome
$fa-font-path: '~@fortawesome/fontawesome-pro/webfonts';   # <-- base URL variable can be set, for subsequent url(...) usage
@import '~@fortawesome/fontawesome-pro/scss/fontawesome';
@import '~@fortawesome/fontawesome-pro/scss/regular';
@import '~@fortawesome/fontawesome-pro/scss/light';
@import '~@fortawesome/fontawesome-pro/scss/solid';
@import '~@fortawesome/fontawesome-pro/scss/brands';

Actual behavior:

After upgrade to shakapacker v7.2.3, these assets are digested to a node_modules/ folder in public/ (i.e. outside the pack):

public/node_modules/font-awesome.  # <-- node_modules folder is now a peer folder of the packs folder.

The upgrade from shakapacker 6 to 7 is essentially the renaming of "webpacker" to "shakapacker". Here is a copy of the webpack config (identical to v6 version):

webpack.config.js

const { globalMutableWebpackConfig: webpackConfig } = require('shakapacker');
const webpack = require('webpack')
const { merge } = require('webpack-merge');
const WebpackAssetsManifest = require('webpack-assets-manifest');
const path = require('path');

// This section creates application.css as a webpack manifest asset.
// Requires use of css-loader and style-loader.
webpackConfig.module.rules.map((module) => {
  if (module.test && module.test.toString().includes('scss')) {
    module.use.splice(-1, 0, {
      loader: require.resolve('resolve-url-loader'),
    });
  }
  return module;
});

const options = {
  context: path.resolve(__dirname, '../../app/frontend'),    // <-- we renamed from the default 'javascript' convention.
};

module.exports = merge(options, webpackConfig);

shakapacker.yml

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/frontend
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  check_yarn_integrity: false
  webpack_compile_output: true

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  additional_paths: ['vendor/dashkit']

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: true

development:
  <<: *default
  compile: true

  # Verifies that correct packages and versions are installed by inspecting package.json, yarn.lock, and node_modules
  check_yarn_integrity: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: webpacker
    port: 3000
    public: cdn.easyredir.test
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    live_reload: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    pretty: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'
      aggregateTimeout: 300
      poll: 500
    client:
      overlay: false
      webSocketURL: "wss://cdn.easyredir.test/ws"
    # Note that apps that do not check the host are vulnerable to DNS rebinding attacks
    allowed_hosts: "all"
    static:
      watch:
        ignored: '**/node_modules/**'

test:
  <<: *default
  compile: true

  # Compile test packs to a separate directory
  public_output_path: packs-test

staging:
  <<: *default

  # Staging depends on precompilation of packs prior to booting for performance.
  compile: false

  # Cache manifest.json for performance
  cache_manifest: true

preview:
  <<: *default

  # Preview depends on precompilation of packs prior to booting for performance.
  compile: false

  # Cache manifest.json for performance
  cache_manifest: true

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Cache manifest.json for performance
  cache_manifest: true

The generated manifest entries for the font assets appear technically correct:

 "../../node_modules/@fortawesome/fontawesome-pro/webfonts/fa-brands-400.eot": "https://my.assets.host/node_modules/@fortawesome/fontawesome-pro/webfonts/fa-brands-400-2ac0cbf0ae3aa5e15f77.eot",

However, any attempt to access the file in question is met with a 404 from our application, and these assets render as missing.

Small, reproducible repo: https://github.com/chuttam/shakapacker-scss-test

(run bin/shakapacker to see results in public/ and public/packs)

Setup environment:

• Ruby version: 3.1.6p260
• Rails version: 7.0.8.7
• Shakapacker version: Shakapacker v7.2.3 upgrade attempt from shakapacker v6.5.2 (currently working fine in production)