Is there a way to tell Metals to ignore a sub-project's build.sbt and instead treat all sub-projects as part of the main project?

[JL102]

(related to: #580)

Hi,

There's a very large project called Chipyard which has one big build.sbt file, which loads all sub-projects inside it, and (most importantly) declares all of the sub-projects' dependencies with each other. For example, most sub-projects depend on one specific sub-project named rocketchip. There is a version of rocketchip that's published, but it's old and to get the most recent version of rocketchip, it has to be loaded as a dependency from the source code.

For example, the project I'm currently working on has a line in that main build.sbt which declares its dependencies and settings:

lazy val messagequeue = (project in file("generators/message-queue"))
  .settings(commonSettings).dependsOn(rocketchip, constellation)
  .settings(libraryDependencies ++= rocketLibDeps.value)

I can get Metals to load that main build.sbt, but then when I open a file in my subproject (see https://github.com/JL102/lbl-message-queue-module/blob/main/src/main/scala/modules/MessageQueue.scala) Metals doesn't recognize it as part of a project. I intentionally have a typo on line 45, as a way to make sure Metals is checking for errors. It seems to refuse to do any sort of processing on that file when I tell it not to load message-queue's build.sbt. And when I run :MetalRunDoctor with that file open, it tells me explicitly that it doesn't have a project open.

In the Chipyard project, that file's relative path is ./generators/message-queue/src/main/scala/modules/MessageQueue.scala.

If I define library dependencies inside message-queue's build.sbt like this:

libraryDependencies ++= Seq(
  "edu.berkeley.cs" %% "chisel3" % chiselVersion,
  "edu.berkeley.cs" %% "chiseltest" % "0.5.4" % "test",
  "edu.berkeley.cs" % "rocketchip_2.12" % "1.2.+"
)

then the Chipyard repo's setup process will fail, because crucially, the version of rocketchip is different than that defined in the main build.sbt (as described earlier). So due to this, I can't define message-queue's dependencies inside that build.sbt. And if I define the library dependencies like this:

lazy val root = (project in file("."))
  .settings(
    name := "messagequeue",
    libraryDependencies ++= Seq(
      "edu.berkeley.cs" %% "chisel3" % chiselVersion,
      "edu.berkeley.cs" %% "chiseltest" % "0.5.4" % "test",
      "edu.berkeley.cs" % "rocketchip_2.12" % "1.2.+",
    ),
    scalacOptions ++= Seq(
      "-language:reflectiveCalls",
      "-deprecation",
      "-feature",
      "-Xcheckinit",
      "-P:chiselplugin:genBundleElements",
    ),
    addCompilerPlugin("edu.berkeley.cs" % "chisel3-plugin" % chiselVersion cross CrossVersion.full),
  )

the setup process will also fail for a different reason (something about "messagequeue" not being a valid project). So I also can't define dependencies that way. Really, I'm only able to define organization, name, version, and scalaVersion in message-queue's build.sbt.

But then if I tell Metals to load message-queue's build (as it prompts me any time I load that sub-project), and then open MessageQueue.scala, it doesn't recognize the dependencies as defined in the main build.sbt, and so it throws a thousand errors (because it doesn't recognize any of the imports). So I'm really stuck between a rock and a hard place. What I need to do is have Metals load the main build.sbt, and ignore the barebones build.sbt within the subproject, so that it recognizes all my dependencies. Is that possible?

[JL102]

Note: It's worth mentioning that if I run the Metals doctor when I have the main build.sbt open, messagequeue DOES appear, with all green check marks, under the "Build target" section. So I don't know, I think all I need is to make Metals associate all the files within message-queue with that build.sbt?

[ckipp01]

So I'll start off with saying I'm not sure how helpful I'll be if I can't actually try to reproduce what you're experiencing, but I'll try my best.

So I think in general what might be happening is that you're hitting on this #417. Basically nvim-metals knows how to traverse down and find the root of your project since the idea of a "project" inside of Neovim is quite abstract. So we have to somehow determine what files belong to what project root. I think in your situation what you end up having is:

build.sbt <- your root buildfile
generators/message-queue/build.sbt <- the subproject build file

Is this right? If so nvim-metals doesn't know how to go that many layers deep. It will only check one layer deep. It does this because if not, if you would have something like

projectA/build.sbt
scala-projects/projectB/build.sbt

It would end up thinking these two are one project. So a reasonable default is it finds the projectB/build.sbt looks one layer down and if there is none there, it will stop. If there is one there it will keep traversing.

So in your scenario the moment you open up https://github.com/JL102/lbl-message-queue-module/blob/main/src/main/scala/modules/MessageQueue.scala it thinks you're in a standalone project with the root being generators/message-queue/build.sbt.

Again if I'm understanding right, this is a bit of an unconventional setup in Scala and one thing that you could try is to hack the find_root_dir function yourself in your config since it is exposed. For example someone hit on this before as you can see in the example here. Could you clean the workspace and try that with find_root_dir to see if it fixes this?

[JL102]

Again if I'm understanding right, this is a bit of an unconventional setup in Scala and one thing that you could try is to hack the find_root_dir function yourself in your config since it is exposed. For example someone hit on this before as you can see in the example #257 (comment). Could you clean the workspace and try that with find_root_dir to see if it fixes this?

YES!!!! thank you so much!!! I managed to get it to work by modifying find_root_dir to check one level higher. This is now my config (in AstroNvim's user.lua, if anyone happens upon the same issue:

  {
    "scalameta/nvim-metals",
    dependencies = "nvim-lua/plenary.nvim",
    init = function()
      vim.api.nvim_create_autocmd("FileType", {
        pattern = { "scala", "sbt", "java" },
        callback = function()
          local Path = require "plenary.path"
          -- has_pattern taken from nvim-metals source code
          local has_pattern = function(patterns, target)
            for _, pattern in ipairs(patterns) do
              local what_we_are_looking_for = Path:new(target, pattern)
              if what_we_are_looking_for:exists() then return pattern end
            end
          end
          -- modified find_root_dir from nvim-metals, which searches one level deeper
          local find_root_dir = function(patterns, startpath)
            local path = Path:new(startpath)
            for _, parent in ipairs(path:parents()) do
              local pattern = has_pattern(patterns, parent)
              if pattern then
                local grandparent = Path:new(parent):parent()
                local gpattern = has_pattern({ pattern }, grandparent)
                if gpattern then
                  vim.print("returning grandparent " .. grandparent)
                  return grandparent.filename
                else
                  -- if grandparent not found, then try great-grandparent
                  local great_grandparent = Path:new(grandparent):parent()
                  local ggpattern = has_pattern({ pattern }, great_grandparent)
                  if ggpattern then
                    return great_grandparent.filename
                  else
                    return parent
                  end
                end
              end
            end
          end
          local metals = require "metals"
          local config = require("astronvim.utils.lsp").config "metals"
          config.find_root_dir = find_root_dir -- Attach find_root_dir to config
          metals.initialize_or_attach(config)
        end,
        group = vim.api.nvim_create_augroup("nvim-metals", { clear = true }),
      })
    end,
  },

I wonder if it would be helpful to make this a recursive function, and add a user-configurable value to define how many steps deep they want it to go. In my case, I need it to be 3 steps deep, but maybe for some even weirder projects someone might even want it to be 4 deep. @ckipp01 if I coded that up and opened a PR, would you be cool with that? Or do you prefer to keep it the way it is?

[ckipp01]

YES!!!! thank you so much!!! I managed to get it to work by modifying find_root_dir to check one level higher.

Awesome, I'm glad to hear this!

@ckipp01 if I coded that up and opened a PR, would you be cool with that? Or do you prefer to keep it the way it is?

Yes, I'd be open to this. I've been meaning to do it for a while, but never got around to it. There are also other problems I need to figure out because while some files you'll want to continue to traverse and search for like pom.xml, build.sbt, etc, there are others you don't want to ever do that for. Aka .metals/ or .scala-build. So I need to add logic to determine as well between files that it should continue to traverse for and those that it should.

[JL102]

there are others you don't want to ever do that for. Aka .metals/ or .scala-build. So I need to add logic to determine as well between files that it should continue to traverse for and those that it should.

Maybe make a key-value dictionary, instead of a list, containing the maximum number of parent directories to traverse, for each file type? (e.g., 1 for .metals / .scala-build and 3 for build.sbt / build.sc)