How to reduce startup time besides using vim.loader.enable()?

[s-cerevisiae]

v0.9.0 did fix the path separator problem on Windows. However during testing I found that the startup time (from v0.8 on) is ~2x slower than v0.7.0 and I think that's due to recent versions removing the byte code cache.
I added vim.loader.enable() to the first line of my bootstrapping lua file but that made little (~10ms) difference.

This happens to me on both Linux and Windows machines so there might be something wrong with my config.

I'd like to know the currently recommended way to use hotpot.nvim with lazy.nvim. Thank you in advance.

My current (rather convoluted) init.lua with lazy.nvim and hotpot.nvim

vim.loader.enable()

local function bootstrap(plugin)
  local _, name = unpack(vim.split(plugin, '/'))
  local plugin_path = vim.fn.stdpath('data')..'/lazy/'..name
  if not vim.loop.fs_stat(plugin_path) then
    vim.notify('Installing '..plugin..' to '..plugin_path, vim.log.levels.INFO)
    vim.fn.system({
      'git', 'clone',
      '--filter=blob:none',
      '--single-branch',
      'https://github.com/'..plugin,
      plugin_path,
    })
  end
  vim.opt.runtimepath:prepend(plugin_path)
end

bootstrap('folke/lazy.nvim')
bootstrap('rktjmp/hotpot.nvim')

require('hotpot').setup({
  provide_require_fennel = true,
  -- BTW `:StartupTime` shows that `hotpot.fennel` takes up ~70ms even this is not set
  -- only happens on >0.8.0
})

local plugins = {
  { 'rktjmp/hotpot.nvim' },
}

local plugin_def_path = vim.fn.stdpath('config')..'/fnl/config/plugins'

if vim.loop.fs_stat(plugin_def_path) then
  for file in vim.fs.dir(plugin_def_path) do
    local name = file:match("^(.*)%.fnl$")
    plugins[#plugins + 1] = require("config.plugins."..name)
  end
end

require('lazy').setup(plugins)

require('config')

[rktjmp]

I think by :StartupTime you mean --startuptime? Or some plugin?

I see its https://github.com/dstein64/vim-startuptime

If I run your init, without your config obviously, hotpots load times (on an SSD, but old processor), is a bit under 1ms.

003.150  000.110  000.110: require('vim.uri')
003.176  000.224  000.114: require('hotpot.loader.record')
003.228  000.050  000.050: require('hotpot.lang.fennel')
003.275  000.046  000.046: require('hotpot.loader.record.module')
003.401  000.124  000.124: require('hotpot.loader.sigil')
003.441  000.773  000.188: require('hotpot.loader')
003.514  000.071  000.071: require('hotpot.common')
003.565  000.049  000.049: require('hotpot.neovim.ftplugin')
003.651  000.084  000.084: require('hotpot.api.make')
003.826  001.587  000.592: require('hotpot')
003.912  000.083  000.083: require('hotpot.runtime')
003.995  000.080  000.080: require('hotpot.api.diagnostics')
004.331  000.316  000.316: require('lazy')

Can you post your --startuptime log? Also is a 2x increase going from 4ms to 8ms, or 40ms to 80ms?

[rktjmp]

Hm, probably either

• The index files are not being saved (or loaded?) because of some error
• The "needs compilation" check is always returning true (which could happen if the index files cant be saved).

Hotpot has two artefacts, the lua code and "index records" which basically track metadata about the lua source: where its fnl file is, when it was last modified, related files (the source, macros that it uses), etc.

You can check the needs compilation one pretty easy, add the following line to loader/init.fnl, after line 65:

(print :missing? (lua-missing?) :changed? (files-changed?) (vim.inspect record))

You can get the cache dir from, :lua= require("hotpot.api.cache")["cache-prefix"](), then check ../index from that. You should see a bunch of files in there, you should have something ending in %....options.lua-metadata.mpack etc if they are getting saved.

I saw some nix lines in those logs which has caused issues before but we only write to (vim.fn.stdpath :cache), which should be allowed by nix.

Looking at it, I think it can only be the needs compilation check is what is failing, since the index stuff failing would stop us loading the lua at all I think -- unless there is a bug in the saving that isnt getting surfaced.

Its late here now, I might open a logging branch tomorrow.

[s-cerevisiae]

Yup, my config files are always considered stale. Their metadata (logged from files-changed?) seems a bit funny:

path: $USER/.config/nvim/fnl/config/init.fnl historic_size: 0 current_size: 340 hsec: 0 hnsec: 0 csec: 1693764242 cnsec: 568441700
path: $USER/.config/nvim/fnl/config/options.fnl historic_size: 0 current_size: 1310 hsec: 0 hnsec: 0 csec: 1692345982 cnsec: 95838691
path: $USER/.config/nvim/fnl/config/events.fnl historic_size: 0 current_size: 187 hsec: 0 hnsec: 0 csec: 1688093119 cnsec: 903753686
path: $USER/.config/nvim/fnl/config/commands.fnl historic_size: 0 current_size: 89 hsec: 0 hnsec: 0 csec: 1688093119 cnsec: 902753667
path: $USER/.config/nvim/fnl/config/diagnostics.fnl historic_size: 0 current_size: 1022 hsec: 0 hnsec: 0 csec: 1688093119 cnsec: 903753686
path: $USER/.config/nvim/fnl/config/keymaps.fnl historic_size: 0 current_size: 4746 hsec: 0 hnsec: 0 csec: 1693744521 cnsec: 436370847

[rktjmp]

"historic_size" is the metadata for files when we first save the record, it defaults to 0 when we create the record

hotpot.nvim/fnl/hotpot/loader/record.fnl

Lines 127 to 128 in c5ae8f2

	;; Include the source file with invalid data so the compiler runs.
	:files [{:path src-path :mtime {:sec 0 :nsec 0} :size 0}]})))

So they "should" be 0 the first time we see a file, the "changed?" check will fail the first time in record-loadfile (which compiles the record if needed and calls luas loadfile on the record target).

In record-loadfile, we check, compile if needed and update the meta data in replace-index-files:

hotpot.nvim/fnl/hotpot/loader/init.fnl

Lines 75 to 89 in c5ae8f2

	(if (needs-compilation? record)
	(case-try
	(compile-record record modules-options macros-options preprocessor) (true deps)
	(replace-index-files record deps) record
	(save-index record) record
	(bust-vim-loader-index record) _
	(needs-cleanup) :TODO
	(loadfile record.lua-path)
	(catch
	;; fennel compiler errors are hard errors but everything else should
	;; just nil out and pass to another loader.
	(false e) (values e)
	_ nil))
	;; Lua is up to date so we can return that as is
	(loadfile record.lua-path))))

What we want to see from that code is:

• record.files should be a table of files with 0 meta data.
• deps has a list of files related to the record (might be empty if they dont use any macros)
• after the call to replace-index-files, record.files should have real meta data
• the record is getting saved out (might have to dip into

  hotpot.nvim/fnl/hotpot/loader/record.fnl

  Line 81 in c5ae8f2

  (fn save [record]

  and make sure...)

eg:

You should be able to drop this code in for logging:

(case-try
        (if (record.src-path:find "fnl/config")
          (dprint "initial record" &record)) _
        (compile-record record modules-options macros-options preprocessor) (true deps)
        (if (record.src-path:find "fnl/config")
          (dprint "compiled record" &deps)) _
        (replace-index-files record deps) record
        (if (record.src-path:find "fnl/config")
          (dprint "post-replace" &record)) _
        (save-index record) record
        (if (record.src-path:find "fnl/config")
          (dprint "post-save" &record)) _
        (bust-vim-loader-index record) _
        (needs-cleanup) :TODO
        (loadfile record.lua-path)
        (catch
          ;; fennel compiler errors are hard errors but everything else should
          ;; just nil out and pass to another loader.
          (false e) (values e)
          _ nil))

If all that is working (metadata is getting filled, record is getting saved) then I guess its down to checking

hotpot.nvim/fnl/hotpot/loader/record.fnl

Line 51 in c5ae8f2

(fn load [lua-path]

and making sure they're being loaded up. Particularly that the index-path is uniform both times.

• Are any of your files or dirs symlinked?
• What is your windows user name? What does vim.fn.stdpath("cache") return, specifically does it have <TRUNCATED-USERNAME>~1 in it?

[rktjmp]

Possibly the windows issue was related to an upstream bug: neovim/neovim#25019 which should be mitigated as of a21771e

[rktjmp]

Is this still an issue? Sorry I cant give better help, cant reproduce it here.

[monkoose]

I guess I have some related issue. But i'm on linux.
I'm using latest neovim (maybe it is an issue, and can't downgrade that simply to stable 0.9, because of using a lot of new api from 0.10 in my config).

When i loading neovim it's startup is more than needed

004.854  000.084  000.084: require('hotpot.fs')
005.099  000.115  000.115: require('hotpot.runtime')
005.119  000.262  000.147: require('hotpot.loader.record')
005.194  000.072  000.072: require('hotpot.lang.fennel')
005.336  000.140  000.140: require('hotpot.loader.record.module')
005.435  000.096  000.096: require('hotpot.loader.sigil')
005.514  000.984  000.330: require('hotpot.loader')
005.683  000.167  000.167: require('hotpot.common')
005.778  000.091  000.091: require('hotpot.neovim.runtime')
006.020  000.239  000.239: require('hotpot.api.make')
006.209  002.341  000.819: require('hotpot')
007.683  000.150  000.150: require('hotpot.lang.fennel.compiler')
024.555  000.068  000.068: require('hotpot.searcher')
024.901  000.048  000.048: require('hotpot.dependency-map')
024.949  000.044  000.044: require('hotpot.lang.fennel.dependency-tracker')
098.081  073.110  070.333: require('hotpot.fennel')
098.186  000.096  000.096: require('hotpot.traceback')

I have added (print :missing? (lua-missing?) :changed? (files-changed?) (vim.inspect record)) and here is its result. As can be seen some files recognized as changed? true for some reason, even if though they shouldn't.

missing? false changed? false {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 44707856,
        sec = 1699831346,
        <metatable> = <1>{
          __tostring = <function 1>
        }
      },
      path = "/home/monkoose/.config/nvim/fnl/conf/options.fnl",
      size = 2785,
      <metatable> = <table 1>
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/options.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/conf/options.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/options.lua",
  ["lua-path-mtime-at-save"] = {
    nsec = 989829669,
    sec = 1699884224,
    <metatable> = <table 1>
  },
  ["lua-path-size-at-save"] = 2063,
  modname = "conf.options",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/conf/options.fnl",
  type = 1,
  <metatable> = <table 1>
}
missing? false changed? false {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 344214841,
        sec = 1699710410,
        <metatable> = <1>{
          __tostring = <function 1>
        }
      },
      path = "/home/monkoose/.config/nvim/fnl/custom/utils.fnl",
      size = 442,
      <metatable> = <table 1>
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/custom/utils.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/custom/utils.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/custom/utils.lua",
  ["lua-path-mtime-at-save"] = {
    nsec = 326496388,
    sec = 1699884231,
    <metatable> = <table 1>
  },
  ["lua-path-size-at-save"] = 752,
  modname = "custom.utils",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/custom/utils.fnl",
  type = 1,
  <metatable> = <table 1>
}
missing? false changed? false {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 825196487,
        sec = 1699812872,
        <metatable> = <1>{
          __tostring = <function 1>
        }
      },
      path = "/home/monkoose/.config/nvim/fnl/plugins/lazy.fnl",
      size = 9281,
      <metatable> = <table 1>
    }, {
      mtime = {
        nsec = 358124770,
        sec = 1699727931,
        <metatable> = <table 1>
      },
      path = "/home/monkoose/.config/nvim/fnl/conf/macros.fnl",
      size = 5217,
      <metatable> = <table 1>
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/plugins/lazy.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/plugins/lazy.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/plugins/lazy.lua",
  ["lua-path-mtime-at-save"] = {
    nsec = 186496408,
    sec = 1699884234,
    <metatable> = <table 1>
  },
  ["lua-path-size-at-save"] = 7321,
  modname = "plugins.lazy",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/plugins/lazy.fnl",
  type = 1,
  <metatable> = <table 1>
}
missing? false changed? true {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 0,
        sec = 0
      },
      path = "/home/monkoose/.config/nvim/fnl/conf/autocmds.fnl",
      size = 0
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/autocmds.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/conf/autocmds.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/autocmds.lua",
  ["lua-path-mtime-at-save"] = 0,
  ["lua-path-size-at-save"] = 0,
  modname = "conf.autocmds",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/conf/autocmds.fnl",
  type = 1
}
missing? false changed? true {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 0,
        sec = 0
      },
      path = "/home/monkoose/.config/nvim/fnl/statusline.fnl",
      size = 0
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/statusline.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/statusline.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/statusline.lua",
  ["lua-path-mtime-at-save"] = 0,
  ["lua-path-size-at-save"] = 0,
  modname = "statusline",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/statusline.fnl",
  type = 1
}
missing? false changed? true {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 0,
        sec = 0
      },
      path = "/home/monkoose/.config/nvim/fnl/custom/view.fnl",
      size = 0
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/custom/view.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/custom/view.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/custom/view.lua",
  ["lua-path-mtime-at-save"] = 0,
  ["lua-path-size-at-save"] = 0,
  modname = "custom.view",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/custom/view.fnl",
  type = 1
}
missing? false changed? true {
  ["cache-root-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim",
  ["colocation-root-path"] = "/home/monkoose/.config/nvim/",
  files = { {
      mtime = {
        nsec = 0,
        sec = 0
      },
      path = "/home/monkoose/.config/nvim/fnl/conf/maps.fnl",
      size = 0
    } },
  ["lua-cache-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/maps.lua",
  ["lua-colocation-path"] = "/home/monkoose/.config/nvim/lua/conf/maps.lua",
  ["lua-path"] = "/home/monkoose/.cache/nvim/hotpot/compiled/nvim/lua/conf/maps.lua",
  ["lua-path-mtime-at-save"] = 0,
  ["lua-path-size-at-save"] = 0,
  modname = "conf.maps",
  namespace = "nvim",
  ["sigil-path"] = "/home/monkoose/.config/nvim/.hotpot.lua",
  ["src-path"] = "/home/monkoose/.config/nvim/fnl/conf/maps.fnl",
  type = 1
}

If I use .hotpot.lua for my config, then neovim startup time has comparable time to lua config, but it compiles files into lua/ directory obviously and seems like loader module does not invoke, because I don't see similar message at startup.

Any ideas what is going on?

[rktjmp]

Is your config dir a plain old dir, or symlinks? Or anything else unusual going on? Funny mounting? What linux flavour?

[monkoose]

I'm on arch linux. And there are no symlinks related to neovim config.