Packer v2 Rewrite

ROADMAP.md

@@ -0,0 +1,61 @@
+# Roadmap for Packer v2
+
+`packer` has become bloated, compilation (at least as it currently stands) has proven confusing and probably not necessary, and the codebase has become messy and hard to maintain.
+
+As such, I'm proposing/working on (very slowly, as my OSS time is quite limited in my current job) an incremental rewrite of `packer`.
+The below is a high-level overview of this plan.
+Comments and additions are welcome and encouraged!
+
+## General principles
+
+`packer`'s code currently suffers from some poor software development practices largely stemming from getting "discovered" before it had time to become polished.
+We want to avoid these pitfalls in the rewrite:
+
+- Document all functions with emmylua-style comments: `packer` has very sparse documentation in its codebase. New/updated functions need to be appropriately documented.
+- Add tests where possible: testing a package manager can be a pain because of requirements to interact with the file system, etc. However, `packer`'s current tests do not cover much, and new code should at least strongly consider adding unit tests.
+- Avoid giant functions: `packer`'s logic contains some monstrous functions (e.g., `manage`, the main compilation logic, etc.) that are difficult to work with. New code should strive to use more, simpler functions in a more modular design.
+- Prioritize clean, performant code over total flexibility: `packer`'s design allows for a very flexible range of input formats. Although we don't want to completely lose this property, supporting all of these formats (and other use cases) has contributed to code bloat. If there's a choice to be made between losing some flexibility and significantly increasing the complexity of the code, seriously consider removing the complexity.
+- Reduce redundancy: `packer` currently has an annoying amount of redundant/overlapping functionality (the worst offenders are the `requires`/`wants`/`after` family of keywords, but there's also duplication of logic in the utilities, etc.). The rewrite should aim to reduce this.
+
+## Stage 1: Remove compilation and clean up main interface. Breaking changes
+
+We want the rewrite process to be as unintrusive as possible while still allowing for significant, potentially breaking changes.
+As such, the first stage should contain all of the significant breaking changes to reduce the number of times users need to adapt.
+The goals of the first stage are:
+
+- [ ] Mostly remove compilation: this step involves moving to dynamic handlers for plugin specs and only "compiling" (really, caching) information that we can automatically recompile as needed, like additional filepaths to source, etc.
+  - [x] Make main module lightweight: if `packer` is no longer relying on the compiled file being lightweight, its main module must become cheaper to `require`, as this will be necessary at all startups. This is mostly a question of reducing the amount that `packer` pulls in at the top level (e.g., moving things like management operations more fully into their own modules, reducing the reliance on utils modules, etc.).
+  - [x] Implement fast runtime handler framework: To replace the compiled file, we will introduce a notion of "keyword handlers": simple functions which are responsible for implementing, at runtime, the behavior of a single previously-compiled keyword. See `lua/packer/handlers.lua` for the current spec and implementation of this idea.
+  - [ ] Implement compiled keywords as handlers: Work in progress/current state of the rewrite. All the existing keywords from `packer/compile.lua` need to be ported to handlers. This involves dealing with `compile.lua`'s tricky logic in some places, and is a key point for simplification.
+  - [ ] Implement `load` function to process and execute handlers: The final step of moving away from compilation is to write a function that runs the handlers on the plugin specs to generate lazy-loaders, run `setup` and `config` functions, etc. The trick here will be making this sufficiently fast.
+  - [ ] Potentially implement path caching, etc.: Once the bulk of compilation is gone, we may wish to investigate adding a little bit back by seeing where the runtime system spends time during startup. If this includes a significant amount of time checking information that only changes when a plugin is installed or updated (e.g., searching for runtime paths), then we may want to start caching this information in a file that gets updated on updates to save startup time.
+- [ ] Simplify `requires`/`wants`/etc.
+  - [ ] Unify `requires`/`wants` into `depends_on`: `requires` and `wants` currently duplicate functionality. I propose merging the two into a single keyword `depends_on`, which will (1) ensure that dependencies are installed and (2) ensure that dependencies are loaded before the dependent plugin.
+  - [ ] Make interface for `depends_on` and `after` consistent: the correct way to specify dependent plugins/sequential load plugins is confusing, since it's based on a plugin's short name in some cases but the full name in others (I think?). We should define and implement a clear way of referring to plugins in these settings.
+- [ ] Clean up interface
+  - [ ] Use new Neovim v0.7.0 functions instead of `vim.cmd`: the codebase makes use of a lot of `vim.cmd` with string building for things like generating keymaps, commands, etc. Neovim v0.7.0 introduced the ability to create these directly from Lua, with direct Lua callbacks. `packer` should move to use these functions.
+  - [ ] Make management operations sequenceable steps: To clean up and simplify the code for management operations (e.g., installs, updates, cleans, etc.), we want to redesign the operation interface into sequenceable steps (e.g., "check current FS state", "clone missing plugins", "pull installed plugins", etc.) so that the actual operations become a sequence of calls to functions implementing these steps, rather than the current highly-duplicated logic.
+
+## Stage 2: Simplify internal git functions and use of async
+
+The nastiest part of the codebase is the `git` logic.
+We would like to simplify this as much as possible.
+Also, a significant contributor to the difficulty of working on `packer` is its aggressive use of `async` everywhere.
+We should investigate how to minimize the async surface area to allow more flexibility and require less of the codebase to need to think about `async`.
+
+## Stage 3: Issue triage
+
+`packer` has accumulated a large number of outstanding issues.
+At this point, we should go through and try to reproduce issues with the updated codebase.
+During this process, issues should either be fixed or closed as "Wontfix" or "No longer relevant" unless they are feature requests or questions.
+The goal of this stage is to more or less close out the open issues.
+
+## Stage 4: Gradual porting to Teal
+
+Teal offers significant benefits for maintainability and has improved since the last time I took a look at porting `packer`.
+After `packer`'s code is at a cleaner, more reliable state, it makes sense to start porting modules incrementally to Teal.
+
+## Stage 5 and onward: new features, Luarocks improvements
+
+Finally, we can start thinking about new features and improvements.
+Things like the proposed unified plugin specification format, improvements to Luarocks (e.g., updating rocks, putting binary rocks on the Neovim `PATH`, etc.)

lua/packer.lua

@@ -682,6 +682,12 @@ packer.compile = function(raw_args, move_plugins)
   local a = require 'packer.async'
   local async = a.sync
   local await = a.wait
+  -- TODO: check if the user wants any compilation at all via a new config variable
+  -- TODO: remove a bunch of complexity from the compile module/maybe here once we don't need to
+  -- compile functions or sequencing
+  -- TODO: implement, for the compiled file, logic to check its values against the current config.
+  -- We can do this by computing an efficient hash of the lazy-loader strings and storing this in
+  -- the compiled file
 
   manage_all_plugins()
   async(function()
@@ -994,4 +1000,11 @@ packer.startup = function(spec)
   return packer
 end
 
+packer.load = function()
+  -- TODO: This will be responsible for (1) sourcing the new, lighter, compiled file and (2)
+  -- handling things like running setup and config functions, sequencing, etc. 90% of this logic
+  -- already exists in the load module; the main thing to add is loading the new compiled file. This
+  -- function needs to be called at the end of startup()
+end
+
 return packer

lua/packer/config.lua

@@ -0,0 +1,86 @@
+--- Configuration
+
+local path_utils = require 'packer.path'
+local stdpath = vim.fn.stdpath
+local join_paths = path_utils.join_paths
+local path_separator = path_utils.path_separator
+
+local defaults = {
+  package_root = join_paths(stdpath 'data', 'site', 'pack'),
+  compile_path = join_paths(stdpath 'config', 'plugin', 'packer_compiled.lua'),
+  plugin_package = 'packer',
+  max_jobs = nil,
+  auto_clean = true,
+  compile_on_sync = true,
+  disable_commands = false,
+  opt_default = false,
+  transitive_opt = true,
+  transitive_disable = true,
+  auto_reload_compiled = true,
+  git = {
+    mark_breaking_changes = true,
+    cmd = 'git',
+    subcommands = {
+      update = 'pull --ff-only --progress --rebase=false',
+      install = 'clone --depth %i --no-single-branch --progress',
+      fetch = 'fetch --depth 999999 --progress',
+      checkout = 'checkout %s --',
+      update_branch = 'merge --ff-only @{u}',
+      current_branch = 'rev-parse --abbrev-ref HEAD',
+      diff = 'log --color=never --pretty=format:FMT --no-show-signature HEAD@{1}...HEAD',
+      diff_fmt = '%%h %%s (%%cr)',
+      git_diff_fmt = 'show --no-color --pretty=medium %s',
+      get_rev = 'rev-parse --short HEAD',
+      get_header = 'log --color=never --pretty=format:FMT --no-show-signature HEAD -n 1',
+      get_bodies = 'log --color=never --pretty=format:"===COMMIT_START===%h%n%s===BODY_START===%b" --no-show-signature HEAD@{1}...HEAD',
+      submodules = 'submodule update --init --recursive --progress',
+      revert = 'reset --hard HEAD@{1}',
+    },
+    depth = 1,
+    clone_timeout = 60,
+    default_url_format = 'https://github.com/%s.git',
+  },
+  display = {
+    non_interactive = false,
+    open_fn = nil,
+    open_cmd = '65vnew',
+    working_sym = '⟳',
+    error_sym = '✗',
+    done_sym = '✓',
+    removed_sym = '-',
+    moved_sym = '→',
+    header_sym = '━',
+    header_lines = 2,
+    title = 'packer.nvim',
+    show_all_info = true,
+    prompt_border = 'double',
+    keybindings = { quit = 'q', toggle_info = '<CR>', diff = 'd', prompt_revert = 'r' },
+  },
+  luarocks = { python_cmd = 'python' },
+  log = { level = 'warn' },
+  profile = { enable = false, threshold = nil },
+}
+
+local M = { _hooks = {} }
+M.config = {}
+function M.configure(user_config)
+  user_config = user_config or {}
+  local config = M.config
+  vim.tbl_deep_extend('force', defaults, user_config)
+  config.package_root = string.gsub(vim.fn.fnamemodify(config.package_root, ':p'), path_separator .. '$', '', 1)
+  config.pack_dir = join_paths(config.package_root, config.plugin_package)
+  config.opt_dir = join_paths(config.pack_dir, 'opt')
+  config.start_dir = join_paths(config.pack_dir, 'start')
+  config.display.non_interactive = #vim.api.nvim_list_uis() == 0
+  for i = 1, #M._hooks do
+    M._hooks[i](config)
+  end
+
+  return M.config
+end
+
+function M.register_hook(hook)
+  M._hooks[#M._hooks + 1] = hook
+end
+
+return M