refactor: make progress tracking better follow LSP semantics

[Moosieus]

This pull request overhauls the way progress reporting is handled in Expert. The top-level API remains largely the same for callers.

The prior architecture started a per-project stateful GenServer that handled progress tracking logic internally:

• It was difficult to reason about due to the depth and statefulness of the call stack.
  • Or at least, I found it difficult. YMMV, but my brain works better with shallow call stacks.
• The Progress GenServers were initiated per-project, even though workDoneTokens aren’t scoped as such in LSP.
• Had no means to handle client-initiated workDoneTokens
  • I had a with_client_progress function incorporated here, but I removed it for the sake of review. There’s already a lot here.
• Had no means to deallocate workDoneTokens should they be cancel-able.
  • Requests for long-running tasks can be canceled by the client, in which case it cancels the request by ID, but not any workDoneTokens associated.
    • Hence why the work-done-tokens now live in the caller’s context.

The new progress reporting modules are organized as follows:

• Forge.Progress - a behaviour that defines the with_progress and with_tracked_progress helper functions, permitting the implementing module provides the following functions:
  • begin/2
  • report/2
  • complete/2
• Expert.Progress - a process-less module that directly handles progress reporting, implements the aforementioned
• Engine.Progress - a thin wrapper that calls Expert.Progress via erpc.
• Forge.Progress.Tracker - An ephemeral GenServer for reporting progress between concurrently running tasks.
  • Supports straight percentage reporting by default, or custom callbacks like turbo-encabulating files: 3/25 should that be preferable.

All in all: Less processes, more sequential Elixir, more flexibility, better alignment with LSP semantics.

Some additional changes:

• Make tests that wait the project_compiled message less flaky by increasing the timeout from 100ms -> 5s.
• Debounce namespacing logs
• When the engine starts, store its manager name in a persistent term and use that, instead of building it.

[badosu]

Hmm.. Perhaps it would be an interesting addition for gen_lsp to support something like:

{:reply, reply, {:continue, term, state}} |
{:noreply, {:continue, term, state}}

Then support a handle_continue callback?

[Moosieus]

Matching GenServer semantics? Yeah, that'd be sweet.

The key to resolving the timing issue here is to make sure the language-client gets the initialize response, before the language-server initializes anymore workDoneTokens.

[badosu]

Yeah, that would be the idea. So that we can support a situation like this where we want to perform work on a continuation after the current work is done, with some guarantee the reply was sent - although there might be other valid reasons to want to do a continuation I guess. E.g.:

def handle_request(req, lsp) do
  reply = get_reply(req)
  {:reply, reply, {:continue, :initialize_engine, lsp}}
end

def handle_continue(:initialize_engine, lsp) do
  # maybe something related but we want to offload that particular loop iteration
  # maybe something unrelated and we want to separate scope
  # maybe something that should run in some continuous "work" manner
  # maybe we wanted some guarantee the reply was sent
  {:noreply, lsp}
end

Anyway, not an issue for this particular PR, I wondered if you'd tell me if this sounds like a reasonable "solution" for the sleep issue.

[Moosieus]

I took a crack at this: https://github.com/Moosieus/gen_lsp/tree/moo/handle-continue

Not tested in-anger against Expert at time of writing, though.

[badosu]

Look's great!