Xterm.js Terminal Display Element Example

[natankeddem]

I am working on this for possible usage in a project. I don't know if there is much interest in this sort of element, and there are already so many, I know there is concern over the current bloat of libraries. Let me know if anyone wants me to make this a built in element or make a formal example. In any case this discussion can be used to reference integrating a random third-party JS library into a project and some asyncio interactions.
google_screen_recording_2023-10-20T15-55_08.310Z.webm
Here is the quick and dirty code:

import asyncio
import contextlib
import shlex
from nicegui import app, ui

app.add_static_files("/static", "static")
ui.add_head_html('<link href="static/xterm.css" rel="stylesheet">')
ui.add_head_html('<script type="text/javascript" src="static/xterm.js"></script>')
app.storage.general["last_run"] = []
display_queue = asyncio.Queue()
terminate_event = asyncio.Event()


async def _read_stream(stream, seperator=None):
    while True:
        if seperator is None:
            buf = await stream.read(140)
        else:
            try:
                buf = await stream.readuntil(seperator)
            except asyncio.exceptions.IncompleteReadError as e:
                buf = e.partial
            except asyncio.exceptions.LimitOverrunError as e:
                if stream._buffer.startswith(seperator, e.consumed):
                    del stream._buffer[: e.consumed + len(seperator)]
                else:
                    stream._buffer.clear()
                stream._maybe_resume_transport()
                raise ValueError(e.args[0])
        if buf:
            await display_queue.put(buf.decode("utf-8"))
            app.storage.general["last_run"].append(buf.decode("utf-8"))
        else:
            break


async def _control_process(process):
    while process.returncode is None:
        if terminate_event.is_set():
            process.terminate()
        with contextlib.suppress(asyncio.TimeoutError):
            await asyncio.wait_for(process.wait(), 0.1)


async def stream_subprocess():
    run.disable()
    terminate_event.clear()
    cmd = shlex.split(command.value, posix=False)
    process = await asyncio.create_subprocess_exec(*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
    app.storage.general["last_run"].clear()
    await asyncio.gather(_control_process(process), _read_stream(process.stdout, b"\r"), _read_stream(process.stderr, b"\r"))
    terminate_event.clear()
    if terminate_event.is_set():
        terminal.call_terminal_method("write", "\n")
    run.enable()
    return await process.wait()


async def update_display():
    while True:
        line = await display_queue.get()
        terminal.call_terminal_method("write", line)


async def show_last():
    for line in app.storage.general["last_run"]:
        terminal.call_terminal_method("write", line)


async def terminate():
    terminate_event.set()


with ui.row() as row:
    row.tailwind.align_items("center")
    command = ui.input("Command", value="curl --limit-rate 1k -O http://speedtest.ftp.otenet.gr/files/test100k.db")
    command.tailwind.width("[480px]")
    run = ui.button("run", on_click=stream_subprocess)
    ui.button("show last", on_click=show_last).bind_enabled_from(run, "enabled")
    ui.button("clear", on_click=lambda _: terminal.call_terminal_method("reset")).bind_enabled_from(run, "enabled")
    ui.button("terminate", on_click=terminate)


class Terminal(ui.element, component="static/terminal.js", libraries=["static/xterm.js"]):
    def call_terminal_method(self, name: str, *args) -> None:
        self.run_method("call_api_method", name, *args)


terminal = Terminal()
terminal.tailwind.width("full")

ui.timer(0.1, update_display, once=True)
if __name__ in {"__main__", "__mp_main__"}:
    ui.run()

terminal.js

export default {
  template: "<div></div>",
  mounted() {
    this.terminal = new Terminal({convertEol: true});
    this.terminal.open(this.$el);
  },
  beforeDestroy() {
    this.terminal.dispose();
  },
  beforeUnmount() {
    this.terminal.dispose();
  },
  methods: {
    call_api_method(name, ...args) {
      this.terminal[name](...args);
    },
  }
};

You can get xterm.js and xterm.css from npm or a CDN like this:
https://cdn.jsdelivr.net/npm/[email protected]/
Then stick all the js and css files in a folder called static in the root of the project.

[stdusr]

Sorry for noob question, but I have zero knowledge in JS. Where do I get xterm.js and xterm.css files? I downloaded github repo https://github.com/xtermjs/xterm.js, but these files are absent.

[natankeddem]

You can get them with npm or from a CDN like this:
https://cdn.jsdelivr.net/npm/[email protected]/

[natankeddem]

I updated the original post with better instructions and some better structured code since it appears there was some interest.

[falkoschindler]

Very nice, @natankeddem!

I'm also not sure if there's demand for xterm as a built-in NiceGUI element. Isn't it similar to ui.log? Maybe adding such streaming capabilities would be a useful extension. 🤔

As a side note: I'm currently struggling with a leaflet integration (see PR #1217), which also comes with both JavaScript and CSS files. Getting a proof of concept up and running is always relatively easy thanks to ui.add_head_html, but once you want to wrap it into a handy module and start to put multiple of them on the same page, it gets messy. That's where I'm struggling right now. Apart from that there's currently no way to register third-party CSS files for custom components, only for JavaScript files via libraries= etc.

[natankeddem]

XTerm.js is a large project with a built out api.
http://xtermjs.org/docs/
It seems to have become the defacto standard for showing "terminal" like displays utilizing JS. They have a huge list of places it is used but notably, we all use xterm.js inside VS Code. Theoretically, with enough glue (maybe using pty?) you could create a fully interactive terminal using NiceGUI.

[rodja]

I can think of quite some use cases where an xterm.js browser terminal could become handy. Foremost in combination with On Air it would allow an easy remote access to your deployment.

[natankeddem]

I am continuing to work with this, and exercise it in some various situations. I did find a strange issue, and I am not really sure how to diagnose what is going on. What exactly does using @ui.page("/") do to page rendering? It appears to break some aspect of xterm.js loading correctly.
First with no ui.page:

with ui.row() as row:
    row.tailwind.align_items("center")
    cmd = ui.input("Command", value="curl --limit-rate 1k -O http://speedtest.ftp.otenet.gr/files/test100k.db")
    cmd.tailwind.width("[480px]")
    run = ui.button("run", on_click=execute)
    ui.button("show last", on_click=show_last).bind_enabled_from(run, "enabled")
    ui.button("clear", on_click=lambda _: terminal.call_terminal_method("reset")).bind_enabled_from(run, "enabled")
    ui.button("terminate", on_click=cli.terminate)
terminal = el.Terminal(options={"rows": 30, "cols": 120, "convertEol": True})

renders correctly:

Now I encapsulate the same exact layout in ui.page:

@ui.page("/")
def page_layout():
    with ui.row() as row:
        row.tailwind.align_items("center")
        cmd = ui.input("Command", value="curl --limit-rate 1k -O http://speedtest.ftp.otenet.gr/files/test100k.db")
        cmd.tailwind.width("[480px]")
        run = ui.button("run", on_click=execute)
        ui.button("show last", on_click=show_last).bind_enabled_from(run, "enabled")
        ui.button("clear", on_click=lambda _: terminal.call_terminal_method("reset")).bind_enabled_from(run, "enabled")
        ui.button("terminate", on_click=cli.terminate)
    terminal = el.Terminal(options={"rows": 30, "cols": 120, "convertEol": True})

Now this happens:

I see no obvious errors in the python console or browser console. I understand I didn't give a fully executable example here but if you want to piece it together with my original post you can recreate my issue I think. I am hoping there is something obvious I missed in my setup which would cause this.

[natankeddem]

I have narrowed it down to the CSS not being loaded when using ui.page. Maybe a similar issue to the leaflet css problems you were alluding to @falkoschindler?

[natankeddem]

It appears it might be:

export default {
  template: "<div></div>",
  async mounted() {
    await this.load_resource("https://cdn.jsdelivr.net/npm/[email protected]/css/xterm.css");
    this.terminal = new Terminal(this.options);
    this.terminal.open(this.$el);
  },
  beforeDestroy() {
    this.terminal.dispose();
  },
  beforeUnmount() {
    this.terminal.dispose();
  },
  methods: {
    load_resource(url) {
      return new Promise((resolve, reject) => {
        const dataAttribute = `data-${url.split("/").pop().replace(/\./g, "-")}`;
        if (document.querySelector(`[${dataAttribute}]`)) {
          resolve();
          return;
        }
        let element;
        if (url.endsWith(".css")) {
          element = document.createElement("link");
          element.setAttribute("rel", "stylesheet");
          element.setAttribute("href", url);
        } else if (url.endsWith(".js")) {
          element = document.createElement("script");
          element.setAttribute("src", url);
        }
        element.setAttribute(dataAttribute, "");
        document.head.appendChild(element);
        element.onload = resolve;
        element.onerror = reject;
      });
    },
    call_api_method(name, ...args) {
      this.terminal[name](...args);
    },
  },
  props: {
    options: Object,
  }
};

I stole load_resource from your PR and it seems to fix the issue.

[falkoschindler]

If you put

ui.add_head_html('<link href="static/xterm.css" rel="stylesheet">')
ui.add_head_html('<script type="text/javascript" src="static/xterm.js"></script>')

in the global scope (and not the page generator function), it will not be part of the page. I guess this is why these resources were missing in #1846 (comment).

[falkoschindler]

Now that you got it working: Can you place multiple terminals on one page?
In #1217 load_resource seemed to have problems in this case.

[natankeddem]

Yes, moving those into the page generator seems to be the key. I forgot about the scoping on that method. Thank you for that.

Now just for testing. I removed all ui.add_head_html and utilized just the load_resource. I put up 2 terminals and I had no issues.

I think long term I am going to just inject the resources at the page scope and revert back to a more simple non-async mount and remove load_resource since that seems to be working. Waiting for the resources to load and managing that was more complicated than required for this library I think.

[paco-sevilla]

I just created PR #4520, adding an example for integrating Xterm.js to run Bash in a pty.

As I wrote there, I would be happy to close that PR and work on a proper integration, if you think this has value for nicegui 🙂

[gohilurvish]

I was trying to implement "Command Runner" since last two weeks using asyncio.create_subprocess_shell().
While it worked great, it was creating a new process for each command making it loose the cwd.

Banged head with asyncio.create_subprocess_exec() and keeping it open for sending and receiving command, while it worked, the issue I found was to know when the command execution ended.

Then came across Xterm.js but somehow could not integrate it. I saw thing thread long back but there was no update.
Thanks @natankeddem for starting this and @paco-sevilla for the PR.

Going to try this soon!