Proposal for Deployed Dashboards Enhancement Using Drag & Drop Functionality #2052

jannikbecher · 2023-07-11T06:19:38Z

TODOs:

handle direct /canvas access (992e9fd)
when removing output from notebook add information to cell
implement deployed canvas as Jose suggests
update canvas_settings as Jonatan suggests
use phx-hook to update gridstack entries
write missing docs
write tests

Known Bugs:

when canvas is closed, you have to click twice on canvas to re enable (91e956d)
canvas options menu doesn't hide when "close" is clicked
when pop out window refreshes, it sends the same event as if it was closed
when in canvas mode and sidebar is open, the indicator buttons are in the wrong position

Original First Message

screen-capture.2.webm

Heyho,

It would be really nice to have the functionality to create deployed dashboards by rearranging cell outputs via Drag&Drop.
This is just a PoC and I thought it would be best to reach out to you before spending to much time with it since this will introduce a new javascript dependency and I'm not sure whether it is a good idea.
Is this something worth exploring or should I completely abandon this idea?

josevalim · 2023-07-11T08:10:05Z

Hi @jannikbecher, this is beautiful. ❤️ Curiously, this is something we had in mind for a while and I have opened an issue about this yesterday (see #2048).

However, I would make it so the canvas is not only related to deployment but always visible. Imagine something like this:

In the bottom right, on presentation view, we will add a canvas option
Once you click it, the current page splits in two, with the notebook and the canvas side by side. You can also click a button to pop the canvas out to its own page, which is very useful for separate monitors or if you want more control to arrange it
You can now send any output to the canvas by clicking a button (or drag-and-drop if on split mode)
You can deploy the canvas

It is very similar to what you did, except we want the canvas to always be running, since the canvas can be very useful during development/exploration too.

josevalim · 2023-07-11T08:14:39Z

Some additional notes:

We should save the canvas information within the notebook (I assume you already do, as it is required for deployment)
The canvas will be hidden under the "presentation view" but maybe perhaps folks have more ideas on how to expose it
Perhaps dashboard is a better name than canvas :D

hugobarauna · 2023-07-11T12:51:35Z

Perhaps dashboard is a better name than canvas :D

I'm not sure.

Given the definition of a dashboard:

A dashboard is a type of graphical user interface which often provides at-a-glance views of key performance indicators relevant to a particular objective or business process (Wikipedia)

I can see how one can use that feature to build other things besides dashboards. For example, this feature would be useful to help one structure the layout of their Livebook App in a non-linear notebook-like way.

So maybe canvas is still a better name to catch potential use cases. Or, I don't know, maybe "free-form layout."

jannikbecher · 2023-07-11T14:33:30Z

In the bottom right, on presentation view, we will add a canvas option

Once you click it, the current page splits in two, with the notebook and the canvas side by side. You can also click a button to pop the canvas out to its own page, which is very useful for separate monitors or if you want more control to arrange it

Is this going in the right direction?

screen-capture.3.webm

You can now send any output to the canvas by clicking a button (or drag-and-drop if on split mode)

Drag & Drop not implemented yet. I'm not sure whether gridstack.js is the best library to use. Do you have any preferences? I thought gridstack would be nice, because you can write <div ... x={x_pos} y={y_pos} w={width} h={height} > and I thought it would be useful. Unfortunately it conflicts with liveview DOM updates and I couldn't figure out how to run it in parallel without using phx-update="ignore"

jannikbecher · 2023-07-11T17:19:31Z

Pop canvas out

screen-capture.4.webm

TODO:

open pop out in new window (use js hook and window.open())
only show notebook when canvas is popped out
remove output from canvas

josevalim · 2023-07-11T22:24:05Z

Is this going in the right direction?

Yes, fantastic!

Drag & Drop not implemented yet.

Which is OK! maybe we should give up on it because it won't work with popped-out canvas anyway.

I thought gridstack would be nice, because you can write <div ... x={x_pos} y={y_pos} w={width} h={height} > and I thought it would be useful.

Maybe no library is going to work like that you should be using a phx-hook to update the entries. :) And then emit live events to update them.

Some nitpicks:

Please keep the bottom right indicators on the notebook and not on the canvas. The pop-out button can stay at the top right of the canvas (and maybe you can "pop-out / pop-in" and "close" as actions . :)
Once you choose the canvas, the "presentation view" needs to show as activated/green. The question is, if you pop-out, should we still show it as green? I would say no.
When you send the output to the canvas, should we remove it from the notebook? We can either leave it duplicated or maybe collapse it and say it is in the canvas, with an option to expand it back... but I think duplicated is a great start? Thoughts @jonatanklosko?

jonatanklosko · 2023-07-11T22:45:39Z

When you send the output to the canvas, should we remove it from the notebook? We can either leave it duplicated or maybe collapse it and say it is in the canvas, with an option to expand it back... but I think duplicated is a great start?

I would keep it to make it easy to identify where the output is coming from and where it can be adjusted.

jonatanklosko · 2023-07-11T23:06:16Z

lib/livebook/notebook/canvas_settings.ex

+          y: non_neg_integer(),
+          w: non_neg_integer(),
+          h: non_neg_integer(),
+          outputs: list(Cell.indexed_output())


We can't keep the outputs directly, because they wouldn't update on reevaluation and more importantly that's not something we could store in the .livemd and load later.

Because of reevaluation, it may be tricky to identify a specific output. We could store cell id + index, but if a random logger output pops up (or if the user adds a temporary inspect for debugging) it can make the index point to a wrong output. I think we should treat all outputs of specific cell atomically in this context. In most cases each rich output would come from a different cell (or it should be easy to make it so). We can store canvas placement in each cell struct and persist in cell metadata (we don't have cell ids in .livemd, so we don't want to store references to that).

And one other note, I wonder what's the best way to store the placement. Coords + size seems natural, but that is not responsive and we will need scroll when the screen is smaller than the canvas. This is an issue for deployed apps where we want things to be more responsive. Perhaps we could make the x and width quantized, so it scales with the screen, but then height cannot be a fixed value either. I'm not sure what the best solution is, just pointing out.

@josevalim if you have concerns with any of the above let me know :)

Coords + size seems natural, but that is not responsive and we will need scroll when the screen is smaller than the canvas.

Ah, I looked at gridstack and as far as I understand x, y, w, h are already quantized and it is responsive, so ignore that part :)

unfortunately I will be on a short vacation till sunday, I will change this next week :)

@jannikbecher there is absolutely no rush, thanks for working on this! Enjoy the vacation :D

jannikbecher · 2023-07-12T07:25:30Z

Please keep the bottom right indicators on the notebook and not on the canvas. The pop-out button can stay at the top right of the canvas (and maybe you can "pop-out / pop-in" and "close" as actions . :)

screen-capture.5.webm

Once you choose the canvas, the "presentation view" needs to show as activated/green. The question is, if you pop-out, should we still show it as green? I would say no.

What should happen if the canvas is popped out and the user clicks on "Canvas"?

When you send the output to the canvas, should we remove it from the notebook? We can either leave it duplicated or maybe collapse it and say it is in the canvas, with an option to expand it back... but I think duplicated is a great start? Thoughts @jonatanklosko?

I thought it was easier to remove it from the notebook, so there aren't any duplicated id conflicts.

I would keep it to make it easy to identify where the output is coming from and where it can be adjusted.

Maybe scroll in the notebook to the selected output in the canvas and highlight it?

josevalim · 2023-07-12T07:45:21Z

A related question is: the button that sends an output to canvas should most likely always be available (so buttons don't move around as we change the presentation mode).

So what happens if you click send to canvas and there is no canvas? I would say we should automatically open up the canvas. But what happens if you click to send to canvas and the canvas is in another page? We don't want to open up the canvas locally, so we should have an indicator that the canvas is running on another page. However, can we know the canvas is running on another page? I think we can only do that if we literally use the browser pop-out, but could that lead to a subpar experience? If the user experience is great and pop-out is the way to go, then we can answer further questions. :)

Btw, the canvas management buttons should only exist inside the canvas, never in the notebook side. Basically, we want to do our best to avoid adding more buttons to the notebook UI. In the canvas side, I think we can simply have a triple-vertical-dot on the top right (perhaps with a rounded border):

And when you click it, a submenu appears with two options: "Pop-out / pop-in" and "Close". There is one last question: if you access /canvas directly, there is no way we can pop-in, so in those cases we most likely don't show any menu whatsoever? Perhaps the trick is to always render it but we use JS to hide it if we are not in the current notebook or in a popped-out session.

I thought it was easier to remove it from the notebook, so there aren't any duplicated id conflicts.

Maybe when we render them, we can pass an id-suffix or something? If for some reason we have to remove them, then we need to have an indicator saying "Output was moved to canvas." with a button to restore it back. But ideally I would keep both for now.

Btw, did you find/consider other libraries for grid management besides gridstack.js?

josevalim · 2023-07-12T07:48:49Z

hide move to canvas button when output is already on canvas

Instead of hiding it, show the button as active. If you click it again, you bring the output back. :)

josevalim · 2023-07-12T09:08:41Z

Btw, did you find/consider other libraries for grid management besides gridstack.js?

Taking a quick look around, another suitable alternative I could find is https://muuri.dev/ - it seems to be smaller in size but I am unsure which one provides the best API.

I also found many other libraries but they are all unmaintained - so we may need to accept that, whatever we choose, it will likely go unmaintained, and then perhaps choose the smallest thing possible. In this scenario, draggable.js from Shopify may be an option. It is no longer maintained but it seems to be quite small.

…tion is not possible anymore

jannikbecher · 2023-07-18T09:33:38Z

Heyho,

managed to waste some time playing around with the Broadcast Channel API. What do you think using this for client side PubSub? I have to admit I don't fully understand the client side communication (especially the iframe part) but since the current canvas implementation uses multiple windows my naive self thinks this is a good idea :-D

josevalim · 2023-07-18T11:06:18Z

@jannikbecher I am actually not sure if we should use pubsub for this one. The canvas is shared but different users have full control if they are seeing the canvas or not. Ideally we would use the browser for communication between those parts if possible. Can that be the case?

jonatanklosko · 2023-07-18T11:49:53Z

@jannikbecher Broadcast Channel API works across all tabs, but in general we don't want events to leak across tabs (e.g. if a cell is focused in one tab, we don't want to tell that to other tabs). I think sending messages directly between the windows is fine :)

For more context, the client side pub sub is expected to work only on the current page, it's just a way to communicate between all the separate hooks.

For iframes we need to use window messaging either way, because they are served from different origin.

jonatanklosko · 2023-07-18T12:02:31Z

lib/livebook_web/live/session_live.ex

+  defp after_operation(
+         %{assigns: %{canvas_pid: pid}} = socket,
+         _prev_socket,
+         {:move_output_to_canvas, _client_id, _cell_id}
+       )
+       when not is_nil(pid) do
+    send(pid, {:new_data, socket.private.data.notebook.canvas_settings})
+    socket
+  end


Let's not tie the LVs together. The canvas should subscribe to data updates and work effectively as a separate client (which is very similar to what AppSessionLive does).

…ast function is not possible anymore" This reverts commit 1c5a61d.

This reverts commit 386f0ab.

jannikbecher · 2023-07-20T12:41:33Z

Heyhey,
I finally made some progress :)

screen-capture.9.webm

In the meantime I stumbled across @zachdaniel's tweet implementing a node-based editor in elixir. I could add some collision detection and a grid for snapping and than it could completely replace gridstack.js. WDYT?

assets/js/lib/pub_sub.js

assets/js/hooks/cell.js

jannikbecher · 2023-07-20T15:13:34Z

lib/livebook/notebook/cell.ex

+          w: non_neg_integer(),
+          h: non_neg_integer()


Is there even a need for resizing the output? For charts this would be really nice, but for this to happen we would need the ability to change the code cell source

As per the other comment, outputs generally use the available width. This is especially important for outputs that use iframes, because they live outside the main layout and we position them absolutely, so without specific width they would freak out :D

jannikbecher · 2023-07-20T15:26:56Z

assets/js/hooks/canvas.js

+      Array.from(this.grid.el.children).forEach((item) => {
+        console.log(item.attributes);
+        const output_id = `[id^=outputs-${item.attributes["gs-id"].value}]`;
+        const output_el = document.querySelector(output_id);
+        item.firstChild.appendChild(output_el);
+      });


Probably very bad idea to do it this way, but I couldn't come up with another solution...
I just remount the existing outputs into the gridstack items

didn't put more effort into it yet, because I'm tinkering around with implementing a liveview compatible drag&drop solution

jannikbecher · 2023-07-20T15:31:03Z

lib/livebook/notebook.ex

+        access_by_id(cell.id)
+      ],
+      fn cell ->
+        %{cell | output_location: nil}


maybe add another variable to the cell, so when moving the output back to the notebook, the location gets saved.
I think it is more expected behaviour when the output moves back to the same location

I think it's fine, moving output back and forth is not likely and when they remove an output it's likely that the user rearranges things to fill the empty space :)

jannikbecher · 2023-07-20T15:32:16Z

lib/livebook/notebook.ex

+        access_by_id(cell.id)
+      ],
+      fn cell ->
+        %{cell | output_location: %{x: 0, y: 0, w: 4, h: 2}}


Is there a way to figure out the size of an output?

Not really, most outputs don't have a specific size and just take whatever width they have, so we need to allocate a specific space.

jannikbecher · 2023-07-20T15:33:12Z

lib/livebook/notebook/cell/code.ex

@@ -10,6 +10,7 @@ defmodule Livebook.Notebook.Cell.Code do
    :id,
    :source,
    :outputs,
+    :output_location,


definitive needs some clearer naming

jannikbecher · 2023-07-20T15:37:39Z

lib/livebook_web/live/session_live/popout_window_live.ex

@@ -0,0 +1,141 @@
+defmodule LivebookWeb.SessionLive.PopoutWindowLive do


The popped out window creates a new connection to the server. Wouldn't it be better to fetch the data from the main window? Any tips to accomplish that?

This is totally fine, it's the same as two users collaborating and everything is built under this assumption anyway. Pretty sure syncing everything on the client wouldn't really be viable.

jannikbecher · 2023-07-20T15:38:12Z

lib/livebook_web/live/session_live/popout_window_live.ex

+     |> assign(
+       session: session,
+       # TODO
+       client_id: "",


What is the best way to fetch the client_id from the main window?

Each process needs to register as a separate client, so we need to register this LV and it will have its own client_id.

jonatanklosko · 2023-07-20T17:54:37Z

In the meantime I stumbled across @zachdaniel's tweet implementing a node-based editor in elixir. I could add some collision detection and a grid for snapping and than it could completely replace gridstack.js. WDYT?

Hey @jannikbecher thanks the updates! I talked with @josevalim and we agreed that we would get the most value if the canvas is a zoomable infinite, workspace like Count.co or Figma. I'm not yet sure if there's a library that would get us there, but that's the direction we would like to explore :)

jannikbecher · 2023-07-20T18:07:41Z

Okey perfect, I will look into it :)

josevalim · 2023-07-20T19:58:52Z

I talked with @josevalim and we agreed that we would get the most value if the canvas is a zoomable infinite, workspace like Count.co or Figma. I'm not yet sure if there's a library that would get us there, but that's the direction we would like to explore :)

Actually, let's hold on this for now.

Hi @jannikbecher, before we move forward, can ask you what was your original motivation for the feature? Because we have noticed we (members of the Livebook team) have two conflicting views:

An exploratory view, where we can place graphs, tables, and overall outputs. Here a canvas, potentially infinite, can be useful.
A way to better visualize and build apps.

Maybe those can be unified in a layer but perhaps not. We will discuss a bit more between us and follow-up. Sorry for the back and forth.

jannikbecher · 2023-07-21T06:02:45Z

Sorry for the back and forth.

Hey @josevalim,
no need for apologies. I'm always amazed by the quick and excellent feedback I get from you guys :)

before we move forward, can ask you what was your original motivation for the feature?

TL;DR
So my original motivation was a combination of the following:

I would like to see some low code/no code features in elixir for companies to empower the individual
- e.g. have the ability to freely design the app page
since I had a great time adding some small features to livebook I thought it would be time to tackle some more advanced problems :)

Maybe some background if anyone is interested :)
I worked as an ERP consultant and at some point I really got frustrated with the software. Odoo's vision is to provide a comprehensive suite of business applications that are open-source, highly customizable, and user-friendly, enabling small to medium-sized companies to manage all their operations from a single platform.
Though I liked this a lot, I came to the conclusion that it's almost impossible to have one software to manage all business processes and although Odoo comes really close to it, it fails in being stable in the long term and this is really frustrating for the end users.
While I was a passive consumer of the Elixir programming language for many years, I knew there are solutions for building scalable and bullet prove software and in the beginning of this year I started to dig a little deeper into livebook because I thought it has the potential to solve many problems in the business world. Especially with the introduction of livebook apps and smart cells I could see some use cases.
Following the principle of easy to use, but powerful when needed is exactly the kind of tool you want to empower individual users in a company without being dependent on IT professionals.
I would really like to see some api/endpoint functionality to write some "glue" code with livebook to connect different software systems and maybe gradually migrate these legacy software to elixir :)

But this is not a plan or anything I try to enforce by submitting PRs. I just had some ideas and much fun hacking around with it :)

An exploratory view, where we can place graphs, tables, and overall outputs. Here a canvas, potentially infinite, can be useful.

A way to better visualize and build apps.

So I definitely would like to see something like the second point in livebook, but since I really like to implemented stuff in elixir/livebook as the sake of it, I'm completely open with whatever direction this will go :)

josevalim · 2023-07-21T18:12:56Z

Hi @jannikbecher, we talked about this today and we have a couple updates. Basically, a grid system will not work for us because we do not know the heights of elements up-front. For example, a text element may take certain height in some screens but wrap in others.

That said, we need to go with a row-based design. You can think that each output goes to its own row:

|-------------|
|      1      |
|-------------|

|-------------|
|      2      |
|-------------|

|-------------|
|      3      |
|-------------|

However, I can also drag and drop 2 into 1 and split that row in two columns:

|------|------|
|   1  |   2  |
|------|------|

|-------------|
|      3      |
|-------------|

You should be able to control how much width each column has, but that doesn't need to be a concern for now. The important to keep in mind in that we don't know how much height a row takes. If 2 has fixed height and 1 is text, 2 may be a smaller than 1 in some screen and bigger in others! Once a row ends, the next row starts.

Unfortunately I think this approach pretty much neglects the grid design but it should hopefully be simpler in the end.

Some additional considerations:

The popped-in design should use an iframe, pointing to the same URL as the popped out version. This should simplify the implementation.
Once this feature is ready, we will remove "rich outputs only" from deployed apps, but don't worry about it for now, this is mostly a TODO for us later.

We also won't call it canvas, most likely something like "Output grid" or "Output workspace". Internally you can call it "Output grid", we will figure out the external name later. :)

josevalim · 2023-07-21T20:14:49Z

For the name, let's go with "Output panel", unless you have better suggestions. :)

jannikbecher · 2023-07-25T13:06:03Z

Closing in favor of #2109

jannikbecher added 7 commits July 7, 2023 13:35

add gridstack.js to create app dashboard pages

cd87376

save layout and show in deployed app

953dadb

WIP

c83c4e6

WIP

2526763

WIP

a10fbdd

WIP

e17bb5b

WIP

eeebf8d

jannikbecher added 2 commits July 11, 2023 15:20

Add new canvas view

a7a5cc6

Send canvas_settings update to canvas

5bf643e

jannikbecher added 4 commits July 11, 2023 18:24

Render iframes in canvas

1e92ccd

remove old deploy functionality

e3b3b2d

add pop out functionality

46ae045

formatting

408ba9a

WIP deploy app

6d55169

jonatanklosko reviewed Jul 11, 2023

View reviewed changes

Apply indicators suggestions

1489455

hide move to canvas button when output is already on canvas

e0473e6

Add move output to notebook button

3dc3635

hide canvas options in sub menu

8a3e33c

jannikbecher added 3 commits July 17, 2023 08:32

Fix click twice to enable canvas mode bug

91e956d

Use pubsub for window popin/popout communication

386f0ab

Since Broadcast Channel API is used for client PubSub, broadcast func…

1c5a61d

…tion is not possible anymore

jonatanklosko reviewed Jul 18, 2023

View reviewed changes

jannikbecher added 9 commits July 18, 2023 14:28

Revert "Since Broadcast Channel API is used for client PubSub, broadc…

9bbe1b2

…ast function is not possible anymore" This reverts commit 1c5a61d.

Revert "Use pubsub for window popin/popout communication"

bfe0ebe

This reverts commit 386f0ab.

Move eventlistener to gridstack hook

f0241db

Some refactoring

50bbf44

Add gridstack to popped out window

e106553

Add show canvas options to app settings

59a66f1

Add canvas deployment

23449fd

Remove canvas settings

0859c43

Get session updates in popout window

d30b795

Some cleanup

ff8b531

jannikbecher commented Jul 20, 2023

View reviewed changes

jannikbecher requested a review from jonatanklosko July 20, 2023 15:42

Add client_id to popout window & handle /popout-window navigation

992e9fd

Fix not rendering markdown

ade354f

jannikbecher closed this Jul 25, 2023

		@@ -0,0 +1,141 @@
		defmodule LivebookWeb.SessionLive.PopoutWindowLive do

Proposal for Deployed Dashboards Enhancement Using Drag & Drop Functionality #2052

Proposal for Deployed Dashboards Enhancement Using Drag & Drop Functionality #2052

Conversation

jannikbecher commented Jul 11, 2023 • edited Loading

josevalim commented Jul 11, 2023 • edited Loading

josevalim commented Jul 11, 2023 • edited Loading

hugobarauna commented Jul 11, 2023

jannikbecher commented Jul 11, 2023

jannikbecher commented Jul 11, 2023 • edited Loading

josevalim commented Jul 11, 2023

jonatanklosko commented Jul 11, 2023

jonatanklosko Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jannikbecher commented Jul 12, 2023

josevalim commented Jul 12, 2023 • edited Loading

josevalim commented Jul 12, 2023

josevalim commented Jul 12, 2023

jannikbecher commented Jul 18, 2023

josevalim commented Jul 18, 2023

jonatanklosko commented Jul 18, 2023

jonatanklosko Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

jannikbecher commented Jul 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonatanklosko commented Jul 20, 2023

jannikbecher commented Jul 20, 2023

josevalim commented Jul 20, 2023

jannikbecher commented Jul 21, 2023

josevalim commented Jul 21, 2023

josevalim commented Jul 21, 2023

jannikbecher commented Jul 25, 2023

jannikbecher commented Jul 11, 2023 •

edited

Loading

josevalim commented Jul 11, 2023 •

edited

Loading

josevalim commented Jul 11, 2023 •

edited

Loading

jannikbecher commented Jul 11, 2023 •

edited

Loading

jonatanklosko Jul 11, 2023 •

edited

Loading

josevalim commented Jul 12, 2023 •

edited

Loading

jonatanklosko Jul 18, 2023 •

edited

Loading