Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter Server Notes 2021 #4

Closed
Zsailer opened this issue Dec 17, 2020 · 47 comments
Closed

Jupyter Server Notes 2021 #4

Zsailer opened this issue Dec 17, 2020 · 47 comments

Comments

@Zsailer
Copy link
Member

Zsailer commented Dec 17, 2020 •
edited
Loading

Hello everyone,

Gitter

Welcome to the Jupyter Server Team!

You can find us in our Gitter chat room or visit us at our weekly meetings!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel. You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

Meeting Notes
@Zsailer Zsailer pinned this issue Dec 17, 2020
@Zsailer
Copy link
Member Author

Zsailer commented Jan 7, 2021

January 7th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Steven Silvester Apple @blink1073
Afshin T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Jan 15, 2021

January 14th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Jan 21, 2021 •
edited
Loading

January 21st, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Pierre-Olivier Simonard ioapps.io / Quansight @pierrotsmnrd
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf
Gonzalo Peña-Castellanos Quansight @goanpeca
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Jan 28, 2021 •
edited
Loading

January 28th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Afshin T. Darian Two Sigma @afshin
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf

Agenda + Minutes

  • Failing tests on Windows, Python 3.6

  • ExtensionApp not compatible with extending ServerApp?

    • Issue: ExtensionApp not compatible with extending ServerApp jupyter_server#395
    • Why doesn't ExtensionApp subclass ServerApp?
      • Jupyter Server is meant to be a headless server.
      • All extensions are equal in the eyes of Jupyter Server.
      • Today versus previously:
        • Previously, extensions (e.g. LabApp from JupyterLab) extended the server two different ways based on how it were launched:
          1. The extension module could include an extension application class (e.g. LabApp); it subclassed the server application. If the extension was launched through its entrypoint, this extension application class was used.
            • The result is something like, LabApp(ServerApp).
          2. Alteratively, if the server application was launched and the extension was enabled, the extension application was not used; rather, just the the handler appending and config loaded.
            • The result is something like ServerApp + "separate logic to appending handlers and load Lab specific config".
        • Today, extensions offers a single, consistent way to extend the server:
          1. Initialize the ServerApp, then initialize all ExtensionApps. Configure the server to decide which extension is the default.
            • The result is always, ServerApp + ExtensionApp(s)
            • ServerApp has an extension_manager; ExtensionApps have a serverapp property.
            • ExtensionApps do have a launch_instance method. This is simply a shortcut to automatically configure and initialize a ServerApp. This allows ExtensionApps to behave like a server application, but not drastically change how they extend the server.
      • In the previous approach, one extension was elevated above the rest—the one that subclasses the server application.
      • Today, all extensions are handled the same.
        • This makes it cleaner to run a single Jupyter Server with e.g. nbclassic, jupyterlab, voila, etc. side-by-side.
        • ExtensionApps are not, themselves, server applications.
          • Their traits and methods are self encapsulated in their own class
          • They should be able to affect the core server application and append handlers.
    • What's the best way to extend Jupyter Server? (e.g. like JupyterHub which spawns server subclasses with auth patched-in)?
      • Subclass ServerApp, not ExtensionApps.
      • To choose a specific frontend application as the default,
        1. enable the extension, e.g. put jpserver_extensions={"jupyterlab": True} somewhere.
        2. set the server applications default_url to the extension's endpoint, e.g. ServerApp.default_url="/lab"

  • Review Issues:

    • Jupyter Server repo name: Repository name jupyter_server#356
      • Discussed not to change the name so that it matches the Python package name.
    • Have web.authenticated be the default: Have handlers be @web.authenticated by default ? jupyter_server#389
      • This might be too invasive of a change, but the principal is sound in general.
      • Alternative approach:
        • Add method in ServerApp that checks whether handlers are authenticated.
        • Add trait (and flag) in ServerApp to warn, exclude, modify, or allow handlers that are not authenticated. The default is allow, to keep backwards compatibility.

@jasongrout jasongrout mentioned this issue Jan 28, 2021
Open
@Zsailer Zsailer mentioned this issue Jan 28, 2021
Closed
@kevin-bates
Copy link
Member

February 4th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Vidar Fauske JPMorgan @vidartf
Zach Sailer Apple @Zsailer
Steve Silvester Apple @blink1073
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Feb 12, 2021

February 11th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Pierre-Olivier Simonard Quansight / ioapps @pierrotsmnrd
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Feb 22, 2021 •
edited
Loading

February 18th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Zach Sailer Apple @Zsailer
Carol Willing Noteable willingc
Pierre-Olivier Simonard ioapps.io / Quansight @pierrotsmnrd
Steven Silvester Apple @blink1073
Sylvain Corlay QuantStack @SylvainCorlay
Eric Charles Datalayer / Quansight @echarles
Vidar T Fauske JP Morgan Chase @vidartf
Jason Grout Bloomberg @jasongrout
A. T. Darian Two Sigma @afshin
Jeremy Tuloup QuantStack @jtpio

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Mar 3, 2021

February 25th, 2021

Attendees

Name affiliation username
Vidar T Fauske JP Morgan Chase @vidartf
Zach Sailer Apple @Zsailer
Pierre-Olivier Simonard ioapps.io / Quansight @pierrotsmnrd
A. T. Darian Two Sigma @afshin
Mariko Wakabayashi Twitter @mwakaba2
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
Sylvain Corlay QuantStack @SylvainCorlay

Agenda + Minutes

  • Change translation function alias
  • Unable to open a kernel if HTTP PATCH request is blocked by proxy
    • Unable to open a kernel if HTTP PATCH request is blocked by proxy jupyter_server#430
    • Why was this not an issue in Jupyter Lab 2.x?
      • PATCH requests were still made when updating sessions in JupyterLab.
      • Let's track down this change in JLab and post it on the issue for reference.
    • Falling back to GET/POST requests is a major change to Jupyter REST API
      • It would require a JEP.
      • Clients that want to support this change also need to handle this.
      • Is this common enough problem to warrant such a change?
        • For now, it makes the most sense to wait and see if this issue appears repeatedly before launching such a wide scale change.
  • jupyter/telemetry: nested category filtering

@Zsailer
Copy link
Member Author

Zsailer commented Mar 4, 2021

March 4th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf
Jeremy Tuloup QuantStack @jtpio
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@blink1073
Copy link
Contributor

March 11th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
Afshin Darian Two Sigma @afshin

Agenda + Minutes

  • Daylight savings time. Which TZ is the time linked to? 😃
  • Updated/updating the notebook port list
  • Improved automated publishing and packaging work in progress
    • leverage Github Actions to:
      • generate a changelog
      • generate a (draft) Github release that contains the tar/wheel files
    • changes to jupyter-packaging
      • thin layer on top of setuptools
      • new build backend that adds a pre-build step to pyproject.toml to build jupyter-like packages, i.e. includes an NPM builder before building the python package.
      • can use entrypoints instead of data_files
        • data_files still work, but we'll drop support for using data_files in develop mode.
        • static JS assets found using entrypoints

@blink1073
Copy link
Contributor

March 18th, 2021

Attendess

Name affiliation username
Steven Silvester Apple @blink1073
Kevin Bates IBM @kevin-bates
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@blink1073 blink1073 mentioned this issue Mar 18, 2021
Open
@afshin afshin mentioned this issue Mar 18, 2021
Closed
@blink1073
Copy link
Contributor

April 1st, 2021

Attendess

Name affiliation username
Steven Silvester Apple @blink1073
Kevin Bates IBM @kevin-bates
A. T. Darian Two Sigma @afshin
Vidar T Fauske JP Morgan Chase @vidartf
Jeremy Tuloup QuantStack @jtpio
Sylvain Corlay QuantStack @SylvainCorlay

Agenda + Minutes

@blink1073
Copy link
Contributor

April 8th, 2021

Attendess

Name affiliation username
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
Vidar T. Fauske JP Morgan Chase @vidartf
David Brochart QuantStack @davidbrochart
Zach Sailer Apple @Zsailer

Agenda + Minutes

  • Status of authorization PR 165, rebased via PR 467?

    • David is going to update the original PR (165) with master and we'll try to keep this PR up to date
    • Clients/applications (e.g., RTC) can then use this branch so we can get a feel for what additions are necessary

  • 1.6.0 released - thanks Steve! Changelog produced using https://github.com/executablebooks/github-activity

  • Server event notification - Get notify on server event jupyter_server#466

    • Server-side event publisher with JSON schema backed events
    • Possible events include kernelspec added/modified, file modified, kernel started/stopped, really any REST call other than GET. Could also add file system polling for contents events
    • For the "server shutdown event", we would have a contract that consumers of the event should handle it synchronously
    • Possible overlap with telemetry effort, might be efficient to share logic/infrastructure
    • Short term solutions are to merge telemetry as a private implementation and use it for this emitter, or to create a trait on ServerApp that is the "status", and could be observed using trailets observer.

@kevin-bates
Copy link
Member

April 15th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
Vidar T. Fauske JP Morgan Chase @vidartf
A. T. Darian Two Sigma @afshin

Agenda + Minutes

@blink1073
Copy link
Contributor

April 22nd, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf
Steven Silvester Apple @blink1073
A. T. Darian Two Sigma @afshin
Tony Fast Quansight @tonyfast

Agenda + Minutes

  • Release 1.6.4 is out - thanks Steve!

  • Refactor gateway kernel management (#483)

    • Compatible with previous behavior
    • Confirmed Papermill-level functionality using Elyra
    • Kevin will rename GatewayKernelManagers to GatewayMappingKernelManager - preserving the current terminology

  • Anyio Version 3 is affecting AsyncContentsManagement (#487)

    • Capped in last release

  • Reading/writing large notebooks

@Zsailer
Copy link
Member Author

Zsailer commented Apr 30, 2021

April 29th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf
Steven Silvester Apple @blink1073
Kien Dang National University of Singapore @kiendang
A. T. Darian Two Sigma @afshin
Zach Sailer Apple @Zsailer

Agenda + Minutes

This was referenced May 3, 2021
Closed
Open
Open
@blink1073
Copy link
Contributor

May 6th, 2021

Attendees

Name affiliation username
Max Klein JP Morgan Chase @telamonian
Steven Silvester Apple @blink1073
Vidar T Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Mariko Wakabayashi Twitter @mwakaba2
Kien Dang NUS @kiendang

Agenda + Minutes

  • Jupyter Releaser is released and we've used it to make two prereleases of server
  • Terminal testing on Windows
    • Notebook tests are failing
    • Server tests are skipped
    • For terminado we only run the basic test
    • jupyter_client has no terminal logic, but it does have Windows-specific logic we can emulate for shutting down processes.
  • Update on "Anyio version v3.0.0 breaks AsyncContentsManager" (#487)
    • Fix: Upgraded anyio to version v3.0.1 for python3.7 and higher. (#492)
    • New version is not working for python 3.6. The anyio team is looking into a fix for python 3.6. (#505)
  • Download folder as archive issue #68. Previously discussed in #251/#252.
  • Institute a process for adopting third party extensions similar to JupyterLab?
    • We will add language to both our team compass and JuptyerLab team compass to point to jupyterlab-contrib
  • Regarding issue #496, Kevin will bump the dependency of jupyter_core to include the appropriate lower version

@blink1073 blink1073 mentioned this issue May 10, 2021
Merged
@blink1073
Copy link
Contributor

Jupyter Server Weekly Meeting

  • Where: jovyan Zoom (pwd: c0JZTHlNdS9Sek9vdzR3aTJ4SzFTQT09)
  • Historical notes can be found here.

May 13th, 2021

Attendees

Name affiliation username
Steven Silvester Apple @blink1073
Vidar Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Kien Dang NUS @kiendang

Agenda + Minutes

  • Added branch protection to multiple repos
  • 1.7.0 released this week. There was one bug in the release process that was fixed in jupyter_releaser.

@kevin-bates kevin-bates mentioned this issue May 18, 2021
Open
@Zsailer
Copy link
Member Author

Zsailer commented May 27, 2021

May 27th, 2021

Attendees

Name affiliation username
Nick Bollweg Georgia Tech, deathbeds @bollwyvl @nrbgt
Kevin Bates IBM @kevin-bates
Vidar Fauske JP Morgan Chase @vidartf
Zach Sailer Apple @Zsailer
Ryan C Cooper UConn @cooperrc
Mariko Wakabayashi Twitter @mwakaba2
Pierre-Olivier Simonard Quansight @pierrotsmnrd
Afshin T. Darian Two Sigma @afshin

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Jun 3, 2021

June 3rd, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Afshin T. Darian Two Sigma @afshin
Tony Fast Quansight @tonyfast
Pierre-Olivier Simonard Quansight @pierrotsmnrd
Max Klein JP Morgan Chase @telamonian

Agenda + Minutes

  • Running tasks in Jupyter Server's ioloop at start-up.

  • Python namespace packages with extensions

    • extensions: allow extensions in namespace packages jupyter_server#523
    • Checking if __file__ is in the package root might not be the most robust way to identify namespace packages.
    • Tony will post some ideas on using importlib versus __import__
      • one might be able to inspect the importlib object to see which loader (i.e. NamespaceLoader) was used to load the package.

  • Access files outside of notebook-dir (root-dir)

    • Access files outside of notebook-dir jupyter_server#534
    • Approach:
      • Add a new trait to represent the "preferred" or "starting" directory (e.g., preferred_dir)
      • preferred_dir
        • will default to root_dir (aka notebook_dir)
        • is required to be an existing directory residing at or within root_dir
        • is a "suggestion" to client applications. Client applications can honor preferred_dir and start their file-browser at preferred_dir instead of root_dir.

  • Support various config in jupyter_server_config.d/*.json

    • eg: purge filetree, clean up a few misc items jpmorganchase/jupyter-fs#108 (comment)

      {
  "ServerApp": {
    "contents_manager_class": "jupyterfs.metamanager.MetaManager",
    "jpserver_extensions": {
      "jupyterfs.extension": true
    }
  }
}

    • Talked about previously, but unresolved in issue #207

    • Jupyter Server only looks for jpserver_extensions trait in the static config dir (conf.d), since this dir was originally intended for discovering extensions.

    • Open question: should we allow extensions to affect other server traits in this static config?

      • How do we handle conflict? —just load alphabetically and raise a warning in --debug.
      • User config supersedes static config, so users can manually override anything set here if there's conflict.

    • Let's continue the conversation in #207.

@Zsailer
Copy link
Member Author

Zsailer commented Aug 12, 2021

August 5th, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
Vidar Fauske JPMC @vidartf
A. T. Darian Two Sigma @afshin
Eric Charles Datalayer @echarles
Jessica Xu Quansight @jess-x
Zach Sailer Apple @Zsailer

Agenda + Minutes

  • Look and consider:

  • Next steps for Parameterized Kernels?

    • Let's use a 15-minutes chunk of Jupyter Server meetings each week to discuss this topic.
    • Can hopefully reuse the settings editor for Lab for UI for editing parameters (JSON schema -> form)

  • Add a Security Policy for Server?

    • Template includes supported versions and how to handle a vulnerability report - we could link to our security docs on RTD for now until jupyter.org/security is available

@Zsailer
Copy link
Member Author

Zsailer commented Aug 12, 2021 •
edited
Loading

August 12th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
A. T. Darian Two Sigma @afshin
Steven Silvester Apple @blink1073
Rollin Thomas NERSC @rcthomas
Jessica Xu Quansight @jess-x
Oliver Sanders Met Office @oliver-sanders
Vidar T Fauske JPMorgan Chase @vidartf

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Aug 19, 2021

August 19th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
A. T. Darian Two Sigma @afshin
Steven Silvester Apple @blink1073
Vidar T Fauske JPMorgan Chase @vidartf
Eric Charles Datalayer @echarles
Max Klein JPM @telamonian
Jessica Xu Quansight @jess-x

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Aug 27, 2021

August 26th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
A. T. Darian Two Sigma @afshin
Max Klein JPM @telamonian
Karla Spuldaro IBM @karlaspuldaro

Agenda + Minutes

  • Issues to review:
  • Jupyter Resource Usage
    • discussed roadmap/potential requirements last week
    • extension demo next week
    • requirements in-progress
  • Long term, re-occuring topics:
    • There a few larger efforts on the horizon for Jupyter Server.
    • We should have a place where we post documentation about these efforts (separate repo?).
    • Multi-user server
      • identity, storage (DB), auth ...
    • Kernel parameterization

@kevin-bates
Copy link
Member

kevin-bates commented Sep 2, 2021 •
edited
Loading

September 2nd, 2021

Attendees

Name affiliation username
Kevin Bates IBM @kevin-bates
Vidar T Fauske JP Morgan Chase @vidartf
Karla Spuldaro IBM @karlaspuldaro
Jessica Xu Quansight @jess-x
A. T. Darian Two Sigma @afshin
Max Klein JPM @telamonian
Mariko Wakabayashi Twitter @mwakaba2
Thi Nguyen Stealth-mode startup @duongnt

Agenda + Minutes

  • Darian: What does a Jupyter server data layer look like?
    • By "data layer", we are talking about things like user settings, workspaces, event logs, etc. that would be a natural fit for a SQL database
    • Discusssion Notes
      • Use cases
        • Multi-user environment where users want to track own settings.
        • extensions like jupyterlab-notifications to store user specific notification data
      • Possible storage solutions: Key Value Store vs Sqlite vs SQL
      • Prevent X individual extensions to connect to X different databases to store configuration etc.
      • Use standard sqlite version?
      • Define some kind of standard on the Jupyter level for the database schema to ensure the tables can be joined with Jupyter server maintained tables (e.g. user, logging, events, metrics)
      • Who would work on this? (e.g. jupyterlab, jupyter_server)
    • Conclusion (for now) We should first pursue a storage solution for the known data that we plan to offer in a multi-user server:
      • user-specific settings
      • user-specific workspaces
      • event log
      • real-time collaboration CRDT store

@Zsailer
Copy link
Member Author

Zsailer commented Sep 9, 2021

September 9th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Luciano Resende IBM @lresende
Thi Nguyen Startup @duongnt
David Brochart QuantStack @davidbrochart
Karla Spuldaro IBM @karlaspuldaro
Adrien Delsalle QuantStack @adriendelsalle
Kevin Bates IBM @kevin-bates
Steven Silvester Apple @blink1073
A. T. Darian Two Sigma @afshin

Agenda + Minutes

  • New Release: v1.11.0 (thanks, Steve!)
  • Proposal: Towards a new generic and composable server
    • Towards a new generic and composable server #11
    • Should the following repos move into the Jupyter Server organization?
    • Technology stack overview:
      • Pydantic and Typer:
        • Embrace Python's native type hints/annotation.
        • Together, they remove most of the need for Traitlets
        • Handles most data validation automatically, and offers a simple way to define custom types.
        • Typer offers a simple API for adding command-line interfaces to applications for configuring Python objects.
        • Can we drop traitlets? What's left in Traitlets?
          • “Observe” other traits. Necessary for ipywidgets.
          • loading configuration from file.
            • Comparatively, this is a small amount of code to maintain.
            • We can likely contribute these things upstream too.
      • FastAPI
        • Built on top of Pydantic.
        • Native syntax for type annotations/hints
        • Auto generate (developer) documentation and unit-tests (via hypothesis) for free.
        • ASGI implementation, i.e. async by default.
      • Pluggy
        • Provides a significantly more robust mechanism for discovering and loading application extensions.
        • As many people have experienced, today’s Jupyter Server's extension discovery/loading is quite brittle.
    • Questions:
      • Does this change the kernel messagine spec? Will this invalidate current kernels?
        • No.
      • Will server extensions have to migrate?
        • Yes.
        • Is it possible to somehow prevent or minimize this in the future?
          • Today's server extension API isn't (and was never) clearly specified. Pluggy defines a clear specification for plugin systems. It's more stable and less likely to change in the future.
      • Will this change the current REST API?
        • It's not a immediately clear. Right now, no. It's possible this new server implementation would expose needed changes in the current REST API, but that would require a JEP.
      • Does this need to wait for classic Notebook to retire?
        • This is an issue to be addressed by the new software steering council. Yes, this should coincide the the retirement of the classic notebook server.

@jtpio jtpio mentioned this issue Sep 10, 2021
Open
@Zsailer
Copy link
Member Author

Zsailer commented Sep 16, 2021

September 16th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Adrien Delsalle QuantStack @adriendelsalle
Raman Tehlan Raman Tehlan @ramantehlan
Sylvain Corlay QuantStack @SylvainCorlay
David Brochart QuantStack @davidbrochart
Vidar T Fauske JP Morgan Chase @vidartf
Thi Nguyen Startup @duongnt
Karla Spuldaro IBM @karlaspuldaro
Jeremy Tuloup QuantStack @jtpio
Steven Silvester
Vidar Tonas Fauske
Adam Patterson
Corey Smith
Kevin Bates IBM @kevin-bates

Agenda + Minutes

  • [VOTE] Move the following repos into the Jupyter Server organization:
    • Repos to move:
    • This simply makes it easier to collaborate on these projects while they are experimental and being developed.
    • It does not mean that the Jupyter Server team is adopting the proposed server, yet. We'll have another vote at a later time for the adoption of this work.
    • If so, we'll need to have clear messaging that says these projects are experimental.
    • [APPROVED] David and Adrien will move these repos to Jupyter Server org sometime over the next couple of weeks.
  • Thought about the backend extension system brought up...
    • Enable each server extension runs its own server and exposes a port.
    • connect to that extension's port using jupyter-server-proxy.
    • This works for simpler server extensions that aren't coupled to one another or don't use any of the managers, but more complex extensions that need access to the core managers can't use this approach.
    • A major thing to keep in mind— auth, loginhandler, etc. needs to be feature complete. We don't want to make multiple requests to the auth provider per request.

@kevin-bates
Copy link
Member

September 16th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Adrien Delsalle QuantStack @adriendelsalle
Raman Tehlan Raman Tehlan @ramantehlan
Sylvain Corlay QuantStack @SylvainCorlay
David Brochart QuantStack @davidbrochart
Vidar T Fauske JP Morgan Chase @vidartf
Thi Nguyen Startup @duongnt
Karla Spuldaro IBM @karlaspuldaro
Jeremy Tuloup QuantStack @jtpio
Steven Silvester
Vidar Tonas Fauske
Adam Patterson
Corey Smith
Kevin Bates IBM @kevin-bates

Agenda + Minutes

  • [VOTE] Move the following repos into the Jupyter Server organization:
    • Repos to move:
    • This simply makes it easier to collaborate on these projects while they are experimental and being developed.
    • It does not mean that the Jupyter Server team is adopting the proposed server, yet. We'll have another vote at a later time for the adoption of this work.
    • If so, we'll need to have clear messaging that says these projects are experimental.
  • Thought about the backend extension system brought up...
    • Enable each server extension runs its own server and exposes a port.
    • connect to that extension's port using jupyter-server-proxy.
    • This works for simpler server extensions that aren't coupled
    • A major thing to keep in mind— auth, loginhandler, etc. needs to be feature complete. We don't want to make multiple requests to the auth provider per request.

@Zsailer
Copy link
Member Author

Zsailer commented Sep 23, 2021 •
edited
Loading

September 23th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Karla Spuldaro IBM @karlaspuldaro
Jeremy Tuloup QuantStack @jtpio
Kevin Bates IBM @kevin-bates
Adrien Delsalle QuantStack @adriendelsalle

Agenda + Minutes

  • No planned discussion in the notes.
  • Open discussion around Jupyverse
    • Can Jupyverse work with Jupyter Server's managers?
      • It would make it easier for people to transition to this new server if the old managers worked.
        • The benefit of the old managers is that you can subclass those classes, borrowing most of their logic and only replacing the things you need.
        • Multi-managers are critical. We might need multimanagers in every service, i.e. contents, kernels etc. They handle logic that applies to all single managers, e.g. kernel culling.
      • This would be a good opportunity to refactor the managers and possibly replace traitlets for something like Pydantic and fps.config.
    • How do we handle plugins where we don't want to include all the routes?
      • Today, we can loop through the Tornado handlers and remove (via monkeypatch) handlers that we don't want.
      • In JupyterLab plugins, you can disable portions of a plugin we don't want to used.
      • In Jupyverse, we can adopt a similar model. Add a way to disable parts of a plugin.
      • Jupyverse will not allow two plugins to have the same route (today, this would silently happen in tornado). If they do, you must configure your plugins to disable one of the duplicate plugin routes.
    • How will Jupyverse handle services that are coupled, like today's sessions service in Jupyter Server?
      • Plugins can depend on other plugins in jupyverse/fps via a dependency injection system.
    • Identity is already being handled in jupyverse (although, it's in a very early stage). See this new endpoint for listing users. Already moving in the direction of multiuser server!

@jtpio jtpio mentioned this issue Sep 24, 2021
Closed
@Zsailer
Copy link
Member Author

Zsailer commented Sep 30, 2021 •
edited
Loading

September 30th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Rick Wagner UCSD @rpwagner
Adrien Delsalle QuantStack @adriendelsalle
David Brochart QuantStack @davidbrochart
Karla Spuldaro IBM @karlaspuldaro
Kevin Bates IBM @kevin-bates
Sylvain Corlay QuantStack @SylvainCorlay

Agenda + Minutes

  • Jupyter security-related documentation survey by Trusted CI @rpwagner
    • Please review and add comments.
    • This document is helpful in identifying gaps in the Jupyter Server docs.
    • Thank you, Kay Avila, for compiling this survey!
  • FPS demo, shown by @adriendelsalle (Fast Pluggable Server)
    • Showed the "Hello World" plugin
    • Showed hooks for registering custom exceptions, configuration, and server routes.
    • Showed autodocumentation generated by FastAPI.
    • (Briefly) showed a full Jupyverse server built on FPS.
  • Jupyverse conversation
    • This is a good time to start trying out the new server.
    • There are questions that need answering soon around auth
      • Can we make an opinionated decision and choose OIDC as the only supported auth protocol? This enables us make opinionated decisions moving forward for RTC, etc.

@kevin-bates
Copy link
Member

October 7th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
Mariko Wakabayashi Twitter @mwakaba2
Vidar Tonaas Fauske JP Morgan chase @vidartf
David Brochart QuantStack @davidbrochart

Agenda + Minutes

  • jp_serverapp pytest fixture clears all logging handlers jupyter_server#585
    • The jp_serverapp fixture is clearing all logging handlers on the main logger
    • This prevents us from testing the output from the default logger
    • The default logging handler is a StreamHandler that points at stdout/stderr. This gets cleared by pytest.
    • add a pytest fixture for capturing logging stream jupyter_server#588
      • Reroutes the logs to a different stream and prints the results at the end of the test.
  • Issue in nbclassic that's breaking nbextensions_configurator
    • The order of the handlers is different between classic notebook and nbclassic
    • To address this, we'll have to inject the handlers in the right place.
  • Mariko: my team at Twitter is working on a blog post to share how Twitter uses Jupyterlab internally - will share blog post link when that gets published (around 10/30th)

@Zsailer
Copy link
Member Author

Zsailer commented Oct 14, 2021

October 14th, 2021

Attendees

Name affiliation username
Vidar Tonaas Fauske JP Morgan chase @vidartf
Steve Silvester Apple @blink1073
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
Carlos Herrero QuantStack @hbcarlos
Jeremy Tuloup QuantStack @jtpio

Agenda + Minutes

  • Slow Starting Kernels Proposal (We'll convert this to an issue for further discussion)

    • Problem: Jupyter Notebook was originally built with the assumption that kernels would start quickly. This turns out
      to not be true for some local kernels and most remote kernels.

    • We previously proposed changing the REST API to reflect
      kernels/sessions that were "pending". The downside to a REST API change is that the server would need to advertise capability through a versioned API or some other status, and clients would need to be updated to accommodate the changes.

    • An alternative method is to leave the current REST APIs intact and instead introduce the concept of a "pending" kernel that
      acts like a regular kernel from the client's perspective.

    • A POST to /api/sessions or /api/kernels would create a "pending" kernel and return immediately before starting the kernel.

    • It remains to be seen during implementation what changes need to be made to handlers and managers, but at the very least we will use a scheduled callback to actually start the kernels when we are handling the POST.
      The MappingKernelManager will also need to be updated to handle pending kernels internally in its public methods.

    • We should use the kernel manager to get the kernel id

    • We need to think about how kernel failure to start is handled for the user. Previously, it could be given to the user in the response to a POST

    • We might even be able to add the pending logic to the handlers without needing to affect the managers (e.g by calling save_state on the managers directly)

@Zsailer
Copy link
Member Author

Zsailer commented Oct 28, 2021

October 21st, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
Jessica Xu Quansight @jess-x

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Oct 28, 2021 •
edited
Loading

October 28th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Adrien Delsalle QuantStack @adriendelsalle
Steve Silvester Apple @blink1073
Jason Weill AWS @jweill-aws
Karla Spuldaro IBM @karlaspuldaro
Piyush Jain AWS @3coins
Jessica Xu Quansight @jess-x

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Nov 4, 2021

November 4, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Karla Spuldaro IBM @karlaspuldaro
Jason Weill AWS @jweill-aws
Kevin Bates IBM @kevin-bates
Mariko Wakabayashi Twitter @mwakaba2
Steven Silvester Apple @blink1073
Jeremy Tuloup QuantStack @jtpio
Jessica Xu Quansight @jess-x

Agenda + Minutes

  • Jupyter Server v1.11.2 released this week.
  • Logging: all logs should use the same format jupyter_server#605
    • Assigned!
  • Pending Kernels
    • All three PRs are ready, but we can't really try it out until all three are merged and available. Question about whether the JupyterLab PR can be released as a patch release (also raised the question yesterday at the Lab meeting).
    • We decided that the JupyterLab PR could be merged and released in an alpha so we can test it and consider whether to backport it.
    • We also questioned what the behavior would be in nbclassic with pending kernels. After some testing, the behavior with a pending kernel is not as seamless as a non-pending kernel in terms of the UI interactions, but all of the lifecycle methods work: starting, changing kernels, restarting, shutting down.
  • [Discussion] Telemetry in server
    • Doing some initial explorations with auto-instrumenting Python classes in jupyter_telemetry (PR posted soon).
    • Goals for jupyter-server telemetry:
      • Straight forward configuration
      • Uninvasive in the source code
      • Well defined JSON schemas
      • Opt-in
    • What's the right level of granularity?
      • Schemas should be defined at the (Python) class level
        • This makes telemetry configuration straight-forward.
        • "Turn on" telemetry at each class level.
      • Individual events are collected at the function/method level.
    • Current progress:
      • record decorator that wraps a method and emits: 
        {
    "method": ...,
    "args": ...,
    "kwargs": ...,
    "returned": ...,
    "time": ...
}
      • Instrument metaclass in Python
        • Decorate (listed) methods with a record wrapper than emits an event.
        • Comment: metaclasses can cause all kinds of issues, particularly with ABCs. Verify this won't break things for traitlets objects, etc.
      • Instrument the APIHandler in jupyter_server
        • log all get, post, patch, put, and delete methods.
        • Defined JSON schemas for each service.
    • Expect to see PRs (in jupyter_server, and jupyter_telemetry) by next week.

@Zsailer
Copy link
Member Author

Zsailer commented Nov 11, 2021 •
edited
Loading

November 11th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Jason Weill AWS @jweill-aws
Vidar T Fauske JP Morgan Chase @vidartf
Kevin Bates IBM @kevin-bates
Piyush Jain AWS @3coins

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Nov 18, 2021

November 18th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Kevin Bates IBM @kevin-bates
Piyush Jain AWS @3coins
Karla Spuldaro IBM @karlaspuldaro
Mariko Wakabayashi OpenZeppelin @mwakaba2
Jason Weill AWS @jweill-aws
Rick Wagner UCSD @rpwagner
Steven Silvester N/A @blink1073
Sylvain Corlay QuantStacck @SylvainCorlay

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Dec 3, 2021

December 2nd, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
David Brochart Quantstack @davidbrochart
Steve Silvester MongoDB @blink1073

Agenda + Minutes

  • Save document through Yjs websocket jupyter_server#624
    • If y-py is found at runtime, pull contents from the server-side (y-py) document model that's syncing (i.e. as a peer) of the Y.js patch server on "save".
    • Question: what does "save" mean when RTC is enabled?
      • In Google Docs, there is not "save" feature. The document is automatically saved on intervals. The RTC is always keeping the latest version of the document
      • Checkpoints become useful. They are previous versions of the document.
      • The "save" API should probably pass-through in an RTC context.
    • Alternative to this PR—create a custom ContentsManager in JupyterLab server that is applied when collaborative=True.

@Zsailer
Copy link
Member Author

Zsailer commented Dec 9, 2021

December 9th, 2021

Attendees

Name affiliation username
Vidar T Fauske JP Morgan Chase @vidartf
Jason Weill AWS @jweill-aws
Brian Granger AWS @ellisonbg
Rahul Goyal AWS @rahul26goyal
Zach Sailer Apple @Zsailer
Mariko Wakabayashi OpenZeppelin @mwakaba2
Karla Spuldaro IBM @karlaspuldaro
Kevin Bates IBM @kevin-bates
Carlos Herrero QuantStack @hbcarlos

Agenda + Minutes

  • Memory leak detected with "remote" kernels
    • Jupyter Server solution for this should take pending kernels into account.
  • Terminal Idle State PR
    • Should this be in a contrib repo? (we don't currently have one). It's marked as "experimental".
    • Maybe it's time to make terminals into an extension.
      • Move termainls into a separate jupyter server extension repository.
      • This extension can, then, be disabled. There are certainly contexts where people don't want to allow terminals (the authorization conversations below is one instance).
      • Make Jupyter Server require this as a dependency temporarily.
      • In a future release, we can drop it as a dependency.
  • Min is picking up the Authorization layer work.
    • Security concerns for users
      • Many people deploying Server don't fully understand all the pieces.
      • We don't want to offer a system that could cause deployers to unknowingly deploy an insecure system.
      • We need to be sure to document what "execute" permissions means.
      • Once you have arbitrary code execution, you can change permissions on the server.
      • "execute" on one document essentially means you have "rwx" on all documents (i.e. the whole server) in a server with local kernels.
      • Ensure that users of Jupyter Server understand the risk of enabling "execute".
    • Add the ability to set permissions on individual documents.

@Zsailer Zsailer mentioned this issue Dec 9, 2021
Merged
2 tasks
This was referenced Dec 13, 2021
Closed
Closed
@Zsailer
Copy link
Member Author

Zsailer commented Jan 6, 2022

December 16th, 2021

Attendees

Name affiliation username
Zach Sailer Apple @Zsailer
Jason Weill AWS @jweill-aws
Rahul Goyal AWS @rahul26goyal
Vidar Fauske JP Morgan Chase @vidartf
Carlos Herrero QuantStack @hbcarlos
Craig Citro Google @craigcitro
Mariko Wakabayashi OpenZeppelin @mwakaba2

Agenda + Minutes

@Zsailer
Copy link
Member Author

Zsailer commented Jan 6, 2022

It's time to close out this thread for a new year. Congratulations, all, on a fantastic year around the Jupyter-Server project! 🎉

Thank you to everyone who participated in these meetings. I'm grateful for this awesome community and look forward to another great year ahead. ❤️

@Zsailer Zsailer closed this as completed Jan 6, 2022
@Zsailer Zsailer unpinned this issue Jan 6, 2022
@kevin-bates kevin-bates mentioned this issue Feb 1, 2023
Closed
@Zsailer Zsailer mentioned this issue Jan 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@blink1073 @Zsailer @kevin-bates