[Telemetry] Add basic telemetry to Jupyter server and begin emitting server events #364

kiendang · 2020-12-17T04:34:46Z

original pr #233

This is for emitting server events. For client events using the eventlog endpoint see #501.

Bump telemetry extension commit as well

Experiments here informed the schema naming recommendations in jupyter/telemetry#11

python 3.5 is still supported

This lets us add detailed documentation & description to our schemas, which is very hard to do in JSON. We also add a lot of documentation to the one JSON schema we have

Primary advantage over JSON is that we can do multi-line strings for more detailed documentation. We also expect humans to read & write these, so YAML is a much better format there. All JSON is also valid YAML, so that helps. Depends on jupyter/telemetry#13

We made a v0.0.1 release!

Solution came from https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html

robertpyke · 2021-02-18T06:35:52Z

Hi all, I was just curious if this is planned for a specific release / date? I noticed a few issues have been replaced by this one.

We are investigating adding telemetry event logging for certain actions, and it looks like this work covers use cases we would want to cover.

We're still planning how to approach our work, but we use the zero-to-jupyterhub-k8s for a deployment on AWS. We planned to do something to the effect of configure the event handler (like shown here: https://jupyter-telemetry.readthedocs.io/en/latest/pages/configure.html )

c.EventLog.handlers = [handler]

to use a CloudWatch or S3 EventLogger (as we're in AWS). e.g. https://pypi.org/project/cloudwatch/ (cloudwatch log handler).

Then we would import https://github.com/jupyterlab/jupyterlab-telemetry , and start injecting our events via our own extension (by subscribing to "signals", or customizing code as necessary).

Is that approach aligned with the work you're doing here? I assume we'd just start getting the additional events you're adding, once this code launches.

kiendang · 2021-02-18T10:27:23Z

Hi there's currently no specific release date for this but this is something that we've been actively working on and trying to get done as soon as we can. Could you elaborate more about your use cases? What events are you looking to record? These could potentially help us make better informed design decisions. Thanks!

tmlohman · 2021-02-18T19:17:06Z

Hello! I work with @robertpyke and can speak to the details of this project. We are in the early stages and still nailing down specific requirements. But I can give a bit of context. We have a large user base on JupyterHub and are trying to get a better idea of usage patterns to inform our development. For instance, we'd like to know when people are using R kernel vs Python. When we have a better list I'll let you guys know.

kiendang · 2021-02-19T07:30:47Z

Our plan is to have telemetry events for Jupyterhub and Jupyter server REST APIs. Events for Jupyter server would be tracked under this PR while those for Jupyterhub would be under jupyterhub/jupyterhub#3218.

For instance, we'd like to know when people are using R kernel vs Python.

Starting a kernel would be one of those events being recorded, along with information on the user and kernel. That should work for your example use case?

When we have a better list I'll let you guys know.

Great!

Currently we are working on the telemetry backend itself. Once that is settled we will come back to adding telemetry events under these two aforementioned PRs.

yuvipanda

Mostly LGTM, but then I wrote some good chunk of it :D Minor comments, but I'd also love to get this merged.

yuvipanda · 2021-03-11T10:44:33Z

jupyter_server/serverapp.py

        self.init_server_extensions()
        self.init_configurables()
        self.init_components()
+        self.init_eventlog()


This is being called twice.

yuvipanda · 2021-03-11T10:47:52Z

docs/source/eventlog.rst

@@ -0,0 +1,61 @@
+Eventlogging and Telemetry


Let's add a security note about not relying on this for security or auditing purposes, unless the user can categorically not mess with the python environment this is running in? Otherwise you can just change the code to emit whatever you want. Likewise, user code can just hit the eventlogging endpoint to add any spurious events it wants.

Almost everyone I talk to wants to use this for security, and this caveat is very important.

yuvipanda · 2021-03-11T10:49:26Z

jupyter_server/services/eventlog/handlers.py

+
+        # Profile, may need to move to a background thread if this is problematic
+        try: 
+            self.eventlog.record_event(schema_name, version, event)


This requires the schema to be registered serverside before clients can post to it. How do we do that? Perhaps a traitlet, with corresponding entrypoint based loading as well?

Ok let me create a separate PR for this.

Pinged you on gitter about this but Imma just copy paste here.

This means something like a jupyter telemetry [register|remove|list] <schema> command?
register would put the schema file in a event-schemas directory. The directory structure would be something like event-schemas/<name>/version/<schema>.[json|yaml]

That sounds great! I'm thinking that most things that want to submit events will be classic notebook extensions or jupyterlab extensions. These will be python packages, so perhaps we can begin by just allowing entrypoints? You call it, and it returns a bunch of schemas to register.

Ah that makes more sense. I was thinking of another type of entrypoint. I'll make a separate PR on this one.

yuvipanda · 2021-03-11T10:52:02Z

jupyter_server/event-schemas/contentsmanager-actions/v1.yaml

+        6. delete
+           Delete a file or empty directory at given path
+  path:
+    category: personally-identifiable-information


We decided to use URIs for category filtering, and there are a few well-known ones defined in https://jupyter-telemetry.readthedocs.io/en/latest/pages/schemas.html#property-categories.

This requires changes in jupyter/telemetry master branch that haven't been made a released yet. Should we make a new release for telemetry?

Yeah, making a new release there seems appropriate. Would you be able to do that?

Nope I don't have the permission.

I have updated this one and use the latest commit on jupyter/telemetry master for now, will change it once we have a new release.

Also should the unrestricted category be URI or just a generic one? Seems like we're currently using a universal unrestricted category
https://github.com/jupyter/telemetry/blob/d2c19ffa03fde7d1903759de4a265ef711b333ef/jupyter_telemetry/eventlog.py#L165-L176

While we're on the topic of URI may I have your thoughts on jupyter/telemetry#56? Thanks!

Zsailer · 2021-04-27T20:09:07Z

Thanks, @kiendang. I've added this to the Jupyter Server Team Meeting agenda for this week.

In the meantime, I'm going move this PR to "draft" state until we merge and release jupyter/telemetry#60.

Move to a separate branch

instead of raw paths since raw paths do not work if e.g. install package from a zip file https://importlib-resources.readthedocs.io/en/latest/using.html#example

docs/source/operators/telemetry.rst

kiendang · 2021-05-03T16:35:24Z

related jupyter/telemetry#61

mlucool · 2021-11-30T18:50:06Z

Is this still being worked on?

kiendang · 2022-03-24T14:30:10Z

@Zsailer sorry for not working on this for quite a while. I have been busy for the last year. I do have some free time now and would like to resume working on this and getting this merged.

The original todo list was

submit a PR for Handle category-filtered fields jupyter/telemetry#61
cut a new release for telemetry
merge this

Does that plan still look valid? anything new/changed regarding telemetry?

Zsailer · 2022-08-29T19:21:32Z

This work has been superceded by #954 and #862.

Closing this PR in favor of the others.

yuvipanda and others added 20 commits May 19, 2020 14:42

Add initial eventlog hook

344bb57

Install jupyter_telemetry from source

a0f40ea

Set up an eventlog API endpoint

96bf2f0

Bump telemetry extension commit as well

Use different naming convention & add test for it

06b91e0

Experiments here informed the schema naming recommendations in jupyter/telemetry#11

Don't use f-strings

716ff1b

python 3.5 is still supported

Derive JSON Schema files from YAML files

8e122fc

This lets us add detailed documentation & description to our schemas, which is very hard to do in JSON. We also add a lot of documentation to the one JSON schema we have

Keep event schemas in YAML

f9a0dfb

Primary advantage over JSON is that we can do multi-line strings for more detailed documentation. We also expect humans to read & write these, so YAML is a much better format there. All JSON is also valid YAML, so that helps. Depends on jupyter/telemetry#13

Depend on the jupyter_telemetry package

c7428e8

We made a v0.0.1 release!

read schemas from new utils function

9437e88

Add fix for tables in RTD theme sphinx docs.

6e3c80c

Solution came from https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html

add event schema auto-documentation to jupyter notebook docs

4035fd5

format paths in recorded events

23d50a3

add documentation for eventlog endpoint

3c94970

return exception as 400 error in eventlog endpoint

e76c91b

normalize path in emitted event

2ce7c54

initial tests

5794d31

add initial telemetry docs

7c9d3d5

fix jupyter_telemetry dependency

ef8573d

point telemetry at correct dev branch

ea9e352

add tests for eventlog

b06f7d6

kiendang mentioned this pull request Dec 17, 2020

Jupyter server telemetry and event logging #352

Closed

kiendang added 2 commits December 17, 2020 13:31

Merge branch 'master' into jupyter_telemetry

4d7fc23

Use correct fixture names

7302396

kiendang force-pushed the jupyter_telemetry branch from bd8aa7d to 7302396 Compare December 17, 2020 05:32

Merge branch 'master' into jupyter_telemetry

2c79b1a

yuvipanda requested changes Mar 11, 2021

View reviewed changes

kiendang marked this pull request as ready for review April 26, 2021 10:07

kiendang requested a review from yuvipanda April 26, 2021 10:07

Zsailer self-assigned this Apr 27, 2021

Zsailer marked this pull request as draft April 28, 2021 18:21

Zsailer mentioned this pull request Apr 30, 2021

Jupyter Server Notes 2021 jupyter-server/team-compass#4

Closed

Remove eventlog endpoint

a67bb25

Move to a separate branch

kiendang mentioned this pull request May 3, 2021

[Telemetry] Add eventlog endpoint for collecting client-side events #501

Closed

kiendang added 4 commits May 3, 2021 18:12

Use telemetry upstream

7d906af

Use importlib.resources to read event schemas

82e70b8

instead of raw paths since raw paths do not work if e.g. install package from a zip file https://importlib-resources.readthedocs.io/en/latest/using.html#example

Fix docs

22b682f

Doc change notebook server to jupyter server

e0c38d2

kevin-bates reviewed May 3, 2021

View reviewed changes

docs/source/operators/telemetry.rst Outdated Show resolved Hide resolved

kiendang added 2 commits May 3, 2021 23:37

Remove eventlog API endpoint doc

31585e4

Merge remote-tracking branch 'upstream/master' into jupyter_telemetry

5703d94

Zsailer changed the title ~~Jupyter telemetry~~ Add basic telemetry to Jupyter server and begin emitting server events May 3, 2021

Zsailer changed the title ~~Add basic telemetry to Jupyter server and begin emitting server events~~ [Telemetry] Add basic telemetry to Jupyter server and begin emitting server events May 3, 2021

Zsailer added the Telemetry Discussion or issues related to Telemetry/Event-logging label Jun 14, 2021

kiendang mentioned this pull request Sep 8, 2021

Use fastjsonschema for json schema validation jupyter/telemetry#64

Draft

yuvipanda mentioned this pull request Mar 14, 2022

Update for compatibility with JupyterLab 3 jupyterlab/jupyterlab-telemetry#23

Closed

5 tasks

Zsailer mentioned this pull request Mar 17, 2022

Meeting Notes 2022 jupyter-server/team-compass#15

Closed

kiendang mentioned this pull request May 12, 2022

Basic Event Bus #820

Merged

1 task

manics mentioned this pull request May 21, 2022

Add telemetry events for Jupyterhub REST APIs jupyterhub/jupyterhub#3218

Draft

Zsailer mentioned this pull request Aug 29, 2022

Emit events from the Contents Service #954

Merged

Zsailer closed this Aug 29, 2022

kiendang deleted the jupyter_telemetry branch August 29, 2022 21:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Telemetry] Add basic telemetry to Jupyter server and begin emitting server events #364

[Telemetry] Add basic telemetry to Jupyter server and begin emitting server events #364

kiendang commented Dec 17, 2020 •

edited

Loading

robertpyke commented Feb 18, 2021

kiendang commented Feb 18, 2021

tmlohman commented Feb 18, 2021

kiendang commented Feb 19, 2021

yuvipanda left a comment

yuvipanda Mar 11, 2021

yuvipanda Mar 11, 2021

yuvipanda Mar 11, 2021

kiendang Mar 19, 2021

kiendang Mar 19, 2021

yuvipanda Mar 19, 2021

kiendang Mar 24, 2021

yuvipanda Mar 11, 2021

kiendang Mar 18, 2021

yuvipanda Mar 19, 2021

kiendang Mar 19, 2021

kiendang Mar 24, 2021

Zsailer commented Apr 27, 2021

kiendang commented May 3, 2021

mlucool commented Nov 30, 2021

kiendang commented Mar 24, 2022 •

edited

Loading

Zsailer commented Aug 29, 2022

[Telemetry] Add basic telemetry to Jupyter server and begin emitting server events #364

[Telemetry] Add basic telemetry to Jupyter server and begin emitting server events #364

Conversation

kiendang commented Dec 17, 2020 • edited Loading

robertpyke commented Feb 18, 2021

kiendang commented Feb 18, 2021

tmlohman commented Feb 18, 2021

kiendang commented Feb 19, 2021

yuvipanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zsailer commented Apr 27, 2021

kiendang commented May 3, 2021

mlucool commented Nov 30, 2021

kiendang commented Mar 24, 2022 • edited Loading

Zsailer commented Aug 29, 2022

kiendang commented Dec 17, 2020 •

edited

Loading

kiendang commented Mar 24, 2022 •

edited

Loading