diff --git a/docs/_static/images/how-to/ht-sagemaker-notebook_runs_01.png b/docs/_static/images/how-to/ht-sagemaker-notebook_runs_01.png new file mode 100644 index 00000000..f25749fb Binary files /dev/null and b/docs/_static/images/how-to/ht-sagemaker-notebook_runs_01.png differ diff --git a/docs/_static/images/notebooks/activate_button.png b/docs/_static/images/notebooks/activate_button.png new file mode 100644 index 00000000..098fe55e Binary files /dev/null and b/docs/_static/images/notebooks/activate_button.png differ diff --git a/docs/_static/images/notebooks/buttons_11.png b/docs/_static/images/notebooks/buttons_11.png new file mode 100644 index 00000000..28760119 Binary files /dev/null and b/docs/_static/images/notebooks/buttons_11.png differ diff --git a/docs/_static/images/notebooks/compare.png b/docs/_static/images/notebooks/compare.png new file mode 100644 index 00000000..c205b9a9 Binary files /dev/null and b/docs/_static/images/notebooks/compare.png differ diff --git a/docs/_static/images/notebooks/configure_connect.png b/docs/_static/images/notebooks/configure_connect.png new file mode 100644 index 00000000..56144c02 Binary files /dev/null and b/docs/_static/images/notebooks/configure_connect.png differ diff --git a/docs/_static/images/notebooks/connect_button.png b/docs/_static/images/notebooks/connect_button.png new file mode 100644 index 00000000..31d8b379 Binary files /dev/null and b/docs/_static/images/notebooks/connect_button.png differ diff --git a/docs/_static/images/notebooks/download_dialog.png b/docs/_static/images/notebooks/download_dialog.png new file mode 100644 index 00000000..b6ee1a12 Binary files /dev/null and b/docs/_static/images/notebooks/download_dialog.png differ diff --git a/docs/_static/images/notebooks/extension_manager.png b/docs/_static/images/notebooks/extension_manager.png new file mode 100644 index 00000000..500618c6 Binary files /dev/null and b/docs/_static/images/notebooks/extension_manager.png differ diff --git a/docs/_static/images/notebooks/get_api_token.png b/docs/_static/images/notebooks/get_api_token.png new file mode 100644 index 00000000..995229c1 Binary files /dev/null and b/docs/_static/images/notebooks/get_api_token.png differ diff --git a/docs/_static/images/notebooks/link_new_notebook.png b/docs/_static/images/notebooks/link_new_notebook.png new file mode 100644 index 00000000..20b03b6c Binary files /dev/null and b/docs/_static/images/notebooks/link_new_notebook.png differ diff --git a/docs/_static/images/notebooks/nb-view-11.png b/docs/_static/images/notebooks/nb-view-11.png new file mode 100644 index 00000000..c5ddd077 Binary files /dev/null and b/docs/_static/images/notebooks/nb-view-11.png differ diff --git a/docs/_static/images/notebooks/nb-view-22.png b/docs/_static/images/notebooks/nb-view-22.png new file mode 100644 index 00000000..c5ddd077 Binary files /dev/null and b/docs/_static/images/notebooks/nb-view-22.png differ diff --git a/docs/_static/images/notebooks/upload_dialog.png b/docs/_static/images/notebooks/upload_dialog.png new file mode 100644 index 00000000..96a4e0bb Binary files /dev/null and b/docs/_static/images/notebooks/upload_dialog.png differ diff --git a/docs/index.rst b/docs/index.rst index 2a457df2..5b420762 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -92,12 +92,12 @@ Documentation contents .. toctree:: :maxdepth: 2 - :caption: Notebooks + :caption: Jupyter Notebooks in Neptune Introduction Installation for Jupyter and JupyterLab Configuration - Integrations (AWS, SageMaker, Colab) + Integrations with AWS and SageMaker Troubleshoot .. toctree:: diff --git a/docs/notebooks/configuration.rst b/docs/notebooks/configuration.rst index 8e19933b..10756a2a 100644 --- a/docs/notebooks/configuration.rst +++ b/docs/notebooks/configuration.rst @@ -1,46 +1,44 @@ -Configuration -============= -Simple configuration wizard lets you to setup your work with Neptune. +Connecting the Jupyter Extension to Your Neptune Account +======================================================== -**Step 1** +After you have successfully `installed the Jupyter extension for Neptune `_, +you connect it to your Neptune account. -Click **Configure** button (if you cannot see it, just hit **F5**). +**Procedure** -.. image:: ../_static/images/notebooks/buttons_01_1.png - :target: ../_static/images/notebooks/buttons_01_1.png +1. In Jupyter, click **Connect to Neptune**. + +.. image:: ../_static/images/notebooks/connect_button.png + :target: ../_static/images/notebooks/connect_button.png :alt: image -**Step 2** -You see **Configure connection to Neptune**. Paste API Token first, then select project: +The **Configure your connection to Neptune** dialog appears. -* *API Token* - take one from your account. :ref:`Check this `, if you do not know where *API Token* is located. This field is validated automatically. -* *Select project* - pick one from the drop down list. -* Click **Create notebook**. +.. image:: ../_static/images/notebooks/configure_connect.png + :target: ../_static/images/notebooks/configure_connect.png + :alt: image -.. warning:: Your *API Token* is private and unique. Never share it. It's like sharing password. -.. image:: ../_static/images/notebooks/checkpoint_01.png - :target: ../_static/images/notebooks/checkpoint_01.png - :alt: image +2. Leave the dialog open and switch to the Neptune UI. -.. image:: ../_static/images/notebooks/checkpoint_02.png - :target: ../_static/images/notebooks/checkpoint_02.png - :alt: image +3. In the Neptune UI, obtain your API Token and copy it to the clipboard. -.. image:: ../_static/images/notebooks/checkpoint_03.png - :target: ../_static/images/notebooks/checkpoint_03.png - :alt: image + a. In the upper right corner, click the avatar, and then click **Get API Token**. + + .. image:: ../_static/images/notebooks/get_api_token.png + :target: ../_static/images/notebooks/get_api_token.png + :alt: image -.. note:: Success! You just created new notebook with first checkpoint in the project you selected. + b. In the dialog that appears, click the **Copy to clipboard** button on the right. Then click **Close**. -**Step 3** +4. Switch back to Jupyter. In the dialog you left open, paste the token you copied to the clipboard. Click **Connect**. -* You can use *this link* at the top, to go directly to this notebook in Neptune. -* Click **Integrate** button to run environment variables export. It allows Neptune to checkpoint your work correctly. Moreover, you will be able to run Neptune experiments directly from notebook. + A confirmation message is displayed. It contains a link through which you can go directly to this Notebook in Neptune. -.. image:: ../_static/images/notebooks/integration_01.png - :target: ../_static/images/notebooks/integration_01.png - :alt: image +.. warning:: Your *API Token* is private and unique. Never share it. It's like sharing password. + + +5. To conclude, to see experiments that you will run associated with this Notebook, click **Activate**. +In the dialog that appears, click **Activate**. -.. note:: Re-run **Integrate** each time you reset or restart your kernel. diff --git a/docs/notebooks/installation.rst b/docs/notebooks/installation.rst index 66d7447e..ad558ff9 100644 --- a/docs/notebooks/installation.rst +++ b/docs/notebooks/installation.rst @@ -1,77 +1,177 @@ -Installation for Jupyter and JupyterLab -======================================= +Installing neptune-notebooks +============================ +This page provides instructions for installing neptune-notebooks -- the Neptune Jupyter extension. -Jupyter -------- -Install `neptune-notebooks extension `_: +The extension enables the integration of Neptune and Jupyter or JupyterLab. + +When the extension is successfully installed, +you will be able to upload notebooks to Neptune, check out previously uploaded Notebooks, +and track experiments and metrics directly from the Jupyter UI. +Neptune versions the Notebook automatically once an experiment has started. + +About neptune-notebooks +----------------------- + +neptune-notebooks is a Python package hosted `here `_. It includes: + +- A very basic CLI (command line interface) for uploading notebooks. +- Standard Jupyter extension. The extension is written in Javascript and is visible in the web-based Jupyter UI. It is not hosted anywhere separately, only as part of the python package. +- JupyterLab extension. The extension is written in Javascript and is visible in the web-based Jupyter UI. It is hosted `here `_. + + +Installation for Jupyter +------------------------ + +.. tip:: When installing Python packages, it is best practice to work in a `virtual environment `_. + +Procedure + +1. Install `neptune-notebooks extension `_: .. code-block:: bash pip install neptune-notebooks -Then enable extension for your Jupyter: +2. Enable the extension for Jupyter: .. code-block:: bash jupyter nbextension enable --py neptune-notebooks -Remember to install Neptune-client, if you did not do so already: +3. Install Neptune client, if you did not do so already: .. code-block:: bash pip install neptune-client -JupyterLab ----------- -Requirements -^^^^^^^^^^^^ -`Node.js `_ and `npm `_ are required. +Installation for JupyterLab +--------------------------- -To check if you have Node.js installed, run this command in your terminal: +JupyterLab is the next-generation web-based UI for Project Jupyter. -.. code-block:: bash +When you install JupyterLab using ``pip install jupyterlab``, the standard Jupyter is also installed, as a dependency. + + +**Installation methods** + +You can install the extension using any of the following three methods: + +- Through the command line (full neptune-notebooks package) +- Through the command line (JupyterLab extension only) +- Using the Extension Manager (JupyterLab extension only) + +Details for these methods follow below. + +**Requirements** + +Irrespective of which method you use, the following must be preinstalled on your system before you begin. + +- `Node.js `_ +- `npm `_ +- `JupyterLab `_ + +.. tip:: When installing Python packages, it is best practice to work in a `virtual environment `_. + + +Method 1: Install the full neptune-notebooks package through the command line +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +**Pros** + +This is the most comprehensive type of installation: - node -v +- It includes extensions for both standard Jupyter and JupyterLab. +- It also installs a CLI for uploading Notebook files. +- The the neptune-client package is a dependency. +**Cons** -To confirm that you have npm installed you can run this command in your terminal: +- After the installation, you still have to perform some manual actions in JupyterLab. +- This method may not be the easiest way of installing extensions in JupyterLab. + +**Procedure** + +1. In the console, run: + +.. code-block:: bash + + pip install neptune-notebooks + +2. Start JupyterLab by running: .. code-block:: bash - npm -v + jupyter lab + +3. In the "Build Recommended" notification that is displayed, click **Build**. + +.. note:: The build process takes up to several minutes. + +4. In the "Build Complete" notification that is displayed, click **Reload**. + +The extension is loaded. + +Method 2: Install the JupyterLab extension only through the command line +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This method installs only the extension to JupyterLab. -If you do not have them, please refer to: `installation instructions `_. +**Cons** -Using command line -^^^^^^^^^^^^^^^^^^ -Install `neptune-notebooks `_ for your JupyterLab. In terminal run: +The neptune-notebooks pip package, neptune-client pip package and extension to the standard Jupyter are not installed. + +**Pros** + +This method gives you more flexibility. For example, the extension can be running on a +different machine than your code that is executed from the Notebook. + +**Procedure** + +1. In the console, run: .. code-block:: bash jupyter labextension install neptune-notebooks -Remember to install Neptune-client, if you did not do so already: +2. Start JupyterLab by running: .. code-block:: bash - pip install neptune-client + jupyter lab + +3. In the "Build Recommended" notification that is displayed, click **Build**. +4. In the "Build Complete" notification that is displayed, click **Reload**. -Using JupyterLab extension manager (alternative) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -**Step 1** +Method 3: Install the JupyterLab extension only, using the JupyterLab Extension Manager +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Enable extension manager. Select "Commands" (third icon on left panel), then click on "Enable Extension Manager (experimental)" +**Procedure** -.. image:: ../_static/images/notebooks/ext-manager-1.png - :target: ../_static/images/notebooks/ext-manager-1.png - :alt: enable extension manager +1. In JupyterLab, enable Extension Manager, by clicking **Settings** > **Enable Extension Manager (experimental)**. -**Step 2** +.. image:: ../_static/images/notebooks/extension_manager.png + :target: ../_static/images/notebooks/extension_manager.png + :alt: Enable extension manager -Go to "Extension Manager" (third icon on left panel) and search for "neptune-notebooks". Click "Install", then "Enable". +2. Find neptune-notebooks and click **Install**. .. image:: ../_static/images/notebooks/ext-manager-2.png :target: ../_static/images/notebooks/ext-manager-2.png :alt: go to extension manager and search for neptune-notebooks + + +Checking if the Installation was Successful +------------------------------------------- + +Provided you have `configured `_ Jupyter correctly, the following Neptune buttons appear in Jupyter notebooks: + +.. image:: ../_static/images/notebooks/buttons_11.png + :target: ../_static/images/notebooks/buttons_11.png + :alt: image + +- **Neptune**: For modifying configuration. +- **Upload**: For creating a new checkpoint in Neptune. +- **Download**: For downloading a specific checkpoint from Neptune. +- **Activate**: Associates experiments you will be running with this Notebook. \ No newline at end of file diff --git a/docs/notebooks/integrations.rst b/docs/notebooks/integrations.rst index ada15783..4ea2f4b5 100644 --- a/docs/notebooks/integrations.rst +++ b/docs/notebooks/integrations.rst @@ -1,279 +1,245 @@ -Integrations with AWS, SageMaker and Google Colab -================================================= +Integrations with AWS and SageMaker +=================================== + +Setting up a Neptune-enabled JupyterLab on AWS +---------------------------------------------- +You can run Neptune and track experiments that you run on AWS cloud. -Setup Neptune-enabled JupyterLab on AWS ---------------------------------------- -I would like to run Neptune and track experiments that I run on AWS cloud. -How do I do that? .. image:: ../_static/images/others/aws_neptuneml.png :target: ../_static/images/others/aws_neptuneml.png :alt: AWS neptune.ai integration -**Register to AWS** - -Follow the `registration instructions `_ from official webpage to create your AWS account. - -**Start EC2 instance** +1. Register to AWS. -Start a new EC2 instance. Select `ubuntu` as your instance type and choose a worker type you need. -You can go with `t2.micro` just to test it out. + Follow the `instructions `_ to create your AWS account. -**ssh to your instance** +2. Start an EC2 instance. -Connect to your instance by going to the terminal and running: + Select ``ubuntu`` as your instance type and choose the worker type you need. + You can use ``t2.micro`` just to test it out. -.. code-block:: Bash +3. SSH to your instance. - ssh -i /path_to_key/my_key.pem ubuntu@public_dns_name + From the terminal, run: -*(make sure that you put correct key and public_dns_name)* + .. code-block:: Bash -**Install docker** + ssh -i /path_to_key/my_key.pem ubuntu@public_dns_name -Create a new file `install_docker.sh`: + Make sure that you specify the correct key and public_dns_name. -.. code-block:: Bash +4. Install Docker. - nano install_docker.sh + a. Create a new file ``install_docker.sh``: -Copy the following commands to it: + .. code-block:: Bash -.. code-block:: Bash + nano install_docker.sh - sudo apt-get update - sudo apt-get install \ - apt-transport-https \ - ca-certificates \ - curl \ - gnupg-agent \ - software-properties-common - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - sudo apt-key fingerprint 0EBFCD88 - sudo add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) \ - stable" - sudo apt-get update - sudo apt-get install docker-ce docker-ce-cli containerd.io + b. Copy the following commands to it: -Run the installation script: + .. code-block:: Bash -.. code-block:: Bash + sudo apt-get update + sudo apt-get install \ + apt-transport-https \ + ca-certificates \ + curl \ + gnupg-agent \ + software-properties-common + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - + sudo apt-key fingerprint 0EBFCD88 + sudo add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) \ + stable" + sudo apt-get update + sudo apt-get install docker-ce docker-ce-cli containerd.io - source install_docker.sh + c. Run the installation script: -**Define your secrets** + .. code-block:: Bash -| Go to Neptune web app, get your ``NEPTUNE_API_TOKEN`` and copy it. Then, create a password for your JupyterLab server. -| Set those two secrets to your environment variables ``NEPTUNE_API_TOKEN`` and ``JUPYTERLAB_PASSWORD``, like below: + source install_docker.sh -.. code-block:: Bash +5. Define your secrets. - export NEPTUNE_API_TOKEN='your_api_token==' - export JUPYTERLAB_PASSWORD='difficult_password' + a. Go to Neptune, `get your NEPTUNE_API_TOKEN `_ and copy it to the clipboard. + b. Create a password for your JupyterLab server. + c. Set the following two secrets to your environment variables, ``NEPTUNE_API_TOKEN`` and ``JUPYTERLAB_PASSWORD``: -**Build docker image** + .. code-block:: Bash -Create a new file `Dockerfile`: + export NEPTUNE_API_TOKEN='your_api_token==' + export JUPYTERLAB_PASSWORD='difficult_password' -.. code-block:: Bash +6. Build the Docker image. - nano Dockerfile + a. Create a new file ``Dockerfile``: -Copy insights of the following `Dockerfile` to your newly created file: + .. code-block:: Bash -.. code-block:: Docker + nano Dockerfile - # Use a miniconda3 as base image - FROM continuumio/miniconda3 + b. Copy insights of the following Dockerfile to your newly created file: - # Installation of jupyterlab and extensions - RUN pip install jupyterlab==0.35.6 && \ - pip install jupyterlab-server==0.2.0 && \ - conda install -c conda-forge nodejs + .. code-block:: Docker - # Installation of Neptune and enabling neptune extension - RUN pip install neptune-client && \ - pip install neptune-notebooks && \ - jupyter labextension install neptune-notebooks - - # Setting up Neptune API token as env variable - ARG NEPTUNE_API_TOKEN - ENV NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN + # Use a miniconda3 as base image + FROM continuumio/miniconda3 - # Adding current directory to container - ADD . /mnt/workdir - WORKDIR /mnt/workdir + # Installation of jupyterlab and extensions + RUN pip install jupyterlab==1.2.6 && \ + pip install jupyterlab-server==1.0.6 && \ + conda install -c conda-forge nodejs -| *(If you want to run on GPU make sure to change your `Dockerfile` to start from nvidia docker images)*. + # Installation of Neptune and enabling neptune extension + RUN pip install neptune-client && \ + pip install neptune-notebooks && \ + jupyter labextension install neptune-notebooks -Run following command to build your docker image: + # Setting up Neptune API token as env variable + ARG NEPTUNE_API_TOKEN + ENV NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN -.. code-block:: Bash + # Adding current directory to container + ADD . /mnt/workdir + WORKDIR /mnt/workdir - sudo docker build -t jupyterlab --build-arg NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN . + If you want to run on a GPU, make sure to change your Dockerfile to start from Nvidia Docker images. -**Start JupyterLab server** + c. Run the following command to build your Docker image: -Spin up JupyterLab server with docker: + .. code-block:: Bash -.. code-block:: Bash + sudo docker build -t jupyterlab --build-arg NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN . - sudo docker run --rm -v `pwd`:/work/output -p 8888:8888 jupyterlab:latest \ - /opt/conda/bin/jupyter lab --allow-root --ip=0.0.0.0 --port=8888 --NotebookApp.token=$JUPYTERLAB_PASSWORD +7. Spin up a JupyterLab server with Docker: -**Forward ports via ssh tunnel** + .. code-block:: Bash -Open a new terminal on your local machine and run: + sudo docker run --rm -v `pwd`:/work/output -p 8888:8888 jupyterlab:latest \ + /opt/conda/bin/jupyter lab --allow-root --ip=0.0.0.0 --port=8888 --NotebookApp.token=$JUPYTERLAB_PASSWORD -.. code-block:: Bash +8. Forward ports via SSH tunnel. Open a new terminal on your local machine and run: - ssh -L 8888:localhost:8888 ubuntu@public_dns_name & + .. code-block:: Bash -*(make sure that you put correct public_dns_name)* + ssh -L 8888:localhost:8888 ubuntu@public_dns_name & -**Open JupyterLab server in your browser** + Make sure that you specify the correct public_dns_name. -Go to `localhost:8888` and enjoy your JupyterLab server with Neptune! +9. Open the JupyterLab server in your browser. -**Final result** + Go to `localhost:8888`. + +Enjoy your JupyterLab server with Neptune. -Neptune extensions are enabled and ``NEPTUNE_API_TOKEN`` is already in the environment variable so you can work with Notebooks and run experiments with no problems. +Neptune extensions are enabled and ``NEPTUNE_API_TOKEN`` is already in the environment variable so you can work with +Notebooks and run experiments with no problem. -Setup Neptune-enabled AWS SageMaker Jupyter instance ----------------------------------------------------- -I would like to use Neptune to track experiments that I run on AWS SageMaker. -How do I do that? +Setting up a Neptune-enabled AWS SageMaker Jupyter instance +----------------------------------------------------------- +You can use Neptune to track experiments that you run on AWS SageMaker. .. image:: ../_static/images/others/sagemaker_neptuneml.png :target: ../_static/images/others/sagemaker_neptuneml.png :alt: Amazon SageMaker neptune.ai integration -**Register to AWS** - -Follow the `registration instructions `_ from official website to create your AWS account. +1. Register to AWS. -**Create Lifecycle configuration** + Follow the `instructions `_ to create your AWS account. -Go to SageMaker *Lifecycle configurations* and click on *Create configuration*. +2. Create Lifecycle configuration. -.. image:: ../_static/images/how-to/ht-sagemaker-create_configuration.png - :target: ../_static/images/how-to/ht-sagemaker-create_configuration.png - :alt: image + a. Go to SageMaker **Lifecycle configurations** and click **Create configuration**. -You can choose whatever name you want just make sure to remember it. + .. image:: ../_static/images/how-to/ht-sagemaker-create_configuration.png + :target: ../_static/images/how-to/ht-sagemaker-create_configuration.png + :alt: image + | -Now, you should modify the *Create notebook* script to run it only once at creation of your SageMaker notebook instance. + You can choose whatever name you want -- just make sure to remember it. -.. image:: ../_static/images/how-to/ht-sagemaker-config_specs.png - :target: ../_static/images/how-to/ht-sagemaker-config_specs.png - :alt: image + b. Modify the **Create Notebook** script to run it only once at the creation of your SageMaker Notebook instance. -Copy and paste the script below to your *Create notebook* tab. -Choose in which environments you want to install *neptune-client* in the *PARAMETERS* section. + .. image:: ../_static/images/how-to/ht-sagemaker-config_specs.png + :target: ../_static/images/how-to/ht-sagemaker-config_specs.png + :alt: image -.. code-block:: Bash - - #!/bin/bash - - set -e - - sudo -u ec2-user -i <<'EOF' - - # PARAMETERS - ENVS=( - python3 - pytorch_p36 - ) - - # Install Neptune client library - for env in $ENVS; do - source /home/ec2-user/anaconda3/bin/activate $env - pip install neptune-client - source /home/ec2-user/anaconda3/bin/deactivate - done + | + + c. Copy and paste the script below to your **Create Notebook** tab. - # Install jupyter extension - source /home/ec2-user/anaconda3/bin/activate JupyterSystemEnv - pip install neptune-notebooks - jupyter nbextension enable --py neptune-notebooks --sys-prefix - jupyter labextension install neptune-notebooks - source /home/ec2-user/anaconda3/bin/deactivate + In the **PARAMETERS** section, choose in which environments you want to install neptune-client. - EOF + .. code-block:: Bash -**Create notebook instance** + #!/bin/bash -Go to SageMaker *Notebook instances* and click on *Create notebook instance*. + set -e -.. image:: ../_static/images/how-to/ht-sagemaker-create_instance.png - :target: ../_static/images/how-to/ht-sagemaker-create_instance.png - :alt: image + sudo -u ec2-user -i <<'EOF' -Select an *instance name* and click on *Additional configuration* to add the *Lifecycle configuration* you have just created: + # PARAMETERS + ENVS=( + python3 + pytorch_p36 + ) -.. image:: ../_static/images/how-to/ht-sagemaker-create_instance_specs.png - :target: ../_static/images/how-to/ht-sagemaker-create_instance_specs.png - :alt: image + # Install Neptune client library + for env in $ENVS; do + source /home/ec2-user/anaconda3/bin/activate $env + pip install neptune-client + source /home/ec2-user/anaconda3/bin/deactivate + done -You can now click *Create notebook instance* instance at the bottom to start your instance. - -.. image:: ../_static/images/how-to/ht-sagemaker-create_notebook_run.png - :target: ../_static/images/how-to/ht-sagemaker-create_notebook_run.png - :alt: image - -**Start notebook** - -If everything went well your AWS SageMaker instance should be *InService* and you can now open Jupyter Notebook or Jupyter lab with Neptune notebook-versioning enabled! - -.. image:: ../_static/images/how-to/ht-sagemaker-notebook_run.png - :target: ../_static/images/how-to/ht-sagemaker-notebook_run.png - :alt: image - -You can now version your notebooks and track experiments in Amazon SageMaker with Neptune! - -.. image:: ../_static/images/how-to/ht-sagemaker-notebook_runs.png - :target: ../_static/images/how-to/ht-sagemaker-notebook_runs.png - :alt: image - - -Track Google Colab experiments with Neptune -------------------------------------------- -I would like to run my experiments on Google Colab and track them with Neptune. How do I do that? - -.. image:: ../_static/images/others/colab_neptuneml.png - :target: ../_static/images/others/colab_neptuneml.png - :alt: Google Colaboratory neptune.ai integration + # Install jupyter extension + source /home/ec2-user/anaconda3/bin/activate JupyterSystemEnv + pip install neptune-notebooks + jupyter nbextension enable --py neptune-notebooks --sys-prefix + jupyter labextension install neptune-notebooks + source /home/ec2-user/anaconda3/bin/deactivate -**Install Neptune client** + EOF -Go to your first cell in Google Colab and install `neptune-client`: +3. Create a Notebook instance. -.. code-block:: Bash + a. Go to SageMaker **Notebook instances** and click **Create notebook instance**. - ! pip install neptune-client + .. image:: ../_static/images/how-to/ht-sagemaker-create_instance.png + :target: ../_static/images/how-to/ht-sagemaker-create_instance.png + :alt: image -**Set Neptune API token** + | -Go to Neptune web app and get your API token. Set it to the environment variable `NEPTUNE_API_TOKEN`: + b. Type in an instance name and click **Additional configuration** to add the Lifecycle configuration you have just created. -.. code-block:: Bash + .. image:: ../_static/images/how-to/ht-sagemaker-create_instance_specs.png + :target: ../_static/images/how-to/ht-sagemaker-create_instance_specs.png + :alt: image - % env NEPTUNE_API_TOKEN='your_private_neptune_api_token==' + | -Delete this cell. + c. Click **Create notebook instance** at the bottom to start your instance. -.. warning:: + .. image:: ../_static/images/how-to/ht-sagemaker-create_notebook_run.png + :target: ../_static/images/how-to/ht-sagemaker-create_notebook_run.png + :alt: image - It is very important that you delete this cell not to share your private token with anyone. +4. Start Notebook. -**That's it. Run your training script with Neptune.** + If everything went well, your AWS SageMaker instance should have *InService* status and you can open a Jupyter Notebook or JupyterLab + with Neptune Notebook versioning enabled. -.. code-block:: Python + .. image:: ../_static/images/how-to/ht-sagemaker-notebook_run.png + :target: ../_static/images/how-to/ht-sagemaker-notebook_run.png + :alt: image - import neptune - neptune.init('USER_NAME/PROJECT_NAME') + +You can now version your Notebooks and track experiments in Amazon SageMaker with Neptune. - with neptune.create_experiment(): - neptune.send_metric('auc', 0.92) + .. image:: ../_static/images/how-to/ht-sagemaker-notebook_runs_01.png + :target: ../_static/images/how-to/ht-sagemaker-notebook_runs_01.png + :alt: image \ No newline at end of file diff --git a/docs/notebooks/introduction.rst b/docs/notebooks/introduction.rst index 8e7de746..13328105 100644 --- a/docs/notebooks/introduction.rst +++ b/docs/notebooks/introduction.rst @@ -1,81 +1,176 @@ -Notebooks in Neptune -==================== +Using Jupyter Notebooks in Neptune +================================== .. image:: ../_static/images/others/notebooks_neptuneml.png :target: ../_static/images/others/notebooks_neptuneml.png - :alt: Jupyter notebooks neptune.ai integration + :alt: Jupyter Notebooks neptune.ai integration -Introduction + +Jupyter Notebooks are a useful and popular tool for data scientists, regardless of their area of specialization. +They allow data scientists to work interactively, keeping code and results - like visualizations - in a single document. + +While Neptune is essentially a platform for tracking experiments, it provides Jupyter and JupyterLab extensions that also let +you track Jupyter Notebooks. + +Key Features ------------ -Notebooks are essential tool for data scientists, regardless their area of specialization. They allow data scientists to work interactively, keeping code and results - like visualizations - in a single document. Neptune builds on top of this experience and comes with Jupyter and JupyterLab extensions that let you track Notebooks in Neptune. -Notebooks in Neptune - key features: +* In Neptune, each Notebook consists of a collection of checkpoints that you upload directly from the Jupyter user interface. +* In any project, an unlimited number of Notebooks and checkpoints is allowed. +* You can browse checkpoints history across all `Notebooks in the project `_. +* You can `share `_ a notebook as a link. +* You can `compare `_ two Notebooks side-by-side, like source code. -* In Neptune, each notebook consist of collection of checkpoints that you upload directly from the Jupyter User interface. -* In the project, unlimited number of notebooks and checkpoints is allowed. -* Browse checkpoints history across all `notebooks in the project `_, -* `Share `_ notebook as a link, -* `Compare `_ two notebooks side-by-side like source code. -Quick-start +Quick Start ----------- -* To start working with notebooks in Neptune, `install `_ and `configure `_ open source extension for Jupyter or JupyterLab. When you have it done, you can start working with notebooks immediately. -* To try it now, without registering to Neptune, look at example notebooks in public project `onboarding `_. Use public user's API token (username: *neptuner*) to upload some snapshots to this project (you still need to `install `_ and `configure `_ Jupyter extension). +To start working with Notebooks in Neptune, `install `_ and `configure `_ the open +source extension for Jupyter or JupyterLab. When you are done, you can start working with Notebooks immediately. -.. code-block:: bash +To try it now, without registering to Neptune, look at the sample Notebooks in the public project `onboarding `_. +Use the public user's API token that appears below, and the username *neptuner* to upload some snapshots to this project. +You still need to `install `_ and `configure `_ Jupyter extension. + +**Public user’s API token**: - eyJhcGlfYWRkcmVzcyI6Imh0dHBzOi8vdWkubmVwdHVuZS5tbCIsImFwaV9rZXkiOiJiNzA2YmM4Zi03NmY5LTRjMmUtOTM5ZC00YmEwMzZmOTMyZTQifQ== +.. code-block:: bash -Tour ----- -`Notebooks view `_ is a collection of all notebooks in the project - each can have multiple checkpoints. + eyJhcGlfYWRkcmVzcyI6Imh0dHBzOi8vdWkubmVwdHVuZS5haSIsImFwaV9rZXkiOiJiNzA2YmM4Zi03NmY5LTRjMmUtOTM5ZC00YmEwMzZmOTMyZTQifQ== -Notebooks view -~~~~~~~~~~~~~~ -* See what your team members are working on now. -* Review details and checkpoints associated with Notebook. -* Share, compare or download notebook. +Notebook UI +----------- +The `Notebooks tab `_ in the Neptune UI provides a table of all the Notebooks in the current project. -.. image:: ../_static/images/notebooks/nb-view-1.png - :target: ../_static/images/notebooks/nb-view-1.png +.. image:: ../_static/images/notebooks/nb-view-11.png + :target: ../_static/images/notebooks/nb-view-11.png :alt: image + +This view lets you see what your team members are working on, review details and checkpoints associated with a Notebook, as well as share or download a Notebook and compare two or more Notebooks. + +The Notebook data is arranged in the following columns: + +* Name +* Owner +* Latest checkpoint +* Description + +In addition, for each Notebook, there are buttons for downloading the Notebook, comparing it with another Notebook, or for sharing a link to it. + +A **Compare** button at the top right displays a Notebooks Comparison pane. See `Compare Notebooks `_. + Notebook contents ~~~~~~~~~~~~~~~~~ -Once you entered notebook, you can see all its content, that is: code and markdown cells, outputs and execution count. +Once you select a Notebook, you can see all its contents, that is: code and markdown cells, outputs and execution count. + +There are two tabs on the right: -* Look at the snapshot of the work with notebook. -* Download, share or compare this checkpoint. +- **Details**: Here are shown the ID, size, creation date, latest checkpoint, owner, description and associated experiments of the selected Notebook. +- **Checkpoints**: Here are listed all the checkpoints of the Notebook. Click a checkpoint to see the details in the main pane. From this tab, you can also access the experiments that are associated with the checkpoint. +You can also view snapshots of the work with the Notebook, as well as download, share or compare this checkpoint with another checkpoint. -.. image:: ../_static/images/notebooks/nb-view-2.png - :target: ../_static/images/notebooks/nb-view-2.png +.. image:: ../_static/images/notebooks/nb-view-22.png + :target: ../_static/images/notebooks/nb-view-22.png :alt: image -Notebooks comparison -~~~~~~~~~~~~~~~~~~~~ -Select two notebooks and compare their contents - code and markdown cells, outputs and execution count - side-by-side just like source code. `Compare view `_ let you look at diff between checkpoints of the same notebook, or two entirely different notebooks (Try yourself `here `_). +Compare Notebooks +~~~~~~~~~~~~~~~~~ + +The Notebooks Comparison pane lets you compare Notebook checkpoints. + +You display the pane by clicking the **Compare** button anywhere it is visible in the Notebooks pane. + +.. image:: ../_static/images/notebooks/compare.png + :target: ../_static/images/notebooks/compare.png + :alt: image + +In the Notebooks Comparison pane, select two Notebook checkpoints, then click **Compare** to see a side-by-side comparison, just like source code. + +.. Commented out. Doesn't seem to be working. +.. `Compare view `_ let you look at the difference between checkpoints of the same Notebook, or two entirely different Notebooks (Try yourself `here `_). + + +* Differences in code, markdown, output and execution count are highlighted. +* Summary information about the differences is displayed at the top of the pane. .. image:: ../_static/images/notebooks/nb-view-cmp-1.png :target: ../_static/images/notebooks/nb-view-cmp-1.png :alt: image -* Code, markdown, output and execution count differences highlighted. -* Top bar displays summary information about diff. -Upload notebook checkpoint --------------------------- -Once `configured `_\ , you can see two Neptune buttons. +Uploading and Downloading Notebook Checkpoints +------------------------------------------- -* - **n** button is for configuration changes -* - **Upload** button is for making checkpoint in Neptune (*see below*) +Notebooks are stored as files on your computer. -.. image:: ../_static/images/notebooks/buttons_02_1.png - :target: ../_static/images/notebooks/buttons_02_1.png - :alt: image +Each Notebook file (.ipynb) is a JSON containing everything that the user can see in a Notebook and some metadata. +Neptune uses metadata to associate particular files with Notebook entities on Neptune servers. That means that after a Notebook +is uploaded to Neptune, the file on disk is changed to include the ID of the entity on the Neptune server. -* - Click **Upload**\ , whenever you want to create new checkpoint in Neptune. -* - You will see tooltip with link as a confirmation. +**Name changes** -.. image:: ../_static/images/notebooks/buttons_03_1.png - :target: ../_static/images/notebooks/buttons_03_1.png - :alt: image +If you copy a Notebook file (let’s call it "Notebook A") and +edit it with the intention of creating something completely separate from Notebook A, +the association with Notebook A on the Neptune server remains. If the name of the Notebook changes from "Notebook A", +you will be warned. + + +**Global accessibility** + +When you download a Notebook checkpoint, the ID in the metadata is preserved, so that when, after some work, +you click **Upload**, Neptune knows that this may be another checkpoint in a particular Notebook. +You can do some work, upload some intermediate snapshot, go to another computer +(or another SageMaker instance, and so on), download the Notebook and keep on working on it. + +The capability is comparable to Google Docs in that there’s a place where you store your work and you can access +it easily from wherever you choose. + +**Collaboration** + +Depending on their roles, members of a project can view and download all Notebooks (and their checkpoints) in the project. + +- Viewers can download Notebooks. +- Contributors and Owners can also upload them. + +When uploading a new Notebook, a user becomes the owner of this Notebook. Only the owner of a Notebook can upload +new checkpoints of this Notebook. + +Uploading a Notebook +~~~~~~~~~~~~~~~~~~~~ + +You can upload Notebook checkpoints from Jupyter to Neptune. + +**To upload the current Notebook as a checkpoint**: + +1. Click **Upload**. + + .. image:: ../_static/images/notebooks/upload_dialog.png + :target: ../_static/images/notebooks/upload_dialog.png + :width: 450 + :alt: Upload Notebook dialog + +2. In the dialog that is displayed, select a project from the list. +3. (Optional) Type in a checkpoint name and description. +4. Click **Upload checkpoint**. + +A confirmation message is displayed. You can click the link in the message to open the Notebook in Neptune. + +Downloading a Notebook +~~~~~~~~~~~~~~~~~~~~~~ + +You can download a specific Notebook checkpoint from Neptune to Jupyter. + +**To download a Notebook checkpoint**: + +1. Click **Download**. + + .. image:: ../_static/images/notebooks/download_dialog.png + :target: ../_static/images/notebooks/download_dialog.png + :width: 450 + :alt: Download Notebook dialog + +2. In the dialog that is displayed, select the following from the respective lists: + - Project + - Notebook + - Checkpoint +3. Click **Download**. diff --git a/docs/notebooks/troubleshoot.rst b/docs/notebooks/troubleshoot.rst index 40a32ba4..2e9e7d09 100644 --- a/docs/notebooks/troubleshoot.rst +++ b/docs/notebooks/troubleshoot.rst @@ -1,21 +1,22 @@ Troubleshoot ============ -Typical problems with notebooks, together with their solutions are listed here. +You may experience issues while working with Jupyter Notebooks and Neptune. +The following presents possible solutions to some of the issues. .. contents:: :local: :depth: 1 :backlinks: top -I can't see 'configure' button. What to do? -------------------------------------------- -Make sure to install notebook extension: +I can't see the **Connect to Neptune** button. What do I do? +------------------------------------------------------------ +Make sure you have installed the Notebook extension: .. code-block:: bash pip install neptune-notebooks -then enable extension for your Jupyter: +then enable the extension for your Jupyter: .. code-block:: bash @@ -27,38 +28,43 @@ Don't forget to install Neptune client: pip install neptune-client -How to enable notebook extension in my jupyter? ------------------------------------------------ -Enable extension for your jupyter: +For more details, see `Installing neptune-notebooks `_. + +How do I enable the Notebook extension in my Jupyter? +----------------------------------------------------- +Enable extension for your Jupyter: .. code-block:: bash jupyter nbextension enable --py neptune-notebooks -I do not know where my notebook was uploaded. How to check it? --------------------------------------------------------------- -#. Click on the **n** button in your jupyter menu. -#. Click on **Checkpoint**. +I do not know where my Notebook was uploaded. How do I check it? +---------------------------------------------------------------- +#. In Jupyter, click **Connect to Neptune**. +#. Click **Checkpoint**. #. Bottom drop-down is your current project. .. _token-location: Where is *NEPTUNE_API_TOKEN*? ----------------------------- -#. Log in to `neptune `_. -#. Click on your avatar (top-right part of the screen) and select **Get API Token** +#. `Log in `_ to Neptune. +#. In the upper right corner of the UI, click the avatar, and then click **Get API Token** to copy the token to the clipboard. .. image:: ../_static/images/notebooks/token.png :target: ../_static/images/notebooks/token.png :alt: image -My integration does not work, but it worked well previously. What do to? + +My integration does not work, but it worked well previously. What do I do? ------------------------------------------------------------------------ -Most likely, you restarted kernel. Here is a solution: +Most likely, you restarted the kernel. + +If that is the case, the experiments are not associated with the notebook. + +In Jupyter, click **Activate**. -#. Go to configuration (**n** button). -#. Click **Integrate**. -.. image:: ../_static/images/notebooks/integration_01.png - :target: ../_static/images/notebooks/integration_01.png +.. image:: ../_static/images/notebooks/activate_button.png + :target: ../_static/images/notebooks/activate_button.png :alt: image